Tracing

Export traces from a self-hosted server to an OpenTelemetry collector, so a slow request becomes a waterfall instead of a stopwatch.

When a request through your server is slow, the useful question is which part was slow: building the executor stack, initializing plugins, a storage query, or the upstream integration answering. The server already produces spans for all of that — pointing it at a collector is what makes them readable.

Export is off by default and stays off until you set an endpoint. With none set there is no exporter and no buffer, and nothing leaves the process.

Turn it on

Set the collector’s base URL. The server appends /v1/traces.

docker run -d \
  -p 4788:4788 \
  -v executor-data:/data \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.local:4318 \
  ghcr.io/usefulsoftwareco/executor-selfhost:latest

Traces are sent as OTLP over HTTP with a JSON payload, which every collector accepts — the OpenTelemetry Collector, Grafana Alloy, Jaeger, or a hosted backend. Port 4318 is the OTLP/HTTP convention, but the URL is whatever you point it at.

Variable	Default	Purpose
`OTEL_EXPORTER_OTLP_ENDPOINT`	unset	Collector base URL. Turns on trace export; `/v1/traces` is appended.
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`	unset	Full traces URL, used as-is. Overrides the base for this signal.
`OTEL_EXPORTER_OTLP_HEADERS`	unset	`key=value,key2=value2` headers, for a collector that needs authentication.
`EXECUTOR_OTEL_EXPORT_LOGS`	`false`	Also export logs. A separate opt-in — logs carry more than timings.
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`	unset	Full logs URL. Required if only the traces endpoint is set.

Traces report under the service name executor-selfhost. On boot the server logs the endpoint it will export to, so a missing line means export is off.

For a hosted backend, put the credential in the headers:

-e OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example.com \
-e OTEL_EXPORTER_OTLP_HEADERS='authorization=Bearer <token>'

Using motel locally

motel is a local OTLP store and viewer: one command, a SQLite file, no Docker and no account. It is the quickest way to look at a trace on your own machine.

It needs Bun, and listens on port 27686:

bunx @kitlangton/motel

That starts the ingest server and opens the terminal UI. For ingest without the UI, run bunx @kitlangton/motel server.

If Executor runs directly on the same machine, point it at motel:

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:27686

From a container, motel is on the host, not in the container’s network — and it binds loopback by default, so it also has to listen on an address the container can reach:

# start motel so it accepts connections from outside the host
MOTEL_OTEL_HOST=0.0.0.0 bunx @kitlangton/motel server

docker run -d \
  -p 4788:4788 \
  -v executor-data:/data \
  --add-host host.docker.internal:host-gateway \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:27686 \
  ghcr.io/usefulsoftwareco/executor-selfhost:latest

Reading a trace

A single API request looks like this — each line a span, indented under its parent, with the time it took:

http.server GET                        31ms
  executor.stack.http.resolve          28ms
    executor.stack.build               28ms
      executor.stack.scoped_executor   27ms
        executor.plugins.init          16ms
        executor.stack.create_executor  2ms
        executor.subject.touch          7ms
          fumadb.subject.findFirst      2ms
          fumadb.subject.create         5ms
  fumadb.integration.findMany           1ms

The indentation is where the answer lives. Time held by http.server but not by any child is time spent outside the instrumented code — the network in front of the server, or a tunnel. Time inside a single child is the phase to look at next.

Requests arriving with a traceparent header continue that trace rather than starting a new one, so a request through a proxy or tunnel keeps one trace id from end to end.