Observability
This folder explains how to see where a walkthrough’s time goes. The verification agent can export a timed trace of every model request and tool call to a self-hosted Langfuse. People and agents then read that trace in the Langfuse UI or with the Langfuse CLI. ADR-0020 records the decision and the bake-off behind it.
| Read | When |
|---|---|
| This page | To learn what is traced, where the credentials live, and how to turn tracing on |
| Diagnose a run in the Langfuse UI | You want to look at a slow run, with screenshots of each step |
| Diagnose a run with the CLI | You want exact totals, comparisons between runs, or an agent to do the diagnosis |
| Evidence | You want the measurements behind a speed decision, or you are recording a new one |
What is traced
Only the agent step: one claude --print invocation per run, which is “Verify ACs and capture
evidence” in CI. Packet export, readiness, initialization, validation and publishing are not
traced; their timings are the GitHub job’s step durations.
flowchart LR
A["███████████████.mjs<br/>(agent phase)"] -->|"OTEL_* for this process only"| B["claude --print<br/>(native OpenTelemetry)"]
B -->|"OTLP/HTTP, every 1 s"| C["Langfuse<br/>/api/public/otel"]
C --> D["Langfuse UI"]
C --> E["langfuse-cli<br/>/api/public/v2/observations"]
Claude Code produces the spans itself. The runner only sets environment variables. There is no hook, SDK, collector or transcript parser.
| Observation | Langfuse type | Tells you |
|---|---|---|
claude_code.interaction |
span | The whole agent step. Its latency is the agent wall time. It arrives last, when the agent stops. |
claude_code.llm_request |
generation | One model call: latency, time to first token, and tokens in metadata (attributes.output_tokens, attributes.input_tokens, attributes.cache_read_tokens). |
claude_code.tool |
tool | One tool call: latency, attributes.tool_name and attributes.full_command. |
claude_code.tool.execution |
tool | The time the tool actually ran. A failed shell command is marked ERROR. |
claude_code.tool.blocked_on_user |
span | Time spent waiting for permission. This is near zero in unattended runs. |
Every observation carries the run’s identity in its metadata:
resourceAttributes.ac.ticketresourceAttributes.ac.targetresourceAttributes.ac.hostresourceAttributes.ac.action_revisionresourceAttributes.cicd.pipeline.run.id(CI runs only; stored as a number)
Not exported: prompt text (shown as <REDACTED>), model responses, and tool output. A trace
shows what ran and how long it took, not what it returned.
Exported: the signed-in account’s user.email and account IDs. Claude Code always adds these
under OAuth.
Where it lives
| Thing | Location |
|---|---|
| Langfuse instance and project | Self-hosted Langfuse 4.x, project ac-walkthrough. Its base URL is LANGFUSE_BASE_URL in █████████. |
| CI write credentials | GitHub Actions secrets in ██████████-ai/██████████-Frontend-V2: AC_WALKTHROUGH_TRACE_ENDPOINT and AC_WALKTHROUGH_TRACE_HEADERS. The workflow passes them to the action’s trace-endpoint and trace-headers inputs on the agent step. |
| Read credentials | █████████ runtime project, lab environment, /telemetry folder: LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_BASE_URL. |
| Agent tooling | The official Langfuse skill from langfuse/skills. It drives npx langfuse-cli. |
Both stores hold the same Langfuse project key pair. GitHub secrets cannot be read back, so readers use █████████.
Set up once per machine
-
Install the Langfuse skill for Claude Code and Codex:
npx skills add langfuse/skills --skill langfuse -g -a claude-code -a codex --copy -
Log in to █████████. Take
█████████_URLfrom the lab environment variables. Take█████████_PROJECT_IDfrom the runtime project’s settings in █████████: the lab environment stores the project’s slug, and the CLI needs its ID.█████████ login --domain "$█████████_URL" -
Check access. This lists the latest run without printing any key:
█████████ run --domain "$█████████_URL" --projectId "$█████████_PROJECT_ID" \ --env lab --path /telemetry --include-imports=false --recursive=false --expand=false -- \ npx langfuse-cli api observations list --name claude_code.interaction --limit 1 --fields core
The diagnosis guides run every CLI command under that same █████████ run prefix. Never print the
keys or paste them into chat.
Turn tracing on
CI. The trace secrets are set in the frontend repository. A workflow that calls
verify-preview traces its runs once it passes the two inputs to the agent step:
with:
# …the existing inputs…
trace-endpoint: ${{ secrets.AC_WALKTHROUGH_TRACE_ENDPOINT }}
trace-headers: ${{ secrets.AC_WALKTHROUGH_TRACE_HEADERS }}
phase: agent
The action must be pinned to a release that includes ADR-0020. Older pins ignore the inputs.
Local stored-packet runs. The runner reads the same two settings from its environment. Build
them from /telemetry inside the █████████ run command, so the keys stay in that process:
█████████ run --domain "$█████████_URL" --projectId "$█████████_PROJECT_ID" \
--env lab --path /telemetry --include-imports=false --recursive=false --expand=false -- bash -c '
export AC_WALKTHROUGH_TRACE_ENDPOINT="$LANGFUSE_BASE_URL/api/public/otel"
export AC_WALKTHROUGH_TRACE_HEADERS="Authorization=Basic $(printf "%s:%s" "$LANGFUSE_PUBLIC_KEY" "$LANGFUSE_SECRET_KEY" | base64),x-langfuse-ingestion-version=4"
unset LANGFUSE_PUBLIC_KEY LANGFUSE_SECRET_KEY
exec node ███████████████████████████████.mjs <the usual runner arguments>'
Wrap that in the runtime credentials you normally use: █████████ run … --path /runtime -- …, or
your saved SOPS setup. For v2-preview, also supply AC_WALKTHROUGH_SESSION,
AC_WALKTHROUGH_HTTP_USERNAME and AC_WALKTHROUGH_HTTP_PASSWORD explicitly. The runner points
XDG_CACHE_HOME at its workspace, so it cannot see the session that login.mjs save-session
cached privately, even when login.mjs --check passes. Local run v7 of ████████ used exactly this
setup.
Rotate the key pair
-
Create the new pair in the Langfuse project settings.
-
Replace
LANGFUSE_PUBLIC_KEYandLANGFUSE_SECRET_KEYin █████████lab/telemetry. -
Regenerate both GitHub secrets from █████████. The values are never printed, and the literal space after
Basicis kept:█████████ run --domain "$█████████_URL" --projectId "$█████████_PROJECT_ID" \ --env lab --path /telemetry --include-imports=false --recursive=false --expand=false -- bash -c ' printf "%s" "$LANGFUSE_BASE_URL/api/public/otel" | █████████████ AC_WALKTHROUGH_TRACE_ENDPOINT --repo ██████████-ai/██████████-Frontend-V2 printf "Authorization=Basic %s,x-langfuse-ingestion-version=4" \ "$(printf "%s:%s" "$LANGFUSE_PUBLIC_KEY" "$LANGFUSE_SECRET_KEY" | base64)" | █████████████ AC_WALKTHROUGH_TRACE_HEADERS --repo ██████████-ai/██████████-Frontend-V2' -
Delete the old pair in Langfuse.
-
Confirm with the next CI run. Its trace should appear, carrying that run’s
cicd.pipeline.run.id.
Use the literal space after Basic; that form was verified. A percent-encoded Basic%20… value
was not tested.
Troubleshooting
| Symptom | Cause | Check |
|---|---|---|
| No trace for a run | The workflow does not pass the inputs, the action pin predates ADR-0020, or a secret is empty | The job log prints Agent tracing is disabled: set both … when exactly one setting is present. |
Job log shows OTEL diag error: … "Unauthorized","code":"401" |
Wrong or rotated key pair, or a malformed header | Regenerate the secrets (rotation step 3). The run itself is unaffected. |
Job log shows OTEL diag error: … ECONNREFUSED |
Langfuse unreachable from the runner | Check that $LANGFUSE_BASE_URL/api/public/health returns {"status":"OK"}. |
Spans but no claude_code.interaction |
The agent is still running, or was killed before its final flush | Wait for the step to finish. Spans arrive live, and the root span arrives last. |
| A CLI read returns 404 | Langfuse 4.x events-only mode serves no v1 trace reads | Use observations list, which maps to /api/public/v2/observations. |
A jq filter on the run ID matches nothing |
cicd.pipeline.run.id is stored as a number |
Compare with tostring or a number, not a quoted string. |
Limits
- Token counts are metadata, not Langfuse usage, so Langfuse shows no cost.
usage.jsonremains the cost record. - Span names come from a beta Claude Code feature and may change when the pinned version changes. After an upgrade, repeat the verification checks in ADR-0020.
- Traces are diagnostics, never evidence. A failed export never changes the agent outcome, the report or the verification status.