Effort level comparison: ████████ (2026-10-05)
This is the evidence behind #116: which reasoning effort the verification agent should run at. The raw per-run numbers are in 2026-10-05-effort-level-att-5933.json. They were read from Langfuse with the queries in Diagnose a run with the CLI.
Result: high cut the local agent step by about 23% (365 s against 473 s on average) and
produced equivalent verdicts. medium was faster still, but it omitted required evidence.
high is pinned.
Why effort matters
-
Model time is output generation. Every run writes at about 104–114 output tokens/s, and the model takes 90–93% of the agent step. Tools take 28–33 s.
-
Most output is hidden reasoning. Two CI runs show it most clearly:
CI run Visible output (text + tool inputs) Reported output ███████████3,298 + 27,814 characters, about 7.8k tokens 31,356 tokens ███████████4,364 + 26,784 characters, about 7.8k tokens 36,697 tokens That makes about 75–80% of the output hidden reasoning. The
███████████transcript holds 23thinkingblocks whose content is hidden; only their signatures are stored. -
The effort level came from a personal settings file. The runner started
claude --print --model sonnetwithout--effort. The self-hosted runner’s account had"effortLevel": "xhigh"in~/.claude/settings.json. Claude Code applies that setting when nothing explicit sets the level (resolution order). The default for Sonnet 5 ishigh.
Method
All runs use the same ticket and the same inputs:
| Input | Value |
|---|---|
| Ticket | ████████ |
| Target | v2-preview |
| Implementation PR | #915 |
| Code reference | 877251ce055c5a694e840886faa18516e6c5b5a6 |
| Runtime revision | ead4b6e (v20.1.0); the first CI run used fa890e9 |
| Model | claude-sonnet-5 (--model sonnet) |
| Claude Code | 2.1.241 |
-
Local runs are traced stored-runner runs on the self-hosted Mac. Credentials come from █████████
lab/runtime,/previewand/telemetry. Each run setsCLAUDE_CODE_EFFORT_LEVELexplicitly, except the first, which inheritedxhigh. -
Override check. Before the runs, the same reasoning prompt was sent at three levels. All three answered correctly, and output tokens fell with effort, so the variable overrides the inherited
xhigh:Level Output tokens Time Inherited xhigh583 9.5 s medium363 7.5 s low7 4.8 s -
Verdicts come from each report’s AC1
assessmentblock. Render outcomes come from the agent transcript.
Results
| Effort | Run | Langfuse trace | Agent | Model | Output tokens | Requests | Verdict | Outcome |
|---|---|---|---|---|---|---|---|---|
xhigh (inherited) |
local | ████████████████████████████████ |
505.9 s | 465.8 s | 41,164 | 51 | AC1 pass, mixed (E1,E2,T1,T2) | delivered (v7) |
xhigh |
local, back-to-back | ████████████████████████████████ |
440.4 s | 399.0 s | 36,868 | 37 | AC1 pass, mixed (E1,E2,T1,T2,T3) | delivered (v13) |
high |
local | ████████████████████████████████ |
372.7 s | 332.4 s | 31,383 | 37 | AC1 pass, mixed (E1,E2,T1,T2,T3) | delivered (v10) |
high |
local, back-to-back | ████████████████████████████████ |
357.6 s | 319.8 s | 29,608 | 36 | AC1 pass, mixed (E1,E2,T1,T2) | delivered (v14) |
medium |
local | ████████████████████████████████ |
268.6 s | 242.0 s | 20,919 | 35 | AC1 pass, mixed (E1,E2,T1,T2) | rejected: no run-inspection video |
For reference, two CI runs at inherited xhigh on GitHub-triggered self-hosted runs:
| CI run | Langfuse trace | Agent | Model | Output tokens |
|---|---|---|---|---|
███████████ |
████████████████████████████████ |
446.0 s | 403.6 s | 36,697 |
███████████ |
████████████████████████████████ |
372.0 s | 336.3 s | 31,356 |
Excluded: one high run (trace ████████████████████████████████) failed for an
environment reason that has nothing to do with effort:
- At 10:45 UTC,
session.mjs openreturnedAUTH_APP_UNAVAILABLEtwice. - The agent then spent 105 s searching the disk for session files (
find ~/Library …, thenfind / -maxdepth 6 …). - The run ended with
completed capture receipt is incomplete.
The next run, seven minutes later, authenticated normally.
Reading
highagainstxhigh(local runs only, same machine):highaverages 365 s, against 473 s forxhigh: −23%, about 108 s per run.- Output tokens average 30.5k against 39.0k: −22%.
- Both
highruns are faster than bothxhighruns.
- Verdict equivalence. Every delivered report gives AC1 pass with mixed evidence: two
screenshots, two or three code references, and the run-inspection video. The number of code
references varies within both levels. A reviewer compares the
highdrafts v10 and v14 with thexhighdraft v13 in the Drive store, under████████/stages/20-ac-walkthrough/runs/. mediumis rejected. It was the fastest by far, but validation stopped it:current visual section AC1 requires one screenshot-bounded run-inspection video. Lower effort traded away correctness, not just time.- The first render fails at every level. Every compared run hit V10
(
block ::: technical-evidence is outside the canonical order) on its first render. That is tracked separately in #117.
Acceptance: the pinned runner
After the pin (--effort high, plus dropping an ambient CLAUDE_CODE_EFFORT_LEVEL), one traced
local run used the branch at f523707. It was started on purpose with an ambient
CLAUDE_CODE_EFFORT_LEVEL=low, while the personal "effortLevel": "xhigh" was still in
settings.
| Run | Langfuse trace | Agent | Model | Output tokens | Requests | Verdict | Outcome |
|---|---|---|---|---|---|---|---|
Pinned high |
████████████████████████████████ |
387.5 s | 346.6 s | 32,250 | 42 | AC1 pass, mixed (E1,E2,T1,T2,T3) | delivered (v15) |
- The pin held. The run matches the earlier
highruns: 29.6–31.4k output tokens and 358–373 s. It does not matchlow, and it does not match localxhigh(36.9–41.2k tokens, 440–506 s). - Run times for ████████ on this machine. The three
highruns average 373 s, against 473 s for the twoxhighruns: −21%, about 100 s per run. - V10 again. The first render failed V10, as in every other run (#117).
Limits
- Few runs, one ticket. Two runs per compared level, all on ████████, which has one AC. A ticket with more ACs, or more judgment, could react differently to lower effort. Recheck after a change of model or of Claude Code version.
- Local and CI absolute times differ. CI
xhighruns took 372–446 s. Compare like with like. - Personal Serena configuration. Local runs also received failures from it (
Too many consecutive read calls without using symbolic tools), because the runner account’s personal configuration reaches the agent. It affects every level equally and is out of scope here.