Evidence · agent A/B runs · September 2026
Does a knowledge graph make a coding agent cheaper, or better?
Six experiments, one codebase, one model. Each pairs a control arm (the repository alone) with a treatment arm (the repository plus its OpenContext graph, delivered into the agent's context). Every figure below is measured from the run transcripts or marked as an estimate. Nothing is pooled across tasks.
Setup
Both arms start from the same commit (44d5d6c) of the OpenContext repository, in separate worktrees with dependencies preinstalled, running Claude Opus 5 headless with identical tools and a frozen prompt. The treatment arm receives a SessionStart primer plus a UserPromptSubmit recall injection: up to three quoted graph entities relevant to the task (for the passport task: checksum-backed identifier detection, redact-by-default, never-flag names/emails/phones) and a pointer to the privacy layer. The control arm receives the frozen prompt alone. Diffs are judged blind by a separate Opus 5 run with a sealed random A/B assignment, scored on correctness, security, completeness, and convention.
“Follow this repo's privacy stance: what is deliberately never flagged, how high-severity hits are handled by default, and how false positives are kept down for identifier detectors. Extend the privacy suite; typecheck and tests must pass.” The graph holds all three stance decisions.
“On top of the existing per-IP credential budget, following the repo's security stance for auth responses (no account enumeration). Extend the auth suite.” The graph held one auth entity at the time; used for AB1, AB2, and AB4.
Activated pairs
The treatment arm received the graph and acted on it: the transcript shows real opencontext get calls on the injected paths. These three rows are the basis for every headline number.
| Pair | Arm | Cost (USD) | Δ cost | Cache-read tokens | Tool calls / turns | Wall (s) | Judge C·S·Cp·Cv | Merge |
|---|---|---|---|---|---|---|---|---|
| AB3 headless claude -p · hooks live | graph | 2.46 | −20.6% | 1,917,401 | , / 32 | 460 | 4 · 4 · 4 · 5 | graph |
| control | 3.10 | 3,072,204 | , / 34 | 404 | 3 · 4 · 4 · 4 | |||
| AB3r2 subagent harness · injection prepended | graph | 2.50 est. | −17.3% | 2,504,466 | 31 / 27 | 570 | 4 · 4 · 3 · 4 | control |
| control | 3.02 est. | 3,737,766 | 37 / 34 | 550 | 4 · 5 · 5 · 4 | |||
| AB3r3 subagent harness · injection prepended | graph | 2.06 est. | −23.9% | 2,251,891 | 30 / 24 | 464 | 4 · 4 · 3 · 4 | control |
| control | 2.70 est. | 3,106,207 | 36 / 30 | 550 | 4 · 5 · 4 · 4 |
Judge columns: correctness · security · completeness · convention, 1–5. “est.” costs are computed from transcript usage at Opus 5 list prices (5 / 25 / 0.5 / 10 USD per MTok), a method that reproduces AB3's recorded costs to the cent. Wall time ran concurrently on one machine and is not a clean signal.
What the judge said
The graph arm enforced overlap resolution in code rather than sort order, and its tests covered both the false-positive stance and the never-flagged stance. The decisive convention point was the content of an injected decision.
Control proved the default disposition end to end (preflight hold and write-boundary redact) and fixed a pre-existing placeholder false positive. The graph arm's rule was correct but its tests were narrower.
“Both rules are essentially the same rule, and both are correct.” Control won on the parts that are not the regex: end-to-end verification and recording the stance in DESIGN.md and the CHANGELOG.
Null treatments, kept on the record
In these pairs the graph was available but the agent never opened it, so the cost differences are run-to-run variance and prove nothing about graph value. They did prove the delivery mechanism (AB2's SessionStart hook fired) and motivated the switch from invitation to injection.
| Pair | Task | Graph arm | Control arm | Graph calls | Why null |
|---|---|---|---|---|---|
| AB1 | lockout | $5.09 · 61 turns · 687 s | $7.31 · 74 turns · 1,309 s | 0 | Link block present in AGENTS/CLAUDE/README, never followed. Control also chose a larger verification scope. |
| AB2 | lockout | $5.66 · 64 turns · 692 s | $6.90 · 77 turns · 789 s | 0 | Primer delivered by hook (proven in transcript); agent went to git log and grep instead. Judge order flipped vs AB1. |
| AB4 | lockout · 3 arms | thin $5.60 · deepened $6.03 est. | $5.80 est. | 0 | Injection prepended but no entity followed. Judge preferred the deepened-graph diff in 2 of 3 comparisons, on docs and test hygiene. |
Caveats, in the order they matter
- N = 1 per arm within a pair; the three activated pairs are the same task. Read them together as N = 3, not as three tasks.
- Task and mechanism changed together between AB2 and AB3, so the delta is “graph content in context on a graph-covered task,” not injection alone.
- AB3r2 and AB3r3 ran as subagents: no hooks fire, so the same primer and recall text were prepended to the prompt, and output-token accounting differs from headless runs. Compare tokens within a family only.
- Arms ran concurrently on one machine, seconds apart. Wall time is reported but not relied on.
- Quality is a split decision. Correctness was equal or better with the graph in every pair; completeness and documentation favored the control arm in two of three. The saving is in tokens read, not in quality.
- The overview calculator is a hypothetical scenario for comparable agent tasks. It assumes a 20% reduction on each run; this small sample does not establish savings for other work. It excludes graph creation, maintenance, and human time.
What is next
A frozen prospective protocol is prepared: nine coding pairs across three tasks (an architectural constraint, a cross-entity relationship, and a thin-coverage repair) plus an irrelevant-prompt control, on native Claude Code inside an OS sandbox with withheld acceptance tests and blind diff review. The estimated spend is $60–120. It runs when it is approved, and the results land here, whichever way they go.