Evidence · agent A/B runs · September 2026
Does a knowledge graph make a coding agent cheaper, or better?
Agents using OpenContext completed the same coding task at lower model cost and with fewer cache-read tokens across three paired comparisons. Explore the measurements, setup, and code review results below.
Averages from three comparisons of the same task. Costs combine recorded and transcript-estimated usage; the per-run figures and review findings are below.
Setup
Both arms start from the same commit (44d5d6c) of the OpenContext repository, in separate worktrees with dependencies preinstalled, running Claude Opus 5 headless with identical tools and a frozen prompt. The treatment arm receives a SessionStart primer plus a UserPromptSubmit recall injection: up to three quoted graph entities relevant to the task (for the passport task: checksum-backed identifier detection, redact-by-default, never-flag names/emails/phones) and a pointer to the privacy layer. The control arm receives the frozen prompt alone. Diffs are judged blind by a separate Opus 5 run with a sealed random A/B assignment, scored on correctness, security, completeness, and convention.
“Follow this repo's privacy stance: what is deliberately never flagged, how high-severity hits are handled by default, and how false positives are kept down for identifier detectors. Extend the privacy suite; typecheck and tests must pass.” The graph holds all three stance decisions.
“On top of the existing per-IP credential budget, following the repo's security stance for auth responses (no account enumeration). Extend the auth suite.” The graph held one auth entity at the time; used for AB1, AB2, and AB4.
Measured comparisons
Each comparison pairs an agent using OpenContext with an agent working from the repository alone. These three comparisons supply the cost and token averages above.
| Pair | Arm | Cost (USD) | Δ cost | Cache-read tokens | Tool calls / turns | Wall (s) | Judge C·S·Cp·Cv | Merge |
|---|---|---|---|---|---|---|---|---|
| AB3 headless claude -p · hooks live | graph | 2.46 | −20.6% | 1,917,401 | , / 32 | 460 | 4 · 4 · 4 · 5 | graph |
| control | 3.10 | 3,072,204 | , / 34 | 404 | 3 · 4 · 4 · 4 | |||
| AB3r2 subagent harness · injection prepended | graph | 2.50 est. | −17.3% | 2,504,466 | 31 / 27 | 570 | 4 · 4 · 3 · 4 | control |
| control | 3.02 est. | 3,737,766 | 37 / 34 | 550 | 4 · 5 · 5 · 4 | |||
| AB3r3 subagent harness · injection prepended | graph | 2.06 est. | −23.9% | 2,251,891 | 30 / 24 | 464 | 4 · 4 · 3 · 4 | control |
| control | 2.70 est. | 3,106,207 | 36 / 30 | 550 | 4 · 5 · 4 · 4 |
Judge columns: correctness · security · completeness · convention, 1–5. “est.” costs are computed from transcript usage at Opus 5 list prices (5 / 25 / 0.5 / 10 USD per MTok), a method that reproduces AB3's recorded costs to the cent. Wall time ran concurrently on one machine and is not a clean signal.
What the judge said
The graph arm enforced overlap resolution in code rather than sort order, and its tests covered both the false-positive stance and the never-flagged stance. The decisive convention point was the content of an injected decision.
Control proved the default disposition end to end (preflight hold and write-boundary redact) and fixed a pre-existing placeholder false positive. The graph arm's rule was correct but its tests were narrower.
“Both rules are essentially the same rule, and both are correct.” Control won on the parts that are not the regex: end-to-end verification and recording the stance in DESIGN.md and the CHANGELOG.
Earlier experiments
These earlier experiments informed how OpenContext delivers knowledge to agents. Their transcripts show no graph reads, so they are documented separately from the cost and token averages above.
| Pair | Task | Graph arm | Control arm | Graph calls | Observations |
|---|---|---|---|---|---|
| AB1 | lockout | $5.09 · 61 turns · 687 s | $7.31 · 74 turns · 1,309 s | 0 | Link block present in AGENTS/CLAUDE/README, never followed. Control also chose a larger verification scope. |
| AB2 | lockout | $5.66 · 64 turns · 692 s | $6.90 · 77 turns · 789 s | 0 | Primer delivered by hook (proven in transcript); agent went to git log and grep instead. Judge order flipped vs AB1. |
| AB4 | lockout · 3 arms | thin $5.60 · deepened $6.03 est. | $5.80 est. | 0 | Injection prepended but no entity followed. Judge preferred the deepened-graph diff in 2 of 3 comparisons, on docs and test hygiene. |
How to read the results
- The three comparisons repeat the same task on one codebase and model. They do not measure performance across three different tasks.
- Task and mechanism changed together between AB2 and AB3, so the delta is “graph content in context on a graph-covered task,” not injection alone.
- AB3r2 and AB3r3 ran as subagents: no hooks fire, so the same primer and recall text were prepended to the prompt, and output-token accounting differs from headless runs. Compare tokens within a family only.
- Arms ran concurrently on one machine, seconds apart. Wall time is reported but not relied on.
- Quality is a split decision. Correctness was equal or better with the graph in every pair; completeness and documentation favored the control arm in two of three. The saving is in tokens read, not in quality.
- The overview calculator is a hypothetical scenario for comparable agent tasks. It assumes a 20% reduction on each run; this small sample does not establish savings for other work. It excludes graph creation, maintenance, and human time.
What is next
The next evaluation is designed to cover more tasks: architectural constraints, relationships across components, and areas with less existing context. It will compare results using independent acceptance tests and blind code review, with per-run measurements published alongside the findings.