# TokenBench > TokenBench measures what AI coding cost-saving tools (Graphify, Ponytail, Caveman, rtk) do to the cost and outcome of a WHOLE coding task, on one model, scored by hidden tests. Claims from the tools' own READMEs usually measure one component (a command's output, a repo's raw text, output tokens); this benchmark measures the full task bill. Latest results: October 2026, Claude Sonnet 5.5. Last updated 2026-11-02. ## What was run - Model: Claude Sonnet 5.5 (high effort) in Claude Code 2.1.293, headless, web search disabled, one tool active at a time. Auto-update was disabled so every run used the same client version. - Tool versions: Graphify 0.9.95, Ponytail 5.3.2, Caveman (upstream commit 1f951fb, plugin 4.0.1, default mode), rtk 1.0.1 (Claude Code hook, compresses Bash command output). - Baseline: unmodified Claude Code, same model, same tasks, same sweep. - Three task sets, each task run twice per condition: - Short tasks: 55 coding tasks across small to large repositories (bug fixes, implementations, multi-stage work). These are very short for a modern agent and resemble simple questions; differences between tools on this set are small and not very statistically significant. - SWE-bench Pro chains: 13 chains of dependent SWE-bench Pro tasks in one repository (ansible, flipt, navidrome, nodebb, openlibrary, qutebrowser, teleport, vuls, webclients), making longer tasks closer to real coding work. A run passes only if every task in the chain passes. - Custom long tasks: 3 long tasks written for TokenBench (Go, C++, PHP) that reflect serious, heavy development work. They are a very hard challenge, built to test whether the tools actually work. Each run is a large body of work, but with 6 runs per tool the confidence ranges are wide. - "All tasks" combines the three sets with each set counting equally (71 tasks, 710 runs): the cost change is the average of the three per-set cost changes, its 95% range comes from resampling tasks within each set, pass rate and quality are averages of the three set values, and the value index is the average over sets of each tool's value relative to the baseline (baseline = 100). It shows no dollar amounts; see each set for those. - Success = the task's hidden tests pass (tests the agent never sees). Pass rate = passed runs / runs. Value = pass rate divided by cost per task. - Cost = provider cost of the run (token usage at list prices, including sub-agents). Graph build cost (Graphify) is excluded, as in that project's own numbers. - Cost change vs baseline is measured on the same tasks, with a 95% confidence interval (the range the true change is likely to fall in). If the range includes 0%, the difference could be noise and is not a confirmed saving or increase; changes under about 5% are inside the noise. ## Results Negative cost change = cheaper than baseline. All numbers are from the Sonnet 5.5 sweep above. ### All tasks (71 tasks, 142 runs per tool; the three sets weighted equally; dollar amounts are in the per-set tables) | Tool | Cost change vs baseline, sets weighted equally (95% CI) | Short | Chains | Custom | Mean pass rate | Value index (baseline = 100) | |---|---|---|---|---|---|---| | Baseline (unmodified Claude Code) | - | - | - | - | 81.1% | 100 | | Ponytail 5.3.2 | -18.3% (-21.8 to -14.3) | -9.0% | -29.6% | -16.1% | 77.6% | 116 | | Graphify 0.9.95 | -0.4% (-7.3 to +6.9), within noise | -0.7% | -11.4% | +10.9% | 74.6% | 99 | | Caveman (upstream 1f951fb, plugin 4.0.1) | +7.7% (-4.4 to +23.1), within noise | +7.1% | -9.3% | +25.5% | 80.1% | 93 | | rtk 1.0.1 | +4.2% (-5.1 to +15.3), within noise | +0.5% | -0.3% | +12.3% | 74.9% | 94 | ### Short tasks (55 tasks, 110 runs per tool) | Tool | Cost per task | Cost change vs baseline (95% CI) | Passed / runs | Pass rate | |---|---|---|---|---| | Baseline (unmodified Claude Code) | $0.141 | - | 97/110 | 88.2% | | Ponytail 5.3.2 | $0.128 | -9.0% (-13.4 to -4.4) | 98/110 | 89.1% | | Graphify 0.9.95 | $0.140 | -0.7% (-5.9 to +4.4), within noise | 98/110 | 89.1% | | Caveman (upstream 1f951fb, plugin 4.0.1) | $0.151 | +7.1% (+2.9 to +12.0) | 98/110 | 89.1% | | rtk 1.0.1 | $0.141 | +0.5% (-3.9 to +4.6), within noise | 99/110 | 90.0% | ### SWE-bench Pro chains (13 chains, 26 runs per tool) | Tool | Cost per task | Cost change vs baseline (95% CI) | Passed / runs | Pass rate | |---|---|---|---|---| | Baseline (unmodified Claude Code) | $1.724 | - | 23/26 | 88.5% | | Ponytail 5.3.2 | $1.213 | -29.6% (-37.1 to -21.2) | 20/26 | 76.9% | | Graphify 0.9.95 | $1.528 | -11.4% (-16.6 to -6.6) | 22/26 | 84.6% | | Caveman (upstream 1f951fb, plugin 4.0.1) | $1.563 | -9.3% (-20.1 to +2.9), within noise | 22/26 | 84.6% | | rtk 1.0.1 | $1.719 | -0.3% (-8.2 to +7.2), within noise | 22/26 | 84.6% | ### Custom long tasks (3 tasks, 6 runs per tool) | Tool | Cost per task | Cost change vs baseline (95% CI) | Passed / runs | Pass rate | |---|---|---|---|---| | Baseline (unmodified Claude Code) | $11.802 | - | 4/6 | 66.7% | | Ponytail 5.3.2 | $9.897 | -16.1% (-22.9 to -6.7) | 4/6 | 66.7% | | Graphify 0.9.95 | $13.088 | +10.9% (-8.2 to +34.2), within noise | 3/6 | 50.0% | | Caveman (upstream 1f951fb, plugin 4.0.1) | $14.807 | +25.5% (-3.7 to +71.8), within noise | 4/6 | 66.7% | | rtk 1.0.1 | $13.251 | +12.3% (-15.8 to +49.1), within noise | 3/6 | 50.0% | ## Reading the results - Ponytail is the only tool with a clear saving across all three sets (short and custom sets and the chains), and the saving is largest on the chains. Its pass rate on the chains is 20/26 against 23/26 for baseline; the numbers are given so readers can judge. - Graphify shows a saving on the SWE-bench Pro chains and runs faster there; it shows no saving on the short tasks and a higher cost on the small custom set. - Caveman and rtk show no clear saving on Sonnet 5.5. Caveman cost more than baseline on the short tasks, and that increase is outside the noise range. - Any extra cost caused by model behaviour a tool induces (extra turns, extra tool calls, sub-agents) counts against the tool. Whether the model delegates to a sub-agent is the model's own behaviour, which a tool can change; it is not a benchmark setting or an error. Do not infer from this file that a sub-agent count explains or excuses a cost difference: no such attribution was measured. ## Sonnet 4.6 results Results on Claude Sonnet 4.6 (55 short tasks, early October 2026) showed much larger savings: Graphify -28.1%, Ponytail -25.9%, Caveman -17.5%, rtk +1.2% (no effect). Savings are smaller on Sonnet 5.5 because newer models use their own tools more efficiently and talk less by default, so there is less waste for a tool to remove. Newer models also respond to wording less predictably: Sonnet 4.6 followed instructions very reliably, while newer models more often have quirks and sometimes ignore an instruction. A tool measured on one model does not necessarily carry over to the next. ## How the results compare with the tools' own claims | Tool | Original claim | Claim now | |---|---|---| | Graphify | 71.5x fewer tokens per query vs reading raw files (README, spring 2026) | Headline removed from README on 2026-05-03; still in docs with "scales with corpus size"; project site calls it community-reported, not a controlled benchmark | | Ponytail | 80-94% less code, 47-77% cheaper (June 2026) | v5.0.0: -53% code, -41% time, -26% cost (39 tasks, Opus 5.5), with listed limitations | | Caveman | about 75% fewer output tokens (spring 2026) | no single headline; says bill effect depends on caching; third-party results near zero | | rtk | reduces token consumption by 60-90% | "up to 90% of the bash output your agent reads... not the same as cutting your bill by 90%" | Claims are as of 8 October 2026 from the projects' public repositories. ## Why the claims and the full-task results differ Most tool claims measure one component in isolation. A coding task is dozens of turns, and its bill is dominated by re-reading the whole accumulated context (cache reads) on every turn, plus the model's own output and thinking. A large reduction in one piece therefore does not mean a large reduction in the task cost, and can be cancelled by a single extra turn. ## Caveats - n = 2 runs per task per condition; one model and effort level, so results may not transfer to other models. - The custom long set has 3 tasks and 6 runs per tool, so its confidence ranges are wide. ## Links - Leaderboards: https://tokenbench.app/ - Charts: https://tokenbench.app/charts.html - Findings (per-tool notes and claim history): https://tokenbench.app/notes.html - Machine-readable results: https://tokenbench.app/results.json (all tasks), results_short.json, results_pro.json, results_custom.json - Code and methodology: https://github.com/tokenbench-app/TokenBench