Blog · 2026-09-16 · 9 min
Raw efficiency and OS footprint, measured: same models, same tasks, external verifiers, anv, jcode, codex, aider, claude and opencode, with every raw record published. Token economics, long-horizon accuracy, swarms, hallucination resistance and memory phases follow.
Direct answer
Phase 1 of our open benchmark program asks the first question: what does an agentic CLI cost as a process on your OS? Six harnesses, three gateway models, ten objective task classes, 238 runs (including an n=3 replication of the two heaviest classes) headless, auto-approved, on the same operator gateways, every completion checked by a script that fails on the untouched seed. 228 of 238 verified, zero timeouts. Later phases cover token economics, long-horizon accuracy and stability, swarms, hallucination resistance, and persistent memory. The full records are public.
01
Three questions, answered independently. Memory: peak RSS of the harness's own process, sampled from the whole process tree at 4 Hz so daemons, git children and language runtimes are counted, plus the kernel's footprint metric because macOS RSS includes shared pages and hides compressed ones. Speed: wall time and time to first output. Completion: an external verifier that fails on the pristine seed and passes on a known-good solution, a model saying it finished is never evidence.
Fairness was a constraint set before the campaign, not a footnote after it. Every harness pointed at the same gateway models (qwen-3.8-27b, deepseek-v4.1-flash, glm-5.3-flash) through its own supported provider mechanism; identical harness-neutral prompts (they say read, write, run, never a tool name); fresh workspace seeds; MCP and telemetry off. The memory ranking is quoted from a serial anchor run, because concurrent runs under memory pressure deflate RSS on macOS.
02
Median own-process RSS across the ten task classes: anv in --no-mind mode 25.6 MB (21 MB idle floor), jcode 45.0 MB, codex 97.2 MB, aider 248.9 MB, claude 432.1 MB, opencode 582.3 MB. Whole-tree totals barely move the order: anv has no auxiliary process at all in this mode, codex adds a git child, claude up to 75 MB of helpers.
Median wall time: anv 15.7 s, opencode 21.3 s, aider 22.8 s, jcode 25.1 s, claude 55.9 s, codex 68.0 s. Wall time is dominated by model latency rather than harness overhead, so read that column as indicative. Completion: anv 49/49, opencode 48/48, jcode 44/44, codex 17/17, claude 32/34, aider 38/46.
The order tracks language runtime as much as engineering: Rust harnesses (anv, jcode, codex) lead on memory; Node (opencode, claude) and Python (aider) carry floors no context management removes. That is still what a user pays, which is why we measure it rather than correcting it away.
03
Aider lost seven cells to one root cause: it has no filesystem search tool. Asked to find one module among two hundred, it read its repo-map summary and replied: “Could you add the source files to the chat so I can inspect their contents?” For the arithmetic task it wrote and applied a script, then asked the user to run it, the answer file never appeared. That is not a benchmark artifact; it is the difference between an edit assistant and an agentic harness.
Claude Code aborted one run at 843 seconds on the most invasive read task: four ~350 KB briefs against a 128K window, where its own compaction guard tripped three times in a row. The same cell passes on the 1M model. Two non-zero exits in 176 matrix runs, and one of those still produced the correct answer before exiting. The n=3 replication then classified both: claude's abort is intermittent (2 of 4 attempts); opencode's exit code was a transient with three clean follow-ups.
04
Not “the world's lightest agentic harness”, and not a verdict on agent quality. This is Phase 1 of a program, scoped to raw efficiency and footprint. Token economics (uncached tokens per completed task), long-horizon accuracy and stability, swarm throughput, hallucination resistance and persistent memory are their own phases, each with its own corpus and criteria; until they land, a memory number is a cost statement, not a capability statement.
Even inside Phase 1, the lightest claim needs four things this edition does not have: anv's --no-mind mode shipped in a public release (these numbers come from a pre-release build), a default-configuration comparison, more than six harnesses, and at least one non-macOS platform. We wrote the four gates down on the benchmarks page.
Also stated plainly: anv is our product. That is exactly why the corpus generator, measurement scripts, raw JSONL records, corpus hash and build fingerprints are public, including the failed attempts and superseded runs. Rerun it, or dispute the method, both are accounted for in the methodology.
05
The repository has the deterministic corpus generator, the three-wave campaign driver, the measurement runner and the report script: github.com/NaMan6122/Anvaya-Benchmark. The v3 report covers 238 runs (the 176-run matrix plus a 62-run n=3 replication of the two heaviest classes) on top of v2's 66, append-only, each record carrying its corpus hash and harness build fingerprint. Full criteria (fairness rules, metric definitions, claim policy) are in CRITERIA.md there and summarized at anvayahq.com/benchmarks.
Questions
Yes. We disclose it in the methodology and in every published artifact, and that is why the raw records, corpus and scripts are public. The benchmark also publishes the result we do not get to cherry-pick: with its Mind enabled, anv's tree is ~213 MB (the largest measured) so the Mind is excluded from the harness comparison as a separate, unoptimised subsystem.
Because that is the pure-harness configuration. The Mind is an experiential store whose daemon runs an ONNX embedding pipeline at request time; it is young and unoptimised, so pricing it as harness overhead would measure the feature, not the agent loop. The Mind-on configuration is documented separately (~190 MB daemon) and deferred to a Mind-optimisation round.
That is the Node/V8 runtime floor before any work happens, it costs about 530 MB to answer “reply with the word READY”. No context-management improvement changes a language runtime's baseline. The same is true in kind for Python (aider) and, in reverse, for Rust harnesses.
No statistical claim is made from it. Session-to-session noise was measured at ±6%, and differences below that are not claimed; the headline gaps here are 1.7x to 23x, which is well outside that band. Floors were replicated n=2–3. They were replicated at n=3: the codex memory spike was a one-off serial event (92-146 MB in three follow-ups), claude's abort reproduces intermittently (2 of 4 attempts), and opencode's rc=1 was a single transient that still produced the correct answer.
github.com/NaMan6122/Anvaya-Benchmark: JSONL records, corpus generator, measurement and report scripts, MIT for code and CC BY 4.0 for the data. The 483-file corpus regenerates deterministically to hash 205507698927dd67.
25.6 MB median RSS. 25 agents ran in parallel on a Core 2 Duo with 4 GB RAM. Hundreds on your machine. Zero cloud required on the Ollama path.
Requires Rust/cargo to build from source. Linux and macOS today, Windows not yet supported. Pre-1.0, public beta. Pricing TBD.