Open Benchmark Program  ·  PHASE 1 MEASURED · 2026-09-16

Benchmarks In Phases.Phase One, Measured.

Phase 1 asks what each agentic CLI costs as a process on your OS, raw efficiency and footprint, six harnesses on the same models, externally verified, 238 runs including an n=3 replication of the heaviest classes. Later phases cover token economics, long-horizon accuracy, swarms, hallucination resistance and persistent memory. Raw records are public from day one.

6phases planned, 1 measured
1030runs across the field, 10 task classes
994/1030verified completions, 0 timeouts
483deterministic corpus files, hash published

The Program

Six Phases, Measured As We Go.

One benchmark program covering what actually decides whether an agent is usable: cost, accuracy over time, coordination, truthfulness, and whether it remembers. Each phase ships with its own corpus, scripts and raw records.

PhaseFocusStatusQuestion it answers
01Raw efficiency & OS footprint →MEASUREDWhat does each harness cost as a process on your machine? Eleven harnesses, two lanes, every failure published.
02Token efficiency & the re-reading taxIN DESIGNUncached tokens per completed task, cache behavior, compounding across sessions, the economics, not just the footprint.
03Long-horizon accuracy & stabilityPLANNEDDo decisions survive compaction, turn 12, session 12? Retention, drift and degradation under forced pressure.
04Swarm & multi-agent throughputPLANNEDCoordinated agents: parallel execution rates, conflict handling, supervision overhead, cost per merged change.
05Hallucination & groundingPLANNEDUnsupported claims, fabricated APIs and files, citation accuracy, measured against ground truth, not opinion.
06Persistent memory & experience →PLANNEDDoes session N+1 cost less and know more than session N? The Anvaya Mind's compounding thesis, measured.

Feature Benchmarks

Measured On Their Own, Outside The Phases.

The phases above track program-wide questions. A feature that is new in kind gets its own benchmark, scoped to what it claims.

FeatureFocusStatusQuestion it answers
Shadow TreesVirtual agent workspaces vs git worktrees →MEASURED · macOSWhat does isolating 32 agents cost in time and disk, and where does the advantage end?

Phase 1 · Measured 2026-09-16, updated 2026-10-04

Raw Efficiency & OS Footprint.

What the harness costs as a process. Median peak RSS of the harness’s own process across ten task classes, its idle floor, median wall time, and externally verified completion — one cumulative list of every harness measured so far.

Full Phase 1 report →Measured recordsRaw records
HarnessMeasuredMedian own RSSIdle floorMedian wallVerified
1anv (--no-mind)2026-10-0423.4 MB20 MB26.2 s90/90
2jcode2026-09-1645.0 MB43 MB25.1 s44/44
3evot2026-10-0556.7 MB51.8 MB29.5 s81/81
reasonix2026-10-0465.6 MB59.8 MB37.3 s81/81
goose2026-10-0496.5 MB96 MB26.0 s81/81
codex2026-09-1697.2 MB93 MB68.0 s17/17
crush2026-10-04107.8 MB94.6 MB27.2 s81/81
pi2026-10-04129.3 MB128 MB26.2 s81/81
hermes2026-10-04201.4 MB199 MB38.5 s81/81
aider2026-10-04255.7 MB253 MB18.3 s55/81
copilot2026-10-04297.8 MB291 MB33.3 s54/54
kimi-code2026-10-04393.5 MB386 MB26.1 s81/81
claude2026-09-16432.1 MB390 MB55.9 s32/34
opencode2026-09-16582.3 MB537 MB21.3 s48/48

Sorted by median own RSS, lightest first. Gold, silver and bronze mark the podium; the sky-blue glow marks each column’s best (lowest) value across the field. One cumulative list. The 2026-10-04 rows are nine harnesses on the opencode gateway (n=3 per cell: 495 runs, 469 verified, three flash models; copilot via BYOK, 54 runs, 54 verified, two models; the Go pair — crush and reasonix — 162 runs, 162 verified, three models; evot, native arm64, 81 runs, 81 verified, three models on 2026-10-05); the 2026-09-16 rows are the four earlier incumbents on the operator’s qwen/deepseek/glm gateways (238 runs, 228 verified, n=1 plus n=3 on the two heaviest classes). Two measurement dates are provenance, not a ranking split — the aider bridge (253 vs 242 MB idle) sits inside the ±6% noise. Floors are each harness’s quiet-box anchor; aider’s 26 unverified cells are the documented model-lane refusal pattern, published, not hidden.

228 Of 238 Verified. Here Are The Eight.

Zero timeouts and zero idle-kills. The misses are capability and context results, not benchmark noise.

01aider: 7 cells, one root causeNo filesystem search tool. Asked to find one module among two hundred it read its repo-map summary and replied: “Could you add the source files to the chat?” For the arithmetic task it wrote a script, applied the edit, and asked the user to run it, the answer file never appeared.
02claude: 1 context abortOn six turns of pressure reads at a 128K window, Claude Code’s own guard aborted after 843 s: “compact within 3 turns of the previous compact, 3 times in a row.” The same cell passes on the 1M model.
03opencode, completed, exited non-zeroOne pressure_read run wrote the correct answer, then exited rc=1. Recorded as verified with clean_exit=false. Replicated at n=3: one transient in four attempts, three clean follow-ups, not a pattern.

Records · Updated 2026-09-17

Beyond Phase 1: Token, Swarm, Eval.

The measured ledger outside the harness comparison, including the experiments that lost. Every row carries its n, scope and artifact path on the records page.

All measured records →
−11.0%median uncached tokens, micro-fold (n=5, verification 19/19)
97.6%dead page tokens in live traces (251 rounds)
7.55MBswarm supervisor max RSS, N=32 soak
12/12swe-mini round 1 resolve (round 2 16/16 · round 3 8/8)

Method & Provenance

What We Measured, And How We Kept It Fair.

Three questions in Phase 1, measured independently and never blended into one score: what the CLI costs in memory, how fast it finishes the same work, and whether the work actually got done.

01Same models: same gatewaysNo vendor defaults, no per-tool model lanes. All six harnesses were pointed at the same operator gateways (qwen-3.8-27b, deepseek-v4.1-flash, glm-5.3-flash) through each tool's supported provider mechanism.
02Objective completion onlyEvery task has an external verifier that fails on the untouched seed and passes on a known-good solution. A run counts only if the verifier passes and the harness's own output shows it responded. Model self-reports never count.
03Process-tree accountingMemory is sampled from the whole process tree at 4 Hz, so a daemon, a git child or a language runtime is counted per harness rather than hidden. The ranking is quoted from a serial anchor run, parallel waves deflate RSS on macOS.
04Everything publishedCorpus generator, measurement scripts, raw JSONL records, corpus hash and build fingerprints are public. Failed runs and superseded attempts are kept, not rewritten. Disclosure: anv is our product, which is exactly why the data is open.
records/238 original runs + 495 current-field runs, append-only JSONL · github.com/NaMan6122/Anvaya-Benchmark
205507698927dd67corpus v3 hash (483 files, ~2.9 MB)
42542492c9d0a058anv build fingerprint (pre-release --no-mind build)
MIT · CC BY 4.0code · data licenses

What we claim: the world’s lightest agentic harness, measured. Across fourteen harnesses, 1030 runs and ten task classes on identical models, corpus and machine, anv in --no-mind mode has the lowest measured memory (20 MB idle floor / 23.4 MB median own-process RSS) and the lightest whole field behind it only at jcode (45 MB), evot (56.7 MB) and the Go pair (reasonix 65.6, crush 107.8). The scope is stated, not hidden: macOS/ARM, the opencode-gateway models, --no-mind mode, n=3 per cell. Two gates remain open against the strongest possible form of the claim — a non-macOS rerun and a default-configuration comparison — and both are on the roadmap. anv’s Mind is deliberately excluded from Phase 1: it is an experience subsystem measured by Phase 6 under its own criteria, not priced as harness overhead.

What Moves The Program Next.

The gates between this edition and stronger claims, and the next phase in the queue.

01Ship Phase 1's buildanv's --no-mind mode is not in the current public release; these numbers come from a pre-release build. Ship it, then re-run so every Phase 1 number reproduces from a release.
02Close Phase 1's outliersThree cells carry the remaining curiosity, codex +214% on multi-file edits and two pressure_read rc=1 events. n=3 will say whether they are stable behaviours or incidents.
03Open Phase 2 (token economics)Uncached tokens per completed task, cache-hit behavior and cross-session compounding, the second phase, now in design.
04A non-macOS rerun“Lightest” cannot be claimed from one platform. A Linux rerun with the same corpus and scripts gates any cross-platform claim.

Run Agents That Fit On Your Laptop.

25.6 MB median RSS. 25 agents ran in parallel on a Core 2 Duo with 4 GB RAM. Hundreds on your machine. Zero cloud required on the Ollama path.

Get Started →Read the FAQ

Requires Rust/cargo to build from source. Linux and macOS today, Windows not yet supported. Pre-1.0, public beta. Pricing TBD.