Q
Shadow eval vs A/B testing?
Same spirit, zero user impact: historical queries replay against candidates offline. Promote on measured wins.
Glossary · Shadow Evaluation
Shadow Evaluation.
Canary deploys for knowledge.
Direct answer
Shadow eval scores candidate knowledge against real historical queries in parallel with live serving (measured, never served) promoting only what beats baseline. UNKNOWN is never treated as zero; absence of evidence stays absence, not failure. It is the final gate between extraction ('this might help') and serving ('this earns tokens every turn'), and the mechanism that lets memory systems learn aggressively without ever regressing answers.
In Anvaya
How We Implement It.
01Shadow EvaluationAnvaya keeps candidate knowledge in a shadow namespace and scores it against live history in an offline comparison harness before any promotion; the experience-benchmark protocol's S0–S4 fixtures define the suite.
Questions
Asked About Shadow Evaluation.
Q
What is the S0–S4 protocol?
Anvaya's experience benchmark fixtures: fail→fix headless, denial handling, tool-only exploration, producer-off baselines. The protocol is published; knowledge-regret and injection-precision report UNKNOWN until the shadow harness runs live.
Q
Why not serve-then-measure?
Serving unproven memory taxes every turn's attention. Measure first; attention is the scarcest budget.
Run Agents That Fit On Your Laptop.
25.6 MB median RSS. 25 agents ran in parallel on a Core 2 Duo with 4 GB RAM. Hundreds on your machine. Zero cloud required on the Ollama path.
Requires Rust/cargo to build from source. Linux and macOS today, Windows not yet supported. Pre-1.0, public beta. Pricing TBD.