Q
Shadow eval vs A/B testing?
Same spirit, zero user impact: historical queries replay against candidates offline. Promote on measured wins.
Glossary · Shadow Evaluation
Shadow Evaluation.
Canary deploys for knowledge.
Direct answer
Shadow eval scores candidate knowledge against real historical queries in parallel with live serving — measured, never served — promoting only what beats baseline. UNKNOWN is never treated as zero; absence of evidence stays absence, not failure. It is the final gate between extraction ('this might help') and serving ('this earns tokens every turn'), and the mechanism that lets memory systems learn aggressively without ever regressing answers.
In Anvaya
How We Implement It.
01Shadow EvaluationAnvaya's shadow_nodes/links plus shadow_eval harness gate every promotion; experience-benchmark protocol S0–S4 defines the fixture suite.
Questions
Asked About Shadow Evaluation.
Q
What is the S0–S4 protocol?
Anvaya's experience benchmark fixtures: fail→fix headless, denial handling, tool-only exploration, producer-off baselines — open trials Q3 2026.
Q
Why not serve-then-measure?
Serving unproven memory taxes every turn's attention. Measure first; attention is the scarcest budget.
Stop Starting From Zero.
One binary. 11+9 Rust crates. 545 tests. Hand-written HNSW index. Three transport modes. Four providers, Ollama, Anthropic, OpenAI, Siemens. Zero API keys required to start. Mind remembers everything after the first session.