Smoke or pilot run
- 10, 100, or 200 unique BEAM-small questions across all four tiers.
- Useful for connection, retrieval, and scoring diagnostics.
- Too small to support a full benchmark claim.
Benchmark governance
Integrity
requirements.
AMBIENT supplies the test, fixed controls, and evidence requirements. Hugging Face runs remain participant-owned; this site does not rank or publish them automatically.
Evidence gates
What a valid claim must prove
A generated score is not automatically a trustworthy comparison. The claim must stay within the evidence actually produced by the selected run.
Controlled memory run
- All 400 unique BEAM-small questions and all 1,600 judged tier rows.
- The same fixed reader, judge, classifier, prompts, corpus, budgets, and design across systems.
- Balanced sampling, paired uncertainty, evidence fingerprints, and zero run or judge errors.
Architecture capability suite
- The separate 15-probe suite tests memory-system behaviors beyond BEAM answer accuracy.
- Each probe retains its raw output and pass/fail witness.
- A BEAM run does not substitute for these architecture checks.
Before you enter
What an integrity review involves
AMBIENT is a controlled, evidence-producing evaluation—not a single prompt or a model popularity contest. Expect four memory conditions, independent judging, trace inspection, and a verifiable artifact bundle.
Expose an observable memory query
The adapter must return the exact external context served for each question. If the harness cannot observe a non-empty store response, the answer cannot earn memory credit.
Run every question four ways
T1 disables memory, T2 enables reference capture, T3 combines both paths, and T4 isolates the selected memory. A complete hosted run is 400 unique questions and 1,600 judged tier rows.
Keep the full evidence trail
Expect manifests, raw transcripts, served context, verdicts, summaries, fingerprints, and SHA-256 checks. Missing pairs, drift, mock output, or any reader, adapter, or judge error invalidates the claim.
Match the claim to the evidence
Space results are private and unreviewed. Architecture claims require fixed controls and the separate capability suite; a native full-system result cannot be presented as isolated memory lift.
Architecture qualification
The fifteen abilities AMBIENT tests
This is the same capability model shown on the benchmark page—not the ten source labels inside the BEAM corpus. Each card explains the observable behavior AMBIENT looks for and the failure pattern that makes the test difficult.
Abstention
- What it tests
- Declines when the stored record cannot support an answer.
- Why it’s difficult
- Fluent readers tend to guess from prior knowledge, and an empty retrieval must not be presented as evidence.
Adoption
- What it tests
- Routes a new memory into the correct governed store so it remains available later.
- Why it’s difficult
- Multi-project routing and fallback behavior can silently place data in the wrong store or an ungoverned side file.
Anteriority
- What it tests
- Establishes that one record existed before another using verifiable ordering evidence.
- Why it’s difficult
- A self-reported timestamp is easy to invent; credible ordering requires evidence that survives reopening and independent inspection.
Calibration
- What it tests
- Keeps confidence proportional to the quality and independence of supporting evidence.
- Why it’s difficult
- Repeated weak claims or many correlated sources can launder uncertainty into unjustified confidence.
Concurrency
- What it tests
- Preserves simultaneous writes and keeps independent operations from corrupting one another.
- Why it’s difficult
- Lock races can silently drop updates, while write skew can leave individually valid writes jointly inconsistent.
Contradiction resolution
- What it tests
- Surfaces incompatible claims and avoids treating either side as settled fact.
- Why it’s difficult
- Conflicts may be paraphrased or separated by time, and last-write-wins can hide uncertainty instead of resolving it.
Enumeration
- What it tests
- Returns the complete requested set without omissions, duplicates, or unrelated members.
- Why it’s difficult
- Top-k retrieval is optimized for a few relevant hits, not proof that every qualifying item was found exactly once.
Expiry
- What it tests
- Stops serving a claim as current when its declared or semantic validity has ended.
- Why it’s difficult
- Temporal scope is often implicit in language, and stale records can remain highly relevant to a lexical search.
Federation
- What it tests
- Reads across independent stores while preserving origin and cross-store disagreement.
- Why it’s difficult
- Deduplication, identifier collisions, or member order can erase one source or collapse a real conflict.
Holonomy
- What it tests
- Detects cyclic ordering contradictions such as A before B, B before C, and C before A.
- Why it’s difficult
- Every pair can look locally plausible; the impossibility appears only when the full dependency cycle is evaluated.
Modality
- What it tests
- Keeps hypothetical, proposed, and actual events in distinct states.
- Why it’s difficult
- The same entities and actions appear in every version, so lexical similarity can make a possibility look like history.
Provenance
- What it tests
- Preserves who produced a claim, where it came from, and which evidence supports it.
- Why it’s difficult
- Summarization, merging, and retrieval can detach content from its origin and make dependent sources appear independent.
Reactivity
- What it tests
- Updates or alerts on dependent conclusions when supporting memory changes.
- Why it’s difficult
- The affected state may be several edges away, and cached conclusions can remain stale without explicit propagation.
Reader independence
- What it tests
- Shows that memory supplied information the fixed reader could not recover alone.
- Why it’s difficult
- A strong model may already know or guess the answer, so attribution requires a paired no-memory control and traceable facts.
Set integrity
- What it tests
- Proves that served items belong to the recorded set and that later history only appends.
- Why it’s difficult
- Ordinary logs can be rewritten; tamper detection needs reproducible membership and consistency proofs.