How we measure
Two benchmarks, and one of them we publish every answer for.
LongMemEval
92.8% (464/500).
| reader / planner / judge | deepseek-v4-flash |
| judge prompts | the LongMemEval benchmark's own evaluate_qa.py templates, byte-identical |
| store | longmemeval_s_500_Hobs_entitysubj, 500 users / 235,840 facts |
Every answer and every retrieved memory is published in benchmarking/artifacts/ — 500 answers verbatim, 8,427 evidence rows.
Re-grade them with any judge, or hand the retrieved context to your own reader and see what the number does. Neither needs access to anything of ours.
LoCoMo
86.88% (1338/1540), categories 1–4. End-to-end QA accuracy, LLM-as-judge.
| reader / planner / judge | deepseek-v4-flash |
| judge prompts | the LongMemEval benchmark's own templates, byte-copied |
| store | locomo10_Hobs, 10 user stores / 2,905 fact events |
| code | a pre-release build — the repository's published history begins at v0.1.0 |
Category 5 (adversarial) is excluded, following common practice; the scores use the 1,540 non-adversarial questions.
No per-question artifacts are published for LoCoMo — no answers, no retrieved context, no run directory. What is published is the LoCoMo section of benchmarking/README.md: the per-category breakdown, the per-conversation spread, provenance and repro steps.
How to read these numbers
Three things the artifacts say about themselves, which we would rather state than have someone discover:
Reader and judge are the same model. That is a self-grading loop and it runs loose; how loose depends on the answer's shape. Score the published answers with your own judge — that is why they are published.
The reader is a budget model. Systems reporting 94–96% on LongMemEval generally use frontier readers, and reader tier moves the score more than most architectural differences do.
The retrieval is reproducible; the reader's prompt is not. That run had SODAMEM_BENCH_CAPTURE_INPUT off, so the assembled prompt was never recorded. What is reproducible was measured rather than assumed: the store's content fingerprint still matches, and the same query through two freshly opened stores returns the same evidence ids in the same order.
No web access
The planner's tools are all prefixed browser_, and every one of them browses the memory store — browser_search queries stored facts, browser_inspect_session opens one recorded session, browser_count_evidence counts matching evidence rows, browser_timeline_events walks one entity's history.
These runs made no network request of any kind beyond the LLM API. SodaMem ships no web-search tool; there is nothing in sodamem/tools/ that can reach the open internet.
Reproducing
The judge half needs nothing from us: hypothesis and golden_answer are both in the published file — point any judge at them.
The answer half needs no access to our service either: take retrieved_evidence for a question, feed it to any reader, and judge the result against golden_answer.
Regenerating the retrieval itself needs the frozen store, which is 12 GB and not distributed.