The problem
An organisation's agents no longer run on one model. Each person picks the harness built for their work, so what the company knows cannot live in any one set of weights; the memory layer is the only place every harness can reach. And it has to do more than recall. When a team rule and an org rule disagree, which one wins for this person? Did last week's decision reach the agent of someone who wasn't in the room? Does a superseded fact stop driving work while staying findable?
I audited ten published memory benchmarks and none of them measured that. Most measure plain recall, which long context now saturates.
What I did
memory-bench is the instrument for that question. A generator builds a simulated software organisation from a seed, people, teams, policies, and weeks of meetings, DMs, docs, and PRs, and delivers each event only to the people who witnessed it. Every fact sits in a hidden ledger with its tier, authority, and time, so the belief each person should hold at each moment is computed rather than asserted. The system under test ingests the stream however it likes and is never inspected. It gets work tasks, not quiz questions, and is scored against the ledger.
Every item has a counterfactual twin, a stream byte-identical except for the probed fact, and credit needs both sides to pass, so an answer from the model's priors earns nothing. The rule underneath the whole build: no language model decides what is true.
I built it alone over roughly two months, froze the dataset on 15 August 2026 before any system was run, and on 19 September posted two preprints to Zenodo, with the dataset on Hugging Face and the harness on GitHub. The first paper is the dataset and its validity study. The second is the engineering account of building a benchmark when every component is a language model.
Method
Every threshold was written down, with a date, before the evidence existed. Every item was screened against a memoryless floor and a fact-injected ceiling before it counted as valid, and judging is blinded in code, not in the spec. Across three simulated organisations, 371 paired instances survive a five-gate screen, and all three seeds miss the prespecified instance gate: 125, 131, and 115 against 135. The misses are marked, not repaired. A gate that is relaxed once it binds was never a gate.
Then on 4 September the provider deprecated the pinned worker model mid-pilot. The comparison between memory systems never ran, so the papers claim no ranking of anything; they release the partial rows as provenance and the procedure for re-anchoring on a successor model.
What I found
The one clean positive result is the floor. A memoryless worker earns pair credit on 4 of 125 instances on the first seed, with 95% upper bounds of 10.6%, 44.1%, and 19.4% by rung, so the twins are doing their job. The rest is about the measuring. Probe validity depends on the worker: under identical rules, 37 of 54 probe clusters survive for one model and 17 for a smaller sibling of the same family. Identical weights behind two agent harnesses agreed on 65% of the hardest condition, against a prespecified 90% bar. The harness is part of the worker.
The finding I trust most is that the judge fails by the grammar of the criterion. It over-accepts every absence-phrased error, because "does not present X as current" is satisfied by silence, and under-accepts every scope error, and the two cancel to a 14–14 split that looks like honest noise. Judge–human agreement on 150 blinded pairs was κ 0.537, which fails the κ ≥ 0.75 gate. Four judges from one vendor agreed with each other at κ 0.93–0.97 and with a human at 0.52–0.56. Consensus among language-model judges is not evidence of validity.
Open
The human anchor is one rater, and the rater is me. A second independent rater on the 150-pair packet is being arranged, and it is the one thing that would turn the judge reading from interpretation into fact. Comparative results wait on a pinned successor worker and a re-screen. No system I built, Marshmallow included, is evaluated anywhere in this work. Both papers are preprints and have not been peer reviewed.