Return on Memory: 91.17% (#1 Overall), 81.33% ($1.35 GPT 5.6 Luna) & 54.17% ($0.00 Open-Weights) Across 11,400 Official DolphinBench Traces

1. The Write-Time Distillation Fallacy, Formal Return on Memory (RoM) & 19-Run Ensemble Oracles

Prevailing production memory layers—including Mem0, Honcho, Hindsight, and Supermemory—rely on write-time LLM distillation: every incoming conversation session is fed through an extraction model that emits condensed atomic facts, entity-relation triples, or persona summaries while discarding or demoting verbatim turn transcripts. On Official DolphinBench (13,539 chronological sessions totaling 1,500,265 o200k_base tokens across three workspace personas, 1,058 canonical facts, and 1,920 YAML assertions), write-time LLM distillation and memory-service processing dominate total benchmark cost before test-time agent inference even begins: in official-results.json, GPT 5.6 Luna + Mem0 spends $96.21 total (only $3.23 in test-call tokens vs. $92.98 in ingestion and memory-layer processing), Honcho spends $142.85 ($4.94 test calls), and Supermemory spends $345.64 ($7.38 test calls).

Worse than the upfront dollar tax is the information-theoretic defect: write-time distillation executes lossy compression without conditioning on the future test-time query. When an executive notes in March, "Keep investor updates under four bullets unless Q3 ARR slips below plan," and an engineering incident log in August mentions Q3 ARR tracking 4% under target, a query-agnostic write-time extractor frequently flattens the March turn into a generic style preference or drops the conditional exception clause altogether. Once stripped from the index at write time, no amount of test-time reasoning can recover the missing premise.

In What 5,535 DolphinBench Runs Taught Me About Local AI Memory and my ICLR 2026 BEAM 10M benchmark, I took the opposite architectural stance—Zero-Distillation Append-Only Verbatim Raw-RAG inside SovereignMemoryStore (M_0):

• Verbatim Dual-Granularity Indexing (0 LLM Calls, $0.00, 153.4s Total): Every session is appended verbatim with its explicit [YYYY-MM-DD] narrative timestamp and segmented into overlapping 256-token Raw-RAG windows (32-token stride, 7,935 windows across 2,547 Morgan + 2,691 Alex + 2,697 Riley). Both full turns and 256/32 windows are embedded locally in a single 153.4s pass (11.33 ms/turn) using 1024d snowflake-arctic-embed2:latest alongside dual SQLite Porter-stemmed FTS5 BM25 tables (turns_fts, windows_fts).
• 4-Way Reciprocal Rank Fusion + Deterministic Salience & Reranking: At read time, SovereignMemoryStore fuses Turn-Dense (weight 1.00), Turn-BM25 (1.15), Window-Dense (0.85), and Window-BM25 (0.95) via RRF(k = 60), applies regex directive and state-transition priors (catching temporal overrides like "switched from", "no longer", "effective immediately"), cross-encoder reranks candidates, and packs a 4,200-token turn-0 briefing in strict chronological turn_index order—augmented by an active search_persona_memory MCP tool for iterative multi-hop query reformulation.
• Cryptographic 0o444 Read-Only Freezing: After ingestion, WAL journals are truncated and morgan.sqlite (44.7 MB), alex.sqlite (53.2 MB), and riley.sqlite (53.6 MB) are locked chmod 0o444 and opened exclusively via file:...?mode=ro URIs, guaranteeing byte-for-byte identical memory state across every model family and harness configuration I evaluate.

Formalizing Return on Memory (RoM) & Empirical Ensemble Oracles (N = 600)

Let M_0 denote my fixed, read-only zero-LLM memory index (0o444 SQLite) and let pi_i denote an agentic tool-use policy parameterized by model family, checkpoint alignment (Stock vs. Wave-4 QLoRA+DPO), thinking budget, interactive search access, and tool harness (Std. Harness vs. + 2PC Harness) across my 19 controlled 600-task evaluations (11,400 official traces):

Two immediate corollaries demolish the write-time distillation hypothesis. First, across all 19 runs reading the exact same $0.00 SQLite files, 582/600 tasks (97.00%)—including 72/75 (96.00%) of the hardest LBF>=4 multi-fact tasks—are solved cleanly, bounding zero-LLM indexing/chunking misses at <= 3.00% (18/600 unsolved tasks). Second, restricting strictly to my eleven $0.00 open-weights runs, the union Oracle@11 solves 480/600 (80.00%: Morgan 182 [91.0%], Alex 156 [78.0%], Riley 142 [71.0%])—surpassing Official #1 GPT 5.6 Luna + Mem0 (70.67% @ $96.21) by +9.33 pp (+56 tasks) at $0.00!

2. Law I — The 19-Run Scaling Matrix, Two-Factor SOTA Decomposition (52.0% Memory + 48.0% Backbone) & Domain Specialization

Expanding my clean-room study to 19 full 600-task Official openai/gpt-5.6-sol (reasoning_effort="medium") archives (11,400 traces: 8 cloud frontier/mini runs + 11 $0.00 local/open-weight runs across 6 model families: Google Gemini [4], OpenAI GPT-5.6 Luna [4], Google Gemma 4 [2], Alibaba Qwen [5], OpenAI GPT-OSS [2], and Nous Hermes [2]) answers both core attribution questions on Official DolphinBench: (1) how much of my +20.50 pp global SOTA lead over #1 Mem0 comes from Zero-LLM Raw-RAG memory alone when holding GPT 5.6 Luna (High) strictly constant, and (2) why general-domain Gemma 4 31B breaks the 50% open-weights barrier at matched ~31B dense scale.

Complete 19-Run Controlled Sovereign Matrix + 13 Official DolphinBench Cloud Baselines

Table 1 reports all 19 verified SovereignMemoryStore (M_0) archives alongside all 13 Official DolphinBench public cloud baselines from upstream official-results.json—explicitly disambiguating model checkpoint (Stock vs. Wave-4 QLoRA+DPO / Wave-4 LoRA+DPO) from tool harness (Std. Harness vs. + 2PC Harness) per INV-TERMINOLOGY-02. Every trace is graded by openai/gpt-5.6-sol (reasoning_effort="medium") across the fixed 1,920 YAML assertions (1,310 mechanical + 610 field_llm_judge semantic checks; invoked judge rates in parentheses):

Exact Two-Factor Decomposition of My +20.50 pp (+123-Task) Global SOTA Lead

Because Official Leaderboard ranks #1–#5 all use OpenAI GPT 5.6 Luna at reasoning_effort="high", running GPT 5.6 Luna (High, Std. Harness) over my frozen 0o444 SovereignMemoryStore cleanly decomposes my +20.50 pp (+123-task) advance from Official #1 Luna High + Mem0 (70.67%, 424/600 @ $96.21) to Gemini 4 Argon (Low) (91.17%, 547/600 @ $24.54) into two orthogonal, additive components:

• Factor 1 — Zero-LLM Raw-RAG Memory Architecture (52.0% of Gain, 71.0x Cheaper): Holding GPT 5.6 Luna (High) fixed, replacing lossy LLM distillation (Mem0) with verbatim 256/32 Raw-RAG + 4-Way RRF lifts pass rate by +10.67 pp (+64 tasks: Morgan +10.0 pp [88.0% vs. 78.0%], Alex +12.0 pp [78.0% vs. 66.0%], Riley +10.0 pp [78.0% vs. 68.0%]) while slashing total benchmark cost from $96.21 to $1.35—71.0x cheaper than Mem0, 255.1x cheaper than Supermemory ($345.64), 1,351x cheaper than Claude Sonnet 5 + Mem0 ($1,830.57), and even 2.38x cheaper than Mem0's $3.23 test-only inference spend! Even GPT 5.6 Luna (Low, Std. Harness) at $1.10 hits 76.67% (460/600), beating #1 Mem0 by +6.00 pp (+36 tasks) at 87.4x lower cost.
• Factor 2 — Frontier Reasoning Backbone (48.0% of Gain, p = 5.96 × 10^-9): Holding M_0 byte-identical, upgrading from GPT 5.6 Luna (High, Std. Harness) (81.33%, 488/600) to Gemini 4 Argon (Low, Std. Harness) (91.17%, 547/600) contributes the remaining +9.83 pp (+59 tasks; 82 Argon-only wins vs. 23 Luna-only wins, 95% CI [+6.58, +13.09], exact two-sided McNemar p = 5.96 × 10^-9).

Post-Training Domain Specialization vs. Parameter Count (The ~31B Open-Weights Control)

• Gemma 4 31B (54.17%) vs. Qwen2.5-Coder-32B (35.33%) — +18.83 pp at Matched Dense Scale: Although Qwen2.5-Coder-32B activates slightly more dense parameters per token (32.8B vs. 30.7B), general-domain tool-trained Gemma 4 31B wins 154 head-to-head tasks vs. 41 losses—a +18.83 pp (+113-task) advantage (95% CI [+14.53, +23.14], paired McNemar p = 1.35 × 10^-16); even with thinking disabled (52.00%), Gemma 4 31B wins by +16.67 pp (p = 2.29 × 10^-14). Code-specialized Qwen2.5-Coder-32B ignores the episodic search tool almost completely (0.03 searches/task, 15 total calls).
• Gemma 4 31B (54.17%) vs. Qwen3-Coder-30B-A3B Wave-4 (Std.) (42.67%) — +11.50 pp Active-Capacity + Domain Lift: Scaling active per-token parameters nearly 10x (3.3B sparse MoE → 30.7B dense) alongside multi-domain post-training lifts accuracy by +11.50 pp (+69 tasks, 128 vs. 59 discordant wins, 95% CI [+7.13, +15.87], McNemar p = 4.96 × 10^-7).
• Nous Hermes-4-14B (34.50%) Statistically Ties 2.2x-Larger Qwen2.5-Coder-32B (35.33%, p = 0.729): Generalist agent-tuned Hermes-4-14B issues 1.99 memory calls/task (1,191 calls, 80.2% of tasks)—matching a 32.8B code model at $0.00 and surpassing commercial Claude Code + Claude Sonnet 5 + Mem0 (32.33% @ $1,830.57).

3. Law II — The 3-Regime Scale-Dependent Goldilocks Reasoning Curve (Gemma 4 31B → GPT 5.6 Luna 2×2 → Gemini 4 Argon 2×2)

Across 6,000 paired reasoning-ablation traces—combining 1,200 Gemma 4 31B traces, 2,400 GPT 5.6 Luna 2×2 traces ({High, Low} × {Std. Harness, + 2PC Harness}), and 2,400 Gemini 4 Argon 2×2 traces ({Low, Medium} × {Briefing-Only, Iterative Search})—test-time reasoning exhibits a non-monotonic, 3-Regime Scale-Dependent Goldilocks Curve governed by whether marginal thinking tokens unlock missing tool queries or induce over-rationalization of retrieved context:

• Regime I — Compact Dense Open-Weights (Gemma 4 31B, 30.7B Dense): +2.17 pp Scratchpad Planning Lift. Without native <|thought|> scratchpads, Gemma 4 31B under-plans multi-app workspace tool chains; enabling thinking lifts accuracy from 52.00% (312/600) to 54.17% (325/600, +2.17 pp, +13 tasks, 4.9x output tokens, searches rising 119 → 156), with 57 vs. 44 complementary task wins producing a 61.50% (369/600) Oracle@2 at $0.00.
• Regime II — Mid-Cost Cloud Reasoning (GPT 5.6 Luna 2×2, 2,400 Traces): +4.67 pp (p = 0.0020) Retrieval-Exploration Lift. On GPT 5.6 Luna, compressing reasoning effort from high (282.0 reasoning tok/task) to low (85.3 reasoning tok/task) triggers a -32.6% drop in active search_persona_memory invocations (476 → 321 searches, 0.79 → 0.54/task; tasks searching 341/600 [56.8%] → 265/600 [44.2%]). Because Luna Low stops querying memory prematurely on multi-constraint tasks (LBF=3 falls 79.17% → 70.83%), Luna High outperforms Luna Low by +4.67 pp on Std. Harness (p = 0.0020) and +4.00 pp on + 2PC Harness (p = 0.0116).
• Regime III — Frontier Over-Thinking Inversion (Gemini 4 Argon 2×2, 2,400 Traces): +2.00 pp Low > Medium (ΔΔ = +3.17 pp, p = 0.0320). At frontier scale where Argon Low already executes aggressive empirical retrieval (1.84 searches/task, 1,105 calls vs. 1,050 on Medium), expanding thinking to Medium over-rationalizes static distractors and over-hedges policy edge cases—inverting the reasoning slope to 91.17% (Low) vs. 89.17% (Medium) while slashing median latency 36.0% (55.27s → 35.38s).

4. Law III — Superlinear LBF Compositional Multipliers (4.19x vs. 31B/30B, 1.31x vs. Luna High) & Corpus-Entropy Resilience

Stratifying all 600 upstream test specifications by ground-truth Load-Bearing Fact depth (LBF)—LBF = 1 (n = 264), LBF = 2 (n = 165), LBF = 3 (n = 96), and LBF >= 4 (n = 75, spanning 4 to 9 simultaneous constraints)—demonstrates both why compact open models degrade on multi-hop synthesis and why Gemini 4 Argon (Low) stays virtually flat across compositional depth while Oracle@19 holds 96.00% (72/75) even at LBF >= 4:

Corpus-Entropy Resilience Across Morgan → Alex → Riley

DolphinBench's three workspace personas form a strict corpus-entropy gradient—from Morgan (3,400 Founder/Ops sessions, 1.77 mean LBF) to Alex (5,011 Staff SWE sessions rich in stack traces and incident noise, 1.97 mean LBF) and Riley (5,128 Growth PM sessions with maximum multi-stakeholder temporal overrides, 2.50 mean LBF). Comparing model-matched GPT 5.6 Luna (High) + Sovereign Raw-RAG (88.0% → 78.0% → 78.0% @ $1.35) directly against Official #1 GPT 5.6 Luna (High) + Mem0 (78.0% → 66.0% → 68.0% @ $96.21) isolates verbatim 256/32 Raw-RAG's noise resistance on every persona:

5. Law IV — The 4-Vertex Global Cost–Accuracy Pareto Frontier & Exact Token Ledgers

Combining zero-LLM local ingestion ($0.00) with prefix-cached turn-0 Raw-RAG briefings establishes a 4-Vertex Global Cost–Accuracy Pareto Frontier—100% owned by Sovereign Raw-RAG—spanning ($0.00, 54.17%) → ($1.10, 76.67%) → ($1.35, 81.33%) → ($24.54, 91.17%). Notably, GPT 5.6 Luna (High, Std. Harness) at $1.35 (81.33%) strictly Pareto-dominates both $15.76/$16.91 Argon Briefing-Only ablations (78.00%/79.17%) and every commercial cloud memory stack ($61.48–$1,830.57, i.e., 3.9x to 1,351x cheaper):

Exact Token & Prefix-Cache Accounting Ledgers ($24.54 Argon Low & $1.35 GPT 5.6 Luna High)

Every token across all 2,388 model responses in sovereign_gemini_4_argon_low.zip (547/600 = 91.17%) and all 1,824 responses in sovereign_gpt_5_6_luna.zip (488/600 = 81.33%) is audited directly by upstream harness.costing:

6. krusch-stack INV-STACK-10 — Generalized 2PC Tool-Action Harness & Full 600-Task Trajectory Recovery

Auditing all 11,400 Official DolphinBench traces across my 19 runs exposed a universal structural invariant of tool-augmented workspace benchmarks: 0/600 passing tasks in Official DolphinBench have app_calls == []—every single passing task inspects or mutates workspace state through at least one MCP app call. Yet across open-weights and compact runs, dozens of tasks failed before grading even examined content because the model emitted zero app calls (printing a prose email draft in chat, asking the user for permission in a single-turn harness, or emitting raw <tool_call> XML/JSON strings inside plain text) or burned its step budget in duplicate read/write spin-loops: Qwen3-Coder-Next-80B (Stock) had 71 zero-app tasks, Hermes-3-8B (Wave-4 LoRA+DPO) had 71, gpt-oss:120b (Wave-4 QLoRA+DPO) had 42 (28 on Stock), Gemma 4 31B had 23 (19 No-Think), and Qwen2.5-Coder-32B (Dense Stock) had 21.

To eliminate these mechanical agentic failure modes without fine-tuning model weights or mutating benchmark prompts, I shipped INV-STACK-10 inside krusch-stack (src/engines/action-harness-engine.js, exposed via POST /action-harness/{start,intercept,outcome,verify-stop,summary} while preserving ALL_STACK_TOOLS.length === 39, verified by 63/63 Node tests and 6/6 Python tests):

• 1. Spin-Loop Read Guard (BLOCK_SPIN_LOOP): Canonicalizes nested JSON tool arguments into SHA-256 call signatures and blocks byte-identical repeat read/search calls (max_identical_reads = 1), returning a deterministic synthetic tool observation instructing the model to act on already-retrieved state.
• 2. Consecutive Read-Streak Guard (max_read_streak = 6): Caps unbounded list_* / get_* polling streaks (such as 12 consecutive list_calendar_events calls) before step exhaustion.
• 3. Duplicate Side-Effect Guard (BLOCK_DUPLICATE_WRITE): Intercepts mutating verbs (send_*, create_*, post_*, book_*, pay_*, update_*, delete_*) and prevents double-sending identical emails, PR comments, or payments.
• 4. 2PC Pre-Commit Action Audit (HOLD_FOR_REVISION): Audits staged mutating arguments prior to execution for unresolved placeholders (TBD, YYYY-MM-DD, [insert ...]), relative unanchored date strings, secret leaks, and optional krusch-context-mcp invariants.
• 5. Verify-on-Stop Draft-in-Chat & Serialized-Tool Gate (CONTINUE_VERIFY): When an imperative task terminates with zero committed mutations while printing a formatted Subject:/Draft: block, asking "Would you like me to send...?", or leaking raw <tool_call>/JSON text—without citing a standing policy abstention rationale—sovereign_harness_krusch_stack.py issues a bounded verification turn. Crucially, the steering prompt is recorded as role: "developer" in recorded_messages, strictly preserving upstream submission.py's single-user-turn invariant (users == [request.dated_message]).

Full 600-Task 2PC Evaluation Across Local MoE & Cloud Frontier Models (1,800 Official 2PC Traces)

Scaling sovereign_harness_krusch_stack.py from my 10-task diagnostic slice to three full 600-task Official gpt-5.6-sol evaluations (1,800 new + 2PC Harness traces paired against 1,800 Std. Harness traces on identical checkpoints and frozen 0o444 SQLite stores) reveals a sharp architectural complementarity between unmediated exploration and two-phase action gating:

Across both open-weight MoE and frontier cloud models, + 2PC Harness preserves overall mean pass rate within statistical noise (-0.17 pp to -0.83 pp, McNemar p ≥ 0.657) while consistently lifting multi-hop compositional synthesis (LBF ≥ 4: +6.67 pp on 30B Wave-4 and +4.00 pp on Luna High; LBF = 3: +5.21 pp on Luna High), eliminating 95.2% of identical spin-loop MCP calls on 30B Wave-4, compressing trace archive size by 17.7% (9.006 → 7.416 MiB), and unlocking 31–46 orthogonal task recoveries per run.

7. Clean-Room Protocol Bridges (argon_openai_bridge.py & luna_openai_bridge.py), 19-Archive SHA-256 Ledger & Reproducibility

Across all 19 Official gpt-5.6-sol archives (11,400 traces), sovereign_memory.py (SHA-256: 16700bc047b0a0e8...), sovereign_harness.py (Std. Harness), and sovereign_harness_krusch_stack.py (+ 2PC Harness) remained clean-room locked, with every archive passing offline harness.export_submission --check with 0 errors. Three zero-dependency localhost protocol proxies bridge provider-specific wire quirks without touching a single line of evaluation or memory code:

• argon_openai_bridge.py (:11439): Transparently caches and re-attaches cryptographic extra_content.google.thought_signature tokens across multi-turn Gemini 4 Argon MCP loops, sanitizes strict OpenAPI tool schemas, and coalesces leading system + turn-0 briefing blocks (driving my 57.1% prefix-cache hit rate).
• luna_openai_bridge.py (:11442): Routes gpt-5.6-luna → openai/gpt-5.6-luna while rewriting response model="gpt-5.6-luna" so upstream pricing/latest.json ($0.20 / $0.02 / $1.25 per 1M) audits exact token costs, maps max_tokens → max_completion_tokens ≥ 16384 so reasoning tokens never starve MCP tool calls, aliases >40-char step-prefixed tool_call_id strings, and coalesces leading system/developer messages for 61.5% $0.02/1M prefix-cache hits.
• ollama_openai_bridge.py: Normalizes <|thought|> channels and tool-call JSON payloads across all 11 $0.00 local and Colab G4 open-weight evaluations.

# argon_openai_bridge.py (:11439) & luna_openai_bridge.py (:11442) — Zero-mutation protocol bridges
STEP_PREFIX_RE = re.compile(r"^s\d+_\d+_")

def normalize_outbound_body(body: dict[str, Any]) -> dict[str, Any]:
    out = dict(body)
    sys_parts, rest_msgs = [], []

    for m in out.get("messages") or []:
        role = m.get("role")
        # 1. Coalesce leading system/developer blocks (SYSTEM_PROMPT + 4,200-tok Briefing) -> 57.1%-61.5% prefix cache hit
        if role in ("system", "developer") and not rest_msgs:
            if isinstance(m.get("content"), str) and m["content"].strip():
                sys_parts.append(m["content"])
        else:
            m_copy = dict(m)
            # 2. Re-attach cached google.thought_signature across multi-turn MCP tool steps (Argon bridge)
            if role == "assistant" and m_copy.get("tool_calls"):
                patched_tcs = []
                for tc in m_copy["tool_calls"]:
                    tc_copy = dict(tc)
                    if "extra_content" not in tc_copy:
                        cid = str(tc_copy.get("id") or "")
                        bare_id = STEP_PREFIX_RE.sub("", cid)
                        fn = tc_copy.get("function") or {}
                        fkey = canonical_fn_key(str(fn.get("name") or ""), fn.get("arguments"))
                        with _SIG_LOCK:
                            extra = _EXTRA_BY_ID.get(cid) or _EXTRA_BY_ID.get(bare_id) or _EXTRA_BY_SIG.get(fkey)
                        if extra:
                            tc_copy["extra_content"] = extra
                    patched_tcs.append(tc_copy)
                m_copy["tool_calls"] = patched_tcs
            rest_msgs.append(m_copy)

    out["messages"] = ([{"role": "system", "content": "\n\n".join(sys_parts)}] if sys_parts else []) + rest_msgs
    if out.get("tools"):
        for t in out["tools"]:
            if isinstance(t.get("function"), dict) and "parameters" in t["function"]:
                t["function"]["parameters"] = sanitize_gemini_schema(t["function"]["parameters"])
    return out

Offline Verification & Reproducibility Commands

# 1. Verify 5-gate static AST clean-room, INV-STACK-10 2PC harness & 0o444 SQLite SHA-256 immutability
.venv/bin/python -m unittest discover -s . -p "test_*.py" -v && .venv/bin/python verify_zero_leakage.py --static-only

# 2. Launch clean-room bridges & execute #1 Overall Gemini 4 Argon (Low) or model-matched GPT 5.6 Luna (High)
.venv/bin/python argon_openai_bridge.py --port 11439 &
OPENAI_BASE_URL=http://localhost:11439/v1 .venv/bin/python run_full_600.py --model gemini-4-argon --reasoning-effort low

.venv/bin/python luna_openai_bridge.py --port 11442 &
OPENAI_BASE_URL=http://localhost:11442/v1 .venv/bin/python run_full_600.py --model gpt-5.6-luna --reasoning-effort high

# 3. Replay & audit all 19 official gpt-5.6-sol archives (11,400 traces: 91.17% Argon Low @ $24.54; 81.33% Luna High @ $1.35; 54.17% Gemma 4 31B @ $0.00)
for z in submissions/sovereign_*.zip; do
  PYTHONPATH=upstream .venv/bin/python -m harness.export_submission --release upstream --check "$z"
done