What 5,535 DolphinBench Runs Taught Me About Local AI Memory

1. Why DolphinBench Breaks Toy Haystacks & My 5-Gate Clean-Room

Toy needle-in-a-haystack benchmarks hide a self-contained fact inside filler text and query the exact same nouns. Real workspace agents fail on stateful action execution under evolving rules. Released by Mem0 (mem0ai/dolphinbench), DolphinBench tests 600 live mock_mcp workspace tasks across 13,539 chronological sessions (1,500,265 o200k_base tokens) and three personas—Morgan (200 Founder/Ops tasks), Alex (200 Staff SWE tasks with 18 KB incident logs and deprecated flags), and Riley (200 Growth PM tasks)—plus 15 pure-deterministic counterfactuals. Passing requires retrieving standing policies that share zero nouns with today's terse prompt, resolving [Date: YYYY-MM-DD] temporal supersession, chaining list_* state lookups before mutating records, and withholding prohibited side effects.

Every run across my 9 × 615 = 5,535-trace local ablation suite and my 28 × 600 = 16,800-trace official openai/gpt-5.6-sol scoreboard is enforced by verify_zero_leakage.py across five gates:

1. Zero-LLM 0o444 SQLite Immutability: Ingests all 13,539 sessions in 153.4s with 0 LLM calls ($0.00), truncates WAL, applies chmod 0o444 + file:...?mode=ro URIs, and verifies pre/post SHA-256 digests across every 600-task run.
2. Static AST Clean-Room Scan: Verifies zero imports from grading rubrics, zero persona proper nouns, and zero test-ID branching.
3. Official 5-Method BenchmarkHarness Replay: Passes upstream python -m harness.export_submission --release upstream --check offline with 0 errors across all 28 archives.
4. Mechanical & Semantic Assertion Suite (1,920 YAML Assertions): Evaluates the exact same 1,310 deterministic mechanical checks (across 599 tasks) and 610 field_llm_judge semantic checks on every 600-task archive, alongside my zero-LLM inner-loop tool-argument gate.
5. Counterfactual Causality Gate (15 Tasks, memory_enabled=True vs. False): Confirms causal lift from retrieved memory (+53.3 pp on both 30B and 120B).

2. Axis A: Zero-LLM Hybrid Memory vs. Raw Cosine (+12.50 pp SWE)

Cloud memory layers run paid LLM extraction over every message during ingestion, whereas naive vector RAG embeds whole turns and fetches top-k cosine neighbors. Holding Stock gpt-oss:120b, 1024d snowflake-arctic-embed2, the 4,200-token budget, and judges fixed, swapping single-query cosine (VanillaSnowflakeHarness) for SovereignMemoryStore lifts overall pass rate from 48.50% (291/600) → 55.83% (335/600, +7.33 pp).

• Morgan Ties (65.0% Vanilla vs. 64.0% Sovereign): Short, self-contained executive notes (flight seating, wire caps) already rank cleanly under dense cosine similarity.
• Alex (32.0% → 44.5%, +12.50 pp) & Riley (48.5% → 59.0%, +10.50 pp) Jump: SWE and PM logs mix 14 KB pasted incident packets with terse channel rules. SovereignMemoryStore fixes three failure modes deterministically: (1) 256/32 Raw-RAG windows (arXiv:2610.30755) prevent long turns from diluting pooled vectors or starving token budgets; (2) Porter-FTS5 BM25 + 4-Way RRF (k = 60) anchors exact symbols like dedup-v2 and #release-ops; and (3) directive/transition regex priors + chronological turn_index packing keep later-dated policy overrides ordered last, backed by active search_persona_memory calls.

# sovereign_memory.py — Read-only 0o444 URI guard + 4-Way Hybrid RRF + Raw-RAG window projection
def retrieve_persona_briefing(path: Path, cache: FrozenCache, query: str, q_vec: list[float], rrf_k: int = 60):
    if (path.stat().st_mode & 0o222) != 0:
        raise PermissionError(f"Refusing benchmark retrieval on writable SQLite store: {path}")

    fts_expr = build_fts5_or_query(query)  # Quoted Porter-stemmed token OR-clause
    conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True, timeout=10.0)
    try:
        top_turn_bm25 = [int(r[0]) for r in conn.execute(
            "SELECT rowid - 1 FROM turns_fts WHERE turns_fts MATCH ? ORDER BY bm25(turns_fts) ASC LIMIT 128",
            (fts_expr,),
        ).fetchall()]
        top_win_bm25 = [int(r[0]) for r in conn.execute(
            "SELECT rowid - 1 FROM windows_fts WHERE windows_fts MATCH ? ORDER BY bm25(windows_fts) ASC LIMIT 128",
            (fts_expr,),
        ).fetchall()]
    finally:
        conn.close()

    # 4-Way RRF: Turn-Dense (1.0) + Turn-BM25 (1.15) + Win-Dense (0.85) + Win-BM25 (0.95)
    turn_rrf: dict[int, float] = {}
    for rank, (t_idx, _) in enumerate(top_turn_dense):
        turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 1.00 / (rrf_k + rank + 1)
    for rank, t_idx in enumerate(top_turn_bm25):
        turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 1.15 / (rrf_k + rank + 1)
    for rank, (w_idx, _) in enumerate(top_win_dense):
        for t_idx in cache.windows[w_idx]["turn_indices"]:
            turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 0.85 / (rrf_k + rank + 1)
    for rank, w_idx in enumerate(top_win_bm25):
        for t_idx in cache.windows[w_idx]["turn_indices"]:
            turn_rrf[t_idx] = turn_rrf.get(t_idx, 0.0) + 0.95 / (rrf_k + rank + 1)
    # Cross-encoder rerank + salience boost -> knapsack-pack 4,200 tokens in chronological order.

3. Axis B: Parameter Scale Phase Transition (8B → 30B MoE → 120B MoE)

With retrieval frozen, every model evaluated on SovereignMemoryStore receives the byte-identical 4,200-token turn-1 briefing across all 600 tasks, isolating parameter scale across my 6-run Phase-1 matrix (3,690 traces across Node-02 Dual RTX 2080 Ti 22 GB NVLink, Node-01 Dual RTX 3060 24 GB, and 96 GB Blackwell G4 burst):

Stock hermes3:latest (8B Q4_0) did not fail on JSON tool syntax—it emitted valid tool calls on 593/600 tasks (66.44% mechanical task gate), yet collapsed to 4.50% (0/15 pure deterministic) when binding multiple policy constraints simultaneously. Stepping up to local Qwen3-Coder-30B-A3B (only 3.3B active parameters on Dual RTX 2080 Ti at $0.00) crosses that reasoning threshold immediately, hitting 80.33% under hermes3 and a suite-high 88.81% (532/599) mechanical compliance.

4. Axis C: Wave-4→Wave-6 QLoRA+DPO, INV-STACK-10 2PC & Scale-Aware Grounded Harness v0.2/v0.3

My most useful discovery came from Run 3's failure, where applying my Wave-3 SRE Code-Auditor adapter to gpt-oss:120b dropped generalist accuracy from 55.83% → 36.50%, alongside a hidden Ollama template bug on 8B:

• Wave-3 Auditor Role Bias (30.0% Side-Effect Withholding): Hardcoding a zero-trust SRE Auditor SYSTEM prompt into the fused Modelfile plus bug-hunting SFT caused 15.6x over-calling of read-only search_persona_memory (701 vs. 45 calls) and 180/600 (30.0%) zero-mutation turns where the model asked for human approval instead of executing routine calendar/email tasks.
• Ollama's Silent Hermes-3 Go Template Bug ({{- else if .System }}): Inspecting ollama show --modelfile hermes3:latest explained why Stock 8B scored near zero—attaching MCP tools silently discarded the entire runtime SYSTEM prompt while duplicating <tool_call> branches on assistant turns:

{{- if .Tools }}
<|im_start|>system
You are a function calling AI model... <tools>{{ .Tools }}</tools><|im_end>
{{- else if .System }}   <-- BUG: .System is silently dropped whenever .Tools is non-empty!
<|im_start|>system
{{ .System }}<|im_end>
{{- end }}

4.1 Deterministic Tool-Loop Governance Without Retraining: krusch-stack INV-STACK-10 2PC Action Harness

Inspecting post-Wave-4 traces revealed mechanical execution pathologies orthogonal to both memory retrieval and adapter weights: 5–12x identical read spin-loops, duplicate mutating MCP writes, unresolved placeholder arguments, and premature zero-app stops. Subclassing BenchmarkHarness in clean-room sovereign_harness_krusch_stack.py over krusch-stack Tier 3A (POST /action-harness/{start,intercept,outcome,verify-stop,summary}, 68/68 Node + 6/6 Python tests) enforces SHA-256 canonical BLOCK_SPIN_LOOP / BLOCK_DUPLICATE_WRITE, pre-commit HOLD_FOR_REVISION staging, and single-user-turn-preserving role: "developer" CONTINUE_VERIFY recovery steps:

• Live 10-Task Diagnostic Failure Slice (Local Qwen3-Coder-30B, $0.00): Eliminated 100% of duplicate MCP calls (10 → 0), reduced zero-app chat-only stops -66.7% (3 → 1), and recovered +20.0 pp (0/10 → 2/10) end-to-end mechanical passes (riley/096 & morgan/058).
• Full-600 Local Dual-2080-Ti Evaluation (Qwen3-Coder-30B-A3B Wave-4 QLoRA+DPO, + 2PC Harness, $0.00): Actively governed 56/600 (9.3%) tasks, cut executed duplicate MCP calls -95.2% (42 → 2), cut zero-app stops -50.0% (2 → 1), shrank archive size -17.7% (9.006 → 7.416 MiB), lifted Morgan (55.5% → 57.0%), Alex (35.0% → 35.5%), and deep compositional LBF>=4 by +6.67 pp (22.67% → 29.33%, 17/75 → 22/75), and unlocked 46 complementary task wins over Wave-4 (Std. Harness)—yielding Oracle@2 = 50.33% (302/600) at $0.00 (Wave-4 Oracle@3 across Std. ∪ 2PC ∪ Grounded v0.3 = 58.50% [351/600]; 5-run 30B Oracle@5 = 61.17% [367/600]).
• Full-600 Cloud Generalization (GPT 5.6 Luna High / Low, + 2PC Harness, 1,200 Traces): On GPT 5.6 Luna (High, + 2PC Harness) (80.50%, 483/600 @ $1.38 vs. 81.33% @ $1.35 Std. Harness), 2PC lifted SWE-heavy Alex +2.0 pp (78.0% → 80.0%), LBF=3 by +5.21 pp (79.17% → 84.38%), and LBF>=4 by +4.00 pp (68.00% → 72.00%) with 38 complementary wins (Oracle@2 = 87.67% [526/600]), while GPT 5.6 Luna (Low, + 2PC Harness) (76.50%, 459/600 @ $1.12) lifted Morgan +3.0 pp (85.0% → 88.0%) and LBF=3 +3.13 pp (Oracle@4 Std. ∪ 2PC Luna = 89.50% [537/600]; Oracle@5 incl. + Grounded = 91.83% [551/600]).

4.2 Scale-Aware Defect-Only Governance: Grounded Harness v0.2/v0.3 (INV-GH-01..10 / INV-STACK-11) Across 4,200 Official Traces

Inspecting why INV-STACK-10 + 2PC lost -3.50 pp on noisy PM persona Riley (78.0% → 74.5% on Luna High; 37.5% → 33.5% on 30B Wave-4) exposed a subtle over-steering failure mode: unconditional Step-1 CONTINUE_VERIFY stop vetoes on zero-mutation turns occasionally pressured models out of valid 1-step policy abstentions or read-only status replies. Synthesizing DolphinBench traces alongside 120 clean-room adyen/DABstep analytics episodes (42/42 upstream scorer.py parity: 90.0% @ $0.81 [-74.5% cost, 2.54x faster] on Gemini 4 Argon and 2x accuracy [40.0% vs. 20.0% @ $0.00] on Gemma 4 31B), I built Tier 3B krusch-grounded-harness v0.2/v0.3 (sovereign_harness_grounded.py + @krusch/toolkit v1.8.1 with 16 <1ms Slop-Breaker detectors, 141/141 toolkit + 13/13 Python tests, 5/5 clean-room gates across 7 600-task runs = 4,200 official traces):

• Why Defect-Only Stop Gating + Turn-0 Deduplication + Temporal Supersession Outperformed Uniform 2PC: Instead of vetoing legitimate 1-step policy abstentions, inspectOutputDefects (INV-GH-05) approves clean completions in 0 extra turns (0 self-doubt nudges) and triggers role: "developer" recovery steers strictly on objective output defects (EMPTY_OR_NULL_BAILOUT, TRUNCATED_OUTPUT, SERIALIZED_TOOL_CALL_IN_TEXT, RAW_DATAFRAME_OR_NUMPY_REPR) or uncommitted staged diffs. Quote-safe call hashing (INV-GH-04) memoizes prior reads and blocks byte-identical spin loops (BLOCK_SPIN_LOOP, maxReadStreak = 0) while permitting fresh post-mutation verification re-reads (committed_mutations > prior_mutations), Turn-0 delta tracking (INV-GH-01..03) strips duplicate memory chunks already present in the 4,200-token briefing, and v0.3 adds deterministic latest-timestamp Turn-0 sorting plus post-search state reminders while preserving frontier recency-spotlight re-reads.
• Gemini 4 Argon (Low Thinking) + Grounded Harness v0.3 — 90.67% (544/600 @ $30.54, 90.50% [181/200] highest Riley slice in self-run 28-run evaluation): Upgrading Argon Low from Grounded v0.2 (535/600 = 89.17% @ $23.92, 8,092 Turn-0 duplicates suppressed, 0 self-doubt nudges, sha256: 5ce8a7f8...) to Grounded v0.3 jumps +1.50 pp (+9 tasks) to 90.67% (544/600), delivers 181/200 = 90.50% on noisy PM persona Riley—the first 90%+ Riley slice across my 28-run self-run evaluation (+2.00 pp over Argon Low Std. 88.5% & +2.50 pp over Grounded v0.2 88.0%), and achieves highest invoked judge accuracy across all 28 runs at 553/590 = 93.73%. Alongside Argon Low Std. (91.17%, 547/600 @ $24.54, uploaded dolphinbench.ai submission UUID c047b063, queued awaiting leaderboard acceptance; exceeds current 70.67% published entry), Grounded v0.3 contributes +21 unique tail-recall wins to lift Argon Low Oracle@2 to 94.67% (568/600), Argon Low Oracle@3 to 95.83% (575/600) with Grounded v0.2, and Argon Oracle@4 to 96.83% (581/600) with Argon Medium.
• krush420/gemma4-31b-sre-lora (Wave-6 QLoRA+DPO, + Grounded v0.3) — 57.33% (344/600 @ $0.00, highest open-weights score in self-run 28-run evaluation; 54.17% [325/600] uploaded stock submission UUID 288671ad queued on dolphinbench.ai): Scores Morgan 149/200 (74.5%), Alex 103/200 (51.5%), and Riley 92/200 (46.0%), with Mech 835/1,310 (63.74%), Sem 354/610 (63.67% invoked), and 50.81s median latency (sha256: b7a26ef9...). This lifts Gemma 4 31B Stock (325/600 = 54.17%, Queued UUID 288671ad-...) by +3.17 pp (+19 tasks: +5 Morgan, +8 Alex, +6 Riley), edges Stock + Grounded v0.2 (340/600 = 56.67%) by +0.67 pp (+4 tasks, +6.0 pp on SWE Alex [97 → 103]), beats Wave-5 (329/600 = 54.83%) by +2.50 pp (+15 tasks) and unmasked Wave-4 (322/600 = 53.67%) by +3.67 pp (+22 tasks), widens the matched-scale dense domain gap over Qwen2.5-Coder-32B (35.33%) to +22.00 pp (+132 tasks), lifts Gemma 4 Oracle@6 to 71.83% (431/600 @ $0.00, +1.16 pp above published #1 Cloud Mem0 70.67%; Grounded Oracle@2 = 65.33% [392/600]), and beats all 8 non-GPT-5.6-Luna commercial cloud stacks up to $1,830.57.
• Google Gemma 4 31B Stock (Thinking ON) + Grounded v0.2 — 340/600 (56.67%, 2nd-highest open-weights in 28-run self-run evaluation @ $0.00): Scores Morgan 146/200 (73.0%), Alex 97/200 (48.5%), and Riley 97/200 (48.5%), with Mech 808/1,310 (61.68%), Sem 349/610 (62.66% invoked), 15 spin loops blocked, 4 policy-floor blocks, 1 output defect healed, 1,228 Turn-0 duplicates suppressed, and 42.47s median latency (sha256: eab60088...)—isolating a pure zero-retraining +2.50 pp (+15 net tasks) harness lift and +5.50 pp (86 → 97) on noisy Riley over 54.17% Stock (Stock + Grounded Oracle@2 = 61.33% [368/600]).
• krush420/qwen3-coder-30b-sre-lora (Wave-4 QLoRA+DPO, + Grounded v0.3) — 258/600 (43.00%, highest Qwen/MoE score in 28-run self-run evaluation @ $0.00): Scores Morgan 125/200 (62.5%), Alex 63/200 (31.5%), and Riley 70/200 (35.0%), with Mech 613/1,310 (46.79%), Sem 266/610 (47.84% invoked), and 42.91s median latency (sha256: efc93d02...)—lifting 30B Stock (42.00%) by +1.00 pp (+6 tasks), lifting Morgan +7.00 pp (111 → 125) over Wave-4 Std. (42.67%), and unlocking +70 new complementary task wins (Oracle@2 = 54.33% [326/600], Wave-4 Oracle@3 = 58.50% [351/600], and 30B Oracle@5 = 61.17% [367/600]).
• GPT 5.6 Luna (High Thinking) + Grounded v0.2 — 489/600 (81.50%) @ $1.35 (1.57x Faster): Scores Morgan 174/200 (87.0%), Alex 157/200 (78.5%), and Riley 158/200 (79.0%) in 30.03s median latency (1.57x faster than Luna High Base 47.02s / 81.33% [488/600])—beating published #1 Hermes + GPT 5.6 Luna + Mem0 (70.67% @ $96.21) by +10.83 pp at 71.1x lower cost, recovering +4.50 pp on Riley (149 → 158) over INV-STACK-10 2PC with 5,318 Turn-0 duplicates suppressed and 0 self-doubt nudges, and lifting Luna High Oracle@2 to 89.33% (536/600) and Luna Oracle@5 to 91.83% (551/600) (sha256: 477047fc...).

4.3 Root-Cause Fine-Tuning Progression (Wave-4 → Wave-5 → Wave-6) & Dense-vs-Sparse Capacity Law

While Wave-4 immediately lifted sparse MoE and edge backbones (30B, 120B, 8B), applying that exact same unmasked recipe to dense 30.7B Google Gemma 4 31B initially dipped -0.50 pp below Stock (322/600 = 53.67% vs. 325/600 = 54.17%, sha256: 05e2d106...). Auditing token-level gradient norms and 4-bit DPO log-ratios isolated two training pathologies—and fixing them in sequence produced a clean monotonic progression from 53.67% → 54.83% → 57.33%:

• Wave-4 Unmasked Full-Sequence SFT (53.67%, 322/600) → Wave-5 Assistant-Only <|channel>thought Completion Masking (54.83%, 329/600): Standard full-sequence causal LM loss spent >80% of gradient norm predicting static Turn-0 Raw-RAG briefings and MCP JSON tool schemas, mildly perturbing dense 30.7B <|channel>thought deliberation. Masking all system, user, and tool-response tokens to -100 (Gemma4AssistantOnlyCollator, 81.56% Stage-1 token accuracy) with a cosine LR schedule immediately restored reasoning fidelity and beat Stock on Std. Harness alone (329/600 = 54.83%, Morgan 71.5% / Alex 47.0% / Riley 46.0%, +0.67 pp vs. Stock, sha256: 5b1a49b1...).
• Wave-6 Dual-Adapter BF16 Reference-Anchored DPO + Partial-Briefing Gap-Search (57.33%, 344/600 @ $0.00, highest open-weights score in self-run 28-run evaluation; 54.17% [325/600] uploaded stock submission UUID 288671ad queued on dolphinbench.ai): Under 4-bit NF4 QLoRA, default DPOTrainer calls disable_adapter() and computes preference log-ratios against the raw pre-SFT NF4 base rather than Stage-1 SFT. Cloning all 460 Stage-1 SFT LoRA tensors into an explicit frozen torch.bfloat16 reference adapter (ref_adapter_name="_sft_ref") with a 0.30 NLL anchor (beta = 0.2, 100.0% DPO preference accuracy across 2,469 pairs) alongside a 3,400-row Partial-Briefing Gap-Search curriculum (teaching targeted search_persona_memory follow-ups whenever Turn-0 omits tail policies) and Grounded Harness v0.3 pushed krush420/gemma4-31b-sre-lora to 57.33% (344/600, +3.17 pp over Stock & +3.67 pp over Wave-4).
• Dense 30.7B <|channel>thought vs. Sparse 3.3B-Active MoE Capacity Law & 4/4 Open-Weights Lifts: Running the identical Wave-6 Gap-Search recipe on non-thinking sparse Qwen3-Coder-30B-A3B (3.05B–3.3B active/token) scored 38.50% (231/600, sha256: f00271bb...) vs. 43.00% (258/600) on Wave-4 + Grounded v0.3: dense 30.7B models hold multi-query retrieval expansions inside their reasoning scratchpad (+14.33 pp above 30B), whereas 3B-active non-thinking MoEs prefer Wave-4's full-turn prompt regularization and concise search while contributing orthogonal wins to 30B Oracle@5 = 61.17% (367/600). Across all four Hugging Face families, @krush420 open-weights adapters beat Stock 4/4: gemma4-31b-sre-lora (54.17% → 57.33%), qwen3-coder-30b-sre-lora (42.00% → 42.67% Std. → 43.00% Grounded), gpt-oss-120b-sre-lora (39.17% → 39.50%), and llama3-hermes3-8b-sre-lora (1.17% → 18.33%, 15.7x).

5. Official 28-Run openai/gpt-5.6-sol Scoreboard (16,800 Traces · 8 Queued)

Building on my 9-run Wave-3/Wave-4 local ablation (5,535 verified traces), I evaluated 28 clean-room 600-task configurations (16,800 official traces across 6 model families: Google Gemini [6], OpenAI GPT-5.6 Luna [5], Google Gemma 4 [6], Alibaba Qwen [7], OpenAI GPT-OSS [2], and Nous Hermes [2]—17 $0.00 open-weights archives + 11 frontier/model-matched cloud controls) on the byte-identical frozen 0o444 SovereignMemoryStore SQLite checkpoints. Every archive was graded with the official openai/gpt-5.6-sol (reasoning_effort="medium") judge across all 1,920 YAML assertions (1,310 mechanical + 610 field_llm_judge), verified offline with 0 errors via harness.export_submission --check (28/28 validated), and 8 flagship archives are queued on dolphinbench.ai awaiting maintainer queue processing alongside 20 clean-room model-matched, 2×2 reasoning/search, Wave-4/5/6 QLoRA+DPO, dense-vs-MoE scaling, + 2PC, and + Grounded v0.2/v0.3 controls:

*Every official archive evaluates the exact fixed 1,310 YAML mechanical checks and 610 field_llm_judge checks (1,920 total); Sem* denotes invoked semantic judge accuracy (553/590 = 93.73%, highest across all 28 runs on Argon Low + Grounded v0.3), and parentheses on Wave-4/Stock open-weights rows show my zero-LLM 1,310-check inner-loop tool-argument gate prior to semantic SQLite post-state verification.

• Gemini 4 Argon (Low) + Grounded Harness v0.3 Reaches 90.67% (544/600 @ $30.54, 90.50% [181/200] Highest Riley Slice in My 28-Run Self-Run Evaluation): Pairing Argon Low with Grounded v0.3 achieves 544/600 (90.67%) @ $30.54, breaks 90% on noisy Growth PM persona Riley at 181/200 = 90.50% (+2.00 pp over Argon Low Std. [91.17%, 547/600 @ $24.54, uploaded dolphinbench.ai submission UUID c047b063, queued awaiting leaderboard acceptance; exceeds current 70.67% published entry] & +2.50 pp over Grounded v0.2), leads all 28 archives in invoked judge accuracy (553/590 = 93.73%), and lifts Argon Low Oracle@2 = 94.67% (568/600), Argon Low Oracle@3 = 95.83% (575/600), Argon Oracle@4 = 96.83% (581/600), and Global Oracle@28 = 98.33% (590/600, <= 1.67% [10/600] zero-LLM Raw-RAG misses).
• Model-Matched GPT 5.6 Luna Isolates +10.83 pp (+10.67 pp Base) Purely from $0.00 Raw-RAG Memory & Runs 1.57x Faster: Running GPT 5.6 Luna (High) over my frozen 0o444 SQLite store reaches 81.50% (489/600, 30.03s = 1.57x faster) with + Grounded Harness and 81.33% (488/600, 47.02s) on Std. Harness for $1.35—beating published #1 Luna High + Mem0 (70.67% @ $96.21) by +10.83 pp (+65 tasks) at 71.1x lower total cost (Luna High Oracle@2 = 89.33% [536/600], Luna Oracle@5 = 91.83% [551/600]), while Luna Low (Std. Harness) hits 76.67% (460/600) for $1.10 (87.4x cheaper).
• 4/4 Open-Weights @krush420 Adapters Beat Stock & Lift $0.00 Oracles Past Cloud Mem0: Dense 30.7B krush420/gemma4-31b-sre-lora climbed 54.17% → 57.33% (325 → 344/600 @ $0.00, highest open-weights score in self-run 28-run evaluation; 54.17% [325/600] uploaded stock submission UUID 288671ad queued on dolphinbench.ai; Gemma 4 Oracle@6 = 71.83% [431/600]); krush420/qwen3-coder-30b-sre-lora climbed 42.00% → 42.67% Std. → 43.00% Grounded v0.3 (258/600, Wave-4 Oracle@3 = 58.50% [351/600], 30B Oracle@5 = 61.17% [367/600] above published #5 Supermemory 59.17%); 120B eliminated Wave-3 prose refusals (39.17% → 39.50%); 8B Q4_0 jumped 15.7x (7/600 → 110/600 = 18.33%); and $0.00 Oracle@17 hits 83.17% (499/600, +12.50 pp above published #1 Mem0 & +1.67 pp above single-run Luna High Grounded).
• Why Local 30B (43.00%) Beats Sparse 120B (39.50%) & 80B (38.50%), While Dense 30.7B Gemma 4 Reaches 57.33%: Qwen3-Coder-30B-A3B chains multi-step list_* state discovery before writes on 320+ tasks (only 2 zero-tool turns; 1 under + 2PC), whereas scaling inactive MoE experts alone (80B / 120B at 3B–5B active params) leaves multi-constraint synthesis bounded near ~40%—and activating all 30.7B general-domain parameters per token in Gemma 4 31B adds another +14.33 pp.

6. 6-Family Synthesis: Gemma 4 31B 57.33%, Model-Matched Luna 81.50%, Argon 91.17% / 90.67% & Scale-Aware Harnesses

6.1 What Kind of Local Model Excels at Multi-App Agentic Memory? (Gemma 4 31B 57.33% Wave-6 / 56.67% Stock+G / 54.17% Stock vs. Coder-32B 35.33%)

Before testing dense non-coder models, it looked plausible that open-weights agents hit a hard ~42% wall on DolphinBench. Evaluating Apache-2.0 Google Gemma 4 31B (30.7B 100% active dense Q4_K_M) disproved that hypothesis across six controlled archives: stock Gemma 4 31B (Dense Stock, Std. Harness) (Queued UUID 288671ad-afe0-40a3-93af-b834c9bbd306) reaches 54.17% (325/600) with native <|channel>thought thinking enabled and 52.00% (312/600) with thinking disabled (4.9x fewer output tokens: 174,226 vs. 853,604; Stock Oracle@2 = 61.50% [369/600]), zero-retraining Stock + Grounded v0.2 reaches 56.67% (340/600, second-highest open-weights score in this self-run evaluation, +2.50 pp overall, +5.50 pp on noisy Riley, 15 spin loops blocked), assistant-masked Wave-5 reaches 54.83% (329/600), and krush420/gemma4-31b-sre-lora (Wave-6 QLoRA + BF16 Ref-Anchored DPO + Grounded v0.3) sets the highest open-weights score in this self-run 28-run evaluation: 57.33% (344/600) at $0.00 total cost (Morgan 149/200 [74.5%], Alex 103/200 [51.5%], Riley 92/200 [46.0%]; +3.17 pp [+19 tasks] over Stock, +6.0 pp on SWE Alex over Stock + Grounded, Grounded Oracle@2 = 65.33% [392/600], and Gemma 4 Oracle@6 = 71.83% [431/600, +1.16 pp above Official #1 Cloud Mem0])—beating all 8 non-GPT-5.6-Luna commercial cloud baselines up to $1,830.57 and surpassing GPT 5.6 Luna + Hindsight/Honcho (144/200 = 72.0%) on Morgan (149/200 = 74.5%).

Holding active parameter count fixed near ~31B dense, general-domain multi-app tool post-training (Gemma 4 31B: 57.33% Wave-6 + Grounded / 56.67% Stock + Grounded / 54.17% Stock) outperforms code-specialized Qwen2.5-Coder-32B (Dense Stock, Std. Harness) (35.33%, 212/600, only 0.03 memory calls/task) by +22.00 pp (+132 tasks; +21.33 pp Stock + Grounded, +18.83 pp Stock, McNemar p = 1.35 × 10⁻¹⁶) and sparse 3.3B-active Qwen3-Coder-30B-A3B (43.00% Wave-4 + Grounded / 42.67% Wave-4 Std.) by +14.33 pp (+86 tasks; +11.50 pp Stock, p = 4.96 × 10⁻⁷). Even 14.8B dense generalist Nous Hermes-4-14B (Dense Stock, Std. Harness) (34.50%, 207/600, 1.99 memory calls/task) statistically ties 2.2x-larger Qwen2.5-Coder-32B (p = 0.729) and beats Claude Sonnet 5 + Mem0 (32.33% @ $1,830.57)—confirming that workplace assistants need dense general-domain social/calendar/email post-training alongside coder backbones.

6.2 Two-Factor SOTA Decomposition, Goldilocks Reasoning & Cross-Suite Sufficiency (Luna 81.50% @ $1.35 · Argon Low 91.17% & 90.67% v0.3 · Oracle@28 = 98.33%)

Because Official Leaderboard ranks #1–#5 all use OpenAI GPT 5.6 Luna (High), running model-matched GPT 5.6 Luna (High) over my frozen 0o444 SovereignMemoryStore isolates the exact return on memory: it achieves 81.50% (489/600) in 30.03s (1.57x faster) with + Grounded Harness and 81.33% (488/600, 47.02s) under Std. Harness for $1.35, beating Official #1 Luna High + Mem0 (70.67%, 424/600 @ $96.21) by +10.83 pp (+65 tasks) / +10.67 pp (+64 tasks—52.0% of my total +20.50 pp SOTA lead) at 71.1x / 71.0x lower cost, while GPT 5.6 Luna (Low, Std. Harness) hits 76.67% (460/600) for $1.10 (87.4x cheaper). Upgrading the agent backbone to Gemini 4 Argon (Low) contributes the remaining +9.67 pp vs. Luna Grounded / +9.83 pp vs. Luna Base (82 vs. 23 unique wins, McNemar p = 5.96 × 10⁻⁹) to reach 91.17% (547/600, uploaded submission UUID c047b063 queued on dolphinbench.ai) at $24.54 under Std. Harness (3.9x cheaper than Mem0) and 90.67% (544/600, self-run clean-room evaluation) at $30.54 under + Grounded Harness v0.3—delivering the first 90%+ score on high-noise Riley (181/200 = 90.50%, highest Riley slice in this self-run evaluation, +2.00 pp over Std.), #1 invoked judge accuracy (553/590 = 93.73%), and a +1.50 pp (+9 tasks) advance over Grounded v0.2 (89.17%, 535/600 @ $23.92), placing every vertex of the global Cost–Accuracy Pareto frontier ($0.00 57.33% / $1.10 76.67% / $1.35 81.50% / $24.54 91.17%) on SovereignMemoryStore.

Comparing reasoning effort across three model families reveals a 3-regime Goldilocks reasoning curve: across my 2×2 Gemini 4 Argon factorial (2,400 traces; 3,600 total incl. + Grounded v0.2 & v0.3), Thinking × Agentic Search interacts significantly (ΔΔ = +3.17 pp, 95% CI [+0.27, +6.06], p = 0.0320) so Low (91.17% Std. / 90.67% Grounded v0.3) beats Medium (89.17%) by reinvesting thinking budget into iterative search_persona_memory calls (+13.17 pp lift over Low Briefing-Only 78.00% @ $15.76); whereas on GPT 5.6 Luna (3,000 traces across 5 runs), High beats Low by +4.67 pp (p = 0.0020) under Std. Harness and +4.00 pp (p = 0.0116) under + 2PC Harness because Low cuts reasoning tokens 3.3x (282.0 → 85.3/task) and prematurely drops active memory searches -32.6% (476 → 321)—matching local Gemma 4 31B (3,600 traces across 6 runs; +2.17 pp from <|channel>thought on Stock, climbing to 57.33% under Wave-6 + Grounded v0.3).

Across all 28 runs, ensemble union reaches Global Oracle@28 = 98.33% (590/600, bounding zero-LLM Raw-RAG misses to <= 1.67% [10/600]), Argon Oracle@4 = 96.83% (581/600), Argon Low Oracle@3 = 95.83% (575/600), Argon Low Oracle@2 = 94.67% (568/600, +21 unique passes), Luna Oracle@5 = 91.83% (551/600; Luna High Oracle@2 = 89.33% [536/600]), Oracle@17 = 83.17% (499/600: 92.0% / 82.0% / 75.5%) across $0.00 open-weights runs alone (+12.50 pp above #1 Mem0 & +1.67 pp above single-run Luna High Grounded!), Gemma 4 Oracle@6 = 71.83% (431/600 @ $0.00, surpassing #1 Mem0), and 30B Oracle@5 = 61.17% (367/600 @ $0.00, surpassing #5 Supermemory), with a 4.19x superlinear multiplier at compositional depth LBF>=4 (89.33% [67/75] vs. 21.33% [16/75] on 30B Stock & Gemma 4 31B Stock) and flatter corpus-entropy decay (-0.50 pp Argon Low Grounded v0.3 [Morgan 91.0% → Riley 90.5%] / -3.50 pp Argon Med vs. -10.00 pp Morgan → Riley and -12.00 pp Morgan → Alex on Mem0). Independent cross-benchmark validation on Official Snap LoCoMo (ACL 2024, 1,540/1,540 Category 1–4 QA probes across 10 multi-month conversations / 5,882 turns) confirms the identical unmodified 0o444 SovereignMemoryStore at $0.00 zero-LLM ingestion achieves 90.26% (1,390/1,540) argon_matched accuracy & 95.77% dia_id recall (181.8 ms median retrieval, +19.86 pp above LivingBrain zero-LLM 70.4%), 89.68% (1,381/1,540) argon_tight @ 4,465 mean tokens (6.73x compression), and 92.66% (1,427/1,540) LoCoMo Oracle@2. See companion Paper #27 (Return on Memory) and @krusch/toolkit/cascade for L1 (8B) → L2 (Gemma 4 31B 57.33% / 30B 43.00% @ $0.00) → L2.5 (Luna 76.67%–81.50% @ $1.10–$1.35) → L3 (Argon Low 91.17% / 90.67% @ $24.54–$30.54) routing.

6.3 Uniform 2PC vs. Scale-Aware Grounded Harness v0.2/v0.3 & Triage Boundaries

• Why Scale-Aware Defect-Only Governance Outperformed Uniform Stop Vetoes (Sections 4.1–4.3): Uniform INV-STACK-10 + 2PC Harness eliminates -95.2% (42 → 2) of duplicate MCP calls at $0.00 and lifts high-branching compositional tasks (+6.67 pp at LBF>=4 on 30B Wave-4, Std. ∪ 2PC Oracle@2 = 50.33% [302/600]; +5.21 pp at LBF=3 and +4.00 pp at LBF>=4 on Luna High), but unconditional Step-1 CONTINUE_VERIFY vetoes incur a minor -0.17 to -0.83 pp over-steering tax on simple LBF=1–2 / Riley abstentions. Replacing uniform stop vetoes with scale-aware Grounded Harness v0.2/v0.3 (@krusch/toolkit v1.8.1, INV-GH-01..10 / INV-STACK-11: 0-turn defect-only inspectOutputDefects with 16 <1ms Slop-Breaker detectors + quote-safe BLOCK_SPIN_LOOP + Turn-0 delta deduplication + latest-timestamp supersession) resolves that tradeoff—lifting Stock Gemma 4 31B +2.50 pp (+5.50 pp on Riley) to 56.67%, pushing gemma4-31b-sre-lora Wave-6 to 57.33% (Gemma 4 Oracle@6 = 71.83%), lifting qwen3-coder-30b-sre-lora Wave-4 to 43.00% (Wave-4 Oracle@3 = 58.50%), accelerating Luna High 1.57x (47.02s → 30.03s) to 81.50% while recovering +4.50 pp on Riley, and elevating Argon Low + Grounded v0.3 to 90.67% (544/600, self-run clean-room evaluation), 90.50% (181/200, first 90%+ highest Riley slice in this self-run evaluation, +2.00 pp over Std.), 93.73% (553/590) #1 invoked judge accuracy, and +21 unique tail-recall wins (Argon Low Oracle@2 = 94.67%, Oracle@3 = 95.83%, Argon Oracle@4 = 96.83%).
• Judge Calibration & 8B Q4_0 Triage Boundary: Local Referees (hermes3 80.33%, gemma4:e4b 53.00%) preserved identical rank order to gpt-5.6-sol (42.67%) for $0.00 inner-loop iteration, while 8B Q4_0's 18.33% ceiling (14.5% SWE/PM) confirms 4-bit 8B models belong in L1 triage rather than multi-app state mutation, and role-neutral Modelfiles prevent auditor refusal drift.

7. Reproducibility, Compute Ledger & Public Artifacts

# 1. Offline unit suites (13/13 Grounded + 6/6 2PC Python tests; 141/141 @krusch/toolkit v1.8.1 + 68/68 krusch-stack Node tests)
.venv/bin/python -m unittest discover -s tests -p "test_*.py" -v && .venv/bin/python verify_zero_leakage.py --static-only

# 2. Local 600-task run, Tier 3A 2PC Action Harness, or Tier 3B Grounded Harness v0.2/v0.3 ($0.00)
.venv/bin/python run_grounded_600_matrix.py --target gemma4_31b --bridge-port 11453 --workers 6 --judge-workers 10
PYTHONPATH=.:upstream .venv/bin/python -m harness.runner --release upstream --adapter sovereign_harness_grounded:BenchmarkHarness

# 3. Official openai/gpt-5.6-sol grading + offline upstream archive check across all 28 archives
.venv/bin/python package_official_submissions.py --only gemma4_31b_wave6_qlora_grounded --workers 12
PYTHONPATH=upstream .venv/bin/python -m harness.export_submission --release upstream --check submissions/sovereign_gemma4_31b_wave6_qlora_grounded.zip

• Compute & Cost Ledger: $0.00 API spend across 153.4s zero-LLM SQLite ingestion and all 17 official 600-task open-weights archives (10,200 official open-weights traces + counterfactuals across Node-02 Dual RTX 2080 Ti, Node-01 Dual RTX 3060, Node-03 RTX 2080 Ti, and on-demand 96 GB Blackwell G4 burst with 0 active sessions post-run), plus $136.24 across all six Gemini 4 Argon runs ($24.54 Low Std., $30.54 Low + Grounded v0.3, $23.92 Low + Grounded v0.2, $24.57 Med Std., $16.91 Med Briefing-Only, $15.76 Low Briefing-Only) and $6.31 across all five GPT 5.6 Luna runs ($1.35 High + Grounded, $1.35 High Std., $1.38 High + 2PC, $1.10 Low Std., $1.12 Low + 2PC—less than 1/15th of a single $96.21 Mem0 run; $142.55 combined across all 11 frontier/model-matched cloud archives / 6,600 cloud traces, well under half of one $345.64 Supermemory run).
• Public Datasets & Open Weights: krush420/sovereign-g4-benchmarks (28 official DolphinBench archives / 16,800 official traces [8 queued on dolphinbench.ai + 20 clean-room controls = 28/28 validated], Official Snap LoCoMo 1,540/1,540 probes [90.26% argon_matched & 95.77% dia_id recall, 92.66% Oracle@2], and 335,000+ total benchmark queries & traces across 14 external suites passing 95/95 SHA-256 clean-room zero-leakage gates) · krush420/sovereign-router-arena (Tab 5) · krush420/sovereign-code-audit-sft (18,131 base rows + Wave-6 Gap-Search & BF16 DPO splits) · 4/4 open-weights adapters (gemma4-31b-sre-lora [57.33% top open-weights self-run result], 30B [43.00% top Qwen adapter], 120B [39.50%], 8B [18.33%]) · github.com/kruschdev/krusch-stack · github.com/kruschdev/krusch-grounded-harness (@krusch/toolkit v1.8.1).