Grounded Memory
Line-level grounded RAG, MCP server, injections quarantined
- abstention precision / recall on 90 frozen test questions (0.81 / 0.45 before)
- 0.95 / 0.79
- recall on 40 never-seen unanswerable questions, with no wrong abstention
- 0.83
- p50 latency (1,012 ms before), 6 ms cached
- 618 ms
- injection traps flagged, 16 / 30 paraphrased injections
- 6/6
- The problem
- A document assistant that invents is worse than a silent one, and a booby-trapped document must never dictate its behaviour.
- For whom
- An MCP-compatible assistant (Claude Desktop or other) and a web UI, over 190 documents split into 1,496 passages, offline.
- The outcome
- On 90 frozen test questions: abstention precision / recall from 0.81 / 0.45 to 0.95 / 0.79, recall@3 from 0.76 to 0.81, p50 latency from 1,012 to 618 ms; traps flagged 6/6.
Context
A dozen projects, 190 documents in French and English that change every day. An assistant must answer from them with the exact source, without hallucinating, and without obeying a document that hides instructions.
What I built
- Structure-aware chunking (headings, tables, code blocks never split) with exact line numbers; incremental indexing by SHA-256 fingerprint.
- Hybrid search: BM25 and multilingual embeddings fused (RRF), then a reranking cross-encoder; abstention on a frozen threshold plus an anchoring rule.
- Injection scanner at ingestion, patterns plus semantic detection: every passage is trusted, suspect or quarantined; quarantine is never served, only reported. 16 of 30 paraphrased injections flagged (0 with patterns alone), 0 false quarantines over 931 real passages.
- A learned, frozen abstention detector: precision / recall 0.95 / 0.79 on 90 test questions, wrong abstentions down from 5 to 2.
- Three entry points: web UI, API and MCP server (search, exact passage, source, inventory), with a ready-to-paste Claude Desktop config.
- Protocol frozen before measuring: 90 test questions (40 from v2, 50 new ones including 40 unanswerable), measured once; the old version replayed on the same questions; 60 tests.
Architecture 7 blocks
- 01 · InputIngestionStructured chunking, exact lines, SHA-256 fingerprints
- 02 · ToolScannerPatterns and semantics: trusted, suspect or quarantined, per passage
- 03 · DataSQLite indexPassages, vectors, trust
- 04 · ToolHybrid recallBM25 and multilingual MiniLM, RRF fusion
- 05 · ModelRerankingmmarco cross-encoder on 24 candidates
- 06 · ToolDecisionCite the line, or abstain (frozen detector)
- 07 · OutputServingWeb UI, API, MCP server
Pipeline 5 steps
- IngestionStructured chunking, injection scan.
- RecallBM25 and dense, RRF fusion.
- RerankCross-encoder on 24 candidates.
- DecisionCite the line, or abstain.
- ServingWeb, API, MCP.
Decisions the choice, the reason, the alternative ruled out
- D1Freeze the question split and its fingerprints before any v2 measurement.
The evaluator refuses to measure if a question changed: the test number is not overfitted.
Ruled outTuning thresholds while looking at the test set.
- D2Replay the old version on the same test questions.
The gain reads on the same set, metric by metric, with the same reference.
- D3Treat document content as untrusted data from ingestion onwards.
Quarantined passages are never served; rendered text is escaped and preceded by a “data, not instructions” warning.
- D4CPU only, checked at load time.
The service runs alongside GPU work without disturbing it; about 0.8 GB resident memory for an MCP session.
Results
v3 detector vs v2 rule, same retrieval run (CPU)
| Metric | Before | After |
|---|---|---|
| Abstention precision / recall (90 questions) | 0.81 / 0.45 | 0.95 / 0.79 |
| Abstention recall, 40 unseen unanswerable questions | 0.48 | 0.83 |
| Wrong abstentions | 5 | 2 |
| Recall@3 (90 questions) | 0.76 | 0.81 |
| Recall@3, 15 original questions | 0.75 | 0.83 |
| Paraphrased injections flagged | 0 / 30 | 16 / 30 |
| False quarantines (931 real passages) | 0 | 0 |
| p50 latency (80-query bench) | 1,012 ms | 618 ms (6 ms cached) |
Thresholds chosen on the tuning set only, then frozen. Strict protocol: a wrong abstention is counted as an error.
Frozen test set (40 questions, CPU): v0 of 18 Oct vs v2 of 20 Oct
| Metric | v0 | v2 |
|---|---|---|
| Recall@1 | 0.33 | 0.67 |
| Source in top 5 (no abstention) | 0.44 | 0.93 |
| MRR | 0.36 | 0.70 |
| Injection traps flagged | 0/6 | 6/6 |
| Quarantined passages served | 0 | 0 |
Deliverables 4
- Web UI (search, evaluation), API and read-only stdio MCP server.
- Ready-to-paste Claude Desktop configuration.
- Frozen protocol (90 test questions, including 40 never-seen unanswerable ones), per-question results and analysis.
- 60 tests; scanner: 0 false quarantines over 931 real passages, 16 / 30 paraphrased injections flagged.
Environment
Hardware and measurement setup
- Everything runs on CPU, models cached locally, about 0.8 GB resident memory per MCP session: the service runs alongside GPU work.
- p50 latency of 618 ms with reranking on CPU, 6 ms for a cached query; faster on a GPU.
Reproduce README commands
Private code: read access under NDA, or a 30-minute live demo. Request access →
.venv/bin/atelier index
.venv/bin/atelier search "Comment RELAIS revient-il en arrière ?" -k 3
.venv/bin/atelier eval verifier
.venv/bin/atelier eval tester
.venv/bin/atelier serve
.venv/bin/atelier mcp
.venv/bin/python -m pytest- UI on 127.0.0.1:8341; everything runs on CPU, models cached locally.
- If the corpus changes, thresholds are recalibrated on the tuning set.
Gallery 3
What this project means for you
The same know-how, on your problem.
A knowledge base that cites
Your procedures and documents queryable by your assistants, every answer with file and lines, and abstention rather than invention.
Resisting booby-trapped documents
A document hiding instructions is set aside and reported, never served to the assistant.
Measure before deploying
A question set written with your teams and frozen: you know the effect of every version before go-live.