CV Mickael Gicquaire · case study

Case studyApplied AI R&D · 09/2025 – present

Two hybrid RAG engines served over MCP, measured with their limits

Hybrid BM25 + vector search, RRF fusion, MCP servers plugged into a coding agent — then an evaluation that separates what the prompt really improves from what a combined score suggests.

Scope
Applied AI R&D — self-directed projects, no client work
Role
Designed, specified, evaluated, optimised; agent-driven development (Claude Code, adversarial review by Codex)
Machine
2014 laptop (i7-4860HQ, GTX 980M 4 GB), Windows + WSL2
Stack
Python · FastMCP · SQLite FTS5 · ChromaDB · NumPy · sentence-transformers

A Figures recomputed from the artefacts (result files, databases, manifest) in October 2026. Their exact scope is detailed in section 04.

Context: learning to build, and to measure

Since September 2025 I have been doing applied AI R&D on self-directed projects, with no client work. The goal: be able to design, end to end, an assistant that answers from documentation and cites its sources — and be able to prove that it does.

Two systems run locally. Same architecture, different scales:

System 1 · technical documentation

Hybrid RAG over technical documentation — “atelier-code” MCP server

MCP server (FastMCP, 8 tools) for hybrid BM25 + vector search fused with RRF, over ~69,000 passages from 31 technical books, integrated with Claude Code.

  • Python
  • FastMCP
  • ChromaDB
  • sentence-transformers
  • SQLite FTS5 (BM25)
  • RRF

System 2 · multilingual

Multilingual hybrid RAG engine over a heritage corpus (ancient texts) — MCP server

Search engine exposed as an MCP server (6 tools): 734,456 passages, 26 corpora, 4 languages, 2 of them in non-Latin scripts; search runs fully locally, answers generated by Claude.

  • Python
  • FastMCP
  • multilingual sentence-transformers
  • PyTorch CUDA
  • NumPy float16 (memory-mapped)
  • SQLite FTS5
  • RRF
How I work

Designed, specified, evaluated and optimised each system; agent-driven development (Claude Code, adversarial review by Codex), run locally.

Method: plans locked through cross-model adversarial review (Claude Code + Codex), golden sets, measured gates and automated tests.

The code is produced by AI agents; my work is the specification, the architecture, the review and the measurement — which is exactly what this page documents.

Architecture: search and client run locally, the model online

Both systems follow the same chain. Indexing, search, the MCP server and the Claude Code client run on my machine. Only the Claude model, hosted online by Anthropic, writes the answer: to do so it receives the question and the retrieved passages.

Architecture: locally, ingestion, dual BM25 and vector index, RRF fusion, MCP server and Claude Code client; online, the Claude modelPDF documents (and EPUB for system 2) are ingested and split into passages, then indexed twice: a BM25 index (SQLite FTS5) and a vector index. Both rankings are fused with RRF and exposed by an MCP server. These steps run locally on a 2014 laptop under Windows and WSL2. The Claude Code client also runs locally: it calls the server's tools and receives passages with their source. It then sends the question and the retrieved passages to the Claude model, hosted online by Anthropic, which writes the cited answer and returns it to the client. The evaluation bench's LLM judge also goes through the online model.LOCAL — 2014 LAPTOP · WINDOWS + WSL2ONLINE — ANTHROPIC01DocumentsPDF (+ EPUB)02Ingestioninto passages03BM25 indexSQLite FTS503Vector indexembeddings04RRF fusionone ranking05MCP serverFastMCP · tools06Claude Codelocal clientcalls the toolsrequestpassages07Claude modelhosted by Anthropicanswers, citesLLM judgeA/B evaluation benchquestion +passages sentcited answersearch on CPU · GTX 980M GPU for indexing the multilingual engine Architecture: locally, ingestion, dual BM25 and vector index, RRF fusion, MCP server and Claude Code client; online, the Claude modelPDF documents (and EPUB for system 2) are ingested and split into passages, then indexed twice: a BM25 index (SQLite FTS5) and a vector index. Both rankings are fused with RRF and exposed by an MCP server. These steps run locally on a 2014 laptop under Windows and WSL2. The Claude Code client also runs locally: it calls the server's tools and receives passages with their source. It then sends the question and the retrieved passages to the Claude model, hosted online by Anthropic, which writes the cited answer and returns it to the client. The evaluation bench's LLM judge also goes through the online model.LOCAL — 2014 LAPTOP · WSL2search on CPU · GTX 980M GPUfor indexing (multilingual engine)01DocumentsPDF (+ EPUB)02Ingestioninto passages03BM25 indexSQLite FTS503Vector indexembeddings04RRF fusionone ranking05MCP serverFastMCP · toolsrequestpassages06Claude Codelocal clientcalls the toolsquestion +passages sentcited answerONLINE — ANTHROPIC07Claude modelhosted by Anthropicanswers, citesLLM judgeA/B evaluation bench
Chain shared by both systems. Solid lines: what stays on the machine. Dashed lines: what is sent to the Claude model online (the question and the retrieved passages) and the answer that comes back. The LLM judge, which also goes through the online model, is only used by the evaluation bench.
  1. 01
    DocumentsPDF; for system 2, EPUB and API collection with retry and resume.
  2. 02
    IngestionText conversion, splitting into passages, idempotent checkpointed indexing.
  3. 03
    Dual indexBM25 (SQLite FTS5) for exact terms, a vector index (sentence-transformers) for meaning: ChromaDB in system 1, memory-mapped float16 NumPy vectors in system 2.
  4. 04
    RRF fusionReciprocal Rank Fusion: both rankings are merged by rank, without having to make their scores comparable.
  5. 05
    MCP serverFastMCP: 8 tools in system 1, 6 tools in system 2. In system 1: lazy loading to stay within the 60 s MCP timeout, and a workaround for SQLite's 999-variable limit.
  6. 06
    Claude Code clientRuns locally: calls the MCP server's tools and receives passages with their source.
  7. 07
    Claude model, onlineReceives the question and the retrieved passages, writes the answer and cites its sources; the answer goes back to the client.

Local

2014 laptop, Windows + WSL2

  • Ingestion, indexing, both indexes and RRF fusion
  • MCP servers and the Claude Code client: search is fully local
  • GTX 980M GPU for indexing the multilingual engine; inference on CPU

Online

Claude model (Anthropic)

  • Receives the question and the retrieved passages, writes the cited answer
  • Generation and LLM judge for the evaluation bench
  • Retrieved passages therefore leave the machine: “fully local” applies to search only, not to the whole chain

Evaluation protocol: one parameter at a time

On system 1 the question was simple: does a stricter answer prompt really improve the answers? The experiment changes one thing only: that prompt.

  1. LocalGolden set61 questions: 56 answerable from the corpus, 5 out of corpus
  2. LocalHybrid searchthe same passages for both modes
  3. OnlineGeneration × 2question and passages sent to the Claude model: one answer with the minimal instruction, one with the answer protocol
  4. OnlineLLM judgethrough the Claude model: each answer compared with the reference answer
Minimal instruction (baseline)
Answer from the passages, no request for citations.
Answer protocol
Strict rules: ground every factual detail in the passages, cite the source of every claim, answer “the corpus does not cover this point” rather than improvise.
Judged axes
Correct answer · citation present · abstention.
Combined score
Answerable question: passes if the answer is judged correct and cites a source, without abstaining. Out-of-corpus question: passes if the model abstains.

Results, and what they do not say

System 1 — A/B test of the answer prompt A

Protocol
A/B test of the answer prompt on a 61-question golden set, LLM judge
Measure
internal score “answer judged correct + citation present”, over the 56 questions covered by the corpus
Result
Answer protocol: 51/56 (91%) · Minimal instruction (baseline): 21/56 (38%); 5/5 out-of-corpus abstentions in both modes.

The number is striking. Broken down axis by axis, it tells a more modest — and more useful — story:

Score broken down by axis, minimal instruction vs answer protocol, over 56 answerable questionsCorrect answer with citation: 21 with the minimal instruction, 51 with the answer protocol. Citation present: 24 vs 55. Accuracy alone: 53 vs 52.02856Correct + citationcombined scoreCorrect + citation — Minimal instruction (baseline): 21/5621/56Correct + citation — Answer protocol: 51/5651/56Citation presentregardless of accuracyCitation present — Minimal instruction (baseline): 24/5624/56Citation present — Answer protocol: 55/5655/56Accuracy aloneanswer judged correctAccuracy alone — Minimal instruction (baseline): 53/5653/56Accuracy alone — Answer protocol: 52/5652/56 Score broken down by axis, minimal instruction vs answer protocol, over 56 answerable questionsCorrect answer with citation: 21 with the minimal instruction, 51 with the answer protocol. Citation present: 24 vs 55. Accuracy alone: 53 vs 52.02856Correct + citation · combined scoreCorrect + citation — Minimal instruction (baseline): 21/5621/56Correct + citation — Answer protocol: 51/5651/56Citation present · regardless of accuracyCitation present — Minimal instruction (baseline): 24/5624/56Citation present — Answer protocol: 55/5655/56Accuracy alone · answer judged correctAccuracy alone — Minimal instruction (baseline): 53/5653/56Accuracy alone — Answer protocol: 52/5652/56
The answer protocol takes citations from 24/56 to 55/56; accuracy alone stays flat (53/56 with the minimal instruction, 52/56 with the protocol). Bars: answerable questions, out of 56.
Show the data as a table
MeasureMinimal instructionAnswer protocol
Answer correct + citation present21/5651/56
Citation present24/5655/56
Accuracy alone53/5652/56
Out-of-corpus abstention5/55/5
Honest reading
  • The gain comes from citations, not accuracy. The protocol does not make answers more correct: it increases how often answers cite a source; citation faithfulness is still to be checked systematically.
  • This combined score measures neither accuracy alone nor a hallucination rate.
  • Both modes abstain on the 5 out-of-corpus questions; the protocol wrongly abstains on 1 answerable question.
  • The judge is an uncalibrated LLM, and citation faithfulness — does the cited passage really say what the answer claims? — was not systematically checked.

System 2 — multilingual engine A

734,456

indexed passages

26

corpora

4

languages, 2 of them in non-Latin scripts

6

MCP tools

Migration: 18.8 GB Chroma file → 1.1 GB of float16 vectors A

Migrated the ChromaDB file (float32 vectors, passage text and internal index: 18.8 GB) to memory-mapped float16 NumPy vectors only (1.1 GB), with text now served from the existing full-text index.

Before: Chroma file, vectors and text. After: float16 vectors only, text served by the existing indexChromaDB SQLite file, float32 vectors and passage text: 18.8 GB. Memory-mapped float16 NumPy vectors only: 1.1 GB. Text is served by the existing full-text index.ChromaDBvectors + textChromaDB — 18.8 GB18.8 GBNumPy float16vectors onlyNumPy float16 — 1.1 GB1.1 GB Before: Chroma file, vectors and text. After: float16 vectors only, text served by the existing indexChromaDB SQLite file, float32 vectors and passage text: 18.8 GB. Memory-mapped float16 NumPy vectors only: 1.1 GB. Text is served by the existing full-text index.ChromaDB · vectors + textChromaDB — 18.8 GB18.8 GBNumPy float16 · vectors onlyNumPy float16 — 1.1 GB1.1 GB
Chroma's SQLite file (18.8 GB) also held the passage text, Chroma's internal full-text index and a queue of float32 vectors. After the migration, vectors are float16, memory-mapped, and text is served from the existing full-text index (2.1 GB, shared by both versions).
Why this matters

The gain comes from two decisions: half precision, and moving the text out of the vector database. Calling it plain compression would be wrong.

v2 → v3 regression bench · 32 questionsB · one-off measurement
2.6 s → 0.45 s
mean hybrid query time: total time divided by the number of queries (a mean, not a median).
97%
top-10 vector overlap between v2 and v3.
88% · 32/32
Recall@10 in open search; 32/32 at rank 5 in corpus-targeted search. Broad relevance criterion: the expected source OR corpus.
Regression
higher latency on the largest corpus.

This bench's output only survives in a session log, not in an archived file: these figures stay out of the headline strip until the bench is re-run and its output archived together with its protocol.

Limits

  • Uncalibrated LLM judge; generation and judging are non-deterministic.
  • Citation faithfulness not systematically checked: “citation present” does not mean “citation accurate”.
  • One variable tested (the answer prompt), on a 61-question golden set.
  • v2/v3 bench not archived: mean latency, regression on the largest corpus, recall measured with a broad relevance criterion.
  • Local search, hosted generation: the question and the retrieved passages are sent to a hosted model, on which the full chain depends.
  • 18.8 GB is Chroma's SQLite file alone, not “the vector index”.
  • Self-directed projects, no client work: no measurement under real operating conditions.
  • The indexed books and corpora are deliberately not named.

Takeaways

  1. Break down before aggregating.A flattering combined score was hiding flat accuracy. The breakdown is what points to the next step: checking citation faithfulness.
  2. Hybrid by default.BM25 finds identifiers and exact terms, vectors find meaning; RRF fusion combines both rankings without calibrating their scores.
  3. Storage is an architecture decision.Separating text from vectors and moving to memory-mapped float16 is what replaced an 18.8 GB Chroma file (vectors and text) with 1.1 GB of float16 vectors only, text being served by the existing index, on a 2014 laptop.
  4. A number without an archived artefact is only a clue.Hence the evidence levels shown here: A when the result can be recomputed from a file, B when it cannot yet.
  5. Driving agents takes a framework.Plans locked through adversarial review by a second model, golden sets, measured gates and automated tests: that framework is what makes agent-driven development checkable.
Next steps
  1. Re-run the v2/v3 bench — and archive its output with its protocol (queries, aggregation, machine).
  2. Check citation faithfulness — on a reviewed sample, then automatically.
  3. Calibrate the LLM judge — against human annotations.