case study · 2026
Handbook RAG
A RAG assistant that shows its evidence and grades itself.
- Role
- Solo: build, evals and deploy
- Year
- 2026
- Eval set
- 17 questions, 5 categories
- Judge
- 17 / 17 correct, 15 / 17 grounded
Answers questions about a fictional company's HR handbook from retrieved passages only, links every citation to the exact passage, and scores retrieval and answers separately so a wrong answer can be traced to the stage that failed.
- Python
- FastAPI
- OpenAI embeddings
- Claude / GPT
- LLM-as-judge
- Vercel
01 / The problem
A RAG answer can fail in two places: retrieval never found the passage, or the model had it and answered wrongly, left something out or added a detail it wasn't given. Grading only the final answer can't tell those apart, and a reader has no way to check a claim. This project makes both stages measurable and every answer checkable.
02 / How it works
- 01Ingest. Ten handbook documents are split on paragraph boundaries into ~500-token chunks with overlap, embedded with text-embedding-3-small and exported, with their vectors, to a committed JSON index.
- 02Retrieve. The question is embedded with the same model and the five nearest chunks are found by an exact squared-L2 search, the measure Chroma used, so distances match earlier runs.
- 03Generate. A system prompt allows answers only from the supplied passages, requires filename citations and has one fixed refusal sentence for questions the handbook doesn't answer.
- 04Inspect. Each answer opens to show every retrieved chunk with its distance, which were cited, timing and tokens; each citation opens the source page with that passage highlighted.
- 05Evaluate. Labelled questions are scored for retrieval (hit, recall, MRR, precision), for keywords and refusals, and optionally by a judge model for correctness and groundedness. Runs are saved with their config, commit and dataset hash, and can be compared on the evals page.
FastAPI with a vanilla-JS UI, no LangChain or LlamaIndex. One adapter module is the only code that knows the Anthropic and OpenAI APIs, so answers and the judge run on either provider. Visitors bring their own key; it's used for that request only.
03 / Hard problems
Knowing which stage failed
Each question is labelled with the documents a correct answer needs, so retrieval is scored before the model writes anything. A separate, larger judge model sees the same passages and returns two verdicts, correct against a written rubric and grounded in the context, through a forced tool call. A judge that returns something invalid counts as an error, not a failed case.
Serverless without the vector database
On Vercel, Chroma at query time meant about 308 MB of dependencies, a database file opened read-write on a read-only disk and a slow cold start. The index has 14 chunks, so the server now does an exact search over a committed JSON file, checked against Chroma's own results. Chroma only builds the index, and a test fails if the server imports it.
Visitor API keys, used once
Keys typed in the UI stay in the tab's session storage and go to the server in a header, never the URL. The server builds a client for that request only, never caches or logs the key, and masks anything key-shaped in provider error messages before passing the rest through.
04 / Measured, not assumed
17 labelled questions across five categories, including out-of-scope and false-premise cases, run on gpt-5-mini and judged by gpt-5. None of the 13 answerable questions was refused. Keyword and refusal checks are deterministic; judge verdicts aren't yet calibrated against human labels, so they're indicative.
| Eval | Passed | Median latency |
|---|---|---|
| Keyword and refusal checks | 17 of 17 | 8.27 s |
| Judge: correct | 17 of 17 | — |
| Judge: grounded | 15 of 17 | — |
| Refused when unanswerable | 4 of 4 | — |
| Retrieval hit @5 | 13 of 13 | 0.30 s |
as of nov 2026
In the first judged run, 16 of 17 answers passed the keyword checks but the judge rated only 12 correct: answers stopped at the headline number and dropped the conditions around it. Asking for the direct answer plus the details the passages state alongside it raised that to 17 of 17, at a cost: output tokens nearly doubled and median answer time went from 3.11 s to 8.27 s. The two grounded failures added a derived figure and a claim the passages didn't make.
05 / What's next
- Retrieval scores are perfect because 14 chunks with k = 5 retrieves a third of the index; a larger corpus is needed before they mean anything.
- Hand-label about 30 answers to measure how often the judge agrees with a person.
- Citations are document-level, inferred from filenames; claim-level citations are the next step.
Want to talk through how it’s built?