RAG Observability & Debugging — Part 2: Golden Sets, Failure Taxonomy, Debugging Playbooks & SLOs — 42 Templates & Checklists for Continuous Evaluation, the RAG Failure Taxonomy, Missing-and-Wrong Debugging Playbooks, Latency & Cost Debugging

The golden set existed. It was good. Nobody ran it the day the embedding model changed — and a retrieval regression lived for a month. 42 production templates & checklists distilled from Chapters 7–12 of RAG Observability & Debugging: golden sets built from real failures and replayed nightly with pins and per-slice gates, a failure taxonomy of trace signatures and distinguishing tests, the missing-and-wrong debugging playbooks, latency & cost playbooks, drift detection with PSI and quality SLOs, and the weekly debugging room. PDF + Markdown source.

## The golden set existed. It was good. Nobody ran it.

A vendor email announced a better embedding model — a drop-in replacement, higher on the public benchmarks. The golden set that would have caught what it did to contract-clause questions lived in a folder, ran by hand, at release time, when someone remembered. That day, nobody remembered. The new model went live on a Thursday; latency improved, error rates did not move, and the answers stayed fluent, well formatted, and confidently cited — while, underneath, the governing clause for a family of questions slipped to rank eleven. The regression was found a month later, not by a dashboard or the golden set, but by a client firm's partner reading an answer and checking the source herself. **RAG Observability & Debugging — Part 2** is the operating half of the discipline that makes that story impossible to repeat: 42 copy-paste templates and checklists distilled from Chapters 7–12 of the dedicated RAG observability book — golden sets built from real failures and replayed nightly against the live index, a taxonomy that gives every failure a name, a signature, and a first distinguishing test, playbooks that pin, replay, and bisect a bad answer until one stage is convicted, the latency / cost / weird-tail playbooks, drift detection and quality SLOs that alert on rot before it becomes ruin, and the weekly debugging room that turns every incident into a golden query and a guard.

Every template carries the book's tested positions built in: no retrieval incident closes until its question joins the golden set or is formally declined with a reason; gold itself rots, so items carry verification dates and are retired in the open, never silently deleted; gates are set per slice because averages are where regressions go to hide (the book's illustrative reconstruction put the clause family's recall@5 at roughly 0.91 under the old model and roughly 0.63 under the new, while the overall average moved only a few points); language-model judges are never the only judge, with deterministic checks carrying the load wherever a check can decide the question; failures are classified by mechanism from trace evidence — never from the fluency of the bad answer; and every alert must pass the 3 A.M. test: harm now, an action tonight, or it is a ticket. Where the book gives an example — its gate shape, its PSI bands, its worked classifier rules — the template labels it as the book's example for you to tune. Bracketed fields are yours to fill; nothing here invents a threshold for your system.

### What's inside — 42 templates & checklists in 7 sections

**1. The Golden Set Never Sleeps — Continuous Evaluation Against a Moving Corpus (6)** — Golden Item Record — provenance, expected chunks, containment & forbidden strings · Incident-to-Golden-Set Closure Rule & Scar Register Log — no incident closes unexamined · Quiet-Corner Coverage Audit Sheet — synthetic items for the corners traffic never samples · Gold-Rot Diagnostic & Re-Verification Ceremony Sheet — is the gold wrong, or the system? · Replay Harness Run Record — nightly, on-change, release-gate; pinned and raw · Gate, Budget & Judge-Governance Sheet — few gates, defended; judges never alone

**2. Name the Failure or Chase It Forever — A Taxonomy of How RAG Goes Wrong (6)** — Four-Stage Incident Re-Classification Exercise — reopen the wall before you rebuild it · Corpus & Chunking Failure Class Cards — missing, stale, amputated · Retrieval & Ranking Failure Class Cards — mismatch, miss, demotion, wrong version · Context & Generation Failure Class Cards — truncation, lost in the middle, ignored, fabricated · Freshness & Tenancy Class Cards + the Impersonator Test Sheet — the pairs that fool responders · Incident Tagging Discipline & Monthly Wall Review Sheet — one primary class, test on the card

**3. Missing and Wrong — The Debugging Playbook for Bad Answers (6)** — Incident Pin Sheet — freeze the case before anything changes · Standing Accusations Notebook — one expectation sentence per stage, written before you look · Stage Bisection Worksheet — test the middle first, halve the suspects · Playbook A: The Absent-Answer Ladder — five rungs, one decisive test per rung · Playbook B: The Wrong-Answer Three-Way Fork — document, span, or generation · Replay Safety Rules & the Incident Note Template — a case a stranger can re-run

**4. Slow, Dear, and Weird — The Debugging Playbook for Latency, Cost, and Edge Cases (6)** — Per-Stage Latency Percentile Sheet — percentiles by stage, never one number · Cost Attribution Worksheet — price every query, stage by stage, from the trace · Costliest-Query Review Sheet — the list, read for shapes, weekly · Weird-Tail Hunt Sheet — tokenisers, Unicode, and monster documents · Guardrail Lever Comparison & Selection Sheet — four levers, prices in view · Budgets & Guardrails That Bite Checklist — the definition-of-done form

**5. Alert on Rot, Not Just Ruin — Drift Detection and Quality SLOs (6)** — Three-Drift Identification Sheet — queries, corpus, embeddings; name the layer that moved · PSI Drift Measurement Worksheet — compare distributions, read the per-bin story · Honest Windows & Baseline Defence Sheet — the choices that make drift numbers true · Quality SLO & Error Budget Sheet — turn taste into arithmetic a product owner can sign · Burn-Rate Alerting Sheet — for rot, threshold the burn; for ruin, threshold the rate · The 3 A.M. Test Alert Routing Sheet — page only what harms and names its action

**6. The Debugging Room — Three Incidents, Traced End to End, and the Operating Model (6)** — Incident Reconstruction Worksheet 1 — The Wrong Year's Policy (candidate-set evidence convicts) · Incident Reconstruction Worksheet 2 — The Reranker That Buried the Addendum (sliced replay + delta telemetry) · Incident Reconstruction Worksheet 3 — Two Vector Spaces in One Index (drift & geometry telemetry convict) · Merged Timeline Reconstruction Sheet — one timeline no single log contains · Three-Layer Evidence Coverage Sheet — no single layer suffices · Operating Model Adoption Sheet — budget, ownership, and the ritual, in that order

**7. Part 2 Gates & Working Artefacts — Appendices A & B and the Chapter Checklists (6)** — Appendix A Production Readiness Checklist — RAG Observability, scored · Appendix B Quick Reference Wall Card — thirteen rules of thumb, on one page · The Two Loops Operating Card — the Golden Set Loop and the Observability Loop · Failure Taxonomy Wall Chart — the tree, its signatures, its tests · Weekly Debugging Room Agenda & Disposition Record — the worst answer, traced live · Part 2 Production Gate — Chapters 7–12 Sign-Off — the book's six chapter checklists consolidated into one evidence-backed gate (46 lines)

### Who this is for

RAG and AI platform engineers, retrieval and evaluation leads, and SREs who operate a retrieval system in production and are tired of incidents that close with "the model is bad". The source book assumes you already run a RAG pipeline and keep traces; Part 1 of this series (Product #33) covers instrumenting the pipeline itself. If your golden set runs when someone remembers, your incident wall has no class names, or your quality monitoring would sleep through a six-week slide — start at Template 1.

### What this pack is not

- **Not Part 1.** Instrumenting ingest, retrieval, and generation — document passports and lineage, candidate-set capture, rerank-delta logging, trace schemas and correlation IDs, privacy-safe query logging, and retrieval-quality signals — is this book's Chapters 1–6. It lives in the Part 1 companion pack, Product #33; nothing here restates it, and this pack is complete and useful without it.

- **Not general AI observability.** Product #10, AI Observability Templates, covers observability for AI systems in general — traces, logs, metrics, dashboards, and alerting across AI workloads — from a different book. This pack is RAG-pipeline observability and debugging only: golden sets for retrieval, a RAG failure taxonomy, retrieval debugging playbooks, and retrieval-quality SLOs. The two packs do not restate each other.

- **No runnable code or tooling.** The book's harness, classifier, PSI, and span-analyser listings are deliberately small teaching models, labelled as such in the book. The pack ships the templates that specify, operate, and govern these components — not a production replay harness, classifier, drift monitor, or alerting platform.

- **No vendor or product guidance.** No evaluation platform, observability vendor, or monitoring product is selected, installed, or configured here; the templates are tool-agnostic by design.

- **No guarantee of quality, and no benchmark claims.** The book's worked figures — the clause-family recall reconstruction, the reranker demotion counts, the cost and latency shapes — are the book's own illustrative examples from fictionalised case stories, and are labelled illustrative wherever this pack carries them. These templates manage, measure, and bound retrieval risk; they never promise that a regression cannot happen.

- **No compliance crosswalk or certification.** The Part 2 gate in Template 42 is a Chapters 7–12 production gate consolidated from the book's own chapter checklists — not a compliance certification, and no framework mapping ships in this pack.

### Companion products

**RAG Observability & Debugging — Part 1: Instrumenting Ingest, Retrieval & Generation** (Product #33) is this pack's sibling, built from Chapters 1–6 of the same book: a companion for when you need the instrumentation layer underneath this pack's operating discipline — never a requirement for this pack to work. **AI Observability Templates** (Product #10) is the general-AI-observability companion from a different book, for the AI workloads beyond your RAG pipeline. This pack stands alone.

*Note: the $99 Architecture Templates Bundle currently contains Products #1–#3 only. This pack launches standalone; no bundle inclusion is promised for this product.*

Built by the author of **RAG Observability & Debugging: Tracing Answers Back to the Chunk That Caused Them**, from the debugging discipline behind production retrieval systems.

### About the author

Naveen Kumar Josyula is the author of 35 AI books on Amazon KDP and the CEO and founder of AISoltrics Technologies Inc. He is the author of the source book, *RAG Observability & Debugging: Tracing Answers Back to the Chunk That Caused Them*, from which every template in this pack is distilled.

### Support promise

If anything in this file is wrong or unclear, email support@aisoltrics.com before you judge it — we fix verified issues within one business day and send you the corrected file free. (You can also reach the founder's office at aadhya@aisoltrics.com.) Updates to this product are free to past buyers via your library. Sold by AISoltrics Technologies Inc.