RAG Eval Free Sample — synthetic toy metrics loop

A tiny free ($0) taste of offline RAG retrieval eval: 4 synthetic docs, 4 gold QAs, and one stdlib Python script that prints Precision@k, Recall@k, and Hit@k. Toy lexical retriever only. Not production accuracy.

You can chat with your docs and still have no idea whether retrieval improved.

This free sample is a minimal offline loop so you can see P@k / R@k / Hit@k on fake data before buying a fuller pack.

What’s inside

• docs.jsonl — 4 synthetic policy/FAQ snippets

• qa.jsonl — 4 questions with gold_doc_ids

• eval_metrics.py — Python 3 standard library only (no pip)

• README.md — metric definitions + honest limits

Run

python3 eval_metrics.py

Honest limits

• Synthetic / educational only

• Toy word-overlap retriever (not embeddings, not a vector DB)

• Faithfulness stays N/A (no generator / judge)

• Not a production RAG platform and not an accuracy guarantee

• Do not put real customer data or secrets in the shareable folder

Want more after you try it?

• Free article: https://dev.to/dicardo9/practice-rag-retrieval-metrics-offline-a-tiny-stdlib-eval-loop-synthetic-data-dm3

• Lite $2.99 — larger scaffold: https://payhip.com/b/cmjJk

• Field Card $4.99 — no-code metrics reading card + checklist: https://payhip.com/b/Lrw6Y

• Pro $11.99 — chunk vs rerank harness: https://payhip.com/b/UdQ0M

This Free Sample is $0. Educational synthetic harness only — not a production RAG stack.

Author: Dicardo9 / Moxuan Ding (I made this).