Yu-Gi-Oh is a trading card game with thousands of cards and famously fiddly rules — cards interact in ways even seasoned players argue about, which makes it a genuinely hard test for a small AI model. So I built the same Gemma 2 2B three ways and put them against the same questions: A the model straight out of the box, B after I fine-tuned it on Yu-Gi-Oh questions and answers, and C that fine-tune paired with a retriever that looks up the real card text first. Every answer is scored 0–10 by a blind judge that is handed the correct answer, so the scores are checkable — and the point is simple: fine-tuning teaches the model how to answer; retrieval is what gives it the facts.
Ask one Yu-Gi-Oh question and the same model answers it three ways — base, fine-tuned, and fine-tuned + retrieval. Pick a question by category (those carry a written answer key, so the judge grades against a reference) or type your own (graded from the judge's own knowledge). Then a blind judge scores all three 0–10.
You ask one question; three versions of the same model answer it, and a blind marker who already has the answer key grades all three out of ten.
Not the live arena — the offline evaluation. Each system answered the same 60 decontaminated held-out questions and a reference-grounded blind judge scored every answer 0–10. Shown on two sets: the original (biased toward card-facts and rulings) and a category-balanced unbiased 60. C wins both; only C survives the balanced mix.
Measured mean judge score /10 per system on both held-out sets — the biased original 60 (left) and the balanced unbiased 60 (right). C wins on both; only C holds up when the mix is balanced.
Bootstrap 95% CI is primary; the paired t-test and Wilcoxon cross-check it. Retrieval (B→C) is the strongest, most significant jump on both sets.
Every held-out question is answered by all three systems and graded blind, then simple statistics check the gaps are real and not luck.
Before C answers, a retriever pulls the five most relevant snippets from the corpus and pastes them into the prompt. Two searches run together — dense (by meaning, MiniLM-L6) and BM25 (by exact words) — and are merged with reciprocal-rank fusion. The merged (hybrid) list is what feeds C.
Measured recall@k on the 60 held-out questions for three retrievers. Hybrid (RRF) — the one that feeds C — reaches 0.93 at k=5, the top-5 actually used.
FAISS flat index over 42,412 chunks, top-5 retrieved. Hybrid recall@5 = 0.93.
Two search methods — one by meaning, one by exact words — are merged, and the best five snippets are pasted into the prompt before the model answers.
Every answer is scored by a judge that never sees which system wrote it and grades one answer at a time. It is handed the question's gold answer and source evidence, so scores are checkable against a known-good reference — four dimensions that add up to /10.
The 10-point rubric, split into its four dimensions.
The marker sees one answer next to the correct answer and gives four small scores that add to ten; make up a card and the honesty score drops to zero.
The fine-tune's supervised set is 2,683 grounded QA pairs, distilled by a teacher model from a curated Yu-Gi-Oh corpus and filtered through a blind-judge gauntlet. Two free sources: Yugipedia editorial prose (CC BY-SA 4.0) and YGOPRODeck card facts (used only as labelled fair-use context).
Measured share of the 27.5 MB corpus by source category — tips and rulings dominate, lore and mechanics trail.
78% free-licensed Yugipedia prose (20.7 MB cleaned, clears the 20 MB floor) + 22% card facts. YGOPRODeck snapshot fetched 2026-08-01.
Raw Yu-Gi-Oh text is cleaned, cut into pieces, and turned into question-answer pairs, and only the ones that pass a strict check are kept.
Two bills built this: Modal for the GPU fine-tune and the Gemini API for generating the QA data and running the judge. The base is Google's (imported free); the retriever, serving and evaluation all ran on a local RTX 3060, so they cost nothing.
Measured invoiced spend by bill — Modal GPU $0.26 and Gemini API $3.00; the local RTX 3060 cost $0. Total $3.26.
Modal is the invoiced GPU total for the fine-tune; Gemini is the combined API spend for teacher generation, QA gating and judging. Only the two totals are measured — no per-stage split inside them.
Almost all the money went to the cloud service that wrote and graded the data; the actual training was cents, and everything on the home GPU was free.