Yu-Gi-Oh is a trading card game with thousands of cards and famously fiddly rules — cards interact in ways even seasoned players argue about, which makes it a genuinely hard test for a small AI model. So I built three of them and put them against the same questions: (A) Google's Gemma 2 2B straight out of the box; (B) the same model after I fine-tuned it on Yu-Gi-Oh questions and answers; and (C) that fine-tune paired with a retriever that looks up the real card text before it answers. Every answer is scored 0–10 by an AI judge that is handed the correct answer, so the scores are checkable — and the whole point is simple: fine-tuning teaches the model how to answer; retrieval is what gives it the facts.
Each system is the same Gemma 2 2B — fine-tune, then add retrieval. Click any one to open its own site (training details, corpus, cost, evaluation) in a new tab.
Pick a question by category — those carry a written answer key, so the AI judge scores each answer against a reference — or type your own, which the judge grades from its own knowledge.
The offline evaluation, not the live arena: each system answered the same 60 decontaminated held-out questions and a reference-grounded blind judge scored every answer 0–10. The component columns split that score into the four rubric dimensions.
This original held-out set is not category-balanced: its 60 questions split as card-facts 45% (27), rulings 45% (27), archetype 8% (5) and mechanics 2% (1) — so rulings and card-facts dominate the average. An unbiased, equal-split evaluation is shown in the second table below.
| # | System | Mean /10 | Correct /5 | Compl /2 | Ground /2 | Clarity /1 | n |
|---|
A second, category-balanced evaluation: 15 questions from each of the four categories (card-facts, rulings, archetype, mechanics), 60 total, scored by the same blind reference-grounded judge. The closed-book systems score lower here — the balanced mix surfaces more niche-fact categories they never learned — so retrieval's advantage is even larger.
| # | System | Mean /10 | Correct /5 | Compl /2 | Ground /2 | Clarity /1 | n |
|---|
| Set | Contrast | Δ mean | 95% CI | t p | Wilcoxon p | Verdict |
|---|---|---|---|---|---|---|
| Biased 60 | A → B fine-tuning | +1.27 | [0.42, 2.17] | 0.007 | 0.012 | significant |
| Biased 60 | B → C retrieval | +2.80 | [1.72, 3.87] | <0.001 | <0.001 | significant |
| Unbiased 60 | A → B fine-tuning | +0.68 | [0.07, 1.33] | 0.041 | 0.042 | marginal |
| Unbiased 60 | B → C retrieval | +5.63 | [4.63, 6.58] | <0.001 | <0.001 | significant |
| Method | @1 | @3 | @5 | @10 |
|---|---|---|---|---|
| Dense (MiniLM-L6) | 0.62 | 0.87 | 0.92 | 0.95 |
| BM25 (lexical) | 0.82 | 0.85 | 0.87 | 0.90 |
| Hybrid (RRF) — feeds C | 0.72 | 0.92 | 0.93 | 0.95 |
The gain is concentrated in groundedness (0.18 → 0.87 → 1.65): fine-tuning teaches the shape, retrieval supplies the facts — correctness only jumps (2.35 → 3.85) once real passages are in the prompt.
Every answer is scored by a Gemini judge (gemini-3.1-flash-lite) that never sees which system produced it, and scores one answer at a time so nothing is flattered by comparison. Each held-out question is handed to the judge with its source evidence and a gold answer, so the score is checkable against a known-good reference rather than a popularity contest.
Four dimensions add up to 10, so a nearly-right answer separates cleanly from a hopeless one:
| Dimension | Points | What it measures |
|---|---|---|
| Correctness | 0–5 | Factual agreement with the gold answer |
| Completeness | 0–2 | Covers the key points, not just one |
| Groundedness | 0–2 | No invented cards, figures or citations — fabrication forces this to 0 |
| Clarity | 0–1 | Answers what was asked, without padding |
The fine-tune's supervised set is 2,683 grounded QA pairs, distilled by a teacher LLM from a curated Yu-Gi-Oh corpus and filtered through a blind-judge gauntlet. That corpus is two free sources: Yugipedia editorial prose (CC BY-SA 4.0, attributed) and YGOPRODeck card facts. Card effect text is © Konami, used only as labelled fair-use context — never a verbatim-recall target.
| Source | Role | License | Size | Share |
|---|---|---|---|---|
| Yugipedia · tips | rulings & interactions | CC BY-SA 4.0 | 8.47 MB | 31% |
| Yugipedia · rulings | rulings | CC BY-SA 4.0 | 5.90 MB | 22% |
| YGOPRODeck · card facts | card identity (stats + fair-use text) | free / © Konami | 6.04 MB | 22% |
| Yugipedia · archetype | mechanics | CC BY-SA 4.0 | 2.74 MB | 10% |
| Yugipedia · lore (ep + char) | lore | CC BY-SA 4.0 | 3.92 MB | 14% |
| Yugipedia · mechanics | glossary | CC BY-SA 4.0 | 0.40 MB | 1% |
78% free-licensed Yugipedia prose (20.7 MB cleaned, clears the 20 MB floor) + 22% card facts. Of the prose, ~82% is on-target rulings/interactions and ~18% is lore. Card knowledge is a YGOPRODeck snapshot fetched 2026-08-01.
Two bills built this: Modal for the GPU training, and the Gemini API for generating the QA data and running the judge. The base is Google's (imported free); the retriever, serving and evaluation all ran on a local RTX 3060, so they cost nothing.
| Bill | Cost | Share |
|---|
Modal is the invoiced GPU total for the fine-tune; Gemini is the combined API spend for teacher generation, QA gating and judging. We don't attribute a per-stage split inside those two bills — only the totals are measured.