# The eval set you keep meaning to write is already in your logs

Every team I have worked with has the same empty file somewhere in the repo: `evals/cases.jsonl`, three examples deep, last touched the week someone decided evals mattered. Writing more feels like homework. You sit down to invent test prompts, and everything you type is either too obvious or too contrived - because you are guessing at what your users ask instead of remembering it.

Here is the thing that quietly bothered me into writing a tool: **the cases you are trying to invent already happened.** They are in your logs. Every prompt a real user sent, every response your model returned, sitting in a JSONL file nobody opens. The golden set is not missing. It is un-mined.

This post is about that gap between raw logs and a usable eval set, why closing it by hand does not scale, and a small offline tool I wrote - **casemine** - that reads your LLM logs and hands back a compact, deduplicated, clustered golden set you can refine and run.

### Why your logs are not a golden set yet

Point someone at a month of production traffic and tell them "these are your eval cases" and they will, correctly, laugh. Raw logs are the raw material, not the product. Three things stand between them and a golden set:

*   **Volume with almost no variety.** A thousand log lines might represent six actual things users do. The same intent shows up hundreds of times with tiny surface differences.
    
*   **Volatile noise on every line.** `"summarize order #48213 for jay@acme.com"` and `"summarize order #90117 for lee@corp.io"` are the *same case*. The order number and the email are variables, not distinctions - but a naive dedup sees two unique strings and keeps both.
    
*   **No structure.** Even if you dedup perfectly, you are left with a flat list. You cannot see that 60% of traffic is one intent and 3% is another, so you cannot sample fairly.
    

Hand-curating past this is the part nobody finishes. You scroll, you copy a few lines that look "representative," you get bored around line 200, and the cases you picked skew toward whatever was on screen. It is not a golden set. It is a convenience sample with extra steps.

### The idea: template, dedup, cluster, sample

casemine is a five-step pipeline, and every step is boring, deterministic, and offline - no model calls, no network, nothing leaves your machine. It reads logs JSONL in, it writes a golden set JSONL out.

The one insight the whole thing rests on: **normalize each prompt into a template before you compare anything.** Mask the parts that vary - numbers, UUIDs, emails, timestamps, long IDs - and the two "different" order-summary prompts above collapse into one shape:

```text
summarize order #<NUM> for <EMAIL>
```

Once prompts are templates, everything downstream gets easy. Near-identical prompts dedup cleanly. Prompts that share a template are, by definition, the same intent - so clustering is just grouping by template. And picking representatives is picking the most common real prompts inside each group.

### Walking a real pile of logs through it

Say you export `logs.jsonl` from your support assistant - 200 lines, looking roughly like this:

```json
{"prompt": "summarize order #48213 for jay@acme.com", "response": "Order #48213 ..."}
{"prompt": "summarize order #90117 for lee@corp.io",  "response": "Order #90117 ..."}
{"prompt": "where is my refund for order 55231",        "response": "Your refund ..."}
{"prompt": "cancel subscription 7c2f-...-a1, effective today", "response": "Done ..."}
```

Two hundred lines, but only a handful of real things going on. Run casemine over it:

```bash
casemine logs.jsonl -o golden.jsonl \
  --min-cluster-size 5 \
  --per-cluster 3 \
  --max-cases 50
```

**Step 1 - templatize.** Each prompt is normalized: `#48213` and `#90117` become `<NUM>`, the emails become `<EMAIL>`, the subscription UUID becomes `<UUID>`. Volatile spans out, shape in.

**Step 2 - dedup.** Near-identical prompts (by shingle hash of the template) collapse. The 200 lines fall to a much smaller set of distinct shapes, and casemine remembers how many raw logs each shape stood for.

**Step 3 - cluster.** Group by template. Three intents rise to the top - *summarize an order*, *refund status*, *cancel a subscription* - plus a long tail of one-off shapes. Clusters smaller than `--min-cluster-size` get dropped as noise.

**Step 4 - sample.** From each surviving cluster, take the `--per-cluster` most frequent (most central) real prompts as representatives, capped at `--max-cases` total.

**Step 5 - emit.** Write the golden set. Here 200 logs became **3 clusters → 9 representative cases**, and the run tells you so:

```text
casemine: mined golden set
  logs read ......... 200
  after dedup ....... 41   (dedup ratio 0.80)
  clusters kept ..... 3    (2 dropped below --min-cluster-size)
  cases written ..... 9
  → golden.jsonl  (format: goldrun)
```

Each emitted case looks like this:

```json
{
  "id": "cluster-1-rep-2",
  "prompt": "summarize order #48213 for jay@acme.com",
  "expected": "Order #48213 shipped on ...",
  "expected_status": "REVIEW",
  "assertions": []
}
```

Note the `expected` field. casemine seeds it from the response that was actually logged for that prompt - because that is your best first guess at the right answer - and then flags it **REVIEW**, loudly, on every single case. That flag is not decoration. It is the whole contract.

## Why `expected` is a draft, not a truth

This is the honesty I want to be blunt about: **the logged response is what your model said, not what it should have said.** If your model was wrong in production, casemine will faithfully seed a wrong `expected`. Shipping that as ground truth would be worse than having no eval at all - you would be locking in the bug as the passing answer.

So `expected` arrives as a draft with `REVIEW` stamped on it. The workflow is: casemine does the mining - the tedious, high-volume, error-prone part of finding and grouping what to test - and you do the judgment, going case by case to confirm or correct each expected answer and add assertions. That division is deliberate. Machines are good at deduping 200 lines into 9. Humans are good at knowing whether the answer is actually right. casemine never pretends to do the second job.

The clustering is honest about its limits too. It matches on **templates and shingles, not meaning** - two prompts that ask the same thing in totally different words will not merge. Sampling by frequency is a heuristic, not a proof of coverage. What you get is a strong, representative first draft that would have taken a day to assemble by hand, not an oracle.

![ Upload image here: flow.svg ](https://cdn.hashnode.com/uploads/covers/6a548baa47c6120a79511a69/738551cf-bc47-476a-876b-f55caa773ec5.svg align="center")

### Where goldrun picks up

casemine **builds** a golden set. It does not run one. That is a different tool on purpose.

If you have read my post on [goldrun](https://jaytank.hashnode.dev/goldrun-llm-golden-set-regression-eval), this is the piece that was missing from it. goldrun answers "did this prompt change regress my cases?" - it runs a golden set, scores it, and fails CI on a regression. But it always assumed you *had* a golden set. casemine is where that set comes from. Emit with `--format goldrun` and the output drops straight into goldrun's runner:

```bash
casemine logs.jsonl -o golden.jsonl --format goldrun
# ... you review every expected, add assertions ...
goldrun golden.jsonl
```

One mines the cases out of production; the other guards them in CI. casemine → review → goldrun. You never invent a test prompt again - you curate the ones your users already wrote.

### Try it

casemine is a single Python CLI, offline, standard-library-lean, no keys and no network:

```bash
pip install casemine

casemine logs.jsonl -o golden.jsonl      # mine a golden set
casemine logs.jsonl --per-cluster 5 --max-cases 100
casemine logs.jsonl --format jsonl --json  # machine-readable summary
```

It exits `0` on a clean run and `2` on a usage or input error, so it slots into a Makefile or a nightly job that re-mines from fresh logs. It is MIT-licensed and on GitHub: [**https://github.com/jay-tank/casemine**](https://github.com/jay-tank/casemine)

### The honest limits

To keep the trade explicit:

*   **Templating is regex-driven, not semantic.** It masks the volatile spans it knows about - numbers, UUIDs, emails, timestamps, long IDs. An unusual identifier format may slip through and split a cluster that should be one.
    
*   **Clustering is by template, not intent.** Same question, different phrasing, different template - casemine will not merge them. It groups surface shape, and surface shape is usually a good proxy for intent, not a guarantee of it.
    
*   `expected` **is seeded from the log and is always wrong until reviewed.** The `REVIEW` flag is there to force the one step a machine cannot do for you.
    
*   **Sampling is a frequency heuristic.** "Most frequent" is a reasonable definition of "representative," but it will under-sample the rare, important edge case. Mine, then look at the tail yourself.
    

None of these are bugs to be fixed later. They are the line casemine draws between what a deterministic offline tool should decide and what a human should. The point was never to auto-generate a perfect eval set. It was to stop staring at an empty `cases.jsonl` and start from the thousands of real cases you already have.

Your best eval cases are already sitting in your logs. casemine just digs them out.