Miatz playground

Evals Runner

Where the code grader and the AI judge disagree, look closer.

Coming to the Build-LabAIE-401BYOK AIAI-Coredev · data

The Demystify signature

The disagreement heat-map is the single artifact that explains why 'just use an LLM judge' isn't enough — it shows the exact cases where rubric and model intuition part ways.

How it works

What goes in, what comes out

What it does

Learner builds a 20-case golden set for a task, then runs it through both a deterministic code grader and an LLM-judge simultaneously; every case where the two disagree is highlighted on a disagreement heat-map, exposing exactly the ambiguity a rubric needs to resolve rather than papering over. A grader-hacking mode challenges the learner to craft an output that fools the LLM-judge while still failing the code grader, making eval design itself an adversarial skill.

You bring

A task definition, a 20-case golden dataset (or an auto-suggested draft from the corpus), and a grading rubric.

You get

Pass/fail per case per grader, a judge-vs-code disagreement heat-map, an overall pass rate against threshold, and (in grader-hacking mode) a fooled/not-fooled verdict.

You control

Golden-set size/source, grader type per case (code, LLM-judge, or both), judge model/rubric, and pass threshold.

Where it's used

In-module lab for AIE-401 (MUST/Core, Evaluator badge); also the shared grading engine every other module's eval-gate calls under the hood.

golden datasetscode-based gradersLLM-as-judgerubric designgrader-hacking / adversarial eval design

Optional · watch & try

Go deeper, elsewhere

Hand-picked public explainers and open tools that complement this one — always optional, never required, never graded.

Introduction to LLM Evaluations – Model Evals vs Application EvalsDistinguishes model-level evals from application-level evals Intro to LLM Evaluation w/ OpenAI Evals [Walk-Thru]Hands-on walkthrough building a golden-set eval How to Systematically Setup LLM Evals (Metrics, Unit Tests, LLM-as-a-Judge)Covers code graders vs. LLM-as-judge tradeoffs PromptfooFree open-source tool for running golden-set evals and red-team tests against prompts

Concepts click when you open the machinery.

Three labs are already live and free — the same hands-on style this playground brings to its module.

Play the free labs Take the DSAT