Measured quality

Evals

Anyone can build a demo. The harder, rarer skill is proving it works — and catching it when it breaks. Here are real numbers from the AI Data Copilot, measured against a reference-SQL golden set, plus the reusable harness I use to keep quality from drifting.

Self-correction loop

The loop re-runs and repairs a failed query. Toggle it to see the measured effect on the golden set.

91.7%
Execution accuracy
11/12 answerable queries
100%
Refusal correctness
every out-of-scope query declined
+8.4
Points from self-correction
83.3% → 91.7%

Golden set · 12 answerable queries

11 pass · 1 fail

✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✕

Highlighted cell is the query the self-correction loop repaired. Comparison is order-independent against reference SQL; runs are deterministic.

Reusable harness

What the evaluation harness measures

The same suite runs as a regression check, so a model or prompt change can't quietly degrade quality.

Retrieval

  • Hit-rate

    Did the right passage make it into the retrieved set?

  • MRR

    How high did the right passage rank?

Generation

  • Faithfulness

    Is the answer grounded in the retrieved context?

  • Answer-relevance

    Does the answer actually address the question?

  • Refusal-rate

    Does it decline when it should, instead of guessing?

Why this matters

Evaluation is what separates a system you can put in front of users from a demo you hope holds up. Deterministic runs make results reproducible, a golden set makes regressions visible, and refusal metrics make sure the system declines when it should instead of guessing. That's the difference between “it worked when I tried it” and “here's how I know it works.”

See the code on GitHub →