LLM eval gate
An LLM eval gate is a CI check that runs a versioned eval suite against a committed golden-file baseline and blocks the merge on any score regression. Caisson's @caisson/ai-evals package compares each eval's mean and per-scorer scores to a JSON baseline file, fails closed when no baseline is committed, and rewrites it only through an explicit BLESS re-baseline step.
In code
export function compareToBaseline(
run: EvalRun,
baseline: BaselineFile,
): BaselineComparison {
const findings: RegressionFinding[] = [];
const prior = baseline.evals[run.name];
if (prior === undefined) {
findings.push({
kind: "missing-baseline",
actual: run.score,
detail: `no committed baseline for eval "${run.name}"`,
});
return { eval: run.name, passed: false, findings, blessed: false };
}
if (run.score + EPS < prior.score) {
findings.push({
kind: "score-regression",
actual: run.score,
baseline: prior.score,
detail: `score ${run.score} worse than baseline ${prior.score}`,
});
}
// ... scorer-level, fewer-cases, and Wilson-CI-floor checks follow the same pattern
return { eval: run.name, passed: findings.length === 0, findings, blessed: false };
}How it holds
Fails closed with no committed baseline
compareToBaseline flags an eval with no matching baseline entry as missing-baseline and fails it outright; the fix is to bless it into existence, never to let an unbaselined eval pass by default.
Checks the mean, every named scorer, and the case count
A run's aggregate score, each individual scorer's mean, and the dataset's case count are all checked against the committed baseline; a shrunk dataset is flagged too, since fewer cases can flatter a mean without the suite actually improving.
One sanctioned rewrite path
BLESS=1 bun run eval is the only way the baseline file changes; every other invocation only compares and never writes, so a baseline update always lands as a reviewable diff in the PR.
An opt-in Wilson-CI floor beyond the mean
wilsonFloor gates the lower confidence bound of a scorer's pass rate on top of the raw threshold, catching a lucky small-sample draw that a flattering mean would let through.