MIS 752 · Lab 7 · LienJ

Catch the AI Lying: an audited pass/fail harness for one small free model

Ten gold clinical question/answer pairs, one small free model, three rulers (exact match, a lenient contains check, and an LLM-as-judge run twice), and the measured reliability of the ruler itself. Built in Google Colab on OpenRouter's free tier: 42 requests, $0, no PHI. These numbers are this run's and will vary.

The 30-second read

THE GOLD STANDARD items : 10 (4 numeric, 3 code-lookup, 1 short-phrase, 1 yes-no, 1 one-word) written before any model output was looked at -- that ordering is what makes it a standard instead of a rationalisation file : data/gold_qa.csv THE MODEL UNDER TEST model : liquid/lfm-2.5-2.6b:free at temperature 0.0 judge : nvidia/nemotron-3-ultra-550b-a55b:free, two passes at 0.7 and 1.1 requests used : 42 this session, of ~50 free per day API cost : $0.000000 (every model here is on OpenRouter's free tier, so the cost column is $0 by construction -- which is exactly why cost is in this table and not on a chart axis) ACCURACY ON THE SAME 10 ITEMS, THREE RULERS exact match : 6/10 = 60% deterministic, free, brittle contains gold : 6/10 = 60% lenient, false PASSES judge, pass A : 7/10 = 70% reads meaning judge, pass B : 7/10 = 70% same questions, new dice IS THE RULER ITSELF RELIABLE? (the part almost nobody ships) judge vs itself : agreement 1.0, kappa 1.0 = almost perfect exact vs judge : agreement 0.9, kappa 0.7826 = substantial mean |score gap| : 0.00 points, over the 10 of 10 items scored by both passes unparseable : 0/10 judge replies ignored the output format resolution : one flipped verdict moves accuracy by 10 points at n=10 DISAGREEMENT DIRECTION judge passed what exact match failed : 1 judge failed what exact match passed : 0 STEP 5 -- YOUR WEEK 4 ANSWERS, RE-SCORED 12 of your Lab 4 answers re-scored; Week 4's keyword rubric averaged 82/100 and this week's judge 61/100, with 4 row(s) 25 or more points apart.

Your judge repeated itself on every item

One row per gold item. The filled circle is judge pass A and the open square is pass B, each at its true 0–10 score, so when the passes agree the circle sits inside the square. Rows are sorted so the biggest score gap is on top, and the column on the right gives pass B minus pass A. Each row label also says whether exact match passed that item. A score the judge never gave sits in its own “no score” lane, never at zero. A judge that cannot repeat itself is not a measuring instrument, and this is the cheapest way to find out.

Exact match is the brittle one

Every ruler’s verdict on every gold item, side by side: exact match, the lenient contains check, and both judge passes, with the judge’s own 0–10 score in its cells. Each cell spells out its verdict, so the grid reads in greyscale. Rows flagged on the right are where the rulers disagree — the rows to read yourself.

The pass/fail table

The week’s deliverable: every gold item, what the model said, and what each ruler decided. Read the scorers_agree and judge_repeated columns first — the False rows are where the information is.

id type gold answer exact_verdict contains_verdict judge_a score_a judge_b score_b judge_repeated scorers_agree
icd10_t2dm code_lookup E11.9 E11.9 PASS PASS PASS 10 PASS 10 True True
icd10_ami code_lookup I21.9 I20.0 FAIL FAIL FAIL 0 FAIL 0 True True
snomed_t2dm code_lookup 44054006 <|tool_call_start|>[google(query='SNOMED CT Di FAIL FAIL FAIL 0 FAIL 0 True True
acetaminophen_max_g numeric 4 4 PASS PASS PASS 10 PASS 10 True True
a1c_8_eag_mgdl numeric 183 183 PASS PASS PASS 10 PASS 10 True True
hypo_level1_mgdl numeric 70 70 PASS PASS PASS 10 PASS 10 True True
hipaa_safe_harbor numeric 18 6 FAIL FAIL FAIL 0 FAIL 0 True True
warfarin_reversal_agent short_phrase 4-factor prothrombin complex concentrate Prothrombin complex concentrate (PCC) FAIL FAIL PASS 10 PASS 10 True False
sjw_warfarin_yesno yes_no NO NO PASS PASS PASS 10 PASS 10 True True
atorva80_intensity one_word high high PASS PASS PASS 10 PASS 10 True True

Auditing the instrument

comparison n agreement disagreements unparseable kappa kappa reads as mean |score gap|
exact match vs judge (pass A) 10 0.9 1 0 0.7826 substantial n/a (two different rulers)
judge pass A vs judge pass B 10 1.0 0 0 1.0000 almost perfect 0.00 points, over the 10 of 10 items scored by both passes
judge_a PASS FAIL UNPARSEABLE
exact_verdict
FAIL 1 3 0
PASS 6 0 0

Confusion matrix: rows are the exact-match verdict, columns the judge pass A verdict. Off-diagonal cells are the disagreements; UNPARSEABLE is a column, not a dropped row, so an instrument failure can never flatter the accuracy.

Your Week 4 answers, re-scored

The same answers you graded in Lab 4 with a keyword rubric, now graded by this week’s judge against a written reference answer. Both series are on a 0–100 scale, sorted so the biggest gap is on top, with the gap in points written on each row. 12 of your Lab 4 answers re-scored; Week 4's keyword rubric averaged 82/100 and this week's judge 61/100, with 4 row(s) 25 or more points apart.

How this was measured

Ten gold question/answer pairs were written before any model output was looked at and saved as data/gold_qa.csv. One model (liquid/lfm-2.5-2.6b:free) answered all ten at temperature 0.0 through OpenRouter, with a fixed system prompt that is identical for every item, because the prompt is part of the instrument. Three rulers scored the same answers. Exact match normalizes case, Unicode look-alikes, Markdown emphasis, answer labels, surrounding quotes and trailing punctuation, then requires the whole string to equal the gold answer or one of its accepted variants. Contains-gold is a deliberately lenient diagnostic that produces false passes. The LLM judge (nvidia/nemotron-3-ultra-550b-a55b:free) reads the question, the gold answer and the candidate, and returns VERDICT / SCORE / REASON in three lines; the parser clamps the score to 0-10 and returns UNPARSEABLE rather than guessing when the format was not respected. The judge was run twice, at temperature 0.7 and again at 1.1, and the two passes are compared with raw agreement and Cohen's kappa (chance-corrected, read on the Landis and Koch scale). Kappa bands are a convention, not a law, and at n=10 a single flipped item moves accuracy by 10 points, so treat every number here as a pilot measurement with its denominator attached. Every model called is on OpenRouter's free tier at $0 per token, verified against the live catalog at run time; prices are fetched, never hard-coded. Answers and verdicts are cached to disk, so re-running a cell never spends quota twice. The harness counts requests against a daily budget and refuses to exceed it. This is a ten-item teaching pilot, not a clinical validation. The gold items are self-contained vignettes with no PHI and no real patients, and nothing here is medical advice or a cleared clinical device.