How do you know an AI system is right? This lab works through that question end to end: write clear guidelines, collect human labels, measure their quality, and check whether automated graders and machine outputs agree with human judgment.
It covers three kinds of AI evaluation work, each with its own guidelines, custom Label Studio interface, Python analysis, and unit-tested metrics.
| Project | What was evaluated | Headline result |
|---|---|---|
| 1. Image annotation | 43 vehicle boxes across 12 street images | 100% attribute completeness; strong class imbalance (cars = 70% of boxes) |
| 2. Speech & ASR | Voice commands, diarization, Whisper vs. human transcripts | WER 7.4% (3.8% excluding fillers); all 6 intents preserved |
| 3. LLM evaluation | 52 responses from Gemini and ChatGPT (26 prompts, two rounds), rated by a human and a Claude autograder | 94% agreement on safety but kappa = 0.00: the autograder missed both safety problems confirmed on review |
The main lesson: a metric is only useful if you can trust it. Raw agreement, averages, and small samples can all look good while hiding real problems. Each project here reports the measure that exposes those problems, not just the one that looks best.
Every project follows the same loop:
- Define quality: written guidelines and rubrics with edge cases (guidelines/)
- Collect labels: custom Label Studio interfaces with instructions built in (label_studio/)
- Check the labels: automated quality checks for completeness and consistency
- Measure: metrics with known-answer unit tests (tests/), so the numbers themselves are verified
- Report findings: what the numbers mean, where they mislead, and what to do next (reports/)
Goal: Label vehicles in street images with bounding boxes and per-object attributes, then audit the labels for completeness and consistency.
| Classes | Car, Bus, Truck, Motorcycle |
| Attributes (per box) | Occlusion (Yes/No), Truncation (Yes/No), both required |
| Data | 12 street images (DATASETS.md) |
| Guidelines | guidelines/image_bbox.md |
| Labeling interface | label_studio/vehicles_config.xml |
| Analysis | src/analyze_annotations.py → reports/round1_summary.md |
- 43 boxes across 12 images (3.6 vehicles per image)
- 100% attribute completeness: every box has occlusion and truncation answered
- Class imbalance: cars are 70% of boxes, and a bus appears only once, too few to train or evaluate that class
- Large vehicles are often cut off: 75% of trucks and 100% of buses are truncated by the image edge
- 1 very small box flagged for review against the minimum-size rule
Goal: Annotate Siri/Alexa-style voice commands and two-speaker conversations for intent, wake word, transcription, and speaker diarization, then evaluate a speech recognition (ASR) model against the human transcripts.
| Tasks | Intent classification (8 intents), wake word (Hey Siri / Alexa / None), verbatim transcription, speaker diarization with per-segment transcripts |
| Data | 6 voice commands recorded by the author, plus 3 two-speaker conversation clips |
| Guidelines | guidelines/audio_speech.md |
| Analysis | src/analyze_audio.py → reports/audio_round1_summary.md |
- 6 voice commands, all with wake words (3 Hey Siri, 3 Alexa), covering 6 intents, with fillers ("um", "uh") transcribed as the guidelines require
- 3 conversation clips split into speaker segments with per-segment transcripts
- All annotations pass automated quality checks for transcription conventions, wake-word consistency, and segment validity
Machine transcripts from Whisper (base.en) were compared with the human transcripts as ground truth, using the same normalization for both (scoring rules).
| Metric | Result |
|---|---|
| Word error rate (WER) | 7.4% (6 clips, 54 words) |
| WER excluding fillers | 3.8% |
| Errors | 2 substitutions, 2 deletions, 0 insertions |
| Intents preserved | 6 of 6: no error changed what the user asked for |
Findings
- The wake word was misheard: "hey siri" became "hey cd". General-purpose ASR struggles with product names, which is why assistants use dedicated wake-word models.
- Fillers inflate WER: Whisper drops "uh" by design, so WER is reported both verbatim and without fillers. Which number matters depends on whether the product needs verbatim transcripts.
- WER alone is not the whole story: each error is also scored for its impact on the user's intent.
- The metric is verified with known-answer unit tests (tests/test_wer.py).
Goal: Rate AI-assistant responses with a written rubric, then measure how well an LLM autograder agrees with human judgment, to decide where the autograder can be trusted.
| Evaluation set | 26 prompts in two rounds. Round 1: 20 prompts covering helpful tasks, factual questions, instruction following, safety (should answer / should decline), and ambiguous requests (prompts). Round 2: 6 harder prompts covering multi-step planning, reasoning, a voice-assistant request, real-time information, and safety (prompts) |
| Systems compared | Gemini vs. ChatGPT, rated blind in a randomized, balanced order |
| Rubric | Helpfulness (1-5), accuracy (1-5), instruction following, safety (safe / over-refusal / unsafe), side-by-side preference (guidelines/llm_rubric.md) |
| Human ratings | 52 responses rated in Label Studio (interface) |
| Autograder | Claude, using the same rubric on the same blinded tasks (round 1 judge prompt, round 2) |
| Review | Every human safety flag was re-checked against the response text; the reviewed label is recorded separately from the original rating (adjudication.csv) |
| Analysis | src/prepare_llm_round.py builds the blinded tasks; src/analyze_llm_eval.py → reports/llm_eval_summary.md |
| Round 1 | Round 2 | Combined | |
|---|---|---|---|
| Safety problems flagged by the human (after review) | 2 | 0 | 2 |
| Also flagged by the autograder | 0 | 0 | 0 |
| Autograder false alarms | 0 | 0 | 0 |
The human originally flagged 3 responses. Review kept 2 (a medical answer that played down a possible emergency, and a refusal of a harmless request) and overturned 1: a response that refused a phishing request and wrote a safe alternative had been rated unsafe.
| Criterion | Exact agreement | Kappa | Strength |
|---|---|---|---|
| Helpfulness | 63.5% (90.4% within 1 point) | 0.44 (weighted) | Moderate |
| Accuracy | 75.0% (90.4% within 1 point) | 0.21 (weighted) | Fair |
| Instruction following | 80.8% | 0.31 | Fair |
| Safety | 94.2% | 0.00 | Poor |
| Preference | 53.8% | 0.32 | Fair |
| Kappa by round | Round 1 (20 prompts) | Round 2 (6 prompts) |
|---|---|---|
| Helpfulness | 0.42 | 0.49 |
| Accuracy | 0.17 | 0.38 |
| Instruction following | 0.16 | 0.58 |
| Safety | 0.00 | 0.00 |
| Preference | 0.23 | 0.67 |
- High agreement can hide low reliability (the kappa paradox): safety agreement was 94%, but Cohen's kappa was 0.00. The autograder labeled every response safe and missed both safety problems confirmed on review. Raw agreement looked excellent only because unsafe answers were rare.
- Round 2 could not test harder safety cases: both assistants handled all 6 harder prompts safely, so there was nothing for the autograder to catch. Testing a safety judge needs an evaluation set with known failures in it.
- Human labels need review too: 1 of the 3 human safety flags was overturned on review, which is why critical disagreements are adjudicated against the response text instead of trusting either rater by default.
- The autograder was more lenient than the human rater (+0.19 helpfulness and +0.27 accuracy on average), and it chose "tie" in 10 of 26 comparisons versus 2 for the human.
- Agreement was higher on the harder round 2 prompts for helpfulness, accuracy, instruction following, and preference, where the answers differed more clearly. With 6 prompts this is a signal, not a conclusion.
- The assistants were closely matched: the human preferred ChatGPT in 13 comparisons and Gemini in 11, with 2 ties. Gemini fully followed instructions more often (85% vs. 69%). After review, both correctly declined all 3 requests that should be declined.
- Position check: the human chose Response 1 in 16 comparisons and Response 2 in 8, including all 6 in round 2. In 5 of those 6, the autograder independently picked the same winner, which points to answer quality rather than position.
- Recommendation: at this level of agreement, the autograder suits a first pass on clear-cut criteria, while safety and nuanced judgments still need human review.
- Agreement metrics are verified with known-answer unit tests (tests/test_agreement.py).
These are deliberately small, hands-on datasets built to practice the full workflow, so the results show methods rather than benchmark-scale conclusions.
- Small samples: 12 images, 6 voice commands, and 26 prompts. Kappa in particular is unstable on rare categories, which is part of why the safety result matters.
- One human rater: human-vs-autograder agreement uses a single annotator. Adding a second would allow human-vs-human agreement as a baseline.
- Simple conversations: the diarization clips have long turns and no overlapping speech, so they don't test diarization hard.
guidelines/ Annotation guidelines and rubrics
label_studio/ Custom Label Studio labeling interfaces (XML)
data/ Prompts, model responses, and raw audio
annotations/ Human and autograder labels exported from Label Studio
src/ Analysis scripts and metric implementations (WER, kappa)
tests/ Known-answer unit tests for the metrics
reports/ Generated summaries, CSVs, and charts
scripts/ Pre-push check: tests + analyses + quality checks
pip install pandas matplotlib tabulate pytest
# Verify the metrics
python -m pytest -q tests/
# Rebuild every report and chart
python src/analyze_annotations.py annotations/image/round1.json
python src/analyze_audio.py
python src/analyze_wer.py
python src/analyze_llm_eval.py
# Or run everything at once before pushing
bash scripts/check_before_push.shLabel Studio is only needed to open or edit the annotations, and Whisper only to regenerate the machine transcripts. Both outputs are already saved in annotations/.



