Skip to content

About

Data annotation and quality evaluation with Label Studio and Python: image, voice-assistant speech annotation (audio), and ASR evaluation (WER).

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

11 Commits

Folders and files

Repository files navigation

Annotation Eval Lab

How do you know an AI system is right? This lab works through that question end to end: write clear guidelines, collect human labels, measure their quality, and check whether automated graders and machine outputs agree with human judgment.

It covers three kinds of AI evaluation work, each with its own guidelines, custom Label Studio interface, Python analysis, and unit-tested metrics.

Python Label Studio Tests


Results at a glance

Project What was evaluated Headline result
1. Image annotation 43 vehicle boxes across 12 street images 100% attribute completeness; strong class imbalance (cars = 70% of boxes)
2. Speech & ASR Voice commands, diarization, Whisper vs. human transcripts WER 7.4% (3.8% excluding fillers); all 6 intents preserved
3. LLM evaluation 52 responses from Gemini and ChatGPT (26 prompts, two rounds), rated by a human and a Claude autograder 94% agreement on safety but kappa = 0.00: the autograder missed both safety problems confirmed on review

The main lesson: a metric is only useful if you can trust it. Raw agreement, averages, and small samples can all look good while hiding real problems. Each project here reports the measure that exposes those problems, not just the one that looks best.


Evaluation approach

Every project follows the same loop:

  1. Define quality: written guidelines and rubrics with edge cases (guidelines/)
  2. Collect labels: custom Label Studio interfaces with instructions built in (label_studio/)
  3. Check the labels: automated quality checks for completeness and consistency
  4. Measure: metrics with known-answer unit tests (tests/), so the numbers themselves are verified
  5. Report findings: what the numbers mean, where they mislead, and what to do next (reports/)

Project 1: Vehicle bounding-box annotation (image)

Goal: Label vehicles in street images with bounding boxes and per-object attributes, then audit the labels for completeness and consistency.

Classes Car, Bus, Truck, Motorcycle
Attributes (per box) Occlusion (Yes/No), Truncation (Yes/No), both required
Data 12 street images (DATASETS.md)
Guidelines guidelines/image_bbox.md
Labeling interface label_studio/vehicles_config.xml
Analysis src/analyze_annotations.py → reports/round1_summary.md

Results

  • 43 boxes across 12 images (3.6 vehicles per image)
  • 100% attribute completeness: every box has occlusion and truncation answered
  • Class imbalance: cars are 70% of boxes, and a bus appears only once, too few to train or evaluate that class
  • Large vehicles are often cut off: 75% of trucks and 100% of buses are truncated by the image edge
  • 1 very small box flagged for review against the minimum-size rule

Boxes per class


Project 2: Voice assistant speech annotation (audio)

Goal: Annotate Siri/Alexa-style voice commands and two-speaker conversations for intent, wake word, transcription, and speaker diarization, then evaluate a speech recognition (ASR) model against the human transcripts.

Tasks Intent classification (8 intents), wake word (Hey Siri / Alexa / None), verbatim transcription, speaker diarization with per-segment transcripts
Data 6 voice commands recorded by the author, plus 3 two-speaker conversation clips
Guidelines guidelines/audio_speech.md
Analysis src/analyze_audio.py → reports/audio_round1_summary.md

Annotation results

  • 6 voice commands, all with wake words (3 Hey Siri, 3 Alexa), covering 6 intents, with fillers ("um", "uh") transcribed as the guidelines require
  • 3 conversation clips split into speaker segments with per-segment transcripts
  • All annotations pass automated quality checks for transcription conventions, wake-word consistency, and segment validity

Speaking time per speaker

ASR evaluation: Whisper vs. human transcripts (WER)

Machine transcripts from Whisper (base.en) were compared with the human transcripts as ground truth, using the same normalization for both (scoring rules).

Metric Result
Word error rate (WER) 7.4% (6 clips, 54 words)
WER excluding fillers 3.8%
Errors 2 substitutions, 2 deletions, 0 insertions
Intents preserved 6 of 6: no error changed what the user asked for

Findings

  • The wake word was misheard: "hey siri" became "hey cd". General-purpose ASR struggles with product names, which is why assistants use dedicated wake-word models.
  • Fillers inflate WER: Whisper drops "uh" by design, so WER is reported both verbatim and without fillers. Which number matters depends on whether the product needs verbatim transcripts.
  • WER alone is not the whole story: each error is also scored for its impact on the user's intent.
  • The metric is verified with known-answer unit tests (tests/test_wer.py).

ASR word error rate per clip


Project 3: LLM response evaluation (human vs. autograder)

Goal: Rate AI-assistant responses with a written rubric, then measure how well an LLM autograder agrees with human judgment, to decide where the autograder can be trusted.

Evaluation set 26 prompts in two rounds. Round 1: 20 prompts covering helpful tasks, factual questions, instruction following, safety (should answer / should decline), and ambiguous requests (prompts). Round 2: 6 harder prompts covering multi-step planning, reasoning, a voice-assistant request, real-time information, and safety (prompts)
Systems compared Gemini vs. ChatGPT, rated blind in a randomized, balanced order
Rubric Helpfulness (1-5), accuracy (1-5), instruction following, safety (safe / over-refusal / unsafe), side-by-side preference (guidelines/llm_rubric.md)
Human ratings 52 responses rated in Label Studio (interface)
Autograder Claude, using the same rubric on the same blinded tasks (round 1 judge prompt, round 2)
Review Every human safety flag was re-checked against the response text; the reviewed label is recorded separately from the original rating (adjudication.csv)
Analysis src/prepare_llm_round.py builds the blinded tasks; src/analyze_llm_eval.py → reports/llm_eval_summary.md

Does the autograder catch safety problems?

Round 1 Round 2 Combined
Safety problems flagged by the human (after review) 2 0 2
Also flagged by the autograder 0 0 0
Autograder false alarms 0 0 0

The human originally flagged 3 responses. Review kept 2 (a medical answer that played down a possible emergency, and a refusal of a harmless request) and overturned 1: a response that refused a phishing request and wrote a safe alternative had been rated unsafe.

Human vs. autograder agreement (52 responses, original ratings)

Criterion Exact agreement Kappa Strength
Helpfulness 63.5% (90.4% within 1 point) 0.44 (weighted) Moderate
Accuracy 75.0% (90.4% within 1 point) 0.21 (weighted) Fair
Instruction following 80.8% 0.31 Fair
Safety 94.2% 0.00 Poor
Preference 53.8% 0.32 Fair
Kappa by round Round 1 (20 prompts) Round 2 (6 prompts)
Helpfulness 0.42 0.49
Accuracy 0.17 0.38
Instruction following 0.16 0.58
Safety 0.00 0.00
Preference 0.23 0.67

Human vs. autograder agreement

Findings

  • High agreement can hide low reliability (the kappa paradox): safety agreement was 94%, but Cohen's kappa was 0.00. The autograder labeled every response safe and missed both safety problems confirmed on review. Raw agreement looked excellent only because unsafe answers were rare.
  • Round 2 could not test harder safety cases: both assistants handled all 6 harder prompts safely, so there was nothing for the autograder to catch. Testing a safety judge needs an evaluation set with known failures in it.
  • Human labels need review too: 1 of the 3 human safety flags was overturned on review, which is why critical disagreements are adjudicated against the response text instead of trusting either rater by default.
  • The autograder was more lenient than the human rater (+0.19 helpfulness and +0.27 accuracy on average), and it chose "tie" in 10 of 26 comparisons versus 2 for the human.
  • Agreement was higher on the harder round 2 prompts for helpfulness, accuracy, instruction following, and preference, where the answers differed more clearly. With 6 prompts this is a signal, not a conclusion.
  • The assistants were closely matched: the human preferred ChatGPT in 13 comparisons and Gemini in 11, with 2 ties. Gemini fully followed instructions more often (85% vs. 69%). After review, both correctly declined all 3 requests that should be declined.
  • Position check: the human chose Response 1 in 16 comparisons and Response 2 in 8, including all 6 in round 2. In 5 of those 6, the autograder independently picked the same winner, which points to answer quality rather than position.
  • Recommendation: at this level of agreement, the autograder suits a first pass on clear-cut criteria, while safety and nuanced judgments still need human review.
  • Agreement metrics are verified with known-answer unit tests (tests/test_agreement.py).

Limitations

These are deliberately small, hands-on datasets built to practice the full workflow, so the results show methods rather than benchmark-scale conclusions.

  • Small samples: 12 images, 6 voice commands, and 26 prompts. Kappa in particular is unstable on rare categories, which is part of why the safety result matters.
  • One human rater: human-vs-autograder agreement uses a single annotator. Adding a second would allow human-vs-human agreement as a baseline.
  • Simple conversations: the diarization clips have long turns and no overlapping speech, so they don't test diarization hard.

Repository structure

guidelines/     Annotation guidelines and rubrics
label_studio/   Custom Label Studio labeling interfaces (XML)
data/           Prompts, model responses, and raw audio
annotations/    Human and autograder labels exported from Label Studio
src/            Analysis scripts and metric implementations (WER, kappa)
tests/          Known-answer unit tests for the metrics
reports/        Generated summaries, CSVs, and charts
scripts/        Pre-push check: tests + analyses + quality checks

How to run

pip install pandas matplotlib tabulate pytest

# Verify the metrics
python -m pytest -q tests/

# Rebuild every report and chart
python src/analyze_annotations.py annotations/image/round1.json
python src/analyze_audio.py
python src/analyze_wer.py
python src/analyze_llm_eval.py

# Or run everything at once before pushing
bash scripts/check_before_push.sh

Label Studio is only needed to open or edit the annotations, and Whisper only to regenerate the machine transcripts. Both outputs are already saved in annotations/.

About

Data annotation and quality evaluation with Label Studio and Python: image, voice-assistant speech annotation (audio), and ASR evaluation (WER).

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages