I'm working on evaluating AI systems and the data behind them: designing rubrics and evaluation sets, collecting and checking human labels, and comparing LLM autograders with human ratings to see where they can be relied on.
- 🎯 Focus: evaluating conversational AI, LLMs, and speech systems
- 💡 Principle: a metric is only useful if it predicts real-world quality and people trust it
- 🔁 Approach: measure → diagnose failures → fix → re-measure → gate the release
- ✍️ Writing: notes on evaluation and data quality on Substack
| Area | What it involves |
|---|---|
| 🤖 LLM autograders & rubrics | Rubric design, evaluation set development, and calibrating an LLM autograder against human ratings |
| 📊 Metrics that mean something | Cohen's kappa for human–LLM agreement (including the kappa paradox), error rates, and checking that findings hold on larger, harder samples |
| 📚 Grounded, factual responses | Groundedness, citation accuracy, and hallucination rate for retrieval-based (RAG) answers |
| 🎙️ Conversational & speech quality | Intent, transcription, and speaker annotation, plus speech recognition accuracy |
| ✅ Quality gates | Golden sets, sampling audits, annotator scorecards, and CI quality gates (GitHub Actions) that block regressions before release |
| 🚗 3D LiDAR annotation | 3D cuboid labeling of driving point clouds in CVAT, evaluated against official ground truth with 3D IoU, center, size, and heading error |
🔬 Featured work: annotation-eval-lab
An end-to-end lab for building trustworthy evaluation, from human annotation to automated grading and release gates.
| Layer | What it involves |
|---|---|
| Human annotation | Image, audio/voice, text, and video labeling with clear guidelines and QA checks |
| Speech evaluation | Comparing automatic transcripts with human ones (WER) |
| LLM evaluation | Rubric ratings, preference judgments, and a Claude autograder calibrated against human ratings with Cohen's kappa, then re-tested on harder prompts |
| RAG evaluation | Groundedness, citation accuracy, and hallucination rate |
| Translation quality | Human error annotation compared with automatic scores |
| Quality framework | Golden sets, sampling audits, annotator scorecards, and CI quality gates with GitHub Actions |
| Red teaming (separate repo) | Find failures, fix, re-scan, and lock in regression tests |
Built mainly with Python and Label Studio.
🚗 Featured work: lidar-3d-annotation-eval
Hands-on 3D LiDAR annotation for autonomous driving and robotics, from labeling guidelines to evaluation against official ground truth.
| Layer | What it involves |
|---|---|
| Annotation guidelines | Classes, box fitting, heading, ground contact, and quality checks |
| 3D annotation | Cuboid labeling of driving point clouds in CVAT, using top, side, and front views |
| Evaluation | Comparing labels with official KITTI boxes: 3D IoU, recall, center, size, and heading error, by distance |
| Results: | 10 of 11 official vehicles found (90.9%) · median center error 0.15 m · heading error ≈ 2° · accuracy drops with distance (3D IoU 0.82 within 20 m → 0.50 beyond 40 m) |
Built mainly with Python and CVAT.