Skip to content
View edara-code's full-sized avatar

Block or report edara-code

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
edara-code/README.md

Typing SVG


👩‍💻 About me

I'm working on evaluating AI systems and the data behind them: designing rubrics and evaluation sets, collecting and checking human labels, and comparing LLM autograders with human ratings to see where they can be relied on.

  • 🎯 Focus: evaluating conversational AI, LLMs, and speech systems
  • 💡 Principle: a metric is only useful if it predicts real-world quality and people trust it
  • 🔁 Approach: measure → diagnose failures → fix → re-measure → gate the release
  • ✍️ Writing: notes on evaluation and data quality on Substack

🧭 Skills & expertise

Area What it involves
🤖 LLM autograders & rubrics Rubric design, evaluation set development, and calibrating an LLM autograder against human ratings
📊 Metrics that mean something Cohen's kappa for human–LLM agreement (including the kappa paradox), error rates, and checking that findings hold on larger, harder samples
📚 Grounded, factual responses Groundedness, citation accuracy, and hallucination rate for retrieval-based (RAG) answers
🎙️ Conversational & speech quality Intent, transcription, and speaker annotation, plus speech recognition accuracy
✅ Quality gates Golden sets, sampling audits, annotator scorecards, and CI quality gates (GitHub Actions) that block regressions before release
🚗 3D LiDAR annotation 3D cuboid labeling of driving point clouds in CVAT, evaluated against official ground truth with 3D IoU, center, size, and heading error

🔬 Featured work: annotation-eval-lab

An end-to-end lab for building trustworthy evaluation, from human annotation to automated grading and release gates.

Layer What it involves
Human annotation Image, audio/voice, text, and video labeling with clear guidelines and QA checks
Speech evaluation Comparing automatic transcripts with human ones (WER)
LLM evaluation Rubric ratings, preference judgments, and a Claude autograder calibrated against human ratings with Cohen's kappa, then re-tested on harder prompts
RAG evaluation Groundedness, citation accuracy, and hallucination rate
Translation quality Human error annotation compared with automatic scores
Quality framework Golden sets, sampling audits, annotator scorecards, and CI quality gates with GitHub Actions
Red teaming (separate repo) Find failures, fix, re-scan, and lock in regression tests

Built mainly with Python and Label Studio.


🚗 Featured work: lidar-3d-annotation-eval

Hands-on 3D LiDAR annotation for autonomous driving and robotics, from labeling guidelines to evaluation against official ground truth.

Layer What it involves
Annotation guidelines Classes, box fitting, heading, ground contact, and quality checks
3D annotation Cuboid labeling of driving point clouds in CVAT, using top, side, and front views
Evaluation Comparing labels with official KITTI boxes: 3D IoU, recall, center, size, and heading error, by distance
Results: 10 of 11 official vehicles found (90.9%) · median center error 0.15 m · heading error ≈ 2° · accuracy drops with distance (3D IoU 0.82 within 20 m → 0.50 beyond 40 m)

Built mainly with Python and CVAT.

Pinned Loading

  1. annotation-eval-lab annotation-eval-lab Public

    Data annotation and quality evaluation with Label Studio and Python: image, voice-assistant speech annotation (audio), and ASR evaluation (WER).

    Python 1

  2. lidar-3d-annotation-eval lidar-3d-annotation-eval Public

    Labeling self-driving car LiDAR data with 3D boxes in CVAT, then measuring accuracy against the official KITTI labels.

    Python 1