Agentic Coding Challenges

A reproducible framework for comparing how AI coding agents perform on real data engineering tasks — across different models and specification styles.

The Question

How much does the spec matter? And when does a better model stop helping?

Most comparisons of AI coding tools are anecdotal. This project replaces vibes with a structured framework: same task, same prompt, different conditions. The results show what actually matters — the model, the spec, or both.

The Matrix

Each challenge runs 3 models against 3 spec variants:

3 models x 3 specs = 9 runs per challenge

	Spec A (agnostic)	Spec B (principled)	Spec C (prescriptive)
Haiku	outcomes only	+ architecture	+ exact tools & schema
Sonnet	outcomes only	+ architecture	+ exact tools & schema
Opus	outcomes only	+ architecture	+ exact tools & schema

Models

• Haiku — fast, cheap, lightweight
• Sonnet — balanced, the workhorse
• Opus — expensive, most capable

Spec Gradient

• Spec A (tool-agnostic): describes only desired outcomes — the agent chooses everything
• Spec B (principled): adds architectural requirements without naming tools
• Spec C (prescriptive): names exact tools, file structure, column names, tests

The agent always sees its spec as SPEC.md — it doesn't know which variant it received.

Quick Start

Prerequisites

• Claude Code CLI installed and authenticated
• A GitHub personal access token (for the github-analytics challenge)
• Python 3.11+
• Git
• jq (for result collection)

1. Set Up

# Clone this repo
git clone <repo-url>
cd agentic-coding-challenges

# Set your GitHub token (needed by the agent during runs)
export GITHUB_TOKEN=ghp_your_token_here

2. Create Worktrees

./scripts/setup-worktrees.sh github-analytics

This creates isolated git worktrees under runs/github-analytics/, each branched from the same seed commit with its spec variant.

runs/github-analytics/
├── seed/          ← base git repo
├── haiku-a/       ← Haiku + Spec A
├── haiku-b/       ← Haiku + Spec B
├── haiku-c/       ← Haiku + Spec C
├── sonnet-a/      ← Sonnet + Spec A
├── ...
└── opus-c/        ← Opus + Spec C

3. Run the Matrix

# Run all 9 sequentially
./scripts/run-matrix.sh github-analytics

# Or run a single cell
./scripts/run-single.sh github-analytics sonnet a

# Or filter by model or spec
./scripts/run-matrix.sh github-analytics --model sonnet
./scripts/run-matrix.sh github-analytics --spec a

Each run:

• Uses the exact same prompt: "Read SPEC.md and build the complete solution. Run it to verify it works."
• Runs in non-interactive print mode — no follow-up, no coaching
• Has a hard cap of $5.00 API spend per run (configurable via MAX_BUDGET_USD)
• Captures the full conversation log, token usage, and git diff

4. Evaluate

# Evaluate all runs (uses Opus as the LLM judge)
./scripts/evaluate-matrix.sh github-analytics

# Or evaluate a single run
./scripts/evaluate-run.sh github-analytics sonnet a

Each evaluation:

1. Runs automated pre-checks (does run.sh work? dbt compile? DuckDB table stats? pattern detection)
2. Sends the rubric + pre-check facts + actual source files + conversation log to an LLM judge
3. Produces a structured scoring report for human review

5. Collect Results

./scripts/collect-results.sh github-analytics

Generates the summary matrix with all individual scoring reports.

Methodology

Reproducibility

Every run uses the exact same prompt. No hints, no corrections, no follow-up guidance. Claude Code runs in -p (print) mode — there is no interactive session, so the agent cannot ask clarifying questions.

Isolation

Each run happens in an isolated git worktree branched from the same seed commit. All 9 runs start from identical state (plus their spec variant). After a run, git diff against the seed commit shows exactly what the agent created.

Evaluation

Each run is scored on a rubric defined before the challenge starts. The rubric is never shown to the agent. An LLM judge (Opus) produces a first-pass scoring report; a human reviewer makes the final call.

Scoring covers:

• Does it run? — end-to-end pipeline execution
• Design decisions — how the agent interpreted the brief
• Metric logic — the core intellectual test
• Engineering practices — testing, project structure, tool choices
• Code quality — readable, shippable, not over-engineered

What We're NOT Testing

• Speed of generation (all models are fast enough)
• Subjective "feel" of the conversation
• Ability to follow detailed instructions (the prescriptive spec tests this; the others don't)

Project Structure

agentic-coding-challenges/
├── README.md
├── challenges/
│   └── github-analytics/
│       ├── SPEC-A.md          # tool-agnostic brief — shown to agent
│       ├── SPEC-B.md          # principled brief — shown to agent
│       ├── SPEC-C.md          # prescriptive brief — shown to agent
│       ├── EVALUATION.md      # scoring rubric — internal, never shown
│       ├── PROMPTS.md         # exact prompts used
│       └── seed/              # minimal starter files
│           ├── pyproject.toml
│           ├── .env.example
│           ├── .gitignore
│           └── README.md
├── scripts/
│   ├── setup-worktrees.sh     # create 9 isolated worktrees
│   ├── run-single.sh          # run one cell
│   ├── run-matrix.sh          # run all 9 sequentially
│   ├── pre-check.sh           # automated mechanical checks
│   ├── evaluate-run.sh        # pre-check + LLM judge for one run
│   ├── evaluate-matrix.sh     # evaluate all 9
│   └── collect-results.sh     # build the summary matrix
├── sessions/                  # results organized by challenge and date
│   └── github-analytics/
│       └── 2026-02-27/
│           ├── RESULTS.md     # summary writeup
│           ├── FINDING-B2.md  # experiment findings
│           ├── logs/          # conversation logs, diffs, metadata
│           ├── pre-checks/    # automated pre-check YAML files
│           └── runs/          # per-run scoring reports
│               ├── haiku-a/SCORE.md
│               ├── sonnet-b/SCORE.md
│               └── ...
└── runs/                      # created by setup-worktrees.sh (gitignored)
    └── github-analytics/
        ├── seed/
        ├── haiku-a/
        └── ...

Challenge Format

Each challenge is a self-contained data engineering task:

• SPEC-A.md (tool-agnostic): describes only desired outcomes
• SPEC-B.md (principled): adds architectural requirements without naming tools
• SPEC-C.md (prescriptive): names exact tools, file structure, column names, tests
• EVALUATION.md is the internal scoring rubric, designed before the runs start and never shown to the agent
• PROMPTS.md documents the exact prompt used, identical across all 9 runs
• seed/ contains the minimal files present in every run (project config, env template, gitignore)

Configuration

Environment Variables

Variable	Default	Description
GITHUB_TOKEN	—	GitHub API token (required for github-analytics)
MAX_BUDGET_USD	5.00	Max dollar spend per run
JUDGE_MODEL	opus	Model used for LLM judging
SESSION_DIR	sessions/{challenge}/{date}	Override session output directory

Adding a New Challenge

Create challenges/your-challenge-name/ with:

• SPEC-A.md, SPEC-B.md, SPEC-C.md — three spec variants
• EVALUATION.md — scoring rubric (never shown to the agent)
• PROMPTS.md — documents the exact prompt used
• seed/ — minimal starter files

Then run:

./scripts/setup-worktrees.sh your-challenge-name
./scripts/run-matrix.sh your-challenge-name
./scripts/evaluate-matrix.sh your-challenge-name
./scripts/collect-results.sh your-challenge-name

Challenges

GitHub Repository Analytics

Build a local analytics pipeline for open source repos. The spec describes what to build, not how — the agent chooses its own tools. The key test: can the agent correctly implement cohort-based responsiveness metrics, bus factor, and contribution concentration — from a brief that only hints at the methodology?

Stack: Agent's choice — the spec is tool-agnostic.

Future Challenges

• Event pipeline with tracking (Segment-style schema)
• Subscription analytics (MRR calculations, cohort retention)
• Reverse ETL pipeline
• Data quality monitoring setup
• Full dbt project refactor