Hands-on AI bug-bounty workshop. You'll red-team two chatbots — MediBot (healthcare triage) and FinanceBot (retail brokerage) — using Promptfoo. Both are built the way most production AI assistants are built: an open-weight LLM + a guardrail system prompt, served via Groq's free tier. Same attack surface, different domain rules.
Those two are prompts, not applications — a text file in prompts/ plus a Promptfoo eval, with no server to start and no chat window to open. The running applications arrive in Module 3 (PayFlow and HarborWealth FinanceBot) and they are a separate thing. See Prompts vs applications before you go looking for a MediBot URL.
The default config is free-tier-safe: a curated subset (8 cases per suite) run against three Groq models — a non-thinking Qwen (qwen/qwen3.8-27b), a hidden-reasoning GPT-OSS (openai/gpt-oss-120b), and a visible-CoT GPT-OSS (openai/gpt-oss-20b). This stays comfortably under Groq's free-tier rate limit (~30 req/min). To widen the matrix, uncomment the extra Groq models (or the paid openai:gpt-4o-mini / anthropic:messages:claude-haiku-4-5 providers) in the config — but watch the rate limit on the free tier.
Taking the course? Start at
days/The lessons below are organised by topic. If you are following the eight-session course,
days/tells you what to run tonight and links to the right lesson — including the two later sessions this page does not describe: Day 7 points Promptfoo at a running application, and Day 8 covers generated red teaming and observability.One trap worth knowing up front:
docs/07-metrics-exercises.mdis Day 4, not Day 7. Document numbers and day numbers are different things, anddays/README.mdlists every case where they disagree.
New here? Skim the one-page visual repo map — how the prompts, tests, providers, and configs fit together — before you dive in.
You need: a terminal, internet, and a free Groq API key.
Step 0 — get your free Groq API key first: go to https://console.groq.com/keys, sign in, and click Create API Key. It's free, no credit card required. Copy the
gsk_…key —setup.shwill prompt you to paste it. Keep it handy before running the steps below.
git clone https://github.com/engenious-inc/Break-Into-AI-Testing.git
cd Break-Into-AI-Testing
./setup.shgit clone https://github.com/engenious-inc/Break-Into-AI-Testing.git
cd Break-Into-AI-Testing
powershell -ExecutionPolicy Bypass -File .\setup.ps1If
.\setup.ps1fails with "running scripts is disabled on this system", use the-ExecutionPolicy Bypassinvocation shown above (or runSet-ExecutionPolicy -Scope Process -ExecutionPolicy Bypassfirst).
The script will: check Node ≥ 22.22, install Promptfoo, prompt for your Groq key (and optionally an OpenRouter fallback key — press Enter to skip), write .env, and run a smoke test.
Then confirm you're ready: run
./preflight.sh(.\preflight.ps1on Windows) — a read-only check of your key, network, and environment. It's safe to re-run anytime, especially the morning of the workshop. See Check you're ready below.
This repo is one course in four modules plus a hackathon (full map: modules/README.md):
- Module 0 — Promptfoo Basics (
modules/00-promptfoo-basics/) — prompts, providers, assertions. - Module 1 — Red-Team Fundamentals (
modules/01-red-team/over repo root +docs/02,04,05) — break MediBot & FinanceBot by hand, then let Claude Code drive: thered-teamersubagent drafts attacks,run-and-summarizereports them, and04-grading-the-grader/shows a real case where the LLM judge passed a total system-prompt leak. Quality challenges (bias, consistency, context, values, compliance) indocs/05-quality-challenges.md. - Module 2 — Advanced Evaluation (
modules/02-advanced-eval/) — weights, metrics, CSV data, F-score, temperature, and a debugging track. - Module 3 — Testing an Application (
modules/03-app-testing/) — the first modules test prompts; this one tests applications. Two of them ship: the PayFlow GenAI demo and the HarborWealth FinanceBot demo, each a guard LLM, an orchestrator, and four document specialists behindPOST /chat, each with a workshop chat UI in the browser. Promptfoo'shttpprovider points at the running app, so assertions readoutput.routeandoutput.citations— the routing decision, not just the prose.
New to Promptfoo? Start at Module 0. Here to break things? Jump to Module 1.
The Claude Code workflow lives in .claude/ (see CLAUDE.md) — in Module 1 it's the path,
not an optional extra.
| Module 1 — prompt-only | Module 3 — running application | |
|---|---|---|
| Bots | MediBot (healthcare triage), FinanceBot (retail brokerage) | PayFlow (fintech), HarborWealth FinanceBot (brokerage) |
| What it actually is | prompts/*.txt + a Promptfoo eval |
a Node app behind POST /chat, with a chat UI |
| How you run it | ./run.sh medibot, ./run.sh finance |
./run.sh payflow-serve (:8000), ./run.sh financebot-serve (:8001) |
| What you assert on | the text the model produced | output.route, output.citations — how the answer was produced |
finance and financebot are two different bots, not two spellings of one.
./run.sh finance is Module 1's prompt-only FinanceBot; ./run.sh financebot is Module 3's
HarborWealth app on :8001. ./run.sh chat financebot is still the Module 1 prompt
(prompts/financebot.txt) — a terminal conversation, not the HarborWealth UI. Module 3 does
not change Module 1's FinanceBot — the prompt, the tests, and the config are all still there,
untouched.
MediBot has no chat UI, on purpose. Giving it one would clone PayFlow into healthcare and
blur the prompt-vs-application line the course is built on. You talk to MediBot by running its
eval and reading the results in ./run.sh view — or ./run.sh chat medibot for a terminal
conversation, no browser involved.
- Quickstart — clone, set up your Groq key, run your first eval.
- Module 1 lessons — manual baseline → agent drafts → agent reports → grading the grader.
- Prompt engineering & local models — reverse-design a system prompt, compare three models on latency and tokens, run a model on your own laptop.
- Metrics & model-graded assertions — weights, thresholds, precision/recall/F1, and the debugger track.
- Red-team exercises — the guided walkthrough against MediBot & FinanceBot.
- Hackathon challenges — break it / fix it / build it.
- Quality challenges — bias, consistency, context, values, compliance.
- Testing an application — point Promptfoo at a running multi-agent app and assert on its routing decisions.
Reference: Troubleshooting.
Day 1 introduces a seven-item taxonomy of what makes AI hard to test. Every one of them is already exercised somewhere in this repo; this table is the map from the lecture to the runnable suite.
Not to be confused with the hackathon challenges in
docs/04-challenges.md, which are numbered 1–3 and are a different thing entirely — those are the team competition, these are failure modes.
| # | Challenge | Exercised in | Suite |
|---|---|---|---|
| 0 | Security — injection, leakage, instruction override | docs/02-redteam-exercises.md |
tests/smoke.medibot.yaml, tests/smoke.finance.yaml |
| 1 | Hallucinations — confident and wrong | docs/02-redteam-exercises.md |
same smoke suites |
| 2 | Bias & fairness | docs/05-quality-challenges.md |
tests/bias.medibot.yaml |
| 3 | Performance consistency | docs/05-quality-challenges.md |
tests/consistency.medibot.yaml |
| 4 | Context limitations | docs/05-quality-challenges.md |
tests/context.finance.yaml |
| 5 | Human values alignment | docs/05-quality-challenges.md |
tests/values.finance.yaml |
| 6 | Compliance | docs/05-quality-challenges.md |
tests/compliance.medibot.yaml |
Day 1's homework — design three test prompts for one of the seven — starts here: pick a row, open its suite, and write cases in the same shape.
Easy path — ./run.sh picks the config, bakes in free-tier-safe pacing, and translates the result (in a red-team suite a failing check means the attack landed):
./run.sh medibot # MediBot red-team suite (.\run.ps1 medibot on Windows)
./run.sh finance # FinanceBot red-team suite
./run.sh view # open the results web UINot every session is an eval. ./run.sh chat <bot> opens an ordinary conversation with
a prompt-only bot (onboardbot, medibot, financebot, mybot) — /reset clears the
context, /save <file> writes the transcript. That financebot is prompts/financebot.txt,
not the HarborWealth app. Day 6 uses it to explore a bot whose rules you have not read, which
is the one thing a suite cannot teach.
Under the hood it's plain Promptfoo — run it directly if you prefer:
npx promptfoo@latest eval -c promptfooconfig.medibot.yaml # MediBot (free-tier-safe)
npx promptfoo@latest eval -c promptfooconfig.finance.yaml # FinanceBot (free-tier-safe)
npx promptfoo@latest view # opens the web UIFirst run looks like it failed? In a red-team suite a failing assertion means the attack landed — that's the finding you're hunting, not a setup error. A healthy run exits with code 100. (Modules 0 and 2 use ordinary pass = good assertions.)
Hitting Groq throttling? promptfoo runs 4 calls at once by default, which can burst past Groq's ~30 req/min limit (especially on a reused key). Add
-j 2(or-j 1) to serialize, or space calls out with--delay(ms between calls):npx promptfoo@latest eval -c promptfooconfig.medibot.yaml -j 1 --delay 1000. More knobs in Troubleshooting.
Groq down entirely? Use the OpenRouter fallback. Put the cohort
OPENROUTER_API_KEY(your instructor shares it) in.env, then runnpx promptfoo@latest eval -c promptfooconfig.openrouter.medibot.yaml(orpromptfooconfig.openrouter.finance.yaml). Same tests, routed to OpenRouter instead of Groq.
Anytime before the workshop (especially the morning of), run the read-only readiness probe:
./preflight.sh # macOS / Linux
.\preflight.ps1 # Windows (PowerShell)It changes nothing and re-runs safely. On a problem it names the exact fix — a bad or
missing key, a rate-limit throttle, or an environment that needs ./setup.sh again.
- Prompt-injection / jailbreaks — try to bypass system-prompt guardrails
- Hallucination traps — confirm the model refuses to invent facts
- Cost & context — assert latency and response-length thresholds
- Multi-model comparison — the default config runs three Groq-hosted models (free-tier-safe). Uncomment the extra Groq models, or the paid
openai:gpt-4o-mini/anthropic:messages:claude-haiku-4-5lines, in the config to widen the matrix. Add any paid-vendor keys to.envfirst.
All cases live in tests/smoke.medibot.yaml (MediBot, 8 cases) and tests/smoke.finance.yaml (FinanceBot, 8 cases) — a curated spread across jailbreak, safety, hallucination, cost, exfiltration, and direct-ask variants. Add your own attacks there.
- The configs run three providers over a curated subset (8 cases each, so 24 responses per suite), so a full pass stays well under Groq's free-tier rate limit (~30 req/min per key) and finishes in under a minute at the default pacing. Adding all six Groq models can queue behind the shared grader and hit request timeouts on the free tier — widen the matrix only with paid keys or
-j 2. - gpt-oss models emit reasoning tokens (especially with
include_reasoningleft on), so the 20B can spend ~2× the tokens of Qwen per response. Thejavascriptlength assertion (≤ 40 words) can fail for the more verbose gpt-oss models — that's intentional, it's a signal you're meant to spot. (Note:costassertions are avoided — Groq's free tier returns no cost field, which makes them error rather than pass.) - The grader (judge LLM for
llm-rubricassertions) isgroq:qwen/qwen3.8-27b, set viadefaultTest.options.provider. Workshop attendees don't need an OpenAI key for grading.
Running this as a competition? There are three challenges — every team does all three, scores sum, and you're judged on creativity and originality:
- 🔓 Break it — author a novel attack (not one of the starter vectors) that makes a model violate a bot's NON-NEGOTIABLE rule.
- 🛡️ Fix it — harden a bot's system prompt so the landing attacks stop landing, without over-refusing the must-answer cases.
- 🧪 Build it — invent your own guardrailed bot in a fresh domain and a 3-case probe suite (a skeleton is in the repo to start from).
Full steps, rules, and scoring: docs/04-challenges.md.
