Skip to content

About

Innovate QA Demo Repo

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

 
 

Latest commit

 

History

228 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Breaking GPT & Claude — Promptfoo Red-Team Workshop

Hands-on AI bug-bounty workshop. You'll red-team two chatbots — MediBot (healthcare triage) and FinanceBot (retail brokerage) — using Promptfoo. Both are built the way most production AI assistants are built: an open-weight LLM + a guardrail system prompt, served via Groq's free tier. Same attack surface, different domain rules.

Those two are prompts, not applications — a text file in prompts/ plus a Promptfoo eval, with no server to start and no chat window to open. The running applications arrive in Module 3 (PayFlow and HarborWealth FinanceBot) and they are a separate thing. See Prompts vs applications before you go looking for a MediBot URL.

The default config is free-tier-safe: a curated subset (8 cases per suite) run against three Groq models — a non-thinking Qwen (qwen/qwen3.8-27b), a hidden-reasoning GPT-OSS (openai/gpt-oss-120b), and a visible-CoT GPT-OSS (openai/gpt-oss-20b). This stays comfortably under Groq's free-tier rate limit (~30 req/min). To widen the matrix, uncomment the extra Groq models (or the paid openai:gpt-4o-mini / anthropic:messages:claude-haiku-4-5 providers) in the config — but watch the rate limit on the free tier.

Scan to clone

Taking the course? Start at days/

The lessons below are organised by topic. If you are following the eight-session course, days/ tells you what to run tonight and links to the right lesson — including the two later sessions this page does not describe: Day 7 points Promptfoo at a running application, and Day 8 covers generated red teaming and observability.

One trap worth knowing up front: docs/07-metrics-exercises.md is Day 4, not Day 7. Document numbers and day numbers are different things, and days/README.md lists every case where they disagree.

New here? Skim the one-page visual repo map — how the prompts, tests, providers, and configs fit together — before you dive in.

Quickstart (≈ 3 minutes)

You need: a terminal, internet, and a free Groq API key.

Step 0 — get your free Groq API key first: go to https://console.groq.com/keys, sign in, and click Create API Key. It's free, no credit card required. Copy the gsk_… key — setup.sh will prompt you to paste it. Keep it handy before running the steps below.

macOS / Linux

git clone https://github.com/engenious-inc/Break-Into-AI-Testing.git
cd Break-Into-AI-Testing
./setup.sh

Windows (PowerShell)

git clone https://github.com/engenious-inc/Break-Into-AI-Testing.git
cd Break-Into-AI-Testing
powershell -ExecutionPolicy Bypass -File .\setup.ps1

If .\setup.ps1 fails with "running scripts is disabled on this system", use the -ExecutionPolicy Bypass invocation shown above (or run Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass first).

The script will: check Node ≥ 22.22, install Promptfoo, prompt for your Groq key (and optionally an OpenRouter fallback key — press Enter to skip), write .env, and run a smoke test.

Then confirm you're ready: run ./preflight.sh (.\preflight.ps1 on Windows) — a read-only check of your key, network, and environment. It's safe to re-run anytime, especially the morning of the workshop. See Check you're ready below.

Curriculum

This repo is one course in four modules plus a hackathon (full map: modules/README.md):

  • Module 0 — Promptfoo Basics (modules/00-promptfoo-basics/) — prompts, providers, assertions.
  • Module 1 — Red-Team Fundamentals (modules/01-red-team/ over repo root + docs/02,04,05) — break MediBot & FinanceBot by hand, then let Claude Code drive: the red-teamer subagent drafts attacks, run-and-summarize reports them, and 04-grading-the-grader/ shows a real case where the LLM judge passed a total system-prompt leak. Quality challenges (bias, consistency, context, values, compliance) in docs/05-quality-challenges.md.
  • Module 2 — Advanced Evaluation (modules/02-advanced-eval/) — weights, metrics, CSV data, F-score, temperature, and a debugging track.
  • Module 3 — Testing an Application (modules/03-app-testing/) — the first modules test prompts; this one tests applications. Two of them ship: the PayFlow GenAI demo and the HarborWealth FinanceBot demo, each a guard LLM, an orchestrator, and four document specialists behind POST /chat, each with a workshop chat UI in the browser. Promptfoo's http provider points at the running app, so assertions read output.route and output.citations — the routing decision, not just the prose.

New to Promptfoo? Start at Module 0. Here to break things? Jump to Module 1. The Claude Code workflow lives in .claude/ (see CLAUDE.md) — in Module 1 it's the path, not an optional extra.

Prompts vs applications — which bot is which

Module 1 — prompt-only Module 3 — running application
Bots MediBot (healthcare triage), FinanceBot (retail brokerage) PayFlow (fintech), HarborWealth FinanceBot (brokerage)
What it actually is prompts/*.txt + a Promptfoo eval a Node app behind POST /chat, with a chat UI
How you run it ./run.sh medibot, ./run.sh finance ./run.sh payflow-serve (:8000), ./run.sh financebot-serve (:8001)
What you assert on the text the model produced output.route, output.citations — how the answer was produced

finance and financebot are two different bots, not two spellings of one. ./run.sh finance is Module 1's prompt-only FinanceBot; ./run.sh financebot is Module 3's HarborWealth app on :8001. ./run.sh chat financebot is still the Module 1 prompt (prompts/financebot.txt) — a terminal conversation, not the HarborWealth UI. Module 3 does not change Module 1's FinanceBot — the prompt, the tests, and the config are all still there, untouched.

MediBot has no chat UI, on purpose. Giving it one would clone PayFlow into healthcare and blur the prompt-vs-application line the course is built on. You talk to MediBot by running its eval and reading the results in ./run.sh view — or ./run.sh chat medibot for a terminal conversation, no browser involved.

Start here (reading order)

  1. Quickstart — clone, set up your Groq key, run your first eval.
  2. Module 1 lessons — manual baseline → agent drafts → agent reports → grading the grader.
  3. Prompt engineering & local models — reverse-design a system prompt, compare three models on latency and tokens, run a model on your own laptop.
  4. Metrics & model-graded assertions — weights, thresholds, precision/recall/F1, and the debugger track.
  5. Red-team exercises — the guided walkthrough against MediBot & FinanceBot.
  6. Hackathon challenges — break it / fix it / build it.
  7. Quality challenges — bias, consistency, context, values, compliance.
  8. Testing an application — point Promptfoo at a running multi-agent app and assert on its routing decisions.

Reference: Troubleshooting.

The 7 core AI testing challenges — where each one lives

Day 1 introduces a seven-item taxonomy of what makes AI hard to test. Every one of them is already exercised somewhere in this repo; this table is the map from the lecture to the runnable suite.

Not to be confused with the hackathon challenges in docs/04-challenges.md, which are numbered 1–3 and are a different thing entirely — those are the team competition, these are failure modes.

# Challenge Exercised in Suite
0 Security — injection, leakage, instruction override docs/02-redteam-exercises.md tests/smoke.medibot.yaml, tests/smoke.finance.yaml
1 Hallucinations — confident and wrong docs/02-redteam-exercises.md same smoke suites
2 Bias & fairness docs/05-quality-challenges.md tests/bias.medibot.yaml
3 Performance consistency docs/05-quality-challenges.md tests/consistency.medibot.yaml
4 Context limitations docs/05-quality-challenges.md tests/context.finance.yaml
5 Human values alignment docs/05-quality-challenges.md tests/values.finance.yaml
6 Compliance docs/05-quality-challenges.md tests/compliance.medibot.yaml

Day 1's homework — design three test prompts for one of the seven — starts here: pick a row, open its suite, and write cases in the same shape.

Run the workshop eval

Easy path — ./run.sh picks the config, bakes in free-tier-safe pacing, and translates the result (in a red-team suite a failing check means the attack landed):

./run.sh medibot     # MediBot red-team suite   (.\run.ps1 medibot on Windows)
./run.sh finance     # FinanceBot red-team suite
./run.sh view        # open the results web UI

Not every session is an eval. ./run.sh chat <bot> opens an ordinary conversation with a prompt-only bot (onboardbot, medibot, financebot, mybot) — /reset clears the context, /save <file> writes the transcript. That financebot is prompts/financebot.txt, not the HarborWealth app. Day 6 uses it to explore a bot whose rules you have not read, which is the one thing a suite cannot teach.

Under the hood it's plain Promptfoo — run it directly if you prefer:

npx promptfoo@latest eval -c promptfooconfig.medibot.yaml   # MediBot (free-tier-safe)
npx promptfoo@latest eval -c promptfooconfig.finance.yaml   # FinanceBot (free-tier-safe)
npx promptfoo@latest view                                   # opens the web UI

First run looks like it failed? In a red-team suite a failing assertion means the attack landed — that's the finding you're hunting, not a setup error. A healthy run exits with code 100. (Modules 0 and 2 use ordinary pass = good assertions.)

Hitting Groq throttling? promptfoo runs 4 calls at once by default, which can burst past Groq's ~30 req/min limit (especially on a reused key). Add -j 2 (or -j 1) to serialize, or space calls out with --delay (ms between calls): npx promptfoo@latest eval -c promptfooconfig.medibot.yaml -j 1 --delay 1000. More knobs in Troubleshooting.

Groq down entirely? Use the OpenRouter fallback. Put the cohort OPENROUTER_API_KEY (your instructor shares it) in .env, then run npx promptfoo@latest eval -c promptfooconfig.openrouter.medibot.yaml (or promptfooconfig.openrouter.finance.yaml). Same tests, routed to OpenRouter instead of Groq.

Check you're ready

Anytime before the workshop (especially the morning of), run the read-only readiness probe:

./preflight.sh        # macOS / Linux
.\preflight.ps1       # Windows (PowerShell)

It changes nothing and re-runs safely. On a problem it names the exact fix — a bad or missing key, a rate-limit throttle, or an environment that needs ./setup.sh again.

What you'll do

  1. Prompt-injection / jailbreaks — try to bypass system-prompt guardrails
  2. Hallucination traps — confirm the model refuses to invent facts
  3. Cost & context — assert latency and response-length thresholds
  4. Multi-model comparison — the default config runs three Groq-hosted models (free-tier-safe). Uncomment the extra Groq models, or the paid openai:gpt-4o-mini / anthropic:messages:claude-haiku-4-5 lines, in the config to widen the matrix. Add any paid-vendor keys to .env first.

All cases live in tests/smoke.medibot.yaml (MediBot, 8 cases) and tests/smoke.finance.yaml (FinanceBot, 8 cases) — a curated spread across jailbreak, safety, hallucination, cost, exfiltration, and direct-ask variants. Add your own attacks there.

Notes on the default lineup

  • The configs run three providers over a curated subset (8 cases each, so 24 responses per suite), so a full pass stays well under Groq's free-tier rate limit (~30 req/min per key) and finishes in under a minute at the default pacing. Adding all six Groq models can queue behind the shared grader and hit request timeouts on the free tier — widen the matrix only with paid keys or -j 2.
  • gpt-oss models emit reasoning tokens (especially with include_reasoning left on), so the 20B can spend ~2× the tokens of Qwen per response. The javascript length assertion (≤ 40 words) can fail for the more verbose gpt-oss models — that's intentional, it's a signal you're meant to spot. (Note: cost assertions are avoided — Groq's free tier returns no cost field, which makes them error rather than pass.)
  • The grader (judge LLM for llm-rubric assertions) is groq:qwen/qwen3.8-27b, set via defaultTest.options.provider. Workshop attendees don't need an OpenAI key for grading.

Hackathon challenges

Running this as a competition? There are three challenges — every team does all three, scores sum, and you're judged on creativity and originality:

  1. 🔓 Break it — author a novel attack (not one of the starter vectors) that makes a model violate a bot's NON-NEGOTIABLE rule.
  2. 🛡️ Fix it — harden a bot's system prompt so the landing attacks stop landing, without over-refusing the must-answer cases.
  3. 🧪 Build it — invent your own guardrailed bot in a fresh domain and a 3-case probe suite (a skeleton is in the repo to start from).

Full steps, rules, and scoring: docs/04-challenges.md.

Troubleshooting

See docs/03-troubleshooting.md.

About

Innovate QA Demo Repo

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages