Repository navigation
feat: GEPA prompt optimizer with Scope adapter (AGENTS.md optimization) - #1031
Draft
Cedric Vidal (cedricvidal) wants to merge 68 commits into
Draft
Cedric Vidal (cedricvidal) wants to merge 68 commits into
Cedric Vidal (cedricvidal) wants to merge 68 commits into
Conversation
Add AGENTS.md support and GEPA experiment grouping to the shared layer:
- TaskPromptDocument gains optional `type` ('task' | 'agents.md') and
`contentBlobUrl`; `text` becomes optional. computeTaskPromptId is
type-aware and backward-compatible (task/undefined keep identical _ids).
- TaskPromptStore: size-based storage (inline <= PROMPT_INLINE_MAX_BYTES,
else blob at prompts/{id}.txt), findOrCreate({type}), getAll({type}),
resolvePromptText(doc), 256 KB hard cap. BlobStorage.uploadText added.
- RequestDocument + schemas gain experimentId, agentsMdPromptId,
agentsMdParentIds (multi-parent lineage). GroupByKey adds 'experiment'.
- Tests: 35 passing covering type namespacing, backward compat, blob
offload on create-miss, resolvePromptText, max-size reject, getAll filter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ompts Wire the shared typed-prompt and experiment capabilities into the API: - POST /api/v1/requests accepts experimentId, agentsMd, agentsMdParentIds; agentsMd is resolved via findOrCreate(type: 'agents.md') and stored as agentsMdPromptId across all construction paths (variation, count>1, single); retry/resubmit/import preserve lineage. - GET /api/v1/requests: ?experimentId= filter + groupBy=experiment. - Task prompts: ?type= list filter, create with type, new GET /:id/content (resolvePromptText), extract-features scoped by prompt type. TaskPromptStore constructed with blobStorage. - grouping.ts buildGroupingPipeline supports 'experiment'. - Tests: typed task prompts, /:id/content, experiment/AGENTS.md request creation + lineage, experiment filter/group. OpenAPI snapshot updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Features are now scoped by prompt type so a feature only applies to prompts
of its own type:
- PromptFeatureStore: getAll({type}), create({type}), seed carries type.
- Schemas: CreatePromptFeatureInput + PromptFeatureResponse add optional type.
- API: GET /prompt-features?type= filter (task includes legacy untyped),
create persists type, seed $setOnInsert carries type, extract-from-text
scopes candidate features by ?type=. task-prompts extract-features already
scopes by the prompt's own type.
- FS provider loads optional `type` from feature YAML.
- Tests: 6 store tests + API filter/create tests. OpenAPI snapshot updated.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add AGENTS.md delivery to the shared queue processor so all workers honor a request's optional agentsMdPromptId: - New PromptClient.getText(id) REST client resolves a prompt's plain-text body via GET /api/v1/task-prompts/:id/content (server downloads blob if the doc is blob-backed), with clear errors on 404 / non-OK / missing text. - writeAgentsMd() resolves the AGENTS.md text once before the run and writes <workspace>/AGENTS.md (constant for the whole run). No-ops when the request has no agentsMdPromptId; throws (fails the run loudly via the base processor's catch) when set but apiBaseUrl is missing or resolution fails. Tests: PromptClient success/404/non-OK/missing-text; writeAgentsMd no-op, missing-apiBaseUrl throw, and writes resolved body into a temp workspace. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GEPA experiments group a seed request and all follow-up requests under a shared experimentId. Both experiment grouping (groupBy=experiment) and AGENTS.md lineage reconstruction filter requests by this field, so add a sparse single-field index to keep those queries efficient. - 018-add-experiment-id-index.ts: try/catch-wrapped createIndex, log-only non-destructive down(). - Registered in required-migrations.ts. - Unit tests cover index creation, idempotent already-exists, and no-op down. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pt features - SubmitRun: optional AGENTS.md textarea + experimentId input - RunDetail: render AGENTS.md as Markdown with lineage/experiment badges - RunsList: add "Experiment" group-by option (client-side grouping by experimentId) - PromptFeatureWizard/List: typed features (task | agents.md) with filter + column - api.ts/types.ts: new request fields, TaskPromptType, getTaskPromptContent Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- run submit: --agents-md <text|@file>, --experiment-id - run list: --experiment-id filter, --group-by (incl. experiment) - task-prompt list/create: --type (task | agents.md); guard blob-backed prompts - prompt-feature list/create: --type filter + create, Type column - Tests for all new options (run/task-prompt/prompt-feature) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add apps/gepa-optimizer, a Python+uv app that drives the GEPA reflective prompt-optimization algorithm to optimize an AGENTS.md file, scoring each candidate with a real Scope benchmark run (Copilot CLI worker + Claude Haiku 4.5). - ScopeAdapter(GEPAAdapter): submit one Scope run per dataset example with agentsMd + experimentId, poll to completion, score from criteriaResults, build per-turn trajectories and reflective dataset (judge + criteria feedback for every turn). Bounded concurrency (default 10, the Linux copilot KEDA ceiling) via asyncio + httpx. - Scoring is derived, never persisted: denominator = requested criteria (skipped/missing count as failed); mean|final|weighted across turns; non-succeeded/empty runs score 0.0, never NaN. - Failure model: per-example errors -> 0.0; systemic majority -> abort, so infrastructure failure never poisons optimization. Idempotent GETs retry with backoff; POSTs are not retried. - Best-effort lineage reconstructed post-hoc from GEPA result.parents. - Preflight validates criterion ids before burning agent runs. - 36 pytest unit tests (scoring, reflective, lineage, client via httpx.MockTransport, adapter via stubbed client). - CI: new gepa job (uv sync --locked + pytest), change filter, ci-summary wiring; root scripts test:gepa/build:gepa; README app table + polyglot exception documented (mirrors apps/gateway). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… optimizer env - app-design.md: typed prompts (task|agents.md), size-based inline/blob storage, experiment grouping, AGENTS.md delivery, scoring-not-persisted - ENV_VARIABLES.md: PROMPT_INLINE_MAX_BYTES + GEPA optimizer (SCOPE_*/GEPA_*) vars - AGENTS.md: gepa-optimizer in repo tree + Key Components section Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…riment flags Captures --type (prompt-feature/task-prompt create), --agents-md, --experiment-id, and --group-by added in the CLI step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TaskPromptDocument.text is now optional (blob-backed docs carry no inline text). Guard all consumers (pickers, list, preview panel) with fallbacks; the preview panel shows a 'stored in blob' note when the body is absent. Fixes tsc build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on 018 Migration 018-add-experiment-id-index was added to REQUIRED_MIGRATIONS in the db-migration step; update the applied/pending lists and totalApplied counts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…y AGENTS.md Replace the two generic Express demo tasks with a single, self-contained optimization target: a Medium-style Django blogging platform backed by a nonrelational cloud database. The seed AGENTS.md is now EMPTY on purpose so GEPA discovers the guidance from scratch. Each run requests the full ancestor closure of the target leaf (uses_azure_documentdb): uses_database -> uses_document_oriented_db, plus uses_azure, both feeding uses_azure_documentdb. Requesting the closure (not just the leaf) gives GEPA graded partial-credit signal instead of a sparse 0/1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The DataTable's getGroupKey switch was missing the 'experiment' case, so selecting Group by > Experiment collapsed every run into a single empty-keyed group instead of grouping by experimentId. Add the missing case and a new Experiment column (visible by default) that shows the short experiment id with the full id on hover. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
compute_run_score gated on outcome == "succeeded", returning 0.0 for any run that did not pass *every* criterion. But Scope only marks a run "succeeded" when all criteria pass; a run that completes and is judged with partial criteria gets outcome "finished". Gating on "succeeded" collapsed GEPA's optimization landscape to a flat 0.0 until a candidate happened to pass all criteria, destroying the partial-credit gradient the reflective loop relies on (a seed passing 3/4 criteria scored 0.0 instead of 0.75). Accept both "succeeded" and "finished" as scoreable outcomes (run completed and was judged); keep "failed" (errored/abandoned) at 0.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #1839
🎉 All tests passed! |
Record the implementation commit/tree, versioned contracts, zero-call validation matrix, current source hashes, and unchanged frozen 256-call evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70
Test Results (Node.js 22)test: Run #358
❌ Some tests failed!
|
Build the example plugin into a unique test-owned project-local directory, use it for process tests, and clean it afterward without touching pre-existing demo output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Use TypeScript transpileModule to emit the trusted process-test plugin into a unique project-local directory, so clean unit jobs need neither example dist nor taxonomy-evaluation declarations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70
Test Results (Node.js 22)test: Run #360
🎉 All tests passed! |
Declare the deterministic handler output schema/type, preserve unattributed retry evidence as indeterminate, and add a disjoint custom prompt corpus with direct Taxonome run/eval/optimize examples. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Record the real custom prompt run, held-out baseline, bounded Ax artifact, artifact replay, full built-in taxonomy run, deterministic readout, and exact 11-call ledger delta separately from the frozen 256-call snapshot. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Append the separate 11-call YAML prompt demo accounting and evidence lineage while keeping the original 256-call snapshot and historical artifacts unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70
Add a separately versioned opaque-identity corpus, verify the exact production prompt projection excludes case metadata and outcome-coded IDs, and mark the first 11-call demo as historical ID-confounded evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Record the opaque-identity run, held-out baseline, bounded artifact, replay, built-in taxonomy run, deterministic readout, and exact 11-call delta without modifying either earlier evidence packet. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Append the separately versioned opaque-ID control, cumulative 278/500 accounting, zero instruction-proposal outcome, and explicit correction of the first demo's identity confound. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70
Test Results (Node.js 22)test: Run #363
🎉 All tests passed! |
Use subject-and-role paths for current trajectories, expected judgments, original-prompt results, optimized artifacts, and optimized-prompt results; expose an offline handler unit-test command distinct from model-backed eval. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Explain that eval scores detector output against gold, counts case executions, makes fresh calls only when model execution is reached, and never caches answers through artifact restore. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Move the built-in Timeout implementation and tests under src/handlers/timeout while preserving scope/timeout-v1 behavior and root package exports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Record clearly named original-prompt results, optimized artifact and results, code-handler test summaries, built-in taxonomy output, readout, and the exact 11-call ledger delta without rewriting earlier packets. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Append the 289/500 cumulative accounting, explicit current filename map, code-handler versus prompt-eval semantics, dedicated Timeout folder, and separate readable-path evidence lineage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Identify dec7cc8 as the committed source used for calls 279-289, record the later Timeout folder move as post-capture validation with zero calls, and test the corrected lineage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Rename the generic runtime to taxonomy-engine, isolate agent-behavior YAML and Timeout in minimal ax-taxonomy, make CLI/static evaluation compose both explicitly, retire unused hard-coded legacy facades, and add package-local neutral/focused test suites. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Record the zero-call taxonomy-engine and ax-taxonomy package split, one-way dependency boundary, package-local/cold-build validation, API cleanup, and unchanged wire/evidence contracts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f3fac40-e7d2-469e-adc2-ef1648ba9310
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70
Preserve the canonical 26-path GEPA feature while inheriting the validated lifecycle and standalone taxonomy stack. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70
Cedric Vidal (cedricvidal)
changed the base branch from
cedricvidal-taxonome-structured-spike
to
cedricvidal-experiment-lifecycle
October 28, 2026 07:45
Test Results (Node.js 22)test: Run #433
🎉 All tests passed! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the GEPA prompt optimizer as the final layer on the canonical Experiment
lifecycle stack.
Restack: now directly based on #1517, branch
cedricvidal-experiment-lifecycleat1b648e711d314557044067f96010a5cbb4861e9c, in native stack 1596. Published head2081c9b62890a735418a0cd14f62ac732c982bcanormally merges the preserved local held heada2f867f7with that exact lifecycle parent; no published history was rewritten. The PR-only diff remains exactly the original 26 GEPA paths, with stable patch ID80883ef2c249dbaff04420ee2f00ecbaed1c85c6. Every non-GEPA path matches the new parent, including removal of stale inherited Taxonome files. This is parent synchronization only: production Taxonome post-processing and Taxonome-aware readout adoption remain deferred.apps/gepa-optimizer/, a Python 3.12 + uv app that evaluates candidateAGENTS.mdprompts through real, project-scoped Scope runs.each evaluated candidate batch through revision-guarded membership updates.
preserving bounded concurrency, derived criteria scoring, reflective
evidence, systemic/per-example failures, idempotent-read retries,
non-retried submissions, and post-hoc candidate lineage.
documentation, and unit tests without duplicating superseded Experiment API,
schema, CLI, Portal, or migration surfaces.
metadata through a focused shared subpath instead of the server-heavy shared
root; this repairs an inherited stacked-base
ioredisbundle failure.Demos
Before
N/A — optimizer/automation layer with no new user-visible Portal or CLI surface.
After
N/A — optimizer/automation layer with no new user-visible Portal or CLI surface.
Testing
Fresh parent-synchronization validation
Actual results at
2081c9b62890a735418a0cd14f62ac732c982bca:pnpm test:gepa && pnpm build:gepa— passed, 44/44 tests and locked uv environment synchronization.pnpm exec vitest run scripts/ci-workflow.test.ts --reporter=default— passed, 24/24 tests.cd apps/gepa-optimizer && uv run python -m compileall -q src tests && uv sync --locked— passed.git diff --check 1b648e71...HEAD— passed.An attempted
uv sync --locked --checkis not supported by the pinned uv 0.6.3; it was not counted as a pass. The supporteduv sync --lockedcommand above passed. No live optimizer, Scope benchmark submission, model inference, recording, or deferred integration was run. Native CI is tracked on the exact published head separately; offline results are not a claim that every whole-stack CI check is green.Preserved historical validation before this parent synchronization
The results below belong to the earlier source implementation, not a new full-suite or live acceptance for the restack. In particular, the older body reported a
--checkvariant as passed; current pinned-tool validation is the supported command documented above.pnpm test:gepa— passed, 44 tests.pnpm exec vitest run scripts/ci-workflow.test.ts— passed, 22 tests.pnpm build:gepa— passed; locked uv environment resolved and audited.cd apps/gepa-optimizer && uv run python -m compileall -q src tests && uv sync --locked --check— historical source report only; not supported or counted as a current pass with pinned uv 0.6.3.pnpm headers:check— passed, 1055 in-scope files.pnpm notice:check— passed after restoring the frozen pnpm dependency store; NOTICE files are current.pnpm build:cli && node --no-global-search-paths apps/cli/dist/scope.mjs --version— passed; isolated bundle prints0.0.0-dev.pnpm vitest run --config vitest.integration.config.ts apps/cli/src/bundle.integration.test.ts apps/cli/src/utils/update-check.integration.test.ts— passed, 24 tests.pnpm vitest run packages/shared/src/experiments/experiment-store.test.ts -t "plans chunked-to-chunked growth and chunked-to-inline shrink independently"— passed; 51ms.pnpm vitest run apps/portal/src/pages/ExperimentDetail.test.tsx -t "starts an asynchronous deletion from the confirmation dialog"— passed; 1.34s.pnpm vitest run --exclude scripts/build-windows-worker.test.ts— passed under full-suite load, 293 files / 3,523 tests; the complete ExperimentStore and ExperimentDetail files passed 14/14 and 16/16.pnpm vitest run --coverage --exclude scripts/build-windows-worker.test.ts— passed under CI-like coverage load, 293 files / 3,523 tests; ExperimentStore completed in 2.34s and ExperimentDetail in 11.59s with all individual tests inside their bounded timeout.pnpm test— both stabilized files passed under full-suite load; the run otherwise had one unrelated localspawnSync pwsh ETIMEDOUTinscripts/build-windows-worker.test.ts.pnpm --filter portal lint && pnpm --filter shared build— passed.pnpm --filter shared lintcould not start because this checkout's local install does not expose theeslintexecutable.http://localhost:3100was unavailable andapps/gepa-optimizer/.envwas absent.Documentation and compatibility
apps/gepa-optimizer/README.md,ENV_VARIABLES.md,docs/architecture/app-design.md,docs/architecture/experiments.md, androot
AGENTS.md.report contracts from the base branch.
subpath; no breaking API/CLI changes and no migration or upgrade step.
Checklist
up()/down()and keep it CosmosDB-compatible. N/A — no database changes.NOTICE/NOTICE-REVIEW.txtwithpnpm noticeas needed.uv.lockis committed andpnpm notice:checkpasses.