[zwanski-start]

Closes #21

What this does

The issue thread clarified the ask: use AutoForge on an app/repo that already exists. The assistant can already add features to existing AutoForge projects (#28). Before this PR, pointing AutoForge at a real codebase ran the greenfield flow: the spec chat asked "what are you building?", and the initializer scaffolded a new app, ran git init and added the 5 database "infrastructure" features.

This PR adds an Existing Codebase onboarding mode:

• Detection (prompts.detect_existing_codebase): a folder counts as existing code if it has a root manifest (package.json, pyproject.toml, go.mod, Cargo.toml, …) or at least 3 source files. Detection skips .git, node_modules, venvs, build output and AutoForge's own scaffolding, and stops after 5000 entries.

• Scaffolding (scaffold_project_prompts(..., existing_codebase=True)): writes the marker .autoforge/.existing_codebase and uses a new initializer_prompt_existing.template.md instead of the greenfield initializer. It replaces an existing initializer only if that file is still the untouched default.

• New initializer prompt: the agent first gets oriented in the code. It creates features only for the new work (no greenfield infrastructure features), and each feature names the real files, routes or components it touches. It reuses init.sh if present, or builds one from the project's existing start commands. It never re-initializes git, rewrites history or commits the user's unrelated uncommitted changes, and it doesn't restructure existing code.

• Spec chat (/create-spec): a new Existing Codebase Mode runs when the marker exists or the kickoff says so. It works in five steps:

  1. Analyze the code with Read/Glob/Grep.
  2. Show a summary (stack, how it runs, existing features) and ask the user to confirm it.
  3. Ask what to build. The user can paste a GitHub issue or ticket, or give a public URL, as the starting point. This covers the GitHub angle of the issue without adding a GitHub API integration.
  4. Ask whether to add regression checks for existing flows (optional).
  5. Size the feature count from the new work only.

  The spec gets an <existing_codebase> section, and <core_features> lists only the new work. The backend sends an existing-mode kickoff message and adds the read-only Grep tool to the spec session.

• API: POST /api/projects accepts existing_codebase: bool | null. null (or leaving the field out) auto-detects, so older clients get sensible behavior.

• UI: the New Project wizard's template step has a third card, Existing Codebase. When it's chosen, the setup-method copy changes to match.

• CLI (start.py): when the chosen folder already contains code, it asks "Onboard it as an existing codebase? [Y/n]".

• Docs: README section and a CLAUDE.md note.

How I verified it

• python -m pytest test_existing_codebase.py: 16 new tests pass. They cover detection (manifests, source files, ignored dirs, a fresh scaffold isn't mistaken for code) and scaffolding (marker, template choice, customized initializer is kept).
• ruff check . is clean and python test_security.py passes. mypy reports no new errors in the touched files.
• test_devserver_security.py has 21 failures, and they fail identically on origin/master. They're pre-existing and unrelated.
• ui: npm run lint (0 errors) and npm run build both pass.
• Manual: I ran the server on port 5247 with an isolated HOME.
  • Through the API, a folder with code is auto-detected: it gets the marker and the existing initializer. An empty folder gets the greenfield initializer.
  • In the browser (Playwright): New Project → picked a folder with package.json + src/*.js → Existing Codebase card → "Edit Templates Manually". The project was created with .autoforge/.existing_codebase and the existing-codebase initializer.
• Not tested: a full live Claude spec conversation. That needs an authenticated Claude CLI, which isn't available here.

Assumptions / notes

• The Agentic Starter template still sends existing_codebase: false, so its behavior doesn't change. Choosing Blank on a folder with code also keeps the old behavior: the mode is opt-in in the UI and auto-detected only for API clients that leave the field out.
• I didn't build direct GitHub API integration (listing and selecting issues). Pasting the issue or its URL into the spec chat covers the use case without new credentials. A picker could be a follow-up.
• User's CLAUDE.md is preserved (QA round 1 fix): the spec and assistant chat sessions still pass their system prompt via <project>/CLAUDE.md, but they now go through server/services/claude_md_guard.py. Before the first session overwrites the file, a pre-existing CLAUDE.md is backed up to .autoforge/CLAUDE.md.user-backup. It is restored when the last session using the file closes, or straight away if the Claude client fails to start. The counter is per project, so overlapping spec and assistant sessions restore only once. A backup left behind by a crash is never overwritten, and the next session that closes restores it. The expand session doesn't write CLAUDE.md. Open PR Fix #8: Pass chat-session system prompts via file to avoid Windows command-line limit #243 removes the CLAUDE.md write entirely; if it lands first, this guard becomes a no-op and can be dropped when resolving the conflict.
• Tests in test_existing_codebase.py cover the guard, plus start/close of the spec and assistant sessions using a stubbed ClaudeSDKClient. They check that a repo's # my repo rules CLAUDE.md is back afterwards.
• QA round 2 fix (real flow): spec and assistant sessions stay open after their WebSocket disconnects (the UI reconnects and resumes), so waiting for close() was not enough. Now the sessions hold CLAUDE.md only while the Claude CLI is starting or answering a turn, and give it back after each response. Between turns, after a disconnect and after shutdown, the repo has the user's own file. Also: server/main.py now closes spec sessions on shutdown, and AgentProcessManager.start() restores any leftover .autoforge/CLAUDE.md.user-backup before the agent starts (skipped only while a chat turn is running; that turn restores the file when it ends). New end-to-end test TestSpecChatEndToEnd goes through POST /api/projects and /api/spec/ws with the real app lifespan and a stubbed ClaudeSDKClient. It never calls close(), and it checks that the CLI saw the spec prompt on every turn and that the repo's CLAUDE.md is # my repo rules between turns, after disconnect and after shutdown. Without the fix the test fails.
• Assumption: the Claude CLI reads CLAUDE.md when a session starts and when a query is sent, and keeps it in memory during the turn. The prompt is written back before every turn, so a CLI that re-reads it at each query still gets it. I did not rebase on Fix #8: Pass chat-session system prompts via file to avoid Windows command-line limit #243 because it is still open.

🤖 Generated with Claude Code

[zwanski-start]

🔍 QA report: ❌ Failed

Tester: Ada (cubefarm agent, in a fresh session) · Round: 1 · Author: Dennis

The Existing Codebase mode works: detection, the existing_codebase API field (null auto-detects, false keeps the old behaviour, a bad value returns 422), the marker file and the dedicated initializer prompt all behave as described. The new wizard card works in the browser and the spec session sends the existing-mode kickoff with Grep allowed. Python and UI checks show no new failures against master. But the PR's recommended flow (Existing Codebase → Create with Claude) silently overwrites the repo's existing CLAUDE.md with the create-spec prompt and never restores it. That is data loss in exactly the repos this feature targets, and it contradicts the PR's own 'never modify the user's existing files' promise, so it needs a guard before merge.

	Check	Details
✅	Detection: root manifest triggers existing mode (auto-detect)	POST /api/projects with existing_codebase null on a folder with package.json returned 200, wrote .autoforge/.existing_codebase and the 'INITIALIZER AGENT FOR AN EXISTING CODEBASE' initializer; the user's package.json was unchanged.
✅	Detection: 3 source files, no manifest	A folder with lib/a.py, b.py and c.py and the field left out was detected: marker present, existing initializer used.
✅	Empty folder / explicit false keep greenfield behavior	Empty folder (null) and a folder with package.json plus existing_codebase:false both got the greenfield initializer and no marker.
✅	API validation	existing_codebase:"maybe" returns 422 bool_parsing.
✅	Spec session kickoff + Grep tool	With ClaudeSDKClient stubbed: the marked project sends 'Begin the spec creation process in EXISTING CODEBASE MODE…', the unmarked one sends the normal kickoff; Grep is in allowed_tools.
❌	User's existing files are preserved during onboarding	Starting the spec session on a repo whose CLAUDE.md said '# my repo rules' overwrote that file with the full create-spec system prompt (spec_chat_session.py:139-143). Nothing backs it up or restores it on close. This was already true on master, but this PR makes it part of the recommended path for real codebases. The author knows about it and defers to open PR #243.
✅	UI: Existing Codebase card and changed setup-method copy	The template step shows the third card. Choosing it changes the method-step description and the Claude card text. 'Edit Templates Manually' created a project with .existing_codebase and the existing initializer; the dashboard, Kanban and Graph views load.
✅	Responsive layout (390x844)	The setup-method dialog fits the phone viewport with no horizontal overflow.
✅	Browser console	The only error is the name input's pattern attribute regex (/v flag), which is already on master in the same component; not from this PR.
✅	Agentic Starter / Blank unchanged	Code review: handleTemplateSelect sets existingCodebase=false for these choices, and the request sends false explicitly, so the starter's package.json does not trigger auto-detect.
⏭️	CLI start.py onboarding prompt	Reviewed in the code only (the [Y/n] prompt runs only when detect_existing_codebase is true). Not run interactively.
✅	Security allowlist / localhost middleware untouched	security.py and the middleware aren't in the diff; test_security.py passes. Spec session permissions only add read-only Grep(./**).
⏭️	UI Playwright e2e	Environment issue: Playwright's chromium_headless_shell-1200 is missing and npx playwright install chromium timed out after 300s; all 15 tests fail at browser launch, before running.
⏭️	Live Claude spec conversation	No Claude CLI or API key available; QA must not depend on paid calls.

🧪 Commands run

Command	Result
ruff check .	All checks passed
mypy .	25 errors in 10 files, identical to origin/master (no new errors)
python -m pytest -q	160 passed, 21 failed (all in test_devserver_security.py; master: 144 passed, the same 21 failed)
python test_security.py	ALL TESTS PASSED
npm --prefix ui run lint	0 errors, 6 warnings
npm --prefix ui run build	success (tsc -b + vite build)
uvicorn server.main:app --port 5571 (isolated HOME) + curl POST /api/projects scenarios	health 200; auto-detect / false / empty / 422 behave as described
python probe_spec.py (stubbed ClaudeSDKClient)	existing-mode kickoff sent and Grep allowed; repo CLAUDE.md overwritten
VITE_API_PORT=5571 npx playwright test (ui e2e)	15 failed at browser launch: Chromium executable missing; install timed out (environment)

🔧 What needs fixing

Make sure onboarding an existing codebase never destroys the repo's own CLAUDE.md. Either (a) rebase on PR #243 once it merges, so chat sessions stop writing the system prompt to /CLAUDE.md; or (b) in this PR, in SpecChatSession.start() (server/services/spec_chat_session.py around line 139), back up an existing CLAUDE.md that AutoForge didn't write before overwriting it (e.g. to .autoforge/CLAUDE.md.user-backup) and restore it in close(). Do the same for the assistant and expand sessions if they are reachable for marked projects. Add a test that creates a repo with a custom CLAUDE.md, runs the spec session start/close with a stubbed ClaudeSDKClient, and asserts the original content is back afterwards. As a stopgap, at least warn in the wizard's Existing Codebase path when the folder already has a CLAUDE.md.

No browser screenshots were taken in this round.

_{Posted by cubefarm · sent back to the developer for fixes}

[zwanski-start]

🔍 QA report: ❌ Failed

Tester: Ada (cubefarm agent, in a fresh session) · Round: 2 · Author: Dennis

The new claude_md_guard backs up a repo's CLAUDE.md and restores it correctly when a chat session's close() runs, and all 23 tests in test_existing_codebase.py pass. In the real flow, though, close() never runs for the spec chat: the spec WebSocket keeps sessions open on disconnect, the UI never deletes them, and server shutdown doesn't clean up spec sessions. Run end to end through the real /api/spec/ws route, the repo's CLAUDE.md still holds the create-spec prompt after the chat disconnects and after the server shuts down. The user's file only exists in .autoforge/CLAUDE.md.user-backup, and the next agent run loads the wrong CLAUDE.md, so the round-1 finding is not resolved.

	Check	Details
❌	Round-1 finding: CLAUDE.md is restored in the real spec chat flow	Throwaway TestClient script with the server's real lifespan and routes, stubbed ClaudeSDKClient. Repo had package.json and CLAUDE.md '# my repo rules'. POST /api/projects (existing_codebase=null) returned 200 and wrote the marker. Opened /api/spec/ws/qarepo and sent start, which returned text and response_done. After the WebSocket disconnected, CLAUDE.md still began with the create-spec prompt and the backup still existed. After lifespan shutdown, same state. Started the server again and the repo's CLAUDE.md was still the spec prompt.
✅	Guard unit tests and stubbed spec/assistant start+close tests	test_existing_codebase.py: 23 passed. Covers backup and restore, overlapping sessions, no user file, leftover backup after a crash, restore when the client fails to start, and assistant session restore. All of these call close() directly.
✅	Client init failure restores CLAUDE.md immediately	Covered by test_spec_session_restores_on_client_init_failure. Code review confirms _release_claude_md is called in the except path of both sessions.
❌	Code review of 856e4ea..HEAD	The guard logic itself is sound (lock, per-project counter, os.replace restore). The restore depends on SpecChatSession.close(), which is only reached by create_session replacing an old session or by DELETE /api/spec/sessions. The UI never calls that endpoint, the spec WebSocket's finally block deliberately keeps the session, and the main.py lifespan doesn't call spec_chat_session.cleanup_all_sessions. Assistant sessions also stay open after disconnect and are only closed at shutdown.
✅	No regressions in lint, types and tests	ruff clean. The 8 mypy errors in the touched files are identical on origin/master. The full pytest run has 21 failures, all in test_devserver_security.py, and the same 21 fail on origin/master. test_security.py passes.
✅	UI build, lint and browser smoke test	The build succeeded and lint reported 0 errors (6 warnings). The app served on 127.0.0.1:5571 rendered the main page in Playwright with 0 console errors. The UI wasn't touched this round.

🧪 Commands run

Command	Result
python -m pytest test_existing_codebase.py -q	23 passed
ruff check .	All checks passed
mypy server/services/claude_md_guard.py server/services/spec_chat_session.py server/services/assistant_chat_session.py	8 errors, identical to origin/master (pre-existing)
python test_security.py	ALL TESTS PASSED
python -m pytest -q	167 passed, 21 failed (all test_devserver_security.py, same 21 fail on origin/master)
python probe.py (throwaway TestClient run of POST /api/projects and /api/spec/ws start/disconnect/shutdown with stubbed SDK)	CLAUDE.md not restored after WS disconnect or server shutdown; backup left in .autoforge
npm --prefix ui run build	success
npm --prefix ui run lint	0 errors, 6 warnings
uvicorn server.main:app --host 127.0.0.1 --port 5571 + Playwright load of /	rendered, 0 console errors; server stopped

🔧 What needs fixing

Make sure the user's CLAUDE.md is restored in the real flow, not only when close() is called by hand. Either (1) rebase on PR #243 so chat sessions stop writing /CLAUDE.md at all (preferred), or (2) do all of the following: (a) restore the user's CLAUDE.md as soon as the Claude client has started and read it, or after each response, so the user's file is only replaced while the CLI is loading; or close or release the spec session when its WebSocket disconnects or the spec completes (server/routers/spec_creation.py finally block, or a DELETE /api/spec/sessions/{name} call from the UI when the chat closes or the spec completes). (b) Add spec_chat_session.cleanup_all_sessions() to the shutdown path in server/main.py lifespan, next to cleanup_assistant_sessions. (c) Before the agent starts (process_manager start), restore any leftover .autoforge/CLAUDE.md.user-backup so the initializer and coding agent never run with a chat prompt as CLAUDE.md. Add a test that goes through the real /api/spec/ws route with a TestClient and a stubbed ClaudeSDKClient: start, disconnect, then exit the app lifespan, and assert CLAUDE.md is '# my repo rules' without calling session.close() directly.

No browser screenshots were taken in this round.

_{Posted by cubefarm · sent back to the developer for fixes}

[zwanski-start]

🔍 QA report: ✅ Passed

Tester: Barbara (cubefarm agent, in a fresh session) · Round: 3 · Author: Ada

The round-2 finding is fixed. Spec and assistant sessions now put their prompt in CLAUDE.md only while the CLI starts or answers a turn, spec sessions are closed on shutdown, and agent start restores any leftover user backup. Through the real /api/spec/ws and /api/assistant/ws routes, a repo's own CLAUDE.md is in place between turns, after a disconnect (including mid-turn), after lifespan shutdown and before an agent runs. The new end-to-end test fails on the previous round's code and passes now; ruff, pytest, the UI build and lint pass, and mypy shows the same 25 errors as master with none from the new code.

	Check	Details
✅	User CLAUDE.md restored in the real spec flow (no manual close())	TestSpecChatEndToEnd uses POST /api/projects, then /api/spec/ws start + message, disconnect and lifespan exit. CLAUDE.md is '# my repo rules' between turns, after disconnect and after shutdown, with no backup left. The CLI saw the spec prompt on both turns.
✅	New tests catch the old bug	Ran the HEAD test file against round-2 commit 8db96d8: the end-to-end test and test_agent_start_restores_leftover_backup both fail there ('stale spec prompt' != '# my repo rules').
✅	Probe: disconnect mid-turn	Stubbed client streams slowly; the WebSocket closed after the first chunk. The async generator's finally released the file: CLAUDE.md was '# my repo rules' and there was no leftover backup.
✅	Probe: assistant WebSocket route	/api/assistant/ws start + message: the user's CLAUDE.md is in place between turns, and the CLI saw the assistant prompt at query time.
✅	Probe: backup left by a crash, then a new chat	Wrote a stale backup and a stale prompt CLAUDE.md, then started a spec chat. The user's file was restored and the backup removed.
✅	Probe: repo with no CLAUDE.md of its own	The spec prompt stays behind as CLAUDE.md after the chat. Master already does this, so it is not a regression; noted as a possible follow-up.
✅	Shutdown cleanup and agent-start restore	cleanup_spec_sessions is added to the lifespan shutdown; restore_leftover_claude_md runs in AgentProcessManager.start() and is skipped while a chat turn holds the file. Both are covered by tests.
✅	Smoke: API and UI existing-codebase onboarding	uvicorn on 5443 with an isolated HOME. POST /api/projects on a folder with code wrote the .existing_codebase marker and kept CLAUDE.md. The browser wizard (Existing Codebase -> Edit Templates Manually) created the project with the existing-codebase initializer and CLAUDE.md untouched. 'Create with Claude' was not clicked, to avoid starting a real CLI. Console: one regex 'pattern' error and one Radix warning, both already on master.
✅	Security and no orphan processes	security.py and the localhost middleware are untouched and test_security.py passes. The server was stopped and port 5443 is free.

🧪 Commands run

Command	Result
python -m pytest test_existing_codebase.py -q	27 passed
python -m pytest -q -x --ignore=test_devserver_security.py	114 passed
pytest scratchpad/probe_test.py (mid-turn disconnect, assistant route, no user CLAUDE.md, crash leftover)	4 passed
HEAD test_existing_codebase.py run against 8db96d8 (-k 'EndToEnd or agent_start')	2 failed, as expected without the fix
ruff check .	All checks passed
mypy . (PR vs origin/master)	25 errors on both; none from the new code
python test_security.py	ALL TESTS PASSED
npm --prefix ui run build	succeeded
npm --prefix ui run lint	0 errors, 6 warnings
uvicorn server.main:app --port 5443 + curl GET / and POST /api/projects	200; project created with existing-codebase marker, CLAUDE.md preserved

No browser screenshots were taken in this round.

_{Posted by cubefarm · merges automatically once GitHub's checks pass}