Repository navigation
[SCR-439] Survive a per-zone API error - #440
itsryanarmani wants to merge 3 commits into
Conversation
A timeout or 429 on one zone printed nothing and aborted the run. Record that zone as a fault and keep going.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
There was a problem hiding this comment.
Devin Review found 3 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| proc = subprocess.run( | ||
| [ | ||
| CURL, "-sS", "--max-time", "20", "-w", "\n%{http_code}", | ||
| CURL, "-sS", "--max-time", "20", "--retry", "3", "--retry-delay", "2", "-w", "\n%{http_code}", |
There was a problem hiding this comment.
🟡 Retries corrupt recovered API responses
When a retried HTTP error later succeeds, curl --retry preserves every response body in stdout. cf_get rejects the concatenated JSON, so recovered requests still fail the audit.
Learn more
Curl's retry machinery reuses the same output stream for every attempt. A transient 429 or 5xx body therefore remains before the final successful body. The status parser sees the final status, but json.loads receives all bodies concatenated and rejects them.
Example: Cloudflare returns {"success":false} with HTTP 429, then {"success":true,"result":[]} with HTTP 200. Curl outputs both JSON objects followed by 200; the audit reports COULD NOT AUDIT instead of accepting the successful retry.
Recommended fix: Retry each request at the Python level and parse only the final attempt, or direct each curl attempt into replaceable output so failed response bodies cannot enter the final JSON payload. Add a stub or local-server test that returns a transient HTTP error before success.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # One slow zone must not end the audit. A 19-zone fleet hits a | ||
| # transient timeout often enough that aborting means the tool is | ||
| # never run, and an audit nobody runs is the state this script | ||
| # exists to end. "Could not look" is not "nothing wrong", so the | ||
| # caller records it as a fault rather than a clean zone. | ||
| raise ApiUnavailable(f"{path}: {err}") |
There was a problem hiding this comment.
🟡 Top-level transport failures exit incorrectly
After the top-level zone request exhausts retries, cf_get raises ApiUnavailable outside the per-zone handler. main emits a traceback and exits 1 instead of the documented API-error code 2.
Learn more
required=True only changes how cf_list handles completed HTTP responses. A curl transport failure occurs earlier in cf_get, which cannot see that flag. No ApiUnavailable handler surrounds the top-level call, so Python handles the exception as an uncaught error.
Example: If the initial active-zone request times out after all retries, the command prints a traceback and returns 1. A completed HTTP 404 for the same request uses die and returns the documented code 2.
Recommended fix: Catch ApiUnavailable around the required top-level cf_list call and pass its message to die, while retaining the per-zone handler for later requests. Add a top-level transport-failure test rather than only an HTTP-status test.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if required: | ||
| die(f"GET {path} failed: API returned success=false") |
There was a problem hiding this comment.
🟡 API failures appear as empty zones
For a per-zone HTTP 200 response with success=false, cf_list returns accumulated items instead of raising ApiUnavailable. The zone appears clean or partially audited despite Cloudflare's explicit failure.
Learn more
Cloudflare communicates application-level failures through the JSON success field even when an HTTP response was received. The required top-level request exits on this value, but a per-zone request falls through to the old return items. On page one this becomes an empty successful list; on a later page it becomes a truncated successful list.
Example: A sending-subdomains request returns HTTP 200 with {"success":false,"result":null}. The report prints sending: (none) and can exit clean instead of printing COULD NOT AUDIT.
Recommended fix: Raise ApiUnavailable for success=false when required is false, mirroring the other malformed or failed per-zone responses. Add a test for an HTTP 200 failure envelope.
Was this helpful? React with 👍 or 👎 to provide feedback.
The closing summary line said "N sending host(s) cannot receive" even when N counted zones that errored out (rate limit, timeout) rather than a confirmed missing MX record, falsely asserting a bounce that was never verified. Track the two fault kinds separately.
- a transport-level failure on the top-level GET /zones call escaped as an uncaught exception instead of the documented exit 2 - a 200 response with success:false on a per-zone subresource was read as a clean, empty zone instead of a fault
There was a problem hiding this comment.
Reviewed head: 648c18de0ce374a5a66a86808e117c14e188c6eb. Pushed a fix on top at 2d76ec0a14bae6a8a4ee223821583a93782a41e1.
Fixed (pushed)
Major — a transport failure on GET /zones crashed instead of exiting 2. cf_get was changed so any transport-level failure (timeout, connection refused) raises ApiUnavailable unconditionally. The per-zone loop catches that, but the top-level zones = cf_list("/zones?status=active", required=True) call in main() sits above the loop and had no except — a real network blip on the very first call (the exact "broken CLOUDFLARE_API_BASE" scenario the PR's own test 5 comment names) produced an uncaught Python traceback and exit 1, not the documented "Exit 2 ... GET /zones itself failed". Fixed by wrapping that call in try/except ApiUnavailable: die(...). Added test 7, which simulates a transport-level /zones timeout and asserts exit 2 with no traceback.
Major — success:false with HTTP 200 on a per-zone subresource was read as a clean, empty zone. cf_list's new required branches correctly turn a non-200 status, non-JSON body, or unexpected result shape into ApiUnavailable when not required — but the sibling success is False branch was left as return items (silently "no sending subdomains") for the non-required (per-zone) case. That's exactly the class of failure this PR exists to stop from looking clean, and it wasn't covered by any of the new tests (test 6 only exercises non-200 statuses; the stub always pairs success:false with a non-200 status by default). Fixed to raise ApiUnavailable there too, symmetric with the other branches. Added test 8, which forces a 200 + success:false response on one zone and asserts it's reported as COULD NOT AUDIT, not silently clean.
Both fixes came with new regression tests mirroring the existing test 4/5/6 pattern (offline, stubbed curl/dig). I could not execute the shell test suite myself in this sandbox (script execution is blocked by the session's permission policy), so CI running mail-fleet-audit.test.sh on the pushed commit is the verification step here — please confirm it's green.
Considered and left alone
mail-fleet-audit.test.shgained fixture plumbing for/zones/*/email/routing/rulesand/zones/*/email/routing/rules/catch_all(newrules/catch_allfixture dirs, new stub cases) that no code inmail-fleet-auditever calls and no test exercises. Harmless dead scaffolding, but it's outside this issue's stated scope (per-zone resilience) — worth dropping in a follow-up rather than carrying it forward silently.- Council's correctness lens returned no findings; the security lens errored (OpenRouter/moonshot timeout) and produced nothing to verify.
- PR-standard checks pass: branch/title/body shape, single
Closes #439,Assisted-by:line, well under the 500-line/40-file caps even with this fix.
🧑⚖️ Council
| Model | Lens | Result |
|---|---|---|
| GPT-5.6 (Codex) | correctness | no findings |
| Kimi K3 | security | errored (no findings produced) |
| Claude (chair) | correctness / proof / conventions | 2 confirmed, fixed |
Closes #439
What
A zone that cannot be read no longer aborts the fleet mail audit. It prints
COULD NOT AUDIT, counts as a fault, and the rest of the fleet is still audited.GET /zonesitself still fails the process.Why
#436 dies on a transient API error. A 19-zone fleet hits a timeout or 429 often enough that aborting means the tool never finishes, and an audit nobody can run is the state this script exists to end. Unknown was indistinguishable from clean.
How I verified
Tests stub both curl and dig and refuse the real network, so none of them send or resolve anything.
Every resilience test was mutation-checked — break the behaviour, confirm that test goes red, restore, confirm green:
/zonescall as zero zones/zonescall reported as cleanOperator: not applicable - a read-only command line audit with no user-visible surface and no writes. Resilience is process behaviour of that same command.
Proof: n/a - a terminal tool with no visual surface; the command output above is the evidence.
Assisted-by: grok:grok-4.6