Skip to content

[SCR-439] Survive a per-zone API error - #440

Closed
itsryanarmani wants to merge 3 commits into
scr-435-mail-fleet-auditfrom
scr-439-per-zone-resilience
Closed

itsryanarmani wants to merge 3 commits into
scr-435-mail-fleet-auditfrom
scr-439-per-zone-resilience

Conversation

@itsryanarmani

@itsryanarmani itsryanarmani commented Sep 16, 2026 •

Copy link
Copy Markdown
Owner

Closes #439

What

A zone that cannot be read no longer aborts the fleet mail audit. It prints COULD NOT AUDIT, counts as a fault, and the rest of the fleet is still audited. GET /zones itself still fails the process.

Why

#436 dies on a transient API error. A 19-zone fleet hits a timeout or 429 often enough that aborting means the tool never finishes, and an audit nobody can run is the state this script exists to end. Unknown was indistinguishable from clean.

How I verified

$ bash mail-fleet-audit.test.sh
ok - sending subdomain without MX exits non-zero
ok - missing MX is reported on the sending host
ok - missing MX is listed under FAULTS
ok - third-party apex MX does not cause a non-zero exit
ok - third-party apex MX is reported as Zoho
ok - third-party MX is not a fault
ok - healthy fleet exits zero
ok - healthy sending subdomain MX is reported
ok - healthy fleet prints the ok line
ok - unreadable zone exits non-zero
ok - unreadable zone is named in the report
ok - a later zone is still audited after an unreadable one
ok - a failed /zones call exits with the API-error code
ok - failed /zones call is not reported as clean
ok - a rate-limited zone exits non-zero
ok - rate-limited zone is named in the report
ok - a later zone is still audited after a rate-limited one

passed 17, failed 0

Tests stub both curl and dig and refuse the real network, so none of them send or resolve anything.

Every resilience test was mutation-checked — break the behaviour, confirm that test goes red, restore, confirm green:

Mutation Red
swallow an unreadable zone instead of recording it unreadable zone exits 0
abort the run on the first failure unreadable zone not reported; audit stopped
treat a 429 as a crash rate-limited zone not reported; later zone skipped
treat a failed /zones call as zero zones failed /zones call reported as clean

Operator: not applicable - a read-only command line audit with no user-visible surface and no writes. Resilience is process behaviour of that same command.

Proof: n/a - a terminal tool with no visual surface; the command output above is the evidence.

Assisted-by: grok:grok-4.6


Devin Review

A timeout or 429 on one zone printed nothing and aborted
the run. Record that zone as a fault and keep going.
@coderabbitai

coderabbitai Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 51a48177-f3ec-4919-b9f5-0ff42fb19916

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread mail-fleet-audit
proc = subprocess.run(
[
CURL, "-sS", "--max-time", "20", "-w", "\n%{http_code}",
CURL, "-sS", "--max-time", "20", "--retry", "3", "--retry-delay", "2", "-w", "\n%{http_code}",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Retries corrupt recovered API responses

When a retried HTTP error later succeeds, curl --retry preserves every response body in stdout. cf_get rejects the concatenated JSON, so recovered requests still fail the audit.

Learn more

Curl's retry machinery reuses the same output stream for every attempt. A transient 429 or 5xx body therefore remains before the final successful body. The status parser sees the final status, but json.loads receives all bodies concatenated and rejects them.

Example: Cloudflare returns {"success":false} with HTTP 429, then {"success":true,"result":[]} with HTTP 200. Curl outputs both JSON objects followed by 200; the audit reports COULD NOT AUDIT instead of accepting the successful retry.

Recommended fix: Retry each request at the Python level and parse only the final attempt, or direct each curl attempt into replaceable output so failed response bodies cannot enter the final JSON payload. Add a stub or local-server test that returns a transient HTTP error before success.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread mail-fleet-audit
Comment on lines +78 to +83
# One slow zone must not end the audit. A 19-zone fleet hits a
# transient timeout often enough that aborting means the tool is
# never run, and an audit nobody runs is the state this script
# exists to end. "Could not look" is not "nothing wrong", so the
# caller records it as a fault rather than a clean zone.
raise ApiUnavailable(f"{path}: {err}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Top-level transport failures exit incorrectly

After the top-level zone request exhausts retries, cf_get raises ApiUnavailable outside the per-zone handler. main emits a traceback and exits 1 instead of the documented API-error code 2.

Learn more

required=True only changes how cf_list handles completed HTTP responses. A curl transport failure occurs earlier in cf_get, which cannot see that flag. No ApiUnavailable handler surrounds the top-level call, so Python handles the exception as an uncaught error.

Example: If the initial active-zone request times out after all retries, the command prints a traceback and returns 1. A completed HTTP 404 for the same request uses die and returns the documented code 2.

Recommended fix: Catch ApiUnavailable around the required top-level cf_list call and pass its message to die, while retaining the per-zone handler for later requests. Add a top-level transport-failure test rather than only an HTTP-status test.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread mail-fleet-audit
Comment on lines +116 to +117
if required:
die(f"GET {path} failed: API returned success=false")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 API failures appear as empty zones

For a per-zone HTTP 200 response with success=false, cf_list returns accumulated items instead of raising ApiUnavailable. The zone appears clean or partially audited despite Cloudflare's explicit failure.

Learn more

Cloudflare communicates application-level failures through the JSON success field even when an HTTP response was received. The required top-level request exits on this value, but a per-zone request falls through to the old return items. On page one this becomes an empty successful list; on a later page it becomes a truncated successful list.

Example: A sending-subdomains request returns HTTP 200 with {"success":false,"result":null}. The report prints sending: (none) and can exit clean instead of printing COULD NOT AUDIT.

Recommended fix: Raise ApiUnavailable for success=false when required is false, mirroring the other malformed or failed per-zone responses. Add a test for an HTTP 200 failure envelope.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

vibecodereview[bot] added 2 commits September 16, 2026 13:42
The closing summary line said "N sending host(s) cannot receive" even
when N counted zones that errored out (rate limit, timeout) rather
than a confirmed missing MX record, falsely asserting a bounce that
was never verified. Track the two fault kinds separately.
- a transport-level failure on the top-level GET /zones call escaped
  as an uncaught exception instead of the documented exit 2
- a 200 response with success:false on a per-zone subresource was
  read as a clean, empty zone instead of a fault

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head: 648c18de0ce374a5a66a86808e117c14e188c6eb. Pushed a fix on top at 2d76ec0a14bae6a8a4ee223821583a93782a41e1.

Fixed (pushed)

Major — a transport failure on GET /zones crashed instead of exiting 2. cf_get was changed so any transport-level failure (timeout, connection refused) raises ApiUnavailable unconditionally. The per-zone loop catches that, but the top-level zones = cf_list("/zones?status=active", required=True) call in main() sits above the loop and had no except — a real network blip on the very first call (the exact "broken CLOUDFLARE_API_BASE" scenario the PR's own test 5 comment names) produced an uncaught Python traceback and exit 1, not the documented "Exit 2 ... GET /zones itself failed". Fixed by wrapping that call in try/except ApiUnavailable: die(...). Added test 7, which simulates a transport-level /zones timeout and asserts exit 2 with no traceback.

Major — success:false with HTTP 200 on a per-zone subresource was read as a clean, empty zone. cf_list's new required branches correctly turn a non-200 status, non-JSON body, or unexpected result shape into ApiUnavailable when not required — but the sibling success is False branch was left as return items (silently "no sending subdomains") for the non-required (per-zone) case. That's exactly the class of failure this PR exists to stop from looking clean, and it wasn't covered by any of the new tests (test 6 only exercises non-200 statuses; the stub always pairs success:false with a non-200 status by default). Fixed to raise ApiUnavailable there too, symmetric with the other branches. Added test 8, which forces a 200 + success:false response on one zone and asserts it's reported as COULD NOT AUDIT, not silently clean.

Both fixes came with new regression tests mirroring the existing test 4/5/6 pattern (offline, stubbed curl/dig). I could not execute the shell test suite myself in this sandbox (script execution is blocked by the session's permission policy), so CI running mail-fleet-audit.test.sh on the pushed commit is the verification step here — please confirm it's green.

Considered and left alone

  • mail-fleet-audit.test.sh gained fixture plumbing for /zones/*/email/routing/rules and /zones/*/email/routing/rules/catch_all (new rules/catch_all fixture dirs, new stub cases) that no code in mail-fleet-audit ever calls and no test exercises. Harmless dead scaffolding, but it's outside this issue's stated scope (per-zone resilience) — worth dropping in a follow-up rather than carrying it forward silently.
  • Council's correctness lens returned no findings; the security lens errored (OpenRouter/moonshot timeout) and produced nothing to verify.
  • PR-standard checks pass: branch/title/body shape, single Closes #439, Assisted-by: line, well under the 500-line/40-file caps even with this fix.
🧑‍⚖️ Council
Model Lens Result
GPT-5.6 (Codex) correctness no findings
Kimi K3 security errored (no findings produced)
Claude (chair) correctness / proof / conventions 2 confirmed, fixed

@itsryanarmani
itsryanarmani deleted the branch scr-435-mail-fleet-audit September 16, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant