Repository navigation
fix(services): apply fault injection to worker jobs, guard concurrent fault transitions, clarify magnitude convention - #13
Merged
Harikeshav-R merged 3 commits intoSep 11, 2026
Conversation
worker_loop only called fault_manager.wait_if_stalled, so LATENCY, ERROR_RATE, and CPU_SPIN faults applied via POST /admin/fault were no-ops against job processing -- only STALL worked. Every other service's HTTP middleware calls pre_request_hook once per request; the worker has no HTTP request in its polling loop, so call it directly once per iteration instead. Factor the loop body into _worker_iteration so it can be exercised directly under test without driving a real polling loop, and so pre_request_hook's ERROR_RATE branch (which raises HTTPException -- meaningful in an HTTP handler, not here) can be handled the same way a real job-processing failure already is: logged and the job skipped for that iteration, rather than crashing the polling task. Implements build-plan step A1.2 (fix on an already-completed service). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
FaultManager.apply_fault/clear_fault mutate shared state (active_fault, expires_at, _timer_task, _leaked_buffers, _held_connections) across multiple await points with no synchronization. Two admin apply_fault/clear_fault calls racing each other, or the independent _ttl_watcher background task clearing a fault out from under a fresh apply_fault, could leave the manager in a torn state -- e.g. a TTL expiry cancelling the wrong timer, or a connection appended to _held_connections during clear_fault's iteration over the same list. Add an asyncio.Lock guarding the whole state transition. apply_fault internally calls clear_fault's logic, so factor that logic into _clear_fault_locked (no lock acquisition) called by both the public clear_fault (which acquires the lock) and apply_fault (which acquires the lock once and calls the internal version), avoiding a self-deadlock on the non-reentrant lock. Route _ttl_watcher's clear through the same locked path so it can't race a fresh apply_fault. Implements build-plan step A1.3 (fix on an already-completed service). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
…ention FaultRequest.magnitude means something different per FaultKind, and for ERROR_RATE specifically it silently overloads two conventions: <= 1.0 is a fraction, > 1.0 is a percentage. A caller who means "1%" and passes 1.0 gets 100% instead, since 1.0 lands on the fraction side of that boundary. This is intentional, existing behavior (test_faults.py::test_fault_error_rate already asserts 1.0 and 100.0 both mean 100%) -- not changed here, only documented: an inline comment at the branch, and an expanded field docstring spelling out the per-kind units and the footgun. Implements build-plan step A1.3 (documentation follow-up on an already-completed service). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
This was referenced Sep 11, 2026
Harikeshav-R
added a commit
that referenced
this pull request
Sep 11, 2026
Merged
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build-Plan Step Implemented
Checkpoint Execution & Output
Two pre-existing, unrelated Windows-local-only items excluded (constructing a
real
httpx.AsyncClient()with the default transport crashes on this devmachine -- see PR #12 for root-cause detail); not present on CI
(
ubuntu-latest).ADRs Relied On
worker fault-application fix don't change seeding; the concurrency
regression test confirms determinism-relevant state stays consistent.
Mock Registry Disclosures
Documentation Updates
services/_common/faults.py: expandedFaultRequest.magnitude's docstringand added an inline comment documenting the ERROR_RATE fraction/percentage
convention (no behavior change).
Definition of Done
make checkgreen (see checkpoint output; two pre-existing Windows-local-only items excluded, unrelated)# MOCKED:marker without a row in docs/MOCKS.mdContext
Three related findings from a
/code-reviewpass against PR #6, all in thefault-injection subsystem, bundled into one PR as three commits:
worker_loopnever calledfault_manager.pre_request_hook(), so LATENCY/ERROR_RATE/CPU_SPIN faultswere no-ops against job processing (only STALL worked, via a separate
call). Fixed by calling it once per polling iteration, factored into a
testable
_worker_iterationhelper.faults.py--apply_fault/clear_faultmutatedshared state across
awaitpoints with no synchronization, and theindependent
_ttl_watcherbackground task could race a freshapply_fault. Fixed with anasyncio.Lockguarding the whole statetransition (the first use of this primitive in the repo).
ERROR_RATE'sfraction-vs-percentage overload was intentional (an existing test already
asserts
1.0means 100%, not 1%) but entirely undocumented. Documented,not changed.
Also noted but explicitly out of scope:
apply_fault'sPOOL_EXHAUSTIONbranch only catches
PoolTimeout, leaking held connections with no TTLwatcher on any other exception from
pool.getconn. Filing as a follow-uprather than fixing here, since it wasn't one of the original review findings.
Base branch is
feat/policies-and-loadgen(PR #6, still open), sinceservices/only exists on that branch's stack.🤖 Generated with Claude Code
https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh