Skip to content

fix(services): apply fault injection to worker jobs, guard concurrent fault transitions, clarify magnitude convention - #13

Merged
Harikeshav-R merged 3 commits into
feat/policies-and-loadgenfrom
fix/worker-fault-lock-magnitude-docs
Sep 11, 2026
Merged

Harikeshav-R merged 3 commits into
feat/policies-and-loadgenfrom
fix/worker-fault-lock-magnitude-docs

Conversation

@rsanghvi21

Copy link
Copy Markdown
Collaborator

Build-Plan Step Implemented

  • Step: A1.2/A1.3 (fix + documentation follow-up on already-completed services).

Checkpoint Execution & Output

$ uv run pytest tests/unit/services/test_faults.py tests/unit/services/test_worker.py -v
30 passed in 3.57s

$ uv run pytest tests/unit/services --ignore=tests/unit/services/test_edge_gateway.py -k "not test_run_loadgen_with_owned_client_and_zero_rps" -v
84 passed, 1 deselected

$ uv run pytest tests/unit --ignore=tests/unit/services/test_edge_gateway.py -k "not test_run_loadgen_with_owned_client_and_zero_rps" --cov=understudy --cov-branch --cov-fail-under=100
193 passed, 1 deselected, 100.00% coverage

$ uv run ruff check . && uv run ruff format --check . && uv run mypy --strict src tests && uv run lint-imports && uv run python scripts/check_mocks.py
All green.

Two pre-existing, unrelated Windows-local-only items excluded (constructing a
real httpx.AsyncClient() with the default transport crashes on this dev
machine -- see PR #12 for root-cause detail); not present on CI
(ubuntu-latest).

ADRs Relied On

  • ADR-015 (fault injection determinism/seeding) -- the new asyncio.Lock and
    worker fault-application fix don't change seeding; the concurrency
    regression test confirms determinism-relevant state stays consistent.

Mock Registry Disclosures

  • None.

Documentation Updates

  • services/_common/faults.py: expanded FaultRequest.magnitude's docstring
    and added an inline comment documenting the ERROR_RATE fraction/percentage
    convention (no behavior change).

Definition of Done

  • Implementation complete for the named build-plan step
  • Unit tests written; 100% line and branch coverage of the added lines
  • make check green (see checkpoint output; two pre-existing Windows-local-only items excluded, unrelated)
  • Checkpoint command run and output matches, OR marked unverified with a reason
  • Docs updated where this change made them untrue
  • No new # MOCKED: marker without a row in docs/MOCKS.md
  • Conventional commit(s), conventional branch, PR opened, never pushed to main

Context

Three related findings from a /code-review pass against PR #6, all in the
fault-injection subsystem, bundled into one PR as three commits:

  1. Worker fault-injection gap -- worker_loop never called
    fault_manager.pre_request_hook(), so LATENCY/ERROR_RATE/CPU_SPIN faults
    were no-ops against job processing (only STALL worked, via a separate
    call). Fixed by calling it once per polling iteration, factored into a
    testable _worker_iteration helper.
  2. Race condition in faults.py -- apply_fault/clear_fault mutated
    shared state across await points with no synchronization, and the
    independent _ttl_watcher background task could race a fresh
    apply_fault. Fixed with an asyncio.Lock guarding the whole state
    transition (the first use of this primitive in the repo).
  3. Magnitude ambiguity at exactly 1.0 -- ERROR_RATE's
    fraction-vs-percentage overload was intentional (an existing test already
    asserts 1.0 means 100%, not 1%) but entirely undocumented. Documented,
    not changed.

Also noted but explicitly out of scope: apply_fault's POOL_EXHAUSTION
branch only catches PoolTimeout, leaking held connections with no TTL
watcher on any other exception from pool.getconn. Filing as a follow-up
rather than fixing here, since it wasn't one of the original review findings.

Base branch is feat/policies-and-loadgen (PR #6, still open), since
services/ only exists on that branch's stack.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh

rsanghvi21 and others added 3 commits September 11, 2026 00:00
worker_loop only called fault_manager.wait_if_stalled, so LATENCY,
ERROR_RATE, and CPU_SPIN faults applied via POST /admin/fault were
no-ops against job processing -- only STALL worked. Every other
service's HTTP middleware calls pre_request_hook once per request;
the worker has no HTTP request in its polling loop, so call it
directly once per iteration instead.

Factor the loop body into _worker_iteration so it can be exercised
directly under test without driving a real polling loop, and so
pre_request_hook's ERROR_RATE branch (which raises HTTPException --
meaningful in an HTTP handler, not here) can be handled the same way
a real job-processing failure already is: logged and the job skipped
for that iteration, rather than crashing the polling task.

Implements build-plan step A1.2 (fix on an already-completed
service).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
FaultManager.apply_fault/clear_fault mutate shared state
(active_fault, expires_at, _timer_task, _leaked_buffers,
_held_connections) across multiple await points with no
synchronization. Two admin apply_fault/clear_fault calls racing each
other, or the independent _ttl_watcher background task clearing a
fault out from under a fresh apply_fault, could leave the manager in
a torn state -- e.g. a TTL expiry cancelling the wrong timer, or a
connection appended to _held_connections during clear_fault's
iteration over the same list.

Add an asyncio.Lock guarding the whole state transition. apply_fault
internally calls clear_fault's logic, so factor that logic into
_clear_fault_locked (no lock acquisition) called by both the public
clear_fault (which acquires the lock) and apply_fault (which
acquires the lock once and calls the internal version), avoiding a
self-deadlock on the non-reentrant lock. Route _ttl_watcher's clear
through the same locked path so it can't race a fresh apply_fault.

Implements build-plan step A1.3 (fix on an already-completed
service).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
…ention

FaultRequest.magnitude means something different per FaultKind, and
for ERROR_RATE specifically it silently overloads two conventions:
<= 1.0 is a fraction, > 1.0 is a percentage. A caller who means "1%"
and passes 1.0 gets 100% instead, since 1.0 lands on the fraction
side of that boundary. This is intentional, existing behavior
(test_faults.py::test_fault_error_rate already asserts 1.0 and 100.0
both mean 100%) -- not changed here, only documented: an inline
comment at the branch, and an expanded field docstring spelling out
the per-kind units and the footgun.

Implements build-plan step A1.3 (documentation follow-up on an
already-completed service).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
@Harikeshav-R
Harikeshav-R merged commit 86c4abe into feat/policies-and-loadgen Sep 11, 2026
1 check passed
Harikeshav-R added a commit that referenced this pull request Sep 11, 2026
@Harikeshav-R
Harikeshav-R deleted the fix/worker-fault-lock-magnitude-docs branch September 11, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants