Skip to content

fix(sync): short messages and in-run retries when the Quant API stalls - #40

Merged
stooit merged 6 commits into
developfrom
fix/http-friendly-errors-retry
Nov 15, 2026
Merged

stooit merged 6 commits into
developfrom
fix/http-friendly-errors-retry

Conversation

@stooit

@stooit stooit commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

When the Quant API stalls, content pushes fail with raw transport text such as cURL error 28: Operation timed out after 30001 milliseconds with 0 bytes received. The dashboard queue card, the push progress and the diagnostics show it as-is. Each job gets one HTTP attempt per run; the queue retry (3 attempts, 300 s x attempts backoff) only happens when WP-Cron runs.

Changes

Short messages, raw text kept apart

  • TransportError maps WP_Error transport failures to a short reason: "Quant did not answer in time." for timeouts and other http_request_failed; DNS, TLS and connection failures have their own text. 502/503/504 → "Quant is busy or unavailable (HTTP n).", other 5xx → "Quant had a server error (HTTP n).". 4xx answers keep the API's own message.
  • The next step is added only when it is true. JobProcessor adds "The plugin will try again." when the job has attempts left (attempts + 1 < MAX_ATTEMPTS) and background processing runs (Scheduler::backgroundProcessingEnabled(), the DISABLE_WP_CRON check the dashboard warning already uses). Otherwise it adds "Select Retry failed on the dashboard." (the progress card button now also reads "Retry failed"). The connection test (ping) adds "Test the connection again in a minute.". Permanent (4xx) errors get no next step. ApiResponse, QuantApiException and FileSyncResult carry a transient flag; SyncJob carries attempts.
  • WordPressHttpClient::send returns error (short), detail (raw code: message) and transport (true only for a matched http_request_failed). QuantContentApi::parse applies the 5xx mapping and keeps the API text or an excerpt of the error page as the detail. ApiResponse has a detail field.
  • Sync handlers throw QuantApiException (message + detail). JobProcessor passes both to markFailed.
  • Queue table: new error_detail TEXT NULL column, DB_VERSION 2.0.6, added by the existing Activator::maybeUpgrade / dbDelta path (same as the 2.0.3 dedupe_key change). Cleared on completion, manual retry and stale-failure.
  • The queue card and the progress card show only the message. The diagnostics report download, the diagnostics API section, wp quant queue errors and wp quant ping/status keep the detail.

Retry inside one run

  • RetryPolicy: at most 2 retries, after 1 s and 3 s plus up to 500 ms jitter, only for a transport failure (an http_request_failed that matches timeout, DNS, TLS or connection) or 502/503/504. Other WP_Errors ("The request to Quant failed.": not executed, invalid URL, too many redirects) and local refusals are not retried.
  • A retry starts only when the call so far + the wait + another attempt as long as the last one fit in 45 s: fast failures retry twice; a request that used the full 30 s timeout is not retried in the run.
  • One instance per request (passed explicitly in Plugin.php) with request-wide state: a circuit breaker (once one call used all its retries and still failed, later calls get none until a call succeeds) and one shared 45 s retry budget. Against a constant 503, N uploads send N + 2 requests.
  • Background processing counts as on when WP-Cron is enabled or the scheduled run recorded quant_last_queue_run in the last 15 minutes (system cron with DISABLE_WP_CRON). Diagnostics report it.
  • Activator records quant_db_version only after error_detail exists.
  • Used only for the idempotent calls: publish (POST /v1 JSON) and file upload (upload, uploadContents, keyed by Quant-File-Url). Unpublish, purge, redirects, search and ping are sent once.

Timeouts

  • JSON publish keeps the configured timeout (default 30 s). A JSON body of 512 KB or more gets the 120 s upload minimum. 512 KB is above a typical page (50–300 KB) and takes about 16 s at a slow 256 kbit/s uplink, which leaves about half of the 30 s for Quant to process the page.
  • Connect timeout: WordPress does not pass connect_timeout to Requests, so the Requests default of 10 s already applies (cURL CURLOPT_CONNECTTIMEOUT and fsockopen). No change needed; documented in code and readme.
  • Readme documents QUANT_HTTP_REQUEST_TIMEOUT, adds an FAQ entry, and a 2.0.6 changelog entry. No version bump.

Tests

  • PHPUnit 468 tests (circuit breaker bounded call count, shared budget, timeout-aware retry, neutral fallback, last-run detection, schema version check; (includes one case per next-step rule: attempts left, last attempt, WP-Cron off, busy text both ways, permanent error, ping): new TransportErrorTest, RetryPolicyTest; new cases in WordPressHttpClientTest, QuantContentApiTest, DatabaseQueueTest, JobProcessorTest, PublisherTest, FileHandlerTest.
  • PHPStan clean, PHPMD clean.
  • JS: queueErrors.test.js (card shows only the message; report keeps the detail); ESLint and stylelint clean; build reproduces.
  • e2e: full suite 50/50. schema-upgrade.spec.js upgrades a 2.0.5 queue table with rows to 2.0.6 and checks the rows and the card. A third new case: on the last attempt the card shows "Quant is busy or unavailable (HTTP 503). Select Retry failed on the dashboard." and no "will try again". New transport-errors.spec.js: a 503 is retried in the same run (2 publish requests, no failure); a stalled mock (8 s, timeout 5 s) produces 3 attempts, the short message in the card and REST, and cURL error 28 … (3 attempts) only in the diagnostics download. The mock API can delay a simulated failure (delayMs). content-sync.spec.js now expects the short server-error message.
  • An existing 2.0.3-schema e2e database upgraded in place to 2.0.6 with the new column.

When the Quant API did not answer, a push failed with raw transport text
such as "cURL error 28: Operation timed out after 30001 milliseconds with
0 bytes received", shown as-is in the dashboard queue card and the push
progress. Each job got one attempt per run, and the queue only retried it
when WP-Cron ran.

- WordPressHttpClient maps WP_Error failures (timeout, DNS, TLS,
  connection) to a short message and keeps the raw text as a detail.
  QuantContentApi does the same for 5xx answers.
- Publish and upload (idempotent, keyed by URL) go through a RetryPolicy:
  at most 2 retries after about 1 s and 3 s plus jitter, only for a
  transport failure or 502/503/504, and no retry starts after 45 s.
  Delays, sleep and clock are injectable.
- The queue stores the message and the detail separately (new
  error_detail column, DB_VERSION 2.0.6, added by dbDelta on upgrade).
  The cards show only the message; the diagnostics report, the
  diagnostics API section and wp quant queue errors keep the detail.
- A JSON publish body of 512 KB or more gets the 120 s upload minimum.
- Readme: QUANT_HTTP_REQUEST_TIMEOUT, an FAQ entry, and the 2.0.6
  changelog.
…card

The mock API can delay a simulated failure (delayMs) so the plugin's request times out. The failed-push spec now expects the short server-error message instead of the API text.
The short messages promised "The plugin will try again." even when the
job had used all its attempts or WP-Cron was disabled, and on the
connection test.

- TransportError reasons no longer say what happens next. ApiResponse,
  QuantApiException and FileSyncResult carry a transient flag (no answer
  or 5xx).
- JobProcessor adds "The plugin will try again." only when the job has
  attempts left (attempts + 1 < MAX_ATTEMPTS) and background processing
  runs (Scheduler::backgroundProcessingEnabled, the DISABLE_WP_CRON check
  the dashboard warning already uses). Otherwise it adds "Retry it from
  the queue.". Permanent errors get no next step.
- SyncJob carries attempts from the queue row.
- The connection test (ping) adds "Test the connection again in a
  minute.".
The manual next step now reads "Select Retry failed on the dashboard.",
the label of the dashboard button. The push progress card said "Retry
Failed"; it now says "Retry failed" too, so the two controls match the
message.
Review fixes for the in-run retry and the short messages.

- RetryPolicy keeps request-wide state (one instance per request, now
  passed explicitly in Plugin.php): a circuit breaker stops retries for
  later calls once one call used all its retries and still failed (a
  success closes it again), and all calls share one 45 s retry budget.
  A job with many uploads against a failing API sends one request per
  upload after the first, not three.
- A retry starts only when the call so far, the wait and another attempt
  as long as the last one fit in 45 s. Fast failures still retry twice;
  a request that used the full 30 s timeout is not retried in the run.
- Only an http_request_failed error that matches a known transport
  failure is marked as transport (retryable). Other WP_Errors (request
  not executed, invalid URL, too many redirects) get "The request to
  Quant failed." and no retry.
- The scheduled queue run records quant_last_queue_run. Background
  processing counts as on when WP-Cron is enabled or a run happened in
  the last 15 minutes (DISABLE_WP_CRON with a system cron). Diagnostics
  report both, and the last run time.
- Activator records quant_db_version only after error_detail exists, so
  a failed dbDelta runs again on the next request.
- Readme and changelog say exactly when a push is retried.
@stooit
stooit merged commit dd58d52 into develop Nov 15, 2026
12 checks passed
@stooit stooit mentioned this pull request Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant