cron-init Jobs can accumulate when dependencies stall: missing deadlines, pagination guards and failure exit codes #280

Problem

cron-init can keep earlier Jobs active while creating another Job every minute. The operator generates no concurrency restriction or Job deadline, while the executor has unbounded HTTP waits and pagination paths. Several dependency failures are also reported as a successful process exit.

This issue separates the reproducible code/configuration defects from the initial cause of the reported production incident, which remains unproven.

Affected version and source verification

Reported image: apache/openserverless-operator:0.1.0-incubating.2510012114.

The public tag resolves to c562668a35c8f383c8ec6d0065da29baa39f0bcd. The five files below, plus cron.sh and pyproject.toml, were extracted from both published amd64 and arm64 images and matched byte-for-byte against that tag: 14/14 comparisons.

Registry index checked on 2026-10-23: sha256:d41201ded2a544c7affc8cbe8084d0f5ac10d6b8031897c96a383d6fd4d55847.

The deployed Pods' immutable imageIDs were not supplied, so tag reuse or a different cached image cannot be completely excluded.

The same relevant behavior was reproduced in the current public main, pinned to fb481d243d902e4f995a14bd6bb1137c79045b6c.

Reported incident observations

On 2026-10-23:

• Approximately 9,500 active cron-init Jobs, more than 9,000 Pending Pods and approximately 211 Running Pods.
• A one-minute schedule, effective concurrencyPolicy: Allow, restartPolicy: OnFailure, no activeDeadlineSeconds, and no container resource requests/limits.
• Scheduler events included Too many pods.
• Some running Pods were eight days old; their last messages referred to a CouchDB nuvolaris_whisks/_find query.
• CouchDB logged timeouts and HTTP 500 responses during the overload.
• After suspending the CronJob and cleaning up accumulated workloads, the query responded in approximately 27 ms and services became Ready again.

These are reported incident observations, not measurements collected by accessing the affected cluster during this investigation.

Original code paths

1. nuvolaris/couchdb_util.py, lines 152–170: find_doc() uses session.post() without a timeout. A non-200 response is logged and converted to None.
2. nuvolaris/actionexecutor.py, lines 94–120: find_docs() has no page bound or bookmark progress guard. A non-empty page with a blank/repeated bookmark can repeat indefinitely; a missing bookmark raises KeyError. A failed later page returns the documents collected so far.
3. nuvolaris/actionexecutor.py, lines 148–196: OpenWhisk POST and PUT also lack timeouts. blocking=false does not bound the HTTP client's wait.
4. nuvolaris/actionexecutor.py, lines 317–396: failed invocation/update results are not propagated to the process exit status. Failed readiness also returns normally.
5. nuvolaris/cronjob.py, lines 24–58, nuvolaris/templates/cron-init.yaml, and deploy/scheduler/cron.yaml: the actual generated CronJob has the unbounded defaults described above.

The actual cronjob.create(), template helpers and Kustomize 4.5.7 were exercised locally, replacing only the Kubernetes apply boundary with object capture. Both the original and current source generate those same missing controls. The generator correctly replaces the stale example image and empty SCHEDULER_CONFIG in the base manifest; those placeholders are not being assumed to be the generated production configuration.

Local fault reproduction

The actual original/current executor functions were run against loopback-only synthetic HTTP servers, with Kubernetes imports stubbed. HTTP/cron dependencies were extracted from the original image: requests 2.32.4, urllib3 2.3.0, croniter 1.4.1, python-dateutil 2.9.0.post0. The host Python was 3.12.3; this was not a Kubernetes controller test or execution of the full image.

The mock CouchDB returns HTTP 200 for readiness, then supplies the selected fault on POST /nuvolaris_whisks/_find (or the corresponding rebranded path). Controller faults are supplied after valid synthetic action/subject pages.

The watchdog termination is an observation limit, not an application-generated timeout or exit code. No claim is made that a one-second test reproduces eight days of production behavior.

Diagnosis boundaries

The unbounded executor plus overlapping Jobs is a demonstrated accumulation mechanism. At one Job/minute, eight days permits 11,520 executions before accounting for completed, failed or missed runs; approximately 9,500 is consistent with that order of magnitude.

However:

• Missing timeouts alone do not identify why the first request stalled.
• A completed HTTP 500 by itself does not explain Jobs remaining active for days: the tested process exits.
• A missing bookmark also exits; the non-advancing-bookmark case requires non-empty documents and has not been established from the production evidence.
• The 27 ms response after cleanup supports overload feedback but does not establish or exclude an initial network, storage, CouchDB or query-planning problem.
• Missing indexes/full scans have not been demonstrated: no production _explain, index inventory or pre-overload execution statistics were supplied.

History limits only clean up finished Jobs. Default Job backoff does not terminate a live blocked process, and an incorrectly successful exit does not trigger failure retry. A Job execution deadline must be set at jobTemplate.spec.activeDeadlineSeconds; startingDeadlineSeconds is not a runtime bound.

Existing later fix

PR #144 / 14e7a309 isolates malformed cron annotations and bounds their size. It is already in the main revision checked above, but does not add HTTP timeouts, pagination progress checks, Job deadlines or a concurrency policy.

Operational patches on the affected cluster have not been treated as upstream fixes.

Expected correction and regression coverage

Before implementing a production fix, the intended behavior should include:

• Bounded HTTP connect/read waits and explicit scan failures, including a 500 on a later page.
• Pagination progress validation and a finite work bound.
• A nonzero process exit for dependency failures, while preserving per-action isolation.
• A concurrency policy and Job deadline that prevent unlimited overlap, plus explicit resource settings.
• No blind retries of action-invocation POSTs: acceptance followed by a lost response can otherwise cause duplicate invocations.

Regression coverage should include every fault above, normal multi-page scans, healthy invocation/autoexec updates, generated manifests, and real Kubernetes deadline/backoff behavior with Pending Pods.

The concurrency/deadline choice also needs scheduling semantics reviewed: Forbid can skip ticks when a scan takes longer than the interval, and the existing executor has no durable catch-up cursor or exactly-once guarantee.

No change has been deployed as part of this investigation, and no fix PR accompanies this report.