Repository navigation
feat(proof): pin real proof-eval digest and operator topic path - #221
Mathis (echobt) wants to merge 2 commits into
Conversation
Ship a harvest-compatible proof-eval binary at /usr/bin/proof-eval with PROOF_METRICS=/PROOF_EVAL_OK markers, baked proxy Qwen/Qwen3.8-0.6B, and 12.5 Gbit/s / no-IB / no-NVLink / no-NCCL-fast-fabric enforcement. The scoring image (CUDA + torch) is what the control plane will pin; the contract-only digest is not a pin. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Stack on the 2000/8000 + WTA/discovery control plane. Pin the scoring image published by run 33892650063 (sha256:78b614a1…, baked proxy Qwen/Qwen3.8-0.6B). Empty digest still 503s until a real sha256 is present; this one was pulled and proved after push. Operator script covers holdout → baseline-on-image → xtask proof-topic → admin POST. Does not touch challenges.toml or the topic schema. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Greptile SummaryThis change adds the Proof evaluator and supporting operational path, but live scoring cannot reliably run yet. Required holdout data is not delivered to rented pods, long-context evaluation is shortened, bandwidth enforcement can be falsely reported, the baseline command does not transport its files, and a non-public image can still be published for use. Confidence Score: 0/5Not safe to merge: the reproduced failures prevent reliable live scoring and can invalidate benchmark measurements. Five independently reproduced blocking failures affect holdout delivery, evaluation context, bandwidth enforcement, baseline generation, and evaluator image availability. One of them permits a reported resource control to be bypassed. Files Needing Attention: eval/src/proof_eval/harness.py, eval/src/proof_eval/fabric.py, deploy/scripts/proof-operator-path.sh, and .github/workflows/publish-proof-eval-image.yml
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| store = Path(os.environ.get("PROOF_HOLDOUT_STORE", "/opt/proof-eval/holdout")) | ||
| path = store / digest | ||
| if not path.is_file(): | ||
| raise ContractError( | ||
| f"holdout shard {digest[:12]}… is not in PROOF_HOLDOUT_STORE; refuse scoring" | ||
| ) |
There was a problem hiding this comment.
The scorer requires each holdout payload at PROOF_HOLDOUT_STORE/<content_sha256>, but the live pod path supplies only request.json after the pod has been rented. No shard staging or holdout-store mount is provided, so live scoring fails when this lookup cannot find the required private shard. Deliver the sealed shards to the pod before scoring, or verify that delivery is available before renting it.
Knowledge Base Used: Deployment, runtime operations, and image updates
Artifacts
- Runs the scorer's actual content-addressed holdout lookup once with no shard and once with the expected shard file, showing that delivery is required.
- Executed the validation script against an empty configured holdout store; the scorer refused the shard and exited 2, confirming missing delivery prevents scoring.
- Executed the same validation script with the expected content-addressed shard present; the scorer read it and exited 0, proving the required delivery condition.
- Ran focused Proof-harvest and harvest-pod tests; they confirm request metadata staging but do not provide a shard-delivery operation, supporting the missing transport conclusion.
| for dev in _data_ifaces(): | ||
| subprocess.run( # noqa: S603 | ||
| [tc, "qdisc", "replace", "dev", dev, "root", "tbf", | ||
| "rate", f"{kbit}kbit", "burst", "256kb", "latency", "50ms"], | ||
| check=False, | ||
| capture_output=True, | ||
| ) | ||
| applied[f"tc:{dev}"] = f"{kbit}kbit" |
There was a problem hiding this comment.
The cap installation ignores a failed tc qdisc replace call and still records the interface as limited. If the scoring pod lacks NET_ADMIN, the claimed 12.5-Gbit/s limit is not installed while the reported evaluation state says it is, allowing unrestricted network bandwidth to affect constrained-topic throughput results. Fail scoring when the qdisc cannot be installed and only report a cap after successful installation.
How this was verified: A no-capability execution produced an operation-permitted error while the cap function still returned a successful recorded limit.
Artifacts
- Runs a failing tc qdisc operation and directly exercises the changed bandwidth-cap function with the same failure, ending with the reproducible validation method.
- The executed tc qdisc command ran with zero effective capabilities and returned Operation not permitted, proving the rate limiter cannot be installed.
- The executed `_cap_bandwidth(12.5)` call returned normally and recorded eth0 as capped after the same failing tc path, proving the enforcement record is false.
| t0 = time.perf_counter() | ||
| with torch.no_grad(): | ||
| for split, text in texts: | ||
| enc = tok(text, return_tensors="pt", truncation=True, max_length=1024) |
There was a problem hiding this comment.
Every holdout record is tokenized with a fixed 1,024-token limit, including the longctx stratum. This reduces the required 8,192–32,768-token sequences to short prefixes, so the long-context metric cannot detect submissions that only work at short context lengths. Score longctx records at their required context length and reject unsupported lengths instead of silently truncating them.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Artifacts
- The authored script installs fake runtime components, creates 9,000-token shards for every scored split, and invokes the real harness measure function, demonstrating the token-selection behavior.
- The generated validation script was compiled with Python before execution and exited successfully, showing it was executable.
- The real harness path was run with 9,000-token inputs and shows every stratum, including longctx declared at 8,192 tokens, encoded to 1,024 tokens, confirming the defect.
- The Rust contract test passed while enforcing that an undersized longctx record is refused, establishing the required long-context lower bound.
- The existing proof evaluator tests completed successfully with 8 passing tests, showing the current suite does not catch the exercised truncation defect.
| # The image enforces 12.5 Gbit/s / no IB / no NVLink / no NCCL fast path | ||
| # before it will emit numbers. | ||
| if [ -n "${digest}" ]; then | ||
| echo "docker run --rm --entrypoint /usr/bin/proof-eval ${IMAGE}@${digest} baseline --request /tmp/proof_eval/request.json --out /tmp/proof_eval/baseline.json" |
There was a problem hiding this comment.
The generated docker run --rm command reads and writes paths inside the container but supplies neither a request-file mount nor an output mount. The evaluator cannot read /tmp/proof_eval/request.json, and any manually created unmounted output is removed with the container, preventing operators from producing the persistent baseline needed to open a topic. Mount or explicitly copy both the request and retained output.
Knowledge Base Used: Deployment, runtime operations, and image updates
Artifacts
- A non-source-changing validation script builds a deterministic local evaluator fixture, executes the generated command shape, and compares it with a mounted persistent-output invocation; it reproduces the missing transport defect.
- The executed generated-command equivalent exits 66 with the request missing, and Docker inspection reports automatic removal with no mounts; it confirms the helper command cannot consume its declared request.
- The same evaluator invocation with a bind mount exits 0 and writes the baseline into the host work directory; it demonstrates the required delivery and persistence mechanism.
| "https://api.github.com/orgs/${OWNER}/packages/container/${enc}/visibility" \ | ||
| -d '{"visibility":"public"}' || true) | ||
| echo "pkg=proof-eval http=${code} body=$(head -c 200 /tmp/vis.json || true)" | ||
| - name: report the digest to pin | ||
| env: | ||
| DIGEST: ${{ steps.push.outputs.digest }} | ||
| run: | | ||
| set -eu | ||
| case "${DIGEST}" in |
There was a problem hiding this comment.
The workflow ignores a failed GHCR package-visibility update, then independently accepts and uploads the image digest. If publicization fails, the published pin can remain private and anonymous scoring workers cannot pull it, causing scoring to fail after the digest is pinned. Require a successful visibility update and verify that the exact digest is anonymously pullable before publishing it.
Knowledge Base Used: Deployment automation and infrastructure
Artifacts
- Executable shell reproduction of the relevant workflow paths using a controlled HTTP 500 response and a valid digest, showing the failure is ignored before the digest is emitted.
- Executed visibility-phase reproduction returned exit code 0 after a simulated GHCR HTTP 500, showing the workflow proceeds despite failed publicization.
- Executed digest-phase reproduction returned exit code 0 and wrote the sha256 value as the upload payload, showing no visibility-success dependency.
Summary
Rebased onto
mainafter #220 merged (7bb56c3b). This PR does not touchchallenges.tomlor the topic schema.Pins a real scoring image digest from
publish-proof-eval-imagerun 33892650063 (packaging commit51f937c7). That digest was pulled after push; harvest-PATH + selftest + baked proxy + 12.5 Gbit/s fabric enforcement proved on those bytes. Contract-only image is not pinned.Empty digest remains fail-closed 503 in code (
can_rent/EvalImageUnpinned). This pin is a real sha256, not invented.Operator path:
deploy/scripts/proof-operator-path.sh(prints holdout → baseline-on-pinned-image →xtask proof-topic→ adminPOST /v1/admin/proof/topics).can_scorestill needs LIUM harvest wired + ≥1 open signed topic + verified holdout + sealed baseline on the host.Greptile
Every PR is reviewed by Greptile before merge. Config:
.greptile/.@greptileai reviewTest plan
cargo test -p proof-task -p proof-http -p proof-eval -p trustrootcargo fmt --all -- --checkcommitted_pin_is_proof_with_a_real_eval_digestempty_digest_and_unwired_harvest_are_503(fail-closed path still present)cargo test --workspacenot run (pin + eval packaging + operator script)Risk
Once merged, Proof submits stop 503-ing on empty digest. Remaining 503s: unwired harvest, no open topic, unsealed baseline. 8000 bps still burns until those land on the droplet. No ceremony/key rotation in this PR.
Naming
I did not rename
BASE_*environment variables, deployed host paths(
/opt/base,/run/base, …), GHCRbaseintelligence/basepackage names, orbase-*-v1cryptographic domain tags.