Repository navigation
test(indexer): retain failed whole-token accuracy experiment for #255 - #259
Conversation
…#255 Evidence-only tree; candidate not promoted. Development MRR gate failed; heldout not evaluated. Exact reviewed local tree from 0c197de3bb3e39df4eef01be74dc238bbcc0ed99; connector transport preserves content, not local commit metadata. Signed-off-by: Codex <codex@openai.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Dong1017
left a comment
There was a problem hiding this comment.
Scoped LGTM for this evidence-only delivery at bdfee4e against main 1147c5b. No production source changes or candidate promotion.
Independent verification reproduced the retained development arithmetic: Hit@5 4/6 to 5/6, MRR@5 7/12 to 13/24; the frozen nondecreasing-MRR gate therefore FAILS. All 38 emitted contexts per arm were checked for complete required resources and exact bytes/hash/revision; saved default outputs match. Publication audit verified 33 hashes, 16 compressed outputs and the exact retained candidate patch/bundle. Provenance wording now distinguishes source-review access from heldout exposure: heldout was not exposed to the candidate implementer and was not evaluated. No tuning or new timing campaign occurred.
Affected evidence tests passed (17); current Ready CI https://github.com/InferMatrixLab/InferRSI/actions/runs/40987495415 passes both full-check and installed-package Python versions. The exact-head native review completed with no findings and no inline threads remain. The earlier draft-trigger run was superseded by this successful Ready run.
This accepts honest, reproducible evidence delivery only. Accuracy improvement was not established, runtime remains unchanged, heldout remains unevaluated, and #255 stays open. No model/hardware efficacy claim, formal APPROVE or tracker closure.
Scope and decision
Evidence-only follow-up to #255. No production runtime or corpus changes; the single experimental weighted-token-v2 candidate is retained only as a patch/Git bundle for reproducibility. Do not promote the candidate or close #255.
The prospectively frozen development gate failed:
The Hit gain does not override the MRR regression. Evaluation stopped without tuning. The six heldout cases were not exposed to the candidate implementer and were not evaluated; case-author/source-review access for preparing labels is acknowledged. Their contents are not included here.
Artifacts
1147c5bb41724826b64ad13cebc8a4dce5cc278b; rejected candidate:55a458328195335586eb6c6c1301ba34a8218e63.Verification and limits
ruff check src tests toolspassed.