Skip to content

test(indexer): retain failed whole-token accuracy experiment for #255 - #259

Merged
Dong1017 merged 1 commit into
mainfrom
issue255-token-v2-evidence
Oct 30, 2026
Merged

Dong1017 merged 1 commit into
mainfrom
issue255-token-v2-evidence

Conversation

@Dong1017

Copy link
Copy Markdown
Collaborator

Scope and decision

Evidence-only follow-up to #255. No production runtime or corpus changes; the single experimental weighted-token-v2 candidate is retained only as a patch/Git bundle for reproducibility. Do not promote the candidate or close #255.

The prospectively frozen development gate failed:

Partial-label metric Baseline Candidate
Hit@5 4/6 5/6
MRR@5 0.583333 0.541667
Complete relevant deliveries 4 5

The Hit gain does not override the MRR regression. Evaluation stopped without tuning. The six heldout cases were not exposed to the candidate implementer and were not evaluated; case-author/source-review access for preparing labels is acknowledged. Their contents are not included here.

Artifacts

  • Readable per-case ranks, output/context bytes, decision and limits: report.
  • Accepted protocol, development inputs, complete lossless gzip outputs, raw hashes, default-output parity captures and candidate patch/bundle.
  • Evidence inventory adds 33 hashes (232 total, 11 bundles). Original campaigns remain unchanged.
  • Historical experiment baseline: 1147c5bb41724826b64ad13cebc8a4dce5cc278b; rejected candidate: 55a458328195335586eb6c6c1301ba34a8218e63.

Verification and limits

  • Candidate focused compatibility/synthetic tests: 41 passed; six baseline/candidate default serialized outputs identical.
  • Evidence-only tree affected inventory tests: 17 passed; ruff check src tests tools passed.
  • Independent artifact audit verified integrity, gzip outputs, patch/bundle, and all 38 emitted contexts per arm against complete required resources, exact bytes/hash/revision.
  • Current hosted CI is pending. Local tests/artifact review do not imply model efficacy, speed nonregression, or Issue acceptance.
  • No new timing campaign, model/service calls, hardware experiments or spending. No merge, formal approval or Issue closure requested by this PR.

…#255

Evidence-only tree; candidate not promoted. Development MRR gate failed; heldout not evaluated. Exact reviewed local tree from 0c197de3bb3e39df4eef01be74dc238bbcc0ed99; connector transport preserves content, not local commit metadata.

Signed-off-by: Codex <codex@openai.com>
@Dong1017
Dong1017 marked this pull request as ready for review October 30, 2026 01:43
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T01:45:55.344706Z bdfee4e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped LGTM for this evidence-only delivery at bdfee4e against main 1147c5b. No production source changes or candidate promotion.

Independent verification reproduced the retained development arithmetic: Hit@5 4/6 to 5/6, MRR@5 7/12 to 13/24; the frozen nondecreasing-MRR gate therefore FAILS. All 38 emitted contexts per arm were checked for complete required resources and exact bytes/hash/revision; saved default outputs match. Publication audit verified 33 hashes, 16 compressed outputs and the exact retained candidate patch/bundle. Provenance wording now distinguishes source-review access from heldout exposure: heldout was not exposed to the candidate implementer and was not evaluated. No tuning or new timing campaign occurred.

Affected evidence tests passed (17); current Ready CI https://github.com/InferMatrixLab/InferRSI/actions/runs/40987495415 passes both full-check and installed-package Python versions. The exact-head native review completed with no findings and no inline threads remain. The earlier draft-trigger run was superseded by this successful Ready run.

This accepts honest, reproducible evidence delivery only. Accuracy improvement was not established, runtime remains unchanged, heldout remains unevaluated, and #255 stays open. No model/hardware efficacy claim, formal APPROVE or tracker closure.

@Dong1017
Dong1017 merged commit dabd184 into main Oct 30, 2026
8 of 12 checks passed
@Dong1017
Dong1017 deleted the issue255-token-v2-evidence branch October 30, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Tracking] Skill Indexer: retrieval efficiency and accuracy

1 participant