Repository navigation
Add unvalidated LLM focal-effect flags to FReD - #166
Draft
LukasWallrich wants to merge 4 commits into
Draft
LukasWallrich wants to merge 4 commits into
LukasWallrich wants to merge 4 commits into
Conversation
LukasWallrich
marked this pull request as draft
November 10, 2026 16:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft pending source reconciliation. The post-pilot audit (
validation/focal-sol-pilot/source_audit.json) recovered the tenth report from its existing source-sheet Drive link. The original pilot remains archived unchanged. One claim-quotation error (additional_studies15, archived effect 61 / permanent source ID fe01694) has an exact before/after correction prepared.add_ai11andav_121contain test-pairing mismatches requiring an explicit estimand-preservation rule before recoding. No source edits have been applied. Earlier additional_studies94 claim fixes and curatescience15/resciencex_01 added effects are already present in the live source. Resolve these issues and the study/sample selection unit before extending coding or treating this PR as ready.FReD users need to select the effect a replication targeted while retaining the assignment's unvalidated LLM provenance. Implements the focal-effect decision in #162 and supersedes #148, carrying its flag CSV forward unchanged (context: #30).
The preparation pipeline adds
focal_llm,focal_role_llm,focal_coverage_llm,focal_source_llm, andfocal_confidence_llm. The codebook and documentation explain their meaning, use, unvalidated status, and provenance. The recorded model isgemini-flash-lite-latest; its resolved model version, the report-reading agent model, prompt version, and exact coding date were not recorded. These gaps are explicit; the source commit identifies the coding snapshot.Because
effect_idis positional, the join checks unique keys and frozen identity/claim/statistic anchors. Missing keys or changed effects stop the pipeline for reconciliation rather than guessing a mapping; new rows remain NA.reported_successand effect signs are unchanged.Validation:
Rscript tests/focal_effect_flags.Rand isolated execution of the new pipeline chunk against the committed workbook passed. All 2,164 keys matched uniquely, with no unmatched or ambiguous keys.focal_llmfocal_role_llmfocal_coverage_llm1,970 rows have a nonmissing flag. Each of the 957 resolved pairs has one TRUE. Existing columns and codebook entries are preserved. Regression checks cover stale, missing, and duplicate identities, row order, and new rows. No live source-sheet download or re-rendered FReD.xlsx is included.
A blinded full-text pilot is archived under
validation/focal-sol-pilot/using GPT-6.1 Sol at medium reasoning through Codex. Nine full texts were read across ten sampled multi-effect cases (40 candidates). Comparison with the unchanged old labels: two identical focal selections; five multiple-target decisions; two mapping abstentions; one unavailable full text. Input/prompt/source hashes and supporting quotations were checked. Runpython3 validation/focal-sol-pilot/compare.pyto reproduce the comparison (source-hash checks require the private cached reports).The pilot is not an accuracy estimate or replacement coding. Some fred_ids contain several replication studies; a larger run first needs an explicit study/sample unit to distinguish independent studies from correlated outcomes. The model, evidence scope, and coding protocol changed together, so disagreement cannot be attributed to model quality alone. Possible claim/statistic mismatches and a manipulation-check selection are documented for review. Existing flags, workbook, signs, and reported_success remain unchanged.
Created with GPT-6.1 Sol in T3 Code (Codex harness), including a delegated GPT-6.1 Sol medium pilot.