Skip to content

Add unvalidated LLM focal-effect flags to FReD - #166

Draft
LukasWallrich wants to merge 4 commits into
mainfrom
t3/focal-effect-llm-flag
Draft

LukasWallrich wants to merge 4 commits into
mainfrom
t3/focal-effect-llm-flag

Conversation

@LukasWallrich

@LukasWallrich LukasWallrich commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Draft pending source reconciliation. The post-pilot audit (validation/focal-sol-pilot/source_audit.json) recovered the tenth report from its existing source-sheet Drive link. The original pilot remains archived unchanged. One claim-quotation error (additional_studies15, archived effect 61 / permanent source ID fe01694) has an exact before/after correction prepared. add_ai11 and av_121 contain test-pairing mismatches requiring an explicit estimand-preservation rule before recoding. No source edits have been applied. Earlier additional_studies94 claim fixes and curatescience15/resciencex_01 added effects are already present in the live source. Resolve these issues and the study/sample selection unit before extending coding or treating this PR as ready.

FReD users need to select the effect a replication targeted while retaining the assignment's unvalidated LLM provenance. Implements the focal-effect decision in #162 and supersedes #148, carrying its flag CSV forward unchanged (context: #30).

The preparation pipeline adds focal_llm, focal_role_llm, focal_coverage_llm, focal_source_llm, and focal_confidence_llm. The codebook and documentation explain their meaning, use, unvalidated status, and provenance. The recorded model is gemini-flash-lite-latest; its resolved model version, the report-reading agent model, prompt version, and exact coding date were not recorded. These gaps are explicit; the source commit identifies the coding snapshot.

Because effect_id is positional, the join checks unique keys and frozen identity/claim/statistic anchors. Missing keys or changed effects stop the pipeline for reconciliation rather than guessing a mapping; new rows remain NA. reported_success and effect signs are unchanged.

Validation: Rscript tests/focal_effect_flags.R and isolated execution of the new pipeline chunk against the committed workbook passed. All 2,164 keys matched uniquely, with no unmatched or ambiguous keys.

Field Distribution
focal_llm TRUE 957; FALSE 1,013; NA 194
focal_role_llm focal 957; constituent 166; none 847; NA 194
focal_coverage_llm resolved 1,781; constituent_set 169; target_absent 13; focal_in_sibling 7; not_applicable 60; uncoded 134

1,970 rows have a nonmissing flag. Each of the 957 resolved pairs has one TRUE. Existing columns and codebook entries are preserved. Regression checks cover stale, missing, and duplicate identities, row order, and new rows. No live source-sheet download or re-rendered FReD.xlsx is included.

A blinded full-text pilot is archived under validation/focal-sol-pilot/ using GPT-6.1 Sol at medium reasoning through Codex. Nine full texts were read across ten sampled multi-effect cases (40 candidates). Comparison with the unchanged old labels: two identical focal selections; five multiple-target decisions; two mapping abstentions; one unavailable full text. Input/prompt/source hashes and supporting quotations were checked. Run python3 validation/focal-sol-pilot/compare.py to reproduce the comparison (source-hash checks require the private cached reports).

The pilot is not an accuracy estimate or replacement coding. Some fred_ids contain several replication studies; a larger run first needs an explicit study/sample unit to distinguish independent studies from correlated outcomes. The model, evidence scope, and coding protocol changed together, so disagreement cannot be attributed to model quality alone. Possible claim/statistic mismatches and a manipulation-check selection are documented for review. Existing flags, workbook, signs, and reported_success remain unchanged.

Created with GPT-6.1 Sol in T3 Code (Codex harness), including a delegated GPT-6.1 Sol medium pilot.

@LukasWallrich
LukasWallrich marked this pull request as draft November 10, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant