Repository navigation
feat(chemistry): ingest isotope results (BDMS-1461) - #1002
Open
ksmuczynski wants to merge 10 commits into
Open
ksmuczynski wants to merge 10 commits into
ksmuczynski wants to merge 10 commits into
Conversation
The isotope ingest has to archive what it reads before mapping it, the same way the field sheet does. That step was a private helper tied to the field sheet's dataset, so the isotope ingest had no way to call it. The step now lives in ingest_raw_zone and takes the dataset as an argument. The field sheet behaves exactly as before. Refs BDMS-1461
The isotope report needs the same top and bottom as the field-sheet report. Both show the outcome, the database, the source and the archive, and both list where the failed row workbook and loadable copy went. Two copies of that code would drift apart. The field-sheet report now gets both parts from two helpers. Its output is unchanged. Refs BDMS-1461
Isotope results use four unit abbreviations that the legacy tables already hold, pm, pmc, TU and yrs. Nothing in the code said what they mean, and each ingest spells its units its own way. The lexicon's unit category doesn't fit. It belongs to the Observation model, where unit is a foreign key. The NMA_* units columns are plain text, so a lexicon term there would check nothing. The lexicon also writes units differently, deg C where the field sheet writes °C. LEGACY_UNITS goes in services/legacy_chemistry.py instead, beside LEGACY_ANALYTE_NAMES. A test checks that every unit the isotope ingest writes is in it, so a new abbreviation can't appear without its meaning recorded. Refs BDMS-1461
The IsotopeResults tab of the field spreadsheet had no way into the database. The LIMS ingest can't read it. It handles one lab's export and records every result as NMBGMR, while this tab holds results from NMBGMR, Beta Analytic and the University of Miami Tritium Laboratory, with qualifiers and uncertainties. The ingest writes to NMA_MinorTraceChemistry. Legacy isotopes already live in that table, and /chemistry/results and the OGC views already read from it. Choices worth knowing: - Each result attaches to a sample that already exists, found by its field sample ID, SamplePointID. The ingest never creates a sample. The tab has no collection date, and the other ingests can't match a sample without one. - Sample, analyte and lab together identify a result. Beta Analytic and NMBGMR both report H2r and O18r for 33 samples, and the ingest keeps both. - The ingest skips a result it already stored unchanged, so a re-run is safe. A changed result stops the run. It may be a re-analysis, and a person decides which value stands. - The lab's sample id goes on each result, not on the sample. The sample has one lab id slot, and the LIMS ingest only adopts a sample whose slot is empty. - The ingest leaves nma_GlobalID and nma_chemistry_sample_info_uuid empty. Those columns hold NM_Aquifer's values only. - Labs come from a fixed list. A misspelled lab would otherwise store a duplicate instead of matching. - The ingest refuses a result analysed before its sample was collected. Like the other chemistry ingests, one failed row stops the whole run and nothing gets written. Refs BDMS-1461
Isotope results come back weeks or months after the sample info and field parameters. Loading the tab with sync-sheet or field-upload would make the crew's data wait on the slowest lab. The tab gets its own commands instead. isotope-upload reads a downloaded copy and isotope-sync-sheet reads Drive. Both take the field sheet commands' options and name the database in the report. Refs BDMS-1461
The isotope ingest is new, so each rule gets a test against the test database: - each measurement becomes one result, with its qualifier and uncertainty - lab ids are stored as text, without a trailing .0 - each kind of bad row is refused - three labs on one sample load side by side, each with its own lab id - the sample record is left alone, and legacy audit columns stay empty - a re-run skips, and a changed value or qualifier stops the run - a missing, ambiguous or unknown-well sample fails its row - a result analysed before collection fails its row - conflicting rows fail, and a repeated row loads once - dry run, both reports, archive and replay - the LIMS ingest still adopts the sample afterwards The last test guards the lab id rule. If the isotope ingest filled the sample's lab id, the LIMS ingest would refuse that sample. The raw_zone fixture moves to conftest, since the field-sheet and isotope tests both use it now. Refs BDMS-1461
The new commands get the same checks as field-upload. The report lists skips and empty rows, names the database, and writes the failed row workbook and the loadable copy. Refs BDMS-1461
The field-sheet doc, the LIMS runbook and CLAUDE.md all said the IsotopeResults tab had no ingest. They now point to a new doc, docs/chemistry-isotope-ingest.md. The new doc covers the commands, the column mapping, how results find their sample, re-runs, failures and known gaps. The biggest gap is the ogc_minor_chemistry_wells layer, which shows one value per well and analyte. Where two labs report H2r, only one appears. Refs BDMS-1461
The isotope ingest reads only the measurement columns it knows. If a lab starts reporting something new and someone adds a column for it, those results would never load, and the report gave no sign of it. The report now names any column on the tab that the ingest doesn't read, under WARNINGS. The run still loads everything else. Refs BDMS-1461
Whoever loads isotope results may not be an engineer. The field sheet has a step-by-step guide for them, but the isotope ingest only had a reference doc written for whoever maintains the code. The new guide walks through checking the tab, loading the sample info first, the practice run, fixing failed rows, the real run, and what to do when a stored result changes. It links to the field sheet guide for setup instead of repeating it, so the two can't drift apart. The field sheet guide and the reference doc now point to it, and the reference doc lists the unread-column warning among its known gaps. Refs BDMS-1461
Contributor
Coverage✅ 81.68% total — gate is 75%. Coverage for the Python files changed in this PR
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
IsotopeResultstab of the chemistry field spreadsheet now loads intoNMA_MinorTraceChemistry, through two new commands,oco water-chemistry isotope-uploadandoco water-chemistry isotope-sync-sheet. Each result attaches to a sample the field-sheet or LIMS ingest already recorded.Why
This PR addresses the following problem / context:
IsotopeResultstab had no ingest path. The LIMS ingest only reads lab exports from Bonnie's lab, and the field-sheet ingest intentionally skips theIsotopeResultstab.How
Implementation summary - the following was changed / added / removed:
services/chemistry_isotopes.py, which turns each value cell into oneNMA_MinorTraceChemistryrow and attaches it to its sample by field sample ID.isotope-uploadreads a downloaded copy of the spreadsheet, andisotope-sync-sheetreads it from Google Drive. Both take the field-sheet commands' options: dry run, failed rows, loadable copy and the raw-zone flags.LEGACY_UNITStoservices/legacy_chemistry.py.services/ingest_raw_zone.py, and the report header and footer into CLI helpers, so both ingests share them.docs/chemistry-isotope-ingest.md, and pointed the field-sheet doc, the LIMS runbook and CLAUDE.md to it.docs/chemistry-isotope-ingest-walkthrough.md, a step-by-step guide for whoever runs the ingest.Notes
Any special considerations, workarounds, or follow-up work to note?
nma_WCLab_ID, which the LIMS ingest needs empty to adopt a sample.ogc_minor_chemistry_wellsshows one value per well and analyte, so where two labs report H2r, only one appears there. That layer also has no tritium column. Both are left for later.WL-0433), two IDs without a letter, and two analysis dates before collection.Refs BDMS-1461