Skip to content

Improve schema validation, ID mapping, and IMPROVE export - #492

Merged
sgosline merged 1 commit into
schema-phosphoproteomicsfrom
validation-and-improve-prep
Sep 26, 2026
Merged

sgosline merged 1 commit into
schema-phosphoproteomicsfrom
validation-and-improve-prep

Conversation

@jjacobson95

@jjacobson95 jjacobson95 commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Pipeline Hardening/Debugging PR # 14

Improve schema validation, ID mapping, and IMPROVE export

Makes the post-build validation and drug-descriptor harmonization scripts faster and able to handle very large inputs. Adding a change so the mRESCIST will be changed to mrescist in the improve mapping file.

align_drug_descriptors.py: scale descriptor harmonization to huge files

  • Harmonizes descriptor values so the same drug gets a consistent value across datasets, using "first value seen wins" (files processed in order).
  • Rewritten to build the reference table in an on-disk SQLite database instead of holding every (drug_id, descriptor) -> value pair in memory at once. The previous in-memory approach ran out of memory on the largest inputs; this version scales to files like nci60 (~89M rows / ~2.7 GB).
  • Skips fingerprint descriptors (for example Morgan fingerprints) from harmonization, since those are per-molecule and don't need cross-dataset reconciliation.

check_schema.py: faster validation

  • Reworked so each worker compiles the LinkML schema once and reuses it, rather than restarting a fresh linkml-validate process (and recompiling the schema) for every file. That per-file startup cost dominated the previous runtime.
  • Uses the same closed-world JSON Schema validation as before, so results are unchanged; only the speed differs.

map_improve_drug_ids.py

  • Aligns its field-size limit with align_drug_descriptors.py so the two scripts stay consistent when reading large descriptor files.

Scope: 4 files. Base: schema-phosphoproteomics.

Comment thread scripts/prepare_data_for_improve.py Outdated
experiment.loc[experiment['mRESCIST'] == 'PR', 'mRESCIST'] = "0.2"
experiment.loc[experiment['mRESCIST'] == 'SD', 'mRESCIST'] = "0.5"
experiment.loc[experiment['mRESCIST'] == 'PD', 'mRESCIST'] = "1.0"
experiment.loc[experiment['mRESCIST'] == 'CR', 'mRESCIST'] = 0.1

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wasn't this changed to lower case in another PR?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great find. I didn't actually intend to make any changes to this file. I had an old version of it on my branch and accidentally re-committed that old version here.

Coincidentally, I didn't know about this logic in this file so I missed that "mRECIST" would have to be changed in this file as well.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted to Yannicks (up to date) version, changed the mrescist mismatch.

@ymahlich tagging for visibility. And adding you for review as this does touch one of the files you maintain.

@jjacobson95
jjacobson95 force-pushed the validation-and-improve-prep branch from 5300f52 to d550f70 Compare September 24, 2026 23:02
@sgosline
sgosline merged commit d1ab41c into schema-phosphoproteomics Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants