Repository navigation
Improve schema validation, ID mapping, and IMPROVE export - #492
Merged
Merged
Conversation
sgosline
reviewed
Sep 24, 2026
| experiment.loc[experiment['mRESCIST'] == 'PR', 'mRESCIST'] = "0.2" | ||
| experiment.loc[experiment['mRESCIST'] == 'SD', 'mRESCIST'] = "0.5" | ||
| experiment.loc[experiment['mRESCIST'] == 'PD', 'mRESCIST'] = "1.0" | ||
| experiment.loc[experiment['mRESCIST'] == 'CR', 'mRESCIST'] = 0.1 |
Member
There was a problem hiding this comment.
wasn't this changed to lower case in another PR?
Collaborator
Author
There was a problem hiding this comment.
Great find. I didn't actually intend to make any changes to this file. I had an old version of it on my branch and accidentally re-committed that old version here.
Coincidentally, I didn't know about this logic in this file so I missed that "mRECIST" would have to be changed in this file as well.
Collaborator
Author
There was a problem hiding this comment.
Reverted to Yannicks (up to date) version, changed the mrescist mismatch.
@ymahlich tagging for visibility. And adding you for review as this does touch one of the files you maintain.
jjacobson95
force-pushed
the
validation-and-improve-prep
branch
from
September 24, 2026 23:02
5300f52 to
d550f70
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipeline Hardening/Debugging PR # 14
Improve schema validation, ID mapping, and IMPROVE export
Makes the post-build validation and drug-descriptor harmonization scripts faster and able to handle very large inputs. Adding a change so the mRESCIST will be changed to mrescist in the improve mapping file.
align_drug_descriptors.py: scale descriptor harmonization to huge files(drug_id, descriptor) -> valuepair in memory at once. The previous in-memory approach ran out of memory on the largest inputs; this version scales to files like nci60 (~89M rows / ~2.7 GB).check_schema.py: faster validationlinkml-validateprocess (and recompiling the schema) for every file. That per-file startup cost dominated the previous runtime.map_improve_drug_ids.pyalign_drug_descriptors.pyso the two scripts stay consistent when reading large descriptor files.Scope: 4 files. Base: schema-phosphoproteomics.