Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 52 additions & 2 deletions schema/coderdata.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ slots:
entrez_id:
description: Gene id according to NCBI
range: integer
phosphosite_id:
description: Unique identifier for a phosphorylation site (gene + residue + position)
identifier: true
range: integer
improve_sample_id:
description: Unique sample identifier for this dataset, generated by this package
identifier: true
Expand Down Expand Up @@ -49,6 +53,24 @@ classes:
gene_symbol:
required: true
description: HGNC Gene symbol or synomym
Phosphosite:
description: Reference table of phosphorylation sites; each row is a unique gene + residue + position combination.
slots:
- phosphosite_id
- entrez_id
- other_id
- other_id_source
attributes:
gene_symbol:
required: true
description: HGNC gene symbol
residue:
description: Phosphorylated amino acid residue (S, T, or Y)
position:
range: integer
description: Position of the residue in the canonical protein sequence
modification:
description: Type of modification (e.g. phospho)
Sample:
description: Unique samples identified for each experiment, used to cross reference omics data to drug data.
slots:
Expand Down Expand Up @@ -122,6 +144,16 @@ classes:
proteomics:
range: float
description: Log-normalized log ratio of proteomics measurements of sample at baseline.
Phosphoproteomics:
slots:
- phosphosite_id
- improve_sample_id
- source
- study
attributes:
phosphoproteomics:
range: float
description: Log-normalized, batch-corrected phosphoproteomics abundance value (correctedAbundance).
Copy Number:
slots:
- entrez_id
Expand Down Expand Up @@ -157,7 +189,7 @@ classes:
- time_unit
attributes:
dose_response_metric:
range: CurveMetric
range: ResponseMetric
description: Metric by which dose response value is measured
dose_response_value:
description: Value of metric
Expand Down Expand Up @@ -187,6 +219,8 @@ classes:
- improve_sample_id
- source
- study
- time
- time_unit
attributes:
improve_drug_1:
description: improve_drug_id of first drug
Expand All @@ -195,6 +229,7 @@ classes:
drug_combination_metric:
description: metric calculated for synergy, or other metric of two drugs
drug_combination_value:
range: linkml:Any
description: value of metric for synergy or combination
enums:
ResponseMetric:
Expand All @@ -218,7 +253,7 @@ enums:
auc:
description: Area under the curve measured by integrating data points
dss:
description: I believe this is the drug sensitivity score
description: Drug sensistivity score.
mrecist:
description: For PDX data this value should be either Progressive Disease, Stable Disease, Partial Response or Complete Response.
published_auc:
Expand All @@ -233,6 +268,19 @@ enums:
description: tumor growth inhibitiojn
kulgap:
description: kl divergence based metric
abc:
description: Area between curves for PDX in-vivo response
lmm:
description: Linear mixed model coefficient for PDX tumor growth response
uM_viability:
description:
Single-dose viability fraction at 1 μM. Values near 1.0 indicate no
effect (full viability); values near 0.0 indicate full kill. Values
above 1.0 are possible when treated cells proliferate beyond the
untreated control. Used for organoid drug screens where dose-response
curves were not collected for that drug.
delta_auc:
description: Difference in AUC between single-drug and combination treatment; used for MPNST treated-microtissue combination experiments.
CellPerturbation:
permissible_values:
gene_ko:
Expand All @@ -248,6 +296,8 @@ enums:
xenograft derived organoid:
patient derived organoid:
ex vivo:
normal_tissue:
3D-MEDS:
Variant:
permissible_values:
3'UTR:
Expand Down
33 changes: 32 additions & 1 deletion schema/expected_files.yaml

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am guessing that the same as with the other issues that contain CNF applies here as well, i.e. we keep the code but don't include the CNF data in the bulid?

Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ datasets:
file: /tmp/mpnst_drugs.tsv
- target_class: Drug Descriptor
file: /tmp/mpnst_drug_descriptors.tsv
- target_class: Combinations
file: /tmp/mpnst_combinations.tsv

cptac:
- target_class: Sample
Expand Down Expand Up @@ -294,4 +296,33 @@ datasets:
- target_class: Drug
file: /tmp/novartis_drugs.tsv
- target_class: Drug Descriptor
file: /tmp/novartis_drug_descriptors.tsv
file: /tmp/novartis_drug_descriptors.tsv


cnf:
- target_class: Sample
file: /tmp/cnf_samples.csv
- target_class: Transcriptomics
file: /tmp/cnf_transcriptomics.csv
- target_class: Proteomics
file: /tmp/cnf_proteomics.csv
- target_class: Phosphoproteomics
file: /tmp/cnf_phosphoproteomics.csv
# - target_class: Mutations
# file: /tmp/cnf_mutations.csv
# - target_class: Copy Number
# file: /tmp/cnf_copy_number.csv
- target_class: Experiments
file: /tmp/cnf_experiments.tsv
- target_class: Drug
file: /tmp/cnf_drugs.tsv
- target_class: Drug Descriptor
file: /tmp/cnf_drug_descriptors.tsv

phosphosites:
- target_class: Phosphosite
file: /tmp/phosphosites.csv

genes:
- target_class: Gene
file: /tmp/genes.csv
107 changes: 87 additions & 20 deletions scripts/align_drug_descriptors.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,45 @@
#!/usr/bin/env python3
"""Harmonize drug_descriptor values across datasets.

For each (improve_drug_id, structural_descriptor) seen across the per-dataset
`*_drug_descriptors.tsv[.gz]` files, the FIRST value encountered (files processed
in sorted order) is treated as canonical, and any later file whose value differs
is rewritten to match.

Memory-bounded implementation
-----------------------------
The reference map is kept in an on-disk SQLite database, not a Python dict, so the
step scales to very large descriptor files (e.g. nci60 ~89M rows / 2.7 GB). The
previous in-memory dict held every (drug_id, descriptor) -> value pair at once and
was OOM-killed by the container on large builds.

Fingerprint descriptors are skipped from harmonization: a fingerprint is a
deterministic function of chemical structure, so the same drug always yields the
same value -- there is nothing to harmonize, and these rows dominate both memory
and file size. They are passed through unchanged.
"""
import os
import sys
import gzip
import shutil
import csv
import argparse
import sqlite3
import tempfile

# Descriptor values (e.g. Morgan fingerprints) can be long strings.
csv.field_size_limit(min(sys.maxsize, 2**31 - 1))

# Descriptor names (case-insensitive substring match) excluded from harmonization.
SKIP_DESCRIPTOR_SUBSTRINGS = ("fingerprint",)

_INSERT_BATCH = 100_000


def _skip_descriptor(name):
n = name.lower()
return any(s in n for s in SKIP_DESCRIPTOR_SUBSTRINGS)


# Helper scripts
def decompress_gz_if_needed(path):
Expand All @@ -15,13 +51,15 @@ def decompress_gz_if_needed(path):
return out, True
return path, False


def recompress_if_needed(decompressed, was_gz, original):
"""If was_gz, recompress decompressed back to original and remove decompressed."""
if was_gz:
with open(decompressed, 'rb') as f_in, gzip.open(original, 'wb') as f_out:
shutil.copyfileobj(f_in, f_out)
os.remove(decompressed)


def find_descriptor_files(directory):
"""Find all *_drug_descriptors.tsv and .tsv.gz files in directory."""
files = []
Expand All @@ -32,23 +70,39 @@ def find_descriptor_files(directory):


# Actual work
def build_reference_map(files):
"""Build map from (drug_id, descriptor) with first found value."""
ref = {}
def build_reference_db(files, conn):
"""Populate SQLite `ref` with the first-seen value per (drug_id, descriptor).

INSERT OR IGNORE keeps the first value encountered, matching the original
'first found wins' semantics. Fingerprint descriptors are not stored.
"""
cur = conn.cursor()
cur.execute(
"CREATE TABLE ref (drug_id TEXT, descriptor TEXT, value TEXT, "
"PRIMARY KEY (drug_id, descriptor)) WITHOUT ROWID"
)
for fp in files:
path, gz = decompress_gz_if_needed(fp)
with open(path, newline='', encoding='utf-8') as f:
reader = csv.DictReader(f, delimiter='\t')
batch = []
for row in reader:
key = (row['improve_drug_id'], row['structural_descriptor'])
val = row['descriptor_value']
if key not in ref:
ref[key] = val
desc = row['structural_descriptor']
if _skip_descriptor(desc):
continue
batch.append((row['improve_drug_id'], desc, row['descriptor_value']))
if len(batch) >= _INSERT_BATCH:
cur.executemany("INSERT OR IGNORE INTO ref VALUES (?,?,?)", batch)
batch.clear()
if batch:
cur.executemany("INSERT OR IGNORE INTO ref VALUES (?,?,?)", batch)
conn.commit()
recompress_if_needed(path, gz, fp)
return ref

def rewrite_files(files, ref):
"""Go back and rewrite any mismatches in-place."""

def rewrite_files(files, conn):
"""Rewrite any mismatched descriptor_value in-place to the canonical value."""
cur = conn.cursor()
for fp in files:
path, gz = decompress_gz_if_needed(fp)
tmp = path + '.tmp'
Expand All @@ -62,13 +116,16 @@ def rewrite_files(files, ref):
writer.writeheader()

for row in reader:
key = (row['improve_drug_id'], row['structural_descriptor'])
correct = ref.get(key)
if correct is not None and row['descriptor_value'] != correct:
# print(f"Fixing {key} in {os.path.basename(fp)}: "
# f"{row['descriptor_value']} to {correct}")
row['descriptor_value'] = correct
changed = True
desc = row['structural_descriptor']
if not _skip_descriptor(desc):
cur.execute(
"SELECT value FROM ref WHERE drug_id=? AND descriptor=?",
(row['improve_drug_id'], desc),
)
res = cur.fetchone()
if res is not None and row['descriptor_value'] != res[0]:
row['descriptor_value'] = res[0]
changed = True
writer.writerow(row)

if changed:
Expand All @@ -78,21 +135,31 @@ def rewrite_files(files, ref):

recompress_if_needed(path, gz, fp)


def main():
parser = argparse.ArgumentParser(
description="Harmonize drug_descriptor values across multiple files."
)
parser.add_argument('--local_dir', default='.', help='Folder containing *_drug_descriptors.tsv[.gz]')
parser.add_argument('--version', help=argparse.SUPPRESS) # ignore the version input
parser.add_argument('--version', help=argparse.SUPPRESS) # ignore the version input
args = parser.parse_args()
files = find_descriptor_files(args.local_dir)
if not files:
print("No drug_descriptor files found in", args.local_dir)
return

ref = build_reference_map(files)
rewrite_files(files, ref)
with tempfile.TemporaryDirectory() as tmpdir:
conn = sqlite3.connect(os.path.join(tmpdir, "descriptors.sqlite"))
# Temp DB: durability is irrelevant, so favour speed.
conn.execute("PRAGMA journal_mode=OFF")
conn.execute("PRAGMA synchronous=OFF")
try:
build_reference_db(files, conn)
rewrite_files(files, conn)
finally:
conn.close()
print("Done.")


if __name__ == '__main__':
main()
Loading