Repository navigation
Description
Problem
It seems that NCI60 contain "duplicate" entries in the drug response data causing the experiment data formatting of the NCI60 dataset object to fail (dataset.format('experiments', shape='wide')).
Likely related to #455.
This time the entries are not exact duplicates.
Steps to reproduce:
0. optionally create clean python environment
1. install most recent version of coderdata.
Currently that is commit 432a205.
I did that by directly installing coder data via pip
pip install git+https://github.com/PNNL-CompBio/coderdata@432a205586136f8355cabba029b309be79a72cd9Alternatively via
git clone https://github.com/PNNL-CompBio/coderdata coderdata.git
cd coderdata.git
pip install .2. Validate the coderdata version we are running
coderdata --versionThis returns
package version: 2.2.2.dev86+g432a20558
dataset version: 2.43. Download the NCI60 dataset
mkdir coderdata-debug
cd coderdata-debug
coderdata download -n nci604. Load dataset and format experiment data
Running the following python code will result in an ValueError: Index contains duplicate entries, cannot reshape error:
>>> import coderdata as cd
>>> nci60 = cd.load('nci60')
>>> exp = nci60.format("experiments", shape="wide", metrics=["auc", "fit_auc", "fit_r2"])5. Narrowing down the offending entries
The following set of python commands 827 entries that contain duplicate values (we have 10 different drug response metrics, 20 is double that - see output).
>>> exp = nci60.experiments
>>> grouped_exp = exp.groupby(['source', 'improve_sample_id', 'improve_drug_id', 'study', 'time', 'time_unit']).size().reset_index(name='row_count')
>>> grouped_exp[grouped_exp['row_count'] > 10]
source improve_sample_id improve_drug_id study time time_unit row_count
1516 NCI60_24 46 SMI_11379 NCI60 48 hours 20
3589 NCI60_24 46 SMI_13316 NCI60 48 hours 20
11505 NCI60_24 46 SMI_2079 NCI60 48 hours 20
11570 NCI60_24 46 SMI_2085 NCI60 48 hours 20
13716 NCI60_24 46 SMI_22850 NCI60 48 hours 20
... ... ... ... ... ... ... ...
2901262 NCI60_24 1916 SMI_41659 NCI60 48 hours 20
2902229 NCI60_24 1916 SMI_42719 NCI60 48 hours 20
2904523 NCI60_24 1916 SMI_453 NCI60 48 hours 20
2908077 NCI60_24 1916 SMI_49184 NCI60 48 hours 20
2911558 NCI60_24 1916 SMI_53042 NCI60 48 hours 20
[827 rows x 7 columns]6. Double checking the results
Explicitly checking the top result from the output above yields this result. As can be seen we get 2 reported values per drug response metric.
>>> exp[(exp['improve_drug_id'] == 'SMI_11379') & (exp['improve_sample_id'] == 46)]
source improve_sample_id improve_drug_id study time time_unit dose_response_metric dose_response_value
1516 NCI60_24 46 SMI_11379 NCI60 48 hours fit_auc 0.8788
48658 NCI60_24 46 SMI_11379 NCI60 48 hours fit_auc 0.8993
2920510 NCI60_24 46 SMI_11379 NCI60 48 hours fit_ic50 4.7260
2967652 NCI60_24 46 SMI_11379 NCI60 48 hours fit_ic50 4.6020
5839504 NCI60_24 46 SMI_11379 NCI60 48 hours fit_ec50 4.7260
5886646 NCI60_24 46 SMI_11379 NCI60 48 hours fit_ec50 4.6020
8758498 NCI60_24 46 SMI_11379 NCI60 48 hours fit_r2 0.9855
8805640 NCI60_24 46 SMI_11379 NCI60 48 hours fit_r2 0.9944
11677492 NCI60_24 46 SMI_11379 NCI60 48 hours fit_ec50se 10.5500
11724634 NCI60_24 46 SMI_11379 NCI60 48 hours fit_ec50se 26.8700
14596486 NCI60_24 46 SMI_11379 NCI60 48 hours fit_einf 0.0000
14643628 NCI60_24 46 SMI_11379 NCI60 48 hours fit_einf 0.0000
17515480 NCI60_24 46 SMI_11379 NCI60 48 hours fit_hs 2.7670
17562622 NCI60_24 46 SMI_11379 NCI60 48 hours fit_hs 2.9000
20434474 NCI60_24 46 SMI_11379 NCI60 48 hours aac 0.1818
20481616 NCI60_24 46 SMI_11379 NCI60 48 hours aac 0.1208
23353468 NCI60_24 46 SMI_11379 NCI60 48 hours auc 0.8182
23400610 NCI60_24 46 SMI_11379 NCI60 48 hours auc 0.8792
26272462 NCI60_24 46 SMI_11379 NCI60 48 hours dss 0.1975
26319604 NCI60_24 46 SMI_11379 NCI60 48 hours dss 0.2218To make sure that this is not a weird coderdata package behaviour, the following command was run in bash.
❯ zcat < nci60_experiments.tsv.gz | awk '$2 == "46" && $3 == "SMI_11379"'
NCI60_24 46 SMI_11379 NCI60 48 hours fit_auc 0.8788
NCI60_24 46 SMI_11379 NCI60 48 hours fit_auc 0.8993
NCI60_24 46 SMI_11379 NCI60 48 hours fit_ic50 4.726
NCI60_24 46 SMI_11379 NCI60 48 hours fit_ic50 4.602
NCI60_24 46 SMI_11379 NCI60 48 hours fit_ec50 4.726
NCI60_24 46 SMI_11379 NCI60 48 hours fit_ec50 4.602
NCI60_24 46 SMI_11379 NCI60 48 hours fit_r2 0.9855
NCI60_24 46 SMI_11379 NCI60 48 hours fit_r2 0.9944
NCI60_24 46 SMI_11379 NCI60 48 hours fit_ec50se 10.55
NCI60_24 46 SMI_11379 NCI60 48 hours fit_ec50se 26.87
NCI60_24 46 SMI_11379 NCI60 48 hours fit_einf 0.0
NCI60_24 46 SMI_11379 NCI60 48 hours fit_einf 0.0
NCI60_24 46 SMI_11379 NCI60 48 hours fit_hs 2.767
NCI60_24 46 SMI_11379 NCI60 48 hours fit_hs 2.9
NCI60_24 46 SMI_11379 NCI60 48 hours aac 0.1818
NCI60_24 46 SMI_11379 NCI60 48 hours aac 0.1208
NCI60_24 46 SMI_11379 NCI60 48 hours auc 0.8182
NCI60_24 46 SMI_11379 NCI60 48 hours auc 0.8792
NCI60_24 46 SMI_11379 NCI60 48 hours dss 0.1975
NCI60_24 46 SMI_11379 NCI60 48 hours dss 0.2218As can be seen the output (except for formatting) is identical, hence to problem lies with the dataset that has been generated by coderbuild and uploaded to figshare.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
sgosline commentedon Oct 19, 2026
This is concerning, is it just a merge errory? @jjacobson95 what tests do you have in place to avoid this type of thing in the future?
jjacobson95 commentedon Oct 19, 2026
Still trying to track down exactly why this is occurring, but its not due from a change to the input data like the other updates were.
Also just adding a note that this affects
prismdata as well.jjacobson95 commentedon Oct 19, 2026
We could add a test right before the schema check that checks for duplicate values across all datasets.
Currently the schema check is the main source of testing we have, but it might be worth having a small suite of tests at that point in the build as well.