Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 37 additions & 6 deletions coderbuild/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ are added.
This script initializes all docker containers, builds all datasets, validates them, and uploads them to figshare.

It requires the following authorization tokens to be set in the local environment depending on the use case:
`SYNAPSE_AUTH_TOKEN`: Required for beataml and mpnst datasets. Join the [CoderData team](https://www.synapse.org/#!Team:3503472) on Synapse and generate an access token.
`SYNAPSE_AUTH_TOKEN`: Required for the beataml, mpnst, bladder, pancreatic, liver, sarcoma, novartis, colorectal and broad_sanger (DepMap files) datasets. Join the [CoderData team](https://www.synapse.org/#!Team:3503472) on Synapse and generate an access token.
`FIGSHARE_TOKEN`: This token is required to upload to Figshare.
`GITHUB_TOKEN`: This token is required to upload to GitHub.

Expand All @@ -24,11 +24,17 @@ It requires the following authorization tokens to be set in the local environmen
- `--omics`: Processes and builds the omics data files.
- `--drugs`: Processes and builds the drug data files.
- `--exp`: Processes and builds the experiment data files.
- `--all`: Executes all available processes above (docker, samples, omics, drugs, exp). This does not run the validate or figshare commands.
- `--validate`: Validates the generated datasets using the schema check scripts. This is automatically included if data upload occurs.
- `--misc`: Runs the final post-build step (e.g. splitting broad_sanger into its datasets).
- `--prev_drugs`: Comma-separated list of previously built drug files. They seed the PubChem lookup so only new drugs are queried.
- `--all`: Executes all available processes above (docker, samples, omics, drugs, exp, misc). This does not run the validate or figshare commands.
- `--validate`: Validates the generated datasets with the build data checks (see below) and the schema check scripts. This is automatically included if data upload occurs.
- `--previous-summary`: Optional. `validation_summary.json(.gz)` from the previous release, compared against by the build data checks.
- `--skip-build-checks`: Skips the build data checks during `--validate` (not recommended).
- `--figshare`: Uploads the datasets to Figshare. FIGSHARE_TOKEN must be set in local environment.
- `--high_mem`: Utilizes high memory mode for concurrent data processing. This has been successfully tested using 32 or more vCPUs.
- `--dataset`: Specifies the datasets to process (default='broad_sanger,hcmi,beataml,mpnst,cptac').
- `--dataset`: Specifies the datasets to process (default='broad_sanger,beataml,pancreatic,bladder,sarcoma,liver,novartis,colorectal,mpnst,cptac,hcmi').
- `--depmap-ready`: Confirms that the DepMap files on Synapse are up to date for the current DepMap release. Required for any build that includes broad_sanger.
- `--continue`: Resumes a partial build, skipping steps whose output already exists in `local/`. Progress is logged to `local/build_progress.log`.
- `--version`: Specifies the version number for the Figshare upload title (e.g., "0.1.29"). This must be a higher version than previously published versions.
- `--github-username`: GitHub username matching the GITHUB_TOKEN. Required to push the new Tag to the GitHub Repository.
- `--github-email`: GitHub email matching the GITHUB_TOKEN. Required to push the new Tag to the GitHub Repository.
Expand All @@ -37,7 +43,7 @@ It requires the following authorization tokens to be set in the local environmen
- Build all datasets and upload to Figshare and GitHub.
Required tokens for the following command: `SYNAPSE_AUTH_TOKEN`, `FIGSHARE_TOKEN`, `GITHUB_TOKEN`.
```bash
python coderbuild/build_all.py --all --high_mem --validate --figshare --version 0.1.41 --github-username jjacobson95 --github-email jeremy.jacobson3402@gmail.com
python coderbuild/build_all.py --all --high_mem --validate --depmap-ready --figshare --version 0.1.41 --github-username jjacobson95 --github-email jeremy.jacobson3402@gmail.com
```

- Build only the experiment files.
Expand All @@ -51,14 +57,16 @@ This script builds a single dataset for **debugging purposes only**. It can help

It requires the following authorization tokens to be set in the local environment depending on the dataset:

`SYNAPSE_AUTH_TOKEN`: Required for beataml and mpnst datasets. Follow the directions above to use gain access.
`SYNAPSE_AUTH_TOKEN`: Required for most datasets (see the list above). Follow the directions above to gain access.

Available arguments:
- `--dataset`: Required. Name of the dataset to build. At a minimum, this will build the docker images.
- `--use_prev_dataset`: Optional. Prefix of the previous dataset for sample and drug ID continuation. The previous dataset files must be in the "local" directory.
- `--build`: Optional. Build the desired Dataset.
- `--validate`: Optional. Run the schema checker on the built files.
- `--continue`: Optional. Continues from where the build left off by skipping existing files in "local" directory.
- `--depmap-ready`: Required when building broad_sanger. Confirms the DepMap files on Synapse are up to date.

Example usage:

Build the broad_sanger dataset:
Expand All @@ -78,6 +86,29 @@ Build the broad_sanger dataset but skip previously built files in "local" direct
python coderbuild/build_dataset.py --dataset broad_sanger --continue
```

## Build data checks (validate_build.py)

These checks look for problems the schema check cannot catch: duplicate results, ids that do not match between files, sample ids shared between datasets, copied responses, and large changes from the previous release. `build_all.py --validate` runs them automatically. They can also be run on their own against any folder of CoderData files, either a local build (`local/all_files_dir`) or a release downloaded from Figshare. Files may be plain or `.gz`. They need `pandas` and `numpy`; the pytest form also needs `pytest`.

As a command:
```bash
python coderbuild/utils/validate_build.py --dir local/all_files_dir
```
- `--dir`: Required. Folder of files to check.
- `--datasets`: Optional. Check only these datasets (default: all found in `--dir`).
- `--previous-summary`: Optional. Summary of an earlier build to compare counts with. Each release publishes one as `validation_summary.json.gz`.
- `--write-summary`: Optional. Save this build's counts, to use as the next baseline.

Results are printed to the terminal. The command exits with code 1 if there are errors; warnings do not change the exit code.

As tests:
```bash
pytest tests/test_build_data.py --build-dir local/all_files_dir
```
Add `--build-datasets beataml,hcmi` to check only some datasets, or `--previous-summary <file>` to compare with a previous build. Each dataset is one test. Errors fail the test, and warnings show as pytest warnings. Without `--build-dir` these tests are skipped.

In `build_all.py` the checks run after the id mapping and before the schema check, and write `validation_summary.json` into `local/all_files_dir` so it is published with the release. If they find errors, the build stops after the schema check.




Expand Down
42 changes: 41 additions & 1 deletion coderbuild/build_all.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,11 @@ def main():
parser.add_argument('--high_mem',dest='high_mem',default=False,action='store_true',help = "If you have 32 or more CPUs, this option is recommended. It will run many code portions in parallel. If you don't have enough memory, this will cause a run failure.")
parser.add_argument('--dataset',dest='datasets',default='broad_sanger,beataml,pancreatic,bladder,sarcoma,liver,novartis,colorectal,mpnst,cptac,hcmi',help='Datasets to process. Defaults to all available.')
parser.add_argument('--version', type=str, required=False, help='Version number for the Figshare upload title (e.g., "0.1.29"). This is required for Figshare upload. This must be a higher version than previously published versions.')
parser.add_argument('--previous-summary', dest='previous_summary', default=None,
help='validation_summary.json(.gz) from an earlier build or release; --validate then '
'warns on large changes in samples, drugs, pairs, omics coverage and model types.')
parser.add_argument('--skip-build-checks', dest='skip_build_checks', default=False, action='store_true',
help='Skip validate_build.py data checks during --validate (not recommended).')
parser.add_argument('--github-username', type=str, required=False, help='GitHub username for the repository.')
parser.add_argument('--github-email', type=str, required=False, help='GitHub email for the repository.')
parser.add_argument('--depmap-ready', dest='depmap_ready', default=False, action='store_true',
Expand Down Expand Up @@ -1019,10 +1024,45 @@ def get_latest_commit_hash(owner, repo, branch='main'):
drug_mapping_command_2 = ['python3', 'coderbuild/utils/align_drug_descriptors.py', '--local_dir', "/tmp"] + version_args
run_docker_upload_cmd(drug_mapping_command_2, 'all_files_dir', 'Align_Drug_Descriptors', args.version)

# Data checks the schema cannot express (duplicate results, orphan or
# overlapping ids, copied responses, large changes from the previous
# build). The summary is written into all_files_dir so it is published
# with the release and can be the next build's --previous-summary.
# Errors stop the build after the schema check has also reported.
build_check_failed = False
if not args.skip_build_checks:
summary_path = os.path.join(all_files_dir, 'validation_summary.json')
check_cmd = [sys.executable, 'coderbuild/utils/validate_build.py', '--dir', all_files_dir,
'--write-summary', summary_path]
if args.previous_summary:
check_cmd += ['--previous-summary', args.previous_summary]
_log('Running build data checks')
# The summary is written once every check has run, so its absence
# means the checker never finished.
if os.path.exists(summary_path):
os.remove(summary_path)
sys.stdout.flush()
# stdin can be closed when the build runs under nohup ... &, and a
# new Python then dies at startup (init_sys_streams: Bad file descriptor).
rc = subprocess.run(check_cmd, stdin=subprocess.DEVNULL).returncode
build_check_failed = rc != 0
if rc == 0:
_log('Build data checks passed')
elif os.path.exists(summary_path):
_log(f'Build data checks found errors (exit code {rc}); see the results above')
else:
# The 2.4 rebuild stopped here with no output.
_log(f'Build data checks did not finish (exit code {rc}). '
f'Rerun by hand: {" ".join(check_cmd)}')

# Run schema checker - This will always run if uploading data.
schema_check_command = ['python3', 'coderbuild/utils/check_schema.py', '--datasets'] + datasets
run_docker_upload_cmd(schema_check_command, 'all_files_dir', 'validate', args.version)


if build_check_failed:
raise RuntimeError('Build data checks failed; see the log above. '
'Fix them, or rerun with --skip-build-checks to override.')

_log("Validation complete. Proceeding with file compression/decompression adjustments")

# Compress or decompress files based on specific conditions after checking
Expand Down
Loading