Skip to content

MoleculeNet Classification Data : Modularization, dynamic splits, Clean Code - #174

Open
aditya0by0 wants to merge 86 commits into
devfrom
fix/molecule_net_dynamic_split
Open

aditya0by0 wants to merge 86 commits into
devfrom
fix/molecule_net_dynamic_split

Conversation

@aditya0by0

@aditya0by0 aditya0by0 commented Jul 17, 2026 •

Copy link
Copy Markdown
Member

Changes related to Molecule Net classification introduced in #130

General Changes

  • Check before dataloader to assert if the data is empty (as pytorch or lightning doesn't check this)

  • Generalize _DynamicDataset class to all datasets by replacing _graph_to_raw_dataset to _preprocess_data_into_dataframe

  • Modularize splitting logic into separate classes (so we dont need to specifiy the split logic for every new data class, rather just inherent the required splitting logic class )

Suggested Metric as per data type by https://arxiv.org/abs/1703.00564 for Molecule Net dataset

Dataset Task Type Classes Recommended Metric Why
BACE Binary classification 2 ROC-AUC Standard benchmark metric; moderate class imbalance
BBBP Binary classification 2 ROC-AUC MoleculeNet standard; threshold-independent
HIV Binary classification 2 ROC-AUC Highly imbalanced dataset
MUV Multi-task binary (17 tasks) 2 PR-AUC (Average Precision) Extremely imbalanced; ROC-AUC can be misleading
PCBA Multi-task binary (128 tasks) 2 PR-AUC (Average Precision) Very sparse positives; PR-AUC is preferred
SIDER Multi-task binary (27 tasks) 2 ROC-AUC Standard MoleculeNet evaluation
ClinTox Multi-task binary (2 tasks) 2 ROC-AUC Standard benchmark metric
Tox21 Multi-task binary (12 tasks) 2 ROC-AUC Widely used benchmark metric
ToxCast Multi-task binary (hundreds of tasks) 2 ROC-AUC (sometimes PR-AUC as secondary) Standard MoleculeNet metric

@aditya0by0 aditya0by0 changed the title Dynamic Splits for MoleculeNet Classification datasets MoleculeNet Classification Data : Modularization, dynamic splits, Clean Code Jul 19, 2026
@aditya0by0
aditya0by0 requested review from schnamo and sfluegel05 July 19, 2026 12:22
@aditya0by0

Copy link
Copy Markdown
Member Author

@sfluegel05, once I complete the gnn experiments with molecule net dataset, then we could merge the PR. Until then we could keep this PR in the draft state.

* Rename 'val' to 'validation' in data splits

* Update chebi-utils version requirement in pyproject.toml
@aditya0by0
aditya0by0 marked this pull request as ready for review October 27, 2026 07:05
@sfluegel05

Copy link
Copy Markdown
Collaborator

One change introduced in this PR that is worth pointing out: The model checkpoint callbacks changed from saving every 25 epochs and the top 3 models (micro-f1) to only saving the top model (macro-f1). Since we typically optimize for macro-f1, this makes sense, but should get documented for the next release

Comment thread chebai/callbacks/epoch_metrics.py Resolved
Comment thread pyproject.toml Resolved
Comment thread chebai/preprocessing/collate.py Resolved
@sfluegel05

sfluegel05 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

I implemented some minor (mostly cosmetic) changes. The only behaviour-changing commits are b05d32a where I extended the BCE loss for other reductions than mean and d97d230 where I retrieve the splits instead of raising an error (otherwise, a user would have to first build the dataset, then run the learn command with the right file path. now, they can run learn directly as usual).

@aditya0by0

aditya0by0 commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

Unit test failure reason:

The Install dependencies step failed before tests ran. On Python 3.12, pip could not install the pinned dependency: “ERROR: No matching distribution found for deepchem==2.8.0”, causing exit code 1.

This is a dependency/Python-version compatibility issue, not flaky infrastructure. Likely fix: use a Python version supported by deepchem==2.8.0 or update the DeepChem requirement.

@sfluegel05, Could you please provide a suggestion regarding this.

@aditya0by0
aditya0by0 requested a review from sfluegel05 November 9, 2026 13:55
@sfluegel05

Copy link
Copy Markdown
Collaborator

yeah, deepchem is a bit behind regarding releases. I am working with the latest pre-release which supports Python 3.12 https://pypi.org/project/deepchem/2.8.1.dev20260821122258/.

Based on the w&b logs, you used Python 3.10 and deepchem 2.8.0 in your experiments. So I would suggest pinning deepchem to >= 2.8.0 (with a comment that it only supports Python <3.12) and mention in the paper that we ran the experiments with Python 3.10 and deepchem 2.8.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update model checkpointing metric and periodic checkpointing Refactor: Generalize dataset base classes & consolidate dynamic splitting logic

3 participants