Skip to content

MoleculeNet datasets - #48

Open
aditya0by0 wants to merge 69 commits into
devfrom
feature/molenet_dataset
Open

aditya0by0 wants to merge 69 commits into
devfrom
feature/molenet_dataset

Conversation

@aditya0by0

@aditya0by0 aditya0by0 commented Aug 8, 2026 •

Copy link
Copy Markdown
Member

Also merge: sfluegel05/HiMol#3

@aditya0by0

Copy link
Copy Markdown
Member Author

How Missing Labels Are Handled in HiMol Finetuning

HiMol uses a three-valued label encoding for classification: +1 (positive/active), -1 (negative/inactive), and 0 (missing/unmeasured). The 0 sentinel is established during data loading in finetune/loader.py — DeepChem weights of 0 and NaN values are both mapped to 0, while real 0/1 labels are converted to -1/+1.

Loss Computation

In finetune/finetune.py (train() and eval() functions), the key mechanism is:

is_valid = y**2 > 0                          # 0²=0 → invalid; (±1)²=1 → valid
loss_mat = criterion(pred.double(), (y+1)/2)  # {-1,+1} → {0,1} for BCE
loss_mat = torch.where(is_valid, loss_mat, torch.zeros(...))  # zero out missing
loss = torch.sum(loss_mat) / torch.sum(is_valid)              # normalize by valid count

Missing labels are zeroed out in the loss matrix and excluded from the normalization denominator (torch.sum(is_valid)), so they contribute nothing to the gradient. The same pattern appears in finetune/optimization.py.

Metrics Computation

  • ROC-AUC (per-task): Only computed for tasks with both positive and negative examples. Within each task, the is_valid mask filters out missing entries before passing to roc_auc_score. Tasks that can't be computed are skipped, and a "Some target is missing!" warning with the missing ratio is printed.

  • Macro F1 (finetune/metrics.py): Classes with zero positive labels are excluded from the macro average via a mask. Classes with positive labels but no positive predictions get NaN precision, which is converted to 0.

  • Micro F1: Uses torchmetrics' MultilabelF1Score (includes all labels in averaging).

  • Regression: No missing label handling — MSE/MAE/RMSE computed directly on all predictions.

Note: The eval() function has a bug where loss is computed using only the last batch's batch.y rather than the full concatenated dataset. The eval_chebi() function fixes this by using the full y_true array.

@aditya0by0

Copy link
Copy Markdown
Member Author

How Missing Labels Are Handled in HiMol Finetuning

HiMol uses a three-valued label encoding for classification: +1 (positive/active), -1 (negative/inactive), and 0 (missing/unmeasured). The 0 sentinel is established during data loading in finetune/loader.py — DeepChem weights of 0 and NaN values are both mapped to 0, while real 0/1 labels are converted to -1/+1.

Loss Computation

In finetune/finetune.py (train() and eval() functions), the key mechanism is:

is_valid = y**2 > 0                          # 0²=0 → invalid; (±1)²=1 → valid
loss_mat = criterion(pred.double(), (y+1)/2)  # {-1,+1} → {0,1} for BCE
loss_mat = torch.where(is_valid, loss_mat, torch.zeros(...))  # zero out missing
loss = torch.sum(loss_mat) / torch.sum(is_valid)              # normalize by valid count

Missing labels are zeroed out in the loss matrix and excluded from the normalization denominator (torch.sum(is_valid)), so they contribute nothing to the gradient. The same pattern appears in finetune/optimization.py.

Metrics Computation

  • ROC-AUC (per-task): Only computed for tasks with both positive and negative examples. Within each task, the is_valid mask filters out missing entries before passing to roc_auc_score. Tasks that can't be computed are skipped, and a "Some target is missing!" warning with the missing ratio is printed.
  • Macro F1 (finetune/metrics.py): Classes with zero positive labels are excluded from the macro average via a mask. Classes with positive labels but no positive predictions get NaN precision, which is converted to 0.
  • Micro F1: Uses torchmetrics' MultilabelF1Score (includes all labels in averaging).
  • Regression: No missing label handling — MSE/MAE/RMSE computed directly on all predictions.

Note: The eval() function has a bug where loss is computed using only the last batch's batch.y rather than the full concatenated dataset. The eval_chebi() function fixes this by using the full y_true array.

fixed in ChEB-AI/python-chebai#174

@aditya0by0 aditya0by0 mentioned this pull request Oct 27, 2026
@sfluegel05

Copy link
Copy Markdown
Contributor

I see some cleanup potential here: Some of the files are specific to the JCI paper and not relevant to a user that just wants to run a GNN model. This includes

  • most of the config files - ideally, we have a maximum of 2 config files per dataset (augmented / not augmented). Everything else is just confusing. Even I would have to check the paper to find the one I am supposed to be using
  • the .sh evaluation scripts
  • results/combine_json_file.py
  • preprocessing/bin/* (I am not sure why we have different tokens lists for different datasets. Long-term, we should move to a fixed-set token system similar to the new ELECTRA reader where possible)
  • scripts/*

Most of these files would actually be useful for reproducing the experiments, so I am not suggesting to delete them. Instead, we should put them on zenodo alongside the checkpoints / datasets as supplementary material. My point is mainly about removing them from git so we have a cleaner repository for future projects.

@sfluegel05 sfluegel05 changed the title Molenet dataset MoleculeNet datasets Nov 9, 2026
@aditya0by0

aditya0by0 commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author
  • preprocessing/bin/* (I am not sure why we have different tokens lists for different datasets. Long-term, we should move to a fixed-set token system similar to the new ELECTRA reader where possible)

Context: The dataset-specific token directories were originally introduced intentionally because we wanted to support different sets of tokens for different datasets (e.g., dataset-specific functional group tokens). See PR: #47

However, this design introduced a limitation when it came to transferring knowledge from pretrained models to downstream tasks. In particular, it prevented us from directly fine-tuning a model pretrained on ZINC on MoleculeNet benchmarks for the following reasons:

  1. Different vocabulary sizes: Each dataset could have a different number of tokens, resulting in different input dimensions for the model's node and edge properties. Consequently, the pretrained model's architecture might not be compatible with the downstream dataset.
  2. Inconsistent token indexing: Even if two datasets happened to have the same number of tokens, the corresponding tokens could be assigned different indices. This would lead to inconsistencies in the node and edge representations, even though the input dimensions were identical.

To address these issues, in I introduced a common _shared token directory in this PR, which provides a fixed and consistent set of tokens across datasets, following an approach similar to the new ELECTRA reader.

The idea is to support two distinct workflows:

  • Pretraining and subsequent fine-tuning: When pretraining a model that will later be fine-tuned on other datasets, we use the common token vocabulary by explicitly setting --data.init_args.tokens_directory=_shared. This ensures that the same token set, with consistent indexing, is used during both pretraining and fine-tuning, making the model's learned representations transferable across datasets.

  • Dataset-specific training from scratch: When training or fine-tuning a model directly on a particular dataset without relying on a pretrained model from another dataset, we can continue using the default dataset-specific token directory. In this case, the vocabulary is constructed based on the tokens available in the training portion of that dataset, allowing us to retain support for dataset-specific tokens. This could be ideal, as we might not want to expose new tokens available from the test and validation part of the dataset. In this scenario if we don't follow this data-set specific tokens, it could be associated with exposing some aspect of test data during training.

Therefore, having separate token directories is intentional rather than redundant. The _shared directory addresses the need for a consistent vocabulary across datasets, particularly for transfer learning, while dataset-specific directories preserve the flexibility to train models using dataset-specific vocabularies.

For this reason, I would suggest retaining the current structure rather than moving everything to a single fixed vocabulary.

@aditya0by0

Copy link
Copy Markdown
Member Author

I see some cleanup potential here: Some of the files are specific to the JCI paper and not relevant to a user that just wants to run a GNN model.

I agree that there is room for cleanup here. However, I would suggest addressing the removal or reorganization of JCI-paper-specific files in a separate PR, as the scope of this PR is specifically focused on the MoleculeNet dataset changes. This would help keep the current PR focused and make the changes easier to review.

  • most of the config files - ideally, we have a maximum of 2 config files per dataset (augmented / not augmented). Everything else is just confusing. Even I would have to check the paper to find the one I am supposed to be using

Similarly, I think we could address the cleanup and consolidation of configuration files in a separate PR.

  • the .sh evaluation scripts

To address the usability aspect, I have updated the README in this PR to provide clear instructions for reproducing our experiments ([README](https://github.com/ChEB-AI/python-chebai-graph/tree/feature/molenet_dataset)).

I have specifically included references to the corresponding shell scripts so that users can reproduce the experiments without having to navigate through the configuration files or understand the various PyTorch Lightning-specific command-line arguments and hyperparameters.

Given that we currently have only five such shell scripts, I think retaining them is reasonable. Moreover, documenting their usage in the README makes the reproduction process more straightforward and user-friendly.

  • results/combine_json_file.py.

I see this as a standalone helper script which lives outside of chebai_graph package. However, I don't think its presence or removal has a significant impact on the current PR, so I am fine with either keeping it for now or removing it as part of a future cleanup.

@aditya0by0
aditya0by0 requested a review from sfluegel05 November 9, 2026 14:54
@sfluegel05

Copy link
Copy Markdown
Contributor

For this reason, I would suggest retaining the current structure rather than moving everything to a single fixed vocabulary.

I agree that we should keep the structure as it is for now.

I have specifically included references to the corresponding shell scripts so that users can reproduce the experiments without having to navigate through the configuration files or understand the various PyTorch Lightning-specific command-line arguments and hyperparameters.

The scripts and README are indeed useful documentation for the experiments. However, if we merge this now (and move consolidation to another PR), this would put the paper-specific code on the dev branch. I think removing the scripts / config files now would help me with reviewing this PR.

One solution I can suggest: We could tag the current version (including the paper-specific instructions) and refer to that version in the paper. This would allow to guide readers directly to the scripts, but keep it out of the regular versioning.

@aditya0by0

aditya0by0 commented Nov 12, 2026 •

Copy link
Copy Markdown
Member Author

@sfluegel05, I have made the suggested changes. Please review.

  • most of the config files - ideally, we have a maximum of 2 config files per dataset (augmented / not augmented). Everything else is just confusing. Even I would have to check the paper to find the one I am supposed to be using

removed, only atmost 2 config files per dataset retained, as suggested

  • the .sh evaluation scripts

removed

  • results/combine_json_file.py

removed

  • scripts/*

removed

@aditya0by0

Copy link
Copy Markdown
Member Author
  • preprocessing/bin/* (I am not sure why we have different tokens lists for different datasets. Long-term, we should move to a fixed-set token system similar to the new ELECTRA reader where possible)
  • Dataset-specific training from scratch: When training or fine-tuning a model directly on a particular dataset without relying on a pretrained model from another dataset, we can continue using the default dataset-specific token directory. In this case, the vocabulary is constructed based on the tokens available in the training portion of that dataset, allowing us to retain support for dataset-specific tokens. This could be ideal, as we might not want to expose new tokens available from the test and validation part of the dataset. In this scenario if we don't follow this data-set specific tokens, it could be associated with exposing some aspect of test data during training.

@sfluegel05, I gave this another thought, and I agree with your point. Ideally, we should maintain a single, shared token directory across all datasets rather than having separate token lists for each dataset.

I don't think using a shared vocabulary introduces any data leakage or exposes information from the test or validation sets. The atom properties we use are fundamental and independent of any particular dataset. For example, we already know that atomic numbers range from 1 to 119. Even if a particular dataset contains only a small subset of these elements, that should not prevent us from using the full vocabulary rather than constructing a vocabulary exclusively from the training portion of that dataset. Similarly for functional groups, we already know from our FG detection algorithm that we can detect around 101 non-ring FG groups, and ring-related FG are fundamental as discussed earlier.

Moreover, I don't expect using a full, shared vocabulary instead of a vocabulary generated from the training split to have any significant impact on the model's input representation or computational requirements. In both cases, the input property vectors are zero-padded to match the size of the largest tensor (203 in our case). Therefore, using the full vocabulary would primarily provide consistency across datasets without introducing any meaningful additional overhead.

Hence, I will remove the tokens related to other datasets while retaining only _shared token directory and setting the --data.init_args.tokens_directory=_shared as default

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants