Skip to content

Dataset consistency #229

Closed as not planned
Closed as not planned

Description

@ymahlich

Downloading the data from figshare I realized that some of the dataset files are tab separated, some comma separated, some compressed, some uncompressed.

Is there a specific reason for that? I understand that everything is handled internally so as long as I am directly interacting with the data through coderdata objects, it might not be important but maybe we want to be consistent about this?

As far as I can tell, currently the "schema" behind what datatype is which format is as follows:

data type csv csv.gz tsv.gz
copy_number x
drug_descriptor x
drugs x
experiments x
mutations x
proteomics x
samples x
transcriptomics x

Activity

  1. sgosline commented on Oct 11, 2024

    @sgosline
    Member

    Yes! First off, pandas can read in whatever so it doesn't matter. I default to csv unless it has drug information, which requires tabs (because drug descriptors and names have commas and quotes in them). we could probably gzip the samples files for consistency.

  2. jjacobson95 commented on Oct 14, 2024

    @jjacobson95
    Collaborator

    I like that we can preview the samples files on figshare when not gzipped, but I don't have a strong opinion either way.

    This could very easily be modified in lines 321,323 of build_all.py during the validation step.

  3. moved this to Backlog in CoderDataon Oct 16, 2024
  4. sgosline commented on Nov 12, 2024

    @sgosline
    Member

    Duplicate of #246.

  5. moved this from Backlog to Done in CoderDataon Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions