load_table_from_dataframe -- support alternative serialization formats #383

Activity

1. product-auto-label

   added

   api: bigqueryIssues related to the googleapis/python-bigquery API.Issues related to the googleapis/python-bigquery API.

   on Nov 11, 2020
2. tswast

   self-assigned this

   on Nov 11, 2020
3. tswast

   added

   type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.

   on Nov 11, 2020
4. tswast

   assigned

   cguardia

   and unassigned

   tswast

   on Nov 17, 2020
5. tswast commented on Nov 17, 2020

   

   tswast

   on Nov 17, 2020

   ContributorAuthor

   More actions

   To go from DataFrame to CSV, we can use the same tempfile logic as we have for parquet, but instead use to_csv + a few parameters to ensure proper data types:

   https://github.com/pydata/pandas-gbq/blob/ac2d2fe4ac0025109f8df3723e3f03a337face94/pandas_gbq/load.py#L16-L30
6. tswast commented on Nov 17, 2020

   

   tswast

   on Nov 17, 2020

   ContributorAuthor

   More actions

   This would also require adding CSV to the allowed serialization formats here:

   python-bigquery/google/cloud/bigquery/client.py

   Lines 2201 to 2206 in 809e4a2

   	if job_config.source_format != job.SourceFormat.PARQUET:
   	raise ValueError(
   	"Got unexpected source_format: '{}'. Currently, only PARQUET is supported".format(
   	job_config.source_format
   	)
   	)
7. cguardia commented on Nov 24, 2020

   

   cguardia

   on Nov 24, 2020

   Contributor

   More actions

   @tswast I have the initial work for this in #399. Looking for comments about which are the best parameters to use in the to_csv call, and whether this is going in the right direction.
8. tswast

   closed this as completedin #399on Dec 7, 2020