Repository navigation
BigQuery: Allow partial schema in load_table_from_dataframe #8140
Description
As discussed in #8105 (comment) and implemented in pandas-gbq at googleapis/python-bigquery-pandas#218, often times the default schema detection works well enough for pandas -> BQ, and only one or two of many columns need to be overridden.
Example:
df = pandas.DataFrame({... lots of columns ..., "needs_manual_schema": [...]})
job_config = bigquery.LoadJobConfig(schema=[
bigquery.SchemaField("needs_manual_schema", "BOOLEAN"),
])
client.load_table_from_dataframe(
df, "my_dataset.my_table", job_config=job_config,
)
# All columns from df are uploaded, but only df["needs_manual_schema"] is
# coerced into a specific type (BOOLEAN).
Activity
tswast commentedon Aug 16, 2019
Now that I think about this, #9044 may be considered a sub-issue of this. Partial schemas make the most sense if we can determine the rest of the schema automatically.