Repository navigation
BigQuery: provide option to omit 'insertId' for 'insert_rows' and 'insert_rows_json'. #9539
Description
Streaming inserts to BigQuery can achieve 1GB per second but only when insertId is omitted. Currently both methods from the google-cloud-bigquery client that implement the BigQuery insertAll REST API method, insert_rows and insert_rows_json, automatically add insertId preventing anyone from making use of the higher throughput limits.
Proposing that an argument be added to one or both of these methods like "insertId=True" (defaulting to true). When false, uuid is not used to add insert IDs. Furthermore, documentation is updated to clearly articulate that omitting insertId can result in duplicate records upon retry of failed API calls according to this guide.
Activity
[-]bigquery client.py methods insert_rows and insert_rows_json don't provide option to omit insertId[/-][+]BigQuery: Client methods 'insert_rows' and 'insert_rows_json' don't provide option to omit 'insertId'[/+][-]BigQuery: Client methods 'insert_rows' and 'insert_rows_json' don't provide option to omit 'insertId'[/-][+]BigQuery: provide option to omit 'insertId' for 'insert_rows' and 'insert_rows_json'.[/+]tseaver commentedon Oct 25, 2019
@tswast Can you please comment?
tswast commentedon Oct 25, 2019
This is already supported for
insert_rows_json. You can explicitly pass in an iterable ofNones for therow_idsargument. Probably worth keeping this issue open to make a sample demonstrating that, though.tswast commentedon Oct 25, 2019
Oh, it also should work for
insert_rows, as it passes any additional keyword arguments toinsert_rows_json.tseaver commentedon Oct 25, 2019
@tswast That seems like an odd DX to have to document / defend. Instead, maybe we could special-case
row_ids=Noneto expand to that? E.g.:later
Ugh, backward compat would require changing the default value for
row_idsto some marker object, which would then expand to UUIDs as currently happens forNone. Or else we would need to define a marker object which signalled "expand to array of None".tswast commentedon Oct 26, 2019
It's only a handful of people who'll want to trade more duplicate rows to get to 1 GB of writes per second. I think the current default of populating row IDs with UUID when
row_id=Noneis the correct choice.We could potentially make a special marker value, but I don't see all that much benefit. At a minimum, we should do the following, though.
Nonevalues continues to work for theinsert_rows*()methods.insert_rows_jsonwithrow_ids=[None]*len(rows)for use on the page where we document how to achieve the higher 1GB / s write limit.