Client.insert_rows_json(): add option to disable best-effort deduplication

Currently, the Client.insert_rows_json() method for streaming inserts always inserts an insertId unique identifier for each row provided.
This row identifier can be user-provided; if the user doesn't provide any identifiers, the library automatically fills the row IDs by using UUID4.

Here's the code:

        for index, row in enumerate(json_rows):
            info = {"json": row}
            if row_ids is not None:
                info["insertId"] = row_ids[index]
            else:
                info["insertId"] = str(uuid.uuid4())
            rows_info.append(info)

However, insert IDs are entirely optional, and there are actually valid use cases not to use them. From the BigQuery documentation:

You can disable best effort de-duplication by not populating the insertId field for each row inserted. When you do not populate insertId, you get higher streaming ingest quotas in certain regions. This is the recommended way to get higher streaming ingest quota limits.

The BigQuery Python client library provides no way of omitting the insertIds. it would be nice to have a parameter for that.

I'll double check why this is the case, but at a glance this request sounds reasonable. 👍

I recall that you can explicitly pass in a list of None values to omit these.

It might be useful to have a more discoverable way to do this, though.

I recall that you can explicitly pass in a list of None values to omit these.

It might be useful to have a more discoverable way to do this, though.

It seems to work. However I don't see the point in allocating memory just to generate a list of Nones, maybe it can be improved? :-)

A memory efficient alternative would be to provide a fake sequence-ish object that returns None for every index:

class NoneItems:
    def __getitem__(self, index):
        return None

>>> insert_ids = NoneItems()
>>> assert insert_ids[0] is None
>>> assert insert_ids[1] is None
>>> assert insert_ids[42] is None

A bit hacky, but should be a good enough workaround until more user friendly support is added. :)

Update: This was confirmed, we'll add support for this in a more user-friendly way.

Amazing, thanks!

Navigation Menu

Repository navigation

`Client.insert_rows_json()`: add option to disable best-effort deduplication #720

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Client.insert_rows_json(): add option to disable best-effort deduplication #720

Description

Activity

plamut commented on Jun 28, 2021

tswast commented on Jun 28, 2021

pietrodn commented on Jun 29, 2021

plamut commented on Jun 29, 2021

plamut commented on Jun 29, 2021

pietrodn commented on Jul 1, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`Client.insert_rows_json()`: add option to disable best-effort deduplication #720