Repository navigation
BigTable: On read_row(), provide default to retrieve only most recent cell values #4468
Description
The current signature on the Bigtable Table read_row() method has a default of filter_=None:
def read_row(self, row_key, filter_=None):
...In cases where a cell value may have been updated multiple times, the default will be to return the full time series with timestamps for each value which can slow down read performance in a non-obvious way.
In the current Python API the cells() method on row_data (PartialRowData) makes a deep copy of the cells, which compounds the performance issue.
@property
def cells(self):
"""Property returning all the cells accumulated on this partial row.
:rtype: dict
:returns: Dictionary of the :class:`Cell` objects accumulated. This
dictionary has two-levels of keys (first for column families
and second for column names/qualifiers within a family). For
a given column, a list of :class:`Cell` objects is stored.
"""
return copy.deepcopy(self._cells)Consider:
- Making a default filter on
read_row()to retrieve only the most recent value of any cell unless the full or partial time series is requested. - Allowing a
ColumnFamilyto implicitly or explicitly limit cells to only one value (no timeseries). - Adding a
cell_value(column_family_id, column, index=0)method torow_data(PartialRowData) to allow more efficient retrieval of a single cell value.
Activity
dhermes commentedon Nov 28, 2017
Thanks for filing @zakons!
I don't think it's a good idea to have any default filter. However, I think we could provide some kind of easy to access filter, i.e. as a constant in the package
google.cloud.bigtable.MOST_RECENT_FILTERor maybe as a class constant inTableso you could write code likeI'm not quite clear on your
ColumnFamilysuggestion. The feature set of the actual backend API is all we can really expose. Is there a particular method where you'd like to see different behavior? (Most API calls that actually refer to a column family just use a string ID for the column family, not an actualColumnFamilyinstance.)Adding
cell_value()seems very easy to implement and not at all controversial to add to the API surface./cc @garye
sduskis commentedon Nov 28, 2017
I like the table.MOST_RECENT_FILTER approach. We can also add in table.KEY_ONLY_FILTER, which is useful for just doing things like row counts.
I'm not certain about
ColumnFamilyeither. Is it the same as 1.?+1
@dhermes, @garye is less work on the client-side these days. I am working with other developers across all languages for Cloud Bigtable.
zakons commentedon Nov 28, 2017
Thanks.
Appreciate the quick feedback.
[-]On BigTable read_row(), provide default to retrieve only most recent cell values[/-][+]BigTable: On read_row(), provide default to retrieve only most recent cell values[/+]zakons commentedon Dec 14, 2017
Note that the cell_value(column_family_id, column, index=0) method returns an immutable value, so the underlying Cell's, which are mutable, are not exposed to the user.
Also, add a cell_values(column_family_id, column) iterator or generator method to row_data (PartialRowData) which will provide an iteration over all the cell values, returning index, timestamp, value for each iteration - each of which are immutable. The first index in the iteration would be 0, or most recent value.
sduskis commentedon Dec 14, 2017
@dhermes, can we please discuss design issues for PR #4564 here?
dhermes commentedon Dec 14, 2017
Sure. Go ahead and discuss? I am 100% for dropping the
deepcopy, but I'd prefer to do it as an opt-in, so users can knowingly get mutate-able values.sduskis commentedon May 25, 2018
@zakons: can this be closed?