You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Looking at the performance graph for reading a large number (1000) of rows, each row having 10 cells in one column family, the validation of chunks takes a large percentage of time (>25%), as measured by cProfile. Areas for tuning are identified below.
Validation of timestamp_micros ordering
While a row is being read, a validation is made to make sure that cells for one qualifier are in timestamp_micros order (most recent first):
def _validate_chunk_row_in_progress(self, chunk):
...
previous = self._previous_cell
_raise_if(self._same_as_previous(chunk) and
chunk.timestamp_micros <= previous.timestamp_micros)
def _same_as_previous(self, chunk):
previous = self._previous_cell
return (chunk.row_key == previous.row_key and
chunk.family_name == previous.family_name and
chunk.qualifier == previous.qualifier and
chunk.labels == previous.labels)
This validation may be extraneous because:
A RowFilter may have been applied - CellsColumnLimitFilter(1) - which will limit the result to the cell with the most recent timestamp.
There may only be one cell value for the qualifier.
The BigTable server may ALWAYS be providing the cells in timestamp order.
Two possible tunings:
Remove the validation if the condition is unlikely to occur in the ReadRowResponse.
If the validation is kept, it could be performed only when qualifier cell lists with length > 1 are detected AFTER the chunks for the qualifier are consumed.
Activity