Repository navigation
GH-34235: [Python] Add join_asof binding
#34234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
078b9a9
2aa990b
dad0416
fd33b8c
09bdd04
27e5862
312378e
8303310
46a1d2a
d399267
367a17d
28ea072
0a517f3
984b159
f68d7b2
6f8db25
102f965
f645e65
d66a2e8
84e8284
acb082e
8a141fe
ae7e9e0
a280fda
cae3e92
7f7b877
5fcce5f
7dd4e21
e6da4df
0553f1d
df46f9e
6e87867
c2339b4
adbf06c
f30de79
88b3c38
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -880,6 +880,70 @@ cdef class Dataset(_Weakrefable): | |
| output_type=InMemoryDataset | ||
| ) | ||
|
|
||
| def join_asof(self, right_dataset, on, by, tolerance, right_on=None, right_by=None): | ||
| """ | ||
| Perform an asof join between this dataset and another one. | ||
|
|
||
| This is similar to a left-join except that we match on nearest key rather | ||
| than equal keys. Both datasets must be sorted by the key. This type of join | ||
| is most useful for time series data that are not perfectly aligned. | ||
|
|
||
| Optionally match on equivalent keys with "by" before searching with "on". | ||
|
|
||
| Result of the join will be a new Dataset, where further | ||
| operations can be applied. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| right_dataset : dataset | ||
| The dataset to join to the current one, acting as the right dataset | ||
| in the join operation. | ||
| on : str | ||
| The column from current dataset that should be used as the "on" key | ||
| of the join operation left side. | ||
|
|
||
| An inexact match is used on the "on" key, i.e. a row is considered a | ||
| match if and only if left_on - tolerance <= right_on <= left_on. | ||
|
|
||
| The input table must be sorted by the "on" key. Must be a single | ||
| field of a common type. | ||
|
|
||
| Currently, the "on" key must be an integer, date, or timestamp type. | ||
| by : str or list[str] | ||
| The columns from current dataset that should be used as the keys | ||
| of the join operation left side. The join operation is then done | ||
| only for the matches in these columns. | ||
| tolerance : int | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If the "on" key is a timestamp column, what value can be used here? (not an int?)
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a good question actually... I'm not 100% sure how this is intended to work. The C++ implementation exclusively accepts an |
||
| The tolerance for inexact "on" key matching. A right row is considered | ||
| a match with the left row `right.on - left.on <= tolerance`. The | ||
| `tolerance` may be: | ||
|
|
||
| - negative, in which case a past-as-of-join occurs; | ||
| - or positive, in which case a future-as-of-join occurs; | ||
| - or zero, in which case an exact-as-of-join occurs. | ||
|
|
||
| The tolerance is interpreted in the same units as the "on" key. | ||
| right_on : str or list[str], default None | ||
| The columns from the right_dataset that should be used as the on key | ||
| on the join operation right side. | ||
| When ``None`` use the same key name as the left dataset. | ||
| right_by : str or list[str], default None | ||
| The columns from the right_dataset that should be used as by keys | ||
| on the join operation right side. | ||
| When ``None`` use the same key names as the left dataset. | ||
|
|
||
| Returns | ||
| ------- | ||
| InMemoryDataset | ||
| """ | ||
| if right_on is None: | ||
| right_on = on | ||
| if right_by is None: | ||
| right_by = by | ||
| return _pac()._perform_join_asof(self, on, by, | ||
| right_dataset, right_on, right_by, | ||
| tolerance, output_type=InMemoryDataset) | ||
|
|
||
|
|
||
| cdef class InMemoryDataset(Dataset): | ||
| """ | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5333,6 +5333,92 @@ cdef class Table(_Tabular): | |
| output_type=Table | ||
| ) | ||
|
|
||
| def join_asof(self, right_table, on, by, tolerance, right_on=None, right_by=None): | ||
| """ | ||
| Perform an asof join between this table and another one. | ||
|
|
||
|
Comment on lines
+5338
to
+5339
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would still like to see a bit more expanded explanation (apart from the individual keyword explanations) about what and asof join exactly is. Something indicating it does 1) an inexact join, 2) on a sorted dataset, 3) potentially first joining on other attributes, and 4) typically useful for time series data that are not perfectly aligned. Are that the most relevant characteristics?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| This is similar to a left-join except that we match on nearest key rather | ||
| than equal keys. Both tables must be sorted by the key. This type of join | ||
| is most useful for time series data that are not perfectly aligned. | ||
|
|
||
| Optionally match on equivalent keys with "by" before searching with "on". | ||
|
|
||
| Result of the join will be a new Table, where further | ||
| operations can be applied. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| right_table : Table | ||
| The table to join to the current one, acting as the right table | ||
| in the join operation. | ||
| on : str | ||
| The column from current table that should be used as the "on" key | ||
| of the join operation left side. | ||
|
|
||
| An inexact match is used on the "on" key, i.e. a row is considered a | ||
| match if and only if left_on - tolerance <= right_on <= left_on. | ||
|
|
||
| The input dataset must be sorted by the "on" key. Must be a single | ||
| field of a common type. | ||
|
|
||
| Currently, the "on" key must be an integer, date, or timestamp type. | ||
| by : str or list[str] | ||
| The columns from current table that should be used as the keys | ||
| of the join operation left side. The join operation is then done | ||
| only for the matches in these columns. | ||
| tolerance : int | ||
| The tolerance for inexact "on" key matching. A right row is considered | ||
| a match with the left row ``right.on - left.on <= tolerance``. The | ||
| ``tolerance`` may be: | ||
|
|
||
| - negative, in which case a past-as-of-join occurs; | ||
| - or positive, in which case a future-as-of-join occurs; | ||
| - or zero, in which case an exact-as-of-join occurs. | ||
|
|
||
| The tolerance is interpreted in the same units as the "on" key. | ||
| right_on : str or list[str], default None | ||
| The columns from the right_table that should be used as the on key | ||
| on the join operation right side. | ||
| When ``None`` use the same key name as the left table. | ||
| right_by : str or list[str], default None | ||
| The columns from the right_table that should be used as keys | ||
| on the join operation right side. | ||
| When ``None`` use the same key names as the left table. | ||
|
|
||
| Returns | ||
| ------- | ||
| Table | ||
|
|
||
| Example | ||
| -------- | ||
| >>> import pyarrow as pa | ||
| >>> t1 = pa.table({'id': [1, 3, 2, 3, 3], | ||
| ... 'year': [2020, 2021, 2022, 2022, 2023]}) | ||
| >>> t2 = pa.table({'id': [3, 4], | ||
| ... 'year': [2020, 2021], | ||
| ... 'n_legs': [5, 100], | ||
| ... 'animal': ["Brittle stars", "Centipede"]}) | ||
|
|
||
| >>> t1.join_asof(t2, on='year', by='id', tolerance=-2) | ||
| pyarrow.Table | ||
| id: int64 | ||
| year: int64 | ||
| n_legs: int64 | ||
| animal: string | ||
| ---- | ||
| id: [[1,3,2,3,3]] | ||
| year: [[2020,2021,2022,2022,2023]] | ||
| n_legs: [[null,5,null,5,null]] | ||
| animal: [[null,"Brittle stars",null,"Brittle stars",null]] | ||
| """ | ||
| if right_on is None: | ||
| right_on = on | ||
| if right_by is None: | ||
| right_by = by | ||
| return _pac()._perform_join_asof(self, on, by, | ||
| right_table, right_on, right_by, | ||
| tolerance, output_type=Table) | ||
|
|
||
| def __arrow_c_stream__(self, requested_schema=None): | ||
| """ | ||
| Export the table as an Arrow C stream PyCapsule. | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.