Repository navigation
to_dataframe dtypes argument could allow functions that take pandas.Series-like object and return new series #807
Description
Is your feature request related to a problem? Please describe.
Some timestamp-related methods don't quite work with the current string/dtype object logic. For example, I'm having a hard time getting TIME data types to parse as the timedelta64[ns] dtype.
I also think this feature would be useful for extension dtypes like Fletcher (https://github.com/xhochy/fletcher) and GeoPandas (https://geopandas.org/)
Describe the solution you'd like
Look at the dtypes dictionary. If any value is a function / callable, save them for later. Create the initial dataframe using the other dtypes, then follow-up and call the desired transformation functions.
Describe alternatives you've considered
Keep dtypes a simple pass-through to pandas and either:
- Let the developer call their own transformation functions.
- Have second argument for transformations
Additional context
Activity
tswast commentedon Jul 23, 2021
On second-thought, I think keeping
dtypessimple is more important. We can have some custom transformations for default dtypes without this feature.tswast commentedon Jul 23, 2021
On third-thought, for
flectchersomething like this could be useful, but the transformation function should take the arrow buffer(s) and return the pandas series. Definitely shouldn't usedtypesfor this, though.