Repository navigation
Evaluate alternatives to pandas-datareader for downloading World Bank data #987
Description
Is your feature request related to a problem? Please describe.
The pandas-datareader package has not been updated since 2021-07-13 and is now breaking with Python 3.12 and newer versions due to the removal of distutils. This package is currently used in util.finance.py for downloading World Bank data and in related tests.
Describe the solution you'd like
Identify and migrate to an alternative for downloading World Bank data. The steps could include:
- Researching viable replacements for
pandas-datareader, such aswbdata(see here) or direct API usage via libraries likerequestsorpandas. - Refactoring
util.finance.pyto use the new solution. - Updating related tests to ensure compatibility with the new implementation.
Describe alternatives you've considered
- Continuing to use
pandas-datareaderby forking and maintaining a private version. However, this would increase the maintenance burden on the project. - Temporarily patching the current implementation to support Python 3.12 and beyond, though this is not a sustainable long-term solution.
For reference, a related issue in pandas-datareader highlights these problems: pydata/pandas-datareader#977
Reacted by Emanuel Schmid
Reacted by Emanuel Schmid
Activity
[-]Evaluate Alternatives to pandas-datareader for Downloading World Bank Data[/-][+]Evaluate alternatives to pandas-datareader for downloading World Bank data[/+]peanutfun commentedon Mar 6, 2025
Check if the data is JSON.
pandas.read_jsonsupports URLs, other read functions might as wellspjuhel commentedon Mar 17, 2025
Note: Update the litpop tutorial here:
peanutfun commentedon Mar 24, 2025
wbdata actually looks quite nice. It has a small set of dependencies and even provides a persistent cache for the downloaded data.
The World Bank data is queried via an API, so building our own solution via
requestscan be a bit cumbersome. I tried directly accessing the data throughpandas.read_jsonusing an API request URL, but that is not straightforward. So I think we should go withwbdata.spjuhel commentedon Mar 24, 2025
I see you are assigned, are you implementing this then? Happy to review it once done if that is the case :)
peanutfun commentedon Mar 26, 2025
Found a solution without using
wbdata, see #1033