Repository navigation
Conversation
The FRED API seems to have implemented a hard cap of 2000 vintages to be retrieved at once. It comes into play if one tries to run get_series_all_releases on any daily data series present in ALFRED (for example 'T1YFF'). Requests for data come back with a "Bad Request. This exceeds the maximum number of vintage dates allowed (2000)" error. The fix splits data retrieval into batches of <= 2000 vintages.
|
hi - I have PRd an additional commit into dgerbc branch. Is it possible for @dsgerbc to merge that, and then we merge this PR? as of now I forked this and pushed to our private PYPI but i would prefer the upstream to have it. |
|
Hi guys, The resulting "realtime_start" and "realtime_end" fields change depending on the "realtime_start" and "realtime_end" parameters. For example: from fredapi import Fred
fred = Fred(api_key='xyz')
df = fred.get_series_all_releases(series_id='GNPCA', realtime_start='1776-09-24', realtime_end='2001-03-27')
df[df['date'] == '1929-01-01']
The value "709.6" is duplicated in the resulting DataFrame, because it falls into two ranges: 1958-12-21 -> 1987-01-22 <observation realtime_start="1958-12-21" realtime_end="1965-08-18" date="1929-01-01" value="181.8"/>
<observation realtime_start="1965-08-19" realtime_end="1976-01-15" date="1929-01-01" value="203.6"/>
<observation realtime_start="1976-01-16" realtime_end="1980-12-22" date="1929-01-01" value="314.7"/>
<observation realtime_start="1980-12-23" realtime_end="1985-12-19" date="1929-01-01" value="315.7"/>
<observation realtime_start="1985-12-20" realtime_end="1991-12-03" date="1929-01-01" value="709.6"/>
<observation realtime_start="1991-12-04" realtime_end="1992-12-21" date="1929-01-01" value="."/>
<observation realtime_start="1992-12-22" realtime_end="1996-01-18" date="1929-01-01" value="827.4"/>
<observation realtime_start="1996-01-19" realtime_end="1997-04-29" date="1929-01-01" value="."/>
<observation realtime_start="1997-04-30" realtime_end="1999-10-28" date="1929-01-01" value="796.8"/>
<observation realtime_start="1999-10-29" realtime_end="2000-04-26" date="1929-01-01" value="."/>
<observation realtime_start="2000-04-27" realtime_end="2001-03-27" date="1929-01-01" value="828.9"/>Even if it would be possible to calculate realtime_end (shift Series) from realtime_start, it is not possible to remove realtime_start duplicates with certainty on the client side. That's why I think it's not possible to apply pagination to vintages data. FRED's proposal is wrong, when the vintages limit is exceeded, pagination should be enforced directly by FRED (limit/offset). In addition, this is an undocumented functionality. But FRED has more such problems... Tomas |
|
@TomasKoutek : thanks for the investigation! I'll also look into it soon. We, specifically, are only interested in the original release date, which - if I understand it correctly - can be still extracted for these series. Do you know if this statement is correct? |
jmalicki
added a commit
to EconGraph/econ-graph
that referenced
this pull request
Oct 25, 2026
…tage dates (ECO-498) (#478) The FRED crawler asked for a series' whole ALFRED history in one real-time window (`realtime_start=1776-07-04`, `realtime_end=9999-12-31`). FRED rejects a window that covers too many vintage dates with HTTP 400, `Bad Request. This exceeds the maximum number of vintage dates allowed (2000).` That was classified Permanent, so daily series with long revision histories (DFF ~5,151 vintage dates, DGS10 ~5,136, DGS1, EFFR ~2,673, T5YIFR ~3,151) never got data. Part of https://linear.app/econgraph/issue/ECO-498 (the live check on DFF/DGS10 and re-queueing the missing FRED series happen after deploy) ## The constraint The `vintage_dates` parameter of [fred/series/observations](https://fred.stlouisfed.org/docs/api/fred/series_observations.html) says: "Your query will be limited to a specific number of vintage dates depending on the series_id, file_type and output_type specified", with "any output_type or other series: 1000 csv, 1000 xlsx, 2000 json, 2000 xml". The adapter uses JSON with output_type 1, so the limit is 2,000. FRED applies it to a `realtime_start`..`realtime_end` window as well. The error text above comes from a report against the fredapi client ([mortada/fredapi#52](mortada/fredapi#52)). FRED's docs don't print the message, so the adapter matches only its stable part, "maximum number of vintage dates", on an HTTP 400. ## Request plan - Every fetch that reads observations first lists the vintage dates in its real-time window with [fred/series/vintagedates](https://fred.stlouisfed.org/docs/api/fred/series_vintagedates.html), paged at 10,000 dates per page. That is one extra request per fetch. A series whose `last_updated` hasn't changed still skips both calls. - It then requests `/series/observations` in consecutive real-time windows of at most `FRED_MAX_VINTAGE_DATES_PER_REQUEST` = 1,000 vintage dates. Each window is still paged with `limit`/`offset`. - **Why 1,000, half the documented limit:** - FRED's exact count isn't documented. Every window after the first starts the day before its first vintage, and that day can itself be a vintage date or be counted for the values clamped to it. - The listing leaves out release dates on which the series didn't change. - 1,000 is FRED's own limit for the other formats, so a JSON limit lowered to match them still isn't hit. - The cost: a series with at most 1,000 vintage dates is still exactly one observations request, the same request as before this change. DFF takes six windows instead of three. - **Why real-time windows instead of `vintage_dates=` batches:** windows keep output_type 1's real-time-period rows, the ones the adapter already stores. Each row's `realtime_start` is its revision date. This is the mechanism the incremental fetch already uses. A `vintage_dates=` list changes what the rows mean: values as of each listed date, not when each value started. The docs don't fully specify those rows, and they could not be mapped to the stored revisions without guessing. - **Fallback:** if FRED still answers a window with the vintage-limit 400 (for example because it lowered the limit), the adapter logs a warning, halves that window, and retries, down to a single vintage date. After that it fails Permanent as before. No other 400 is retried or split. "Series does not exist" is still NotFound. - Every request, including the listing, goes through the shared `HttpFetcher`. Windows run one after another under FRED's rate limiter, with the same User-Agent and in-process retries. ## Semantics preserved - **Each revision stored exactly once.** - The first window starts where the fetch does: 1776-07-04, or the day before the known vintage. - Each later window starts the day before its first vintage and ends the day before the next window's first vintage. The last one ends at 9999-12-31. - A row is kept only by the window whose range, from its first vintage through its end, holds the row's `realtime_start`. Those ranges partition the real-time axis, so every row is kept once, whether or not FRED clamps `realtime_start` at the window edge. - **Original releases unchanged.** At each window start, FRED also returns the values already in effect that day, clamped to it. They aren't kept, but they count when choosing each date's earliest row. So a date first published in an earlier window has no original release in a later one. `is_original_release` is part of the upsert key, and it comes out the same as with one request. - **Incremental fetch unchanged.** It still starts the day before the newest stored vintage, drops rows before that vintage, and re-reads the vintage itself. A long gap is windowed the same way. A short refresh is one listing plus one observations request. - **Same points for small series.** A series under the default gets the same single observations request and the same points as before. No migration, no schema change. ## Tests `src/sources/fred/vintage_tests.rs` runs against a fake ALFRED (wiremock). It holds a known daily revision history and answers `vintagedates` and `observations` the way FRED does: windows clamped, `limit`/`offset` paging, and the vintage-limit 400. Each test compares the fetched points with the points the whole history gives in a single request. - **Normal operation:** 2,500 vintage dates at FRED's real limit. Three windows, no 400, no request over 1,001 vintage dates. Revisions on the boundary vintages, revisions across windows, and a revision to a missing value all come back with the right original-release flags. - **Small series:** one observations request for the whole window. - **Gappy history:** weekly vintages, so the day before a window is never a vintage date. Every 5th vintage is left out of the `vintagedates` listing, and one row starts a decade before the first vintage. A full fetch and incremental fetches from v[0], v[700] (listed) and v[1599] (unlisted) match the single-request oracle at mock limits of 2000, 300 and 37. The last two force the fallback. - **Regression:** - The deployed request shape (one window, 5,151 vintage dates) gets the 400, and the follow-up windows collect the complete history. - With the mock's limit (300) below the default, rejected windows are halved until FRED accepts them. Each window is requested at most once, and the merged result is complete. - **Paging:** observation paging inside each window, and paging of the vintage-date listing. - **Incremental:** - A known vintage with 2,300 newer vintage dates is windowed from the day before it, with no points before it. - A short refresh is one observations request. - **Errors:** - An unrelated 400 (bad parameter) stays Permanent and isn't split. - "Series does not exist" on the listing is NotFound. - A window rejected at a single vintage fails after a bounded number of requests. - **Rate limiter:** at 50 requests/s with burst 1, the 19 requests of a windowed fetch take at least 18 intervals, both client-side and as they reach the mock. - **Unit tests:** - window splitting and boundaries (every day is kept by exactly one window, including a boundary that is itself a vintage date); - original-release choice when a row comes back from two windows; - recognising only the vintage-limit 400. Existing FRED tests, the contract tests, the worker's FRED vintage test, the soak upstream and the e2e seed manifest now also serve `/series/vintagedates` from new fixtures. `cargo test -p econ-graph-crawler --all-features -- --test-threads=1` passes against Postgres (511 tests), `seed-fixtures` loads both FRED e2e series, and fmt and the crawler clippy job are clean.
jmalicki
added a commit
to EconGraph/econ-graph
that referenced
this pull request
Oct 26, 2026
…tage dates (ECO-498) (#490) The FRED crawler asked for a series' whole ALFRED history in one real-time window (`realtime_start=1776-07-04`, `realtime_end=9999-12-31`). FRED rejects a window that covers too many vintage dates with HTTP 400, `Bad Request. This exceeds the maximum number of vintage dates allowed (2000).` That was classified Permanent, so daily series with long revision histories (DFF ~5,151 vintage dates, DGS10 ~5,136, DGS1, EFFR ~2,673, T5YIFR ~3,151) never got data. Part of https://linear.app/econgraph/issue/ECO-498 (the live check on DFF/DGS10 and re-queueing the missing FRED series happen after deploy) ## The constraint The `vintage_dates` parameter of [fred/series/observations](https://fred.stlouisfed.org/docs/api/fred/series_observations.html) says: "Your query will be limited to a specific number of vintage dates depending on the series_id, file_type and output_type specified", with "any output_type or other series: 1000 csv, 1000 xlsx, 2000 json, 2000 xml". The adapter uses JSON with output_type 1, so the limit is 2,000. FRED applies it to a `realtime_start`..`realtime_end` window as well. The error text above comes from a report against the fredapi client ([mortada/fredapi#52](mortada/fredapi#52)). FRED's docs don't print the message, so the adapter matches only its stable part, "maximum number of vintage dates", on an HTTP 400. ## Request plan - Every fetch that reads observations first lists the vintage dates in its real-time window with [fred/series/vintagedates](https://fred.stlouisfed.org/docs/api/fred/series_vintagedates.html), paged at 10,000 dates per page. That is one extra request per fetch. A series whose `last_updated` hasn't changed still skips both calls. - It then requests `/series/observations` in consecutive real-time windows of at most `FRED_MAX_VINTAGE_DATES_PER_REQUEST` = 1,000 vintage dates. Each window is still paged with `limit`/`offset`. - **Why 1,000, half the documented limit:** - FRED's exact count isn't documented. Every window after the first starts the day before its first vintage, and that day can itself be a vintage date or be counted for the values clamped to it. - The listing leaves out release dates on which the series didn't change. - 1,000 is FRED's own limit for the other formats, so a JSON limit lowered to match them still isn't hit. - The cost: a series with at most 1,000 vintage dates is still exactly one observations request, the same request as before this change. DFF takes six windows instead of three. - **Why real-time windows instead of `vintage_dates=` batches:** windows keep output_type 1's real-time-period rows, the ones the adapter already stores. Each row's `realtime_start` is its revision date. This is the mechanism the incremental fetch already uses. A `vintage_dates=` list changes what the rows mean: values as of each listed date, not when each value started. The docs don't fully specify those rows, and they could not be mapped to the stored revisions without guessing. - **Fallback:** if FRED still answers a window with the vintage-limit 400 (for example because it lowered the limit), the adapter logs a warning, halves that window, and retries, down to a single vintage date. After that it fails Permanent as before. No other 400 is retried or split. "Series does not exist" is still NotFound. - Every request, including the listing, goes through the shared `HttpFetcher`. Windows run one after another under FRED's rate limiter, with the same User-Agent and in-process retries. ## Semantics preserved - **Each revision stored exactly once.** - The first window starts where the fetch does: 1776-07-04, or the day before the known vintage. - Each later window starts the day before its first vintage and ends the day before the next window's first vintage. The last one ends at 9999-12-31. - A row is kept only by the window whose range, from its first vintage through its end, holds the row's `realtime_start`. Those ranges partition the real-time axis, so every row is kept once, whether or not FRED clamps `realtime_start` at the window edge. - **Original releases unchanged.** At each window start, FRED also returns the values already in effect that day, clamped to it. They aren't kept, but they count when choosing each date's earliest row. So a date first published in an earlier window has no original release in a later one. `is_original_release` is part of the upsert key, and it comes out the same as with one request. - **Incremental fetch unchanged.** It still starts the day before the newest stored vintage, drops rows before that vintage, and re-reads the vintage itself. A long gap is windowed the same way. A short refresh is one listing plus one observations request. - **Same points for small series.** A series under the default gets the same single observations request and the same points as before. No migration, no schema change. ## Tests `src/sources/fred/vintage_tests.rs` runs against a fake ALFRED (wiremock). It holds a known daily revision history and answers `vintagedates` and `observations` the way FRED does: windows clamped, `limit`/`offset` paging, and the vintage-limit 400. Each test compares the fetched points with the points the whole history gives in a single request. - **Normal operation:** 2,500 vintage dates at FRED's real limit. Three windows, no 400, no request over 1,001 vintage dates. Revisions on the boundary vintages, revisions across windows, and a revision to a missing value all come back with the right original-release flags. - **Small series:** one observations request for the whole window. - **Gappy history:** weekly vintages, so the day before a window is never a vintage date. Every 5th vintage is left out of the `vintagedates` listing, and one row starts a decade before the first vintage. A full fetch and incremental fetches from v[0], v[700] (listed) and v[1599] (unlisted) match the single-request oracle at mock limits of 2000, 300 and 37. The last two force the fallback. - **Regression:** - The deployed request shape (one window, 5,151 vintage dates) gets the 400, and the follow-up windows collect the complete history. - With the mock's limit (300) below the default, rejected windows are halved until FRED accepts them. Each window is requested at most once, and the merged result is complete. - **Paging:** observation paging inside each window, and paging of the vintage-date listing. - **Incremental:** - A known vintage with 2,300 newer vintage dates is windowed from the day before it, with no points before it. - A short refresh is one observations request. - **Errors:** - An unrelated 400 (bad parameter) stays Permanent and isn't split. - "Series does not exist" on the listing is NotFound. - A window rejected at a single vintage fails after a bounded number of requests. - **Rate limiter:** at 50 requests/s with burst 1, the 19 requests of a windowed fetch take at least 18 intervals, both client-side and as they reach the mock. - **Unit tests:** - window splitting and boundaries (every day is kept by exactly one window, including a boundary that is itself a vintage date); - original-release choice when a row comes back from two windows; - recognising only the vintage-limit 400. Existing FRED tests, the contract tests, the worker's FRED vintage test, the soak upstream and the e2e seed manifest now also serve `/series/vintagedates` from new fixtures. `cargo test -p econ-graph-crawler --all-features -- --test-threads=1` passes against Postgres (511 tests), `seed-fixtures` loads both FRED e2e series, and fmt and the crawler clippy job are clean. Forward-port of #478 from release/v4.0 (cherry-pick -x, applied cleanly). The FRED tests pass on main (52) and the workspace compiles.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The FRED API seems to have implemented a hard cap of 2000 vintages to be retrieved at once. It comes into play if one tries to run get_series_all_releases on any daily data series present in ALFRED (for example 'T1YFF'). Requests for data come back with a "Bad Request. This exceeds the maximum number of vintage dates allowed (2000)" error.
The fix splits data retrieval into batches of <= 2000 vintages.