[AnikDey-exe]

Summary

Replaces the schedule scraper's HTML page-scraping with Sidearm's calendar JSON
API, and gives every game a stable event_id so it can be matched by identity
instead of by guessing from its fields.

The old scraper fetched one /schedule page per sport and parsed it with
BeautifulSoup, then identified a stored game by comparing sport, gender, date,
city, state and location. Those fields are not unique — two games on the same
day at the same venue are indistinguishable — and they are not stable, because
the page writes a state five different ways and puts an end date in the time
column for multi-day events. A missed match creates a duplicate rather than
updating the row.

The calendar feed gives each event a stable numeric id. Matching on it makes
duplicates structurally impossible for any game that has one.

What changed

• games_scraper.py — parse_calendar_data reads the feed a week at a
  time; calendar_event_to_game_data maps event fields to DB fields.
  process_game_data now looks a game up by event_id first and only falls
  back to field matching when there is no id. Box-score and recap fetching is
  folded into it, since the feed carries no play-by-play.
• event_id added to the model, the repository, the service, and as a
  sparse unique index. Sparse so existing rows without one are not rejected.
• normalize_state reduces states to a canonical two-letter form. The feed
  writes "N.Y.", "NY", "Conn." and "Ohio" interchangeably, and state is part of
  the fallback lookup key.
• update_game catches DuplicateKeyError and warns instead of raising, so
  one colliding row cannot abort a whole scrape cycle.
• Dead code removed — 184 lines. The HTML schedule parsers
  (parse_game_links, parse_schedule_location, parse_schedule_date_and_time,
  extract_season_years, infer_game_year, absolute_url), is_game_active,
  and 30 now-unreferenced CSS-selector constants.
• Four scripts under scripts/, all dry-run by default:
  backfill_event_id.py, dedupe_games.py, dry_run_scraper.py,
  snapshot_games.py.

game_details_scrape.py is unchanged and still required — it is what produces
box_score and score_breakdown.

Deploying this (CRITICAL WARNING: PLEASE READ THIS)

app.py calls scrape_schedules() at import time, and gunicorn imports with
no_scrape=False, so the new scraper runs as soon as the container starts.

scripts/backfill_event_id.py must run against the target database before
this code deploys to it. Without ids, every event falls through to field
matching, and the rows that miss on time or state formatting are created as
duplicates instead of updated.

Per environment, in this order:

1. mongodump the game collection.
2. python3 scripts/dedupe_games.py then --apply.
3. python3 scripts/backfill_event_id.py then --apply.
4. python3 scripts/dry_run_scraper.py — confirm updates dominate creates.
   Creates in the hundreds means the backfill did not take; stop.
5. Merge. master auto-deploys to dev; prod comes from release.

Running on dev first and watching a full cycle is worth it before prod.

Verification

• scripts/dry_run_scraper.py: 122 updates, 4 creates across the fetch window.
• scripts/snapshot_games.py <file> before a full run and --diff after:
  0 added, 0 changed, 0 removed, confirming the dead-code removal changed no
  behaviour.
• Backfill assigned 1,174 of 1,361 stored games an event_id, 0 duplicates.
• 0 violations of the unique index after deduping.

Wins

Stable identity, 7× fewer requests, duplicates structurally impossible for 86% of rows and 100% of new ones, no CSS-selector fragility, 184 lines of dead code gone.

Known limitations

• 187 games have no event_id. The backfill only assigns one where exactly
  one stored row matches; anything ambiguous is reported and skipped rather than
  guessed at. 100 of those have a time column holding a date or placeholder
  ("Sep 27 (Sun)", "All Day") that can never match the feed's real time, and
  most of the rest are multi-day meets the old scraper stored once and the feed
  lists per day. They continue to match on fields.
• State formatting is only canonical inside the fetch window (±2 weeks),
  since nothing rewrites rows outside it. 90% canonical within 14 days, 42%
  beyond. Rows heal as they come into range.
• Box scores exist for 7 of 35 sports — soccer, football, ice hockey, field
  hockey, lacrosse, baseball, softball. The rest log "Sport parser not found",
  which is pre-existing and expected.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 9ca67e3c-12be-45b8-9082-e04420db5162

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn