Skip to content

fix(db): handle MySQL post-upgrade startup delay gracefully (re: #1137) - #1138

Open
riteshyadav33302111-cell wants to merge 1 commit into
shift-org:mainfrom
riteshyadav33302111-cell:fix/mysql-startup-healthcheck
Open

riteshyadav33302111-cell wants to merge 1 commit into
shift-org:mainfrom
riteshyadav33302111-cell:fix/mysql-startup-healthcheck

Conversation

@riteshyadav33302111-cell

Copy link
Copy Markdown

After a MySQL version update, the server runs upgrade/migration steps before it can fully accept connections. This caused a 502 error that required a manual ./shift down && ./shift up to clear.

Three-layer fix:

  1. docker-compose.yml: add a healthcheck on the db service using mysqladmin ping, with a 30s start_period and 10 retries (5s interval) to cover the extended init window after a version upgrade.

  2. docker-compose.yml: make the node service depend on db with condition: service_healthy, so Docker Compose holds node until MySQL is fully ready -- not just started.

  3. app/db.js: add waitForConnection() with exponential-backoff retry (up to ~63s across 6 attempts) as a defence-in-depth layer. If a timing edge-case slips past the healthcheck, the node process retries the connection itself instead of crashing and requiring a human to bounce the server.

  4. shift: raise MAX_TRIES from 5 to 20 so ./shift up waits long enough when a MySQL upgrade genuinely takes extra time before node can print its ready message.

…t-org#1137)

After a MySQL version update, the server runs upgrade/migration steps before
it can fully accept connections. This caused a 502 error that required a manual
./shift down && ./shift up to clear.

Three-layer fix:

1. docker-compose.yml: add a healthcheck on the db service using mysqladmin
   ping, with a 30s start_period and 10 retries (5s interval) to cover the
   extended init window after a version upgrade.

2. docker-compose.yml: make the node service depend on db with
   condition: service_healthy, so Docker Compose holds node until MySQL is
   fully ready -- not just started.

3. app/db.js: add waitForConnection() with exponential-backoff retry (up to
   ~63s across 6 attempts) as a defence-in-depth layer. If a timing edge-case
   slips past the healthcheck, the node process retries the connection itself
   instead of crashing and requiring a human to bounce the server.

4. shift: raise MAX_TRIES from 5 to 20 so ./shift up waits long enough when a
   MySQL upgrade genuinely takes extra time before node can print its ready
   message.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant