Skip to content

Db next - #1117

Draft
ionous wants to merge 56 commits into
shift-org:mainfrom
ionous:db-next
Draft

Db next#1117
ionous wants to merge 56 commits into
shift-org:mainfrom
ionous:db-next

Conversation

@ionous

@ionous ionous commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

. new table format and preliminary migration tool
. significant changes to manage event, delete event, and unit tests to accommodate new tables
. jumps the calendar api version from 3.63.0 -> 3.64.0. the manage event response no longer returns the full event data.

https://github.com/ionous/shift-docs/blob/5106a1cf0eee1b9cde87f5dc8448a7a55df52d7b/docs/TABLES.md

related:
. extra command line control over app and unit test startup ( ex. to select mysql or sqlite for local testing )
. unit test files and test support code now live separately ( tests vs testing ) -- to make test file filtering easier
. tests now uses a temp directory for images rather than the source control directory
. better handling of thrown errors ( used by manage and delete ) to help with tests when they fail;
. more throwable errors ( for manage and delete ) to simplify some code.
. removes various unneeded functions
. moves dev.env to root: this makes more sense to me
. moves test images to their own directory ( separate from where images get uploaded )

ionous added 16 commits October 11, 2026 12:37
. new table format and preliminary migration tool
. significant changes to manage event, delete event, and unit tests to accommodate new tables
. jumps the calendar api version from 3.63.0 -> 3.64.0. the manage event response no longer returns the full event data.

related:
. extra command line control over app and unit test startup ( ex. to select mysql or sqlite for local testing )
. unit test files and test support code now live separately ( tests vs testing ) -- to make test file filtering easier
. tests now uses a temp directory for images rather than the source control directory
. better handling of thrown errors ( used by manage and delete ) to help with tests when they fail;
. more throwable errors ( for manage and delete ) to simplify some code.
. removes various unneeded functions
. moves dev.env to root: this makes more sense to me
. moves test images to their own directory ( separate from where images get uploaded )
includes the new tables, and support files; but doesn't change any of the existing code: purely additive
1. fix order of public schedule view  -- sqlite was okay with it; mysql needed the order reversed
2. change view_combined_events to a regular join to avoid generating null pkids for events with no days
1. my development branch is using "shift root" but the migration tool and narrow db change needs to use "app root" ( based on the location of config.js ) to work like it used to
2. for easier testing with mysql in docker, add the ability to test against a fixed port
1. dont make all days today
2. clone days from the event data so munging a day doesn't affect all of the shared data
3. improve logged output for skipped days and invalid images
required giving the two methods both called getJSON separate names
- getStatusesByEventId was 'view_event_schedule' which only includes published days: it needed to use `view_all_days` to get at data for events which haven't yet been published.
- updated those view names to be a little bit more self-documenting.
- tweaked some of the tests to avoid creating data when the test didn't need to.
- improved some comments
@ionous

ionous commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author

still remaining before changing the draft into a regular pr:

  • double check whether "delisted" days are needed ( bikefun maybe? )
  • additional manual verification of new and old rides. ( ex. maybe grab json for a span of time and compare the returned values )

updates and fixes since the initial draft pr was created:

  • the original code would autogenerate tinytitle and store it if necessary. the v2 tables store null if the tinytitle isn't specified. for the editing endpoints, the client will generate a html placeholder for the tinytitle input; for the other endpoints, the server generates and return the tinytitle to match the previous returned values.
  • double check image adding: the db changes included moving app/eventimages to app/images to simplify some things about the test setup. i've added docs about the (preexisting) image setup, and the rationale for the move of the test/dev files.
  • merge mainline into branch: slightly complicated due to new error codes for endpoints. took the opportunity to review and improve the calendar api doc.
  • days weren't reported correctly for unpublished (new) events
  • length of route during should be optional, and it wasn't.
  • endtime was reporting in iso format instead of 24-hour time ( due to tweaks needed for the join'd views )
  • verify migrated data has a decent creation/modified time
  • verify data modification times work okay ( and that created times don't change )

ionous added 24 commits October 26, 2026 19:11
in events.js:
- correctly handles if both id and range are specified
- throws errors when possible because it simplifies the code.
was reporting iso date/time. regression introduced during the refactoring needed to support join'd views
app/eventimages was where php used to upload files, and it got reused image testing files. this... was confusing. and it meant new image files for testing couldn't be added without also updating gitignore.
all the dev and test files live in /app/images now. and the gitignore can exclude the app/eventimages directory entirely
in v1 data, a missing tinytitle was autogenerated and stored into the db
with v2 data, the tinytitle is null unless the organizer specifically provided one. the summary endpoint and the client code needs some adjustment to handle that.
- adds a 'revoked' column to cal_series to handle deleted rides
- when deleting rides, set revoked info and delist all days
- updates queries to handle delisted and deleted rides
- restore 'all' query params to events and ics endpoints
- updates migration to include deleted and delisted rides
sqlite is devonly, faker and proxy should be included by app
root shouldn't have any direct dependencies
i don't think anything was using "express-http-proxy
the hide/print flags were always false on the viewing endpoints  ( non-editing ) because the view didn't include those columns.
the actual data was always null because the code was asking for their v2 column names, not the legacy names ( ex. it was asking for ".private_email" which exists in the db, but the view renamed it to ".email", exactly so the code didn't have to change. )
1. the web info views weren't producing hideweb, printweb like they should have.
2. the field name changes for editing needed to be updated since the ones for viewing did ( ex. private_email is just email in the view )
@ionous

ionous commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

doing manual comparisons of the events endpoint (ex. /api/events.php?startdate=2026-10-19&enddate=2026-10-21 ) i found and fixed some issues.

  1. the email, contact, phone info wasn't being returned even when the organizer specifically requested it to.
  2. the loopride flag wasn't being properly migrated
  3. tinytitle wasn't being trimmed when autogenerated.

my migration script replaces windows style newlines (\r\n) to unix newlines (\n). i think that's okay?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant