Skip to content

Added lazy per-resource template config and structural skeleton cache key - refs #1481 - #1532

Draft
philayres wants to merge 15 commits into
consected:developfrom
hmsrc:handlebars-lazy-config-phase2-1481
Draft

philayres wants to merge 15 commits into
consected:developfrom
hmsrc:handlebars-lazy-config-phase2-1481

Conversation

@philayres

@philayres philayres commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1481 (Phase 2; does not close #1481 or #1480).

Stacked on #1527 (Phase 1). The first commit (d92f4b1a7) is Phase 1 and will drop out of this diff once #1527 merges. The remaining 14 commits are Phase 2 and are kept separate for review: the original change (5f1da02dc), refactors R1-R8 and review fixes, each described by its commit message. Squash on merge is fine. Phase 1 keeps generation_key stable across admin saves via Application.clear_cache_preserving_generation, and prewarm markers include Application.cache_epoch.

What this does

A config-only edit to a dynamic model, activity log or external identifier no longer rebuilds the masters skeleton fragment. The client is told the resource's config version has moved on and fetches only that resource's template_config.

  • Version tokens. A live definition carries template_version_token: the latest history-row id, or a composite integer (>= 1e12) when it references config libraries with use_current_version, so a library edit changes the token. def_version is unchanged and still nil for live definitions.
  • Emitters. State blobs are emitted under v<token> plus a v0 alias. The four hardcoded 'v' slots are now v0. Forms carry their own nonce-bearing inline state block. Shared base aliases (the ones list templates read) are promoted to v0 only from the definition's default_option_type_name option type, which set_defaults always generates for dynamic models and external identifiers. This keeps the list chrome independent of YAML order, and activity logs, which have no shared base, are unaffected.
  • Server. Live template_config responses are state-only (no re-emitted Handlebars tags). The external identifier endpoint now renders the shared set; it previously returned an empty body.
  • Client (_fpa.js). A fetch happens only when the v<n> slot is absent, sends one representative record per type, option type and version, and patches v<n> and v0. fpa_state_item falls back to v0 for live tokens.
  • Skeleton cache key. A new master_templates_skeleton_cache_key is a DB-derived set signature: active definitions, structural fingerprint (option types, references, embed), activity-log item_type/rec_type, page-layout panels, access timestamps including global rows, and the page-layout/app-config history token. A config-only edit does not move it. partial_cache_key is unchanged.
  • Freshness. Data-response ETags include vdef_version. Live tokens are refreshed from the DB on index, show, create and update, so a second worker does not report a stale token. template_config ETags for all three controllers are keyed on the resource's own tokens.

Needs a human decision

Rails.cache.clear is no longer called on DynamicModel, ExternalIdentifier and YAML Admin::ConfigLibrary saves. The issue does not authorise this. It is required for the target behaviour: the clear deleted server_cache_version, which rotated the compiled-template generation and wiped the skeleton on every definition save. ActivityLog and AppConfiguration already behaved this way via SelectorCache. Non-YAML libraries and Admin::PageLayout still clear everything.

To mitigate caches that depended on the clear, I added explicit shared-cache invalidation: definition-metadata keys (db_column_comments, db_foreign_keys, per-table data dictionary), the selector-with-config-overrides cache, and viewable_tables. Please check this against #1289.

  • Metadata invalidation runs after the migration. Admin::MigrationGenerator#run_migration drops the metadata keys only once the migration has succeeded, so a failed or skipped migration leaves them alone and a concurrent reader cannot re-cache pre-migration catalog data.
  • Column comments are read fresh when written to config. DynamicModel#set_comments_from_table and the update_config_from_table path call column_comments(force: true), so comments written back by a migration are never stale.
  • Data dictionary cache expires. data_dic caches under the basename (e.g. q2), but invalidation deleted the q2_datadic key and never matched. Invalidation now deletes both, and the cached rows expire after 5 minutes because dictionary rows change without any migration. A SELECT * plus array conversion measured 1-10 ms on the largest dev dictionary (1,313 rows).

Behaviour changes to know about

Verification

Passing on this branch: pages requests, views/masters, helpers (343), initializer, prewarm, concerns, classification, config library, user access control, activity-log/admin dynamic-model/external-identifier controller specs, activity-log/dynamic-model/external-identifier request specs, template_version_cache_poisoning_spec, dynamic_model_show_if_spec (browser, 4 examples), Jasmine (164 specs).

A combined run of pages requests, views/masters, prewarm, concerns, classification, helpers and the dynamic model, external identifier and activity log request specs (629 examples) had two prewarm failures, both spec issues fixed on this branch: candidates_spec.rb collided with recently signed-in users left in the test DB, and master_templates_spec.rb read a skeleton signature memoized on the process-wide ApplicationController.helpers by earlier pages specs (the shared reset helper now clears the new memos).

spec/models/dynamic_model_spec.rb passes (11 examples) on a freshly cleaned test database. spec/models/option_configs/dynamic_model_options_spec.rb ('gets the correct version of extra options based on creation date of the instance') fails identically on up-develop.

Issue verification items covered by dedicated specs: a brand-new definition (including a new details model) appears on the next full page load without a restart (template_new_definition_discovery_spec.rb, dynamic_model_new_definition_discovery_spec.rb); data index responses return 304 for a matching ETag, and 200 with a new ETag after a config-only edit or a change to only a model-referenced record's definition (template_definition_edit_freshness_spec.rb).

These fail the same way on the Phase 1 tip, so are not caused by this PR: notes_field_plain_format_spec.rb, active_model_configurations_fail_open_spec.rb:27, def_config_triggers_spec.rb:89, dynamic_models_versions_spec.rb:40 and :91, Jasmine "YAML top-level array first". Running spec/models/dynamic as a whole directory also fails in file order (before_save_trigger_spec, model_reference_handler_spec, 32 of them pass when run alone); the same specs fail on the Phase 1 baseline.

Review follow-ups (round 2)

A second independent review and a regression found these, all fixed with failing specs first:

  • Embedded forms use their own definition version. Regression introduced by this PR: client-side show_if of an embedded item (e.g. a dynamic model in an activity log form) was looked up under the parent's token and silently skipped in New and Edit mode. show_items now resolves the version per item, embedded forms carry their own data-vdef-version and render their own state-only template, and data_from_form carries the version for edit mode. System spec embedded_form_own_version_spec.rb and a Jasmine spec cover it (parent and embedded on different versions; the spec passes on the pre-Phase-1 baseline).
  • Index cache and authorization. index_cache_key includes the current definition's live token so a tightened showable_if is not bypassed by a cached index for records on an older version. A worker whose loaded definition is stale (another worker saved it) now reloads it (reload_if_stale) before reading the token, so a cache key never holds results computed from an older config.
  • Monotonic v0 ordering. Library-derived composite tokens contain a hash and are not ordered. The token stays the slot key/ETag; a separate live_order (definition history row id + latest config library history row id) is emitted with each live slot and compared for v0 promotion.
  • Option-specific slots. Records of a non-default option type now fetch their own slot when absent, instead of being masked by the base slot.
  • Freshness dedupe. template_config tokens/ETag include the dependencies of every requested root, not only the first.
  • Skeleton signature. Now covers the activity-log inputs baked into the skeleton and the dynamic model/external identifier inputs from round 1; extracted to MasterTemplates::SkeletonSignature (values for unchanged inputs are unchanged for DMs/EIs; activity-log skeleton keys change once).
  • Cache invalidation without Rails.cache.clear. One Dynamic::DefinitionCacheVersion token replaces the per-class and shared tokens (viewable tables, access control resource names, selector overrides, reports); the info-block-gs-block fragment is invalidated; selector-cache key tracking is guarded by a mutex.
  • State and refactors (R1-R8). state_only is an explicit caller decision, the external identifier skeleton has its own item partial, v0 promotion and embedded-dependency logic are single _fpa functions, template config freshness is Dynamic::TemplateConfigFreshness, the edit form state is a helper, and EmbeddedItemHandler is no longer used by the external identifier controller.
  • Specs. A prohibited conditional skip was removed; ordering/leak problems between specs were fixed (config_library_versioned_spec, embedded form spec uses its own aa_own_version activity log).

Review follow-ups (round 1)

  • Version-handler memo. refresh_outdated clears the class-level all_versions_memo, so an older record resolves to its historical version after another worker edits the definition. Reads, population and invalidation are coordinated with a mutex and a generation counter, so a reader can no longer see nil mid-clear or store a stale list after a clear.
  • Template dedup keys. _common_search_results_template_set dedup keys include the definition class, id and def_version, so same-named definitions of different types no longer suppress each other.
  • Session-scoped template_config URL. The client adds &s=<salt>.<hash>: a random, non-secret per-Rails-session salt plus a hash of user, sign-in time, app type and roles. A browser-cached body cannot be reused across sessions or access changes.
  • v0 from forms. A form of a current definition (New or Edit) promotes v0 through _fpa.promote_template_state_v0, which refuses to replace a v0 with a larger live_order, so a form from a stale worker cannot replace a newer v0. Historical forms never touch v0.
  • Spec fixes. The freshness spec reloads routes after creating a dynamic model; the show-if system fixtures are disabled on teardown.
  • Filed separately: Definition refresh ignores edits made within 2 seconds of the last loaded update #1533 (refresh_outdated/up_to_date? 2 s timestamp tolerance can hold a stale definition).

Review notes

  • Tokens are derived from the history table even when DisableVDef is set, so definition edits still reach clients in development; def_version stays nil and records still use the current definition.
  • Edit forms for records on an older definition version now emit that record's own versioned config under its historical slot, never the live slot or v0.
  • Skeleton structure digests are memoized per definition (id, updated_at, token).

Not covered / known limits

  • Panel/chrome and sibling activity-log configuration changes need a page reload on an already open page (accepted).
  • A list that is empty and never opens a New form keeps its old list-level v0 (caption, add label, orientation) until a record fetch or skeleton rebuild, because the endpoint needs a record id. The plan's optional version-map endpoint would remove this.
  • A config library change alone is still picked up by another worker through refresh_outdated with its existing short tolerance (Definition refresh ignores edits made within 2 seconds of the last loaded update #1533); definition changes are detected immediately via reload_if_stale.
  • For a dynamic model or external identifier that declares option types but no default, the list chrome now comes from the generated default, not the first declared option type.
  • max_age: 30 on template_config is kept; the ?v= query discriminator avoids serving a stale body to a changed token.
  • Classification::ItemFlagName is still not in any key (existing gap, Cache key audit: under-scoped and shared-cache/local-state mismatches across Rails.cache usages #1289).
  • Not run: the complete spec/ suite. After the final fixes: spec/system (excluding nfs_store and redcap) 645 examples with only play_ipa_process_spec.rb:83 and study_info_process_spec.rb:56 failing (both also fail on up-develop); Jasmine 230 specs; the model, helper, request, view and controller groups for the changed areas pass. A few specs are order-dependent in very large combined runs and pass alone (message_template_spec.rb:233, item_flags_controller_spec.rb:130, switch_id_display_spec.rb:197/252, manage_users_spec.rb:169, admin_master_records_spec.rb:167, upgrade_to_2fa_spec.rb:156).
  • template_config ETags change once on deploy (definition resource tokens are now part of the key), as do activity-log skeleton keys.

@philayres
philayres force-pushed the handlebars-lazy-config-phase2-1481 branch from adb17f0 to 3405923 Compare November 4, 2026 15:01
@philayres
philayres force-pushed the handlebars-lazy-config-phase2-1481 branch 6 times, most recently from 4b8cdb4 to 5f1da02 Compare November 11, 2026 13:31
Phil Ayres and others added 15 commits November 12, 2026 21:22
…nsected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…item partials - refs consected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… - refs consected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…consected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… refs consected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…istory_id fallback assertion - refs consected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nsected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…efs consected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…- refs consected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…endencies in template config freshness - refs consected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ecific slot fetching - refs consected#1481

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cs and embedded spec robustness - refs consected#1481

Includes the finding 4 and 6 code and spec changes that were picked up earlier in 66c6705.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@philayres
philayres force-pushed the handlebars-lazy-config-phase2-1481 branch from fbb3280 to 93202c3 Compare November 14, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build the Handlebars bundle once per server uptime and lazy-patch changed resource configs

1 participant