Repository navigation
fix(server): keep schema caches current across HStore Servers - #3481
Merged
Merged
Conversation
With several Servers on HStore, a schema change made through one Server reached the others only for adds. Updates published no clear event, removes published one only with task.sync_deletion=false, and an event sent while a Server's PD watch was reconnecting was lost. A Server that kept an old copy then wrote it back over newer schema. Complete the existing clear event instead of adding a new mechanism: - publish after every committed add, update, remove and store clear or truncate, and once when a graph is opened - KvClient.listen() accepts a callback that runs each time a watch goes live; the Server clears its V2 schema caches when it runs - a publish that fails after the schema write no longer fails the call; the graph is kept in a pending set and published again with backoff and when the watch is live again - a generation counter and a lock on the shared cache keep a read, a listing or a write-through that started before a clear from caching its value after the clear - every event clears at once, events are not merged Not covered here: a write in flight on another Server while an index is created can still miss the index, a writer that stops for good between the schema write and its publish, two Servers changing the same element at once, serving cached schema while cut off from PD, and a watch stream that dies without an error. They are documented in docs/server-runtime.md.
A second pass over the schema cache clear path found four defects. - A schema write is two or three PD puts. One that failed after the first put left the change partly in PD, published nothing and kept the writer's own copy. It now drops that copy, publishes, and still reports the error. The publish also runs if caching the written element fails. - The generation was advanced before the caches were changed. A read running at that moment could note the new generation, take the old value from the id cache and copy it into the array cache afterwards, where it stayed until the next clear. The generation is now advanced as the last step of every change. - A watch start during a running retry scheduled a second retry chain, which doubled the publishes and the backoff. Only the newest retry reschedules now. - After a refused write-through the "whole list is cached" flag of that schema type was never set again, so each listing read PD until the next clear. Tests: one per fix, plus tests for the publish order, the callback passed by PdMetaDriver, the guard across transactions of one graph and local invalidation events. The listing-lock test now holds a real listing, and the concurrency test checks every read while it runs. A V2 graph opened without PD no longer leaves a retry behind that fires in a later test.
Follow-up to review comments on this change. - MultiClusterSchemaCacheTest: the readers that run while an index is created were submitted and never checked, so a reader that stopped on an error left the test passing with fewer reads. Their futures are read once the index is created. - docs/server-runtime.md: a PD watch that ends with a gRPC status the PD client does not retry is not started again, so that Server keeps its cached schema until it is restarted. This is how master behaves for every meta listener and is tracked in apache#3478. - A TODO in the test's waitFor notes that the shared RestClient sets no timeouts.
- Distinguish watch readiness from transport heartbeats. - Bound cluster HTTP calls and assert successful reader responses. - Drain in-flight retries before replacing the test publisher. - Clarify schema writer routing and propagation limits.
Batch normal clear events per graph in a fixed one-second window. Reuse the existing scheduler and retry path, keeping graph-open recovery immediate. Cover batching, concurrent flushes and test cleanup races; document delayed visibility.
Serialize manual window flushes with the existing scheduler so assertions wait for an already-running publication. Add a regression that blocks a detached window and verifies the helper waits for its publication. Exercise concurrent retry failure through the immediate graph-open publish path.
This was referenced Nov 15, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3481 +/- ##
============================================
+ Coverage 58.52% 58.66% +0.14%
- Complexity 2717 3104 +387
============================================
Files 779 780 +1
Lines 69439 69825 +386
Branches 9577 9731 +154
============================================
+ Hits 40637 40964 +327
- Misses 25108 25171 +63
+ Partials 3694 3690 -4
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
A Server that stops for good while a clear event waits in the 1-second window never sends it. Before the window this gap lasted milliseconds. Measured on a 3-Server cluster: after one schema change and a kill of the writing Server within the window, the other Servers kept the old schema.
imbajin
force-pushed
the
fix/schema-cache-events-3235
branch
4 times, most recently
from
November 19, 2026 06:45
f6966ea to
6dc4691
Compare
Isolate setup windows and verify publish-after-write ordering using existing test gates. Clarify batching semantics and safe index recovery without altering runtime behavior.
Keep the registration-start timeout active until Starting or Started for watches using a started callback. Preserve legacy first-frame readiness for ordinary watch consumers. Add deterministic recovery test.
imbajin
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of the PR
Closes #3235. Keep HStore schema caches current across Servers within seconds and coalesce invalidations during DDL bursts. The patch uses the existing PD event and wire format.
Main Changes
Before → After
master8828d8238)task.sync_deletion.For normal DDL, each Server coalesces clear events per graph in a fixed 1 s window. Peers reload from PD on the next schema access after invalidation.
Verifying these changes
6dc4691c5. 114 targeted tests passed across the two schema cache test classes andKvClientTest.Three-Server results
Independent REST run on one host with 3 PD, 3 Store and 3 Servers. The pre-batching control uses the
d2d54fe0eschema cache class with the same binaries as6dc4691c5. Each Server took a turn writing 200 schema updates while both peers read schema.TODOs and rollout limits
StartingorStartedconfirms registration, and add a regression test. In a controlled gRPC probe, anAlive-only stream cancelled the timeout and neither retried nor ran the cache-clear callback.The org PR shares this branch. Earlier validation and review history are available in #3477.
Server runtime guide · Helm operations guide