Skip to content

fix(server): keep schema caches current across HStore Servers - #3481

Merged
imbajin merged 9 commits into
apache:masterfrom
hugegraph:fix/schema-cache-events-3235
Nov 19, 2026
Merged

imbajin merged 9 commits into
apache:masterfrom
hugegraph:fix/schema-cache-events-3235

Conversation

@bitflicker64

@bitflicker64 bitflicker64 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the PR

Closes #3235. Keep HStore schema caches current across Servers within seconds and coalesce invalidations during DDL bursts. The patch uses the existing PD event and wire format.

Main Changes

Before → After

Before (upstream master 8828d8238) After (this PR)
Schema changes Updates and index status changes could stay on the writer. Removal and truncate notifications depended on task.sync_deletion. Adds, updates, removals, clear and truncate queue an invalidation. Partial writes invalidate caches and still return the original error.
Cache invalidation Each published event invalidated the peer's whole graph schema cache. A fixed 1 s window per Server and graph coalesces notifications. Local cache updates stay immediate. A generation guard prevents in-flight reads from refilling cleared caches.
Recovery A reconnect could leave missed updates cached. A failed publish could surface as a failed DDL after the write committed. A confirmed watch start or reconnect clears local V2 caches. Failed publications retry in the background. Opening a graph publishes a fresh clear, which covers a writer restart.
image schema-cache1 schema-cache2

For normal DDL, each Server coalesces clear events per graph in a fixed 1 s window. Peers reload from PD on the next schema access after invalidation.

Verifying these changes

  • Tested on Java 17 at 6dc4691c5. 114 targeted tests passed across the two schema cache test classes and KvClientTest.
  • ASF Server CI passed on the same commit, including the multi-node cluster job. ASF CI · Cluster job

Three-Server results

Independent REST run on one host with 3 PD, 3 Store and 3 Servers. The pre-batching control uses the d2d54fe0e schema cache class with the same binaries as 6dc4691c5. Each Server took a turn writing 200 schema updates while both peers read schema.

Metric Before batching After batching
Clear events received by each peer per 200 updates 200 / 200 / 200 8 / 6 / 6 (~97% fewer)
Combined schema reads/s on both peers during the burst 111 / 121 / 496 527 / 1,018 / 1,189
Median time for an isolated update to become visible on a peer 0.045 s 1.05 s (max 1.08 s)
  • The final burst update was visible on both peers within 0.72 s of the last DDL response. Three concurrent writers updating distinct elements also converged. Sync and async deletion, truncate, and vertex/edge label updates passed.
  • Killing the PD leader recovered visibility in ~3.2 s. A single-Server reconnect cleared missed updates in ~1 s. An injected publish failure was retried ~1 s after connectivity returned.

TODOs and rollout limits

  • Keep the watch start timeout active until Starting or Started confirms registration, and add a regression test. In a controlled gRPC probe, an Alive-only stream cancelled the timeout and neither retried nor ran the cache-clear callback.
  • Repeat the index-write race with explicit unique vertex IDs before quoting a miss rate. The race already reproduced index queries missing vertices after the rebuild reported success. Pause writes to the affected label until the rebuild finishes and all Servers see the new schema.
  • Pin DDL for the same element to one Server. Verify visibility before switching writers. An immediate handoff reproduced a lost append. Conditional writes are tracked in feat(pd): commit schema keys and a per-graph record as one raft entry #3376.
  • A writer killed between commit and publish can leave peers stale. In the SIGKILL run, peers stayed stale throughout the 6 s outage and saw the update ~6.4 s after the writer restarted. Atomic publication, an index rebuild barrier, and recovery of permanently stopped watches ([Bug] A PD watch stopped by a non-retryable gRPC status is never started again, so Server meta listeners stay deaf until restart #3478) remain follow-up work. Upgrade all Servers for update notifications from every writer.

The org PR shares this branch. Earlier validation and review history are available in #3477.

Server runtime guide · Helm operations guide

bitflicker64 and others added 6 commits November 13, 2026 11:26
With several Servers on HStore, a schema change made through one
Server reached the others only for adds. Updates published no clear
event, removes published one only with task.sync_deletion=false, and
an event sent while a Server's PD watch was reconnecting was lost.
A Server that kept an old copy then wrote it back over newer schema.

Complete the existing clear event instead of adding a new mechanism:

- publish after every committed add, update, remove and store clear
  or truncate, and once when a graph is opened
- KvClient.listen() accepts a callback that runs each time a watch
  goes live; the Server clears its V2 schema caches when it runs
- a publish that fails after the schema write no longer fails the
  call; the graph is kept in a pending set and published again with
  backoff and when the watch is live again
- a generation counter and a lock on the shared cache keep a read,
  a listing or a write-through that started before a clear from
  caching its value after the clear
- every event clears at once, events are not merged

Not covered here: a write in flight on another Server while an index
is created can still miss the index, a writer that stops for good
between the schema write and its publish, two Servers changing the
same element at once, serving cached schema while cut off from PD,
and a watch stream that dies without an error. They are documented
in docs/server-runtime.md.
A second pass over the schema cache clear path found four defects.

- A schema write is two or three PD puts. One that failed after the
  first put left the change partly in PD, published nothing and kept
  the writer's own copy. It now drops that copy, publishes, and still
  reports the error. The publish also runs if caching the written
  element fails.
- The generation was advanced before the caches were changed. A read
  running at that moment could note the new generation, take the old
  value from the id cache and copy it into the array cache afterwards,
  where it stayed until the next clear. The generation is now advanced
  as the last step of every change.
- A watch start during a running retry scheduled a second retry chain,
  which doubled the publishes and the backoff. Only the newest retry
  reschedules now.
- After a refused write-through the "whole list is cached" flag of
  that schema type was never set again, so each listing read PD until
  the next clear.

Tests: one per fix, plus tests for the publish order, the callback
passed by PdMetaDriver, the guard across transactions of one graph and
local invalidation events. The listing-lock test now holds a real
listing, and the concurrency test checks every read while it runs. A
V2 graph opened without PD no longer leaves a retry behind that fires
in a later test.
Follow-up to review comments on this change.

- MultiClusterSchemaCacheTest: the readers that run while an index is
  created were submitted and never checked, so a reader that stopped on
  an error left the test passing with fewer reads. Their futures are
  read once the index is created.
- docs/server-runtime.md: a PD watch that ends with a gRPC status the
  PD client does not retry is not started again, so that Server keeps
  its cached schema until it is restarted. This is how master behaves
  for every meta listener and is tracked in apache#3478.
- A TODO in the test's waitFor notes that the shared RestClient sets no
  timeouts.
- Distinguish watch readiness from transport heartbeats.
- Bound cluster HTTP calls and assert successful reader responses.
- Drain in-flight retries before replacing the test publisher.
- Clarify schema writer routing and propagation limits.
Batch normal clear events per graph in a fixed one-second window.
Reuse the existing scheduler and retry path, keeping graph-open recovery immediate.
Cover batching, concurrent flushes and test cleanup races; document delayed visibility.
Serialize manual window flushes with the existing scheduler so assertions wait for an already-running publication.

Add a regression that blocks a detached window and verifies the helper waits for its publication.

Exercise concurrent retry failure through the immediate graph-open publish path.
@codecov

codecov Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.89189% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.66%. Comparing base (8828d82) to head (ccb000c).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...graph/backend/cache/CachedSchemaTransactionV2.java 92.62% 7 Missing and 9 partials ⚠️
...n/java/org/apache/hugegraph/meta/PdMetaDriver.java 50.00% 3 Missing ⚠️
...ain/java/org/apache/hugegraph/meta/MetaDriver.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3481      +/-   ##
============================================
+ Coverage     58.52%   58.66%   +0.14%     
- Complexity     2717     3104     +387     
============================================
  Files           779      780       +1     
  Lines         69439    69825     +386     
  Branches       9577     9731     +154     
============================================
+ Hits          40637    40964     +327     
- Misses        25108    25171      +63     
+ Partials       3694     3690       -4     
Flag Coverage Δ *Carryforward flag
commons 96.37% <ø> (ø) Carriedforward from a5fa1fa
hstore 36.33% <62.71%> (+0.11%) ⬆️
pd 57.01% <100.00%> (+0.17%) ⬆️
server_hbase 42.60% <0.00%> (+0.11%) ⬆️ Carriedforward from a5fa1fa
server_memory 57.05% <90.78%> (+0.40%) ⬆️
server_rocksdb 42.84% <3.07%> (-0.17%) ⬇️
store 36.13% <0.00%> (-0.07%) ⬇️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A Server that stops for good while a clear event waits in the 1-second
window never sends it. Before the window this gap lasted milliseconds.
Measured on a 3-Server cluster: after one schema change and a kill of
the writing Server within the window, the other Servers kept the old
schema.
@imbajin
imbajin force-pushed the fix/schema-cache-events-3235 branch 4 times, most recently from f6966ea to 6dc4691 Compare November 19, 2026 06:45
Isolate setup windows and verify publish-after-write ordering using
existing test gates. Clarify batching semantics and safe index recovery
without altering runtime behavior.
Keep the registration-start timeout active until Starting or Started
for watches using a started callback. Preserve legacy first-frame
readiness for ordinary watch consumers. Add deterministic recovery test.
@imbajin
imbajin merged commit f60b628 into apache:master Nov 19, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Stale schema caches on HStore Servers: updates stay invisible, appends get overwritten, new indexes miss data

2 participants