Refresh Babel discovery in the background and cache status health

README.md

@@ -102,6 +102,7 @@ index names:
   "version": "1.1.0",
   "biolink_model_toolkit_version": "v4.4.3",
   "babel": {
+    "checked_at": "2026-10-23T12:00:00+00:00",
     "default_version": "2026jul22",
     "available_versions": {
       "2025sep1": "https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1/README.md",
@@ -116,15 +117,16 @@ index names:
 ```
 
 Index names in this example are illustrative. Each `indices` array contains the
-concrete targets selected at startup and pinned until restart. The default release
+concrete targets selected by the latest successful discovery snapshot. The default release
 uses every backing index of its configured alias; other releases select one index.
 Available releases come from Elasticsearch metadata, rather than a configured version
 list; these maps do not indicate backend health.
 
 ### Discovering available releases
 
-At startup, the server uses its existing `ES_HOST` client to discover release indices
-from `mappings._meta.src.nodenorm` metadata on the associated cluster. No version list,
+At startup and every **10 minutes** in the background, the server uses its existing
+`ES_HOST` client to discover release indices from `mappings._meta.src.nodenorm` metadata
+on the associated cluster. No version list,
 version-to-index mapping, or per-release alias is required. The client's Elasticsearch
 permissions must allow reading mappings across that cluster. Discovery uses an explicit
 `version` field, or a dated release tag in the legacy source URL; it does not infer
@@ -145,17 +147,32 @@ release and its exact concrete backing indices. All targets must identify the sa
 Babel release. The default keeps those targets even if newer builds exist elsewhere
 on the cluster; a multi-index default queries all its backing indices. Explicitly
 requesting the default Babel version uses those same alias targets. After the alias
-moves to another release and the API restarts, the former default becomes a supported
+moves to another release and discovery refreshes, the former default becomes a supported
 nondefault release and uses the build-date rule above.
 
-The default release has no separate Babel configuration override. Configuration and
-metadata discovery are loaded at startup. Adding a release or a newer build makes
-it available after restart; adding a release does not automatically change the
-default. To change the default, move the configured alias (or change the configured
-default target) and restart.
-
-All selected concrete indices stay pinned until restart, so moving an alias cannot
-change queries in a running process. Keep indices serving existing processes until
-the restart rollout completes; removing an index earlier makes requests using it fail.
-Verify `/status` and sample results before rollout; discovery relies on metadata
-accurately describing each index's contents.
+The default release has no separate Babel configuration override. Configuration is
+loaded at startup. Adding a release or a newer build makes it available on the next
+successful background refresh; adding a release does not automatically change the
+default. Moving the configured default alias takes effect on the next successful
+refresh, without restarting the API. Changing the configuration itself requires a restart.
+
+Each process refreshes independently, even without incoming requests. A refresh has a
+10-second total deadline. It publishes the new release registry and default query
+indices together, so `/status` and new requests use the same snapshot. Each request
+keeps its selected concrete indices throughout processing, including conflation lookups.
+Keep previous indices available until every process has refreshed and requests using
+the previous snapshot have completed. Verify `/status` and sample results before
+removing indices; discovery relies on metadata accurately describing each index's contents.
+
+`babel.checked_at` is the UTC time of the last successful discovery. A failed refresh
+retains that snapshot and timestamp, adds `babel.refresh_error` with code
+`discovery_failed`, and retries after 30 seconds. A successful retry clears the error.
+Discovery failures alone do not determine service health.
+
+`GET /status` checks that every selected default index still exposes the expected
+Babel source version. These separate health results are cached for **5 seconds** on
+success and **1 second** on failure, with concurrent requests sharing one check and a
+3-second total deadline. Changing the default version or its concrete indices causes
+the next status request to use a new health check. A failed check returns HTTP 503;
+an expired healthy result is never served after a failed check. Health checks do not
+trigger Babel discovery. All status responses send `Cache-Control: no-store`.

src/nodenorm/babel.py

@@ -80,6 +80,16 @@ def __init__(self, default_version: str, releases: dict, unavailable: dict = Non
         self._releases = {version: BabelRelease(version, index) for version, index in releases.items()}
         self._unavailable = dict(unavailable)
 
+    @classmethod
+    def from_default_mappings(cls, default_mappings, mappings):
+        """Build one registry from a validated default target and a cluster scan."""
+        versions = {get_babel_version(mapping) for mapping in default_mappings.values()}
+        if len(versions) != 1:
+            raise ValueError("The default Elasticsearch target must contain exactly one Babel release")
+        backing_indices = tuple(default_mappings)
+        default_index = backing_indices[0] if len(backing_indices) == 1 else backing_indices
+        return cls.from_mappings(versions.pop(), {**mappings, **default_mappings}, default_index=default_index)
+
     @classmethod
     def from_mappings(cls, default_version, mappings, default_index=None):
         """Discover release names and targets from source metadata on concrete indices."""

src/nodenorm/handlers/status.py

@@ -2,12 +2,13 @@
 import json
 from functools import lru_cache
 
-from elasticsearch import AsyncElasticsearch
+from tornado.web import HTTPError
 
 import nodenorm
-from nodenorm.babel import get_babel_release_url, get_babel_version
+from nodenorm.babel import get_babel_release_url
 from nodenorm.biolink import BIOLINK_MODEL_VERSION
 from nodenorm.handlers.base import NodeNormalizationBaseHandler
+from nodenorm.status_health import StatusHealthCache
 
 
 @lru_cache(maxsize=None)
@@ -23,23 +24,32 @@ class NodeNormHealthHandler(NodeNormalizationBaseHandler):
 
     name = "health"
 
-    async def get(self):
-        async_client: AsyncElasticsearch = self.biothings.elasticsearch.async_client
-        search_indices = self.biothings.elasticsearch.indices
-
-        mapping_response = await async_client.indices.get_mapping(index=search_indices)
-        index_mapping = next(iter(mapping_response.values()))
-        # Validate live default metadata while reporting the releases resolved at startup.
-        get_babel_version(index_mapping)
+    def set_default_headers(self):
+        super().set_default_headers()
+        self.set_header("Cache-Control", "no-store")
 
-        releases = self.biothings.elasticsearch.babel
+    async def get(self):
+        elasticsearch = self.biothings.elasticsearch
+        releases = elasticsearch.babel
         babel_status = {
             "default_version": releases.default_version,
             "available_versions": {version: get_babel_release_url(version) for version in releases.available_versions},
             "indices": {version: releases.resolve(version).search_indices for version in releases.available_versions},
+            "checked_at": elasticsearch.babel_checked_at,
         }
         if releases.errors:
             babel_status["errors"] = releases.errors
+        refresh_error = elasticsearch.babel_refresh_error
+        if refresh_error is not None:
+            babel_status["refresh_error"] = refresh_error
+
+        release = releases.resolve()
+        health = getattr(elasticsearch, "status_health_cache", None)
+        if health is None or health.configuration != (release.version, tuple(release.search_indices)):
+            health = StatusHealthCache(elasticsearch.async_client, release)
+            elasticsearch.status_health_cache = health
+        if not await health.check():
+            raise HTTPError(503, reason="Elasticsearch health check failed")
 
         status_response = {
             "status": "running",

src/nodenorm/namespace.py

@@ -1,3 +1,4 @@
+import asyncio
 import importlib.resources
 import importlib.util
 import json
@@ -7,17 +8,22 @@
 import re
 import sys
 import types
+from contextlib import suppress
+from datetime import datetime, timezone
 
 import tornado.options
 
 from biothings.web import connections
 from biothings.web.services.metadata import BiothingsESMetadata
 
 import nodenorm
-from nodenorm.babel import BabelReleases, get_babel_version
+from nodenorm.babel import BabelReleases
 
 
 logger = logging.getLogger(__name__)
+BABEL_REFRESH_INTERVAL = 600
+BABEL_REFRESH_RETRY_INTERVAL = 30
+BABEL_REFRESH_TIMEOUT = 10
 
 
 class NodeNormalizationAPINamespace:
@@ -83,27 +89,71 @@ def configure_elasticsearch(self) -> types.SimpleNamespace:
         elasticsearch_namespace.async_client = connections.es.get_async_client(
             elasticsearch_configuration["ES_HOST"], **elasticsearch_configuration["ES_ARGS"]
         )
-        # Preserve the alias's exact targets for the default release, even when
-        # the cluster contains a newer build of that Babel release.
         configured_indices = self._validate_elasticsearch_index(elasticsearch_namespace)
-        default_mappings = elasticsearch_namespace.client.indices.get_mapping(index=configured_indices)
-        versions = {get_babel_version(mapping) for mapping in default_mappings.values()}
-        if len(versions) != 1:
-            raise ValueError("The default Elasticsearch target must contain exactly one Babel release")
-        default_version = versions.pop()
-        backing_indices = tuple(default_mappings)
-        default_index = backing_indices[0] if len(backing_indices) == 1 else backing_indices
-        mappings = elasticsearch_namespace.client.indices.get_mapping(filter_path="*.mappings._meta")
-        # Preserve targets resolved above if the cluster-wide scan omits them.
-        # Prefer the scan's metadata when an index is present in both responses.
-        releases = BabelReleases.from_mappings(
-            default_version, {**default_mappings, **mappings}, default_index=default_index
+        default_mappings = elasticsearch_namespace.client.indices.get_mapping(
+            index=configured_indices, filter_path="*.mappings._meta"
         )
-        # Pin every selected concrete index until restart.
+        mappings = elasticsearch_namespace.client.indices.get_mapping(filter_path="*.mappings._meta")
+        releases = BabelReleases.from_default_mappings(default_mappings, mappings)
         elasticsearch_namespace.indices = releases.resolve().search_indices
         elasticsearch_namespace.babel = releases
+        elasticsearch_namespace.babel_checked_at = datetime.now(timezone.utc).isoformat()
+        elasticsearch_namespace.babel_refresh_error = None
         return elasticsearch_namespace
 
+    async def _discover_babel_releases(self):
+        client = self.elasticsearch.async_client
+        configuration = self.config.elasticsearch
+        index = configuration["ES_INDEX"]
+        alias = configuration["ES_ALIAS"]
+        if index and await client.indices.exists(index=index):
+            target = index
+        elif await client.indices.exists_alias(name=alias):
+            target = alias
+        else:
+            raise RuntimeError("Unable to validate nodenorm elasticsearch index / alias")
+        default_mappings = await client.indices.get_mapping(index=[target], filter_path="*.mappings._meta")
+        mappings = await client.indices.get_mapping(filter_path="*.mappings._meta")
+        return BabelReleases.from_default_mappings(default_mappings, mappings)
+
+    async def refresh_babel_releases(self):
+        """Publish a complete discovery snapshot, retaining the previous one on failure."""
+        try:
+            releases = await asyncio.wait_for(self._discover_babel_releases(), timeout=BABEL_REFRESH_TIMEOUT)
+        except Exception:
+            logger.exception("Unable to refresh Babel releases; retaining the previous snapshot")
+            self.elasticsearch.babel_refresh_error = {"code": "discovery_failed"}
+            return False
+
+        # There is no await between these writes: handlers see one complete
+        # snapshot, and requests that already selected a release keep that object.
+        self.elasticsearch.babel = releases
+        self.elasticsearch.indices = releases.resolve().search_indices
+        self.elasticsearch.babel_checked_at = datetime.now(timezone.utc).isoformat()
+        self.elasticsearch.babel_refresh_error = None
+        return True
+
+    async def _run_babel_refresh(self):
+        delay = BABEL_REFRESH_INTERVAL
+        while True:
+            await asyncio.sleep(delay)
+            refreshed = await self.refresh_babel_releases()
+            delay = BABEL_REFRESH_INTERVAL if refreshed else BABEL_REFRESH_RETRY_INTERVAL
+
+    def start_babel_refresh(self):
+        """Start one background discovery task on the server's running event loop."""
+        task = getattr(self, "_babel_refresh_task", None)
+        if task is None or task.done():
+            self._babel_refresh_task = asyncio.create_task(self._run_babel_refresh())
+
+    async def stop_babel_refresh(self):
+        task = getattr(self, "_babel_refresh_task", None)
+        if task is not None:
+            task.cancel()
+            with suppress(asyncio.CancelledError):
+                await task
+            self._babel_refresh_task = None
+
     def _validate_elasticsearch_index(self, elasticsearch_namespace: types.SimpleNamespace) -> dict:
         """Validates the elasticsearch index / alias.
 

src/nodenorm/server.py

@@ -61,11 +61,15 @@ def start(self):
             pformat(self.namespace.handlers, width=200),
         )
 
+        loop = tornado.ioloop.IOLoop.instance()
         try:
-            loop = tornado.ioloop.IOLoop.instance()
+            loop.add_callback(self.namespace.start_babel_refresh)
             loop.start()
         except Exception as gen_exc:
             logger.exception(gen_exc)
             raise gen_exc
         finally:
-            loop.close()
+            try:
+                loop.run_sync(self.namespace.stop_babel_refresh)
+            finally:
+                loop.close()

src/nodenorm/status_health.py

@@ -0,0 +1,48 @@
+"""Short health caching, independent of the ten-minute Babel discovery snapshot."""
+
+import asyncio
+import logging
+from time import monotonic
+
+from nodenorm.babel import get_babel_version
+
+
+logger = logging.getLogger(__name__)
+HEALTH_CACHE_TTL = 5
+HEALTH_FAILURE_TTL = 1
+HEALTH_TIMEOUT = 3
+
+
+class StatusHealthCache:
+    def __init__(self, client, release):
+        self.client = client
+        self.configuration = (release.version, tuple(release.search_indices))
+        self._healthy = None
+        self._expires_at = 0
+        self._refresh = None
+
+    async def check(self):
+        if self._healthy is not None and monotonic() < self._expires_at:
+            return self._healthy
+        if self._refresh is None or self._refresh.done():
+            self._refresh = asyncio.create_task(self._load())
+        # A disconnected probe must not cancel the check shared by other callers.
+        return await asyncio.shield(self._refresh)
+
+    async def _probe(self):
+        version, indices = self.configuration
+        mappings = await self.client.indices.get_mapping(
+            index=list(indices), filter_path="*.mappings._meta.src.nodenorm"
+        )
+        return set(mappings) == set(indices) and all(
+            get_babel_version(mapping) == version for mapping in mappings.values()
+        )
+
+    async def _load(self):
+        try:
+            self._healthy = await asyncio.wait_for(self._probe(), timeout=HEALTH_TIMEOUT)
+        except Exception:
+            logger.warning("NodeNorm Elasticsearch health check failed", exc_info=True)
+            self._healthy = False
+        self._expires_at = monotonic() + (HEALTH_CACHE_TTL if self._healthy else HEALTH_FAILURE_TTL)
+        return self._healthy