Translator Shepherd Service

Shepherd is a shared platform for ARA implementation. Incorporated ARAs have access to a plethora of shared ARA functionality while retaining the ability to implement their own custom operations.

TRAPI version

Shepherd speaks TRAPI 2.0 (x-trapi.version in the served OpenAPI document). Everything TRAPI-shaped is described by translator_tom (TOM), the Translator-wide TRAPI object model that Gandalf and Retriever also use, rather than being re-derived here:

• Requests are checked against TOM's TRAPI 2.0 Query (its lightweight TypedDict validation, so the orjson fast path stays fast) and the OpenAPI request/response schemas come from TOM's models. workflow is the exception: Shepherd runs its own operations (aragorn.lookup, score_paths, ...) and checks them against its own list.
• Internals stay plain dicts -- responses can hold hundreds of thousands of edges and are reloaded by every worker -- typed with TOM's TypedDicts where it helps. The 2.0 helpers every worker shares (binding ids, query parameters, stored/delivered response form) live in shepherd_utils/trapi.py, whose module docstring lists the 2.0 rules the pipeline depends on.
• Stored vs delivered responses. Every worker writes a query's response through shepherd_utils.db.save_response, which stores it without the delivery envelope (schema_version, biolink_version, parameters, logs) and with nulls and forbidden empties already pruned. When the response is delivered the envelope is added -- by finish_query directly on the stored bytes, without decoding the response.
• The ARS port validates ARA responses with TOM's 2.0 models.

What changed for clients coming from TRAPI 1.x:

• log_level and bypass_cache live in parameters (with timeout), and the response repeats the parameters it was given.
• A QEdge's qualifier_constraints / attribute_constraints are now one constraints object; a qualifier set is a {qualifier_type_id: value} mapping. A QPath constraint's intermediate_categories is now required_intermediate_categories.
• A query using a 1.x spelling is rejected with HTTP 400 naming the 2.0 replacement, rather than being run with the filter silently ignored.
• Responses use 2.0 bindings (one {"ids": [...]} object per query node / edge / path), carry knowledge_level / agent_type as top-level edge properties, and are stamped with schema_version and biolink_version. Nulls and the empty containers 2.0 forbids (logs, auxiliary_graphs, analyses, ...) are omitted.

Shepherd does not convert between TRAPI versions. The services it calls (Retriever, ARAX, the pathfinder library) are expected to answer in 2.0; a 1.x-shaped answer is not read correctly.

Local Development

All Shepherd services are set up as docker containers. You can learn about docker and install here: https://www.docker.com/

The main entrypoint is ./compose.yml and will spin up the core stack (server, db, broker, monitor, shared workers and the hosted ARAs).

• In the root folder, run docker compose up --build

The Translator ARS services (ars_fanout, ars_premerge, ars_merge, ars_watchdog, ars_notify) are opt-in, behind the ars compose profile:

• docker compose --profile ars up --build, or
• set COMPOSE_PROFILES=ars in your .env to always include them.

Every Shepherd image builds on one shared base image, shepherd_base/Dockerfile: the Python image, runtime settings and Shepherd's dependencies (read from pyproject.toml), but no Shepherd code. Compose builds it first on its own. A code change then rebuilds only each image's last few layers, and only a pyproject.toml change reinstalls the dependencies. To build one image without compose, build and tag the base first:

docker build -f shepherd_base/Dockerfile -t shepherd_base .
docker build -f workers/merge_message/Dockerfile -t merge_message .

Every service in compose.yml carries prod-matching CPU / memory limits (see the header of compose.yml for why), so the stack needs a machine (or Docker Desktop VM) with at least 6 CPUs. To lift them locally, override cpus / mem_limit (0 = unlimited) in a gitignored compose.override.yml.

If you want to add a new operation/worker, add a new service in compose.yml under services (with profiles: [ars] if it's part of the ARS).

Data volumes: Postgres, Redis and RedisInsight keep their data in the named volumes shepherd_db_data, shepherd_broker_data and shepherd_broker_insight_data, not in ./pgdata and ./data folders (a bind mount on Docker Desktop doesn't make their writes durable, so an unclean stop could corrupt them). Docker Desktop's Volumes tab can browse and download their files. With REDIS_AOF_AUTOFIX=yes (the compose default) the broker repairs a damaged AOF at startup, keeping the damaged files in /data/appendonlydir.damaged, instead of restart-looping on it. To start empty, run docker compose down -v, or docker volume rm just the volume to reset (e.g. shepherd_broker_data).

Credentials and admin access:

• The broker's password comes from REDIS_PASSWORD (shepherd_broker/entrypoint.sh passes it to redis-server; it is not in redis.conf). The broker refuses to start without it, and every service must be given the same value. Compose supplies a development default.
• The monitor's POST /api/admin/* routes require Authorization: Bearer $MONITOR_ADMIN_TOKEN, and are disabled until MONITOR_ADMIN_TOKEN is set. Its dashboard and read-only API need no token.
• In compose, Postgres, Redis, RedisInsight, the monitor and Jaeger listen on 127.0.0.1 only; use an SSH tunnel to reach them from another machine. The API server (5439) is published on every interface.

Worker data (LMDB / sqlite) downloads

A couple of workers read from large, read-only sqlite databases and LMDB datasets that are too big to commit to git (they're gitignored and volume-mounted from the host):

• aragorn_omnicorp → ./omnicorp_lmdb/ (curies.lmdb, shared_counts.lmdb)
• score_paths → ./pathfinder_embeddings/ (a directory-style LMDB)
• arax_pathfinder → ./arax_pathfinder_dbs/ (curie_ngd_v1.0_<tier-version>.sqlite, tier0-info-for-overlay_v1.0_<tier-version>.sqlite, general_concepts.json)
• arax (the in-process ARAX port) → ./arax_dbs/ (curie_to_pmids_v1.0_<tier-version>.sqlite, ExplainableDTD_v1.0_<tier-version>-all_with_paths.db, fda_approved_drugs_v1.0.pickle, COHDdatabase_v1.0_KG2.8.0.db), plus the arax_pathfinder sqlite dbs in ./arax_pathfinder_dbs/. The server's ARAX API fetches autocomplete_v1.0_<tier-version>.sqlite into the same ./arax_dbs/ (and keeps its meta-KG backups there). The arax worker also caches the Biolink model and ARAX's Biolink lookup map in ./arax_dbs/biolink/, built at startup so it survives restarts (ARAX_BIOLINK_CACHE_DIR moves it).

So a new developer doesn't have to source these by hand, each worker can fetch its dataset on first startup. Two download mechanisms are supported, depending on where the dataset lives:

LMDB datasets (aragorn_omnicorp, score_paths) are fetched as a .tar.gz from a plain HTTP(S) URL and extracted in place. Add the matching variable to your root .env file:

OMNICORP_LMDB_URL=https://example.org/path/omnicorp_lmdb.tar.gz
PATHFINDER_EMBEDDINGS_URL=https://example.org/path/pathfinder_embeddings.tar.gz

The archive for each dataset should contain the expected files at its top level: curies.lmdb and shared_counts.lmdb for omnicorp, data.mdb (and lock.mdb) for the embeddings.

arax_pathfinder's sqlite databases are served as plain files over HTTPS, no credentials needed. The filenames embed a Knowledge Graph version that changes periodically, so only one variable needs updating when a new Knowledge Graph ships:

ARAX_PATHFINDER_TIER_VERSION=tier0-20260621

The ARAX blocked-concept list (general_concepts.json, fetched from GitHub) lands in this same directory, so it is downloaded once and then persists with the databases rather than being re-fetched by every new container. It is only fetched when absent — delete it from the volume to pick up an updated upstream list.

The ARAX port's data files work the same way as the pathfinder databases: plain files over HTTPS, one volume-mounted directory (ARAX_DBS_DIR, default arax_dbs), fetched on first startup. Each worker requests only the files it opens (ensure_arax_dbs([...]) in shepherd_utils/data_download.py), and arax_db_path(name) is the single place their on-disk paths come from. The tier-versioned filenames are filled from one variable, separate from pathfinder's:

ARAX_TIER_VERSION=tier0-20260621

By default each file is fetched from ARAX_DBS_BASE_URL/<filename>. These URLs are placeholders until the real files are published; a single file can be pointed elsewhere with its own override (ARAX_CURIE_TO_PMIDS_URL, ARAX_EXPLAINABLE_DTD_URL, ARAX_AUTOCOMPLETE_URL, ARAX_FDA_APPROVED_DRUGS_URL, ARAX_COHD_URL), and each filename can be changed with its ARAX_*_FILENAME setting. COHD stays on its KG2.8.0 build and the FDA pickle has no tier version, so ARAX_TIER_VERSION does not affect those two. The two databases the pathfinder already downloads (curie_ngd, tier0-info-for-overlay) are not duplicated here; ARAX workers that need them use arax_pathfinder_sqlite_paths(). The deployment notes below apply to ARAX_DBS_DIR in the same way.

On startup, each worker checks whether its files already exist in the volume-mounted directory. If they're missing and a URL is configured, it fetches them into that directory — which lives on the host, so the data persists across restarts and is only downloaded once. If the files are already present, or no source is configured, the download is skipped (production mounts this data out of band, so it's unaffected).

Downloads are bounded by DATASET_DOWNLOAD_TIMEOUT_SEC (default 60), which applies per socket operation rather than to the whole transfer — a large file downloads for as long as it needs, but a connection that opens and then stalls fails loudly instead of hanging worker startup.

ARAX's KP response cache

The arax worker caches every Retriever (KP) response ARAX's Expand gets, and Connect's PathFinder and xCRG results, as ARAX does (DEC-18 in docs/ARAX_PORT_BASELINE.md). The cache lives in Shepherd's Redis data store, so it is shared by every arax worker replica and survives restarts; the server lists it at /arax/status?mode=kp_cache (the UI's KP-cache view). A query with "query_options": {"bypass_cache": true} skips it. Each worker re-queries entries older than 6 h in the background, one replica at a time, and an entry is dropped 3 days after it was last requested.

ARAX_KP_CACHE_ENABLED=true               # false: never read or write the cache
ARAX_KP_CACHE_TTL_SEC=259200             # an entry's lifetime after its last request
ARAX_KP_CACHE_REFRESH_INTERVAL_SEC=60    # seconds between refresh passes; 0 turns the refresh off

To inspect or clear it, run ARAX's cacher CLI in the arax container, e.g. python -m shepherd_utils.arax.Expand.trapi_query_cacher --summarize (also --list, --dump_response ID, --delete_query ID, --delete_query_url_match URL, --refresh, --initialize_cache).

Mock ARAX data for local testing

Until the real files are published, shepherd_utils/arax_mock_data.py writes small mock versions in exactly their shape (same filenames, tables, columns and value formats), so the whole ARAX stack runs locally. Generate them into the volume-mounted directories before docker compose up; the startup check then finds them and skips the download:

python -m shepherd_utils.arax_mock_data --pathfinder
# writes ./arax_dbs/ (curie_to_pmids, ExplainableDTD, COHD, FDA drugs, autocomplete)
# and, with --pathfinder, ./arax_pathfinder_dbs/ (curie_ngd, tier0 overlay)

The content comes from a seed of about 30 real curies (e.g. MONDO:0005148 type 2 diabetes, CHEBI:6801 metformin, NCBIGene:5468 PPARG) and their edges. The values are made up (deterministically) but consistent: NGD comes from the PMID sets, and every xDTD prediction has explanation paths built from edges in its own edge mapping. To cover more nodes, add the knowledge graph of saved TRAPI responses, e.g. a Retriever response or one from /arax/query:

python -m shepherd_utils.arax_mock_data --pathfinder --force --from-trapi response1.json response2.json

The clinical-info overlay maps curies to OMOP ids through the live cohd.io API at query time, so the generator asks cohd.io for the same ids; with --no-network it makes ids up instead, and COHD then finds nothing at query time. Existing files are never replaced without --force. Use --out and --pathfinder-out to write somewhere other than ARAX_DBS_DIR and ARAX_PATHFINDER_DBS_DIR. Delete the mock files (or run with the real URLs into an empty directory) when switching to the real data, because a file that is already present is never downloaded.

Deploying these workers

The presence check is an exact match on the configured directory and the tier-versioned filenames. A deployment that mounts the data somewhere else, or whose ARAX_PATHFINDER_TIER_VERSION doesn't match the filenames on the volume, will not use the mounted copies — it will decide the dataset is missing and download it again, into whatever path the settings do point at. If that path isn't the mount, the files land on the container's writable layer and the pod is eventually evicted for exceeding its ephemeral storage. So when deploying, set the directory explicitly and confirm it resolves to your mount:

ARAX_PATHFINDER_DBS_DIR=/data/arax_pathfinder_dbs   # default is relative: resolved against /app

$ kubectl exec deploy/arax-pathfinder -- python -c \
    "from shepherd_utils.data_download import arax_pathfinder_sqlite_paths as p; print(*p(), sep='\n')"

Because general_concepts.json now shares that directory, the volume needs to be writable for the first startup that fetches it — or the file can be preloaded alongside the sqlite databases, after which the worker only ever reads it. A read-only mount with no preloaded copy fails at startup with a permission error rather than silently continuing.

Translator ARS

Shepherd also hosts a port of the NCATS Translator ARS (NCATSTranslator/Relay) at /ars/... -- the same /ars/api/submit / messages/<pk>?trace=y / get_status surface the Translator UI uses, backed by five workers (ars_fanout, ars_premerge, ars_merge, ars_watchdog, ars_notify) on the shared Redis Streams fabric instead of Celery/RabbitMQ, and ars_* Postgres tables instead of MySQL. ars_premerge validates each ARA response in a process pool; ars_merge folds each validated response into the query's merged message and post-processes the new version in the same pool child, draining every response that is ready for a query under one lock (the merge_message worker's pattern). Node annotation runs the biothings_annotator package in-process; ARS_ANNOTATION_MODE picks where: premerge (the default -- each ARA response is annotated in ars_premerge, in parallel and outside the merge lock), merge (upstream's placement, over every merged version), or off. Nodes that already carry a biothings_annotations attribute are never re-annotated. The package is configured by its own environment variables (SERVICE_PROVIDER_API_HOST, ANNOTATOR_QUERY_BACKEND), set on the ars_premerge and ars_merge containers. Each merged version's results are ordered by weighted Reciprocal Rank Fusion of the hosted ARAs' own rankings (Aragorn 0.9, ARAX 0.1, as upstream), configured on the ars_merge container by upstream's variables: ARS_RRF_ENABLED (default true), ARS_RRF_ARAGORN_WEIGHT (ARAX gets the rest) and ARS_RRF_C (default 40).

The ARS is de-federated: it talks only to the ARAs this same Shepherd deployment hosts (shepherd_utils/ars/aras.py -- Aragorn, ARAX, BTE), and every hop goes through the broker. A submitted query is fanned out by enqueueing each ARA's own worker task, and when an ARA pipeline finishes, finish_query hands the response back onto the ars.premerge queue. There is no SmartAPI registry lookup, no HTTP round trip to an ARA, and no HTTP result callback; the upstream /agents and /actors registry endpoints are not served. GET /ars/api/aras lists the hosted ARAs instead, with whether each is enabled for fan-out (ARS_ENABLED_ARAS, empty = all) and how many of its workers are currently alive, so it shows what the ARS can reach right now.

Behavior is pinned against the upstream codebase by a parity suite; see docs/ARS_PARITY_REGISTER.md for the invariants, the golden-regeneration procedure, and every documented deviation. Deployment knobs live in shepherd_utils/config.py under the "Translator ARS" block.

ARS response cache

The ARS keeps a whole-response cache in front of /ars/api/submit (shepherd_utils/ars/cache.py; design and rationale in docs/ARS_RESPONSE_CACHE_PLAN.md). There is one message tree per distinct query: a submit whose query graph is structurally identical to a completed earlier submit -- same content regardless of key order, null-vs-missing-vs-empty fields, list order, or the names chosen for query nodes/edges/paths -- is handed that tree's pk. The 201 body is the source parent's envelope, already Done, with the caller's own submit body as data exactly like a fresh parent; the client then fetches merged_version as usual. The merged message keeps the labels of whoever ran the query first and carries a log entry, written when it became the cached answer, saying it is served from the cache. A submit identical to a query still in flight is handed the running query's pk. Hits create no rows and no payloads and decompress nothing, so no ARA, merge or post-process worker runs and nothing is stored.

The cache lives in Postgres and holds no payloads of its own: it indexes a query-graph hash to the source tree's own durable payloads, and source trees backing a live cache generation are exempt from the ARS_DATA_RETENTION_DAYS payload purge (under disk pressure they are evicted last, with their cache entries). Budget disk for one tree per distinct query per generation. Entries have no TTL; old pks keep working after an invalidation until their payloads age out (or forever if retained).

ARS payload storage and retention

Every ARS message keeps a durable copy of its payload once it finishes, so results stay readable after the Redis copy expires. Set ARS_PAYLOAD_DIR to keep those copies as files on a volume instead of in Postgres -- a single stress run of large merged results otherwise fills a small Postgres volume, and space freed inside Postgres is never returned to the disk. The directory must be shared by the server, finish_query and every ars_* worker that writes or reads payloads (ars_fanout, ars_premerge, ars_merge, ars_watchdog), so in Kubernetes it is a ReadWriteMany volume; compose mounts ./ars_payloads. Copies already in Postgres stay readable. Unset, payloads stay in Postgres.

Set PAYLOAD_S3_BUCKET to keep the same store in an S3 bucket instead (with both set, the bucket is used). Objects keep the directory layout as their keys, under PAYLOAD_S3_PREFIX, so a directory can be copied into a bucket as it is. Credentials and region come from the standard AWS chain (the pod's IAM role); PAYLOAD_S3_ENDPOINT_URL points it at an S3-compatible service. A bucket cannot fill, so the disk-pressure eviction below does not apply to it.

Archived ARA responses

The same store archives every ARA response. When a query's pipeline finishes, finish_query keeps the response exactly as the pipeline left it -- before a callback envelope or the ARS's premerge touches it -- for queries submitted directly and ones the ARS dispatched alike. GET /response/{id} falls back to that copy once the Redis one has expired, so an ARA's answer stays readable for ARA_RESPONSE_RETENTION_DAYS rather than REDIS_TTL.

Each query is one self-contained object under ara/, beside the ARS payloads: three zstd frames back to back -- the stored response, the query (which carries the parameters a served response repeats), and a tail with the query's final status, description and logs. The first two are the compressed bytes Redis held, copied without being decompressed, so archiving costs finish_query only the compressed size of a response. The routes take the object apart; clients get an ordinary TRAPI Response. To look at one directly: zstd -dc <file> | jq -s . prints the three documents.

Archiving is best-effort: a failed write is retried (for at most 30 seconds), logged, and the query is delivered regardless. It needs the store configured for finish_query as well as the server (the same volume mounted, or the same bucket settings); with no store configured, nothing is archived.

What stays in Redis

With a store configured, Redis only holds a payload while it is hot. Without one, none of this applies: every payload stays in Redis for REDIS_TTL and every ARS message keeps a durable copy in Postgres.

• A finished query. Once finish_query has archived it, its query, response and logs are left REDIS_HOT_TTL (default 30 minutes) in Redis instead of the rest of REDIS_TTL. If the archive write failed they keep the full REDIS_TTL, since Redis is then the only copy.
• A finished ARS tree. When the parent completes, every Redis copy in the tree gets REDIS_HOT_TTL: long enough for clients still polling it. A later read comes from the cold store and re-warms Redis for the same window.
• ARS children. A child and its ARA's response are one entry: the child's pk is the response id its ARA's query was given. The ARS never writes it. A child's payload is the ARA's own answer, as the ARA would deliver it (with the query's parameters and logs, without the ARS's score normalization or node annotation), read from Redis while hot and from the archive afterwards; a child that is still running has no payload yet. What premerge makes of a child is the merge's working copy (ars:premerged:<pk>), deleted once the merge has attempted it. A child whose ARA never answered keeps a small durable copy of whatever its entry held.
• Merged versions. Only the newest keeps a durable copy; each new one drops the copy of the version it supersedes. Earlier entries in merged_versions_list are readable only while the tree is hot.
• Lookup callbacks. A callback's payload is deleted as soon as it has been folded into its query's response (this one applies with or without a store). A batch given up on as unmergeable keeps its payloads until REDIS_TTL, as evidence.

REDIS_TTL (default 4 hours) remains the TTL of anything still in flight or never finished; it only has to outlast the longest query.

The defaults are sized for a local stack, and a deployment sets its own:

Setting	Default	For a deployment
REDIS_TTL	14400 (4 hours)	How long unfinished work stays in Redis to be looked at. Without a cold store it is also how long a finished response is readable at all, so set it much longer there (it used to be 3 days).
REDIS_HOT_TTL	1800 (30 minutes)	How long finished work stays in Redis after its cold copy exists.
REDIS_MAXMEMORY (broker container)	unset: redis.conf's 2gb	The broker's maxmemory, e.g. 25gb. Keep it at about half the container's memory limit and a third of its volume.
REDIS_MEMORY_LIMIT (monitor)	unset	The broker container's memory limit, for the monitor's footprint alert.

The ARS watchdog keeps it bounded, always working on whole finished trees (a parent with every child and merged version under it) so a tree is never left half there, and never touching retained trees:

Setting	Default	What it does
ARS_PAYLOAD_EVICT_HIGH_PCT / ARS_PAYLOAD_EVICT_LOW_PCT	80 / 65	Checked every pass (ARS_WATCHDOG_INTERVAL_SEC): once the payload volume is this full, evict archived ARA responses (oldest first) and then tree payloads -- uncached oldest first, then cache-backed least recently used (dropping their cache entries) -- until it is back under the low mark.
ARS_DATA_RETENTION_DAYS	30	Hourly: drop the payloads of trees finished this long ago. Rows stay.
ARS_ROW_RETENTION_DAYS	90	Hourly: delete the rows of trees finished this long ago.
ARA_RESPONSE_RETENTION_DAYS	30	Hourly: delete the archived responses of directly submitted queries written this long ago. Under disk pressure they are evicted, oldest first, before any tree.

Every object in the store has one owner. An ARS child's pk is its ARA's response id, so the archive of that response belongs to the child's tree: it is kept as long as the tree keeps its payloads (ARS_DATA_RETENTION_DAYS, a retained tree, a live cache entry) and dropped with them, and its size counts toward what evicting the tree frees. The archive of a query submitted directly has no tree, and is the only kind ARA_RESPONSE_RETENTION_DAYS applies to. Ownership is decided by asking Postgres, so it works the same on a volume and in a bucket; for that reason do not put a bucket lifecycle rule on ara/ (it would delete a retained tree's answers), or set one far longer than every retention above as a backstop.

The rest of the database is aged out by the monitor's janitor (QUERY_RETENTION_DAYS for queries and callbacks, 30 days of monitor history). The monitor alerts when the payload volume passes 85% / 95%, which means eviction isn't keeping up -- typically a volume filling with retained trees, which the alert reports.

Controls:

• parameters.bypass_cache: true -- run fresh; neither read nor write the cache.
• parameters.overwrite_cache: true -- run fresh and point the cached entry for this query at the result.
• Whole-cache invalidation (e.g. after a knowledge-graph release): python scripts/ars_cache.py invalidate --reason "..." from any host that reaches Postgres, or POST /ars/api/cache/invalidate with Authorization: Bearer $ARS_ADMIN_TOKEN (the route is disabled until ARS_ADMIN_TOKEN is set). scripts/ars_cache.py stats|show|evict|key and GET /ars/api/cache cover inspection, including per-entry hit counts.
• ARS_CACHE_ENABLED=false turns the whole thing off.

Because pks are shared, retain/<pk> retains the tree for every reader and block/<pk> edits it for every reader; the row's timestamp and name are the first submitter's.

Worker

Each worker is it's own separate docker container. It spins up and begins to watch a central message broker for tasks to work on. Once it gets a task, it can do that task either synchronously, asynchronously, on a separate process etc. based on its individual resource requirements.

Each worker has access to a shared utilities library that aids in db and message broker interaction as well as other functions that are common across ARAs. Check the shared function library before writing a new function that you think other ARAs might also want to use.

Worker tuning & graceful shutdown

Every worker draws its tasks through shepherd_utils.shared.get_tasks, so the following behavior applies to all of them:

• Concurrency (TASK_LIMIT) — each worker declares a default in-process concurrency, but it can be overridden per deployment with the TASK_LIMIT environment variable (each worker is its own container, so a single TASK_LIMIT per Deployment is unambiguous). No code change or rebuild needed.
• Graceful drain on shutdown — on SIGTERM/SIGINT (Kubernetes sends SIGTERM on every rollout, scale-down and node drain) a worker stops pulling new tasks, waits up to WORKER_DRAIN_TIMEOUT_SEC (default 30s) for in-flight tasks to finish, writes a clean-shutdown marker the monitor reads (so the event is classified as a graceful scale-down rather than a crash), then exits. Tasks that don't finish in the window are left in the stream for Redis reclaim. Set the deployment's terminationGracePeriodSeconds comfortably above WORKER_DRAIN_TIMEOUT_SEC.
• Whole-query timeout budget (QUERY_TIMEOUT_SEC) — see below.

Query timeout budget

The ARS and the other external callers stop waiting for a Shepherd query after about five minutes, and the synchronous /query endpoint gives up around the same point. Work done past that is work nobody receives — it only takes worker slots (and process-pool children) away from queries that can still be answered.

So the server stamps each query with an absolute deadline at intake, and that deadline travels with the task from operation to operation. Every worker checks it as it picks a task up (in shepherd_utils.shared.get_tasks, on both freshly delivered and reclaimed messages). If the budget is spent the worker does not run the operation: it drops the rest of the workflow and routes the query straight to finish_query, which ends it the way any other query ends — state COMPLETED in Postgres with a TIMEOUT status, callback rows reaped, logs saved (including the line explaining why the response is partial), and whatever was gathered POSTed to the callback URL. A synchronous caller therefore gets a partial response instead of waiting out its own timeout for nothing.

Setting	Meaning
QUERY_TIMEOUT_SEC	The budget, in seconds (default 300). 0 disables deadlines entirely — tasks then run however old they are, as they did before.

A client that explicitly asks to wait longer (TRAPI parameters.timeout) is not cut short: the larger of the two wins. Two streams are exempt — finish_query, which is the wrap-up, and merge_message, which folds in callbacks an upstream service has already done the work for. Tasks with no deadline (a payload enqueued before this shipped) are never expired, so a rollout is safe mid-flight.

This is a fast path that settles a query at the moment it goes over. The monitor's abandoned-query reaper (MONITOR_ABANDONED_QUERY_SEC, default 600s) remains the backstop for queries that go over without any worker picking a task up for them — e.g. one whose driving worker died with nothing left in a stream.

Response size limits

A TRAPI response is held in memory as a Python object tree several times the size of its JSON, by every worker that touches it, and merge_message reloads the whole accumulated response on every pass. One big enough is OOM-killed and then retried on the next pod, which dies the same way. Two caps bound that, and both default to 250M — tune them per deployment (Helm) from the pod's memory limit (0 turns a cap off; a value that doesn't parse also reads as 0, and is logged as a warning):

Setting	Meaning
CALLBACK_MAX_REQUEST_SIZE	Largest /callback body the server accepts (e.g. 100M). Checked on the wire before the body is buffered, and on the decompressed size for a zstd-encoded body. Over it: 413, and the query is failed.
MAX_RESPONSE_SIZE	Largest uncompressed response any worker will load (e.g. 300M). Read from the stored blob's zstd header, so nothing is loaded to check it. Every standard worker checks before loading; finish_query checks before delivering; merge_message treats it as the budget for the accumulated response and sizes each merge pass to stay within it.

Sizing: decoding expands the uncompressed JSON roughly 6-7x in memory, and a merge pass can overshoot the budget by at most one callback, so for the merge pod keep 2 x MAX_RESPONSE_SIZE x 7 under its memory limit (about 300M for a 5Gi pod). The callback endpoint records each payload's size on its trace span (callback.payload_bytes), which is the number to tune from.

A breach never yields a partial answer. The whole response is discarded (shepherd_utils/response_limit.py): the stored response is replaced by an empty message whose status/description say it was too large, the query's outstanding callbacks are cleared so the workflow runs on to finish_query promptly, and the query finishes with status RESPONSE_TOO_LARGE (HTTP 500 on the sync endpoint, Failed from /asyncquery_status). Every breach is logged at CRITICAL with a RESPONSE_TOO_LARGE marker, both in the worker's output and in the query's own log list, so it shows up in the delivered response too.

Two more bounds, both always on:

• Per-query logs are capped (QUERY_MAX_LOG_ENTRIES, default 10000, 0 disables). A query's log list is loaded whole by everything that reads it, including the status endpoint the ARS polls, so it is trimmed to the newest N entries on every append. The last thing logged, such as the CRITICAL line a failed query ends with, always survives.
• merge_message only merges callbacks the query is still waiting for. A callback's row in the callbacks table is cleared when the lookup times out, when the query finishes, or once it is merged; a ready callback with no row is dropped instead of being merged into (and logging against) a response that has already been delivered.

Independently of the caps, merge_message keeps a per-query crash counter in Redis: each merge pass is counted before it starts and cleared when it returns, so a pass that took the pod down with it is still counted on the next pod. Once a response's merge has crashed MAX_TASK_DELIVERIES (default 3) times without completing, the query is failed RESPONSE_TOO_LARGE instead of being retried. This is the always-on backstop for the case the caps were set too high to catch.

A finished response is sealed

Tasks are delivered at least once, so work for a query can still be running after the query has finished: a merge pass already in flight, a late callback, a redelivered or reclaimed task, a slow operation that outlived its query. Once finish_query has claimed a query it seals the response (a response_sealed:<response_id> key next to it in the data store), after its own last write -- the too-large rewrite, when there is one -- and before it reads the response to deliver, hand to the ARS or archive. What the caller gets is then exactly what stays stored.

Every write of a stored message checks the seal in the same Redis transaction as the write (WATCH/MULTI/EXEC, retried on a conflict), so a writer cannot slip in between the check and the seal. A refused write leaves the response untouched and raises ResponseSealedError; the task lifecycle (and merge_message, which drops the query's ready callbacks) logs it once at WARNING with the worker and query, ACKs the task, and stops -- the query is not handed to its next operation, and not to finish_query again. Standard workers also check the seal before running an operation, so a task for a finished query is skipped rather than run for nothing.

The seal lives REDIS_TTL from the moment it is set, so it always outlives the response (whose TTL was set by an earlier write, and is only ever shortened after archiving). The initial response written at intake comes before any seal; the abandoned-query reaper does not seal (a late finish_query may still claim and deliver an abandoned query), and every other way a query ends -- a failed or timed-out operation, a response discarded as too large, the post-restart recovery -- goes through finish_query and is sealed there.

Query intake limits

/{ara}/query and /{ara}/asyncquery are unauthenticated, so intake bounds what a single request can cost:

Setting	Meaning
QUERY_MAX_REQUEST_SIZE	Largest query body accepted (default 100MB, 0 disables). Checked on the declared Content-Length before anything is read, then on the streamed total. Over it: 413.
REQUEST_BODY_IDLE_TIMEOUT_SEC	How long a query or /callback body may go without sending a byte (default 30, 0 disables). Over it: 408. This is what stops a client that declares a large body and then stalls (slowloris); a size cap never trips on a stream that stops. It is an idle gap, not a total, so a large body that keeps arriving is never cut off. An ingress read timeout is still worth setting in front.
CALLBACK_ALLOWED_HOSTS	Hosts an /asyncquery callback may name even though they resolve to a private, loopback or link-local address: comma-separated hostnames, IPs or CIDRs. Default empty.

The callback of an async query must be http/https and must resolve only to public addresses, unless the host or address is in CALLBACK_ALLOWED_HOSTS; cloud metadata addresses (169.254.169.254 and friends) are refused even when allowlisted. Anything else is refused at intake with 422, and finish_query checks again before it POSTs (shepherd_utils/callback_url.py). Production callers (the ARS and other Translator services) call back to public hosts and need nothing set. compose.yml allowlists host.docker.internal and the docker bridge range for the local scripts/run_query.py --async flow; a Test deployment whose callers sit on a private network has to list them, on both the server and finish_query.

Kubernetes sizing (Helm)

Production limits live in the Helm chart, not in compose.yml (which is dev-only). Recommended starting point for finish_query, which holds whole decompressed TRAPI payloads in memory while POSTing async callbacks (as JSON bytes: it never decodes a response, the TRAPI 2.0 envelope is written around the stored bytes):

Setting	Value
resources.requests	cpu: 500m, memory: 1Gi
resources.limits	cpu: "2", memory: 4Gi
TASK_LIMIT	32 (down from the in-code default of 100)
terminationGracePeriodSeconds	35

Scale throughput with replicas / an HPA (on CPU or queue depth) rather than a single large pod. On Kubernetes the memory limit (OOMKilled + restart) plus regular rollouts already recycle pods, so leaked-resource cleanup comes for free — add an RSS-based livenessProbe only if the monitor shows OOMKills in practice. CPU-bound pool workers (merge_message, score_paths, arax_rank, aragorn_score, aragorn_omnicorp) size their process pools from the in-code default, so raising TASK_LIMIT for those only deepens the intake queue rather than adding parallelism.

Monitoring Redis

The monitor (workers/monitor/) watches Redis per instance. By default the four Redis roles (broker, data, lock, logs) are one server, and the monitor reports and alerts on it exactly as it always has. When a role is given a Redis of its own (REDIS_<ROLE>_HOST, or REDIS_<ROLE>_SENTINEL_MASTER behind Sentinel), every distinct server is polled with one INFO and evaluated on its own:

• Snapshot. /api/snapshot keeps redis (the instance serving the broker role) and adds redis_instances, keyed by instance (host:port, or sentinel:<master>), each with the roles it serves. An instance that can't be reached reports an error in its own entry. The dashboard shows a block per instance when there is more than one.
• Alert names. The broker role's instance keeps today's names (redis_memory_high, redis_evicting, redis_restarted, ...). Any other instance's alerts append it -- redis_memory_critical@shepherd_data:6379 -- and say which instance and roles in the text. Each instance has its own grace periods and cooldowns, so one never masks another.
• History. The broker role's instance keeps the redis:<metric> series; others record redis@<instance>:<metric>.
• Memory. REDIS_MEMORY_LIMIT is the container limit of the instance serving the broker role. For a role with a Redis of its own set REDIS_DATA_MEMORY_LIMIT (or REDIS_LOCK_MEMORY_LIMIT / REDIS_LOGS_MEMORY_LIMIT); without it that instance is still checked against its maxmemory, just not against a container limit.
• Restarts and failovers. A restart of the data role's instance finishes, as errors, the queries whose payloads it lost; a restart of the broker role's instance finishes the ones whose next task it lost; a shared Redis does both. Behind Sentinel a failover is handled the same way, since writes the old primary had not replicated are gone, and is reported as redis_failover where it can be told from a restart.

Alerts added for multi-server and replicated deployments (all automatic, all no-ops on a single unreplicated Redis unless noted; thresholds in shepherd_utils/config.py, documented in workers/monitor/monitor_alerts.yaml):

Alert	Fires when
redis_failover	Sentinel promoted a replica; unreplicated writes are lost and the recovery sweep runs
redis_replicas_lost / redis_replicas_recovered	an instance has fewer connected replicas than it had (critical at zero) for MONITOR_REDIS_REPLICA_GRACE_SEC
redis_replica_lagging / redis_replica_caught_up	a replica is syncing, silent for MONITOR_REDIS_REPLICA_LAG_SEC, or MONITOR_REDIS_REPLICA_LAG_BYTES behind
redis_eviction_window_short / redis_eviction_window_recovered	an instance near its maxmemory would evict keys younger than the longest query budget (any deployment with maxmemory set)

redis_persistence_failing now says which save failed: a failed AOF write ("writes are not being persisted", critical), a failed AOF rewrite ("no data lost yet, but the AOF will keep growing", warning), or a failed RDB snapshot (critical).

Message Broker Streams

Shepherd uses Redis Streams for its message broker. More info on Redis Streams can be found here Each worker type listens to its own message stream/queue. The shared workers can just use their Translator workflow name for their stream name. For workers that share the same workflow but are for different ARAs need their own custom stream name. The current convention is to use the format {ara_name}.{workflow_name}. Multiple workers of the same type can be in the same GROUP, and redis will make sure not to give out the same task to more than one worker in that group. The current convention is to call the group consumer.

Redis roles

Shepherd uses Redis for four jobs: the broker (task streams, worker heartbeats, the monitor's state), data (query, response and callback payloads while they are hot), locks, and logs. By default they are one Redis, REDIS_HOST, told apart by database number (0-3).

Each role can be given a Redis of its own, so the small, latency-critical broker and locks stop sharing a command thread and a maxmemory with multi-megabyte payload transfers. Set any of:

Setting	Role
REDIS_BROKER_HOST	streams, heartbeats, monitor state
REDIS_DATA_HOST	payloads
REDIS_LOCK_HOST	locks
REDIS_LOGS_HOST	query logs

Each has an optional ..._PORT and ..._PASSWORD, defaulting to REDIS_PORT and REDIS_PASSWORD. A role keeps its database number wherever it lives, so moving one is a configuration change only. Every process must be given the same values (the server and every worker), and a role must be moved while nothing is in flight: its existing keys are not migrated.

A reasonable split is the broker and locks together on one small Redis and the data role on another. They want different settings:

• Broker and locks: maxmemory-policy noeviction. Nothing there may be evicted (a task, a heartbeat, a lock), and the dataset is small. This is the Redis to make highly available first.
• Data: maxmemory-policy volatile-ttl, as shepherd_broker/redis.conf has it, so the oldest payloads go first if the cap is ever reached. With a cold store configured it only holds payloads while they are hot (see "What stays in Redis").

Redis high availability (Sentinel)

A single Redis is a single point of failure: if its node dies, every query in flight is lost and nothing recovers until it is back. With Redis Sentinel (one primary, two replicas, three Sentinels, each on a different node), a replica is promoted in about 15-30 seconds.

Point Shepherd at the Sentinels instead of at Redis:

REDIS_SENTINELS=sentinel-0:26379,sentinel-1:26379,sentinel-2:26379
REDIS_SENTINEL_MASTER=shepherd      # the name the Sentinels monitor it under
REDIS_PASSWORD=...                  # the Redis password, as before
REDIS_SENTINEL_PASSWORD=...         # only if the Sentinels require auth

Clients then ask Sentinel for the current primary and follow it across a failover; REDIS_HOST is not used. With roles split, a role with its own Redis names its master with REDIS_<ROLE>_SENTINEL_MASTER (watched by the same Sentinels). For each role the most specific setting wins: its own sentinel master, then its own host, then REDIS_SENTINEL_MASTER, then REDIS_HOST. Unset, nothing changes: compose and the tests connect directly.

What a failover costs:

• Seconds of errors. Commands fail while no primary is reachable. Workers retry as they do through a broker restart.
• The last writes. Replication is asynchronous, so writes the replica had not received are lost, as are writes the old primary accepted in the moment before it was demoted. The monitor treats a failover like a restart and fails the queries whose payloads are gone.
• Possibly a lock. A lock taken just before the failover may not have reached the replica, so for a moment two workers could merge the same response. This is rare and accepted.

Redis settings to use on each node, on top of shepherd_broker/redis.conf:

Setting	Value	Why
repl-backlog-size	1gb	A replica that briefly disconnects can catch up without a full resync.
client-output-buffer-limit	replica 4gb 2gb 120	The default 256mb overflows with large payloads during a full sync, which would restart the resync forever.
appendonly	yes (already)	A primary without persistence that restarts empty before a failover is copied to its replicas, wiping the dataset.
masterauth	the Redis password	So a demoted primary can sync from the new one.

Sentinel: quorum 2, down-after-milliseconds 10000, failover-timeout 60000. Use a maintained Sentinel Helm chart or operator rather than hand-rolling the failover, and spread the pods with anti-affinity.

Creating your own ARA

Creating a brand new ARA is fairly straightforward. Here are the steps to create a basic ARA that performs all the necessary Translator operations:

• Copy the workers/example_ara folder. This will be the main entrypoint to your new ARA.
  • Towards the top of that file, replace the STREAM variable value with your ARA name.
  • Within the example_ara function in that file, replace the workflow list with your ARA's workflow. This could include analyzing the message to determine a pertinent workflow.
  • Note: If using operations not in the shared workers, your workflow operation ids need to reflect the STREAM name of your custom operation workers. This is how the task get passed to your operation worker.
• (If needed) Copy the workers/example_lookup folder. This will be your ARA's lookup operation.
  • Towards the top of that file, replace the STREAM variable value with {ara_name}.lookup.
  • Within the example_lookup function in that file, replace the contents with your ARA's lookup logic.
• (If needed) Copy the workers/example_score folder. This will be your ARA's score operation.
  • Towards the top of that file, replace the STREAM variable value with {ara_name}.score.
  • Within the example_score funciton in that file, replace the contents with your ARA's scoring logic.
• If you have other custom operations you want to perform that are in your workflow above, pick a similar folder to copy and adjust the code inside to fit your needs.
• If you want to use shared workers (i.e. workers/sort_results_score, workers/filter_results_top_n), you don't need to do anything other than include them in your workflow. They will automatically pick up your query and pass it along like your other operations.
• Open the compose.yml file in the root directory, and for each ARA folder you created, add a service (or copy an existing one), and make sure that the container_name and build/dockerfile reflect you worker names and the path to your worker Dockerfile (keep the copied build/additional_contexts entry: worker Dockerfiles build FROM shepherd_base)
• Run Shepherd with docker compose up --build

Testing your ARA

Shepherd uses pytest and tox for local test running and GitHub Actions. To run tests, simply activate your virtual env, run:

• pip install tox and then run:
• tox

This will run all the tests and then also provide code coverage.

If you would like to run local integration tests, use scripts/run_query.py to send a query to your ARA (or to the ARS) through a running Shepherd. Pick where it goes with a target (<ara>-<env>, e.g. aragorn-local, or a URL), the query type with -t, and the curies with -c:

python scripts/run_query.py --list                                   # targets, query types, default curies
python scripts/run_query.py aragorn-local -c MONDO:0005148           # MVP1: what treats type 2 diabetes
python scripts/run_query.py arax-local -t mvp2-decreased -c NCBIGene:1017
python scripts/run_query.py aragorn-local -t pathfinder -c "CHEBI:45783~MONDO:0004979"
python scripts/run_query.py ars-local -t mvp1 --limit 3              # submit to Shepherd's ARS and poll
python scripts/run_query.py arax-local --async --callback-host 172.17.0.1   # /asyncquery + callback

Responses are saved under responses/ and timings under benchmark_metrics.json; --help lists every option.