diff --git a/CHANGELOG.md b/CHANGELOG.md index 4877b6418..a5e077b9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - gateway: `Admin.SetInstanceReady` takes a CVM instance out of its app's load-balancing rotation without stopping it; instance-id routing stays open so the instance can still be investigated, and the setting survives re-registration - gateway: application-level health polling. The gateway asks each CVM's guest agent (new `Worker.Health` RPC) whether every container declaring a Compose `healthcheck` is healthy, and keeps instances that say no -- or that have not answered since registering -- out of app-id load balancing. Legacy images, apps without healthchecks, and apps where no instance reports healthy are unaffected - guest-agent: container health also covers the `nerdctl-compose` runner, read through `nerdctl inspect` (its output is Docker-compatible). Requires nerdctl >= 2.3.1 for Compose `healthcheck:` to be honoured; the mkosi backend is pinned to 2.3.5 +- app-compose: optional `health_check` (program path, args, interval, timeout) run periodically by the guest agent. Overrides container-derived health for any runner, and is the only way the `bash` runner — or any future non-container runner — can report health. Documented in `docs/app-health-checks.md` ## [0.5.5] - 2025-10-20 diff --git a/docs/app-health-checks.md b/docs/app-health-checks.md new file mode 100644 index 000000000..abd144fe3 --- /dev/null +++ b/docs/app-health-checks.md @@ -0,0 +1,133 @@ +# Application health checks + +The gateway keeps an instance out of its app's load-balancing rotation while the +application is not able to serve. This page describes where that verdict comes +from and how an app influences it. + +## Why this exists + +A CVM registers with the gateway from `dstack-util setup`, during boot. +`app-compose.service` is ordered *after* `dstack-prepare.service`, so at the +moment of registration the application's containers do not exist yet — the image +may still be pulling. Without a health signal the only thing standing between a +freshly booted instance and a real request is the WireGuard handshake, which +says the tunnel is up and nothing about whether anything is listening behind it. + +## Where the verdict comes from + +The guest agent answers `Worker.Health`; the gateway polls it. Sources are +consulted in this order: + +1. **`health_check` in `app-compose.json`** — if declared, it decides, for every + runner. +2. **Container healthchecks** — for `docker-compose` and `nerdctl-compose`, + every container that declares a Compose `healthcheck` must be reporting + healthy. +3. **Nothing** — the app is treated as healthy. This is where an app lands when + it declares no probe and none of its containers declare a healthcheck. + +### Container healthchecks + +Nothing to configure beyond what a Compose file already says: + +```yaml +services: + web: + image: my-app:1.0 + healthcheck: + test: ["CMD-SHELL", "curl -fsS localhost:8080/healthz || exit 1"] + interval: 10s + timeout: 3s + retries: 3 +``` + +Only containers that declare a `healthcheck` are judged. A container without one +is not evidence either way — the app has not said how to test it — so it is +skipped rather than counted as passing. `starting` counts as *not* healthy: that +state is precisely the boot window this exists to keep traffic out of. + +Finding **no containers at all** is not healthy either. A runtime that answers +while having nothing to show means Compose has not got as far as creating +anything. + +The `nerdctl-compose` runner needs **nerdctl >= 2.3.1** for Compose +`healthcheck:` to be honoured; earlier versions parse the field and discard it. +Container presence is still detected on older versions, so the boot-window gate +works regardless. + +### `health_check` in `app-compose.json` + +For the `bash` runner there are no containers to inspect, and an app may anyway +want to answer the question itself. Declare a program and the guest agent runs +it on a timer: + +```json +{ + "manifest_version": 1, + "name": "my-app", + "runner": "bash", + "bash_script": "/opt/my-app/run.sh &", + "health_check": { + "path": "/opt/my-app/healthz.sh", + "args": ["--strict"], + "interval_secs": 10, + "timeout_secs": 5 + } +} +``` + +| Field | Default | Meaning | +|---|---|---| +| `path` | required | Program to run. Executed directly, **not** through a shell | +| `args` | `[]` | Arguments, passed verbatim | +| `interval_secs` | `10` | Seconds between runs | +| `timeout_secs` | `5` | A run that exceeds this is killed and counted as a failure | + +Exit status 0 means healthy; anything else, including a timeout or a program +that cannot be executed, means not. The exit code and up to 512 bytes of stderr +are reported to the gateway for logging. + +`path` is not a shell command line: nothing in it is word-split, glob-expanded +or variable-substituted. If you need shell semantics, point it at a script. + +Declaring `health_check` **overrides** container healthchecks, so a Compose app +that wants one whole-application answer can give one instead of having every +container judged separately. + +Before the first run completes, the app reports **not** healthy. Registration +happens long before the application is up, so "has not run yet" must not read as +a pass. + +## How the gateway uses it + +- Health only gates **app-id** routing. Addressing an instance directly by + instance id is never gated, so an unhealthy instance stays reachable for + investigation. +- **If no instance of an app reports healthy, the gateway routes to all of them + anyway.** Health is inference and can be wrong — a misconfigured probe, an + agent that died, a whole fleet failing for a reason unrelated to the app. + Blackholing an app on that basis is worse than sending traffic to instances + that might be fine. +- Health is a *per-node observation*. Each gateway node polls for itself and + does not share the result, the same way each node reads its own WireGuard + handshakes. +- An agent that predates `Worker.Health` is never polled and is always treated + as healthy, so older guest images keep serving. + +Operators can also take an instance out of rotation by hand with +`Admin.SetInstanceReady`, independently of health. That one is an instruction +rather than an inference, so it does **not** fail open: gating every instance of +an app means the app refuses new connections. + +Polling is configured on the gateway: + +```toml +[core.proxy.health_check] +enabled = true +interval = "5s" +timeout = "2s" +concurrency = 16 +``` + +`enabled = false` restores the behaviour from before health polling existed: +every instance stays eligible regardless of what it reports. diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index 465ba815b..ff6818145 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -225,6 +225,14 @@ pub struct AppCompose { skip_serializing_if = "Vec::is_empty" )] pub init_script: Vec, + /// An app-declared health probe, run periodically by the guest agent. + /// + /// When present it replaces whatever the runner would otherwise report -- + /// including the container runners, whose per-container `healthcheck` + /// declarations it overrides. For a runner that has no containers to + /// inspect (`bash`) it is the only way to report health at all. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub health_check: Option, #[serde(default)] pub public_logs: bool, #[serde(default)] @@ -604,6 +612,41 @@ pub struct PortAttrs { pub pp: bool, } +/// A health probe the app declares in `app-compose.json`. +/// +/// The guest agent runs it on its own schedule and caches the verdict; the +/// gateway reads the cache through `Worker.Health`. Running it locally on a +/// timer rather than on each gateway poll keeps the two cadences independent: +/// several gateway nodes polling the same instance must not turn into several +/// executions of the app's probe. +#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] +pub struct HealthCheck { + /// Path of the program to run. + /// + /// Executed directly, not through a shell, so nothing here is word-split, + /// glob-expanded or variable-substituted. Put arguments in `args`; if you + /// need shell semantics, point this at a script. + pub path: String, + /// Arguments passed to `path`, verbatim. + #[serde(default)] + pub args: Vec, + /// Seconds between runs. + #[serde(default = "default_health_interval_secs")] + pub interval_secs: u64, + /// Seconds a single run may take before it is killed and counted as a + /// failure. A probe that hangs is not a probe that passes. + #[serde(default = "default_health_timeout_secs")] + pub timeout_secs: u64, +} + +fn default_health_interval_secs() -> u64 { + 10 +} + +fn default_health_timeout_secs() -> u64 { + 5 +} + fn default_true() -> bool { true } diff --git a/dstack/guest-agent/src/app_health.rs b/dstack/guest-agent/src/app_health.rs new file mode 100644 index 000000000..6c287671d --- /dev/null +++ b/dstack/guest-agent/src/app_health.rs @@ -0,0 +1,245 @@ +// SPDX-FileCopyrightText: © 2024-2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! The health probe an app declares for itself in `app-compose.json`. +//! +//! [`super::container_health`] can only judge apps whose containers declare a +//! Compose `healthcheck`, which leaves out the `bash` runner entirely -- it has +//! no containers to inspect -- and any runner added later. This is the hook for +//! those: the app names a program, the guest agent runs it on a timer, and the +//! exit status is the verdict. +//! +//! Run on a local timer rather than on each gateway poll, for two reasons: a +//! fleet of gateway nodes polling the same instance must not multiply into that +//! many executions of the app's probe, and the probe's cadence belongs to the +//! app rather than to whatever interval the operator configured on the gateway. + +use std::sync::{Arc, RwLock}; +use std::time::Duration; + +use dstack_types::HealthCheck; +use or_panic::ResultOrPanic; +use tokio::process::Command; +use tracing::{debug, info}; + +use crate::container_health::{HealthReport, UnhealthyContainer}; + +/// Name reported in place of a container, so an operator reading gateway logs +/// can tell an app-declared probe from a container that failed its healthcheck. +const PROBE_NAME: &str = ""; + +/// How much of a failing probe's stderr to keep. Bounded so a probe that dumps +/// a stack trace, or an HTML error page, cannot flood the gateway's logs. +const MAX_REASON_BYTES: usize = 512; + +/// The latest verdict, refreshed on its own schedule. +pub(crate) struct AppHealthProbe { + latest: RwLock, +} + +#[derive(Clone)] +struct Verdict { + healthy: bool, + reason: String, +} + +impl AppHealthProbe { + /// Start the probe loop and hand back the handle the RPC reads. + pub(crate) fn spawn(check: HealthCheck) -> Arc { + info!( + "app declares a health check: {} (every {}s, {}s timeout)", + check.path, check.interval_secs, check.timeout_secs + ); + let probe = Arc::new(Self { + latest: RwLock::new(Verdict { + healthy: false, + // Same rule the gateway applies to an instance that has + // registered but not yet answered a poll: not having run is + // not a pass. Registration happens long before the app is up. + reason: "health check has not run yet".to_string(), + }), + }); + let task = probe.clone(); + tokio::spawn(async move { + // Zero would busy-loop on the app's behalf. + let interval = Duration::from_secs(check.interval_secs.max(1)); + loop { + let verdict = run_once(&check).await; + debug!( + "health check verdict: healthy={} {}", + verdict.healthy, verdict.reason + ); + *task.latest.write().or_panic("app health lock poisoned") = verdict; + tokio::time::sleep(interval).await; + } + }); + probe + } + + /// The cached verdict, in the shape the RPC reports. + pub(crate) fn report(&self) -> HealthReport { + let verdict = self + .latest + .read() + .or_panic("app health lock poisoned") + .clone(); + if verdict.healthy { + return HealthReport { + healthy: true, + unhealthy: vec![], + }; + } + HealthReport { + healthy: false, + unhealthy: vec![UnhealthyContainer { + name: PROBE_NAME.to_string(), + status: verdict.reason, + }], + } + } +} + +async fn run_once(check: &HealthCheck) -> Verdict { + let timeout = Duration::from_secs(check.timeout_secs.max(1)); + let mut command = Command::new(&check.path); + command.args(&check.args); + let output = match tokio::time::timeout(timeout, command.output()).await { + Err(_) => { + return Verdict { + healthy: false, + reason: format!("health check timed out after {}s", timeout.as_secs()), + } + } + Ok(Err(err)) => { + return Verdict { + healthy: false, + reason: format!("failed to run {}: {err}", check.path), + } + } + Ok(Ok(output)) => output, + }; + if output.status.success() { + return Verdict { + healthy: true, + reason: String::new(), + }; + } + Verdict { + healthy: false, + reason: describe_failure(&output), + } +} + +fn describe_failure(output: &std::process::Output) -> String { + let code = match output.status.code() { + Some(code) => format!("exit code {code}"), + // No code means a signal killed it. + None => "killed by signal".to_string(), + }; + let stderr = String::from_utf8_lossy(&output.stderr); + let stderr = stderr.trim(); + if stderr.is_empty() { + return code; + } + format!("{code}: {}", truncate(stderr)) +} + +fn truncate(text: &str) -> String { + if text.len() <= MAX_REASON_BYTES { + return text.to_string(); + } + // Slice on a char boundary so a multi-byte character is never cut in half. + let mut end = MAX_REASON_BYTES; + while end > 0 && !text.is_char_boundary(end) { + end -= 1; + } + format!("{}... (truncated)", &text[..end]) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn check(path: &str, args: &[&str]) -> HealthCheck { + HealthCheck { + path: path.to_string(), + args: args.iter().map(|arg| arg.to_string()).collect(), + interval_secs: 1, + timeout_secs: 2, + } + } + + #[tokio::test] + async fn exit_zero_is_healthy() { + let verdict = run_once(&check("/bin/true", &[])).await; + assert!(verdict.healthy); + assert!(verdict.reason.is_empty()); + } + + #[tokio::test] + async fn a_non_zero_exit_reports_the_code() { + let verdict = run_once(&check("/bin/sh", &["-c", "exit 3"])).await; + assert!(!verdict.healthy); + assert_eq!(verdict.reason, "exit code 3"); + } + + #[tokio::test] + async fn stderr_is_carried_into_the_reason() { + let verdict = run_once(&check( + "/bin/sh", + &["-c", "echo 'db unreachable' >&2; exit 1"], + )) + .await; + assert!(!verdict.healthy); + assert_eq!(verdict.reason, "exit code 1: db unreachable"); + } + + /// A probe that hangs must not be mistaken for one that passes. + #[tokio::test] + async fn a_hanging_probe_times_out_as_unhealthy() { + let mut hang = check("/bin/sh", &["-c", "sleep 30"]); + hang.timeout_secs = 1; + let verdict = run_once(&hang).await; + assert!(!verdict.healthy); + assert_eq!(verdict.reason, "health check timed out after 1s"); + } + + #[tokio::test] + async fn a_missing_program_is_unhealthy_rather_than_an_error() { + let verdict = run_once(&check("/nonexistent/health-probe", &[])).await; + assert!(!verdict.healthy); + assert!(verdict + .reason + .starts_with("failed to run /nonexistent/health-probe")); + } + + /// The gateway must not be handed an unbounded blob of app output. + #[tokio::test] + async fn a_noisy_probe_has_its_output_bounded() { + let verdict = run_once(&check( + "/bin/sh", + &["-c", "head -c 4000 /dev/zero | tr '\\0' 'x' >&2; exit 1"], + )) + .await; + assert!(!verdict.healthy); + assert!(verdict.reason.len() < MAX_REASON_BYTES + 64); + assert!(verdict.reason.ends_with("... (truncated)")); + } + + /// Before the first run there is no evidence the app works, and a CVM + /// registers long before its app is up. + #[test] + fn the_first_report_is_not_healthy() { + let probe = AppHealthProbe { + latest: RwLock::new(Verdict { + healthy: false, + reason: "health check has not run yet".to_string(), + }), + }; + let report = probe.report(); + assert!(!report.healthy); + assert_eq!(report.unhealthy[0].name, PROBE_NAME); + assert_eq!(report.unhealthy[0].status, "health check has not run yet"); + } +} diff --git a/dstack/guest-agent/src/lib.rs b/dstack/guest-agent/src/lib.rs index 4a06879ca..1edac8bba 100644 --- a/dstack/guest-agent/src/lib.rs +++ b/dstack/guest-agent/src/lib.rs @@ -5,6 +5,7 @@ pub const CARGO_PKG_VERSION: &str = env!("CARGO_PKG_VERSION"); pub const GIT_REV: &str = dstack_build_info::git_revision!(); +mod app_health; pub mod backend; pub mod config; mod container_health; diff --git a/dstack/guest-agent/src/rpc_service.rs b/dstack/guest-agent/src/rpc_service.rs index ffe6e7066..38429a892 100644 --- a/dstack/guest-agent/src/rpc_service.rs +++ b/dstack/guest-agent/src/rpc_service.rs @@ -79,6 +79,8 @@ struct AppStateInner { cert_client: CertRequestClient, demo_cert: RwLock, platform: Arc, + /// Present only when the app declared its own probe; see `app_health`. + app_health: Option>, } impl AppStateInner { @@ -169,6 +171,11 @@ impl AppState { let cert_client = CertRequestClient::create(&keys, verifier, vm_config.clone()) .await .context("Failed to create cert signer")?; + let app_health = config + .app_compose + .health_check + .clone() + .map(crate::app_health::AppHealthProbe::spawn); let me = Self { inner: Arc::new(AppStateInner { config, @@ -177,6 +184,7 @@ impl AppState { demo_cert: RwLock::new(String::new()), vm_config, platform, + app_health, }), }; me.maybe_request_demo_cert(); @@ -191,6 +199,10 @@ impl AppState { &self.inner.config } + fn app_health(&self) -> Option<&crate::app_health::AppHealthProbe> { + self.inner.app_health.as_deref() + } + fn quote_response(&self, report_data: [u8; 64]) -> Result { self.inner .platform @@ -602,6 +614,22 @@ impl RpcCall for InternalRpcHandlerV0 { } } +/// Shared conversion so both health sources report the same shape. +fn health_response(report: crate::container_health::HealthReport, error: String) -> HealthResponse { + HealthResponse { + healthy: report.healthy, + unhealthy: report + .unhealthy + .into_iter() + .map(|container| ContainerHealth { + name: container.name, + status: container.status, + }) + .collect(), + error, + } +} + pub struct ExternalRpcHandler { state: AppState, } @@ -625,23 +653,20 @@ impl WorkerRpc for ExternalRpcHandler { } async fn health(self) -> Result { + // An app-declared probe replaces whatever the runner would say. It is + // the app's own statement about itself, which beats anything inferred + // from container state -- and for a runner with no containers to + // inspect it is the only signal there is. + if let Some(probe) = self.state.app_health() { + return Ok(health_response(probe.report(), String::new())); + } // Deliberately infallible: see `HealthResponse.error`. A failure to - // reach Docker has to come back as a verdict, because an RPC error is - // indistinguishable from an agent that predates this method. + // reach the container runtime has to come back as a verdict, because an + // RPC error is indistinguishable from an agent that predates this + // method. let runner = self.state.config().app_compose.runner.clone(); Ok(match crate::container_health::collect(&runner).await { - Ok(report) => HealthResponse { - healthy: report.healthy, - unhealthy: report - .unhealthy - .into_iter() - .map(|container| ContainerHealth { - name: container.name, - status: container.status, - }) - .collect(), - error: String::new(), - }, + Ok(report) => health_response(report, String::new()), Err(err) => { warn!("failed to collect container health: {err:#}"); HealthResponse { @@ -799,6 +824,7 @@ mod tests { port_policy: Default::default(), requirements: None, verity_volumes: Vec::new(), + health_check: None, }; let dummy_appcompose_wrapper = AppComposeWrapper { @@ -951,6 +977,7 @@ pNs85uhOZE8z2jr8Pg== ) .unwrap(), }), + app_health: None, }; (