CyberController Container Watchdog

The CyberController Container Watchdog is a lightweight, autonomous monitoring service that continuously tracks the health of all Docker containers on the host. It detects failures — including crashes, out-of-memory kills, prolonged unhealthy states, and restart loops — and dispatches real-time alerts through one or more configurable channels (Slack, SMTP, SNMP Traps, or Syslog). It also sends a one-time INFO recovered alert through the same channels once a previously-alarmed container returns to normal operation.

1. How It Works

The watchdog runs two parallel monitoring paths:

Path	Mechanism	What It Catches
Event stream	`docker events` (real-time)	Crashes (`die`), OOM-kills (`oom`), health status changes
Active Probes	`docker ps` + Probes	Integrity: Periodic verification of liveness, stuck states, and restart-loops

The watchdog selects one of 4 active probe types per container, chosen automatically based on the container's configuration. Probe selection is cached for the container's lifetime and re-discovered on restart.

Docker HEALTHCHECK (passive) — for containers that have a Docker HEALTHCHECK directive in their Dockerfile. The watchdog reads the result from container.attrs["State"]["Health"] rather than running its own probe. When Docker reports health_status: unhealthy 3 consecutive times, an alert fires. The last health check output is included in the alert's Detection field.
HTTP GET (active) — for containers with no Docker HEALTHCHECK that expose HTTP ports. Auto-discovers a working endpoint by trying common paths (/-/healthy, /health, /healthz, /metrics, /) on each exposed internal port, and across all internal IPs when the container is attached to multiple Docker networks (configurable via auto_health_check.paths in watchdog-config.yaml). Returns healthy if status_code < 500. The discovered URL is cached for the container's lifetime.
TCP connect (active) — if no HTTP path responds on a port, falls back to a raw socket.create_connection() on that port. A successful connect confirms the service is listening; a refused or timed-out connection triggers an unhealthy alert. Used by databases, message brokers, and any non-HTTP service.
/proc alive check (active) — last resort when the container exposes no ports at all, or all TCP connects fail during initial discovery. Reads /proc/1/status inside the container via exec_run and checks that PID 1 is in a live state (R/S/D). A zombie (Z) or stopped (T) PID 1 triggers an unhealthy alert.

Alerts are deduplicated via a cooldown — once an alert fires for a container, the same failure type is suppressed for cooldown_minutes minutes. The cooldown resets automatically when the container recovers.

Two additional deduplication rules suppress redundant alerts at the event level:

OOM → die suppression: when a container is OOM-killed, Docker emits both an oom event and an immediately following die event. The watchdog fires the oom CRITICAL alert and suppresses the subsequent crashed alert to avoid a duplicate for the same incident.
Restart-loop → die suppression: once a restart-loop HIGH alert is active for a container, subsequent per-cycle die events are suppressed — the restart-loop alert is the primary signal until the container stabilises.

Recovery alerts: once a container that was previously alarmed (crashed, oom, unhealthy, or restart-loop) is observed running and healthy again, the watchdog dispatches a one-time INFO recovered alert through the same channels and clears its alarm state. Controlled by alert_on_recovery (default true). Detection depends on the same probe path used for the original failure — a container with a Docker HEALTHCHECK recovers via the health_status: healthy event; containers on active probes (HTTP/TCP//proc) recover on the next successful poll cycle.

2. System Overview

Project Structure

.
├── watchdog.py                  # Main monitor script
├── watchdog-config.yaml         # All configuration (no secrets)
├── docker-compose.yaml          # Production / offline runtime (image only)
├── docker-compose.build.yaml    # Developer / online build
├── Dockerfile                   # Image build definition (python:3.11-slim)
├── requirements-watchdog.txt    # Python dependencies (reference; Dockerfile pip-installs inline)
├── install.sh                   # Offline install script
├── uninstall.sh                 # Stop + remove script
├── VERSION                       # Release version — drives the image tag and installer
├── watchdog.tar                  # Pre-built Docker image (provided, no internet needed)
└── .env                          # Secrets — NOT committed to git

Image Size

Item	Size
Docker image (`watchdog:<VERSION>`)	~180 MB (python:3.11-slim base + dependencies)
Running container (memory)	~50–80 MB baseline; capped at `mem_limit: 256m` in `docker-compose.yaml`
`watchdog.tar` export	~170 MB

Note: The values above are approximate and may vary depending on the host operating system, Docker version, and installed dependencies. If usage approaches the 256 MB limit (check with docker stats docker-container-watchdog), raise mem_limit/mem_reservation in docker-compose.yaml rather than removing the limit — and raise memswap_limit to at least the new mem_limit in both docker-compose.yaml and docker-compose.build.yaml, since Docker rejects a config where memswap_limit is lower than mem_limit.

Container Hardening

By default the container runs as root so docker compose up works with zero configuration — no host-specific group setup needed to read /var/run/docker.sock. install.sh instead auto-detects the socket's GID and opts into running the container as non-root UID/GID 1000 (joined to that group via group_add), with no manual input required; manual installs can do the same by setting WATCHDOG_UID/WATCHDOG_GID/DOCKER_GID in .env (see DEPLOYMENT.md). Either way, the container is constrained by a read-only root filesystem, no extra Linux capabilities (cap_drop: ALL), no-new-privileges, and CPU/memory/PID caps (cpus: "0.50", pids_limit: 200, mem_limit: 256m), which contain a leak or runaway condition to this container instead of the host. Note that access to /var/run/docker.sock is effectively host-root-equivalent for any user (root or not): anything that can reach the socket can create privileged containers or bind-mount the host filesystem, so a compromise of the watchdog process is not contained by these controls — treat docker.sock access as the primary trust boundary when deciding who/what can reach this host.

Python Dependencies

docker==7.1.0
requests==2.32.3
PyYAML==6.0.2
pysnmp>=6.2

pysnmp 6.2+ uses the modern pysnmp.hlapi.v3arch.asyncio API. pysnmp 4.x is no longer supported — its synchronous generator API was removed in 5.x. pyasn1 is a transitive dependency managed by pysnmp itself and no longer needs to be pinned separately.

Runtime (in container): Python 3.11+

3. Configuration

All non-secret settings live in watchdog-config.yaml. Secrets (webhook URLs, passwords) are stored in .env and never committed to version control. Restart the container after editing either file — no image rebuild is needed.

Configuration Keys

Key	Default	Description
`alert_channels`	`["slack"]`	Active destinations: `slack`, `smtp`, `snmp_trap`
`check_interval_seconds`	`60`	How often the poll loop runs
`cooldown_minutes`	`5`	Suppress duplicate alerts per container per failure type
`restart_threshold`	`5`	Restart count within window that triggers a restart-loop alert
`restart_window_minutes`	`10`	Rolling window for restart counting
`unhealthy_cycles_threshold`	`3`	Consecutive unhealthy cycles before alerting
`alert_on_recovery`	`true`	Send an INFO "recovered" alert once a previously-alarmed container returns to normal
`excluded_containers`	`[]`	Container names to never alert on
`ignored_container_events`	MariaDB syntax-check label rule	Label-based event suppressions for intentional temporary containers
`auto_health_check`	enabled	Probe containers reporting `health=none` by discovering an HTTP/TCP endpoint
`container_health_checks`	`{}`	Per-container probe overrides (`type: http` or `type: exec`) for `health=none` containers; takes precedence over `auto_health_check`
`log_level`	`INFO`	`DEBUG` / `INFO` / `WARNING` / `ERROR`
`log_file`	`/var/log/watchdog/watchdog.log`	Bind-mounted to `./watchdog/watchdog.log` on host. Rotates at 10 MB, 5 backups. Set to `null` to disable
`state_file`	`/var/log/watchdog/state.json`	Persists open alerts and the host boot time so recovery and reboot reporting survive a restart. Set to `""` to disable
`boot_grace_seconds`	`180`	After a detected host reboot, hold alerts this long while services start, then send one `reboot-summary`. `0` alerts immediately and sends the summary on the first poll
`runbook_base_url`	—	URL included in every alert

Preferred suppression: Temporary MariaDB HA syntax-check containers should be created with Docker label com.radware.cybercontroller.role=mariadb-ha-syntax-check. The watchdog ignores only configured events for containers carrying that explicit label; normal service containers still alert.

Temporary fallback: until that label exists, dynamic MariaDB dump helper containers are not alerted as crashes only when all identifying signals match together: image kvision_infra_mariadb, container name is not config_kvision-infra-mariadb_1, Docker AutoRemove is true, a mount contains /mnt/cli/tmp/mariadb_dump.sql.gz, and the exit code is 137. The primary MariaDB container remains monitored.

Environment Variables

Variable	Required	Description
`SLACK_WEBHOOK_URL`	Yes (if Slack enabled)	Slack incoming webhook URL
`SMTP_USERNAME`	Yes (if SMTP enabled)	SMTP username or login email used for login
`SMTP_PASSWORD`	Yes (if SMTP enabled)	SMTP account password or api key value
`SNMP_V3_AUTH_KEY`	Yes (if SNMPv3 enabled)	SNMPv3 authentication passphrase — minimum 8 characters (RFC 3414)
`SNMP_V3_PRIV_KEY`	Yes (if SNMPv3 authPriv)	SNMPv3 privacy passphrase — minimum 8 characters; requires auth key also set
`WATCHDOG_CONFIG`	No	Path to config file (default: `/etc/watchdog/watchdog-config.yaml`)
`WATCHDOG_HOST`	No	Hostname shown in alerts (default: system hostname)

4. Supported Alert Channels

Configure at least one channel and add its identifier to alert_channels in watchdog-config.yaml. Multiple channels can be active simultaneously.

Slack

Add slack to alert_channels and set SLACK_WEBHOOK_URL in .env:

slack:
  enabled: true
  webhook_url_env: SLACK_WEBHOOK_URL

Syslog

syslog:
  enabled: true
  host: <syslog-server-ip>
  port: 514
  protocol: udp       # udp or tcp
  facility: local0

SMTP (Email)

Add smtp to alert_channels to enable.

smtp:
  enabled: true
  host: smtp.radware.com
  port: 587
  sender: noc-alerts@radware.com
  username_env: SMTP_USERNAME     # env var: SMTP username or app token
  recipients:
    - ops-team@radware.com
    - oncall@radware.com
  tls: true
  password_env: SMTP_PASSWORD        # env var: SMTP password or app token

SNMP Traps

Add snmp_trap to alert_channels to enable. Sends SNMP traps (v1, v2c, or v3) to your NMS/SIEM on every alert.

snmp_trap:
  enabled: true
  host: 155.1.1.4                          # IP or hostname of your SNMP trap receiver
  port: 162                                # Standard SNMP trap port
  version: v2c                             # SNMP version: v1, v2c, or v3
  community: public                        # SNMPv1/v2c community string (ignored for v3)
  trap_oid: "1.3.6.1.4.1.89.110.0.1"      # Radware enterprise container-alert notification OID
  # SNMPv3 only — add SNMP_V3_AUTH_KEY / SNMP_V3_PRIV_KEY to .env
  # v3_username:        watchdog-user
  # v3_auth_protocol:   SHA                # MD5 | SHA | SHA256 | SHA384 | SHA512
  # v3_auth_key_env:    SNMP_V3_AUTH_KEY   # min 8 chars
  # v3_priv_protocol:   AES                # AES (recommended) | AES192 | AES256 | DES (weak)
  # v3_priv_key_env:    SNMP_V3_PRIV_KEY   # min 8 chars; requires auth key also set
  # v3_local_engine_id: ""                 # hex engine ID pinned from first-run log; see DEPLOYMENT.md

Each trap carries five var-binds, all under the Radware enterprise OID arc (1.3.6.1.4.1.89.110):

OID	Object	Value
`1.3.6.1.4.1.89.110.1.1.0`	`cwHost`	Hostname of the alerting node
`1.3.6.1.4.1.89.110.1.2.0`	`cwSummary`	Human-readable alert summary
`1.3.6.1.4.1.89.110.1.3.0`	`cwContainerName`	Container name
`1.3.6.1.4.1.89.110.1.4.0`	`cwFailureType`	`crashed` / `oom` / `unhealthy` / `restart-loop` / `recovered`
`1.3.6.1.4.1.89.110.1.5.0`	`cwProbeDetail`	Probe failure detail

5. Deployment Instructions

For full deployment instructions — including alert channel setup, offline installation, developer builds, and troubleshooting — see DEPLOYMENT.md.

Quick start (automated):

bash install.sh

6. Failure Types & Severities

Failure	Severity	Trigger
`crashed`	CRITICAL	Container exited with non-zero exit code
`oom`	CRITICAL	Container was OOM-killed by the kernel
`unhealthy`	HIGH	Health probe failing for N consecutive cycles
`restart-loop`	HIGH	Container restarted ≥ threshold times within window
`recovered`	INFO	Previously-alarmed container (`crashed`/`oom`/`unhealthy`/`restart-loop`) is healthy/running again. Fires once per incident; controlled by `alert_on_recovery` (default `true`)
`reboot-summary`	INFO / HIGH	Sent once after a host reboot is detected, when the `boot_grace_seconds` window closes. Reports boot time, how many containers are running, which recovered, and which are still failing. Always sent after a reboot — an INFO summary confirms the node came back cleanly. HIGH if any container is still failing

OOM alerts include memory stats (usage / limit / peak) prepended to the log snippet.

Every alert includes two probe fields that identify exactly how and why the failure was detected:

Probe Type — short label classifying the detection mechanism
Detection — full detail string with path, status code, or error message

Failure	Probe Type	Example Detection value
`crashed`	`Docker event (crash)`	`Crash probe failed — container exited with exit code 1`
`oom`	`Docker event (OOM)`	`OOM probe failed — container was OOM-killed by the kernel`
`unhealthy` (Docker native HEALTHCHECK)	`Docker HEALTHCHECK`	`Docker health probe failed — exit code 1, output: connection refused`
`unhealthy` (HTTP auto-probe)	`HTTP GET`	`HTTP probe failed — error response 503 Service Unavailable, URI http://172.18.0.5:8080/health`
`unhealthy` (TCP connect)	`TCP connect`	`TCP probe failed — connection refused or timed out on 172.18.0.5:6379 (Connection refused)`
`unhealthy` (/proc fallback)	`/proc alive check`	`/proc alive check failed — PID 1 is zombie/stopped (exit code 1)`

Restart-loop detection is not a probe — it is tracked by the Poll loop path and fires when a container restarts ≥ restart_threshold times within restart_window_minutes. See the Poll loop row in the monitoring paths table above.

7. Testing

The sections below describe optional manual tests that validate the watchdog's detection and alerting paths. Each creates a temporary Docker container and cleans up after itself.

OOM Simulation

Two options — choose based on whether the host has internet access.

Option A — `polinux/stress` (requires internet access)

Pulls polinux/stress from Docker Hub.

# --memory-swap=32m disables swap so the kernel is forced to OOM-kill rather than swap out
# Note: the polinux/stress entrypoint IS the stress binary — do not repeat "stress" in the args
docker run -d --name oom-test --memory=32m --memory-swap=32m polinux/stress \
  --vm 1 --vm-bytes 64M --vm-keep

# Clean up
docker rm -f oom-test
docker rmi polinux/stress

Option B — `watchdog:latest` (no internet required)

Uses the Watchdog image that is already present on the host, avoiding the need to pull the image during deployment.

docker run -d --name oom-test --memory=32m --memory-swap=32m watchdog:latest \
  python3 -c "x = b'\xff' * (64 * 1024 * 1024)"

# Clean up — container only (keep the watchdog image)
docker rm -f oom-test

Docker kills the container with exit code 137 and emits an oom event. The watchdog captures memory stats and the last 20 log lines, then dispatches a CRITICAL alert — all within seconds of the kill.

If the container exits cleanly (exit code 0) instead of being OOM-killed, cgroup memory accounting may be disabled on this host. Verify with:
docker info 2>&1 | grep -i "memory limit"
WARNING: No memory limit support means the --memory flag is silently ignored. Enable cgroup memory accounting by adding cgroup_enable=memory swapaccount=1 to GRUB_CMDLINE_LINUX in /etc/default/grub, then run sudo update-grub and reboot.

Probe Testing

Optional — requires internet access. These tests pull images from Docker Hub and create temporary containers on the host. Each section includes full cleanup instructions (container + image).

HTTP Probe

# Start a container where every HTTP path returns 503
# Note: the full server { } block is required — a bare location { } directive is not valid nginx config
docker run -d --name http-probe-test \
  nginx:alpine \
  sh -c "printf 'server { listen 80; location / { return 503; } }' > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"

# Verify the container is running:
docker ps | grep http-probe-test

# Wait one poll cycle (60s), then check watchdog detected it:
grep "http-probe-test" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

# Clean up
docker rm -f http-probe-test
docker rmi nginx:alpine

During auto-discovery the watchdog sends an HTTP GET to /-/healthy on port 80. Nginx returns 503, so the watchdog immediately caches that endpoint as failing and reports unhealthy. After unhealthy_cycles_threshold consecutive unhealthy cycles an alert fires with Detection value HTTP probe failed — error response 503 Service Unavailable, URI http://<ip>:80/-/healthy.

TCP Connect Probe

# Expose port 9999 but run nothing on it — TCP connect will be refused
docker run -d --name tcp-probe-test \
  --expose 9999 \
  alpine \
  sh -c "sleep infinity"

# Watch the watchdog fall through HTTP → TCP and alert:
grep "tcp-probe-test" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

# Clean up
docker rm -f tcp-probe-test
docker rmi alpine

After unhealthy_cycles_threshold consecutive failed cycles an alert fires with Detection value TCP probe failed — connection refused or timed out on <ip>:9999.

`/proc` Alive Check

# No EXPOSE — watchdog falls straight to /proc alive check
docker run -d --name proc-probe-test \
  alpine \
  sleep infinity

# Verify the container is running:
docker ps | grep proc-probe-test

# After one poll cycle, confirm the watchdog selected /proc as the probe type:
grep "proc-probe-test" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

# Clean up
docker rm -f proc-probe-test
docker rmi alpine

With no exposed ports, auto-discovery skips HTTP and TCP and falls back to reading /proc/1/status inside the container via exec_run. Since sleep is PID 1 in a live S (sleeping) state, no alert fires. The log will show falling back to /proc alive-check confirming the probe type was selected. In production, this probe fires when PID 1 enters zombie (Z) or stopped (T) state.

Recovery Alert Test

Validates the INFO recovered alert fires once a previously-alarmed container is healthy again. Use any real, already-monitored container — no throwaway container needed.

# 1. Force a failure (SIGKILL → non-zero exit code → CRITICAL "crashed" alert)
docker kill <container-name>

# 2. Confirm the crashed alert was logged/dispatched
grep "<container-name>" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log | grep CRITICAL

# 3. Bring it back
docker start <container-name>

# 4. Wait for it to report healthy again, then confirm the recovery alert:
grep "<container-name>" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log | grep -i recovered

Expected log line: ALERT [INFO] <container-name> — recovered (channels: [...]).

Containers with a Docker HEALTHCHECK recover via the health_status: healthy event — near-instant once Docker reports healthy (subject to the image's own start_period/interval).
Containers on HTTP/TCP//proc active probes recover on the next poll cycle (check_interval_seconds).
No alert fires if alert_on_recovery: false is set, or if the container was never in an alarmed state (state.alerted_for empty).

8. Troubleshooting

Service Fails to Start

docker compose logs docker-container-watchdog

Common causes:

Missing .env file — run cp .env.example .env and fill in credentials
Docker socket not accessible — ensure /var/run/docker.sock exists and the container has read access
Image not loaded — run docker images watchdog; if empty, set WATCHDOG_VERSION=$(cat VERSION) and build with WATCHDOG_VERSION="$WATCHDOG_VERSION" docker compose -f docker-compose.build.yaml build (see DEPLOYMENT.md for details)

Alert Notifications Not Received

Check alert_channels in watchdog-config.yaml — ensure at least one channel is listed and configured

Check watchdog logs for errors:

grep -E "ERROR|CRITICAL" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

For Slack: verify SLACK_WEBHOOK_URL is set in .env and test manually:

curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Watchdog test"}' "$SLACK_WEBHOOK_URL"

Expected response: ok

Recovery alert (recovered) specifically not received:

Confirm alert_on_recovery is not set to false in the live watchdog-config.yaml.
Confirm the container was actually alarmed first — state.alerted_for must be non-empty; a container that was never flagged won't generate a recovery alert.
For containers with a Docker HEALTHCHECK, confirm Docker actually reports healthy — docker inspect <name> --format '{{json .State.Health}}'. If it's stuck on starting, the watchdog is waiting on Docker, not the other way around.
For containers on active probes (HTTP/TCP//proc), confirm the poll loop reports health=healthy for the container in the log — if it stays unhealthy, the probe (often auto_health_check) is misjudging the service; add a manual entry under container_health_checks instead.

Duplicate and Excessive Alert Notifications

Increase cooldown_minutes in watchdog-config.yaml (default: 5 minutes per container per failure type). To silence a noisy container entirely, add it to excluded_containers:

excluded_containers:
  - debug-shell
  - load-test
  - my-noisy-container

Probe Type Not as Expected

Check which probe was cached at startup:

grep -E "auto-discover|falling back" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

Viewing Logs

Log destinations (simultaneous):

stdout — always on; captured by docker compose logs
file — ./watchdog/watchdog.log on the host (bind-mounted), rotates at 10 MB, keeps 5 backups
syslog — remote server if syslog.enabled: true in config

# Live tail (preferred — host path, no need to exec into the container)
tail -f /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

# Last 100 lines
tail -100 /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

# Alerts and errors only
grep -E "ALERT|ERROR|CRITICAL" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

# Filter by container name
grep "my-container" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

# Search across rotated logs
grep "CRITICAL" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log \
             /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log.1

# Via Docker (stdout only)
docker compose logs -f docker-container-watchdog

9. Container Probe Map (This Deployment)

Probe selection is automatic: containers with a Docker HEALTHCHECK are monitored passively via events; all others get an active probe auto-discovered on the first poll and cached for the container's lifetime.

Container	Probe Type	Endpoint	Why
`config_kvision-infra-fluentd_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-dp-inline-config_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-collector_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-configuration-service_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-anomaly-detection-engine_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-data-persist-service_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-rt-alert_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_policy-editor_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-alerts_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-reporter_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-vrm_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-infra-redis_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-health_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-webui_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-tor-feed_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-infra-efk_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-data-polling-mgr_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-formatter_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-data-polling-scheduler_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-infra-cadvisor_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-scheduler_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-vdirect_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-help_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-ted_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_postgres_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-snmp-trap-collector_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-auto-engine_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-lls_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-assist-service_1`	HTTP	`:3005` (path auto-discovered)	No Docker HEALTHCHECK; Node.js service; path cached on first successful response
`config_kvision-infra-rabbitmq_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`config_kvision-ha-operator_1`	HTTP	`:8080` (path auto-discovered)	No Docker HEALTHCHECK; Java HTTP service; path cached on first successful response
`config_kvision-dc-nginx_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive
`process-exporter`	HTTP	`:9256/metrics`	No Docker HEALTHCHECK; Prometheus exporter — `/metrics` returns 200
`prometheus`	HTTP	`:9090/-/ready`	No Docker HEALTHCHECK; standard Prometheus readiness endpoint
`alertmanager`	HTTP	`:9093/-/healthy`	No Docker HEALTHCHECK; standard Alertmanager health endpoint
`postgres-exporter`	HTTP	`:9187/metrics`	No Docker HEALTHCHECK; Prometheus exporter — `/metrics` returns 200
`mysql-exporter`	HTTP	`:9104/metrics`	No Docker HEALTHCHECK; Prometheus exporter — `/metrics` returns 200
`es-exporter`	HTTP	`:9114/metrics`	No Docker HEALTHCHECK; port 7979 HTTP fails; port 9114 serves `/metrics`
`node-exporter`	HTTP	`:9100/metrics`	No Docker HEALTHCHECK; Prometheus exporter — `/metrics` returns 200
`grafana`	HTTP	`:3000/api/health`	No Docker HEALTHCHECK; Grafana health endpoint returns 200
`config_kvision-infra-mariadb_1`	Docker HEALTHCHECK	Docker managed	Image has HEALTHCHECK directive

Verify what was actually cached after startup:

grep -E "auto-discover|falling back" /opt/radware/storage/scripts/Alert_Container/watchdog/watchdog.log

10. Version History

Version	Date	Author	Changes
1.5.4	2026-09-23	Rahul Kumar	Versioned Docker image (`watchdog:<VERSION>`) driven by the root `VERSION` file; installer loads the packaged `watchdog.tar`, rejects an archive that does not carry the release tag, aborts on Compose failures, and verifies the running image; restart-loop detection now uses Docker `RestartCount` instead of counting poll observations; `container_health_checks` overrides implemented; fixed `.env` migration on root-owned files; `watchdog-config.yaml` made readable by the non-root container user
1.5.3	2026-09-16	Rahul Kumar	Resource Limits Enforced
1.5.2	2026-08-31	Rahul Kumar	Updated error message"Suppressing expected Cyber Controller SQL dump syntax-check container termination (exit 137) alert"
1.5.1	2026-08-31	Rahul Kumar	fixed ignore Dynamic container crash alert
1.5.0	2026-08-27	Rahul Kumar	Added ignore Dynamic container crash alert
1.4.0	2026-08-17	Rahul Kumar	Added INFO "recovered" alert
1.3.3	2026-08-05	Rahul Kumar	Added Auth True/false
1.3.2	2026-07-28	Rahul Kumar	Fixed SMTP Auth issue
1.3.1	2026-07-21	Egor Egorov	Updated Readme and Deployment guides
1.3.0	2026-07-16	Rahul Kumar	Added Upgrade Guide
1.2.0	2026-07-16	Rahul Kumar	Removed sudo and renamed the container
1.1.0	2026-07-09	Rahul Kumar	Added SNMPv3 support
1.0.0	2026-06-25	Rahul Kumar	Added Slack notifications and bug fixes

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
tests		tests
.env.example		.env.example
.gitignore		.gitignore
DEPLOYMENT.md		DEPLOYMENT.md
Dockerfile		Dockerfile
README.md		README.md
VERSION		VERSION
docker-compose.build.yaml		docker-compose.build.yaml
docker-compose.yaml		docker-compose.yaml
install.sh		install.sh
requirements-watchdog.txt		requirements-watchdog.txt
uninstall.sh		uninstall.sh
watchdog-config.yaml.example		watchdog-config.yaml.example
watchdog.py		watchdog.py

Latest commit

History

Folders and files

Repository files navigation