A distributed log aggregator in Go, modeled on Apache Flume's source → channel → sink architecture, built on gRPC + Protocol Buffers.
producers edge agent collector agent
┌──────────┐ gRPC ┌───────────────────────┐ gRPC ┌───────────────────────┐
│ app / │ ───────▶ │ FileTailSource │──────▶ │ GRPCSource │
│ loadgen │ │ │ │ │ │ │
└──────────┘ │ ▼ │ │ ▼ │
│ MemoryChannel (sharded)│ │ FileChannel (WAL) │
│ │ │ │ │ │
│ ▼ │ │ ▼ │
│ GRPCSink (pipelined) │ │ FileSink (rolling) │
└───────────────────────┘ └───────────────────────┘
Any number of these agents can be chained: a sink in one agent is a gRPC client of the next agent's source, exactly like Flume's Avro source/sink pairing. Each agent is independently configured via YAML and can mix and match source/channel/sink types.
Sources (internal/source)
GRPCSource— implements theCollectorgRPC service. Accepts a bidirectional streaming RPC (StreamEvents) so a well-behaved client can pipeline many batches without a round trip per batch, plus a unarySendBatchfor simple/low-volume producers. Used both for external ingestion and for agent-to-agent forwarding.FileTailSource— follows a growing/rotating file liketail -F.
Every source runs its events through an interceptor chain and then a channel selector before they're considered received — see below.
Interceptors (internal/interceptor)
A source can declare an ordered list of interceptors that transform or filter every event before it's routed to a channel. Built-ins:
| type | does |
|---|---|
timestamp |
stamps a header with ingest time (ms since epoch); preserve_existing skips events that already have it |
host |
tags the event with the agent's hostname (or IP with use_ip: true) |
static |
adds a fixed key/value header |
regex_filter |
drops events whose body matches (or, with exclude: true, doesn't match) a pattern |
regex_extractor |
promotes regex capture groups from the body into headers, by name ((?P<name>...)) or 1-based index |
New interceptor types are added by implementing one Intercept(*Event) *Event
method (return nil to drop) — the config schema (type + a string-keyed
params map) never has to change.
Channel Selectors (internal/router)
A source can target more than one channel via channels: [...] plus a
selector:
replicating(default when more than one channel is listed) — every event goes to every listed channel.multiplexing— routes by the value of one header:mappingmaps header values to a list of destination channels,defaultcatches everything else. This is how you'd split one firehose into per-tenant, per-priority, or per-destination channels.
Channels (internal/channel)
-
MemoryChannel— bounded, sharded in-memory queue. Producers hash across N independent Go channels instead of funneling through one shared queue, reducing lock contention under many concurrent goroutines; a fair fan-in read usesreflect.Selectacross all shards. Fast, but not durable — a crash loses whatever's buffered. -
FileChannel— a real write-ahead log: events are appended to disk (flushed immediately, fsynced on a timer) and only exposed to consumers once durably written. Segments roll at a configurable size and are deleted once fully drained. On restart it replays whatever segment files remain on disk. Every entry carries a CRC32 checksum; recovery stops replaying a segment at the first entry that fails verification rather than risking a torn write (or genuine bit rot) cascading into garbage. Delivery is at-least-once: a crash between "consumed" and "segment fully drained" can replay already-delivered events — this is the same trade-off Flume's file channel makes, and downstream consumers should be idempotent or dedupe if that matters.Segment cleanup happens two ways. The common path is reactive: a segment is deleted the moment its last entry is taken, as a normal side effect of consumption. The backstop is an active retention sweep enforcing an optional
RetentionPolicy(retention_max_age/retention_max_bytes, swept everycompaction_interval) - independent of whether consumers have caught up at all. A segment past its retention limit is purged even if it still holds unconsumed entries, the same trade-off Kafka'sretention.ms/retention.bytesmake: a retention limit describes how long you're willing to keep data, period, separate from whether it's been delivered. Purged-but-still-unconsumed events are counted viaPurgedUnconsumed()(surfaced in/statsaspurged_unconsumed) so it's visible when this is actually discarding data versus just reclaiming already-delivered space. -
SpillableMemoryChannel(type: spillable) — Flume's other durability pattern: a fastMemoryChannelprimary with automatic overflow to a durableFileChannelonce the primary fills, rather than blocking or dropping.Take()drains the overflow with priority, since anything sitting there means the system was already falling behind. Reports aSpilled()counter (surfaced in/stats) as a signal that downstream consumers aren't keeping up. The sameRetentionPolicycan be applied to its overflow. -
Any of the above can be wrapped in a
cluster:block to replicate it across multiple agent processes via Raft instead of running single-node, or asharded_cluster:block to partition it across several independent Raft groups for write throughput that scales past one group's single-leader ceiling — see "Clustered collector (Raft)" below.
Sinks (internal/sink)
GRPCSink— drains a channel, batches events (by count or a linger timeout, whichever comes first), and forwards them over a streaming RPC. Runs several worker goroutines (parallelism) each holding its own stream, and each worker keeps several batches in flight (in_flight_batches) rather than waiting for an ack before sending the next one. Supports gzip compression, automatic reconnect with backoff, and requeues events back onto the channel on send failure.FileSink— appends batches as newline-delimited text to files that roll by size or time.StdoutSink/NullSink— for local testing.
Sink Processors / groups (internal/sink, configured under
sink_groups instead of sinks)
A group of backends draining one channel, coordinated by a policy. A target
is normally a gRPC host:port, or file:<dir> for a rolling-file backend
(tuned by file_roll_max_bytes / file_roll_interval, defaults 128MiB / 1m)
- e.g.
targets: ["collector:7000", "file:/var/spool/logflow"]makes a local disk spool the failover when the collector is unreachable. A file target counts as healthy when a probe file can be created in its directory. A write error (e.g. disk full) hands the batch back to the channel, so a partly-written batch may duplicate its first events (at-least-once): load_balance— every backend is active simultaneously;parallelismworkers each bind to one target (round_robin, wrapping if there are more workers than targets, orrandom), all pulling concurrently from the same channel. A target being down only removes that fraction of capacity.failover— exactly one backend is active at a time, chosen by priority (targets[0]first). On failure it's marked down and traffic moves to the next available target; a background proberPings every higher-priority target everyrecovery_probe_intervaland promotes one back the moment it responds, by cancelling the active stream so the drain loop reconnects immediately rather than waiting for the current target to fail on its own.
Agent (internal/agent) wires up sources/interceptors/selectors/
channels/sinks/sink-groups from a config.Config and exposes /metrics
(Prometheus text format), /healthz, and /stats over HTTP.
Every gRPC endpoint (source or sink) is unencrypted, unauthenticated
plaintext by default — the same "secure is opt-in" stance as the rest of
this project. Turn on either or both of two independent layers per
endpoint via internal/security:
TLS / mutual TLS — set tls.enabled: true under a source, sink, or
sink group:
tls:
enabled: true
cert_file: certs/server-cert.pem # this endpoint's own identity
key_file: certs/server-key.pem
client_auth: true # server only: require+verify a client cert (mTLS)
ca_file: certs/ca-cert.pem # server: CA that signs client certs; client: CA that signs the server certOn a sink/sink-group (client role), also set cert_file/key_file if the
server requires a client certificate (mTLS), and optionally server_name
(override the hostname used for certificate verification, e.g. when
target_addr is a bare IP) or insecure_skip_verify: true (skip server
certificate verification — local testing only, never on an untrusted
network).
Bearer token auth — set auth_token on a source (the exact string
every caller must present) and the matching auth_token on whatever
sink/sink-group talks to it. This is checked independently of TLS via gRPC
metadata, using a constant-time comparison, so it composes with or without
encryption — though without TLS the token itself travels in clear text, so
grpc-go refuses to send it over a plaintext connection unless the client
config's TLS is enabled (see RequireTLS in internal/security).
Parallel WALs — a file channel (or a spillable channel's overflow)
funnels every Put through one active write segment. Set wal_shards: N
(N >= 2) to split it into N independent logs under dir/wal-0 ... wal-N-1,
each with its own segment, lock and fsync loop; events are assigned
round-robin and Take fans in across all of them, so ordering holds within
a WAL, not across them (same trade as the memory channel's shards).
capacity is split evenly. Changing the count on a non-empty dir is
refused at startup (switching to/from the single-WAL layout, or shrinking N)
because it would strand unread events - drain first or use a new dir;
growing N is allowed.
Per-caller tokens — instead of one shared auth_token, a source can set
auth_tokens_file: /etc/logflow/tokens.yaml (mutually exclusive with
auth_token). Each entry is one caller with its own identity, optional expiry
and a revoked flag:
tokens:
- id: edge-eu-1
token_sha256: 9f86d0... # sha256 hex of the secret (preferred)
expires_at: 2027-01-01T00:00:00Z
- id: batch-job
token: plain-secret # convenience; hashed on load, never kept
revoked: truego run ./cmd/tokengen -id edge-eu-1 -expires-in 720h mints a secret (shown
once) and the matching entry. The file is re-read when it changes (checked at
most once a second), so adding, expiring or revoking a caller needs no
restart; an unparseable edit keeps the last good set, while a bad file at
startup is a hard error. Open streams re-validate about once a second, so
revoking or expiring a caller also cuts off a stream it already had open
(Unauthenticated), not just new connections. Per-caller metrics:
logflow_auth_accepted_total{caller}, logflow_auth_rejected_total{caller,reason}
(invalid/revoked/expired; caller="unknown" when the token matched
nothing) and logflow_auth_tokens_active. Clients keep using the same
auth_token field with their own secret.
Generate a throwaway CA + server + client certificate for local testing:
./scripts/gen-certs.sh # writes certs/{ca,server,client}-{cert,key}.pemconfigs/secure-standalone.yaml (mTLS + token on a single agent's
ingestion source) and configs/secure-collector-agent.yaml /
configs/secure-edge-agent.yaml (mTLS + token on an agent-to-agent hop)
are ready-to-run examples — see "Verified behavior" below for exactly what
was exercised against them.
cmd/loadgen has matching flags (-tls, -cacert, -cert, -key,
-server-name, -insecure-skip-verify, -token) so it can drive a
TLS/mTLS/token-protected agent directly.
Shutdown is a staged drain, not an instant kill, on the first SIGINT or SIGTERM:
- Stop ingestion. Sources are cancelled immediately - gRPC sources
GracefulStop()(refuse new streams, let any RPC already in flight finish naturally rather than severing it),filetailsources stop polling. No new events are admitted from this point on. - Drain. Sinks keep running against whatever's already buffered in
channels, polling every 100ms, until every channel reports empty or
the agent's
drain_timeout(default 30s) elapses - whichever comes first. The HTTP server (and/stats) stays up through this stage so you can watch progress. - Stop sinks and close. Sinks are cancelled, the HTTP server is shut down, and every channel is closed.
Each stage logs what it's doing and how it concluded (fully drained vs. timed out with N events left, etc). A second SIGINT/SIGTERM at any point during this sequence forces an immediate stop instead of waiting out the current stage's timeout - the conventional "I understand this may lose buffered data, stop now" escape hatch, for a source stuck waiting on a slow producer or a drain that's taking longer than you want to wait for.
drain_timeout is set per agent (top-level config, alongside agent_id).
- Bidirectional gRPC streaming with per-worker pipelining (multiple batches in flight, not send-wait-send-wait)
- Sharded memory channel to cut lock contention across many goroutines
- Configurable batching (size + linger) at both source and sink boundaries
- Optional gzip compression on the wire
- Bounded channels provide real backpressure (
Putblocks / respects context) instead of unbounded queues that can OOM an agent - Parallel sink workers per channel, and parallel/priority-ordered backends via sink groups (load balance / failover)
- Async batch acking (ack for batch N doesn't block batch N+1 from sending)
- Durable file channel with immediate flush + timed fsync, so durability doesn't require fsync-per-event, and per-segment (not per-channel) locking so a slow fsync doesn't stall unrelated writers
- Spillable memory channel: fast in-memory path for the common case, automatic durable overflow under burst instead of blocking or dropping
- CRC32-verified WAL entries with bounded-allocation recovery, so a torn write or on-disk corruption is caught and safely truncated rather than crashing the process or replaying garbage
- Automatic reconnect + exponential-ish backoff on downstream failures, with best-effort requeue instead of silent data loss
- In-flight interceptor chain (filter/enrich/tag) and channel selectors (replicating/multiplexing fan-out) for routing without extra hops
- Zero-copy-oriented I/O throughout the pipeline: an event's body bytes
are shared by reference (never duplicated) from gRPC ingestion through
channel routing, replication fan-out, and sink forwarding; the file
channel additionally pools its write-side encode buffer (
sync.Pool) and aliases the read-side decode buffer directly into the returnedEventinstead of copying it - see "Verified behavior" for measured before/after allocation numbers - Graceful, staged shutdown: on SIGINT/SIGTERM, sources stop admitting new
events first, then sinks keep draining whatever's already buffered until
every channel is empty or
drain_timeoutelapses, and only then are sinks stopped and channels closed - a second signal during that drain forces an immediate quit instead of waiting it out
go build ./...Dependencies are fetched from the Go module proxy and verified against the
committed go.sum (the repo does not vendor them). The first build needs
network access to a module proxy; after that they are served from the local
module cache (~/go/pkg/mod). To build offline, run go mod vendor once
yourself (vendor/ is git-ignored) and build with -mod=vendor. After
changing a dependency, run go mod tidy and commit go.mod and go.sum.
If you regenerate proto/logflow.proto, you'll need protoc,
protoc-gen-go, and protoc-gen-go-grpc on your PATH:
protoc --go_out=gen/logflowpb --go_opt=paths=source_relative \
--go-grpc_out=gen/logflowpb --go-grpc_opt=paths=source_relative \
-I proto proto/logflow.protoQuick single-agent smoke test:
go run ./cmd/agent -config configs/standalone.yaml &
go run ./cmd/loadgen -addr 127.0.0.1:6000 -rate 20000 -batch 500 -duration 10s
curl localhost:9100/statsTwo-tier topology (edge agent tailing a file, forwarding to a collector agent that persists to disk):
go run ./cmd/agent -config configs/collector-agent.yaml &
go run ./cmd/agent -config configs/edge-agent.yaml &
echo "hello world" >> /var/log/app/app.log
curl localhost:9101/stats # collector
curl localhost:9100/stats # edgeconfigs/*.yaml document every tunable (batch size/linger, parallelism,
in-flight batches, retry/backoff, channel capacity/shards, WAL segment
size, roll policy, etc).
Interceptors + multiplexing selector + a load-balance sink group, all in one agent, with two receiving backends:
go run ./cmd/agent -config configs/backend-a.yaml &
go run ./cmd/agent -config configs/backend-b.yaml &
go run ./cmd/agent -config configs/multiplex-demo.yaml &
go run ./cmd/loadgen -addr 127.0.0.1:6000 -rate 5000 -batch 100 -duration 5s
curl localhost:9100/stats # events split between the `prod` and `default` channels
curl localhost:9200/stats # backend-a's share of the load-balanced prod traffic
curl localhost:9201/stats # backend-b's shareFailover sink group — kill and restart backend-a while traffic is
flowing and watch configs/failover-demo.yaml's log switch to
backend-b, then promote backend-a back the moment it's reachable
again:
go run ./cmd/agent -config configs/backend-a.yaml &
go run ./cmd/agent -config configs/backend-b.yaml &
go run ./cmd/agent -config configs/failover-demo.yaml &
go run ./cmd/loadgen -addr 127.0.0.1:6000 -rate 1000 -batch 25 -duration 30s
# in another shell: kill -9 <backend-a pid>, watch it fail over, then
# restart backend-a and watch it get promoted back within recovery_probe_intervalTLS + mutual TLS + bearer token, both for direct ingestion and for an agent-to-agent hop:
./scripts/gen-certs.sh
go run ./cmd/agent -config configs/secure-standalone.yaml &
go run ./cmd/loadgen -addr 127.0.0.1:6000 -rate 5000 -batch 100 -duration 5s \
-tls -cacert certs/ca-cert.pem -cert certs/client-cert.pem -key certs/client-key.pem \
-token s3cret-demo-token
curl localhost:9100/stats # events landed
# omit -cert/-key to see the mTLS rejection, or pass -token wrong-token to
# see the auth rejection - both fail closed with 0 events landingSpillable memory channel — a tiny in-memory primary with a slow sink
forces most of the burst to overflow to disk; watch the spilled counter
climb in /stats while dropped stays at 0:
go run ./cmd/agent -config configs/spillable-demo.yaml &
go run ./cmd/loadgen -addr 127.0.0.1:6000 -rate 20000 -batch 500 -duration 5s
curl localhost:9100/stats # e.g. put=104475 take=104475 dropped=0 spilled=98392
ls data/spillable-demo/overflow/ # the durable WAL holding the spilled backlogGraceful shutdown — start a long-running producer, send SIGTERM once to watch the staged drain, or twice in quick succession to force an immediate quit instead:
go run ./cmd/agent -config configs/shutdown-test.yaml &
AGENT_PID=$!
go run ./cmd/loadgen -addr 127.0.0.1:6000 -rate 20000 -batch 500 -duration 30s &
sleep 1
kill -TERM $AGENT_PID # first signal: graceful drain begins
# watch the log; send a second signal to force an immediate stop instead:
# kill -TERM $AGENT_PIDActive WAL retention/purging — tiny segments and a 3-second retention window with no sink attached, so nothing is ever consumed and every purge visibly discards unconsumed data:
go run ./cmd/agent -config configs/retention-demo.yaml &
go run ./cmd/loadgen -addr 127.0.0.1:6000 -rate 2000 -batch 100 -duration 2s
ls data/retention-demo/wal/ | wc -l # many segments right after the burst
sleep 4
ls data/retention-demo/wal/ | wc -l # down to just the active segment
curl localhost:9100/stats # purged_unconsumed=... in the outputDockerfile + docker-compose.yml package logflow as a container image and
stand up a realistic multi-agent topology with one command.
Image design. The image is built FROM scratch from statically linked
(CGO_ENABLED=0) binaries that are compiled first, on the host, by
make build - not by a golang:... build stage inside docker build.
make build runs go mod download and then go build on the host (no
network is needed inside docker build itself). This means the image build
needs zero container-registry access - only the host's Go module proxy
access for make build - which also happens to make for a
smaller, more auditable image: nothing in it but logflow's own three
binaries (agent, loadgen, and a tiny healthcheck probe - see below)
and a CA bundle, no base-image CVEs to track. If you're building somewhere
with normal registry access and would rather have Docker compile too, the
Dockerfile has a comment showing the one-stage swap back to a conventional
FROM golang:1.24-alpine AS builder pattern.
Why a healthcheck binary. A FROM scratch image has no shell, so the
conventional HEALTHCHECK CMD wget -qO- http://... doesn't work. cmd/healthcheck
is a ~10-line Go program that GETs a URL and exits 0/1 on 2xx/otherwise;
docker-compose.yml's healthchecks (and the Dockerfile's own HEALTHCHECK,
if you add one for standalone docker run use) call it directly as the
container's only "binary available."
Topology. docker-compose.yml stands up two independent edge agents
(each fed by its own loadgen sidecar) load-balancing forwarded batches
round-robin across two collector agents, which persist to a durable WAL and
roll output files onto a named Docker volume per collector:
loadgen-1 --> edge-1 --\ /--> collector-1 (WAL + disk, volume collector1-data)
>--------<
loadgen-2 --> edge-2 --/ \--> collector-2 (WAL + disk, volume collector2-data)
This is the same edge/collector tiering as configs/edge-agent.yaml +
configs/collector-agent.yaml, scaled to two agents per tier (over
configs/docker/*.yaml, which use Docker service names instead of
127.0.0.1) specifically so the load-balance sink group and the WAL's
crash-recovery path both get exercised by the demo, not just described.
make compose-up # build + docker compose up -d
curl -s http://localhost:19100/stats # edge-1
curl -s http://localhost:19101/stats # collector-1Published host ports: 19100/19103 (edge-1/edge-2 HTTP), 16000/16001
(edge-1/edge-2 gRPC ingest, if you want to point your own loadgen at one
from the host), 19101/19102 (collector-1/collector-2 HTTP). Since the
image has no shell, docker compose exec won't give you a prompt to poke
around in - use the published ports, or docker compose logs, instead.
Kill a collector to watch the edges' load-balance groups route around it, then bring it back and watch the WAL survive the restart:
docker compose stop collector-1
curl -s http://localhost:19102/stats # collector-2 now absorbing all the traffic
docker compose start collector-1 # replays its WAL, rejoins the rotationmake compose-down tears everything down including the named volumes.
k8s/ has manifests for the same topology as docker-compose.yml - 2
edges (each with its own loadgen) load-balancing across 2 collectors -
as plain Kubernetes objects (Namespace, ConfigMap, PersistentVolumeClaim,
Deployment, Service), applied in one shot via Kustomize:
make docker-build # build the image first (no registry push - see below)
kind load docker-image logflow:latest # or: docker save logflow:latest | k3s ctr images import -
kubectl apply -k k8s/
kubectl -n logflow get pods -w
kubectl -n logflow port-forward svc/edge-1 19100:9100 &
curl -s http://localhost:19100/statsEach collector and each edge is a named, single-replica Deployment +
Service (collector-1, collector-2, edge-1, edge-2) rather than a
2-replica StatefulSet, so the ConfigMap can carry configs/docker/*.yaml
completely unchanged - those configs address peers by the Docker Compose
service names they were written for, and this gives every agent that same
name as a Kubernetes Service too. k8s/README.md has the full writeup,
including the deliberate tradeoff this makes against horizontal scaling
and what a scaled version would need instead.
No image is pushed to a registry here - see "Docker deployment" under
Verified behavior below for why (this environment blocks every container
registry, not just one), and k8s/README.md for how to load a locally
built image into kind/k3s/minikube without one.
Every collector in the topologies above is a single point of failure for
its own data: if collector-1 dies, whatever was buffered in its WAL is
unavailable until it comes back, and if its disk is lost, that data is
gone. internal/cluster adds an alternative: wrap a channel in a cluster:
block and N agent processes replicate it via Raft
(leader election + log replication) instead of each holding an independent
copy - every replica ends up with a full durable copy of every committed
event, one node is automatically elected leader, and the group survives
losing a minority of its nodes (2 of 3, 3 of 5, ...) without losing data or
needing a human to intervene.
Why a hand-rolled Raft instead of a library. This project already
hand-builds its own channels, retry/backoff, load-balancing and failover
rather than pulling in a message-queue client; the same reasoning applies
here. hashicorp/raft, the obvious off-the-shelf choice, turned out to
need Go 1.25 (newer than this project targets) and pulls in a chain of
indirect dependencies (go-hclog, go-metrics, go-immutable-radix,
golang-lru, fatih/color, and what those need in turn) just to get a
terminal-color-aware logger none of this needs. The core Raft algorithm
(Ongaro & Ousterhout, In Search of an Understandable Consensus
Algorithm) is well-specified and modest in size, so internal/cluster
implements it directly - zero new third-party dependencies, transported
over the stdlib's net/rpc (a separate internal control-plane protocol
from the gRPC data path the rest of logflow uses).
What's in scope, and what's deliberately not. Implemented: leader
election with randomized timeouts, log replication with the standard
consistency check and backtracking, the majority-commit rule (including
the term-matching safety condition from Raft §5.4.2 that keeps an old
leader's uncommitted entries from being silently committed later),
persistence of term/vote/log across a process restart, live cluster
membership changes via joint consensus (see "Live membership changes"
below), and sharding across multiple independent Raft groups to scale
write throughput past one group's single-leader ceiling (see "Sharding"
below), and opt-in Raft log compaction (see "Log compaction" below).
Deliberately out of scope: automatic shard rebalancing
(shard count and each shard's peer list are fixed at config time - see
"Sharding"). Put() calls are batched - see "Batching" below - the same
batch_size/batch_linger trade-off GRPCSink already makes, applied
one layer up to the Raft log itself.
Off by default (snapshot_threshold: 0 - the log is kept forever, the
original behaviour). Set snapshot_threshold: N under a channel's
cluster: (per shard for sharded channels) and, once N entries have been
applied since the last snapshot, each node discards the applied prefix of
its Raft log, keeping snapshot_trailing recent entries (default N/4) so a
follower that is only slightly behind still catches up from the log. What
survives is the boundary - the last included index, its term, and the
cluster configuration in effect at it - persisted with the rest of the Raft
state, so restarts resume from it instead of replaying history, and a
membership change whose config entries have been compacted away is still
remembered (a node restarted with a stale bootstrap peers list picks up the
real membership from the snapshot).
A follower or brand-new member whose next needed entry has been compacted
away is sent an InstallSnapshot RPC and then resumes normal replication
after the boundary. The snapshot is the boundary only, not event data:
this state machine is "append to the local channel", and a channel's
contents (some already consumed) can't be exported, so a node that catches
up this way has no copy of the events in the range it skipped (it logs
installed snapshot ... log positions A..B were not replayed). Those events
are still on every node that applied them, so durability is unchanged while
a majority that applied them survives - but a late joiner is not a full
replica of history. Leave compaction off if every replica must hold
everything; the cost is a log (and a new member's catch-up) that grows
without bound. Compaction only advances over entries a node has applied,
so a node whose local channel is full (see "Stuck-apply detection") stops
compacting until it drains. Observability: raft_log_entries /
raft_snapshot_index on /stats and
logflow_cluster_raft_log_entries / logflow_cluster_raft_snapshot_index
on /metrics.
- Compose:
docker-compose.cluster.ymlruns three Raft collectors (cluster-1..3), an edge that load-balances across them, and a loadgen:make build && docker compose -f docker-compose.cluster.yml up --build.COLLECTOR_CFG=collector-sharded.yamlswitches to two sharded Raft groups on the same three nodes. All three collectors run the same config file (configs/docker/cluster/); the per-node identity comes fromNODE_ID. - Kubernetes:
k8s/cluster/(kubectl apply -k k8s/cluster) is a separatelogflow-clusternamespace with a 3-replicaStatefulSet, a headlesscollector-raftService for stable pod DNS, and an edge. All pods share one ConfigMap (generated from the compose config byscripts/gen-k8s-cluster-config.py);NODE_IDcomes from the pod name. It is the single-group layout (the sharded config is compose-only), andreplicas: 3must match the peer list - resizing needs a live membership change, notkubectl scale. - Two config features make one shared file work:
${VAR}/${VAR:-default}expansion in config files (an unset variable without a default is a startup error, a bare$is left alone), andcluster.listen_addr/client_listen_addr, which set what a node binds separately from the peer address others dial (a pod can bind:8301while advertisingcollector-0.collector-raft:8301, a name that may not resolve to it yet).
How it wires into the existing pipeline. cluster.ReplicatedChannel
implements the same channel.Channel interface every other channel type
does (Put/Take/Stats/Close), wrapping an underlying channel (a
FileChannel, so a replica's copy survives its own restart too, not just
the group's). This is what keeps the integration small:
Put(): if this node is leader, propose the event to the Raft group and block until a majority commits it and this node applies it locally. If not leader, transparently forward the write to whichever node is currently believed to be leader (redirecting once more if that belief was stale) - so a gRPC source needs zero cluster-awareness; any node in the group can accept a client's write.Take(): only the current leader actually drains the underlying channel; a follower'sTake()blocks (polling leadership, not the channel) until it either becomes leader or the caller gives up. Since a collector's sinks just callTake()in a loop already, this is the entire mechanism behind "only the leader's sinks run" - no separate gating logic was needed anywhere else in the agent./statsand/metricsgaincluster_state/cluster_term/cluster_leader/cluster_leader_changes(and Prometheuslogflow_cluster_is_leader/logflow_cluster_term/logflow_cluster_leader_changes_total) on any clustered channel, via the same type-assertion pattern/statsalready used forSpilled()/PurgedUnconsumed().
Batching. An early version of this feature proposed one Raft log entry
per Put() call, which measured at ~113-120 Apply()s/sec on a 3-node
localhost cluster (internal/cluster's TestApplyThroughput keeps that
number pinned as a regression test) - every single event paying for its
own full replication round trip. ReplicatedChannel now batches the same
way GRPCSink already batches a network send: concurrent Put() callers
join a shared pending buffer, which flushes as one logflowpb.EventBatch-
encoded Raft entry once batch_size events have joined it or
batch_linger has elapsed since the first one, whichever comes first
(cluster.batch_size/cluster.batch_linger in a channel's config,
defaulting to 200/50ms - see ClusterConfig in internal/config). Fixing
this took two changes, not one: internal/cluster/channel.go's batching
itself, and a real bottleneck one layer up it exposed -
internal/source/grpc_source.go's GRPCSource.putBatch used to Put()
every event in an incoming gRPC batch sequentially, which meant at most
one event was ever in flight at a time and each one paid out its own full
batch_linger wait individually instead of ever joining a shared batch;
putBatch now fires all of a batch's Put() calls concurrently (safe -
channel.Channel implementations are already required to support
concurrent Put()/Take()), which is what actually lets batching
coalesce in practice. See Verified behavior below for the before/after
numbers, both the raw Apply() measurement and a live end-to-end run.
Live membership changes. A cluster's peer list no longer has to be
static for a node's whole process lifetime. AddServer/RemoveServer
(on cluster.Node, and ReplicatedChannel's thin passthroughs of the
same names) drive Raft's standard two-phase joint-consensus protocol
(Ongaro & Ousterhout §6) to add or remove a member live, with no restart
of any existing node:
- Cold,new - the leader proposes a joint configuration naming both
the current member set and the target one. From the moment this entry
is merely appended to a node's log (not yet committed), that node
requires separate majorities of Cold and Cnew to agree on anything
- elections, commits - closing the window where two disjoint single-majority views could otherwise each elect their own leader mid-change.
- Cnew - once Cold,new commits, the leader proposes the new configuration alone. Once that commits, the change is final. A leader that removed itself from Cnew steps down once this commits (Raft §6.2), rather than going on managing a cluster it's no longer part of.
Adding a server runs a catch-up phase first: the leader replicates its
full log directly to the new node - which doesn't count toward any quorum
yet - until it's caught up, then proposes Cold,new, so a brand-new,
empty-log member can't stall everyone else's quorum the moment it's
added. A node meant to join this way is started standalone, with
cluster.joining: true and cluster.self instead of a full
cluster.peers list (see configs/cluster-demo/collector-4.yaml) - it
answers RPCs and can be caught up and voted for, but deliberately never
originates an election itself until it observes a real multi-member
configuration in its own log, so it can't disrupt a live leader before
it's actually a member.
The reverse problem - a node that was removed, whose process is still
running - gets the same deliberate treatment, and it's worth calling out
because it's exactly the kind of bug that only shows up under live,
multi-process testing, not an in-process unit test: once RemoveServer
commits, the leader's normal replication stops contacting the removed
node at all (it's no longer in the config), so if nothing else were done,
that node's own election timer would eventually fire with nothing to
suppress it - and since a RequestVote handler that updates its own term
before checking cluster membership would still adopt that climbing term
and (if it were leader) step down, a decommissioned-but-still-running
process could disrupt the real cluster indefinitely, forcing pointless
re-elections it could never actually win. Two things close this off:
RequestVote rejects a candidate that isn't part of the voter's own
current configuration before touching its term or stepping down (not
just before granting the vote), and the leader gives a removed node one
best-effort direct courtesy push of the final Cnew entry after commit, so
a still-running removed node actually learns it was removed and stops
calling elections altogether, rather than retrying forever into a void.
Two HTTP endpoints on every agent (not just a cluster node's own Go API)
make this operable without writing code: POST /cluster/add?id=...&addr=...&client_addr=...
and POST /cluster/remove?id=..., both callable against any node in the
group - a non-leader forwards the request (up to 2 redirect hops, the
same bounded pattern ReplicatedChannel.Put already uses to forward a
client write) to whichever node it currently believes is leader. /stats
gained cluster_phase (stable/joint) and cluster_members (the live
member-ID list) alongside the existing cluster_state/cluster_term/
cluster_leader fields.
What clustering doesn't give you for free. Replicating the data
doesn't by itself give clients a single virtual endpoint - there's no
built-in VIP, DNS failover, or service mesh here. An edge still needs to
be configured with all of a cluster's addresses (a failover or
load_balance sink group, same as against any non-clustered multi-collector
topology) so that if the specific node it's connected to dies - leader or
not - it reconnects elsewhere. What Raft buys is that wherever it
reconnects to already has every committed event, not that it never needs
to reconnect at all. configs/cluster-demo/edge.yaml is wired this way on
purpose, and its header comment spells this out.
configs/cluster-demo/collector-{1,2,3}.yaml is a runnable 3-node
bootstrap example (the one exercised in Verified behavior below) -
cluster.peers lists all three members' Raft address and
client-forwarding address, and cluster.node_id is the only thing that
differs between the three files besides the obvious per-node ports/paths.
configs/cluster-demo/collector-4.yaml is a 4th node meant to be started
after those three and added live (cluster.joining: true - see "Live
membership changes" above). Not yet wired into docker-compose.yml/k8s/
- see Known limitations.
Sharding. A single Raft group's write throughput is capped by its one
leader's batched-commit rate - batching (above) raised that ceiling
roughly 7x, but it's still one ceiling, and adding more nodes to one
group doesn't raise it further (more nodes just means more copies of the
same log, replicated by the same one leader). cluster.ShardedChannel
gets past that by partitioning one logical channel's events across N
independent Raft groups ("shards") instead of just one, each a full
ReplicatedChannel - its own leader election, log, replication, and
batching - completely unaware of any other shard. With S shards, S
different physical nodes can each be leading a different shard at the
same time, so aggregate write throughput scales roughly with S instead of
being capped by any single leader.
Routing a Put() to a shard is the whole mechanism: ShardedChannel.Put
picks a shard first (round-robin by default - maximum throughput spread,
no ordering preference; or by a hash of a configured event header, via
sharded_cluster.shard_key_header, to keep everything with the same key
sticky to one shard's single ordered log), then that shard's own
ReplicatedChannel.Put does its usual "I'm leader, apply it" or "forward
to whoever is" dance, completely unaware it's one of several shards. A
gRPC source calling Put() never needs to know sharding is involved at
all - the same client transparency a plain (non-sharded) ReplicatedChannel
already gives any node in its group.
What sharding does NOT give you, and why that's a deliberate, documented trade-off rather than an oversight:
- No ordering across shards. Each shard is its own totally-ordered
Raft log; two events that land on different shards have no defined
relative order, ever, even if one was
Put()before the other returned (this project's channels never promised an ordering guarantee across concurrentPut()callers in the first place, sharded or not). - No cross-shard atomicity. A single
Put()call always goes to exactly one shard - there's no multi-shard transaction. - No automatic rebalancing. The shard count and each shard's peer list
are fixed at config time; splitting or merging shards, or moving a key
range from one shard to another, isn't implemented. Each shard's own
membership can still change live (
AddServer/RemoveServer- see "Live membership changes" above) independently of every other shard, but that changes who replicates that shard, not how many shards there are.
A sharded channel's HTTP surface mirrors the non-sharded one, with one
addition: /stats reports shards=N plus a shard[i]_state/
shard[i]_term/shard[i]_leader/shard[i]_phase/shard[i]_members block
per shard (the sharded analogue of cluster_state/cluster_term/etc.),
and /cluster/add//cluster/remove both require a &shard=N query
parameter identifying which shard's Raft group the request targets, since
each shard has its own independent membership (a request with no shard=
against a sharded channel returns 400, same as an out-of-range or
non-integer shard index does).
configs/sharded-demo/collector-{1,2,3}.yaml is a runnable 3-node x
2-shard example (the one exercised in Verified behavior below): each
physical node participates in both shard-0's and shard-1's Raft group as
one of their 3 members (so sharded_cluster.shards lists two full
cluster:-style blocks, one per shard, with distinct ports), configured
with a channel the gRPC source writes to exactly as if it weren't sharded
at all.
Every agent exposes real Prometheus metrics at /metrics via
client_golang
(internal/metrics) - the standard counter/gauge/histogram types and
promhttp exposition format, plus the usual process/Go-runtime
collectors (goroutines, GC pauses, RSS, open FDs) every
Prometheus-instrumented service gets for free. What's actually
instrumented:
logflow_events_received_total,logflow_events_forwarded_total,logflow_events_forward_failed_total,logflow_events_dropped_total- counters at the source/sink boundary.logflow_channel_size,logflow_channel_capacity,logflow_channel_put_total,logflow_channel_take_total,logflow_channel_dropped_total,logflow_channel_spilled_total,logflow_channel_purged_unconsumed_total- one gauge/counter set per channel, labeled by channel name, collected on demand from each channel's existingStats()(a customprometheus.Collectorininternal/agent/channel_collector.go, not a name-string metric per channel) so there's no separate bookkeeping to keep in sync with the channel's own counters.logflow_batch_send_duration_seconds- a histogram around every gRPC sink batch send (internal/sink/grpc_sink.go), bucketed from 1ms to 10s to cover everything from a loopback send to a loaded, retrying network sink, for real p50/p95/p99 batch latency instead of just an average.
observability/ has a ready-to-go Prometheus scrape config and a
provisioned Grafana dashboard (6 panels: events received, forwarded vs.
failed, channel fill, dropped events, batch latency percentiles, and
cumulative spilled/purged-unconsumed) - wired into both
docker-compose.yml (prometheus/grafana services) and k8s/
(41-prometheus.yaml/42-grafana.yaml) so either one comes up with
metrics and a dashboard already in place, no manual setup. See the
Deployment sections above for exact URLs.
observability/alerts.yml is a Prometheus rule file (loaded via
prometheus.yml's rule_files:) covering the failure modes the metrics
above actually expose:
| Alert | Fires on | Severity |
|---|---|---|
LogflowTargetDown |
a collector/edge stops responding to scrapes for 1m | critical |
LogflowEventsDropped |
the source/sink boundary counter increases at all | warning |
LogflowChannelDropped |
a channel's drop counter increases at all | warning |
LogflowChannelNearFull |
a channel stays over 80% full for 2m (leading indicator before it starts dropping) | warning |
LogflowForwardFailuresHigh |
nonzero forward-failure rate to a sink target | warning |
LogflowPurgedUnconsumed |
a collector purges retention-expired, never-forwarded WAL data on restart - real, already-lost data | critical |
LogflowBatchLatencyHigh |
p95 gRPC batch send latency stays over 1s for 2m | warning |
Prometheus ships firing alerts to observability/alertmanager.yml
(Alertmanager, alerting: alertmanagers: pointed at the alertmanager
service/Service by name) - a minimal routing tree (group by
alertname+job, severity=critical gets a shorter repeat interval) with
one webhook receiver pointed at a placeholder URL, plus an inhibit rule so
a target that's already down doesn't also page for every derived
symptom (drops, latency, forward failures) on the same job. Swap the
receiver's url: for a real webhook (Slack/PagerDuty/OpsGenie) to get
actual notifications; everything else (routing, grouping, inhibition)
works as-is. Wired into both docker-compose.yml (alertmanager service,
port 19093) and k8s/ (43-alertmanager.yaml), using the exact same
config files as the rest of the observability stack.
.github/workflows/ci.yml runs on every push/PR to main, as four
independent jobs:
| Job | What it checks |
|---|---|
test |
go mod verify, gofmt -l (fails on any unformatted file), go vet ./..., go build ./..., go test -race ./... |
docker |
make build (host binaries) + docker build (the real FROM scratch Dockerfile) + docker compose config -q (Compose file syntax) |
manifests |
kustomize build k8s/ - renders every object in k8s/, catching bad kustomization.yaml entries, malformed YAML, or broken ConfigMap/volume references |
observability |
promtool check rules/check config and amtool check-config against the real observability/*.yml files, using the same Prometheus/Alertmanager release binaries the Verified behavior section downloads below |
Dependencies are not vendored: each job downloads modules from the Go module
proxy (go mod download) and go mod verify checks them against go.sum.
The test job also runs go mod tidy and fails if that changes go.mod or
go.sum, so those files can't drift from the imports.
Caching: actions/setup-go (cache: true, cache-dependency-path: go.sum)
caches the module cache (~/go/pkg/mod, keyed on the go.sum hash) and the
Go build cache; the downloaded kustomize, promtool and amtool binaries
are cached by version with actions/cache@v4. The manifests job also renders
k8s/cluster/ and fails if k8s/cluster/01-configmap.yaml is out of sync
with configs/docker/cluster/; the docker job also checks
docker-compose.cluster.yml. Not verified: an actual runner has never
executed any of this (the workflow was only YAML-parsed here).
This was built and exercised end-to-end across two sandboxed environments
with no proxy.golang.org access (dependencies were fetched directly from
GitHub source and vendored locally for this reason, with a replace block in
go.mod that was removed before publishing; a normal machine just needs
go build ./...), using Go 1.22 for
the original feature work and Go 1.24 for the Docker deployment work:
- Standalone agent sustained ~19,900 events/sec end-to-end (loadgen → gRPC source → sharded memory channel → sink) with zero drops over a 5s / ~100k event run.
- Two-tier edge→collector forwarding over gRPC verified with zero data loss under normal operation.
- Crash recovery of the durable file channel verified: killing the
collector agent mid-flight (
SIGKILL) and restarting it replayed the unconsumed backlog with no data loss (and, as documented above, replayed the tail of the already-consumed-but-undeleted active segment, matching the documented at-least-once guarantee). - Multiplexing channel selector verified: events tagged
env=prodwere routed exclusively to theprodchannel (0 landed indefault), and untagged events landed exclusively indefault(0 inprod). - Load-balance sink group verified: 3,000 events split across two independent backend agents (802 / 2,198 — uneven but confirming both were genuinely receiving traffic concurrently).
- Failover sink group verified through a full cycle: traffic flowed
through the priority-0 target only; killing that process mid-stream
(
SIGKILL) triggered failover to the priority-1 target with zero events lost (all subsequent traffic landed on the backup); restarting the priority-0 target triggered automatic promotion back to it within onerecovery_probe_interval, confirmed by post-recovery traffic landing on the primary again. - TLS/mTLS/token auth verified through four scenarios against a source
configured with
client_auth: true+auth_tokenset: a plaintext client is refused at the transport level (can't even complete a TLS handshake); a TLS client with no certificate is refused withtls: certificate required; a client with a valid certificate but the wrong token is refused withUnauthenticated: invalid authorization tokenand 0 events land; a client with the correct certificate and token successfully delivers events (verified with both a direct client and a full edge→collector agent-to-agent hop, including confirming the edge agent's sink keeps retrying safely — no data loss,droppedcount stays at 0 — when given a deliberately wrong token, rather than silently discarding events). - Spillable memory channel verified under real load: a 500-event memory
primary against a ~20K events/sec, ~100K-event burst spilled 98,392
events to the durable file overflow with zero drops and a full
clean drain (
put=104475 take=104475 dropped=0). This surfaced and fixed two real bugs along the way (see below) — worth knowing about since they'd otherwise silently undermine the durability guarantee. - File channel CRC32 corruption detection verified two ways: flipping a byte inside an entry's length prefix (corrupting "how many bytes to read" into a huge number) previously crashed the process with an out-of-memory error — fixed by bounding the trusted length against a sane maximum before allocating; flipping a byte inside an entry's payload/checksum region is caught by the CRC32 mismatch. In both cases, after the fix, recovery now logs the exact byte offset and reason, stops replay of that segment at the corrupted entry, and every entry it does return is confirmed to be genuine original data — never garbage from a misaligned read.
Bugs found and fixed while building this feature (documented here rather than quietly folded in, since they're the kind of thing that matters for a durability feature specifically):
- The spillable channel's Put briefly retried the primary channel with a
time.After(1ms)sleep between attempts before spilling. SinceTake()drains the overflow with priority, the primary stays full once spilling begins, so every event was paying that retry cost — enough that under real load the server fell far behind the client, making a normal test look like a hang. Fixed by making the primary check a single non-blocking attempt with immediate fallback to overflow. - That same retry path was (before the fix above) also inflating the
primary channel's drop counter for every spilled event, since
MemoryChannel.Putcounts any context-timeout as a drop — even one from an intentionally short probe. Fixed by adding aTryPutthat never touches the drop counter on failure. - A corrupted or torn length prefix was used directly to size a byte
slice allocation (
make([]byte, n)) with no upper bound, so a few flipped bits could crash the whole process with an out-of-memory error instead of being caught as the corruption it is. Fixed with a sane maximum entry size checked before allocating.
- Zero-copy I/O verified with a before/after benchmark
(
go test -bench=. -benchmem ./internal/channel) on a full file-channel Put+Take round trip: pooling the write-side encode buffer and aliasing the read-side decode buffer (instead of copying it) took allocation cost from 776 B/op, 14 allocs/op to 592 B/op, 13 allocs/op for the same operation, confirmed by reverting the optimization and re-running the identical benchmark to get the "before" numbers. - Graceful shutdown verified through both paths: (1) a producer streaming
continuously was allowed to finish naturally after SIGTERM -
GracefulStop()correctly waited for the in-flight RPC rather than severing it, and the full sequence (stop ingestion → nothing to drain → stop sinks → close) completed and logged cleanly; (2) with a 30-second producer still streaming, a second SIGTERM sent partway through the wait forced an immediate stop - logged abandoning both the source-stop wait and the drain wait - exiting in about 7 seconds instead of waiting out the remaining ~27. - Active WAL retention/purging verified live: a burst of 3,900 events with
a 4KiB max segment size (forcing 80 rolled segments) and no sink
attached, against a 3-second
retention_max_age. Before the retention window elapsed: 80 segments on disk, 3,900 buffered, 0 purged. After one sweep past the window: 79 of 80 segments purged (only the still-active segment remains),purged_unconsumed=3871correctly surfaced in/stats, with a clear per-segment log line for each (removed segment N (older than retention_max_age (3s)), discarding 49 unconsumed event(s)).
A serious pre-existing bug found and fixed while building this feature
(not introduced by it - present since the file channel's locking was
refactored earlier in this project - but only exposed once a test forced
multiple segment rolls with partial consumption, which no earlier test
had done): openNewWriteSegment created two separate struct instances
for the same logical segment - one for the write path, one for the read
path - so the read-side written counter was never incremented. That
made maybeGC's "has every entry been consumed" check
(consumed >= written) evaluate to true after just the first Take()
from any rolled-but-not-yet-fully-consumed segment, silently deleting it
- discarding every other still-unconsumed entry in it, with no error, no
dropped-count increment, nothing. This would have affected any real
workload where segments actually roll (small
max_segment_bytesrelative to sustained volume) and a consumer only partially drains a segment before the next one rolls; it went unnoticed earlier because every prior test used a segment size large enough that everything fit in one never-rolled segment, where a separate early-return guard (seg.id == fc.writeSeg.id) happens to skip the buggy check entirely. Fixed by using one shared*segmentstruct (holding both a write handle and a read handle) forfc.writeSegandfc.readSegs[id]while a segment is active, so both bookkeeping counters are always consistent regardless of which path touches them. Verified via a dedicated regression test (TestFileChannelDoesNotDropUnconsumedOnPartialTake) that fails cleanly against the unfixed code and passes against the fix.
Docker deployment, verified live with the real docker compose up
topology (2 edges, 2 collectors, 2 loadgens), in an environment where
every container registry (registry-1.docker.io, ghcr.io, gcr.io,
quay.io, mcr.microsoft.com, public.ecr.aws, registry.k8s.io) was
unreachable (org egress policy, confirmed via the proxy's own status
endpoint - not a transient failure). That's what drove the image design
above (FROM scratch + host-compiled binaries instead of a
golang:... build stage, plus the custom healthcheck binary in place of
wget): it's the only design that builds and runs with zero registry
access, and it turned out to also be a legitimately smaller/more
auditable image, not just a workaround.
make buildcompiledagent/loadgen/healthcheckas static Linux binaries from the locally vendored dependencies (at the time: no network calls needed; the published repo now fetches modules viago mod download).docker buildproduced a 40.6MB image (11.7MB uncompressed content) with noFROMpull at all - confirmed via the build log showing every step as a localCOPY, nothing fetched.docker compose up -dbrought up all 6 containers (2 edges, 2 collectors, 2 loadgens) in correct dependency order viadepends_on: condition: service_healthy- collectors healthy first, then edges (which depend on both), then loadgens (which depend on their edge) - confirmed via the compose output'sHealthy/Startedsequencing anddocker compose psshowing all four long-running services(healthy).- End-to-end traffic flow confirmed via
/statson all four agents simultaneously: loadgen → edge (gRPC ingest) → load-balance sink group → collector (gRPC source → durable file channel → file sink) withput == takeanddropped=0at every hop, and the collectors' on-disk output files (pulled out withdocker cpfor inspection, since the scratch image has no shell toexecinto) containing the actual forwarded log lines. - File backends in sink groups: with a gRPC primary down, a
failovergroup spooled 2000/2000 events to itsfile:backup; once the collector started, the prober promoted it and the next 2000 reached it (0 lost). A genuineENOSPC(/dev/full) requeues the batch instead of discarding it (regression test; fails if the requeue is removed). - Per-caller tokens, live: with
edge-a(no expiry) andedge-b(expires 9s out) in a token file,edge-astreamed 500/500; afterrevoked: truewas edited into the file (no restart) its next connection was refused;edge-b's long-running stream was cut at expiry withsent=4000 acked=4000(nothing in flight lost) and the metrics showedrejected{edge-a,revoked}andrejected{edge-b,expired}. Unit tests cover identity, expiry, revocation, hot reload, bad-reload fallback, file validation and cutting off an open stream (mutation-checked: disabling the re-check fails the test). - Parallel WALs (
wal_shards: 4on a file channel): 15000/15000 events acked acrosswal-0..3, then akill -9and restart recovered all 15000 from the four logs; restarting withwal_shards: 2on that directory was refused with an error rather than stranding data. Measured on this 2-core sandbox (go test -bench ParallelPut ./internal/channel, 16 producers): ~2.4µs/Put with one WAL vs ~1.2-1.5µs with 2-8 WALs (~1.6x) - the gain is bounded by the cores/disk here, not by the design. - Raft log compaction, live (
configs/cluster-demo,snapshot_threshold: 50): 16000 events through a 3-node group left every node with ~36 retained log entries andraft_snapshot_index=190(vs. the whole log without it). Withcollector-3kill -9'd, 24000 more events were committed by the other two; on restart it loggedrestored persisted state ... log_len=16, theninstalled snapshot through index 190 ... positions 77..190 were not replayed, converged on the same index, and took part in the next 2000 events - holding 24400 of the 42000 events in its local channel, i.e. the documented gap. Unit tests cover bounded log growth, restart from the boundary, a far-behind follower and a brand-new member catching up by snapshot, membership surviving compaction + a stale-peers restart, and never regressing to an older snapshot (the last was a real bug the live run found: a follower with boundary 228 accepted the leader's 190 and dropped its retained log); mutation-checked by disabling the snapshot send, the membership fallback and the stale-snapshot guard. - Cluster deployment configs:
configs/docker/cluster/{collector,collector-sharded,edge}.yamlwere run as real processes - three collectors with the identical file, differing only in env (NODE_ID, a per-node loopbackBIND_IP, data dir) - plus an edge load-balancing to them by hostname: leader elected, 12000/12000 events replicated to all three nodes and written by the leader's sink; the sharded variant elected different leaders for the two shards (cluster-2,cluster-3) and delivered 12000/12000. The k8s ConfigMap's collector config was loaded withNODE_ID=collector-0and started, binding:8301while advertising an unresolvable DNS name. Not verified here: no container runtime or Kubernetes cluster is available, sodocker-compose.cluster.yml(checked withdocker compose config) andk8s/cluster/(YAML parse only - nokubectl/kubeconform) have never actually been run as containers/pods. One gotcha seen while testing: gRPC honoursHTTP(S)_PROXYfor non-loopback hostnames, so in an environment with a proxy set, add the collector names toNO_PROXY. - Raft fault injection (
Config.LinkFilterdrops outbound RPCs between chosen nodes, test-only): an isolated leader cannot commit, the majority elects a new one and commits, and after healing the isolated leader's uncommitted entry is discarded everywhere. A randomized 5-node chaos test (random isolations/2-3 splits/heals for 8s under 3 concurrent writers; ~650 acknowledged writes over ~7 terms) checks that no term has two leaders, every node applies the identical sequence, and every acknowledged write appears exactly once. Mutation-checked: weakening the quorum rule to "any one node" makes both fail. The chaos test also flaked once under CPU load, which exposed a real liveness gap: a new leader never committed anything of its own until the next client write, so entries inherited from earlier terms could sit uncommitted on a quiet cluster. Fixed with the standard Raft section 8 leader no-op (EntryNoop, never applied to the state machine), with a test that fails without it. This is in-process partition simulation, not Jepsen-style testing (no clock skew, disk faults or real network). - A second chaos-test finding, fixed:
Applyreported success oncelastApplied >= index, which only means some entry at that index was applied. A leader partitioned away mid-write could, after healing, have its slot overwritten by the new leader's entry and still acknowledge its own write - a lost acknowledged write (12 consecutive chaos runs now pass; it failed within 12 before).Applynow acknowledges only if the entry at its index still has its own term; otherwise it returnsErrNotLeaderso the caller retries (at-least-once). Deterministic regression testTestApplyNotAckedWhenEntryOverwrittenfails without the fix. - Load-balance failover under a real container kill:
docker stop collector-1while traffic was flowing - both edges immediately started logging reconnect attempts tocollector-1:7000(DNS removed along with the stopped container:no such host, correctly treated as a connection failure and retried) while continuing to forward everything to collector-2 with zero drops (edge-1put/takekept climbing in lockstep,dropped=0throughout).collector-2alone absorbed the full combined rate of both edges. - WAL durability across a real container restart:
docker start collector-1after the stop above replayed its WAL (seg-...0000.log, 3.5MB) and rolled a fresh write segment (seg-...0001.log), logs showing a clean graceful-shutdown-then-restart cycle, and resumed accepting traffic - confirmed both via/stats(put/takeclimbing again, stilldropped=0on both edges and the collector) and viadocker cp-inspecting the volume directly: WAL andout/both show files from both before and after the restart. docker compose stop(graceful, notkill) drained all 6 containers cleanly - confirmed via each agent's shutdown log sequence (ingestion stop → drain → sinks stop → channel close) rather than an abrupt kill.
Known rough edge, not a correctness issue: this image intentionally
runs as root (uid 0) inside the container. A FROM scratch image has no
/etc/passwd, and Docker initializes a freshly created named volume
(collector1-data/collector2-data here) as root-owned; a non-root
numeric USER would hit permission-denied on the file channel's
MkdirAll into that volume. Since scratch has no shell, package manager,
or setuid binary to escalate from in the first place, this doesn't carry
the risk it would on a full OS base image, but it's worth knowing if
you're adapting this for an environment with stricter container
policies (pre-creating the volume's ownership via an init container, or
switching to a non-root distroless base with its own minimal
/etc/passwd, are the usual fixes).
Kubernetes deployment. k8s/'s manifests were applied against a real,
running single-node cluster (k3s, built from its own GitHub release
binaries + airgap image bundle - kind's node image is itself a container
image, and this environment blocks every container registry, so kind
specifically wasn't an option here either). What that confirmed live:
kubectl apply -k k8s/succeeds end to end against a real API server: theNamespace,ConfigMap, bothPersistentVolumeClaims, all 6Deployments and all 6Services were all actually created and schema-valid, confirmed viakubectl getshowing every object present with the right selectors, ports, and labels.Deployments correctly producedReplicaSets, and everyPodreachedScheduledon the node - the manifests and image references in them aren't the problem.
What wasn't confirmed live: a pod actually reaching Running. This
sandbox's container runtime can create containers directly just fine
(docker run, and ctr run against the same containerd k3s itself uses,
both work) but fails specifically inside kubelet's CRI pod-sandbox
creation path, for every pod regardless of image - including the
cluster's own stock coredns, with nothing of logflow's involved. That
rules out the manifests or the image as the cause; it's a property of
this one sandboxed host's kubelet/runc interaction, not something the
k8s/ directory did wrong. Full details and the exact error are in
k8s/README.md's "What was verified, and what wasn't" section, including
what to do if this looks familiar on a normal cluster (short version:
it isn't this issue - look at cgroup driver or runc/containerd version
skew on that host instead).
Clustered collector (Raft). Verified at two levels: an in-process Go test suite exercising the Raft algorithm directly (fast, deterministic, runs in CI), and a live multi-process demo exercising the full pipeline (real OS processes, real TCP, a real forced leader kill mid-traffic).
In-process (go test ./internal/cluster/... -race, 5 correctness tests +
1 throughput measurement, all passing, -count=25 under -race with zero
failures or data races):
- Exactly one leader is elected and every node agrees who it is
(
TestElectsExactlyOneLeader). - 20 sequential
Apply()s on the leader are replicated, in order, to every node's state machine (TestApplyReplicatesToAllNodes). - A follower's own
Apply()is correctly rejected withErrNotLeader(TestFollowerRejectsApply). - Killing the leader elects a new one (with a strictly higher term) from
the remaining nodes, which continues committing new entries - every
surviving node ends up with the post-failover entry
(
TestLeaderFailoverContinuesCommitting). - A follower stopped mid-cluster, restarted after the leader committed 5
more entries while it was down, catches up to all 5 via normal Raft log
replication (
TestRejoinedNodeCatchesUp).
This test suite is also how a real implementation bug got caught and
fixed before it ever reached a live run: the first version of Close()
stopped a node's listener from accepting new connections but didn't
close its already-open ones, so a "stopped" node's net/rpc server kept
right on being served by a per-connection goroutine over its existing TCP
connection to the leader - a zombie node, logically removed from the test
harness's bookkeeping but still functionally live and applying entries,
indistinguishable from a real node until that stale connection happened to
break on its own. TestRejoinedNodeCatchesUp failed with the rejoining
node reporting entries it couldn't legitimately have yet, which is what
surfaced it. Fixed by tracking every accepted connection and force-closing
them in Close() (see serveTracked in internal/cluster/raft.go).
Live, end-to-end (configs/cluster-demo/: 3 clustered collectors +
1 edge + loadgen, all real OS processes on localhost, built from
make build's real binaries):
- All 3 collectors came up, held a real leader election over real TCP
(
net/rpc, not mocked), and agreed on the result - confirmed via each node's own/stats:cluster_state=leaderon exactly one (collector-1),cluster_state=followerandcluster_leader=collector-1on the other two. - Sent real traffic through a real edge agent. All 3 collectors'
putcounters advanced identically (508each) - genuine replication, not one node forwarding a summary - while only the leader'stakecounter advanced and only the leader's output file had real bytes in it (the followers' sink-rolled files existed but were 0 bytes), confirming the leader-gated-Take()design actually prevents triplicated output rather than just being intended to. kill -9'd the leader process outright mid-traffic. A new leader (collector-2) was elected within ~2 seconds (one real election, no split-vote retries needed this run) and immediately resumed draining - confirmed via/statson the two survivors agreeing on the new leader and term. The edge's own failover sink group independently detected the dead connection (rpc error: ... EOF) and redirected to the new leader about 150ms later, fully independently of the Raft layer underneath it noticing the same failure - exactly the "two separate failure-recovery mechanisms, client-facing and data-layer" split this section's doc comment describes.- Batching, before and after. The run described above (one Raft log
entry per
Put(), no batching) is what first surfaced the throughput ceiling: under 500 events/sec sustained load, the post-failover leader's accept rate couldn't keep up, the edge's channel backed up accordingly and dropped exactly 1 event at capacity, then drained cleanly with zero further drops - correct backpressure behavior, but driven by a real bottleneck, pinned in isolation byTestApplyThroughput: ~113-120 sequentialApply()calls/sec on a 3-node cluster over localhost loopback. Batching was added to fix this (see the "Batching" writeup above), and re-measured at both levels:- In-process, concurrent-caller throughput (
TestBatchedThroughput,internal/cluster/channel_test.go): 50 concurrentPut()callers driving 5,000 events through a 3-node cluster with the package's defaultbatch_size/batch_linger(200/50ms) measured ~794 commits/sec - roughly 7xTestApplyThroughput's unbatched number, from the same algorithm, same localhost loopback, same majority-of-3 commit rule - batching is the only variable that changed. 4 further correctness tests (TestBatchSizeTriggersFlush,TestBatchLingerFlushesSubThresholdBatch,TestBatchCoalescesIntoOneEntry,TestEveryPutGetsResult,TestCloseFlushesPendingBatch) all pass under-race -count=15with zero flakes or data races, confirming the size threshold, the linger timeout, correct per-caller results, and a cleanClose()with in-flightPut()s all actually work, not just the happy-path throughput number. - Live, end-to-end, re-running the exact same demo (3 clustered
collectors + 1 edge + loadgen, 500 events/sec,
kill -9the leader mid-traffic): this also caught a second, real bottleneck thatReplicatedChannel's own batching couldn't fix by itself -GRPCSource.putBatchwasPut()ing every event in an incoming gRPC batch sequentially, so at most one event was ever in flight and each one paid out its own fullbatch_lingerwait alone instead of joining a shared batch (measured: ~20/sec, worse than unbatched, before this was found and fixed - seeinternal/source/grpc_source.go). With both fixes in place (Raft-level batching + concurrentPut()per incoming batch), the same 12,450-event / 500-per-sec run, including the same mid-traffickill -9of the leader and failover to a new one, completed with zero backlog (edge's channel stayed atsize=0throughout -putandtakeadvanced together) and zero drops:edge /statsshowedput=12588 take=12588 dropped=0, the new leader's/statsshowedput=12450 take=12450(matching loadgen'ssent 12450 events), and the leader's output file held exactly12450lines (wc -l) - every event generated, replicated, and written to disk, through a real leader failure, with no perceptible backpressure at this load level.
- In-process, concurrent-caller throughput (
Live membership changes. Verified the same two ways as the rest of clustering: an in-process test suite exercising the joint-consensus protocol directly, and a live multi-process demo.
In-process (go test ./internal/cluster/... -race, 6 tests, all passing,
-count=15 under -race with zero failures or data races):
- Adding a node to a live 3-node cluster (
TestAddServerJoinsLiveCluster): the new node is started standalone (Config.Joining: true), added viaAddServer, and every node - including the new one - converges on a 4-membercluster_phase=stableview; a freshApply()after joining replicates to the new node too, confirming it's a real participant, not just a name on a list. - Removing a node from a live cluster and continuing to commit
(
TestRemoveServerLeavesLiveCluster). - A lone node started with
Joining: trueand no reachable peers never elects itself, even after many uncontested election timeouts (TestJoiningNodeNeverSelfElects) - the safety propertyJoiningexists for. - A second membership change is rejected with
ErrMembershipChangeInProgresswhile one is already in flight (TestChangeMembershipRejectsConcurrentChange) - the standard Raft restriction, actually enforced, not just documented. - A leader that removes itself steps down once Cnew commits, and the
remaining 2 nodes elect a new leader and keep committing
(
TestLeaderStepsDownAfterRemovingSelf) - Raft §6.2. TestRemovedNodeDoesNotDisruptCluster- a regression test for a real bug this feature's own live testing caught (see below): removes an unrelated node from a 3-node cluster without killing its process, then confirms the original leader's identity and term are unchanged 500ms later (anything else means the removed node disrupted it), the cluster still commits, and the removed node itself converges to believing it's no longer a member.
Live, end-to-end (configs/cluster-demo/: the same 3-collector bootstrap
cluster, plus collector-4.yaml started separately and added live):
- Started collector-4 standalone (
cluster.joining: true) alongside the already-running 3-node cluster, confirmed via its own/stats(cluster_state=follower cluster_term=0 cluster_members=collector-4) that it sat idle - never calling an election - for several seconds past its own election-timeout window. curl -X POST .../cluster/addagainstcollector-2(a follower, not the leader) added collector-4 live. The request forwarded to the real leader exactly as documented, and all 4 agents'/statsconverged oncluster_phase=stable cluster_members=collector-1,collector-2,collector-3,collector-4- zero restarts of any of the original 3 processes. Sent real traffic
(
loadgenagainst the leader) and confirmed collector-4's ownputcounter advanced identically to the leader's (980/980) - genuine replication to the newly-joined node, not just list membership.
- zero restarts of any of the original 3 processes. Sent real traffic
(
- Caught a real bug here that no unit test had found: calling
curl -X POST .../cluster/remove?id=collector-2without killing collector-2's own process (the realistic "decommission, forgot to alsokillit" scenario) sent the leader's term climbing from 14 to 46+ within seconds, with repeatedstepping down to follower/starting electionlog lines - the leader was being disrupted by the very node that was just removed, exactly the failure mode described above. Root-caused toRequestVoteadopting a candidate's term (and stepping down if it was leader) before checking whether that candidate was still a cluster member. Fixed by reordering that check first, plus the leader's courtesy push of the final Cnew entry so a still-running removed node actually learns to stop calling elections (both described in "Live membership changes" above, and now covered byTestRemovedNodeDoesNotDisruptCluster). Re-ran the identical scenario after the fix: removed collector-2 (process still running), waited 5 seconds, confirmed collector-1's term and leadership were completely unchanged (cluster_term=13before and after,cluster_leader_changes=1throughout), collector-2's own log shows zerostarting electionlines (it received the courtesy push and quieted down before its own timer ever fired), and a follow-uploadgenrun replicated cleanly to both remaining members with the leader untouched.
Sharding. Verified the same two ways as the rest of clustering: an in-process test suite, and a live multi-process, multi-Raft-group demo.
In-process (go test ./internal/cluster/... -race -run TestSharded, 9
tests, all passing; the 2 tests that run real multi-node Raft groups
additionally checked with -count=5 -race with zero failures):
- Round-robin distribution with no
ShardKeyFuncspreads evenly across every shard (TestShardedChannelRoundRobinDistribution): 100Put()s across 4 shards landed exactly 25 on each. - Key-based routing is sticky (
TestShardedChannelKeyRouting): the same key always resolves to the same shard across 10 rounds, while different keys are free to land on different shards; an empty key correctly falls back to round-robin instead of pinning every such event to shard 0 (TestShardedChannelKeyRoutingEmptyKeyFallsBackToRoundRobin). Take()fan-in aggregates every shard with zero loss and zero duplication (TestShardedChannelTakeAggregatesAllShards): 60 events put across 3 shards, all 60 delivered exactly once each, regardless of which shard'sfanIngoroutine happened to pull them.Stats()sums every shard's counters rather than reporting only one (TestShardedChannelStatsAggregates) -PutCount/TakeCountboth matched the real totals,Capacitymatched the sum of each shard's own capacity.Close()returns promptly (no goroutine hang) and aTake()after it fails cleanly instead of blocking forever (TestShardedChannelCloseIsPromptAndIdempotentPerShard).- An out-of-range shard index is rejected with a clean error from both
AddServerandRemoveServer, not a panic (TestShardedChannelShardIndexOutOfRange) - the same checkhandleClusterMembership's&shard=NHTTP parameter relies on. TestShardedChannelIndependentRaftGroups- the real integration test: 2 shards, each a genuine 3-node Raft group (6 realNodeinstances), wired into oneShardedChannel. Confirmed each shard elects its own leader independently (not necessarily the same-numbered node in both groups), both groups show real non-trivial Raft state (ClusterTerm() >= 1, 3-memberClusterMembers()), and 40 events round-robined across theShardedChannelwere all delivered exactly once end-to-end through two completely independent logs.- Caught and fixed a real bug while writing this suite, not a product bug
but a test-harness goroutine leak worth recording because of how it
presented: an early version registered both groups' cleanup in a single
deferplaced after the loop that creates them; sincet.Fatalfunwinds viaruntime.Goexitand only runs defers already registered by that point, a timeout on the second group's election would skip cleanup for the first group entirely, leaking 3 live Raft goroutines that kept calling uncontested elections into the test binary's background - surfacing not as a failure in that test, but as a completely unrelated-looking later test's leader term having inexplicably climbed into the thousands. Fixed by registering each group'sdefer groups[i].closeAll()immediately as it's created, inside the loop.
Live, end-to-end (configs/sharded-demo/: 3 physical agent processes,
each participating in both of 2 independent shards - 6 real Raft Node
instances total, same topology as the integration test above but as
actual separate OS processes talking over real sockets):
- Started all three
collector-{1,2,3}processes and confirmed via/statsthat the two shards elected different leaders on different physical nodes at the same time -shard[0]_leader=collector-2 shard[1]_leader=collector-3- the concrete scaling property sharding exists for (two different nodes genuinely leading concurrently, not one leader doing all the work). - Sent 20,000 events via a small precise verification client (built
against the real
SendBatchunary RPC, which blocks for each batch's ack before the next - chosen overloadgen's streaming mode specifically because that mode fires sends asynchronously and tears down the connection as soon as its wall-clock duration elapses without waiting for in-flight acks, which this verification caught producing a misleading ~17% apparent "loss" that was actually just unacked in-flight sends at shutdown, not real data loss - a pre-existing imprecision inloadgenitself, unrelated to sharding, not investigated further since the precise client settles the question directly). Result:sent=20000 acked=20000 nacked=0, and every one of the 20,000 unique event bodies was found exactly once across the leader nodes' file sinks (grep -o ... | sort -u | wc -l= 20000) - zero loss, zero duplication, confirmed by actually reading the output files, not just trusting the ack count. - Verified per-shard membership independence end-to-end over HTTP, not
just via the in-process test:
POST /cluster/remove?id=collector-1&shard=0removed collector-1 from shard-0's membership only (shard[0]_members=collector-2,collector-3afterward) while shard-1's membership was completely untouched (shard[1]_members=collector-1,collector-2,collector-3) - confirming the&shard=Nparameter actually scopes the change to one independent Raft group, not the whole sharded channel. - Verified the new HTTP validation paths directly: a
/cluster/removeagainst a sharded channel with noshard=parameter returns400("channel is sharded - shard query parameter is required"), an out-of-rangeshard=5(only 2 shards configured) returns400("shard 5 out of range (have 2 shard(s))"), and a non-integershard=abcreturns400("shard query parameter must be an integer") - none of the three panics or silently no-ops.
Stuck-apply observability, apply ordering, and loadgen accuracy.
Three small fixes closing former Known limitations:
- Stuck follower apply. A node whose local channel stays full used to
block its Raft applier with no external signal.
ReplicatedChannelnow runs a watchdog around each localPut: pastapplyStallWarnInterval(2s,SetApplyStallWarnInterval) it is reported via/stats(apply_stuck=true apply_stuck_seconds=N apply_stall_warnings_total=N), Prometheus (logflow_cluster_apply_stuck,..._stuck_seconds,..._stall_warnings_total), a rate-limited log line, and a newLogflowClusterApplyStuckalert (promtool check rulespasses). ThePutitself deliberately stays unbounded: every channel's ownPutcounts a cancelled attempt as a dropped event, so "retry with a short timeout" would have corrupteddropped_totalwith false drops for entries never actually abandoned. Live: a 1-node cluster on a capacity-50 channel with no sink wentapply_stuck=truewithin seconds withdropped=0. - Apply ordering race (found while doing this).
applyLoopbumpedlastAppliedbefore callingApplyFn, so a clientPutcould be told "applied" before the entry reached the local channel. Now bumped after. Regression tests:TestApplyStuckDetectionAndRecovery,TestApplyOrderingWaitsForApplyFn. Throughput unchanged (827/sec batched vs ~794 baseline; 124/sec unbatched). loadgennow counts on ack, waits up to-ack-grace(10s) for trailing acks, and printssent= acked= nacked=; live run: 10000/10000, matching the agent'sputcounter.
Per-channel drain_timeout. drain_timeout on a channels: entry
overrides the agent-wide value for that channel only (zero inherits it);
each channel is waited on until empty or past its own budget, and one
that blows its budget stops holding up the others. Per channel, not per
sink, because "drained" means a channel emptying - sinks sharing a channel
drain the same events, so a per-sink budget wouldn't be well-defined.
Live: agent default 30s, two undrained channels with 1s/4s overrides -
SIGTERM to exit took 4.0s, and the log named the channels that timed out.
Certificate rotation/reload. TLS certificates are no longer read once
at startup. Every tls: block's cert/key pair (server identity, or the
client certificate presented for mTLS) and CA bundle (client CAs on a
server with client_auth, the server-verifying CA on a client) is
re-read when its file's mtime/size changes (checked lazily at handshake
time, at most once a second - no new dependency, no fsnotify), via
tls.Config callbacks (GetCertificate, GetClientCertificate,
GetConfigForClient, and VerifyConnection for the client-side CA, since
RootCAs is fixed at construction). New connections use the new
material; an already-established gRPC stream keeps the certificate it was
negotiated with until it reconnects (normal TLS behavior). A reload that
fails - a half-written file, a cert/key mismatch mid-rotation - keeps
serving the last good certificate and retries on the next check instead
of breaking every handshake. Verified: 4 tests with real generated CAs and
real TLS handshakes (server cert rotation, torn rotation falling back then
recovering, mTLS client-cert + client-CA rotation, client CA rotation with
hostname/untrusted-CA rejection still enforced), and a mutation check
(disabling reload makes all 4 fail). Live: a running secure-standalone
agent (mTLS + token), server cert replaced on disk, the same process
(never restarted) went from serial 300461AA... to 378B4E6D... for new
connections, and loadgen kept delivering 500/500 events over mTLS
before and after.
Observability (Prometheus + Grafana). The container-registry block
covers Grafana too - unlike most projects, Grafana doesn't publish its
binary releases as GitHub release assets, only via its own CDN
(dl.grafana.com), which this sandbox's egress policy denies the same
way it denies every container registry. So the actual docker-compose.yml/
k8s/ services (standard prom/prometheus and grafana/grafana-oss
images) couldn't be run here. What could be, and was, verified live
instead, entirely outside Docker/Kubernetes:
- Built
logflow's own agent/loadgen binaries and ran a real two-agent edge→collector pair as plain host processes, instrumented with the realclient_golangmetrics described above - confirmed real counters and a real, non-empty latency histogram at/metricson both agents while traffic was flowing (e.g.logflow_channel_put_total{channel="main"} 14570,logflow_batch_send_duration_seconds_bucket{le="0.001"} 50and climbing per bucket). - Downloaded Prometheus's own release binary directly from its GitHub
releases (not a container image - same technique the Kubernetes work
used for k3s) and ran it against
observability/prometheus.yml's exact scrape config, pointed at those two agents: both targets came up"health": "up"in Prometheus's own/api/v1/targets, actually scraping on the configured interval. - Ran every one of the dashboard's 6 panels' exact PromQL expressions
(copied straight out of
observability/grafana/dashboards/logflow.json, not approximations) against that live Prometheus via its HTTP API and got back real, sensible numbers - e.g.rate(logflow_events_received_total[1m])returning ~650 events/sec on both agents, and the p99 batch-latencyhistogram_quantilequery returning0.0064(6.4ms), matching what a loopback gRPC send should look like. Since a Grafana panel is just that query rendered as a chart against that same datasource, this confirms every panel will show real data the moment Grafana itself can run somewhere with registry/CDN access - which is everywhere except this one sandbox.
Alerting (Prometheus + Alertmanager). Unlike Grafana, Alertmanager
publishes real binary releases on GitHub, so this one was run exactly as
it runs in docker-compose.yml/k8s/ - not approximated. Built on the
same real agent/loadgen/Prometheus setup as above, plus a real Alertmanager
(v0.27.0) downloaded from its GitHub releases and run against the exact
observability/alertmanager.yml, with observability/alerts.yml loaded
into Prometheus via rule_files: (only for: durations were shortened,
from 1m/2m to 10s, purely so the run didn't need to sit idle for minutes -
the expressions themselves are unmodified):
- Pointed one scrape target at an address nothing was listening on: within
one
for: 10swindow,LogflowTargetDownmoved tofiringin Prometheus's real/api/v1/rules, and the alert showed up"state": "active"in Alertmanager's real/api/v2/alerts- the whole Prometheus-evaluates → Alertmanager-receives pipeline, not just the rule expression in isolation. SIGKILLed the collector process outright (not just a closed port) while the edge kept taking traffic: itscollector-1andcollector-2scrape jobs both wentdown, firing a second and third independentLogflowTargetDownalert; the edge's memory channel filled solid (500/500) with nowhere to drain to, firing bothLogflowChannelNearFullandLogflowChannelDroppedfrom reallogflow_channel_size/logflow_channel_dropped_totalvalues - at one point Alertmanager's API showed 5 simultaneous real, distinct alerts.- Confirmed the inhibit rule's scope is correct, not just present:
edge-1(itself up, just backpressured) kept itsLogflowChannelNearFull/LogflowChannelDroppedalerts visible throughout, while the inhibit rule only suppresses derived-symptom alerts on ajobthat's also the one firingLogflowTargetDown- verified byequal: ["job"]not cross-suppressing a healthy agent's alerts just because an unrelated one is down. - Restarted the collector process and confirmed the resolution path, not
just the firing path: once its
/metricsendpoint came back, itsLogflowTargetDownandLogflowChannelNearFullalerts both dropped out of Alertmanager's active-alerts list on their own (LogflowChannelDroppedcorrectly stayed active a little longer - itsincrease(...[5m]) > 0condition is still true until the drop event ages out of the 5m window). - Not forced live:
LogflowEventsDropped(same shape asLogflowChannelDropped, one condition and one metric, already proven to evaluate and fire correctly - just not the specific metric that happened to move during this run),LogflowForwardFailuresHigh(this setup's connection-refused failures happen before a batch is ever attempted, sologflow_events_forward_failed_total- which counts failed send attempts, not failed connections - never got a nonzero value to alert on; forcing it needs a target that accepts the TCP connection but then rejects/times out the RPC, not one that's simply down), andLogflowPurgedUnconsumed/LogflowBatchLatencyHigh(both require conditions - WAL retention actually expiring, or sustained >1s p95 latency - that take materially longer to construct than a demo run justifies). All four rely on the samerule_files:/expression evaluation path already proven live by the three that were forced, so this is a real gap in forced conditions, not in mechanism coverage.
CI/CD (GitHub Actions). .github/workflows/ci.yml can't actually be
run by GitHub Actions from inside this sandbox (no runner to dispatch to),
so instead every one of its steps was run here exactly as written, outside
the workflow file, to confirm each one does what the workflow claims:
go mod verify,gofmt -l(after actually fixing the handful of files that weren't formatted - see below),go vet ./...,go build ./..., andgo test -v -race ./...all pass clean against the current tree.make build+docker build -t logflow:ci .+docker compose config -qall succeed, reusing the same real Docker daemon the Docker deployment work verified against earlier.- Installed
kustomizefrom its real GitHub release binary (not a container image -ghcr.io/dl.k8s.ioare both blocked here the same way every registry and Grafana's CDN were) and rankustomize build k8s/- rendered all 25 objects with no error. - Installed
promtool/amtoolthe same way (already downloaded for the Alerting verification above) and ran the workflow's three config-check steps verbatim - this actually caught a real bug before it ever reached GitHub:promtool check configresolves arule_files:entry relative to the config file's own directory, not the caller's cwd, so the original one-linesedsubstitution pointed at a path that didn't exist. Fixed to substitute in an absolute path instead, then re-ran it to confirm the fix. - Ran the whole file through
actionlint(downloaded the same way, from its GitHub release binary) with zero findings, confirming the YAML is schema-valid GitHub Actions syntax, not just valid YAML. - Also fixed, as a prerequisite:
gofmt -lfound 7 files that weren't gofmt-clean (internal/agent/agent.go,internal/channel/{channel,file}.go,internal/config/config.go,internal/sink/{failover_group,file_sink,grpc_sink}.go)- all whitespace/alignment-only diffs (confirmed with
gofmt -dbefore applying),go build/go vet/go testall still pass after.
- all whitespace/alignment-only diffs (confirmed with
- Not verified: an actual GitHub Actions run. The commands behind every
step were proven to work on this machine; whether they behave
identically on a
ubuntu-latestrunner (different base image,sudoavailability, pre-installed tool versions) is unconfirmed until this is pushed and a real workflow run is watched.
- A single Raft group is still single-leader-bound (~794 commits/sec
measured per group, ~7x the unbatched baseline via batching - see the
"Batching" writeup and Verified behavior above). Every write within one
group still funnels through that group's one leader's
batchLoopgoroutine, which flushes strictly in order - largerbatch_size/longerbatch_lingertrade latency for more throughput, but only within that one group's ceiling. Getting past that ceiling is now implemented (see "Sharding" above):ShardedChannelpartitions a channel across multiple independent Raft groups so several leaders can be committing concurrently, on different physical nodes. What sharding itself doesn't give you, by design, not oversight: no ordering or atomicity across shards, and no automatic rebalancing - shard count and each shard's peer list are fixed at config time; splitting/merging shards or moving a key range between them isn't implemented. See "Sharding" above for the full trade-off list. - Membership changes are live now (joint consensus - see "Live
membership changes" above). With
snapshot_thresholdset, a new member catches up from a snapshot boundary plus the retained tail; without it (the default) it still replays the entire Raft log. Also: only one membership change may be in flight at a time (ErrMembershipChangeInProgress, per shard for a sharded channel - different shards' memberships can still change concurrently with each other, independently), and if the node driving a change crashes between committing Cold,new and Cnew, the cluster (or that one shard) is left in the joint phase until the same AddServer/RemoveServer call is retried against whichever node is leader now (safe - joint quorum rules are already protecting it - just incomplete until retried). - Raft snapshots carry no event data. Compaction is opt-in
(
snapshot_threshold; default 0 keeps the whole log, as before). A node brought up to date byInstallSnapshot(a new member, or a follower down longer than the retained tail) gets the log position and cluster membership but not the events in the skipped range, so its local channel lacks them; the nodes that were up hold them. That is the price of a log that doesn't grow forever, because the state machine here is "append to a local channel" whose contents can't be exported - see "Log compaction" below for when it matters. - Deliberate design decision (not a gap): zero-copy reads. Aliasing the file channel's read buffer directly into the returned
Event(rather than copying it) means that buffer's memory can't be recycled until the consumer is done with the event - the standard zero-copy tradeoff of extending an allocation's lifetime in exchange for skipping a copy. The write-side scratch buffer has no such issue since it's only ever used transiently within a singlePut()call. - Deliberate design decision (not a gap): whole-segment retention. Retention purging deletes whole segments, not individual entries - there's no byte-level compaction that rewrites a partially-consumed segment to reclaim just the consumed prefix while keeping the rest. This is a deliberate simplicity/safety trade-off: rewriting a segment while tokens still in the ready queue reference byte offsets into it would require rewriting those offsets too, and getting that wrong risks silent corruption. Real systems facing the same trade-off (Kafka's retention, notably) also delete whole segments rather than byte-rewriting for time/size-based retention.
observability/alertmanager.yml's one receiver points at a placeholderurl:- alerts fire and route correctly (see Verified behavior above) but nothing real gets paged until that's swapped for an actual Slack/PagerDuty/OpsGenie webhook.- Alert thresholds in
observability/alerts.yml(80% channel fill, >1s p95 batch latency, 1m/2mfor:windows) are sized for the demo topology's traffic, not tuned against any real production load profile. - Not verifiable from the authoring sandbox:
.github/workflows/ci.ymlvalidates manifests by rendering them (kustomize build) and validates configs by checking their syntax (promtool/amtool), but doesn't deploy anything or run the live, forced-failure scenarios documented above - those still need a human (or a separate, heavier integration-test job) running them against a real cluster/compose stack. - The
manifestsjob deliberately stops atkustomize build- it doesn't run a Kubernetes schema validator (kubeconform,kubectl --dry-run=client) because this sandbox's egress policy blocks bothghcr.ioanddl.k8s.io, so neither tool could be fetched and test-run here before being committed (see CI/CD section above). Worth adding on a runner that can reach them.