Clean, flexible Kubernetes resource monitoring - Use as a Go library or deploy as an operator.
📚 Library-First Design: Pure mechanisms for Kubernetes resource monitoring
🎯 Two Deployment Modes: Embed in your Go applications or deploy as a standalone operator
🔧 Clean Architecture: Provides tools and mechanisms, not policies
🚀 Production Ready: Comprehensive testing, secure RBAC, Prometheus metrics
go get github.com/T0MASD/faropackage main
import (
"log"
faro "github.com/T0MASD/faro/pkg"
)
func main() {
config, _ := faro.LoadConfig()
client, _ := faro.NewKubernetesClient()
logger, _ := faro.NewLogger(config)
controller := faro.NewController(client, logger, config)
controller.AddEventHandler(&MyHandler{})
controller.Start()
}
type MyHandler struct{}
func (h *MyHandler) OnMatched(event faro.MatchedEvent) error {
log.Printf("Event: %s %s/%s", event.EventType, event.Object.GetNamespace(), event.Object.GetName())
return nil
}# Deploy operator
kubectl apply -k deploy/operator/
# Verify deployment
kubectl get pods -n faro-system
# View metrics
kubectl port-forward -n faro-system svc/faro-operator-metrics 8080:8080
curl http://localhost:8080/metricsContainer Images:
ghcr.io/t0masd/faro-operator:latest- Latest stable releaseghcr.io/t0masd/faro-operator:v1.0.0- Specific version
| Feature | Faro | kubectl watch | Custom Controllers |
|---|---|---|---|
| Server-side Filtering | ✅ Labels + field selectors | ❌ Basic only | |
| JSON Event Export | ✅ Structured output | ❌ Text only | |
| Readiness Callbacks | ✅ Built-in | ❌ No signal | |
| Graceful Shutdown | ✅ Clean cleanup | ❌ Kill only | |
| Dynamic Informers | ✅ Runtime creation | ❌ Static | ✅ Full control |
| Prometheus Metrics | ✅ Built-in | ❌ None | |
| Operator Deployment | ✅ Ready-to-use | ❌ N/A | |
| Library Integration | ✅ Import and use | ❌ CLI only | ✅ Full control |
- 🎯 Dual Deployment: Use as Go library or Kubernetes operator
- 🔍 Server-side Filtering: Efficient label selectors and field matching
- 📊 Prometheus Metrics: Built-in observability (events, informers, health)
- 📝 JSON Event Export: Structured event output for integration
- 🔄 Dynamic Resource Discovery: Add/remove resources at runtime
- 🛡️ Secure RBAC: Read-only access, no secrets, principle of least privilege
- ⚡ Graceful Lifecycle: Clean startup, readiness signals, shutdown handling
- 🧪 Comprehensive Testing: Unit, E2E, integration, and operator deployment tests
When deployed as an operator, Faro provides:
- In-cluster Authentication: Automatic ServiceAccount token handling
- Health & Readiness Probes: Kubernetes-native health checks via
/metrics - Resource Limits: Production-ready CPU/memory constraints
- Security Hardening: Minimal capabilities, non-root user, read-only filesystem
- ConfigMap-driven: Easy configuration updates without redeployment
- Event Persistence: JSON event files stored in persistent volumes
Faro Core Provides (Mechanisms):
- ✅ Informer management and lifecycle
- ✅ Event streaming with work queues
- ✅ Server-side filtering (labels, fields, namespaces)
- ✅ JSON export and structured logging
- ✅ Prometheus metrics and health endpoints
- ✅ Graceful startup, readiness, and shutdown
Users Implement (Policies):
- 🔧 Business logic and event processing
- 🔧 Complex filtering and correlation
- 🔧 Workload detection and CRD discovery
- 🔧 External integrations and workflows
- 🔧 Custom actions and automation
┌─────────────────────────────────────────────────────────────┐
│ Faro Controller │
├─────────────────┬──────────────┬──────────┬─────────────────┤
│ Config Loader │ K8s Client │ Logger │ Metrics Server │
├─────────────────┴──────────────┴──────────┴─────────────────┤
│ Multi-layered Informer Manager │
│ ┌────────────────┐ ┌───────────────┐ ┌────────────────┐ │
│ │ Config-driven │ │ Dynamic │ │ Namespace │ │
│ │ Informers │ │ Informers │ │ Scoped │ │
│ └────────────────┘ └───────────────┘ └────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ Event Processing │
│ ┌────────────────┐ ┌───────────────┐ ┌────────────────┐ │
│ │ Work Queues │ │Event Handlers │ │ JSON Export │ │
│ └────────────────┘ └───────────────┘ └────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Faro supports two configuration formats: namespace-centric and resource-centric.
Monitor multiple resources within specific namespaces:
output_dir: "/var/faro/events"
log_level: "info"
auto_shutdown_sec: 0 # Run indefinitely (operator mode)
json_export: true
metrics:
enabled: true
port: 8080
path: "/metrics"
bind_addr: "0.0.0.0"
namespaces:
- name_selector: "production"
resources:
"v1/pods":
label_selector: "app=nginx"
"v1/services": {}
"apps/v1/deployments": {}
"batch/v1/jobs": {}Monitor specific resources across multiple namespaces:
output_dir: "./logs"
log_level: "info"
json_export: true
metrics:
enabled: true
port: 8080
resources:
- gvr: "v1/configmaps"
namespace_names: ["default", "kube-system"]
label_selector: "app=web"
- gvr: "batch/v1/cronjobs"
namespace_names: ["production"]
field_selector: "metadata.name=backup-job"
- gvr: "v1/events"
namespace_names: [""] # "" is every namespace: one watch, one cache
field_selector: "type=Warning"A namespaced type that names no namespaces is not watched at all (Faro warns).
namespace_names: [""] means the whole cluster, with one informer:
- for a namespaced type, every namespace (Faro says so once at start);
- for a cluster-scoped type, all of it - which is also how to ask for a CRD that is not installed yet, whose scope Faro cannot know until it is.
List(gvr, "") reads that cache, and Get finds an object in it by
namespace/name (or name).
By default an event carries identity and metadata: eventTime, eventType,
gvr, namespace, name, uid, resourceVersion, and labels/annotations
when present.
eventTime is when this collector saw the change; resourceVersion is the
version of the change itself. Two collectors watching one stream stamp one
change with two different eventTimes and the same resourceVersion, which is
what lets a capture be de-duplicated across collectors rather than guessed at
from timestamps.
That answers what changed, not why - a crash-looping pod produces Event
objects whose reason and message explain it, and neither is metadata.
fields names object paths to copy into the emitted event, per GVR:
resources:
- gvr: "v1/events"
fields: ["reason", "message", "type", "involvedObject.name"]
- gvr: "v1/pods"
fields: ["status.phase", "status.containerStatuses.restartCount"]{"eventTime":"...","eventType":"ADDED","gvr":"v1/events","name":"pod.18dd44",
"fields":{"reason":"BackOff","involvedObject":{"kind":"Pod","name":"antrea-agent-86jgg"}}}The config path is dotted; the output is nested. involvedObject.name is
the natural way to name a path, but a flat key would contain a dot - and a
dot is the path separator in every tool that reads the capture afterwards, so
.fields.involvedObject.name in jq and
json_extract(detail,'$.involvedObject.name') in SQLite would both return
null, silently and with no error. Paths sharing a prefix merge into one object.
- a path crossing a list applies to every element, so
status.containerStatuses.restartCountyields one value per container - an absent path is omitted rather than emitted null, so one field list works across clusters where a type may not carry it
- with nothing configured the event is byte-identical to before this existed
Per GVR, deliberately. The reason a capture-everything config is affordable
is that each record is small - measured at 393 bytes across ~5000 events.
Adding reason+message to Events alone costs about 3.5%; a global
"emit everything" would quietly remove that property.
Secrets are refused in code. data and stringData on v1/secrets are
never extracted, whatever the config asks for. A Secret's data is base64, not
encryption, and a capture file is written to disk and copied around for
analysis - a typo must not be able to turn this into a credential leak.
| Field | Type | Description |
|---|---|---|
output_dir |
string | Directory for logs and JSON exports |
fields |
[]string | Per-GVR object paths copied into the emitted event (see above) |
label_selector |
string | Server-side label selector, per resource |
field_selector |
string | Server-side field selector, per resource (e.g. metadata.name=backup-job) |
name_selector |
string | On a resource: exact object name, sent as metadata.name= |
Unknown keys are an error. Entries for one type and namespace share one informer, so they must ask for the same label and field selectors; entries that only name different objects watch all names, with a warning.
| log_level | string | debug, info, warning, error, fatal |
| auto_shutdown_sec | int | Auto-shutdown after N seconds (0 = disabled) |
| json_export | bool | Enable structured JSON event export |
| metrics.enabled | bool | Enable Prometheus metrics server |
| metrics.port | int | Metrics server port (default: 8080) |
| workers | int | Workers delivering events (default: 3). Never two at once for one object |
| resync_sec | int | Hand every watched object to the handlers again as RESYNC on this period (0 = never). Not written to the capture |
| sqlite_path | string | SQLite capture: auto (default for the binary), a path, or off. Unset in a Config built in code means none |
package main
import (
"log"
faro "github.com/T0MASD/faro/pkg"
)
func main() {
// Load configuration
config, err := faro.LoadConfig()
if err != nil {
log.Fatal(err)
}
// Create Kubernetes client (auto-detects in-cluster or kubeconfig)
client, err := faro.NewKubernetesClient()
if err != nil {
log.Fatal(err)
}
// Create logger
logger, err := faro.NewLogger(config)
if err != nil {
log.Fatal(err)
}
// Create controller
controller := faro.NewController(client, logger, config)
// Register event handler
controller.AddEventHandler(&MyEventHandler{})
// Set readiness callback
controller.SetReadyCallback(func() {
log.Println("Faro is ready!")
})
// Start controller
controller.Start()
}
type MyEventHandler struct{}
func (h *MyEventHandler) OnMatched(event faro.MatchedEvent) error {
log.Printf("Event: %s %s %s/%s",
event.EventType,
event.GVR,
event.Object.GetNamespace(),
event.Object.GetName())
return nil
}A Config built in code writes no capture: a program that embeds faro to react
to changes gets its handlers and nothing on disk, even with OutputDir set for
its logs. Ask for one explicitly:
config := &faro.Config{
OutputDir: "./logs",
SqlitePath: faro.SqliteAuto, // <output_dir>/logs/capture.db, or a path
JsonExport: true, // events-*.json alongside it
}The faro binary (and LoadConfig) keeps today's default, a SQLite capture
under output_dir; sqlite_path: "off" turns it off.
type DynamicDiscovery struct {
controller *faro.Controller
}
func (d *DynamicDiscovery) OnMatched(event faro.MatchedEvent) error {
// Watch for new CRDs and dynamically add them
if event.GVR == "apiextensions.k8s.io/v1/customresourcedefinitions" {
if event.EventType == "ADDED" {
gvr := extractGVRFromCRD(event.Object)
d.controller.AddResources([]faro.ResourceConfig{{
GVR: gvr,
NamespaceNames: []string{"default"},
}})
d.controller.StartInformers()
}
}
return nil
}// Controller creation
controller := faro.NewController(client, logger, config)
// Event handlers (implement your business logic). Called on a worker, one
// event at a time per object, in the order observed; different objects run in
// parallel (`workers`, default 3). Return an error to be handed the same
// event again, with backoff - up to 5 retries, then it is logged and skipped.
// A slow handler slows its worker: that is backpressure, not loss.
controller.AddEventHandler(handler EventHandler)
// Current state, from the informer cache (no API calls). Copies; "" namespace
// for a cluster-scoped type. ErrNotSynced until the first list completes.
objs, err := controller.List("v1/configmaps", "default")
obj, err := controller.Get("v1/configmaps", "default/my-config") // nil, nil if absent
// Informer status handlers (hear whether a watch listed, and what it holds)
// An empty result, a refusal, and a type the endpoint does not serve all
// deliver no events, so OnMatched cannot tell them apart. This can.
controller.AddInformerStatusHandler(handler InformerStatusHandler)
// JSON middleware (modify objects before export)
controller.AddJSONMiddleware(middleware JSONMiddleware)
// Readiness callback (initialization complete)
controller.SetReadyCallback(func() {
fmt.Println("Ready!")
})
// Check readiness
if controller.IsReady() {
// All informers synced
}
// Dynamic resource management
controller.AddResources([]faro.ResourceConfig{...})
controller.StartInformers()
// Get active informer counts
configCount, dynamicCount := controller.GetActiveInformers()
// Lifecycle management
controller.Start() // Returns once informers are started; events arrive on handlers from then on
// ...block on your own signal or context here...
controller.Stop() // Graceful shutdown- Kubernetes cluster (1.24+)
kubectlconfigured
# Deploy operator with default configuration
kubectl apply -k deploy/operator/
# Verify deployment
kubectl get pods -n faro-system
kubectl logs -n faro-system -l app.kubernetes.io/name=faro-operator -f
# Check metrics
kubectl port-forward -n faro-system svc/faro-operator-metrics 8080:8080
curl http://localhost:8080/metrics# Deploy operator
bash scripts/deploy-operator.sh
# Cleanup
bash scripts/cleanup-operator.shModify the ConfigMap to customize monitoring:
kubectl edit configmap -n faro-system faro-operator-configExample configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: faro-operator-config
namespace: faro-system
data:
config.yaml: |
output_dir: "/var/faro/events"
log_level: "info"
auto_shutdown_sec: 0
json_export: true
metrics:
enabled: true
port: 8080
path: "/metrics"
namespaces:
- name_selector: "production"
resources:
"v1/pods": {}
"v1/services": {}
"apps/v1/deployments": {}# Get operator pod name
POD=$(kubectl get pod -n faro-system -l app.kubernetes.io/name=faro-operator -o jsonpath='{.items[0].metadata.name}')
# View captured events
kubectl exec -n faro-system $POD -- ls -lh /var/faro/events/
kubectl exec -n faro-system $POD -- cat /var/faro/events/events-*.jsonThe operator exposes Prometheus metrics:
# Port-forward metrics endpoint
kubectl port-forward -n faro-system svc/faro-operator-metrics 8080:8080
# View metrics
curl http://localhost:8080/metricsKey Metrics:
faro_events_total- Total events processed by GVR and typefaro_informer_health- Informer health statusfaro_gvr_per_informer- Resources tracked per informerfaro_informer_last_event_timestamp- Last event timestamp per informer
The operator is deployed with security best practices:
- Read-only RBAC: Can only
get,list,watchresources - No secrets access: Explicitly denied
- Resource limits: CPU and memory constraints
- Security context: Non-root user, dropped capabilities
- Health probes: Kubernetes-native health checks
View RBAC permissions:
# Check what operator can do
kubectl auth can-i list pods --as=system:serviceaccount:faro-system:faro-operator
kubectl auth can-i get secrets --as=system:serviceaccount:faro-system:faro-operator # Should be "no"Faro includes comprehensive test coverage:
make test-unitTests configuration parsing, logger functionality, and core logic.
make test-e2eTests core library functionality with real Kubernetes clusters.
make test-integrationTests library user implementations (dynamic discovery, workload monitoring).
make test-operatorEnd-to-end validation of operator deployment:
- Image building
- RBAC configuration
- Metrics endpoint
- Event capture
- Security restrictions
make test # Runs unit + e2e + integration testsFaro includes scripts for local Kubernetes testing:
# Start local kinc cluster
podman run -d --name kinc-cluster \
--hostname kinc-control-plane \
--cap-add=SYS_ADMIN \
-p 127.0.0.1:6443:6443/tcp \
ghcr.io/t0masd/kinc:latest
# Extract kubeconfig
mkdir -p ~/.kube
podman cp kinc-cluster:/etc/kubernetes/admin.conf ~/.kube/config
sed -i 's|server: https://.*:6443|server: https://127.0.0.1:6443|g' ~/.kube/config
# Run tests
make testFaro uses GitHub Actions for continuous integration:
On every push and pull request:
- ✅ Unit tests
- ✅ E2E tests (parallel job with kinc)
- ✅ Integration tests (parallel job with kinc)
- ✅ Operator deployment tests (parallel job with kinc)
- ✅ Library import validation
- ✅ Test artifact uploads
On version tags (v*):
- ✅ Full test suite validation
- ✅ GoReleaser library release
- ✅ Container image build and push to
ghcr.io/t0masd/faro-operator - ✅ Multi-tag support (
latest,v1.0.0,v1.0,v1)
View CI Status: https://github.com/T0MASD/faro/actions
When json_export: true, events are exported in structured format:
{
"timestamp": "2025-11-10T14:33:02Z",
"eventTime": "2025-11-10T16:04:11.882471903Z",
"eventType": "ADDED",
"gvr": "v1/pods",
"namespace": "default",
"name": "nginx-abc123",
"uid": "12345678-1234-1234-1234-123456789012",
"resourceVersion": "12345",
"labels": {
"app": "nginx",
"version": "1.0"
},
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "..."
}
}timestamp and eventTime answer different questions. timestamp is the
object's creationTimestamp - its age - so every update to one object carries
the same value and a capture cannot be ordered by it. eventTime is when Faro
observed the event, and is what a timeline is built from. timestamp is omitted
when it is unknown, which is the case for a DELETED event reconstructed after
the object is gone.
Events are written to:
- Library mode:
${output_dir}/events-YYYYMMDD-HHMMSS.json - Operator mode:
/var/faro/events/events-YYYYMMDD-HHMMSS.json
Check the examples/ directory for real-world usage:
examples/library-usage/- Basic library integration, event and informer-status handlersexamples/workload-monitor/- Dynamic workload detectionexamples/worker-dispatcher/- Event processing and actions
Each is its own package, so it builds and runs directly:
go run ./examples/library-usage- Architecture Overview - Design principles and patterns
- Component Reference - Detailed component documentation
- Controller - Informer management
- Client - Kubernetes client
- Config - Configuration formats
- Logger - Structured logging
- Metrics - Prometheus metrics
- Examples - Real-world implementations
# Build library binary
make build
# Build with version info
make build-dev
# Build operator image
make operator-image
# Load image into local cluster
make operator-image-loadfaro/
├── pkg/ # Core library code
│ ├── client.go # Kubernetes client
│ ├── config.go # Configuration loading
│ ├── controller.go # Main controller
│ ├── logger.go # Structured logging
│ └── metrics.go # Prometheus metrics
├── main.go # CLI entrypoint
├── Dockerfile # Operator container image
├── deploy/operator/ # Kubernetes manifests
│ ├── namespace.yaml
│ ├── serviceaccount.yaml
│ ├── clusterrole.yaml
│ ├── clusterrolebinding.yaml
│ ├── configmap.yaml
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
├── scripts/ # Deployment, test and release scripts
│ ├── deploy-operator.sh
│ ├── cleanup-operator.sh
│ ├── test-operator.sh
│ └── tag-version.sh # make tag-patch/minor/major
├── tests/ # Test suites
│ ├── unit/
│ ├── e2e/
│ ├── integration/
│ └── operator-ci/
├── examples/ # Usage examples, one package each
└── docs/ # Documentation
make help # Show all available targets
make build # Build faro binary
make test # Run all tests
make test-unit # Run unit tests
make test-e2e # Run E2E tests
make test-integration # Run integration tests
make test-operator # Run operator deployment tests
make clean # Clean build artifacts
make operator-image # Build operator container imageContributions are welcome! Please feel free to submit issues or pull requests.
- Ensure you have a Kubernetes cluster available (kinc recommended for local development)
- Run
make testto validate your changes - Run
make test-operatorto validate operator deployment
This project is released into the public domain under the Unlicense.
- Library provides mechanisms - Informers, events, JSON export, metrics
- Users implement policies - Business logic, filtering, workflows
- No fallbacks or defaults - Errors are surfaced, not hidden
- Clean separation - Core library vs. application concerns
- Dual deployment - Use as library or operator, your choice
Faro gives you the tools. You build the solutions.
Made with ❤️ for the Kubernetes community
