Skip to content
T0MASDPublic

About

Kubernetes resource observation tool with dynamic discovery and configuration-driven informer management.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

banner

Go Reference License Go Version

Faro - Kubernetes Resource Monitoring

Clean, flexible Kubernetes resource monitoring - Use as a Go library or deploy as an operator.

📚 Library-First Design: Pure mechanisms for Kubernetes resource monitoring
🎯 Two Deployment Modes: Embed in your Go applications or deploy as a standalone operator
🔧 Clean Architecture: Provides tools and mechanisms, not policies
🚀 Production Ready: Comprehensive testing, secure RBAC, Prometheus metrics


Quick Start

As a Go Library

go get github.com/T0MASD/faro
package main

import (
    "log"
    faro "github.com/T0MASD/faro/pkg"
)

func main() {
    config, _ := faro.LoadConfig()
    client, _ := faro.NewKubernetesClient()
    logger, _ := faro.NewLogger(config)
    controller := faro.NewController(client, logger, config)
    
    controller.AddEventHandler(&MyHandler{})
    controller.Start()
}

type MyHandler struct{}

func (h *MyHandler) OnMatched(event faro.MatchedEvent) error {
    log.Printf("Event: %s %s/%s", event.EventType, event.Object.GetNamespace(), event.Object.GetName())
    return nil
}

As a Kubernetes Operator

# Deploy operator
kubectl apply -k deploy/operator/

# Verify deployment
kubectl get pods -n faro-system

# View metrics
kubectl port-forward -n faro-system svc/faro-operator-metrics 8080:8080
curl http://localhost:8080/metrics

Container Images:

  • ghcr.io/t0masd/faro-operator:latest - Latest stable release
  • ghcr.io/t0masd/faro-operator:v1.0.0 - Specific version

Why Faro?

Feature Faro kubectl watch Custom Controllers
Server-side Filtering ✅ Labels + field selectors ❌ Basic only ⚠️ Manual
JSON Event Export ✅ Structured output ❌ Text only ⚠️ Custom
Readiness Callbacks ✅ Built-in ❌ No signal ⚠️ Manual
Graceful Shutdown ✅ Clean cleanup ❌ Kill only ⚠️ Manual
Dynamic Informers ✅ Runtime creation ❌ Static ✅ Full control
Prometheus Metrics ✅ Built-in ❌ None ⚠️ Manual
Operator Deployment ✅ Ready-to-use ❌ N/A ⚠️ Build yourself
Library Integration ✅ Import and use ❌ CLI only ✅ Full control

Features

Core Capabilities

  • 🎯 Dual Deployment: Use as Go library or Kubernetes operator
  • 🔍 Server-side Filtering: Efficient label selectors and field matching
  • 📊 Prometheus Metrics: Built-in observability (events, informers, health)
  • 📝 JSON Event Export: Structured event output for integration
  • 🔄 Dynamic Resource Discovery: Add/remove resources at runtime
  • 🛡️ Secure RBAC: Read-only access, no secrets, principle of least privilege
  • ⚡ Graceful Lifecycle: Clean startup, readiness signals, shutdown handling
  • 🧪 Comprehensive Testing: Unit, E2E, integration, and operator deployment tests

Operator Features

When deployed as an operator, Faro provides:

  • In-cluster Authentication: Automatic ServiceAccount token handling
  • Health & Readiness Probes: Kubernetes-native health checks via /metrics
  • Resource Limits: Production-ready CPU/memory constraints
  • Security Hardening: Minimal capabilities, non-root user, read-only filesystem
  • ConfigMap-driven: Easy configuration updates without redeployment
  • Event Persistence: JSON event files stored in persistent volumes

Architecture

Philosophy: Mechanisms, Not Policies

Faro Core Provides (Mechanisms):

  • ✅ Informer management and lifecycle
  • ✅ Event streaming with work queues
  • ✅ Server-side filtering (labels, fields, namespaces)
  • ✅ JSON export and structured logging
  • ✅ Prometheus metrics and health endpoints
  • ✅ Graceful startup, readiness, and shutdown

Users Implement (Policies):

  • 🔧 Business logic and event processing
  • 🔧 Complex filtering and correlation
  • 🔧 Workload detection and CRD discovery
  • 🔧 External integrations and workflows
  • 🔧 Custom actions and automation

Component Architecture

┌─────────────────────────────────────────────────────────────┐
│                       Faro Controller                       │
├─────────────────┬──────────────┬──────────┬─────────────────┤
│  Config Loader  │  K8s Client  │  Logger  │ Metrics Server  │
├─────────────────┴──────────────┴──────────┴─────────────────┤
│               Multi-layered Informer Manager                │
│  ┌────────────────┐  ┌───────────────┐  ┌────────────────┐  │
│  │ Config-driven  │  │    Dynamic    │  │   Namespace    │  │
│  │   Informers    │  │   Informers   │  │     Scoped     │  │
│  └────────────────┘  └───────────────┘  └────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│                      Event Processing                       │
│  ┌────────────────┐  ┌───────────────┐  ┌────────────────┐  │
│  │  Work Queues   │  │Event Handlers │  │  JSON Export   │  │
│  └────────────────┘  └───────────────┘  └────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Configuration

Faro supports two configuration formats: namespace-centric and resource-centric.

Namespace-Centric Configuration

Monitor multiple resources within specific namespaces:

output_dir: "/var/faro/events"
log_level: "info"
auto_shutdown_sec: 0  # Run indefinitely (operator mode)
json_export: true

metrics:
  enabled: true
  port: 8080
  path: "/metrics"
  bind_addr: "0.0.0.0"

namespaces:
  - name_selector: "production"
    resources:
      "v1/pods":
        label_selector: "app=nginx"
      "v1/services": {}
      "apps/v1/deployments": {}
      "batch/v1/jobs": {}

Resource-Centric Configuration

Monitor specific resources across multiple namespaces:

output_dir: "./logs"
log_level: "info"
json_export: true

metrics:
  enabled: true
  port: 8080

resources:
  - gvr: "v1/configmaps"
    namespace_names: ["default", "kube-system"]
    label_selector: "app=web"
  - gvr: "batch/v1/cronjobs"
    namespace_names: ["production"]
    field_selector: "metadata.name=backup-job"
  - gvr: "v1/events"
    namespace_names: [""]          # "" is every namespace: one watch, one cache
    field_selector: "type=Warning"

A namespaced type that names no namespaces is not watched at all (Faro warns). namespace_names: [""] means the whole cluster, with one informer:

  • for a namespaced type, every namespace (Faro says so once at start);
  • for a cluster-scoped type, all of it - which is also how to ask for a CRD that is not installed yet, whose scope Faro cannot know until it is.

List(gvr, "") reads that cache, and Get finds an object in it by namespace/name (or name).

Emitting object fields

By default an event carries identity and metadata: eventTime, eventType, gvr, namespace, name, uid, resourceVersion, and labels/annotations when present.

eventTime is when this collector saw the change; resourceVersion is the version of the change itself. Two collectors watching one stream stamp one change with two different eventTimes and the same resourceVersion, which is what lets a capture be de-duplicated across collectors rather than guessed at from timestamps. That answers what changed, not why - a crash-looping pod produces Event objects whose reason and message explain it, and neither is metadata.

fields names object paths to copy into the emitted event, per GVR:

resources:
  - gvr: "v1/events"
    fields: ["reason", "message", "type", "involvedObject.name"]
  - gvr: "v1/pods"
    fields: ["status.phase", "status.containerStatuses.restartCount"]
{"eventTime":"...","eventType":"ADDED","gvr":"v1/events","name":"pod.18dd44",
 "fields":{"reason":"BackOff","involvedObject":{"kind":"Pod","name":"antrea-agent-86jgg"}}}

The config path is dotted; the output is nested. involvedObject.name is the natural way to name a path, but a flat key would contain a dot - and a dot is the path separator in every tool that reads the capture afterwards, so .fields.involvedObject.name in jq and json_extract(detail,'$.involvedObject.name') in SQLite would both return null, silently and with no error. Paths sharing a prefix merge into one object.

  • a path crossing a list applies to every element, so status.containerStatuses.restartCount yields one value per container
  • an absent path is omitted rather than emitted null, so one field list works across clusters where a type may not carry it
  • with nothing configured the event is byte-identical to before this existed

Per GVR, deliberately. The reason a capture-everything config is affordable is that each record is small - measured at 393 bytes across ~5000 events. Adding reason+message to Events alone costs about 3.5%; a global "emit everything" would quietly remove that property.

Secrets are refused in code. data and stringData on v1/secrets are never extracted, whatever the config asks for. A Secret's data is base64, not encryption, and a capture file is written to disk and copied around for analysis - a typo must not be able to turn this into a credential leak.

Configuration Options

Field Type Description
output_dir string Directory for logs and JSON exports
fields []string Per-GVR object paths copied into the emitted event (see above)
label_selector string Server-side label selector, per resource
field_selector string Server-side field selector, per resource (e.g. metadata.name=backup-job)
name_selector string On a resource: exact object name, sent as metadata.name=

Unknown keys are an error. Entries for one type and namespace share one informer, so they must ask for the same label and field selectors; entries that only name different objects watch all names, with a warning. | log_level | string | debug, info, warning, error, fatal | | auto_shutdown_sec | int | Auto-shutdown after N seconds (0 = disabled) | | json_export | bool | Enable structured JSON event export | | metrics.enabled | bool | Enable Prometheus metrics server | | metrics.port | int | Metrics server port (default: 8080) | | workers | int | Workers delivering events (default: 3). Never two at once for one object | | resync_sec | int | Hand every watched object to the handlers again as RESYNC on this period (0 = never). Not written to the capture | | sqlite_path | string | SQLite capture: auto (default for the binary), a path, or off. Unset in a Config built in code means none |


Library Usage

Basic Integration

package main

import (
    "log"
    faro "github.com/T0MASD/faro/pkg"
)

func main() {
    // Load configuration
    config, err := faro.LoadConfig()
    if err != nil {
        log.Fatal(err)
    }

    // Create Kubernetes client (auto-detects in-cluster or kubeconfig)
    client, err := faro.NewKubernetesClient()
    if err != nil {
        log.Fatal(err)
    }

    // Create logger
    logger, err := faro.NewLogger(config)
    if err != nil {
        log.Fatal(err)
    }

    // Create controller
    controller := faro.NewController(client, logger, config)

    // Register event handler
    controller.AddEventHandler(&MyEventHandler{})

    // Set readiness callback
    controller.SetReadyCallback(func() {
        log.Println("Faro is ready!")
    })

    // Start controller
    controller.Start()
}

type MyEventHandler struct{}

func (h *MyEventHandler) OnMatched(event faro.MatchedEvent) error {
    log.Printf("Event: %s %s %s/%s",
        event.EventType,
        event.GVR,
        event.Object.GetNamespace(),
        event.Object.GetName())
    return nil
}

Capture is opt-in for a library

A Config built in code writes no capture: a program that embeds faro to react to changes gets its handlers and nothing on disk, even with OutputDir set for its logs. Ask for one explicitly:

config := &faro.Config{
    OutputDir:  "./logs",
    SqlitePath: faro.SqliteAuto, // <output_dir>/logs/capture.db, or a path
    JsonExport: true,            // events-*.json alongside it
}

The faro binary (and LoadConfig) keeps today's default, a SQLite capture under output_dir; sqlite_path: "off" turns it off.

Advanced Usage: Dynamic Resource Discovery

type DynamicDiscovery struct {
    controller *faro.Controller
}

func (d *DynamicDiscovery) OnMatched(event faro.MatchedEvent) error {
    // Watch for new CRDs and dynamically add them
    if event.GVR == "apiextensions.k8s.io/v1/customresourcedefinitions" {
        if event.EventType == "ADDED" {
            gvr := extractGVRFromCRD(event.Object)
            d.controller.AddResources([]faro.ResourceConfig{{
                GVR: gvr,
                NamespaceNames: []string{"default"},
            }})
            d.controller.StartInformers()
        }
    }
    return nil
}

API Reference

// Controller creation
controller := faro.NewController(client, logger, config)

// Event handlers (implement your business logic). Called on a worker, one
// event at a time per object, in the order observed; different objects run in
// parallel (`workers`, default 3). Return an error to be handed the same
// event again, with backoff - up to 5 retries, then it is logged and skipped.
// A slow handler slows its worker: that is backpressure, not loss.
controller.AddEventHandler(handler EventHandler)

// Current state, from the informer cache (no API calls). Copies; "" namespace
// for a cluster-scoped type. ErrNotSynced until the first list completes.
objs, err := controller.List("v1/configmaps", "default")
obj, err := controller.Get("v1/configmaps", "default/my-config") // nil, nil if absent

// Informer status handlers (hear whether a watch listed, and what it holds)
// An empty result, a refusal, and a type the endpoint does not serve all
// deliver no events, so OnMatched cannot tell them apart. This can.
controller.AddInformerStatusHandler(handler InformerStatusHandler)

// JSON middleware (modify objects before export)
controller.AddJSONMiddleware(middleware JSONMiddleware)

// Readiness callback (initialization complete)
controller.SetReadyCallback(func() {
    fmt.Println("Ready!")
})

// Check readiness
if controller.IsReady() {
    // All informers synced
}

// Dynamic resource management
controller.AddResources([]faro.ResourceConfig{...})
controller.StartInformers()

// Get active informer counts
configCount, dynamicCount := controller.GetActiveInformers()

// Lifecycle management
controller.Start()  // Returns once informers are started; events arrive on handlers from then on
// ...block on your own signal or context here...
controller.Stop()   // Graceful shutdown

Operator Deployment

Prerequisites

  • Kubernetes cluster (1.24+)
  • kubectl configured

Installation

Option 1: Using Kustomize (Recommended)

# Deploy operator with default configuration
kubectl apply -k deploy/operator/

# Verify deployment
kubectl get pods -n faro-system
kubectl logs -n faro-system -l app.kubernetes.io/name=faro-operator -f

# Check metrics
kubectl port-forward -n faro-system svc/faro-operator-metrics 8080:8080
curl http://localhost:8080/metrics

Option 2: Using Scripts

# Deploy operator
bash scripts/deploy-operator.sh

# Cleanup
bash scripts/cleanup-operator.sh

Configuration

Modify the ConfigMap to customize monitoring:

kubectl edit configmap -n faro-system faro-operator-config

Example configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: faro-operator-config
  namespace: faro-system
data:
  config.yaml: |
    output_dir: "/var/faro/events"
    log_level: "info"
    auto_shutdown_sec: 0
    json_export: true
    
    metrics:
      enabled: true
      port: 8080
      path: "/metrics"
    
    namespaces:
      - name_selector: "production"
        resources:
          "v1/pods": {}
          "v1/services": {}
          "apps/v1/deployments": {}

Accessing Events

# Get operator pod name
POD=$(kubectl get pod -n faro-system -l app.kubernetes.io/name=faro-operator -o jsonpath='{.items[0].metadata.name}')

# View captured events
kubectl exec -n faro-system $POD -- ls -lh /var/faro/events/
kubectl exec -n faro-system $POD -- cat /var/faro/events/events-*.json

Monitoring

The operator exposes Prometheus metrics:

# Port-forward metrics endpoint
kubectl port-forward -n faro-system svc/faro-operator-metrics 8080:8080

# View metrics
curl http://localhost:8080/metrics

Key Metrics:

  • faro_events_total - Total events processed by GVR and type
  • faro_informer_health - Informer health status
  • faro_gvr_per_informer - Resources tracked per informer
  • faro_informer_last_event_timestamp - Last event timestamp per informer

Security

The operator is deployed with security best practices:

  • Read-only RBAC: Can only get, list, watch resources
  • No secrets access: Explicitly denied
  • Resource limits: CPU and memory constraints
  • Security context: Non-root user, dropped capabilities
  • Health probes: Kubernetes-native health checks

View RBAC permissions:

# Check what operator can do
kubectl auth can-i list pods --as=system:serviceaccount:faro-system:faro-operator
kubectl auth can-i get secrets --as=system:serviceaccount:faro-system:faro-operator  # Should be "no"

Testing

Faro includes comprehensive test coverage:

Unit Tests (No Kubernetes Required)

make test-unit

Tests configuration parsing, logger functionality, and core logic.

E2E Tests (Requires Kubernetes)

make test-e2e

Tests core library functionality with real Kubernetes clusters.

Integration Tests (Requires Kubernetes)

make test-integration

Tests library user implementations (dynamic discovery, workload monitoring).

Operator Tests (Requires Kubernetes)

make test-operator

End-to-end validation of operator deployment:

  • Image building
  • RBAC configuration
  • Metrics endpoint
  • Event capture
  • Security restrictions

All Tests

make test  # Runs unit + e2e + integration tests

Local Development with kinc

Faro includes scripts for local Kubernetes testing:

# Start local kinc cluster
podman run -d --name kinc-cluster \
  --hostname kinc-control-plane \
  --cap-add=SYS_ADMIN \
  -p 127.0.0.1:6443:6443/tcp \
  ghcr.io/t0masd/kinc:latest

# Extract kubeconfig
mkdir -p ~/.kube
podman cp kinc-cluster:/etc/kubernetes/admin.conf ~/.kube/config
sed -i 's|server: https://.*:6443|server: https://127.0.0.1:6443|g' ~/.kube/config

# Run tests
make test

CI/CD

Faro uses GitHub Actions for continuous integration:

CI Workflow

On every push and pull request:

  • ✅ Unit tests
  • ✅ E2E tests (parallel job with kinc)
  • ✅ Integration tests (parallel job with kinc)
  • ✅ Operator deployment tests (parallel job with kinc)
  • ✅ Library import validation
  • ✅ Test artifact uploads

Release Workflow

On version tags (v*):

  • ✅ Full test suite validation
  • ✅ GoReleaser library release
  • ✅ Container image build and push to ghcr.io/t0masd/faro-operator
  • ✅ Multi-tag support (latest, v1.0.0, v1.0, v1)

View CI Status: https://github.com/T0MASD/faro/actions


JSON Event Export

When json_export: true, events are exported in structured format:

{
  "timestamp": "2025-11-10T14:33:02Z",
  "eventTime": "2025-11-10T16:04:11.882471903Z",
  "eventType": "ADDED",
  "gvr": "v1/pods",
  "namespace": "default",
  "name": "nginx-abc123",
  "uid": "12345678-1234-1234-1234-123456789012",
  "resourceVersion": "12345",
  "labels": {
    "app": "nginx",
    "version": "1.0"
  },
  "annotations": {
    "kubectl.kubernetes.io/last-applied-configuration": "..."
  }
}

timestamp and eventTime answer different questions. timestamp is the object's creationTimestamp - its age - so every update to one object carries the same value and a capture cannot be ordered by it. eventTime is when Faro observed the event, and is what a timeline is built from. timestamp is omitted when it is unknown, which is the case for a DELETED event reconstructed after the object is gone.

Events are written to:

  • Library mode: ${output_dir}/events-YYYYMMDD-HHMMSS.json
  • Operator mode: /var/faro/events/events-YYYYMMDD-HHMMSS.json

Examples

Check the examples/ directory for real-world usage:

  • examples/library-usage/ - Basic library integration, event and informer-status handlers
  • examples/workload-monitor/ - Dynamic workload detection
  • examples/worker-dispatcher/ - Event processing and actions

Each is its own package, so it builds and runs directly:

go run ./examples/library-usage

Documentation


Development

Building from Source

# Build library binary
make build

# Build with version info
make build-dev

# Build operator image
make operator-image

# Load image into local cluster
make operator-image-load

Project Structure

faro/
├── pkg/                    # Core library code
│   ├── client.go          # Kubernetes client
│   ├── config.go          # Configuration loading
│   ├── controller.go      # Main controller
│   ├── logger.go          # Structured logging
│   └── metrics.go         # Prometheus metrics
├── main.go                # CLI entrypoint
├── Dockerfile             # Operator container image
├── deploy/operator/       # Kubernetes manifests
│   ├── namespace.yaml
│   ├── serviceaccount.yaml
│   ├── clusterrole.yaml
│   ├── clusterrolebinding.yaml
│   ├── configmap.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── scripts/               # Deployment, test and release scripts
│   ├── deploy-operator.sh
│   ├── cleanup-operator.sh
│   ├── test-operator.sh
│   └── tag-version.sh     # make tag-patch/minor/major
├── tests/                 # Test suites
│   ├── unit/
│   ├── e2e/
│   ├── integration/
│   └── operator-ci/
├── examples/              # Usage examples, one package each
└── docs/                  # Documentation

Makefile Targets

make help          # Show all available targets
make build         # Build faro binary
make test          # Run all tests
make test-unit     # Run unit tests
make test-e2e      # Run E2E tests
make test-integration  # Run integration tests
make test-operator     # Run operator deployment tests
make clean         # Clean build artifacts
make operator-image    # Build operator container image

Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.

Running Tests Locally

  1. Ensure you have a Kubernetes cluster available (kinc recommended for local development)
  2. Run make test to validate your changes
  3. Run make test-operator to validate operator deployment

License

This project is released into the public domain under the Unlicense.


Key Principles

  1. Library provides mechanisms - Informers, events, JSON export, metrics
  2. Users implement policies - Business logic, filtering, workflows
  3. No fallbacks or defaults - Errors are surfaced, not hidden
  4. Clean separation - Core library vs. application concerns
  5. Dual deployment - Use as library or operator, your choice

Faro gives you the tools. You build the solutions.


Made with ❤️ for the Kubernetes community

About

Kubernetes resource observation tool with dynamic discovery and configuration-driven informer management.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages