Skip to content

feat(api): add gfsctl, a web dashboard and cluster endpoints (unfinished) - #52

Closed
mattmattox wants to merge 49 commits into
mainfrom
claude/api-wip
Closed

mattmattox wants to merge 49 commits into
mainfrom
claude/api-wip

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

The user's unfinished API work, moved out of #49 unchanged: /api/v1/cluster/{status,nodes,locks}, the gfsctl CLI and the React dashboard in web/dashboard, plus their docs. #49 keeps Prometheus metrics and rate limiting.

#49 has merged and this PR targets main. The branch merges main, and the diff is these 79 files.

Missing before this can be ready:

  • A stats provider. Mount never gives the API one, so /api/v1/stats/* and /api/v1/cluster/* answer 503, and gfsctl and the dashboard have nothing to show. This needs adapters for the cache, journal, TRIM, lock and node managers.
  • Snapshot and admin endpoints. gfsctl snapshot/admin and the dashboard's snapshot pages call /api/v1/snapshots (GET, POST, DELETE) and /api/v1/admin/{flush-journal,clear-cache,force-trim}. The server has none of these, and they would be its first write endpoints. That's a product decision for the user.
  • Serving the dashboard. Only npm run dev runs it. Nothing builds or serves dist, and serving it would add Node to CI and the release. Also a product decision.
  • CORS for the dashboard. fix(api): refuse CORS * without a token and check Host on the loopback API #50 allows only GET, HEAD and OPTIONS, so write endpoints would also need their methods allowed.
  • gfsctl tests, and a place for gfsctl in scripts/dist.sh.
  • The dashboard README is still the Vite template.

Results (.68, head ecd13ce, with main at 55ca3cd merged in): build, vet, gofmt, go mod tidy -diff, golangci-lint (0 issues), parity and go test -race ./internal/... ./cmd/... (20 packages) pass. Integration -short: 99 pass, 18 skip, 0 fail. go.mod keeps main's x/sys v0.50.0 (#53) and adds cobra v1.10.2 and yaml.v2 v2.4.0.

What works today: gfsctl builds and its read-only commands run. The dashboard passes tsc -b, eslint and vite build with the user's existing node_modules. web/ is source only.

🤖 Generated with Claude Code

mattmattox and others added 30 commits November 10, 2026 12:35
Set up the foundation package structure for the HTTP API server that will
serve the management CLI and web dashboard.

Files created:
- doc.go: Package documentation with endpoint overview
- config.go: Config struct with DefaultConfig(), Validate(), and builder methods
- errors.go: Package-specific error definitions
- types.go: JSON request/response type definitions
- routes.go: Route registration with /health endpoint
- server.go: Server struct with Start/Stop lifecycle
- server_test.go: Comprehensive unit tests (11 tests)

The API server follows existing GoClusterFS patterns:
- Config pattern from internal/cache
- Manager lifecycle pattern from internal/journal
- Dependency injection (no global singletons)

All tests pass (11/11).

Task: #5588
- Add /api/v1/ catch-all handler for versioned endpoints
- Add /metrics endpoint for Prometheus scraping
- Add /api/v1/status endpoint returning server status
- Add /api/v1/stats/* placeholder for component stats
- Add method validation (405 Method Not Allowed)
- Add route tests for all endpoint groups

Task: #5591
- Add LoggingMiddleware that logs method, path, status, duration
- Add AuthMiddleware with Bearer token validation and skip paths
- Add CORSMiddleware with configurable origins and preflight support
- Integrate middleware chain into server (logging -> CORS -> auth)
- Add comprehensive middleware tests (11 new tests)

Task: #5592
- Add HealthStatus constants (ok, degraded, unhealthy)
- Add ComponentHealth for individual health checks
- Enhance StatusResponse with capacity, cluster, mount state
- Add CacheStatsResponse matching internal/cache.Stats
- Add JournalStatsResponse matching internal/journal.JournalStats
- Add IOStatsResponse and StatsOverviewResponse
- Add comprehensive type marshaling tests

Task: #5593
- Add HealthChecker interface for component health reporting
- Add AggregateHealth to combine multiple health check results
- Implement priority: unhealthy > degraded > ok
- Return 503 for unhealthy status, 200 for ok/degraded
- Add RegisterHealthChecker to Server for dependency injection
- Add comprehensive health endpoint tests

Task: #5594
…y info

- Add version.go with build-time ldflags support (Version, GitCommit, BuildTime)
- Add StatusProvider interface for filesystem status injection
- Enhance handleStatus with build info and optional capacity
- Add status endpoint tests with mock provider

Task: #5595
- Add fsStatusProvider to expose filesystem status via API
- Start API server on :8080 after FUSE mount succeeds
- Register health checker and status provider
- Add graceful shutdown in signal handler and exit cleanup

Task: #5596
- Add -api flag to enable/disable API server (default: enabled)
- Add -api-addr for listen address (default: :8080)
- Add -api-token for Bearer token authentication
- Add -api-cors for comma-separated CORS allowed origins
- Wire flags to API server configuration via fluent API

Task: #5597
Add thread-safe accessor method to expose snapshot registry for API
endpoints. Uses RLock since we're only reading the pointer and the
Registry itself manages its own concurrent access with internal mutex.

Task: #5598
Add unified statistics collection and aggregation package:

- stats.go: Core types (Snapshot, CacheStats, JournalStats, etc.)
  and provider interfaces for component decoupling
- collector.go: Collector with TTL-based caching to aggregate
  statistics from multiple providers
- cache.go: Cache state utilities (CacheInfo, CachedSnapshot)
- errors.go: Package-specific error definitions
- doc.go: Package documentation
- stats_test.go: Comprehensive unit tests (16 tests)

The collector uses a pull model where providers are queried on
demand, with configurable cache TTL to avoid expensive repeated
queries. Preset configs available for different use cases:
DefaultConfig (1s TTL), HighFrequencyConfig (100ms), LowOverheadConfig (10s).

Task: #5599
Add comprehensive statistics types for TRIM operations, Sparse File
Bitmap allocation, and cluster node information:

- TrimStats: TRIM/discard operation metrics
- SFBStats: block allocation and fragmentation stats
- ClusterInfo: node role, state, and peer information
- PeerInfo: individual peer node details

Also add provider interfaces and Collector support for all new types.

Task: #5600
…sfb, cluster

- Add StatsProvider interface for statistics collection
- Implement /api/v1/stats endpoints:
  - /stats (overview with cache and journal)
  - /stats/cache (inode, extent, negative cache hit rates)
  - /stats/journal (transactions, space, checkpoints)
  - /stats/trim (TRIM operations and bytes)
  - /stats/lock (lock hint cache statistics)
  - /stats/sfb (block allocation statistics)
  - /stats/cluster (node membership and peers)
  - /stats/all (combined snapshot of all stats)
- Add response types for all stats endpoints
- Add comprehensive tests for stats handlers

Task: #5602, #5603, #5604, #5605, #5606, #5607
CI fails on unformatted files. Whitespace and import order only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t a token

The API listened on :8080 on every interface with no auth, so any host
that could reach the node could read cluster state. -api-addr now
defaults to 127.0.0.1:8080. Mount exits before touching the device when
-api-addr can be reached from other hosts and -api-token is empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Broadcast is on by default and its secret is empty by default, so most
clusters accept unauthenticated invalidation and peer messages without
knowing it. Mount now logs a WARNING line before it opens the device.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
internal/api kept its own Version, GitCommit and BuildTime, which
scripts/dist.sh never set, so /api/v1/status always said "dev".
GetBuildInfo now reads internal/version, and goclusterfs-mount -version
prints the same string as goclusterfs-fsck -version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
golangci-lint flagged mockFilesystemProvider as unused. Use it in a test
of SetFilesystemProvider instead of deleting it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Token bucket per client IP, off by default (Config.RateLimitEnabled).
Requests over the limit get 429 with Retry-After: 1.

Recovered from uncommitted work in the user's checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A custom collector reads the StatsProvider on each scrape and exports
cache, journal, TRIM, lock, SFB and cluster metrics plus Go runtime
metrics. Adds github.com/prometheus/client_golang v1.23.2, which raises
golang.org/x/sys from v0.28.0 to v0.35.0.

Recovered from uncommitted work in the user's checkout; prometheus.go
was gofmt'd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
StatsProvider gains ClusterNodes and ClusterLocks. The handlers answer
503 without a provider and 404 when the provider has no cluster data.

Recovered from uncommitted work in the user's checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gfsctl wraps the API: status, health, stats, cluster, snapshot, admin,
watch, config and shell completion. It reads host, port and token from
flags, GFSCTL_* variables or ~/.config/gfsctl/config.yaml. Adds spf13/cobra v1.10.2
and gopkg.in/yaml.v2 v2.4.0.

The snapshot and admin commands call /api/v1/snapshots and
/api/v1/admin/*, which the server does not have yet.

Recovered from uncommitted work in the user's checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vite + React 19 + Tailwind, with pages for the dashboard, cluster,
statistics, snapshots and settings. `npm run dev` serves it on :5173
and proxies /api, /health and /metrics to localhost:8080. Nothing
serves the built files yet.

Source only: node_modules and dist stay ignored by
web/dashboard/.gitignore. The lockfile is the user's from 2025-12-31.

Recovered from untracked work in the user's checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gfsctl kept its own Version, GitCommit and BuildTime, a third copy next
to internal/version and the one internal/api dropped. It now uses
internal/version like goclusterfs-mount and goclusterfs-fsck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
prometheus.NewGoCollector and NewProcessCollector are deprecated, and
staticcheck (SA1019) fails CI's lint job on them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
allocateImage opens an existing file, so the refusal test failed before
it ran the mount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the API port is taken, mount logged a second warning at unmount
("api: server not started"). Every mount after the first on a host
hits this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox and others added 9 commits November 10, 2026 13:29
main's tree equals #48's last head, already merged in the parent commit.
…uncommitted

# Conflicts:
#	internal/api/middleware.go
#	internal/api/server.go
…wn PR

They depend on a stats provider and on snapshot and admin endpoints that
the server does not have yet. This PR keeps Prometheus metrics and rate
limiting. The removed parts are restored unchanged in a stacked draft PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reverts 2505935 on top of the trimmed API PR, so this draft holds the user's
unfinished work unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The limiter took the client IP from X-Forwarded-For or X-Real-IP, so any
client could pick a fresh bucket per request. It now uses the
connection's address only. Its cleanup goroutine ran forever; Server.Stop
now ends it, also for a server that never started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Requests per second per client IP, with bursts of twice that. 0, the
default, keeps rate limiting off. A negative value stops mount before it
opens the device.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox and others added 8 commits November 10, 2026 14:34
Review of #49 found that the limiter, which runs before auth, kept one
bucket per source IP for 10-15 minutes. An IPv6 /64 or a local user on
127.0.0.0/8 could grow it to gigabytes. Now:
- A bucket is dropped once it has been idle long enough to refill
  (burst/rate, at most an hour), swept every 1-60 s.
- At most 65536 clients have buckets; new clients past that get 429.
- IPv6 clients share a bucket per /64; mapped IPv4 counts as IPv4.

Also:
- Validate refuses NaN, infinite and above-1e6 rates.
- Retry-After is ceil(1/rate), not always 1.
- /health and /metrics skip the limiter, so probes and scrapers that
  share 127.0.0.1 never get 429.
- promhttp serves at most 2 scrapes at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dropping the server without Stop leaked the limiter's cleanup goroutine.
-api-rate-limit now gets a clear error for NaN, infinite or huge values
instead of an overflowed burst.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt cap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 100 requests/s a token refills every 10 ms, so on a loaded host the
4th request often found a fresh token. 1 request/s keeps the bucket
empty for the test's duration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	docs/API-CONFIG.md
main's tree equals #49's last head, already merged in the parent commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox changed the base branch from claude/land-api-work-uncommitted to main November 10, 2026 21:40
mattmattox and others added 2 commits November 10, 2026 15:40
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 10, 2026
* feat(gfsctl): add a CLI client for the HTTP API

The user's gfsctl from their uncommitted work, unchanged (5eb160f in
#52). The following commits make it read-only and fix it. Adds
spf13/cobra v1.10.2 and gopkg.in/yaml.v2 v2.4.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(gfsctl): drop the write and cluster commands

snapshot and admin change the filesystem through endpoints the server
does not have yet; they come back with those endpoints. cluster called
/api/v1/cluster/*, which is not on main. The client keeps GET only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(gfsctl): make settings, errors and output behave, add metrics

- Settings resolve per field from a flag that was set, then GFSCTL_*,
  then the config file, then the default. GFSCTL_PORT and GFSCTL_TIMEOUT
  were checked but never read, and the config file loaded at package init.
- Bad GFSCTL_PORT, GFSCTL_TIMEOUT, output format or config file are
  errors, not silently ignored.
- Every error prints once. Some paths exited 1 with no message, and
  watch mode printed errors twice.
- Output goes to the command's writer, not os.Stdout, and is colored
  only on a terminal without NO_COLOR.
- An IPv6 --host gave http://::1:8080; the URL now uses JoinHostPort.
- version reads internal/version, like the other binaries.
- New: gfsctl metrics, and gfsctl stats cluster (/api/v1/stats/cluster).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(gfsctl): check every read command against the server's types

A fake API answers with JSON built from internal/api's response types,
and each command's -o json output must decode back to the same value.
This found that stats overview dropped the server's io section; the
client now has IOStatsResponse. Also tests settings precedence, bad
settings, error output, metrics and the base URL for IPv6 hosts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: build gfsctl with make build and ship it in release archives

dist.sh also collects gfsctl's dependency licenses, and the release
workflow checks that gfsctl reports the build commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(gfsctl): read status, health and metrics from a live mount

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: document gfsctl

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: list spf13/pflag as a direct dependency

The gfsctl tests import it to reset flags between runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox

Copy link
Copy Markdown
Contributor Author

Closing. This draft was split into three PRs:

@mattmattox mattmattox closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant