Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ fusermount3 -u mnt # or Ctrl+C the mo
./goclusterfs-fsck ./disk.img # device is a positional argument
```

`goclusterfs-mount` stays in the foreground until the filesystem is unmounted. `-format` formats and then mounts. It logs a lot of `[DEBUG]` lines even without `-debug`, which only turns on go-fuse request tracing. Flags and defaults: `./goclusterfs-mount -h`, `./goclusterfs-fsck -h`, [docs/API-CONFIG.md](docs/API-CONFIG.md).
`goclusterfs-mount` stays in the foreground until the filesystem is unmounted. `-format` formats and then mounts. `-debug` turns on `[DEBUG]` log lines and `-fuse-debug` go-fuse request tracing. The three commands share exit codes (`internal/cli`), listed in the README. Flags and defaults: `./goclusterfs-mount -h`, `./goclusterfs-fsck -h`, [docs/API-CONFIG.md](docs/API-CONFIG.md).

## Code Map

Expand Down Expand Up @@ -57,6 +57,8 @@ fusermount3 -u mnt # or Ctrl+C the mo
| `internal/metrics` | Atomic counters, not used by `cmd/mount` yet |
| `internal/api` | HTTP API served by `cmd/mount`: `/health`, `/metrics`, `/api/v1/status`, `/api/v1/stats/...`; loopback only unless `-api-token` is set |
| `internal/stats` | Background sampler that feeds `/api/v1/stats/...`, `/metrics` and the capacity in `/api/v1/status` |
| `internal/cli` | Exit codes, flag parsing and error hints shared by the three commands |
| `internal/logging` | The `-debug` switch: `logging.Debugf` prints only with `-debug` |
| `internal/version` | Build version, set with `-ldflags -X` by `scripts/dist.sh` |
| `internal/webui` | Serves the web dashboard under `/ui/`; embeds `make dashboard`'s build, or a placeholder page |
| `web/dashboard` | React dashboard source (Vite, npm); `make dashboard` builds it into `internal/webui/build/dist` |
Expand Down
26 changes: 22 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ df -h mnt
fusermount3 -u mnt
```

The mount process exits after the unmount. To mount the same image again, run the mount command without `-format`.
The mount process exits after the unmount. Ctrl+C or `kill` (SIGINT or SIGTERM) unmounts too: the mount stops its HTTP API first, then unmounts. While files under the mount are open, the unmount fails and the mount keeps running without its API; close the files and send the signal again. To mount the same image again, run the mount command without `-format`.

## Command-Line Options

Expand All @@ -87,7 +87,8 @@ These are the `goclusterfs-mount` flags. Run `./goclusterfs-mount -h` for the fu
| `-mount` | (required) | Mount point directory (must exist) |
| `-format` | `false` | Format the device, then mount it (destroys data) |
| `-bytes-per-inode` | `1048576` | With `-format`: one inode per this many bytes of device (4096 to 16,777,216 inodes) |
| `-debug` | `false` | Log every FUSE request |
| `-debug` | `false` | Print `[DEBUG]` log lines: file operations, locks, allocation |
| `-fuse-debug` | `false` | Log every FUSE request and reply |
| `-version` | `false` | Print the version and exit |

### Cache Options
Expand Down Expand Up @@ -153,7 +154,7 @@ The API only listens on this host by default. Mount refuses to start when `-api-

```bash
./gfsctl status # mount state, version, capacity
./gfsctl health # exits 1 when a component is unhealthy
./gfsctl health # exits 1 when degraded, 4 when unhealthy
./gfsctl stats # overview; also: cache, journal, trim, lock, sfb, cluster, all
./gfsctl metrics # Prometheus text from /metrics
./gfsctl status -w -i 5s # refresh every 5 s
Expand All @@ -169,6 +170,7 @@ It connects to `localhost:8080` by default. Each setting comes from the first of
| `--token` | `GFSCTL_TOKEN` | (none) |
| `--output`, `-o` | `GFSCTL_OUTPUT` | `text` (or `json`) |
| `--timeout` | `GFSCTL_TIMEOUT` | `30s` |
| `--debug` | | off: print each API request with its status and time |

Use `GFSCTL_TOKEN` or the config file for the token: other local users can read `--token` with `ps`. gfsctl and the API speak plain HTTP, so for a remote mount don't send the token across the network: tunnel over SSH (`ssh -L 8080:127.0.0.1:8080 node`, then `gfsctl status`) or put a TLS proxy in front of the API. Statistics answer 503 until mount feeds them to the API, and `gfsctl` prints that error.

Expand Down Expand Up @@ -218,10 +220,26 @@ Run fsck on an unmounted device. The device is a positional argument, and flags
./goclusterfs-fsck -repair ./disk.img
```

Exit codes follow fsck(8) and add up: `0` no errors, `1` errors repaired, `4` errors remain, `8` I/O failure or a check that could not finish or was refused, `16` usage error.
Exit codes follow fsck(8) and add up; see [Exit Codes and Debug Output](#exit-codes-and-debug-output).

`-repair` refuses to run while a node has the device mounted, while a stopped node's journal still needs replay, or when mount would refuse the superblock's layout. Back up the device before you repair it. The [fsck Guide](docs/TROUBLESHOOTING.md#fsck-guide) lists what fsck checks and what `-repair` changes.

## Exit Codes and Debug Output

`goclusterfs-mount`, `goclusterfs-fsck` and `gfsctl` use the same exit codes. They are bits, as with fsck(8), so fsck can add them up: 5 means some errors were repaired and some remain.

| Code | `goclusterfs-mount` | `goclusterfs-fsck` | `gfsctl` |
|------|---------------------|--------------------|----------|
| `0` | Mounted, then unmounted cleanly | No errors (warnings may remain) | Success; `health` is ok |
| `1` | | Errors were repaired | `health` is degraded |
| `4` | | Errors remain on disk | `health` is unhealthy |
| `8` | Could not mount, or stopped on an error | I/O failure, or the check could not finish or was refused, for example while the device is mounted | No answer, or an error from the API |
| `16` | Usage error | Usage error | Usage error |

A usage error is a bad flag, argument or setting; the command changed nothing. Error messages end with what to do next, such as the flag to set or the command to run.

`-debug` (or `--debug`) prints extra `[DEBUG]` lines to stderr and changes nothing else. `goclusterfs-mount` logs file operations, locks and block allocation, `goclusterfs-fsck` logs how long each phase took, and `gfsctl` logs each API request with its status and time. `goclusterfs-mount -fuse-debug` also logs every FUSE request and reply.

## Architecture

GoClusterFS uses a layered architecture:
Expand Down
14 changes: 11 additions & 3 deletions cmd/fsck/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@
// 4: Errors remain on disk
// 8: I/O failure, or the check could not finish or was refused
// 16: Usage error
//
// The exit codes, -debug and error hints match goclusterfs-mount and gfsctl.
package main

import (
Expand All @@ -33,7 +35,9 @@ import (
"fmt"
"os"

"github.com/supporttools/GoClusterFS/internal/cli"
"github.com/supporttools/GoClusterFS/internal/fsck"
"github.com/supporttools/GoClusterFS/internal/logging"
"github.com/supporttools/GoClusterFS/internal/storage"
"github.com/supporttools/GoClusterFS/internal/version"
)
Expand All @@ -51,6 +55,7 @@ func main() {
summaryOnly = flag.Bool("summary", false, "Only print final summary")
showProblems = flag.Bool("show-problems", true, "Show individual problems found")
showVersion = flag.Bool("version", false, "Print version and exit")
debug = flag.Bool("debug", false, "Print [DEBUG] log lines")

listSlots = flag.Bool("list-slots", false, "List node registry slots with their lease and journal, then exit")
releaseSlotN = flag.Int("release-slot", -1, "Check that registry slot N can be released and print what releasing it discards")
Expand All @@ -77,7 +82,10 @@ func main() {
fmt.Fprintf(os.Stderr, " 8: I/O failure, or the check could not finish or was refused\n")
fmt.Fprintf(os.Stderr, " 16: Usage error\n")
}
flag.Parse()
if code, ok := cli.ParseFlags(flag.CommandLine, os.Args[1:]); !ok {
os.Exit(code)
}
logging.SetDebug(*debug)

if *showVersion {
fmt.Println("goclusterfs-fsck", version.String())
Expand All @@ -92,7 +100,7 @@ func main() {

// Validate arguments
if flag.NArg() != 1 {
flag.Usage()
fmt.Fprintf(os.Stderr, "Error: give exactly one device, got %d arguments. Run %s -h to see the flags.\n", flag.NArg(), os.Args[0])
os.Exit(fsck.ExitUsage)
}
if *dryRun && !*repair {
Expand Down Expand Up @@ -129,7 +137,7 @@ func main() {
// Open device
dev, err := storage.Open(devicePath)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: Failed to open device: %v\n", err)
fmt.Fprintf(os.Stderr, "Error: Failed to open device: %v; %s\n", err, cli.OpenHint(err))
os.Exit(fsck.ExitOperational)
}
defer dev.Close()
Expand Down
19 changes: 15 additions & 4 deletions cmd/gfsctl/client/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,26 @@ package client

import (
"context"
"encoding/json"
"fmt"
"net/http"
)

// Health fetches the health status from the API.
// Health fetches the health status from the API. An unhealthy mount answers
// 503 with the same body, which Health returns without an error.
func (c *Client) Health(ctx context.Context) (*HealthResponse, error) {
var resp HealthResponse
if err := c.Get(ctx, "/health", &resp); err != nil {
body, status, err := c.send(ctx, http.MethodGet, "/health")
if err != nil {
return nil, err
}
return &resp, nil
var resp HealthResponse
if (status == http.StatusOK || status == http.StatusServiceUnavailable) && json.Unmarshal(body, &resp) == nil && resp.Status != "" {
return &resp, nil
}
if status < 400 {
return nil, fmt.Errorf("decode health response: %q", body)
}
return nil, apiError(status, body)
}

// Status fetches the filesystem status from the API.
Expand Down
51 changes: 31 additions & 20 deletions cmd/gfsctl/client/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (
"strconv"
"strings"
"time"

"github.com/supporttools/GoClusterFS/internal/logging"
)

// MaxResponseSize bounds how much of one response the client reads.
Expand Down Expand Up @@ -77,51 +79,60 @@ func (c *Client) BaseURL() string {
return c.baseURL
}

// doRequest performs an HTTP request and returns the response body.
// doRequest performs an HTTP request and returns the response body. A
// status of 400 or above is an *APIError.
func (c *Client) doRequest(ctx context.Context, method, path string) ([]byte, error) {
body, status, err := c.send(ctx, method, path)
if err != nil {
return nil, err
}
if status >= 400 {
return nil, apiError(status, body)
}
return body, nil
}

// send performs an HTTP request and returns the body and status whatever the status.
func (c *Client) send(ctx context.Context, method, path string) ([]byte, int, error) {
url := c.baseURL + path

req, err := http.NewRequestWithContext(ctx, method, url, nil)
if err != nil {
return nil, fmt.Errorf("create request: %w", err)
return nil, 0, fmt.Errorf("create request: %w", err)
}

req.Header.Set("Accept", "application/json")
if c.token != "" {
req.Header.Set("Authorization", "Bearer "+c.token)
}

start := time.Now()
logging.Debugf("%s %s (token set: %v, timeout %v)", method, url, c.token != "", c.httpClient.Timeout)
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, fmt.Errorf("request failed: %w", err)
logging.Debugf("%s %s failed after %v: %v", method, url, time.Since(start).Round(time.Millisecond), err)
return nil, 0, fmt.Errorf("request failed: %w", err)
}
defer resp.Body.Close()

body, err := io.ReadAll(io.LimitReader(resp.Body, MaxResponseSize+1))
logging.Debugf("%s %s: %s, %d bytes in %v", method, url, resp.Status, len(body), time.Since(start).Round(time.Millisecond))
if err != nil {
return nil, fmt.Errorf("read response: %w", err)
return nil, 0, fmt.Errorf("read response: %w", err)
}

if len(body) > MaxResponseSize {
return nil, fmt.Errorf("response from %s is larger than %d bytes", path, MaxResponseSize)
return nil, 0, fmt.Errorf("response from %s is larger than %d bytes", path, MaxResponseSize)
}
return body, resp.StatusCode, nil
}

if resp.StatusCode >= 400 {
var errResp ErrorResponse
if json.Unmarshal(body, &errResp) == nil && errResp.Error != "" {
return nil, &APIError{
StatusCode: resp.StatusCode,
Message: errResp.Error,
Details: errResp.Details,
}
}
return nil, &APIError{
StatusCode: resp.StatusCode,
Message: http.StatusText(resp.StatusCode),
}
func apiError(status int, body []byte) *APIError {
var errResp ErrorResponse
if json.Unmarshal(body, &errResp) == nil && errResp.Error != "" {
return &APIError{StatusCode: status, Message: errResp.Error, Details: errResp.Details}
}

return body, nil
return &APIError{StatusCode: status, Message: http.StatusText(status)}
}

// Get performs a GET request and decodes the JSON response into v.
Expand Down
2 changes: 1 addition & 1 deletion cmd/gfsctl/commands/config_cmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ func runConfig(cmd *cobra.Command, args []string) error {
case "init":
return initConfigFile(cmd)
default:
return fmt.Errorf("unknown subcommand: %s (valid: path, show, init)", subcmd)
return usageErrorf("unknown subcommand: %s (valid: path, show, init)", subcmd)
}
}

Expand Down
68 changes: 68 additions & 0 deletions cmd/gfsctl/commands/errors.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
package commands

import (
"errors"
"fmt"
"net"
"syscall"

"github.com/spf13/cobra"

"github.com/supporttools/GoClusterFS/cmd/gfsctl/client"
"github.com/supporttools/GoClusterFS/internal/cli"
)

// gfsctl health adds these to the exit codes in internal/cli.
const (
exitDegraded = 1
exitUnhealthy = 4
)

type usageError struct{ err error }

func (e usageError) Error() string { return e.err.Error() }
func (e usageError) Unwrap() error { return e.err }

func usageErrorf(format string, args ...any) error {
return usageError{fmt.Errorf(format, args...)}
}

type healthError struct{ status string }

func (e healthError) Error() string { return "filesystem is " + e.status }

// ExitCode returns the process exit code for an error from Execute.
func ExitCode(err error) int {
var health healthError
switch {
case err == nil:
return cli.ExitOK
case errors.As(err, &health) && health.status == "degraded":
return exitDegraded
case errors.As(err, &health):
return exitUnhealthy
case errors.As(err, &usageError{}):
return cli.ExitUsage
}
return cli.ExitFailure
}

// hint says what to do about err, or returns "" when its message says enough.
func hint(err error, cmd *cobra.Command) string {
var apiErr *client.APIError
var netErr net.Error
var dnsErr *net.DNSError
switch {
case errors.As(err, &usageError{}):
return "run " + cmd.CommandPath() + " --help"
case errors.Is(err, syscall.ECONNREFUSED):
return fmt.Sprintf("is goclusterfs-mount running with its API on %s? Point --host and --port (or GFSCTL_HOST and GFSCTL_PORT) at its -api-addr", apiClient.BaseURL())
case errors.As(err, &dnsErr):
return "check --host or GFSCTL_HOST"
case errors.As(err, &netErr) && netErr.Timeout():
return fmt.Sprintf("the mount did not answer within %v; check that it is not stuck, or raise --timeout", cfgTimeout)
case errors.As(err, &apiErr) && apiErr.IsUnauthorized():
return "set GFSCTL_TOKEN, --token or token in the config file (gfsctl config path) to the mount's API token"
}
return ""
}
Loading
Loading