diff --git a/CLAUDE.md b/CLAUDE.md index 9bcab9cd..efabca47 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ fusermount3 -u mnt # or Ctrl+C the mo ./goclusterfs-fsck ./disk.img # device is a positional argument ``` -`goclusterfs-mount` stays in the foreground until the filesystem is unmounted. `-format` formats and then mounts. It logs a lot of `[DEBUG]` lines even without `-debug`, which only turns on go-fuse request tracing. Flags and defaults: `./goclusterfs-mount -h`, `./goclusterfs-fsck -h`, [docs/API-CONFIG.md](docs/API-CONFIG.md). +`goclusterfs-mount` stays in the foreground until the filesystem is unmounted. `-format` formats and then mounts. `-debug` turns on `[DEBUG]` log lines and `-fuse-debug` go-fuse request tracing. The three commands share exit codes (`internal/cli`), listed in the README. Flags and defaults: `./goclusterfs-mount -h`, `./goclusterfs-fsck -h`, [docs/API-CONFIG.md](docs/API-CONFIG.md). ## Code Map @@ -57,6 +57,8 @@ fusermount3 -u mnt # or Ctrl+C the mo | `internal/metrics` | Atomic counters, not used by `cmd/mount` yet | | `internal/api` | HTTP API served by `cmd/mount`: `/health`, `/metrics`, `/api/v1/status`, `/api/v1/stats/...`; loopback only unless `-api-token` is set | | `internal/stats` | Background sampler that feeds `/api/v1/stats/...`, `/metrics` and the capacity in `/api/v1/status` | +| `internal/cli` | Exit codes, flag parsing and error hints shared by the three commands | +| `internal/logging` | The `-debug` switch: `logging.Debugf` prints only with `-debug` | | `internal/version` | Build version, set with `-ldflags -X` by `scripts/dist.sh` | | `internal/webui` | Serves the web dashboard under `/ui/`; embeds `make dashboard`'s build, or a placeholder page | | `web/dashboard` | React dashboard source (Vite, npm); `make dashboard` builds it into `internal/webui/build/dist` | diff --git a/README.md b/README.md index fa8a5c88..5ce10bff 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ df -h mnt fusermount3 -u mnt ``` -The mount process exits after the unmount. To mount the same image again, run the mount command without `-format`. +The mount process exits after the unmount. Ctrl+C or `kill` (SIGINT or SIGTERM) unmounts too: the mount stops its HTTP API first, then unmounts. While files under the mount are open, the unmount fails and the mount keeps running without its API; close the files and send the signal again. To mount the same image again, run the mount command without `-format`. ## Command-Line Options @@ -87,7 +87,8 @@ These are the `goclusterfs-mount` flags. Run `./goclusterfs-mount -h` for the fu | `-mount` | (required) | Mount point directory (must exist) | | `-format` | `false` | Format the device, then mount it (destroys data) | | `-bytes-per-inode` | `1048576` | With `-format`: one inode per this many bytes of device (4096 to 16,777,216 inodes) | -| `-debug` | `false` | Log every FUSE request | +| `-debug` | `false` | Print `[DEBUG]` log lines: file operations, locks, allocation | +| `-fuse-debug` | `false` | Log every FUSE request and reply | | `-version` | `false` | Print the version and exit | ### Cache Options @@ -153,7 +154,7 @@ The API only listens on this host by default. Mount refuses to start when `-api- ```bash ./gfsctl status # mount state, version, capacity -./gfsctl health # exits 1 when a component is unhealthy +./gfsctl health # exits 1 when degraded, 4 when unhealthy ./gfsctl stats # overview; also: cache, journal, trim, lock, sfb, cluster, all ./gfsctl metrics # Prometheus text from /metrics ./gfsctl status -w -i 5s # refresh every 5 s @@ -169,6 +170,7 @@ It connects to `localhost:8080` by default. Each setting comes from the first of | `--token` | `GFSCTL_TOKEN` | (none) | | `--output`, `-o` | `GFSCTL_OUTPUT` | `text` (or `json`) | | `--timeout` | `GFSCTL_TIMEOUT` | `30s` | +| `--debug` | | off: print each API request with its status and time | Use `GFSCTL_TOKEN` or the config file for the token: other local users can read `--token` with `ps`. gfsctl and the API speak plain HTTP, so for a remote mount don't send the token across the network: tunnel over SSH (`ssh -L 8080:127.0.0.1:8080 node`, then `gfsctl status`) or put a TLS proxy in front of the API. Statistics answer 503 until mount feeds them to the API, and `gfsctl` prints that error. @@ -218,10 +220,26 @@ Run fsck on an unmounted device. The device is a positional argument, and flags ./goclusterfs-fsck -repair ./disk.img ``` -Exit codes follow fsck(8) and add up: `0` no errors, `1` errors repaired, `4` errors remain, `8` I/O failure or a check that could not finish or was refused, `16` usage error. +Exit codes follow fsck(8) and add up; see [Exit Codes and Debug Output](#exit-codes-and-debug-output). `-repair` refuses to run while a node has the device mounted, while a stopped node's journal still needs replay, or when mount would refuse the superblock's layout. Back up the device before you repair it. The [fsck Guide](docs/TROUBLESHOOTING.md#fsck-guide) lists what fsck checks and what `-repair` changes. +## Exit Codes and Debug Output + +`goclusterfs-mount`, `goclusterfs-fsck` and `gfsctl` use the same exit codes. They are bits, as with fsck(8), so fsck can add them up: 5 means some errors were repaired and some remain. + +| Code | `goclusterfs-mount` | `goclusterfs-fsck` | `gfsctl` | +|------|---------------------|--------------------|----------| +| `0` | Mounted, then unmounted cleanly | No errors (warnings may remain) | Success; `health` is ok | +| `1` | | Errors were repaired | `health` is degraded | +| `4` | | Errors remain on disk | `health` is unhealthy | +| `8` | Could not mount, or stopped on an error | I/O failure, or the check could not finish or was refused, for example while the device is mounted | No answer, or an error from the API | +| `16` | Usage error | Usage error | Usage error | + +A usage error is a bad flag, argument or setting; the command changed nothing. Error messages end with what to do next, such as the flag to set or the command to run. + +`-debug` (or `--debug`) prints extra `[DEBUG]` lines to stderr and changes nothing else. `goclusterfs-mount` logs file operations, locks and block allocation, `goclusterfs-fsck` logs how long each phase took, and `gfsctl` logs each API request with its status and time. `goclusterfs-mount -fuse-debug` also logs every FUSE request and reply. + ## Architecture GoClusterFS uses a layered architecture: diff --git a/cmd/fsck/main.go b/cmd/fsck/main.go index 1d6eb3a9..ba7316b6 100644 --- a/cmd/fsck/main.go +++ b/cmd/fsck/main.go @@ -25,6 +25,8 @@ // 4: Errors remain on disk // 8: I/O failure, or the check could not finish or was refused // 16: Usage error +// +// The exit codes, -debug and error hints match goclusterfs-mount and gfsctl. package main import ( @@ -33,7 +35,9 @@ import ( "fmt" "os" + "github.com/supporttools/GoClusterFS/internal/cli" "github.com/supporttools/GoClusterFS/internal/fsck" + "github.com/supporttools/GoClusterFS/internal/logging" "github.com/supporttools/GoClusterFS/internal/storage" "github.com/supporttools/GoClusterFS/internal/version" ) @@ -51,6 +55,7 @@ func main() { summaryOnly = flag.Bool("summary", false, "Only print final summary") showProblems = flag.Bool("show-problems", true, "Show individual problems found") showVersion = flag.Bool("version", false, "Print version and exit") + debug = flag.Bool("debug", false, "Print [DEBUG] log lines") listSlots = flag.Bool("list-slots", false, "List node registry slots with their lease and journal, then exit") releaseSlotN = flag.Int("release-slot", -1, "Check that registry slot N can be released and print what releasing it discards") @@ -77,7 +82,10 @@ func main() { fmt.Fprintf(os.Stderr, " 8: I/O failure, or the check could not finish or was refused\n") fmt.Fprintf(os.Stderr, " 16: Usage error\n") } - flag.Parse() + if code, ok := cli.ParseFlags(flag.CommandLine, os.Args[1:]); !ok { + os.Exit(code) + } + logging.SetDebug(*debug) if *showVersion { fmt.Println("goclusterfs-fsck", version.String()) @@ -92,7 +100,7 @@ func main() { // Validate arguments if flag.NArg() != 1 { - flag.Usage() + fmt.Fprintf(os.Stderr, "Error: give exactly one device, got %d arguments. Run %s -h to see the flags.\n", flag.NArg(), os.Args[0]) os.Exit(fsck.ExitUsage) } if *dryRun && !*repair { @@ -129,7 +137,7 @@ func main() { // Open device dev, err := storage.Open(devicePath) if err != nil { - fmt.Fprintf(os.Stderr, "Error: Failed to open device: %v\n", err) + fmt.Fprintf(os.Stderr, "Error: Failed to open device: %v; %s\n", err, cli.OpenHint(err)) os.Exit(fsck.ExitOperational) } defer dev.Close() diff --git a/cmd/gfsctl/client/api.go b/cmd/gfsctl/client/api.go index 67b0be85..700f05e4 100644 --- a/cmd/gfsctl/client/api.go +++ b/cmd/gfsctl/client/api.go @@ -2,15 +2,26 @@ package client import ( "context" + "encoding/json" + "fmt" + "net/http" ) -// Health fetches the health status from the API. +// Health fetches the health status from the API. An unhealthy mount answers +// 503 with the same body, which Health returns without an error. func (c *Client) Health(ctx context.Context) (*HealthResponse, error) { - var resp HealthResponse - if err := c.Get(ctx, "/health", &resp); err != nil { + body, status, err := c.send(ctx, http.MethodGet, "/health") + if err != nil { return nil, err } - return &resp, nil + var resp HealthResponse + if (status == http.StatusOK || status == http.StatusServiceUnavailable) && json.Unmarshal(body, &resp) == nil && resp.Status != "" { + return &resp, nil + } + if status < 400 { + return nil, fmt.Errorf("decode health response: %q", body) + } + return nil, apiError(status, body) } // Status fetches the filesystem status from the API. diff --git a/cmd/gfsctl/client/client.go b/cmd/gfsctl/client/client.go index 14b434d3..e1b55eb8 100644 --- a/cmd/gfsctl/client/client.go +++ b/cmd/gfsctl/client/client.go @@ -11,6 +11,8 @@ import ( "strconv" "strings" "time" + + "github.com/supporttools/GoClusterFS/internal/logging" ) // MaxResponseSize bounds how much of one response the client reads. @@ -77,13 +79,26 @@ func (c *Client) BaseURL() string { return c.baseURL } -// doRequest performs an HTTP request and returns the response body. +// doRequest performs an HTTP request and returns the response body. A +// status of 400 or above is an *APIError. func (c *Client) doRequest(ctx context.Context, method, path string) ([]byte, error) { + body, status, err := c.send(ctx, method, path) + if err != nil { + return nil, err + } + if status >= 400 { + return nil, apiError(status, body) + } + return body, nil +} + +// send performs an HTTP request and returns the body and status whatever the status. +func (c *Client) send(ctx context.Context, method, path string) ([]byte, int, error) { url := c.baseURL + path req, err := http.NewRequestWithContext(ctx, method, url, nil) if err != nil { - return nil, fmt.Errorf("create request: %w", err) + return nil, 0, fmt.Errorf("create request: %w", err) } req.Header.Set("Accept", "application/json") @@ -91,37 +106,33 @@ func (c *Client) doRequest(ctx context.Context, method, path string) ([]byte, er req.Header.Set("Authorization", "Bearer "+c.token) } + start := time.Now() + logging.Debugf("%s %s (token set: %v, timeout %v)", method, url, c.token != "", c.httpClient.Timeout) resp, err := c.httpClient.Do(req) if err != nil { - return nil, fmt.Errorf("request failed: %w", err) + logging.Debugf("%s %s failed after %v: %v", method, url, time.Since(start).Round(time.Millisecond), err) + return nil, 0, fmt.Errorf("request failed: %w", err) } defer resp.Body.Close() body, err := io.ReadAll(io.LimitReader(resp.Body, MaxResponseSize+1)) + logging.Debugf("%s %s: %s, %d bytes in %v", method, url, resp.Status, len(body), time.Since(start).Round(time.Millisecond)) if err != nil { - return nil, fmt.Errorf("read response: %w", err) + return nil, 0, fmt.Errorf("read response: %w", err) } if len(body) > MaxResponseSize { - return nil, fmt.Errorf("response from %s is larger than %d bytes", path, MaxResponseSize) + return nil, 0, fmt.Errorf("response from %s is larger than %d bytes", path, MaxResponseSize) } + return body, resp.StatusCode, nil +} - if resp.StatusCode >= 400 { - var errResp ErrorResponse - if json.Unmarshal(body, &errResp) == nil && errResp.Error != "" { - return nil, &APIError{ - StatusCode: resp.StatusCode, - Message: errResp.Error, - Details: errResp.Details, - } - } - return nil, &APIError{ - StatusCode: resp.StatusCode, - Message: http.StatusText(resp.StatusCode), - } +func apiError(status int, body []byte) *APIError { + var errResp ErrorResponse + if json.Unmarshal(body, &errResp) == nil && errResp.Error != "" { + return &APIError{StatusCode: status, Message: errResp.Error, Details: errResp.Details} } - - return body, nil + return &APIError{StatusCode: status, Message: http.StatusText(status)} } // Get performs a GET request and decodes the JSON response into v. diff --git a/cmd/gfsctl/commands/config_cmd.go b/cmd/gfsctl/commands/config_cmd.go index d8020706..1501d667 100644 --- a/cmd/gfsctl/commands/config_cmd.go +++ b/cmd/gfsctl/commands/config_cmd.go @@ -53,7 +53,7 @@ func runConfig(cmd *cobra.Command, args []string) error { case "init": return initConfigFile(cmd) default: - return fmt.Errorf("unknown subcommand: %s (valid: path, show, init)", subcmd) + return usageErrorf("unknown subcommand: %s (valid: path, show, init)", subcmd) } } diff --git a/cmd/gfsctl/commands/errors.go b/cmd/gfsctl/commands/errors.go new file mode 100644 index 00000000..7e8d2a6d --- /dev/null +++ b/cmd/gfsctl/commands/errors.go @@ -0,0 +1,68 @@ +package commands + +import ( + "errors" + "fmt" + "net" + "syscall" + + "github.com/spf13/cobra" + + "github.com/supporttools/GoClusterFS/cmd/gfsctl/client" + "github.com/supporttools/GoClusterFS/internal/cli" +) + +// gfsctl health adds these to the exit codes in internal/cli. +const ( + exitDegraded = 1 + exitUnhealthy = 4 +) + +type usageError struct{ err error } + +func (e usageError) Error() string { return e.err.Error() } +func (e usageError) Unwrap() error { return e.err } + +func usageErrorf(format string, args ...any) error { + return usageError{fmt.Errorf(format, args...)} +} + +type healthError struct{ status string } + +func (e healthError) Error() string { return "filesystem is " + e.status } + +// ExitCode returns the process exit code for an error from Execute. +func ExitCode(err error) int { + var health healthError + switch { + case err == nil: + return cli.ExitOK + case errors.As(err, &health) && health.status == "degraded": + return exitDegraded + case errors.As(err, &health): + return exitUnhealthy + case errors.As(err, &usageError{}): + return cli.ExitUsage + } + return cli.ExitFailure +} + +// hint says what to do about err, or returns "" when its message says enough. +func hint(err error, cmd *cobra.Command) string { + var apiErr *client.APIError + var netErr net.Error + var dnsErr *net.DNSError + switch { + case errors.As(err, &usageError{}): + return "run " + cmd.CommandPath() + " --help" + case errors.Is(err, syscall.ECONNREFUSED): + return fmt.Sprintf("is goclusterfs-mount running with its API on %s? Point --host and --port (or GFSCTL_HOST and GFSCTL_PORT) at its -api-addr", apiClient.BaseURL()) + case errors.As(err, &dnsErr): + return "check --host or GFSCTL_HOST" + case errors.As(err, &netErr) && netErr.Timeout(): + return fmt.Sprintf("the mount did not answer within %v; check that it is not stuck, or raise --timeout", cfgTimeout) + case errors.As(err, &apiErr) && apiErr.IsUnauthorized(): + return "set GFSCTL_TOKEN, --token or token in the config file (gfsctl config path) to the mount's API token" + } + return "" +} diff --git a/cmd/gfsctl/commands/errors_test.go b/cmd/gfsctl/commands/errors_test.go new file mode 100644 index 00000000..07e4b613 --- /dev/null +++ b/cmd/gfsctl/commands/errors_test.go @@ -0,0 +1,92 @@ +package commands + +import ( + "bytes" + "encoding/json" + "log" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "testing" + + "github.com/supporttools/GoClusterFS/internal/api" +) + +func serveHealth(t *testing.T, code int, status api.HealthStatus) *fakeAPI { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/api/v1/status" && r.Header.Get("Authorization") == "" { + w.WriteHeader(http.StatusUnauthorized) + json.NewEncoder(w).Encode(api.ErrorResponse{Error: "unauthorized", Code: http.StatusUnauthorized}) + return + } + w.WriteHeader(code) + json.NewEncoder(w).Encode(api.HealthResponse{Status: status, Checks: map[string]api.ComponentHealth{ + "recovery": {Status: status, Message: "replaying the journal of the crashed node in slot 3"}, + }}) + })) + t.Cleanup(srv.Close) + u, _ := url.Parse(srv.URL) + return &fakeAPI{host: u.Hostname(), port: u.Port()} +} + +func TestExitCodesAndHints(t *testing.T) { + ok := serveHealth(t, http.StatusOK, api.HealthStatusOK) + degraded := serveHealth(t, http.StatusOK, api.HealthStatusDegraded) + down := serveHealth(t, http.StatusServiceUnavailable, api.HealthStatusUnhealthy) + noStats := startFakeAPI(t, map[string]any{"/api/v1/stats/cache": http.StatusServiceUnavailable}) + + tests := []struct { + name string + run func() (string, string, error) + code int + stdout string + stderr string + }{ + {"healthy", func() (string, string, error) { return ok.run(t, "health") }, 0, "ok", ""}, + {"degraded", func() (string, string, error) { return degraded.run(t, "health") }, 1, "slot 3", "filesystem is degraded"}, + {"unhealthy answers 503", func() (string, string, error) { return down.run(t, "health") }, 4, "slot 3", "filesystem is unhealthy"}, + {"API error", func() (string, string, error) { return noStats.run(t, "stats", "cache") }, 8, "", "no stats provider configured"}, + {"no token", func() (string, string, error) { return ok.run(t, "status") }, 8, "", "set GFSCTL_TOKEN"}, + {"nothing listening", func() (string, string, error) { return run(t, "--port", "1", "metrics") }, 8, "", "is goclusterfs-mount running"}, + {"unknown flag", func() (string, string, error) { return run(t, "status", "--bogus") }, 16, "", "run gfsctl status --help"}, + {"unknown command", func() (string, string, error) { return run(t, "bogus") }, 16, "", "--help"}, + {"unknown component", func() (string, string, error) { return run(t, "stats", "bogus") }, 16, "", "unknown component"}, + {"watch with json", func() (string, string, error) { return run(t, "status", "-w", "-o", "json") }, 16, "", "drop one of them"}, + {"bad setting", func() (string, string, error) { return run(t, "-o", "yaml", "metrics") }, 16, "", "unknown output format"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + stdout, stderr, err := tt.run() + if got := ExitCode(err); got != tt.code { + t.Errorf("exit code = %d, want %d (err %v)", got, tt.code, err) + } + if !strings.Contains(stdout, tt.stdout) || !strings.Contains(stderr, tt.stderr) { + t.Errorf("stdout:\n%s\nstderr:\n%s\nwant %q and %q", stdout, stderr, tt.stdout, tt.stderr) + } + if tt.code == 0 && stderr != "" { + t.Errorf("stderr on success: %q", stderr) + } + }) + } +} + +func TestDebugLogsEachRequest(t *testing.T) { + f := startFakeAPI(t, map[string]any{"/metrics": "x\n"}) + var logs bytes.Buffer + log.SetOutput(&logs) + defer log.SetOutput(log.Writer()) + + if _, _, err := f.run(t, "metrics"); err != nil || logs.Len() != 0 { + t.Fatalf("without --debug: err=%v logs=%q", err, logs.String()) + } + if _, _, err := f.run(t, "--debug", "metrics"); err != nil { + t.Fatal(err) + } + for _, want := range []string{"[DEBUG] GET http://", "/metrics", "200 OK"} { + if !strings.Contains(logs.String(), want) { + t.Errorf("--debug logs lack %q:\n%s", want, logs.String()) + } + } +} diff --git a/cmd/gfsctl/commands/health.go b/cmd/gfsctl/commands/health.go index a940c8fe..c40251bb 100644 --- a/cmd/gfsctl/commands/health.go +++ b/cmd/gfsctl/commands/health.go @@ -11,7 +11,10 @@ var healthCmd = &cobra.Command{ Use: "health", Short: "Check filesystem health", Long: `Check the health status of the filesystem and its components. -Returns exit code 0 if healthy, 1 if unhealthy. + +Exit codes: 0 ok, 1 degraded (the mount works with a problem, such as a +journal replay of a crashed node or broadcast down), 4 unhealthy (the +mount cannot work correctly), 8 no answer from the API. Examples: gfsctl health # Check health in text format @@ -59,10 +62,8 @@ func runHealth(cmd *cobra.Command, args []string) error { f.Println() - // Return error if unhealthy (for scripting) - if health.Status == "unhealthy" { - return fmt.Errorf("filesystem is unhealthy") + if health.Status != "ok" { + return healthError{status: string(health.Status)} } - return nil } diff --git a/cmd/gfsctl/commands/root.go b/cmd/gfsctl/commands/root.go index 4cc79f53..acbfe73d 100644 --- a/cmd/gfsctl/commands/root.go +++ b/cmd/gfsctl/commands/root.go @@ -10,6 +10,7 @@ import ( "github.com/spf13/cobra" "github.com/supporttools/GoClusterFS/cmd/gfsctl/client" + "github.com/supporttools/GoClusterFS/internal/logging" ) var ( @@ -19,6 +20,10 @@ var ( cfgToken string cfgTimeout time.Duration cfgOutput string + cfgDebug bool + + // started is set once the flags and settings are valid; earlier errors are usage errors. + started bool // API client (initialized in PersistentPreRunE) apiClient *client.Client @@ -36,6 +41,9 @@ GoClusterFS mount through its HTTP API. It changes nothing. Settings come from flags, then GFSCTL_HOST, GFSCTL_PORT, GFSCTL_TOKEN, GFSCTL_OUTPUT and GFSCTL_TIMEOUT, then the config file (gfsctl config path). +Exit codes: 0 success, 1 health degraded, 4 health unhealthy, 8 no answer +or an error from the API, 16 usage error. + Examples: gfsctl status # Show filesystem status gfsctl health # Check health status @@ -45,9 +53,11 @@ Examples: SilenceUsage: true, SilenceErrors: true, PersistentPreRunE: func(cmd *cobra.Command, args []string) error { + logging.SetDebug(cfgDebug) if err := resolveSettings(cmd); err != nil { - return err + return usageError{err} } + started = true apiClient = client.New( client.WithHost(cfgHost), client.WithPort(cfgPort), @@ -58,12 +68,22 @@ Examples: }, } -// Execute runs the root command and prints any error to stderr. +// Execute runs the root command and prints any error, with what to do about +// it, to stderr. ExitCode maps the error to the exit code. func Execute() error { - err := rootCmd.Execute() - if err != nil { - fmt.Fprintln(rootCmd.ErrOrStderr(), "Error:", err) + started = false + cmd, err := rootCmd.ExecuteC() + if err == nil { + return nil + } + if !started { + err = usageError{err} + } + msg := err.Error() + if h := hint(err, cmd); h != "" { + msg += "; " + h } + fmt.Fprintln(rootCmd.ErrOrStderr(), "Error:", msg) return err } @@ -73,6 +93,7 @@ func init() { rootCmd.PersistentFlags().StringVar(&cfgToken, "token", "", "Bearer token for authentication (visible in ps; prefer GFSCTL_TOKEN or the config file)") rootCmd.PersistentFlags().DurationVar(&cfgTimeout, "timeout", 30*time.Second, "Request timeout") rootCmd.PersistentFlags().StringVarP(&cfgOutput, "output", "o", "text", "Output format (text, json)") + rootCmd.PersistentFlags().BoolVar(&cfgDebug, "debug", false, "Print [DEBUG] lines for each API request") } // resolveSettings fills every setting whose flag was not given from the diff --git a/cmd/gfsctl/commands/stats.go b/cmd/gfsctl/commands/stats.go index add5f1b8..1f4f8af0 100644 --- a/cmd/gfsctl/commands/stats.go +++ b/cmd/gfsctl/commands/stats.go @@ -51,7 +51,7 @@ func init() { func runStats(cmd *cobra.Command, args []string) error { // Watch mode not compatible with JSON output if statsWatch && GetOutputFormat() == "json" { - return fmt.Errorf("watch mode is not compatible with JSON output") + return usageErrorf("--watch does not work with -o json; drop one of them") } // Determine which stats to show @@ -65,7 +65,7 @@ func runStats(cmd *cobra.Command, args []string) error { case "", "cache", "journal", "trim", "lock", "sfb", "cluster", "all": // valid default: - return fmt.Errorf("unknown component: %s (valid: cache, journal, trim, lock, sfb, cluster, all)", component) + return usageErrorf("unknown component: %s (valid: cache, journal, trim, lock, sfb, cluster, all)", component) } // Create the display function diff --git a/cmd/gfsctl/commands/status.go b/cmd/gfsctl/commands/status.go index 45dc80b0..5a898e97 100644 --- a/cmd/gfsctl/commands/status.go +++ b/cmd/gfsctl/commands/status.go @@ -36,7 +36,7 @@ func init() { func runStatus(cmd *cobra.Command, args []string) error { // Watch mode not compatible with JSON output if statusWatch && GetOutputFormat() == "json" { - return fmt.Errorf("watch mode is not compatible with JSON output") + return usageErrorf("--watch does not work with -o json; drop one of them") } // Create the display function diff --git a/cmd/gfsctl/main.go b/cmd/gfsctl/main.go index 2031504e..f215769e 100644 --- a/cmd/gfsctl/main.go +++ b/cmd/gfsctl/main.go @@ -9,7 +9,5 @@ import ( ) func main() { - if err := commands.Execute(); err != nil { - os.Exit(1) - } + os.Exit(commands.ExitCode(commands.Execute())) } diff --git a/cmd/mount/health.go b/cmd/mount/health.go new file mode 100644 index 00000000..7bc02a16 --- /dev/null +++ b/cmd/mount/health.go @@ -0,0 +1,79 @@ +package main + +import ( + "context" + "fmt" + "strings" + + "github.com/supporttools/GoClusterFS/internal/api" +) + +type healthSources struct { + deviceError func() error + journal func() *api.JournalStatsResponse + replaying func() []int + broadcastOn bool + broadcastProblem func() string +} + +func registerHealthChecks(srv *api.Server, src healthSources) { + check := func(name string, fn func() api.ComponentHealth) { + srv.RegisterHealthChecker(api.NewHealthCheckFunc(name, func(context.Context) api.ComponentHealth { return fn() })) + } + check("device", func() api.ComponentHealth { return deviceHealth(src.deviceError()) }) + check("journal", func() api.ComponentHealth { return journalHealth(src.journal()) }) + check("recovery", func() api.ComponentHealth { return recoveryHealth(src.replaying()) }) + check("broadcast", func() api.ComponentHealth { return broadcastHealth(src.broadcastOn, src.broadcastProblem()) }) +} + +func healthy(message string) api.ComponentHealth { + return api.ComponentHealth{Status: api.HealthStatusOK, Message: message} +} + +func degraded(format string, args ...any) api.ComponentHealth { + return api.ComponentHealth{Status: api.HealthStatusDegraded, Message: fmt.Sprintf(format, args...)} +} + +func unhealthy(format string, args ...any) api.ComponentHealth { + return api.ComponentHealth{Status: api.HealthStatusUnhealthy, Message: fmt.Sprintf(format, args...)} +} + +func deviceHealth(lastErr error) api.ComponentHealth { + if lastErr != nil { + return unhealthy("the last device read or write failed: %v; check the device and the path to it, then unmount and run goclusterfs-fsck if it keeps failing", lastErr) + } + return healthy("") +} + +func journalHealth(j *api.JournalStatsResponse) api.ComponentHealth { + switch { + case j == nil: + return healthy("disabled with -journal=false: a crash can leave metadata half written") + case j.State == "corrupt": + return unhealthy("this node's journal is corrupt; unmount the filesystem and run goclusterfs-fsck") + case j.State == "recovering": + return degraded("replaying this node's journal") + } + return healthy("") +} + +func recoveryHealth(slots []int) api.ComponentHealth { + if len(slots) == 0 { + return healthy("") + } + names := make([]string, len(slots)) + for i, slot := range slots { + names[i] = fmt.Sprint(slot) + } + return degraded("replaying the journal of the crashed node in slot %s; files it had locked wait until the replay ends", strings.Join(names, ", ")) +} + +func broadcastHealth(on bool, problem string) api.ComponentHealth { + switch { + case !on: + return healthy("disabled with -broadcast=false: peers see this node's changes after -cache-ttl") + case problem != "": + return degraded("%s; peers see this node's changes only after -cache-ttl until it recovers. Check the network between the nodes and the -mcast-port and -unicast-port firewall rules", problem) + } + return healthy("") +} diff --git a/cmd/mount/health_test.go b/cmd/mount/health_test.go new file mode 100644 index 00000000..654575fe --- /dev/null +++ b/cmd/mount/health_test.go @@ -0,0 +1,114 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "strings" + "syscall" + "testing" + + "github.com/supporttools/GoClusterFS/internal/api" +) + +func TestHealthChecksSeparateDegradedFromDown(t *testing.T) { + tests := []struct { + name string + src healthSources + want api.HealthStatus + code int + problem string + }{ + {name: "all fine", src: fineSources(), want: api.HealthStatusOK, code: http.StatusOK}, + {name: "journal off and broadcast off are settings", src: func() healthSources { + s := fineSources() + s.journal = func() *api.JournalStatsResponse { return nil } + s.broadcastOn = false + return s + }(), want: api.HealthStatusOK, code: http.StatusOK}, + {name: "replaying a crashed peer", src: func() healthSources { + s := fineSources() + s.replaying = func() []int { return []int{3, 7} } + return s + }(), want: api.HealthStatusDegraded, code: http.StatusOK, problem: "recovery"}, + {name: "broadcast down", src: func() healthSources { + s := fineSources() + s.broadcastProblem = func() string { return "multicast is unavailable and no unicast peers are known" } + return s + }(), want: api.HealthStatusDegraded, code: http.StatusOK, problem: "broadcast"}, + {name: "device failing", src: func() healthSources { + s := fineSources() + s.deviceError = func() error { return syscall.EIO } + return s + }(), want: api.HealthStatusUnhealthy, code: http.StatusServiceUnavailable, problem: "device"}, + {name: "journal corrupt", src: func() healthSources { + s := fineSources() + s.journal = func() *api.JournalStatsResponse { return &api.JournalStatsResponse{State: "corrupt"} } + return s + }(), want: api.HealthStatusUnhealthy, code: http.StatusServiceUnavailable, problem: "journal"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + srv, err := api.NewServer(api.DefaultConfig().WithListenAddr("127.0.0.1:0")) + if err != nil { + t.Fatal(err) + } + registerHealthChecks(srv, tt.src) + if err := srv.Start(context.Background()); err != nil { + t.Fatal(err) + } + defer srv.Stop() + resp := api.AggregateHealth(context.Background(), srv.GetHealthCheckers()...) + if resp.Status != tt.want { + t.Errorf("status = %s, want %s: %+v", resp.Status, tt.want, resp.Checks) + } + if len(resp.Checks) != 4 { + t.Errorf("checks = %v, want device, journal, recovery and broadcast", resp.Checks) + } + for name, check := range resp.Checks { + if (name == tt.problem) != (check.Status != api.HealthStatusOK) { + t.Errorf("check %s = %+v", name, check) + } + if check.Status != api.HealthStatusOK && check.Message == "" { + t.Errorf("check %s is %s without saying why", name, check.Status) + } + } + + got, err := http.Get("http://" + srv.Addr() + "/health") + if err != nil { + t.Fatal(err) + } + defer got.Body.Close() + var body api.HealthResponse + if err := json.NewDecoder(got.Body).Decode(&body); err != nil { + t.Fatal(err) + } + if got.StatusCode != tt.code || body.Status != tt.want { + t.Errorf("GET /health = %d %s, want %d %s", got.StatusCode, body.Status, tt.code, tt.want) + } + }) + } +} + +func TestHealthMessagesNameTheSlotsAndTheError(t *testing.T) { + if msg := recoveryHealth([]int{3, 7}).Message; !strings.Contains(msg, "slot 3, 7") { + t.Errorf("recovery message = %q", msg) + } + if msg := deviceHealth(errors.New("input/output error")).Message; !strings.Contains(msg, "input/output error") || !strings.Contains(msg, "goclusterfs-fsck") { + t.Errorf("device message = %q", msg) + } + if got := journalHealth(&api.JournalStatsResponse{State: "recovering"}); got.Status != api.HealthStatusDegraded { + t.Errorf("recovering journal = %+v, want degraded", got) + } +} + +func fineSources() healthSources { + return healthSources{ + deviceError: func() error { return nil }, + journal: func() *api.JournalStatsResponse { return &api.JournalStatsResponse{State: "dirty"} }, + replaying: func() []int { return nil }, + broadcastOn: true, + broadcastProblem: func() string { return "" }, + } +} diff --git a/cmd/mount/main.go b/cmd/mount/main.go index 0d3707a2..ea07238b 100644 --- a/cmd/mount/main.go +++ b/cmd/mount/main.go @@ -11,6 +11,7 @@ import ( "os" "os/signal" "strings" + "sync/atomic" "syscall" "time" @@ -21,6 +22,7 @@ import ( "github.com/supporttools/GoClusterFS/internal/bitmap" "github.com/supporttools/GoClusterFS/internal/cache" "github.com/supporttools/GoClusterFS/internal/cache/broadcast" + "github.com/supporttools/GoClusterFS/internal/cli" "github.com/supporttools/GoClusterFS/internal/dir" "github.com/supporttools/GoClusterFS/internal/extent" "github.com/supporttools/GoClusterFS/internal/fileops" @@ -28,6 +30,7 @@ import ( "github.com/supporttools/GoClusterFS/internal/inode" "github.com/supporttools/GoClusterFS/internal/journal" "github.com/supporttools/GoClusterFS/internal/lock" + "github.com/supporttools/GoClusterFS/internal/logging" "github.com/supporttools/GoClusterFS/internal/node" "github.com/supporttools/GoClusterFS/internal/refcount" "github.com/supporttools/GoClusterFS/internal/stats" @@ -77,8 +80,9 @@ func main() { format = flag.Bool("format", false, "Format the device (WARNING: destroys data)") inodeRatio = flag.Uint64("bytes-per-inode", superblock.DefaultBytesPerInode, fmt.Sprintf("With -format: one inode per this many bytes of device (%d to %d inodes)", superblock.MinInodes, superblock.MaxInodes)) - debug = flag.Bool("debug", false, "Enable FUSE debug output") - showVer = flag.Bool("version", false, "Print version and exit") + debug = flag.Bool("debug", false, "Print [DEBUG] log lines") + fuseDebug = flag.Bool("fuse-debug", false, "Print every FUSE request and reply") + showVer = flag.Bool("version", false, "Print version and exit") // Cache configuration cacheStrict = flag.Bool("cache-strict", false, "Enable strict cache mode (always validate)") @@ -117,22 +121,31 @@ func main() { apiRate = flag.Float64("api-rate-limit", 0, "API requests per second per client IP, with bursts of twice that (0=off)") dashboard = flag.Bool("dashboard", true, "Serve the web dashboard on the API server under /ui/") ) - flag.Parse() + flag.Usage = func() { + fmt.Fprintf(os.Stderr, "Usage: %s -device -mount [-format] [-debug]\n\n", os.Args[0]) + flag.PrintDefaults() + fmt.Fprint(os.Stderr, exitCodesHelp) + } + if code, ok := cli.ParseFlags(flag.CommandLine, os.Args[1:]); !ok { + os.Exit(code) + } + logging.SetDebug(*debug) if *showVer { fmt.Println("goclusterfs-mount", version.String()) - os.Exit(0) + os.Exit(cli.ExitOK) } if *devicePath == "" || *mountPoint == "" { - fmt.Fprintf(os.Stderr, "Usage: %s -device -mount [-format] [-debug]\n", os.Args[0]) - flag.PrintDefaults() - os.Exit(1) + usageFailf("-device and -mount are required") + } + if flag.NArg() > 0 { + usageFailf("unexpected argument %q; give the device with -device and the mount point with -mount", flag.Arg(0)) } token, tokenSrc, err := loadAPIToken(*apiTokenFile, *apiToken) if err != nil { - log.Fatalf("Invalid API settings: %v", err) + usageFailf("Invalid API settings: %v", err) } if tokenSrc == tokenFromFlag { log.Printf("WARNING: -api-token is visible to every local user in ps and /proc//cmdline, and it unlocks API writes. Use -api-token-file or %s instead.", apiTokenEnv) @@ -143,14 +156,14 @@ func main() { } if *apiEnabled { if err := apiCfg.Validate(); err != nil { - log.Fatalf("Invalid API settings: %v", err) + usageFailf("Invalid API settings: %v", err) } if err := apiCfg.CheckExposure(); err != nil { hint := "set -api-addr to a loopback address such as 127.0.0.1:8080" if errors.Is(err, api.ErrWildcardCORSWithoutToken) { hint = "list the allowed origins in -api-cors" } - log.Fatalf("Refusing to start the API: %v. Set -api-token-file (or %s), %s, or pass -api=false", err, apiTokenEnv, hint) + usageFailf("Refusing to start the API: %v. Set -api-token-file (or %s), %s, or pass -api=false", err, apiTokenEnv, hint) } } if *broadcastEnabled && *broadcastSecret == "" { @@ -169,7 +182,7 @@ func main() { Interface: *broadcastIface, } broadcastFailed := func(err error) { - log.Fatalf("Failed to start cache broadcast: %v. Free UDP ports %d and %d, choose others with -mcast-port and -unicast-port, or pass -broadcast=false", err, *multicastPort, *unicastPort) + failf("Failed to start cache broadcast: %v. Free UDP ports %d and %d, choose others with -mcast-port and -unicast-port, or pass -broadcast=false", err, *multicastPort, *unicastPort) } // Bind and release the ports before registering or formatting, so a taken // port fails the mount before it changes anything on disk. @@ -177,17 +190,14 @@ func main() { broadcastFailed(err) } - // Check mount point exists if _, err := os.Stat(*mountPoint); os.IsNotExist(err) { - log.Fatalf("Mount point does not exist: %s", *mountPoint) + failf("Mount point %s does not exist; create it with mkdir -p %s", *mountPoint, *mountPoint) } - // Open device dev, err := storage.Open(*devicePath) if err != nil { - log.Fatalf("Failed to open device: %v", err) + failf("Failed to open device: %v; %s", err, cli.OpenHint(err)) } - defer dev.Close() // Get device size (returns int64, no error) devSize := dev.Size() @@ -200,8 +210,11 @@ func main() { } else { sb, err = superblock.Read(dev) } + if errors.Is(err, superblock.ErrChecksumMismatch) { + failf("Cannot use %s: %v; check it with goclusterfs-fsck %s", *devicePath, err, *devicePath) + } if err != nil { - log.Fatalf("Cannot use %s: %v", *devicePath, err) + failf("Cannot use %s: %v", *devicePath, err) } bitmapOffset := sb.BlockBitmapOffset @@ -224,14 +237,18 @@ func main() { nodeReg.SetJournalArea(sb.JournalRegionOffset, sb.JournalSlotSize()) hostname, _ := os.Hostname() slotIdx, err := nodeReg.Register(ctx, hostname) + if errors.Is(err, node.ErrNoFreeSlots) { + failf("Cannot join the cluster: all %d node slots are taken. List them with goclusterfs-fsck -list-slots %s, and free the slot of a node that will not come back with -release-slot", node.MaxNodes, *devicePath) + } if err != nil { - log.Fatalf("Failed to register node: %v", err) + failf("Cannot join the cluster: %v; check the device, then mount again", err) } nodeReg.SetSlotLostHandler(func(slot int, owner [16]byte) { - log.Fatalf("Node registry slot %d is no longer this node's (owner %x); stopping so two nodes never share a journal", slot, owner[:8]) + failf("Node registry slot %d is no longer this node's (owner %x); stopping so two nodes never share a journal. "+ + "Check that the device is reachable and that node clocks agree within a few seconds, then mount again", slot, owner[:8]) }) if err := nodeReg.StartHeartbeat(); err != nil { - log.Fatalf("Failed to start node heartbeat: %v", err) + failf("Failed to start node heartbeat: %v", err) } dev.SetWriteGate(nodeReg.WriteGate) nodeUUID := nodeReg.LocalUUID() @@ -288,7 +305,7 @@ func main() { // Replay a dirty journal left in this slot's region by its previous owner result, err := recoverOwnJournal(dev, journalOffset, journalSize, recoveryHandler, slotIdx) if err != nil { - log.Fatalf("Journal recovery for slot %d failed: %v (to discard this journal, mount with -skip-recovery=%d)", slotIdx, err, slotIdx) + failf("Journal recovery for slot %d failed: %v. Check the device, then mount again; to discard this journal instead, mount with -skip-recovery=%d", slotIdx, err, slotIdx) } if result != nil && result.WasNeeded { log.Printf("Journal recovery: replayed %d committed transactions (%d bytes scanned) from crashed node %x", @@ -312,7 +329,7 @@ func main() { // Recovery (if needed) has already been performed above j, err = journal.Init(dev, journalOffset, journalSize, nodeUUID, generation, jConfig) if err != nil { - log.Fatalf("Failed to initialize journal: %v", err) + failf("Failed to initialize the journal: %v; check the device, then run goclusterfs-fsck %s", err, *devicePath) } log.Printf("Journal: initialized at offset %d (%d MB)", journalOffset, journalSize/(1024*1024)) @@ -331,7 +348,7 @@ func main() { // Create cache manager for read optimization cacheMgr, err := cache.NewManager(cacheCfg, inodeTbl, lockMgr) if err != nil { - log.Fatalf("Failed to create cache manager: %v", err) + failf("Failed to create the cache: %v; check -cache-ttl", err) } log.Printf("Cache: %d inodes, %d negative entries, TTL=%v, strict=%v", cacheMgr.Config().InodeCacheSize, @@ -403,7 +420,7 @@ func main() { if *format { log.Println("Formatting filesystem...") if err := gfs.InitRoot(ctx); err != nil { - log.Fatalf("Failed to initialize root: %v", err) + failf("Failed to create the root directory: %v; check the device, then format again", err) } log.Println("Filesystem formatted successfully") } @@ -411,13 +428,13 @@ func main() { // Verify root exists rootIno, err := inodeTbl.Read(inode.InodeRoot) if err != nil || rootIno.Magic != inode.InodeMagic { - log.Fatalf("Filesystem not formatted. Use -format to initialize.") + failf("The root directory of %s is missing or damaged. Check it with goclusterfs-fsck %s; -format creates a new, empty filesystem and erases this one", *devicePath, *devicePath) } // Initialize snapshot subsystem after verifying filesystem is valid if *snapshotsEnabled { if err := gfs.InitSnapshots(ctx, snapshotOffset, snapshotSize, lockMgr, slotIdx); err != nil { - log.Fatalf("Failed to initialize snapshots: %v", err) + failf("Failed to load snapshots: %v; check the device with goclusterfs-fsck %s, or mount with -snapshots=false", err, *devicePath) } log.Printf("Snapshots: enabled (registry at %dMB, %dMB capacity)", snapshotOffset/(1024*1024), snapshotSize/(1024*1024)) @@ -431,7 +448,7 @@ func main() { MountOptions: fuse.MountOptions{ FsName: "goclusterfs", Name: "goclusterfs", - Debug: *debug, + Debug: *fuseDebug, MaxReadAhead: 128 * 1024, // 128KB readahead DisableXAttrs: false, // Note: AllowOther requires 'user_allow_other' in /etc/fuse.conf @@ -444,7 +461,7 @@ func main() { log.Printf("Mounting at %s...", *mountPoint) server, err := fs.Mount(*mountPoint, gfs, mountOpts) if err != nil { - log.Fatalf("Failed to mount: %v", err) + failf("Failed to mount at %s: %v. Check that /dev/fuse exists, fusermount3 is installed, and nothing else is mounted there (fusermount3 -u %s)", *mountPoint, err, *mountPoint) } // Start API server for management and monitoring @@ -454,7 +471,7 @@ func main() { var err error apiServer, err = api.NewServer(apiCfg) if err != nil { - log.Printf("Warning: Failed to create API server: %v", err) + log.Printf("Warning: Failed to create the API server: %v; the mount continues without it", err) } else { sampler = stats.StartSampler(statsSources(cacheMgr, lockMgr, j, trimMgr, bm, nodeReg)) apiServer.SetStatusProvider(&fsStatusProvider{ @@ -463,11 +480,17 @@ func main() { sampler: sampler, }) apiServer.SetStatsProvider(sampler) - apiServer.RegisterHealthChecker(api.DefaultHealthChecker()) + registerHealthChecks(apiServer, healthSources{ + deviceError: dev.LastError, + journal: sampler.JournalStats, + replaying: nodeReg.Replaying, + broadcastOn: cacheMgr.BroadcastEnabled(), + broadcastProblem: cacheMgr.BroadcastProblem, + }) apiServer.SetFileService(fileops.New(*mountPoint, gfs.SnapshotRegistry())) apiServer.SetAdminService(adminOps{journal: j, trim: trimMgr, cache: cacheMgr}) if err := apiServer.Start(ctx); err != nil { - log.Printf("Warning: Failed to start API server: %v", err) + log.Printf("Warning: Failed to start the API server: %v; the mount continues without it. Choose a free -api-addr, or pass -api=false", err) apiServer.Stop() apiServer = nil sampler.Stop() @@ -489,57 +512,48 @@ func main() { log.Println("API server: disabled") } - // Handle signals for clean unmount + stopAPI := func() { + if apiServer == nil { + return + } + if err := apiServer.Stop(); err != nil { + log.Printf("Warning: Failed to stop the API server: %v", err) + } + sampler.Stop() + } + + // The API stops before the unmount: its snapshot and clone requests go + // through the mount, and the unmount fails while they hold files open. + var unmounted atomic.Bool sigChan := make(chan os.Signal, 1) signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM) go func() { - <-sigChan - log.Println("Received signal, unmounting...") - - // Stop API server first - if apiServer != nil { - if err := apiServer.Stop(); err != nil { - log.Printf("Warning: Failed to stop API server: %v", err) + for range sigChan { + if unmounted.Load() { + log.Println("Received signal; already unmounted, finishing the shutdown") + continue + } + log.Println("Received signal, unmounting...") + stopAPI() + if err := server.Unmount(); err != nil { + log.Printf("Unmount of %s failed: %v. Close the files and shells that use it, then send the signal again or run fusermount3 -u %s", *mountPoint, err, *mountPoint) } - sampler.Stop() - } - - // Close filesystem (stops snapshot worker, TRIM, journal, etc.) - nodeReg.Stop() - if err := gfs.Close(); err != nil { - log.Printf("Warning: Failed to close filesystem: %v", err) - } - - // Stop broadcaster to notify peers - if err := cacheMgr.StopBroadcast(); err != nil { - log.Printf("Warning: Failed to stop broadcast: %v", err) } - - // Unmount using go-fuse server method - server.Unmount() }() log.Println("Filesystem mounted. Press Ctrl+C to unmount.") - - // Wait for filesystem to be unmounted - this blocks until unmount server.Wait() + unmounted.Store(true) + stopAPI() - // Stop API server - if apiServer != nil { - if err := apiServer.Stop(); err != nil { - log.Printf("Warning: Failed to stop API server on exit: %v", err) - } - sampler.Stop() - } - - // Clean shutdown - close filesystem (handles snapshot worker, TRIM, journal) + exit := cli.ExitOK nodeReg.Stop() if err := gfs.Close(); err != nil { - log.Printf("Warning: Failed to close filesystem on exit: %v", err) + log.Printf("Closing the filesystem failed: %v; run goclusterfs-fsck %s before the next mount", err, *devicePath) + exit = cli.ExitFailure } - if err := cacheMgr.StopBroadcast(); err != nil { - log.Printf("Warning: Failed to stop broadcast on exit: %v", err) + log.Printf("Warning: Failed to stop broadcast: %v", err) } // Print final TRIM stats @@ -566,6 +580,28 @@ func main() { } log.Println("Filesystem unmounted.") + _ = dev.Close() + os.Exit(exit) +} + +const exitCodesHelp = ` +Exit codes: + 0: The filesystem was mounted and later unmounted cleanly + 8: The mount failed, or stopped on an error + 16: Usage error; nothing was changed +` + +// failf logs why the mount cannot go on and exits with cli.ExitFailure. +func failf(format string, args ...any) { + log.Printf(format, args...) + os.Exit(cli.ExitFailure) +} + +// usageFailf reports a bad flag or setting and exits with cli.ExitUsage. +func usageFailf(format string, args ...any) { + fmt.Fprintf(os.Stderr, "Error: "+format+"\n", args...) + fmt.Fprintf(os.Stderr, "Run %s -h to see the flags.\n", os.Args[0]) + os.Exit(cli.ExitUsage) } func statsSources(cacheMgr *cache.Manager, lockMgr *lock.Manager, j *journal.Journal, trimMgr *trim.Manager, bm *bitmap.Region, nodeReg *node.Registry) stats.Sources { diff --git a/docs/API-CONFIG.md b/docs/API-CONFIG.md index ef348f97..67f71f47 100644 --- a/docs/API-CONFIG.md +++ b/docs/API-CONFIG.md @@ -6,9 +6,10 @@ This document provides a comprehensive reference for all configuration options i 1. [Command-Line Flags](#command-line-flags) 2. [HTTP API Statistics](#http-api-statistics) -3. [Code-Level Settings](#code-level-settings) -4. [Tuning Recommendations](#tuning-recommendations) -5. [Environment Variables](#environment-variables) +3. [HTTP API Health and Shutdown](#http-api-health-and-shutdown) +4. [Code-Level Settings](#code-level-settings) +5. [Tuning Recommendations](#tuning-recommendations) +6. [Environment Variables](#environment-variables) --- @@ -33,7 +34,8 @@ These are the `goclusterfs-mount` flags: |------|------|---------|-------------| | `-format` | bool | `false` | Format the device (destroys all data) | | `-bytes-per-inode` | uint | `1048576` | With `-format`: one inode per this many bytes of device, 4096 to 16,777,216 inodes | -| `-debug` | bool | `false` | Log every FUSE request and reply. Mount logs its own `[DEBUG]` lines either way | +| `-debug` | bool | `false` | Print `[DEBUG]` log lines | +| `-fuse-debug` | bool | `false` | Log every FUSE request and reply | | `-version` | bool | `false` | Print the version, commit and commit date, then exit | ### Cache Flags @@ -93,7 +95,7 @@ The kernel caches file attributes and directory entries for `-attr-timeout`. For | `-api-rate-limit` | float | `0` (off) | Requests per second per client IP, with bursts of twice that, up to 1,000,000. Over the limit, the API answers 429 | | `-dashboard` | bool | `true` | Serve the [web dashboard](#web-dashboard) under `/ui/` | -The API serves `GET /health`, `GET /metrics`, `GET /api/v1/status`, `GET /api/v1/stats/...`, the [web dashboard](#web-dashboard) under `/ui/` and, with a token, the [write endpoints](#http-api-write-endpoints). [HTTP API Statistics](#http-api-statistics) lists what `/metrics`, `/api/v1/status` and `/api/v1/stats/...` return. `/health` only checks the API itself. With a token set, `/api/v1/...` requests need `Authorization: Bearer `. `/health`, `/metrics` and the dashboard's files under `/ui/` never need the token. +The API serves `GET /health`, `GET /metrics`, `GET /api/v1/status`, `GET /api/v1/stats/...`, the [web dashboard](#web-dashboard) under `/ui/` and, with a token, the [write endpoints](#http-api-write-endpoints). [HTTP API Statistics](#http-api-statistics) lists what `/metrics`, `/api/v1/status` and `/api/v1/stats/...` return. `/health` checks the device, journal, journal recovery and broadcast; see [HTTP API Health and Shutdown](#http-api-health-and-shutdown). With a token set, `/api/v1/...` requests need `Authorization: Bearer `. `/health`, `/metrics` and the dashboard's files under `/ui/` never need the token. Without a token, the API answers only requests whose `Host` header is `localhost`, a `127.x` address or `[::1]`, so a web page cannot reach it through DNS rebinding, and mount refuses `-api-cors '*'`. List the allowed origins instead. The dashboard at `/ui/` shares the API's origin and needs no CORS. @@ -158,6 +160,23 @@ Mount copies its managers' counters every second. While requests read capacity o --- +## HTTP API Health and Shutdown + +`GET /health` runs four checks and reports the worst of them: + +| Check | `ok` | `degraded` | `unhealthy` | +|-------|------|------------|-------------| +| `device` | The last device read, write or sync worked | | The last device read, write or sync failed | +| `journal` | Clean or dirty, or off with `-journal=false` | This node's journal is being replayed | This node's journal is corrupt | +| `recovery` | No replay is running | This node replays the journal of a crashed node | | +| `broadcast` | Working, or off with `-broadcast=false` | The last invalidation send failed, or multicast is unavailable and no unicast peers are known | | + +`ok` and `degraded` answer 200; `unhealthy` answers 503. Degraded means the mount works with a problem: during a replay, files the crashed node had locked wait until it ends, and with broadcast down, peers see this node's changes only after `-cache-ttl`. Unhealthy means file operations fail. A mount that is not running refuses the connection. A check that is not `ok` has a `message` that says what to do. `gfsctl health` exits 0, 1 or 4 for the three states. + +The API starts after the mount has replayed its own journal, so a mount that is still starting refuses the connection too. + +On SIGINT or SIGTERM, the mount stops the API before it unmounts, because snapshot and clone requests go through the mount. The API stops accepting connections and gives requests in flight 10 seconds to finish. After that it cancels them and closes their connections; snapshot, clone and admin requests that have started still run to the end, and the API waits for them. After `fusermount3 -u`, the API stops the same way once the unmount is done. + ## Code-Level Settings The settings below have no flag unless the table names one. `goclusterfs-mount` uses these defaults; changing the others means editing the Go code and rebuilding. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 79918d2f..e0e61514 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -194,13 +194,13 @@ mkdir -p /mnt/goclusterfs ./goclusterfs-mount -device /path/to/device -mount /mnt/goclusterfs ``` -### Mount with FUSE Request Tracing +### Mount with Debug Output -`-debug` logs every FUSE request. The mount logs its own `[DEBUG]` lines either way. +`-debug` adds the mount's `[DEBUG]` lines, and `-fuse-debug` logs every FUSE request and reply. ```bash ./goclusterfs-mount -device /path/to/device -mount /mnt/goclusterfs \ - -debug + -debug -fuse-debug ``` ### Systemd Service (Single Node) diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 3d2b24ff..2a56d71f 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -16,9 +16,9 @@ This guide covers day-to-day operations for managing GoClusterFS deployments. ## Monitoring -Each mount serves its counters over the HTTP API on `127.0.0.1:8080`: Prometheus metrics at `/metrics`, and JSON under `/api/v1/status` and `/api/v1/stats/...`. [HTTP API Statistics](API-CONFIG.md#http-api-statistics) lists them. `gfsctl status`, `gfsctl stats` and the web dashboard at `/ui/` read the same data. The counters cover this node only and start at zero when it mounts. +Each mount serves its counters over the HTTP API on `127.0.0.1:8080`: Prometheus metrics at `/metrics`, and JSON under `/api/v1/status` and `/api/v1/stats/...`. [HTTP API Statistics](API-CONFIG.md#http-api-statistics) lists them. `gfsctl status`, `gfsctl stats` and the web dashboard at `/ui/` read the same data. The counters cover this node only and start at zero when it mounts. `/health` says whether the mount is ok, degraded or unhealthy; see [HTTP API Health and Shutdown](API-CONFIG.md#http-api-health-and-shutdown). -`goclusterfs-mount` also writes a log to stderr, so capture it (see [Background Mount](#background-mount-with-logging) or the systemd unit in the [Deployment Guide](DEPLOYMENT.md#systemd-service-single-node)). The log has many `[DEBUG]` lines, which are always on. +`goclusterfs-mount` also writes a log to stderr, so capture it (see [Background Mount](#background-mount-with-logging) or the systemd unit in the [Deployment Guide](DEPLOYMENT.md#systemd-service-single-node)). With `-debug`, the log also has `[DEBUG]` lines. ### Totals at Unmount @@ -117,16 +117,16 @@ A mount that was killed leaves its journal for the next mount on the same host t `fusermount3 -u`, `SIGINT` (Ctrl+C) and `SIGTERM` (`kill`, or `systemctl stop` without an `ExecStop`) all shut down cleanly. On a signal the mount: -1. Logs `Received signal, unmounting...` and stops the HTTP API, after API writes in flight finish. -2. Stops its heartbeat and failure detection, after a replay of another node's journal in progress ends. -3. Stops the snapshot worker, flushes queued TRIM, stops the journal (marking it clean only if nothing is pending), and releases the lock records it keeps. -4. Stops broadcast and unmounts. +1. Logs `Received signal, unmounting...` and stops the HTTP API. Requests in flight get 10 seconds, then are cancelled; snapshot, clone and admin requests that have started run to the end. +2. Unmounts. If the mount point is busy, the unmount fails, the log says so, and the process keeps running without its API. Close the open files, then send the signal again or run `fusermount3 -u`. +3. Stops its heartbeat and failure detection, after a replay of another node's journal in progress ends. +4. Stops the snapshot worker, flushes queued TRIM, stops the journal (marking it clean only if nothing is pending), and releases the lock records it keeps. +5. Stops broadcast. -It then logs the totals and `Filesystem unmounted.`, and exits 0. After `fusermount3 -u`, the same steps run after the unmount. +It then logs the totals and `Filesystem unmounted.`, and exits 0, or 8 when closing the filesystem failed. After `fusermount3 -u`, steps 1, 3, 4 and 5 run once the unmount is done. -- On a signal, the mount closes the device before it unmounts, so a file operation still running fails. Stop applications first. -- If the mount point is busy, the unmount fails and the process keeps running with the device closed. Close the open files, then run `fusermount3 -u`. -- A second Ctrl+C does nothing. `kill -9` stops the process at once and leaves its journal for the next mount to replay. +- An open file or a file operation still running makes the unmount fail, so stop applications first. +- A signal after a failed unmount tries again; once the unmount is done, further signals are ignored. `kill -9` stops the process at once and leaves its journal for the next mount to replay. - The node's registry slot stays marked active after a clean unmount. A remount on the same host takes it back at once. Otherwise a running node clears it about 20 seconds later, and `goclusterfs-fsck -repair` waits up to 11 seconds before it trusts that no node is mounted. ### Running fsck diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index e3128c9a..0694dc83 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -479,10 +479,10 @@ fusermount3 -u /mnt/cluster ### Log Output -The mount always logs `[DEBUG]` lines for file operations, lock acquire and release, journal recovery, and broadcast. Capture stderr and filter it: +With `-debug`, the mount logs `[DEBUG]` lines for file operations, lock acquire and release, and block allocation. Journal recovery and broadcast lines are always logged. Capture stderr and filter it: ```bash -./goclusterfs-mount -device /dev/device -mount /mnt/fs 2>&1 | tee mount.log +./goclusterfs-mount -device /dev/device -mount /mnt/fs -debug 2>&1 | tee mount.log grep "lock.Acquire" mount.log grep "\[broadcast\]" mount.log @@ -491,10 +491,10 @@ grep "\[Recovery\]" mount.log ### FUSE Request Tracing (Very Verbose) -`-debug` adds a log line for every FUSE request and reply: +`-fuse-debug` adds a log line for every FUSE request and reply: ```bash -./goclusterfs-mount -device /dev/device -mount /mnt/fs -debug 2>&1 | tee fuse.log +./goclusterfs-mount -device /dev/device -mount /mnt/fs -fuse-debug 2>&1 | tee fuse.log ``` --- diff --git a/internal/api/lifecycle_test.go b/internal/api/lifecycle_test.go new file mode 100644 index 00000000..94004e5c --- /dev/null +++ b/internal/api/lifecycle_test.go @@ -0,0 +1,154 @@ +package api + +import ( + "context" + "errors" + "net" + "net/http" + "runtime" + "strings" + "sync/atomic" + "testing" + "time" +) + +var serverFrames = []string{ + "net/http.(*conn).serve", + "net/http.(*Server).Serve", + "internal/api.(*Server)", + "internal/api.(*RateLimiter)", +} + +func serverGoroutines() (int, string) { + buf := make([]byte, 1<<20) + stacks := string(buf[:runtime.Stack(buf, true)]) + var count int + var found []string + for _, g := range strings.Split(stacks, "\n\n") { + for _, frame := range serverFrames { + if strings.Contains(g, frame) { + count++ + found = append(found, g) + break + } + } + } + return count, strings.Join(found, "\n\n") +} + +func expectNoLeftoverGoroutines(t *testing.T, before int) { + t.Helper() + deadline := time.Now().Add(2 * time.Second) + for { + n, stacks := serverGoroutines() + if n <= before { + return + } + if time.Now().After(deadline) { + t.Fatalf("%d server goroutines left after Stop, %d before Start:\n%s", n, before, stacks) + } + time.Sleep(10 * time.Millisecond) + } +} + +func startWithChecker(t *testing.T, cfg Config, checker HealthChecker) *Server { + t.Helper() + s, err := NewServer(cfg.WithListenAddr("127.0.0.1:0").WithRateLimit(100, 100)) + if err != nil { + t.Fatal(err) + } + s.RegisterHealthChecker(checker) + if err := s.Start(context.Background()); err != nil { + t.Fatal(err) + } + return s +} + +func getStatus(url string) int { + resp, err := http.Get(url) + if err != nil { + return 0 + } + resp.Body.Close() + return resp.StatusCode +} + +func TestStopLetsRequestsInFlightFinish(t *testing.T) { + before, _ := serverGoroutines() + entered, release := make(chan struct{}), make(chan struct{}) + s := startWithChecker(t, DefaultConfig(), NewHealthCheckFunc("slow", func(ctx context.Context) ComponentHealth { + close(entered) + <-release + return ComponentHealth{Status: HealthStatusOK} + })) + addr := s.Addr() + + got := make(chan int, 1) + go func() { got <- getStatus("http://" + addr + "/health") }() + <-entered + + stopped := make(chan error, 1) + go func() { stopped <- s.Stop() }() + select { + case err := <-stopped: + t.Fatalf("Stop returned %v while a request was in flight", err) + case <-time.After(100 * time.Millisecond): + } + if conn, err := net.DialTimeout("tcp", addr, time.Second); err == nil { + conn.Close() + t.Error("the server accepted a new connection while stopping") + } + + close(release) + if code := <-got; code != http.StatusOK { + t.Errorf("request in flight got %d, want 200", code) + } + if err := <-stopped; err != nil { + t.Errorf("Stop = %v, want nil", err) + } + expectNoLeftoverGoroutines(t, before) +} + +func TestStopCancelsRequestsThatOutlastTheTimeout(t *testing.T) { + before, _ := serverGoroutines() + cfg := DefaultConfig() + cfg.ShutdownTimeout = 50 * time.Millisecond + entered := make(chan struct{}) + var returned atomic.Bool + s := startWithChecker(t, cfg, NewHealthCheckFunc("stuck", func(ctx context.Context) ComponentHealth { + close(entered) + <-ctx.Done() + time.Sleep(50 * time.Millisecond) + returned.Store(true) + return ComponentHealth{Status: HealthStatusUnhealthy} + })) + + go getStatus("http://" + s.Addr() + "/health") + <-entered + + if err := s.Stop(); !errors.Is(err, ErrShutdownTimeout) { + t.Errorf("Stop = %v, want ErrShutdownTimeout", err) + } + if !returned.Load() { + t.Error("Stop returned before the cancelled handler did") + } + expectNoLeftoverGoroutines(t, before) +} + +func TestStopIdleServer(t *testing.T) { + before, _ := serverGoroutines() + s := startWithChecker(t, DefaultConfig(), DefaultHealthChecker()) + if code := getStatus("http://" + s.Addr() + "/health"); code != http.StatusOK { + t.Fatalf("GET /health = %d", code) + } + if err := s.Stop(); err != nil { + t.Fatalf("Stop = %v", err) + } + if err := s.Stop(); err != nil { + t.Errorf("second Stop = %v, want the first result", err) + } + if code := getStatus("http://" + s.Addr() + "/health"); code != 0 { + t.Errorf("GET /health after Stop = %d, want a refused connection", code) + } + expectNoLeftoverGoroutines(t, before) +} diff --git a/internal/api/server.go b/internal/api/server.go index 78a04321..6a634e71 100644 --- a/internal/api/server.go +++ b/internal/api/server.go @@ -2,6 +2,7 @@ package api import ( "context" + "errors" "fmt" "log" "net" @@ -39,9 +40,12 @@ type Server struct { fileService FileService adminService AdminService - // Write requests in flight; Stop waits for them so the mount outlives them. - writes sync.WaitGroup - stopping bool + // Requests in flight; Stop waits for them so the mount outlives them. + requests sync.WaitGroup + stopping bool + closed bool + cancelRequests context.CancelFunc + served chan struct{} // Lifecycle state started bool @@ -85,7 +89,7 @@ func NewServer(cfg Config) (*Server, error) { CORSMiddleware(cfg.CORSOrigins), AuthMiddleware(cfg.AuthToken, exempt...), ) - handler := Chain(s.router, middlewares...) + handler := s.track(Chain(s.router, middlewares...)) // Create HTTP server s.server = &http.Server{ @@ -120,8 +124,14 @@ func (s *Server) Start(ctx context.Context) error { s.started = true s.startTime = time.Now() + base, cancel := context.WithCancel(context.Background()) + s.cancelRequests = cancel + s.server.BaseContext = func(net.Listener) context.Context { return base } + s.served = make(chan struct{}) + // Start serving in background go func() { + defer close(s.served) log.Printf("api: server listening on %s", ln.Addr()) if err := s.server.Serve(ln); err != nil && err != http.ErrServerClosed { log.Printf("api: server error: %v", err) @@ -131,9 +141,9 @@ func (s *Server) Start(ctx context.Context) error { return nil } -// Stop gracefully shuts down the API server. -// It waits for active connections to complete up to the configured -// ShutdownTimeout before forcefully closing them. +// Stop shuts down the API server. Requests in flight get ShutdownTimeout to +// finish; after that their contexts are cancelled and their connections +// closed. Stop returns once every handler has returned. func (s *Server) Stop() error { s.stopOnce.Do(func() { if s.rateLimiter != nil { @@ -141,29 +151,32 @@ func (s *Server) Stop() error { } s.mu.Lock() s.stopping = true - if !s.started { - s.mu.Unlock() + started, timeout := s.started, s.config.ShutdownTimeout + s.mu.Unlock() + if !started { s.stopErr = ErrServerNotStarted return } - timeout := s.config.ShutdownTimeout - s.mu.Unlock() log.Printf("api: shutting down server...") - - // Create shutdown context with timeout ctx, cancel := context.WithTimeout(context.Background(), timeout) defer cancel() - - // Attempt graceful shutdown if err := s.server.Shutdown(ctx); err != nil { - if ctx.Err() == context.DeadlineExceeded { + s.cancelRequests() + _ = s.server.Close() + if errors.Is(err, context.DeadlineExceeded) { s.stopErr = ErrShutdownTimeout } else { s.stopErr = fmt.Errorf("shutdown error: %w", err) } } - s.waitForWrites() + <-s.served + + s.mu.Lock() + s.closed = true + s.mu.Unlock() + s.waitForRequests() + s.cancelRequests() log.Printf("api: server stopped") }) @@ -171,18 +184,35 @@ func (s *Server) Stop() error { return s.stopErr } -// waitForWrites waits for snapshot, clone and admin requests that are still -// running. They cannot be cancelled once the filesystem has started them. -func (s *Server) waitForWrites() { +// track counts each request so Stop can wait for it, and refuses requests +// that arrive after Stop stopped waiting. +func (s *Server) track(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + s.mu.Lock() + if s.closed { + s.mu.Unlock() + writeError(w, http.StatusServiceUnavailable, "server is shutting down", "") + return + } + s.requests.Add(1) + s.mu.Unlock() + defer s.requests.Done() + next.ServeHTTP(w, r) + }) +} + +// waitForRequests waits for handlers that are still running. Snapshot, clone +// and admin requests cannot be cancelled once the filesystem has started them. +func (s *Server) waitForRequests() { done := make(chan struct{}) go func() { - s.writes.Wait() + s.requests.Wait() close(done) }() select { case <-done: case <-time.After(time.Second): - log.Printf("api: waiting for write requests in flight to finish") + log.Printf("api: waiting for requests in flight to finish") <-done } } diff --git a/internal/api/writes.go b/internal/api/writes.go index 20feb16a..ef660420 100644 --- a/internal/api/writes.go +++ b/internal/api/writes.go @@ -128,25 +128,24 @@ func (s *Server) requireToken(w http.ResponseWriter) bool { return true } -// beginWrite refuses writes without a token or from another origin, and -// counts the request so Stop can wait for it. -func (s *Server) beginWrite(w http.ResponseWriter, r *http.Request) (func(), bool) { +// beginWrite refuses writes without a token, from another origin, or once +// the server is stopping. +func (s *Server) beginWrite(w http.ResponseWriter, r *http.Request) bool { if !s.requireToken(w) { - return nil, false + return false } if origin := r.Header.Get("Origin"); origin != "" && !sameOrigin(origin, r.Host) { writeError(w, http.StatusForbidden, "cross-origin writes are refused", "") - return nil, false + return false } - s.mu.Lock() - if s.stopping { - s.mu.Unlock() + s.mu.RLock() + stopping := s.stopping + s.mu.RUnlock() + if stopping { writeError(w, http.StatusServiceUnavailable, "server is shutting down", "") - return nil, false + return false } - s.writes.Add(1) - s.mu.Unlock() - return s.writes.Done, true + return true } func sameOrigin(origin, host string) bool { @@ -236,11 +235,9 @@ func (s *Server) listSnapshots(w http.ResponseWriter, r *http.Request) { } func (s *Server) createSnapshot(w http.ResponseWriter, r *http.Request) { - done, ok := s.beginWrite(w, r) - if !ok { + if !s.beginWrite(w, r) { return } - defer done() fs := s.fileServiceOrError(w) if fs == nil { return @@ -260,11 +257,9 @@ func (s *Server) createSnapshot(w http.ResponseWriter, r *http.Request) { } func (s *Server) deleteSnapshot(w http.ResponseWriter, r *http.Request, name string) { - done, ok := s.beginWrite(w, r) - if !ok { + if !s.beginWrite(w, r) { return } - defer done() fs := s.fileServiceOrError(w) if fs == nil { return @@ -281,11 +276,9 @@ func (s *Server) deleteSnapshot(w http.ResponseWriter, r *http.Request, name str } func (s *Server) cloneSnapshot(w http.ResponseWriter, r *http.Request, name string) { - done, ok := s.beginWrite(w, r) - if !ok { + if !s.beginWrite(w, r) { return } - defer done() fs := s.fileServiceOrError(w) if fs == nil { return @@ -314,11 +307,9 @@ func (s *Server) handleClones(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusMethodNotAllowed, "method not allowed", "use POST") return } - done, ok := s.beginWrite(w, r) - if !ok { + if !s.beginWrite(w, r) { return } - defer done() fs := s.fileServiceOrError(w) if fs == nil { return @@ -348,11 +339,9 @@ func (s *Server) handleAdmin(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusMethodNotAllowed, "method not allowed", "use POST") return } - done, ok := s.beginWrite(w, r) - if !ok { + if !s.beginWrite(w, r) { return } - defer done() _, as := s.services() if as == nil { writeError(w, http.StatusServiceUnavailable, "admin not available", "no admin service configured") diff --git a/internal/bitmap/bitmap.go b/internal/bitmap/bitmap.go index 3aa12e67..7f8eb8bc 100644 --- a/internal/bitmap/bitmap.go +++ b/internal/bitmap/bitmap.go @@ -17,6 +17,7 @@ import ( "time" "github.com/supporttools/GoClusterFS/internal/lock" + "github.com/supporttools/GoClusterFS/internal/logging" "github.com/supporttools/GoClusterFS/internal/storage" ) @@ -150,7 +151,7 @@ func (r *Region) Alloc(ctx context.Context, count int) ([]uint64, error) { return nil, r.rollback(allocated, ErrNoFreeBlocks) } - log.Printf("[DEBUG] bitmap.Alloc: allocated blocks %v", allocated) + logging.Debugf("bitmap.Alloc: allocated blocks %v", allocated) return allocated, nil } @@ -204,7 +205,7 @@ func (r *Region) Free(ctx context.Context, blockNum uint64) error { // Check if already free if (sectorData[byteIdx] & (1 << bitIdx)) == 0 { - log.Printf("[DEBUG] bitmap.Free: block %d already free!", blockNum) + logging.Debugf("bitmap.Free: block %d already free!", blockNum) return ErrAlreadyFree } @@ -215,7 +216,7 @@ func (r *Region) Free(ctx context.Context, blockNum uint64) error { return fmt.Errorf("failed to write bitmap sector: %w", err) } - log.Printf("[DEBUG] bitmap.Free: freed block %d", blockNum) + logging.Debugf("bitmap.Free: freed block %d", blockNum) // Queue block for TRIM if manager is configured if r.trimMgr != nil { diff --git a/internal/cache/broadcast/problem_test.go b/internal/cache/broadcast/problem_test.go new file mode 100644 index 00000000..f17eacd3 --- /dev/null +++ b/internal/cache/broadcast/problem_test.go @@ -0,0 +1,40 @@ +package broadcast + +import ( + "net" + "strings" + "testing" +) + +func TestProblemSaysWhyPeersMissInvalidations(t *testing.T) { + tr, err := NewTransport(DefaultConfig()) + if err != nil { + t.Fatal(err) + } + if got := tr.Problem(); got != "broadcast is not running" { + t.Errorf("before Start: %q", got) + } + + tr.running.Store(true) + if got := tr.Problem(); !strings.Contains(got, "no unicast peers") { + t.Errorf("no multicast and no peers: %q", got) + } + tr.AddPeer("peer", &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 9}) + if got := tr.Problem(); got != "" { + t.Errorf("a unicast peer is known: %q", got) + } + + if err := tr.Send(NewMessageBuilder([16]byte{1}).InvalidateAll()); err == nil { + t.Fatal("Send without a socket succeeded") + } + if got := tr.Problem(); !strings.Contains(got, "the last invalidation send failed") { + t.Errorf("after a failed send: %q", got) + } + + disabled := DefaultConfig() + disabled.Enabled = false + off, _ := NewTransport(disabled) + if got := off.Problem(); got != "" { + t.Errorf("broadcast off: %q", got) + } +} diff --git a/internal/cache/broadcast/receiver.go b/internal/cache/broadcast/receiver.go index 958b9803..9d1a8c55 100644 --- a/internal/cache/broadcast/receiver.go +++ b/internal/cache/broadcast/receiver.go @@ -495,6 +495,12 @@ func (b *Broadcaster) ResetStats() { atomic.StoreUint64(&b.messagesLost, 0) } +// Problem says why peers may miss this node's invalidations, or returns "" +// when they should get them. +func (b *Broadcaster) Problem() string { + return b.transport.Problem() +} + // IsEnabled returns whether broadcasting is enabled. func (b *Broadcaster) IsEnabled() bool { return b.transport.config.Enabled diff --git a/internal/cache/broadcast/transport.go b/internal/cache/broadcast/transport.go index d7c9b4db..715f4448 100644 --- a/internal/cache/broadcast/transport.go +++ b/internal/cache/broadcast/transport.go @@ -35,6 +35,7 @@ type Transport struct { unicastRecv uint64 multicastFailures uint64 // total over the transport's life multicast multicastHealth + lastSendErr atomic.Pointer[error] // State running atomic.Bool @@ -298,6 +299,16 @@ func (t *Transport) Send(msg *Message) error { return fmt.Errorf("marshal error: %w", err) } + err = t.deliver(data) + if err != nil { + t.lastSendErr.Store(&err) + } else if t.lastSendErr.Load() != nil { + t.lastSendErr.Store(nil) + } + return err +} + +func (t *Transport) deliver(data []byte) error { // Try multicast first. It goes out on the unicast socket: ListenMulticastUDP // turns off multicast loopback on its socket, so other nodes on this host // would not receive what that socket sends. @@ -420,6 +431,25 @@ func (h *multicastHealth) reset() { h.retryAt = time.Time{} } +// Problem says why peers may miss this node's invalidations, or returns "" +// when they should get them. +func (t *Transport) Problem() string { + if !t.config.Enabled { + return "" + } + if !t.running.Load() { + return "broadcast is not running" + } + if err := t.lastSendErr.Load(); err != nil { + return "the last invalidation send failed: " + (*err).Error() + } + multicast := t.multicastConn != nil && t.multicast.usable(time.Now(), t.config.MaxMulticastFailures) + if !multicast && t.PeerCount() == 0 { + return "multicast is unavailable and no unicast peers are known" + } + return "" +} + // Stats returns transport statistics. func (t *Transport) Stats() TransportStats { return TransportStats{ diff --git a/internal/cache/cache.go b/internal/cache/cache.go index dfcc3ced..e1289e78 100644 --- a/internal/cache/cache.go +++ b/internal/cache/cache.go @@ -271,6 +271,15 @@ func (m *Manager) BroadcastEnabled() bool { return m.broadcaster != nil && m.broadcaster.IsEnabled() } +// BroadcastProblem says why peers may miss this node's invalidations, or +// returns "" when they should get them or broadcast is off. +func (m *Manager) BroadcastProblem() string { + if m.broadcaster == nil { + return "" + } + return m.broadcaster.Problem() +} + // BroadcastInvalidateInode invalidates an inode locally AND broadcasts // the invalidation to peer nodes. Call this after modifying an inode, and // cache the new inode after this call, since it drops the local entry. diff --git a/internal/cli/cli.go b/internal/cli/cli.go new file mode 100644 index 00000000..7e6c7728 --- /dev/null +++ b/internal/cli/cli.go @@ -0,0 +1,47 @@ +// Package cli holds what goclusterfs-mount, goclusterfs-fsck and gfsctl +// share: exit codes, flag parsing and hints for common errors. +package cli + +import ( + "errors" + "flag" + "os" + "syscall" +) + +// Exit codes are bits, as with fsck(8), so goclusterfs-fsck can add its own +// results (1 repaired, 4 errors remain) to them. +const ( + ExitOK = 0 + ExitFailure = 8 // the command could not do its job, or refused to + ExitUsage = 16 // bad flags or arguments; nothing was changed +) + +// ParseFlags parses args. When it returns false the caller exits with the +// code: ExitOK after -h, ExitUsage after a bad flag. The flag package has +// already printed the error and the usage. +func ParseFlags(fs *flag.FlagSet, args []string) (int, bool) { + fs.Init(fs.Name(), flag.ContinueOnError) + if err := fs.Parse(args); err != nil { + if errors.Is(err, flag.ErrHelp) { + return ExitOK, false + } + return ExitUsage, false + } + return ExitOK, true +} + +// OpenHint says what to do about an error from opening a device or image. +func OpenHint(err error) string { + switch { + case errors.Is(err, os.ErrNotExist): + return "check the path" + case errors.Is(err, os.ErrPermission): + return "run as root or give this user read and write access to it" + case errors.Is(err, syscall.EINVAL): + return "its filesystem does not support O_DIRECT (tmpfs before Linux 6.6); put the image on ext4 or xfs" + case errors.Is(err, syscall.EBUSY): + return "another program has it open exclusively" + } + return "check that the device exists and is readable and writable" +} diff --git a/internal/cli/cli_test.go b/internal/cli/cli_test.go new file mode 100644 index 00000000..0c78b8e4 --- /dev/null +++ b/internal/cli/cli_test.go @@ -0,0 +1,48 @@ +package cli + +import ( + "flag" + "fmt" + "io" + "os" + "strings" + "syscall" + "testing" +) + +func TestParseFlagsExitCodes(t *testing.T) { + tests := []struct { + args []string + code int + ok bool + }{ + {nil, ExitOK, true}, + {[]string{"-debug"}, ExitOK, true}, + {[]string{"--debug"}, ExitOK, true}, + {[]string{"-h"}, ExitOK, false}, + {[]string{"-no-such-flag"}, ExitUsage, false}, + {[]string{"-count", "many"}, ExitUsage, false}, + } + for _, tt := range tests { + fs := flag.NewFlagSet("cmd", flag.ExitOnError) + fs.SetOutput(io.Discard) + fs.Bool("debug", false, "") + fs.Int("count", 0, "") + if code, ok := ParseFlags(fs, tt.args); code != tt.code || ok != tt.ok { + t.Errorf("ParseFlags(%q) = %d, %v, want %d, %v", tt.args, code, ok, tt.code, tt.ok) + } + } +} + +func TestOpenHintNamesTheFix(t *testing.T) { + tests := map[error]string{ + &os.PathError{Op: "open", Path: "/x", Err: syscall.ENOENT}: "check the path", + &os.PathError{Op: "open", Path: "/x", Err: syscall.EACCES}: "run as root", + fmt.Errorf("open: %w", syscall.EINVAL): "O_DIRECT", + } + for err, want := range tests { + if got := OpenHint(err); !strings.Contains(got, want) { + t.Errorf("OpenHint(%v) = %q, want it to mention %q", err, got, want) + } + } +} diff --git a/internal/dir/dir.go b/internal/dir/dir.go index 8e8b88a2..d7c083b1 100644 --- a/internal/dir/dir.go +++ b/internal/dir/dir.go @@ -20,11 +20,11 @@ import ( "encoding/binary" "errors" "fmt" - "log" "github.com/supporttools/GoClusterFS/internal/extent" "github.com/supporttools/GoClusterFS/internal/inode" "github.com/supporttools/GoClusterFS/internal/lock" + "github.com/supporttools/GoClusterFS/internal/logging" "github.com/supporttools/GoClusterFS/internal/storage" ) @@ -211,7 +211,7 @@ func (m *Manager) Add(ctx context.Context, dirIno *inode.Inode, name string, chi for attempt := 0; attempt < MaxAddRetries; attempt++ { err := m.addInternal(ctx, dirIno, entry) if err == errRetryAdd { - log.Printf("[DEBUG] dir.Add: retry attempt %d for entry %q in inode %d", + logging.Debugf("dir.Add: retry attempt %d for entry %q in inode %d", attempt+1, name, dirIno.InodeNum) continue } @@ -436,27 +436,27 @@ func (m *Manager) IsEmpty(ctx context.Context, dirIno *inode.Inode) (bool, error // InitDirectory initializes a new directory with "." and ".." entries. func (m *Manager) InitDirectory(ctx context.Context, dirIno *inode.Inode, parentInodeNum uint64) error { - log.Printf("[DEBUG] dir.InitDirectory: inode=%d parent=%d", dirIno.InodeNum, parentInodeNum) + logging.Debugf("dir.InitDirectory: inode=%d parent=%d", dirIno.InodeNum, parentInodeNum) if dirIno.Size > 0 { return fmt.Errorf("directory already has content") } // IMPORTANT: Block allocation must happen BEFORE acquiring inode lock // to respect lock ordering (bitmap lock level 6 < inode lock level 8) - log.Printf("[DEBUG] dir.InitDirectory: allocating block for inode %d", dirIno.InodeNum) + logging.Debugf("dir.InitDirectory: allocating block for inode %d", dirIno.InodeNum) ext, err := m.extentMgr.AllocateBlocks(ctx, dirIno, 0, 1) if err != nil { return fmt.Errorf("failed to allocate directory block: %w", err) } - log.Printf("[DEBUG] dir.InitDirectory: allocated physBlock=%d for inode %d", ext.PhysicalBlock, dirIno.InodeNum) + logging.Debugf("dir.InitDirectory: allocated physBlock=%d for inode %d", ext.PhysicalBlock, dirIno.InodeNum) // Now acquire exclusive lock for writing directory contents if m.lockMgr != nil { - log.Printf("[DEBUG] dir.InitDirectory: acquiring inode lock for inode %d", dirIno.InodeNum) + logging.Debugf("dir.InitDirectory: acquiring inode lock for inode %d", dirIno.InodeNum) if err := m.lockMgr.Acquire(ctx, lock.LockTypeInode, uint64(dirIno.InodeNum), lock.LockModeExclusive); err != nil { return fmt.Errorf("failed to acquire directory lock: %w", err) } - log.Printf("[DEBUG] dir.InitDirectory: acquired inode lock for inode %d", dirIno.InodeNum) + logging.Debugf("dir.InitDirectory: acquired inode lock for inode %d", dirIno.InodeNum) defer m.lockMgr.Release(lock.LockTypeInode, uint64(dirIno.InodeNum)) } @@ -492,11 +492,11 @@ func (m *Manager) InitDirectory(ctx context.Context, dirIno *inode.Inode, parent serializeEntry(dotdotEntry, data[offset:]) // Write to disk - log.Printf("[DEBUG] dir.InitDirectory: writing block %d for inode %d", ext.PhysicalBlock, dirIno.InodeNum) + logging.Debugf("dir.InitDirectory: writing block %d for inode %d", ext.PhysicalBlock, dirIno.InodeNum) if err := m.writeBlock(ext.PhysicalBlock, data); err != nil { return err } - log.Printf("[DEBUG] dir.InitDirectory: block write complete for inode %d", dirIno.InodeNum) + logging.Debugf("dir.InitDirectory: block write complete for inode %d", dirIno.InodeNum) // Update inode size and allocated blocks dirIno.Size = BlockSize @@ -505,11 +505,11 @@ func (m *Manager) InitDirectory(ctx context.Context, dirIno *inode.Inode, parent // CRITICAL: Write the inode to disk so subsequent operations // (which re-read from disk for distributed consistency) see the current state. // Use WriteWithLockHeld since we already hold the inode lock. - log.Printf("[DEBUG] dir.InitDirectory: writing inode %d to disk", dirIno.InodeNum) + logging.Debugf("dir.InitDirectory: writing inode %d to disk", dirIno.InodeNum) if err := m.inodeTbl.WriteWithLockHeld(dirIno); err != nil { return fmt.Errorf("failed to write directory inode: %w", err) } - log.Printf("[DEBUG] dir.InitDirectory: complete for inode %d", dirIno.InodeNum) + logging.Debugf("dir.InitDirectory: complete for inode %d", dirIno.InodeNum) return nil } @@ -707,7 +707,7 @@ func (m *Manager) addToNewBlock(ctx context.Context, dirIno *inode.Inode, entry if !blockUsed { // Free the allocated block since we didn't use it. // Ignore errors - best effort cleanup. - log.Printf("[DEBUG] dir.addToNewBlock: freeing unused block %d for inode %d", + logging.Debugf("dir.addToNewBlock: freeing unused block %d for inode %d", physBlock, dirIno.InodeNum) m.extentMgr.FreePhysicalBlock(ctx, physBlock) } @@ -750,7 +750,7 @@ func (m *Manager) addToNewBlock(ctx context.Context, dirIno *inode.Inode, entry _, _, lookupErr := m.lookupUnlocked(ctx, freshIno, entry.Name) if lookupErr == nil { // Entry was added by a concurrent operation - return ErrExists - log.Printf("[DEBUG] dir.addToNewBlock: entry %q already exists in inode %d (added by another node)", + logging.Debugf("dir.addToNewBlock: entry %q already exists in inode %d (added by another node)", entry.Name, dirIno.InodeNum) return ErrExists } @@ -792,7 +792,7 @@ func (m *Manager) addToNewBlock(ctx context.Context, dirIno *inode.Inode, entry if extErr == nil { // Block position is already taken - signal caller to retry. // The defer will free our allocated block. - log.Printf("[DEBUG] dir.addToNewBlock: logical block %d already occupied in inode %d, retrying", + logging.Debugf("dir.addToNewBlock: logical block %d already occupied in inode %d, retrying", nextBlock, dirIno.InodeNum) return errRetryAdd } @@ -823,7 +823,7 @@ func (m *Manager) addToNewBlock(ctx context.Context, dirIno *inode.Inode, entry // Overlap: another node beat us. Deadlock risk: the reservation was too // small for the fresh inode, so retry outside the lock. if errors.Is(err, extent.ErrExtentOverlap) || errors.Is(err, extent.ErrStaleTree) || errors.Is(err, lock.ErrDeadlockRisk) { - log.Printf("[DEBUG] dir.addToNewBlock: extent insert conflict in inode %d: %v, retrying", + logging.Debugf("dir.addToNewBlock: extent insert conflict in inode %d: %v, retrying", dirIno.InodeNum, err) return errRetryAdd } diff --git a/internal/extent/extent.go b/internal/extent/extent.go index c818124b..e7f85694 100644 --- a/internal/extent/extent.go +++ b/internal/extent/extent.go @@ -25,6 +25,7 @@ import ( "github.com/supporttools/GoClusterFS/internal/bitmap" "github.com/supporttools/GoClusterFS/internal/inode" "github.com/supporttools/GoClusterFS/internal/lock" + "github.com/supporttools/GoClusterFS/internal/logging" "github.com/supporttools/GoClusterFS/internal/refcount" "github.com/supporttools/GoClusterFS/internal/storage" ) @@ -314,7 +315,7 @@ func (m *Manager) FreePhysicalBlock(ctx context.Context, physBlock uint64) error // Truncate removes all extents beyond the given size. func (m *Manager) Truncate(ctx context.Context, ino *inode.Inode, newSize uint64) error { - log.Printf("[DEBUG] extent.Truncate: inode %d, newSize=%d, extentCount=%d", ino.InodeNum, newSize, ino.ExtentCount) + logging.Debugf("extent.Truncate: inode %d, newSize=%d, extentCount=%d", ino.InodeNum, newSize, ino.ExtentCount) if err := m.unmap(ctx, ino, BlocksForSize(newSize), math.MaxUint64); err != nil { return err } diff --git a/internal/fsck/fsck.go b/internal/fsck/fsck.go index 28139644..3ed4b701 100644 --- a/internal/fsck/fsck.go +++ b/internal/fsck/fsck.go @@ -24,6 +24,8 @@ import ( "sync" "time" + "github.com/supporttools/GoClusterFS/internal/cli" + "github.com/supporttools/GoClusterFS/internal/logging" "github.com/supporttools/GoClusterFS/internal/storage" ) @@ -246,11 +248,11 @@ func (r *Result) unrepairedErrors() int { // Exit codes of goclusterfs-fsck. As with fsck(8), they are bits: a repair // that fixed some errors and left others exits 5. const ( - ExitClean = 0 // no errors (warnings may remain) - ExitCorrected = 1 // errors were repaired - ExitUncorrected = 4 // errors remain on disk - ExitOperational = 8 // I/O failure, or the check could not finish or was refused - ExitUsage = 16 // usage error + ExitClean = 0 // no errors (warnings may remain) + ExitCorrected = 1 // errors were repaired + ExitUncorrected = 4 // errors remain on disk + ExitOperational = cli.ExitFailure // I/O failure, or the check could not finish or was refused + ExitUsage = cli.ExitUsage // usage error ) func (r *Result) ExitCode() int { @@ -509,8 +511,11 @@ func (c *Checker) Check() *Result { } c.logf("%s: Starting...\n", p.phase) + started, problems := time.Now(), len(c.result.Problems) + err := p.fn() + logging.Debugf("fsck: %s took %v and found %d problems", p.phase, time.Since(started).Round(time.Millisecond), len(c.result.Problems)-problems) - if err := p.fn(); err != nil { + if err != nil { c.result.AddProblem(Problem{ Phase: p.phase, Severity: Fatal, diff --git a/internal/goclusterfs/fs.go b/internal/goclusterfs/fs.go index 8a0a4345..9f534a68 100644 --- a/internal/goclusterfs/fs.go +++ b/internal/goclusterfs/fs.go @@ -31,6 +31,7 @@ import ( "github.com/supporttools/GoClusterFS/internal/inode" "github.com/supporttools/GoClusterFS/internal/journal" "github.com/supporttools/GoClusterFS/internal/lock" + "github.com/supporttools/GoClusterFS/internal/logging" "github.com/supporttools/GoClusterFS/internal/refcount" "github.com/supporttools/GoClusterFS/internal/sfb" "github.com/supporttools/GoClusterFS/internal/snapshot" @@ -591,29 +592,29 @@ func (f *FS) Getattr(ctx context.Context, fh fs.FileHandle, out *fuse.AttrOut) s // Lookup finds a child of the root directory by name. func (f *FS) Lookup(ctx context.Context, name string, out *fuse.EntryOut) (*fs.Inode, syscall.Errno) { ctx = detach(ctx) - log.Printf("[DEBUG] FS.Lookup: name=%q", name) + logging.Debugf("FS.Lookup: name=%q", name) dirIno, err := f.getInode(inode.InodeRoot) if err != nil { - log.Printf("[DEBUG] FS.Lookup: failed to get root inode: %v", err) + logging.Debugf("FS.Lookup: failed to get root inode: %v", err) return nil, syscall.EIO } childInodeNum, fileType, err := f.dirMgr.Lookup(ctx, dirIno, name) if err != nil { if err == dir.ErrNotFound { - log.Printf("[DEBUG] FS.Lookup: %q not found", name) + logging.Debugf("FS.Lookup: %q not found", name) return nil, syscall.ENOENT } - log.Printf("[DEBUG] FS.Lookup: error: %v", err) + logging.Debugf("FS.Lookup: error: %v", err) return nil, syscall.EIO } - log.Printf("[DEBUG] FS.Lookup: found %q -> inode %d, type %d", name, childInodeNum, fileType) + logging.Debugf("FS.Lookup: found %q -> inode %d, type %d", name, childInodeNum, fileType) // Read the child inode to populate attributes childIno, err := f.inodeTbl.Read(uint32(childInodeNum)) if err != nil { - log.Printf("[DEBUG] FS.Lookup: failed to read child inode %d: %v", childInodeNum, err) + logging.Debugf("FS.Lookup: failed to read child inode %d: %v", childInodeNum, err) return nil, syscall.EIO } @@ -645,7 +646,7 @@ func (f *FS) Lookup(ctx context.Context, name string, out *fuse.EntryOut) (*fs.I Mode: fuse.S_IFDIR, Ino: childInodeNum, }) - log.Printf("[DEBUG] FS.Lookup: returning dir inode %d", childInodeNum) + logging.Debugf("FS.Lookup: returning dir inode %d", childInodeNum) return ch, 0 } @@ -659,14 +660,14 @@ func (f *FS) Lookup(ctx context.Context, name string, out *fuse.EntryOut) (*fs.I Mode: fuse.S_IFREG, Ino: childInodeNum, }) - log.Printf("[DEBUG] FS.Lookup: returning file inode %d, size=%d", childInodeNum, childIno.Size) + logging.Debugf("FS.Lookup: returning file inode %d, size=%d", childInodeNum, childIno.Size) return ch, 0 } // Readdir returns a DirStream for the root directory listing. func (f *FS) Readdir(ctx context.Context) (fs.DirStream, syscall.Errno) { ctx = detach(ctx) - log.Printf("[DEBUG] FS.Readdir: root directory") + logging.Debugf("FS.Readdir: root directory") dirIno, err := f.getInode(inode.InodeRoot) if err != nil { return nil, syscall.EIO @@ -818,7 +819,7 @@ func retryRace(op, name string, try func() syscall.Errno) syscall.Errno { } baseDelay := time.Millisecond * time.Duration(1< inode %d, type %d", name, childInodeNum, fileType) + logging.Debugf("Lookup: found %q -> inode %d, type %d", name, childInodeNum, fileType) // Read the child inode to populate attributes childIno, err := d.fs.inodeTbl.Read(uint32(childInodeNum)) if err != nil { - log.Printf("[DEBUG] Lookup: failed to read child inode %d: %v", childInodeNum, err) + logging.Debugf("Lookup: failed to read child inode %d: %v", childInodeNum, err) return nil, syscall.EIO } @@ -1242,7 +1243,7 @@ func (d *Dir) Lookup(ctx context.Context, name string, out *fuse.EntryOut) (*fs. Mode: fuse.S_IFDIR, Ino: childInodeNum, }) - log.Printf("[DEBUG] Lookup: returning dir inode %d", childInodeNum) + logging.Debugf("Lookup: returning dir inode %d", childInodeNum) return ch, 0 } @@ -1256,7 +1257,7 @@ func (d *Dir) Lookup(ctx context.Context, name string, out *fuse.EntryOut) (*fs. Mode: fuse.S_IFREG, Ino: childInodeNum, }) - log.Printf("[DEBUG] Lookup: returning file inode %d, size=%d", childInodeNum, childIno.Size) + logging.Debugf("Lookup: returning file inode %d, size=%d", childInodeNum, childIno.Size) return ch, 0 } @@ -1287,7 +1288,7 @@ func (ds *dirStream) Close() {} // Readdir returns a DirStream for directory listing. func (d *Dir) Readdir(ctx context.Context) (fs.DirStream, syscall.Errno) { ctx = detach(ctx) - log.Printf("[DEBUG] Readdir: dir=%d", d.inodeNum) + logging.Debugf("Readdir: dir=%d", d.inodeNum) dirIno, err := d.fs.getInode(d.inodeNum) if err != nil { return nil, syscall.EIO @@ -1308,7 +1309,7 @@ func (d *Dir) Readdir(ctx context.Context) (fs.DirStream, syscall.Errno) { mode = fuse.S_IFLNK } - log.Printf("[DEBUG] Readdir: entry %q inode=%d type=%d", entry.Name, entry.InodeNum, entry.FileType) + logging.Debugf("Readdir: entry %q inode=%d type=%d", entry.Name, entry.InodeNum, entry.FileType) result = append(result, fuse.DirEntry{ Ino: entry.InodeNum, Name: entry.Name, @@ -1557,7 +1558,7 @@ func (d *Dir) removeEntryOnce(ctx context.Context, name string, isDir bool) sysc // === WAL: Mark intent complete === if d.fs.journal != nil && intentSeq > 0 { if err := d.fs.journal.CompleteIntent(intentSeq); err != nil { - log.Printf("[DEBUG] removeEntry %q: CompleteIntent failed: %v", name, err) + logging.Debugf("removeEntry %q: CompleteIntent failed: %v", name, err) } } @@ -1591,24 +1592,24 @@ var ( // Open opens the file for reading/writing. // Returns nil FileHandle to indicate the File node itself should be used as the handle. func (f *File) Open(ctx context.Context, flags uint32) (fs.FileHandle, uint32, syscall.Errno) { - log.Printf("[DEBUG] File.Open: inode=%d flags=%d", f.inodeNum, flags) + logging.Debugf("File.Open: inode=%d flags=%d", f.inodeNum, flags) // Verify the inode exists and is a regular file ino, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil { - log.Printf("[DEBUG] File.Open: read failed: %v", err) + logging.Debugf("File.Open: read failed: %v", err) return nil, 0, syscall.EIO } if !ino.IsFile() { - log.Printf("[DEBUG] File.Open: not a file") + logging.Debugf("File.Open: not a file") return nil, 0, syscall.EISDIR } - log.Printf("[DEBUG] File.Open: success, Size=%d", ino.Size) + logging.Debugf("File.Open: success, Size=%d", ino.Size) // Close-to-open consistency: reads through this handle start from the on-disk inode. f.fs.cacheInode(ino) // The kernel stops reads at the size it cached, which a lookup during another // node's write can leave short. Expiring it makes fstat and reads ask again. if errno := f.NotifyContent(-1, 0); errno != 0 && errno != syscall.ENOSYS { - log.Printf("[DEBUG] File.Open: inode %d: expiring kernel attributes failed: %v", f.inodeNum, errno) + logging.Debugf("File.Open: inode %d: expiring kernel attributes failed: %v", f.inodeNum, errno) } // Return nil FileHandle - the File node itself implements Read/Write/etc. // The fuse.FOPEN_KEEP_CACHE flag tells the kernel to keep cached data valid @@ -1618,16 +1619,16 @@ func (f *File) Open(ctx context.Context, flags uint32) (fs.FileHandle, uint32, s // Getattr returns file attributes. func (f *File) Getattr(ctx context.Context, fh fs.FileHandle, out *fuse.AttrOut) syscall.Errno { - log.Printf("[DEBUG] File.Getattr: inode=%d", f.inodeNum) + logging.Debugf("File.Getattr: inode=%d", f.inodeNum) // Read directly from disk to get fresh inode state. // In multi-node scenarios, another node may have updated the inode. ino, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil { - log.Printf("[DEBUG] File.Getattr: read failed: %v", err) + logging.Debugf("File.Getattr: read failed: %v", err) return syscall.EIO } - log.Printf("[DEBUG] File.Getattr: inode=%d Size=%d AllocatedBlocks=%d", f.inodeNum, ino.Size, ino.AllocatedBlocks) + logging.Debugf("File.Getattr: inode=%d Size=%d AllocatedBlocks=%d", f.inodeNum, ino.Size, ino.AllocatedBlocks) // Zero means go-fuse applies the mount's -attr-timeout. out.AttrValid = 0 @@ -1662,18 +1663,18 @@ func (f *File) Read(ctx context.Context, fh fs.FileHandle, dest []byte, off int6 // - Broadcast invalidation propagates changes between nodes ino, err := f.fs.getInode(f.inodeNum) if err != nil { - log.Printf("[DEBUG] Read inode %d: Read failed: %v", f.inodeNum, err) + logging.Debugf("Read inode %d: Read failed: %v", f.inodeNum, err) return nil, syscall.EIO } - log.Printf("[DEBUG] Read inode %d: Size=%d AllocatedBlocks=%d Extents=[%+v, %+v]", + logging.Debugf("Read inode %d: Size=%d AllocatedBlocks=%d Extents=[%+v, %+v]", f.inodeNum, ino.Size, ino.AllocatedBlocks, ino.InlineExtents[0], ino.InlineExtents[1]) offset := uint64(off) // Check bounds using the authoritative size from disk if offset >= ino.Size { - log.Printf("[DEBUG] Read inode %d: offset %d >= Size %d, returning empty", f.inodeNum, offset, ino.Size) + logging.Debugf("Read inode %d: offset %d >= Size %d, returning empty", f.inodeNum, offset, ino.Size) return fuse.ReadResultData(nil), 0 } @@ -1768,7 +1769,7 @@ func (f *File) Write(ctx context.Context, fh fs.FileHandle, data []byte, off int bytesWritten, err := f.writeInternal(ctx, data, uint64(off)) unlock() if err == errRetryWrite { - log.Printf("[DEBUG] Write inode %d: retry attempt %d due to concurrent modification", + logging.Debugf("Write inode %d: retry attempt %d due to concurrent modification", f.inodeNum, attempt+1) conflictBackoff(attempt) continue @@ -1793,7 +1794,7 @@ func (f *File) waitOutReplay(ctx context.Context, physBlock uint64) error { return syscall.EIO } if err := f.fs.lockMgr.ReleaseExtent(physBlock); err != nil { - log.Printf("[DEBUG] Write inode %d: releasing block %d after the replay wait failed: %v", f.inodeNum, physBlock, err) + logging.Debugf("Write inode %d: releasing block %d after the replay wait failed: %v", f.inodeNum, physBlock, err) } return errRetryWrite } @@ -1803,7 +1804,7 @@ func (f *File) waitOutReplay(ctx context.Context, physBlock uint64) error { func (f *File) stillWritableInPlace(blockNum, physBlock uint64) error { ino, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil { - log.Printf("[DEBUG] Write inode %d: re-read for in-place write failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: re-read for in-place write failed: %v", f.inodeNum, err) return syscall.EIO } ext, off, err := f.fs.extentMgr.Lookup(ino, blockNum) @@ -1811,7 +1812,7 @@ func (f *File) stillWritableInPlace(blockNum, physBlock uint64) error { case errors.Is(err, extent.ErrExtentNotFound), errors.Is(err, extent.ErrStaleTree): return errRetryWrite case err != nil: - log.Printf("[DEBUG] Write inode %d: lookup for in-place write at block %d failed: %v", f.inodeNum, blockNum, err) + logging.Debugf("Write inode %d: lookup for in-place write at block %d failed: %v", f.inodeNum, blockNum, err) return syscall.EIO case ext.PhysicalBlock+off != physBlock: return errRetryWrite @@ -1821,7 +1822,7 @@ func (f *File) stillWritableInPlace(blockNum, physBlock uint64) error { } entry, err := f.fs.refcountReg.GetUnlocked(physBlock) if err != nil { - log.Printf("[DEBUG] Write inode %d: refcount read for block %d failed: %v", f.inodeNum, physBlock, err) + logging.Debugf("Write inode %d: refcount read for block %d failed: %v", f.inodeNum, physBlock, err) return syscall.EIO } if entry.NeedsCoW() { @@ -1861,13 +1862,13 @@ var testHookBeforeFreeingUnused func() func (f *File) allocateBlockLocked(ctx context.Context, ino *inode.Inode, blockNum uint64) (*inode.Inode, *extent.Extent, func(), error) { blocks, err := f.fs.bitmapMgr.Alloc(ctx, 1) if err != nil { - log.Printf("[DEBUG] Write inode %d: allocating block %d failed: %v", f.inodeNum, blockNum, err) + logging.Debugf("Write inode %d: allocating block %d failed: %v", f.inodeNum, blockNum, err) return nil, nil, nil, allocErrno(err) } ext := &extent.Extent{LogicalBlock: blockNum, PhysicalBlock: blocks[0], Length: 1, Flags: extent.FlagAllocated} treeCtx, releaseTree, err := f.fs.extentMgr.ReserveTreeBlocks(ctx, ino) if err != nil { - log.Printf("[DEBUG] Write inode %d: reserving extent-tree blocks failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: reserving extent-tree blocks failed: %v", f.inodeNum, err) f.fs.bitmapMgr.Free(ctx, ext.PhysicalBlock) return nil, nil, nil, allocErrno(err) } @@ -1956,7 +1957,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // Read initial inode state ino, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil { - log.Printf("[DEBUG] Write inode %d: Read failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: Read failed: %v", f.inodeNum, err) return 0, syscall.EIO } // persistAndUnlock writes changes made under the inode lock and drops it, so @@ -1968,7 +1969,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i if inoDirty { copiesMayBeOnDisk = true if err := f.fs.inodeTbl.WriteWithLockHeld(ino); err != nil { - log.Printf("[DEBUG] Write inode %d: persisting inode before unlocking failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: persisting inode before unlocking failed: %v", f.inodeNum, err) releaseInodeLock() f.fs.extentMgr.DiscardPending(ctx, ino) return retryOnTreeConflict(err) @@ -1989,7 +1990,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i if !copiesMayBeOnDisk { for _, cow := range cowDone[committedCoW:] { if err := f.fs.extentMgr.FreePhysicalBlock(ctx, cow.NewPhysBlock); err != nil { - log.Printf("[DEBUG] Write inode %d: freeing unused copy %d failed: %v", f.inodeNum, cow.NewPhysBlock, err) + logging.Debugf("Write inode %d: freeing unused copy %d failed: %v", f.inodeNum, cow.NewPhysBlock, err) } } } @@ -2000,7 +2001,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // older build moves out of its SFB slot before the write. if ino.IsSFBBacked() { if err := f.migrateToExtent(ctx, ino); err != nil { - log.Printf("[DEBUG] Write inode %d: SFB migration failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: SFB migration failed: %v", f.inodeNum, err) if err == errRetryWrite { return 0, err } @@ -2021,7 +2022,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // Try to find existing extent first (without lock for read-only check) ext, extOffset, err := f.fs.extentMgr.Lookup(ino, blockNum) if err != nil && err != extent.ErrExtentNotFound { - log.Printf("[DEBUG] Write inode %d: extent lookup at block %d failed: %v", f.inodeNum, blockNum, err) + logging.Debugf("Write inode %d: extent lookup at block %d failed: %v", f.inodeNum, blockNum, err) if errors.Is(err, extent.ErrStaleTree) { return 0, errRetryWrite } @@ -2034,7 +2035,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // Need to allocate - must acquire lock first if !lockHeld && f.fs.lockMgr != nil { if err := f.fs.lockMgr.Acquire(ctx, lock.LockTypeInode, uint64(f.inodeNum), lock.LockModeExclusive); err != nil { - log.Printf("[DEBUG] Write inode %d: failed to acquire lock: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: failed to acquire lock: %v", f.inodeNum, err) return 0, syscall.EIO } lockHeld = true @@ -2042,13 +2043,13 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // Re-read inode after acquiring lock to detect concurrent changes freshIno, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil { - log.Printf("[DEBUG] Write inode %d: re-read failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: re-read failed: %v", f.inodeNum, err) return 0, syscall.EIO } // Check if MetaGeneration changed (concurrent modification) if freshIno.MetaGeneration != ino.MetaGeneration { - log.Printf("[DEBUG] Write inode %d: MetaGeneration changed %d -> %d, retrying", + logging.Debugf("Write inode %d: MetaGeneration changed %d -> %d, retrying", f.inodeNum, ino.MetaGeneration, freshIno.MetaGeneration) return 0, errRetryWrite } @@ -2070,7 +2071,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i if f.fs.lockMgr != nil { freshIno, newExt, releaseTree, allocErr := f.allocateBlockLocked(ctx, ino, blockNum) if allocErr != nil { - log.Printf("[DEBUG] Write inode %d: allocating block %d failed: %v", f.inodeNum, blockNum, allocErr) + logging.Debugf("Write inode %d: allocating block %d failed: %v", f.inodeNum, blockNum, allocErr) return 0, allocErr } lockHeld = true @@ -2083,7 +2084,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // Allocate block (acquires bitmap lock internally, then releases it) ext, err = f.fs.extentMgr.AllocateBlocks(ctx, ino, blockNum, 1) if err != nil { - log.Printf("[DEBUG] Write inode %d: AllocateBlocks at block %d failed: %v", f.inodeNum, blockNum, err) + logging.Debugf("Write inode %d: AllocateBlocks at block %d failed: %v", f.inodeNum, blockNum, err) switch { case errors.Is(err, extent.ErrExtentOverlap), errors.Is(err, extent.ErrStaleTree): // Concurrent allocation by another node @@ -2120,7 +2121,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i } needsCoW, cowErr := f.fs.extentMgr.NeedsCoW(ctx, blockExtent(ext, extOffset), f.fs.refcountReg) if cowErr != nil { - log.Printf("[DEBUG] Write inode %d: CoW check failed for block %d: %v", + logging.Debugf("Write inode %d: CoW check failed for block %d: %v", f.inodeNum, ext.PhysicalBlock, cowErr) return 0, syscall.EIO } @@ -2129,7 +2130,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // the inode lock; the data is copied under the lock, after any in-place write has landed. cowResult, cowErr := f.fs.extentMgr.MapCopy(ctx, ino, blockNum) if cowErr != nil { - log.Printf("[DEBUG] Write inode %d: CoW failed for block %d: %v", + logging.Debugf("Write inode %d: CoW failed for block %d: %v", f.inodeNum, ext.PhysicalBlock, cowErr) f.fs.extentMgr.DiscardPending(ctx, ino) // The unlocked copy may point at tree nodes another node has since rewritten. @@ -2149,29 +2150,29 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // Mapped on an unlocked copy: keep it only if no node wrote the inode meanwhile. if f.fs.lockMgr != nil { if err := f.fs.lockMgr.Acquire(ctx, lock.LockTypeInode, uint64(f.inodeNum), lock.LockModeExclusive); err != nil { - log.Printf("[DEBUG] Write inode %d: failed to acquire lock for CoW: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: failed to acquire lock for CoW: %v", f.inodeNum, err) dropCopy() return 0, syscall.EIO } lockHeld = true freshIno, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil || freshIno.MetaGeneration != ino.MetaGeneration { - log.Printf("[DEBUG] Write inode %d: inode changed during CoW (err=%v)", f.inodeNum, err) + logging.Debugf("Write inode %d: inode changed during CoW (err=%v)", f.inodeNum, err) dropCopy() return 0, errRetryWrite } } if err := f.fs.extentMgr.CopyBlockData(cowResult, f.fs.dataOffset); err != nil { - log.Printf("[DEBUG] Write inode %d: copying block %d failed: %v", f.inodeNum, cowResult.OldPhysBlock, err) + logging.Debugf("Write inode %d: copying block %d failed: %v", f.inodeNum, cowResult.OldPhysBlock, err) dropCopy() return 0, syscall.EIO } inoDirty = true cowDone = append(cowDone, cowResult) - log.Printf("[DEBUG] Write inode %d: CoW completed, old block %d -> new block %d", + logging.Debugf("Write inode %d: CoW completed, old block %d -> new block %d", f.inodeNum, cowResult.OldPhysBlock, cowResult.NewPhysBlock) if ext, extOffset, cowErr = f.fs.extentMgr.Lookup(ino, blockNum); cowErr != nil { - log.Printf("[DEBUG] Write inode %d: extent lookup after CoW failed: %v", f.inodeNum, cowErr) + logging.Debugf("Write inode %d: extent lookup after CoW failed: %v", f.inodeNum, cowErr) return 0, syscall.EIO } } @@ -2188,7 +2189,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i } cleared, clearErr := f.fs.extentMgr.ClearUnwritten(ctx, ino, blockNum) if clearErr != nil { - log.Printf("[DEBUG] Write inode %d: clearing Unwritten flag at block %d failed: %v", + logging.Debugf("Write inode %d: clearing Unwritten flag at block %d failed: %v", f.inodeNum, blockNum, clearErr) f.fs.extentMgr.DiscardPending(ctx, ino) if errors.Is(clearErr, extent.ErrStaleTree) { @@ -2197,14 +2198,14 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i return 0, syscall.EIO } if err := f.fs.lockMgr.Acquire(ctx, lock.LockTypeInode, uint64(f.inodeNum), lock.LockModeExclusive); err != nil { - log.Printf("[DEBUG] Write inode %d: failed to acquire lock for unwritten: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: failed to acquire lock for unwritten: %v", f.inodeNum, err) f.fs.extentMgr.DiscardPending(ctx, ino) return 0, syscall.EIO } lockHeld = true freshIno, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil || freshIno.MetaGeneration != ino.MetaGeneration { - log.Printf("[DEBUG] Write inode %d: inode changed during unwritten clear (err=%v)", f.inodeNum, err) + logging.Debugf("Write inode %d: inode changed during unwritten clear (err=%v)", f.inodeNum, err) releaseInodeLock() f.fs.extentMgr.DiscardPending(ctx, ino) return 0, errRetryWrite @@ -2212,13 +2213,13 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i inoDirty = inoDirty || cleared zeroFirst = cleared if ext, extOffset, err = f.fs.extentMgr.Lookup(ino, blockNum); err != nil { - log.Printf("[DEBUG] Write inode %d: lookup after unwritten clear failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: lookup after unwritten clear failed: %v", f.inodeNum, err) return 0, syscall.EIO } } else if ext.Flags&extent.FlagUnwritten != 0 { cleared, clearErr := f.fs.extentMgr.ClearUnwritten(ctx, ino, blockNum) if clearErr != nil { - log.Printf("[DEBUG] Write inode %d: clearing Unwritten flag at block %d failed: %v", + logging.Debugf("Write inode %d: clearing Unwritten flag at block %d failed: %v", f.inodeNum, blockNum, clearErr) return 0, syscall.EIO } @@ -2248,7 +2249,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i releaseInPlaceLock := func() { if inPlaceLockHeld { if err := f.fs.lockMgr.Release(lock.LockTypeInode, uint64(f.inodeNum)); err != nil { - log.Printf("[DEBUG] Write inode %d: releasing the shared inode lock failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: releasing the shared inode lock failed: %v", f.inodeNum, err) } inPlaceLockHeld = false } @@ -2260,7 +2261,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i mode = lock.LockModeExclusive } if err := f.fs.lockMgr.Acquire(ctx, lock.LockTypeInode, uint64(f.inodeNum), mode); err != nil { - log.Printf("[DEBUG] Write inode %d: failed to acquire inode lock: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: failed to acquire inode lock: %v", f.inodeNum, err) return 0, syscall.EIO } inPlaceLockHeld = true @@ -2285,7 +2286,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i extentCtx = lock.Retryable(ctx) } if err := f.fs.lockMgr.AcquireExtent(extentCtx, physBlock, lock.LockModeExclusive); err != nil { - log.Printf("[DEBUG] Write inode %d: failed to acquire extent lock for block %d: %v", + logging.Debugf("Write inode %d: failed to acquire extent lock for block %d: %v", f.inodeNum, physBlock, err) releaseInPlaceLock() if retryable && errors.Is(err, lock.ErrReplayBlocked) { @@ -2302,7 +2303,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // The rest of a block first written here would otherwise expose old disk contents. if zeroFirst { if err := f.fs.zeroBlock(physBlock); err != nil { - log.Printf("[DEBUG] Write inode %d: zeroing block %d failed: %v", f.inodeNum, physBlock, err) + logging.Debugf("Write inode %d: zeroing block %d failed: %v", f.inodeNum, physBlock, err) if extentLockHeld { f.fs.lockMgr.ReleaseExtent(physBlock) } @@ -2340,7 +2341,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // Write back if _, err := f.fs.dev.WriteAt(buf.Bytes(), alignedOffset); err != nil { - log.Printf("[DEBUG] Write inode %d: WriteAt offset %d failed: %v", f.inodeNum, alignedOffset, err) + logging.Debugf("Write inode %d: WriteAt offset %d failed: %v", f.inodeNum, alignedOffset, err) storage.PutBuffer(buf) if extentLockHeld { f.fs.lockMgr.ReleaseExtent(physBlock) @@ -2378,7 +2379,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i acquiredLockForSize := false if !lockHeld && f.fs.lockMgr != nil { if err := f.fs.lockMgr.Acquire(ctx, lock.LockTypeInode, uint64(f.inodeNum), lock.LockModeExclusive); err != nil { - log.Printf("[DEBUG] Write inode %d: failed to acquire lock for size update: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: failed to acquire lock for size update: %v", f.inodeNum, err) return 0, syscall.EIO } acquiredLockForSize = true @@ -2388,7 +2389,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // Re-read the inode while holding the lock to get the freshest size freshIno, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil { - log.Printf("[DEBUG] Write inode %d: re-read inode failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: re-read inode failed: %v", f.inodeNum, err) return 0, syscall.EIO } // ino was read before the lock, so it can miss another node's extents. @@ -2417,7 +2418,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i MetaGeneration: ino.MetaGeneration, Size: newSize, }); err != nil { - log.Printf("[DEBUG] Write inode %d: LogExtentAdd failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: LogExtentAdd failed: %v", f.inodeNum, err) return 0, syscall.EIO } } @@ -2431,7 +2432,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i InodeNum: uint64(f.inodeNum), UpdateMask: mask, }); err != nil { - log.Printf("[DEBUG] Write inode %d: LogInodeUpdate failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: LogInodeUpdate failed: %v", f.inodeNum, err) return 0, syscall.EIO } @@ -2439,7 +2440,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i var commitErr error intentSeq, commitErr = f.fs.journal.Commit(txn) if commitErr != nil { - log.Printf("[DEBUG] Write inode %d: journal commit failed: %v", f.inodeNum, commitErr) + logging.Debugf("Write inode %d: journal commit failed: %v", f.inodeNum, commitErr) return 0, syscall.EIO } defer f.fs.closeIntent(intentSeq) @@ -2495,7 +2496,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i f.fs.extentMgr.DiscardPending(ctx, ino) if f.fs.journal != nil && intentSeq > 0 { if err := f.fs.journal.CompleteIntent(intentSeq); err != nil { - log.Printf("[DEBUG] Write inode %d: CompleteIntent after a failed write: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: CompleteIntent after a failed write: %v", f.inodeNum, err) if conflict { return errRetryWrite } @@ -2513,7 +2514,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i } for _, b := range unusedBlocks() { if err := f.fs.extentMgr.FreePhysicalBlock(ctx, b); err != nil { - log.Printf("[DEBUG] Write inode %d: freeing block %d after tree conflict failed: %v", f.inodeNum, b, err) + logging.Debugf("Write inode %d: freeing block %d after tree conflict failed: %v", f.inodeNum, b, err) } } // The retry finds these blocks already copied, so release the old ones now. @@ -2531,12 +2532,12 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i copiesMayBeOnDisk = true if lockHeld { if err := f.fs.inodeTbl.WriteWithLockHeld(ino); err != nil { - log.Printf("[DEBUG] Write inode %d: WriteWithLockHeld failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: WriteWithLockHeld failed: %v", f.inodeNum, err) return 0, writeFailed(err) } } else { if err := f.fs.inodeTbl.Write(ctx, ino); err != nil { - log.Printf("[DEBUG] Write inode %d: final inodeTbl.Write failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: final inodeTbl.Write failed: %v", f.inodeNum, err) return 0, writeFailed(err) } } @@ -2561,7 +2562,7 @@ func (f *File) writeInternal(ctx context.Context, data []byte, offset uint64) (i // On recovery, this prevents replaying already-completed operations. if f.fs.journal != nil && intentSeq > 0 { if err := f.fs.journal.CompleteIntent(intentSeq); err != nil { - log.Printf("[DEBUG] Write inode %d: CompleteIntent failed: %v", f.inodeNum, err) + logging.Debugf("Write inode %d: CompleteIntent failed: %v", f.inodeNum, err) } } @@ -2707,7 +2708,7 @@ func (f *File) migrateToExtent(ctx context.Context, ino *inode.Inode) error { // === WAL: Mark intent complete === if f.fs.journal != nil && intentSeq > 0 { if err := f.fs.journal.CompleteIntent(intentSeq); err != nil { - log.Printf("[DEBUG] migrateToExtent inode %d: CompleteIntent failed: %v", f.inodeNum, err) + logging.Debugf("migrateToExtent inode %d: CompleteIntent failed: %v", f.inodeNum, err) } } @@ -2750,7 +2751,7 @@ func (f *File) setattr(ctx context.Context, fh fs.FileHandle, in *fuse.SetAttrIn if _, resizing := in.GetSize(); resizing && ino.IsSFBBacked() { if err := f.migrateToExtent(ctx, ino); err != nil { - log.Printf("[DEBUG] Setattr inode %d: SFB migration failed: %v", f.inodeNum, err) + logging.Debugf("Setattr inode %d: SFB migration failed: %v", f.inodeNum, err) if err == errRetryWrite { return 0, err } @@ -2897,7 +2898,7 @@ func (f *File) setattr(ctx context.Context, fh fs.FileHandle, in *fuse.SetAttrIn // === WAL: Mark intent complete === if f.fs.journal != nil && intentSeq > 0 { if err := f.fs.journal.CompleteIntent(intentSeq); err != nil { - log.Printf("[DEBUG] Setattr inode %d: CompleteIntent failed: %v", f.inodeNum, err) + logging.Debugf("Setattr inode %d: CompleteIntent failed: %v", f.inodeNum, err) } } @@ -2921,7 +2922,7 @@ func (f *File) Allocate(ctx context.Context, fh fs.FileHandle, off uint64, size const FALLOC_FL_KEEP_SIZE = 0x01 if mode != 0 && mode != FALLOC_FL_KEEP_SIZE { // PUNCH_HOLE and ZERO_RANGE not yet implemented - log.Printf("[DEBUG] Allocate inode %d: unsupported mode %d", f.inodeNum, mode) + logging.Debugf("Allocate inode %d: unsupported mode %d", f.inodeNum, mode) return syscall.EOPNOTSUPP } @@ -2945,7 +2946,7 @@ func (f *File) allocate(ctx context.Context, off uint64, size uint64, mode uint3 // Read current inode ino, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil { - log.Printf("[DEBUG] Allocate inode %d: read failed: %v", f.inodeNum, err) + logging.Debugf("Allocate inode %d: read failed: %v", f.inodeNum, err) return syscall.EIO, nil } @@ -2972,7 +2973,7 @@ func (f *File) allocate(ctx context.Context, off uint64, size uint64, mode uint3 endBlock := (endOffset + BlockSize - 1) / BlockSize // Round up totalBlocks := endBlock - startBlock - log.Printf("[DEBUG] Allocate inode %d: offset=%d length=%d blocks=%d-%d (total=%d)", + logging.Debugf("Allocate inode %d: offset=%d length=%d blocks=%d-%d (total=%d)", f.inodeNum, off, size, startBlock, endBlock-1, totalBlocks) // Find contiguous ranges of unallocated blocks and allocate each as a single extent. @@ -2987,7 +2988,7 @@ func (f *File) allocate(ctx context.Context, off uint64, size uint64, mode uint3 continue } if lookupErr != extent.ErrExtentNotFound { - log.Printf("[DEBUG] Allocate inode %d: extent lookup at block %d failed: %v", f.inodeNum, blockNum, lookupErr) + logging.Debugf("Allocate inode %d: extent lookup at block %d failed: %v", f.inodeNum, blockNum, lookupErr) return syscall.EIO, nil } @@ -3003,13 +3004,13 @@ func (f *File) allocate(ctx context.Context, off uint64, size uint64, mode uint3 } rangeCount := rangeEnd - rangeStart - log.Printf("[DEBUG] Allocate inode %d: allocating contiguous range blocks=%d-%d (count=%d)", + logging.Debugf("Allocate inode %d: allocating contiguous range blocks=%d-%d (count=%d)", f.inodeNum, rangeStart, rangeEnd-1, rangeCount) // Allocate the entire contiguous range as a single unwritten extent _, allocErr := f.fs.extentMgr.AllocateUnwritten(ctx, ino, rangeStart, rangeCount) if allocErr != nil { - log.Printf("[DEBUG] Allocate inode %d: AllocateUnwritten blocks %d-%d failed: %v", + logging.Debugf("Allocate inode %d: AllocateUnwritten blocks %d-%d failed: %v", f.inodeNum, rangeStart, rangeEnd-1, allocErr) if errors.Is(allocErr, extent.ErrExtentTreeFull) { return syscall.EFBIG, nil @@ -3028,7 +3029,7 @@ func (f *File) allocate(ctx context.Context, off uint64, size uint64, mode uint3 // Acquire inode lock for metadata update if f.fs.lockMgr != nil { if err := f.fs.lockMgr.Acquire(ctx, lock.LockTypeInode, uint64(f.inodeNum), lock.LockModeExclusive); err != nil { - log.Printf("[DEBUG] Allocate inode %d: lock failed: %v", f.inodeNum, err) + logging.Debugf("Allocate inode %d: lock failed: %v", f.inodeNum, err) return syscall.EIO, nil } defer f.fs.lockMgr.Release(lock.LockTypeInode, uint64(f.inodeNum)) @@ -3036,7 +3037,7 @@ func (f *File) allocate(ctx context.Context, off uint64, size uint64, mode uint3 // ino holds the new extents, so keep it only if no node wrote the inode since it was read. fresh, err := f.fs.inodeTbl.Read(f.inodeNum) if err != nil { - log.Printf("[DEBUG] Allocate inode %d: re-read failed: %v", f.inodeNum, err) + logging.Debugf("Allocate inode %d: re-read failed: %v", f.inodeNum, err) return syscall.EIO, nil } if fresh.MetaGeneration != ino.MetaGeneration { @@ -3060,7 +3061,7 @@ func (f *File) allocate(ctx context.Context, off uint64, size uint64, mode uint3 // Write updated inode if err := f.fs.inodeTbl.Write(ctx, ino); err != nil { - log.Printf("[DEBUG] Allocate inode %d: write inode failed: %v", f.inodeNum, err) + logging.Debugf("Allocate inode %d: write inode failed: %v", f.inodeNum, err) if errors.Is(err, extent.ErrTreeConflict) { return 0, err } @@ -3069,7 +3070,7 @@ func (f *File) allocate(ctx context.Context, off uint64, size uint64, mode uint3 } committed = true - log.Printf("[DEBUG] Allocate inode %d: success, AllocatedBlocks=%d WrittenBlocks=%d Size=%d", + logging.Debugf("Allocate inode %d: success, AllocatedBlocks=%d WrittenBlocks=%d Size=%d", f.inodeNum, ino.AllocatedBlocks, ino.WrittenBlocks, ino.Size) return 0, nil @@ -3099,7 +3100,7 @@ func (f *File) Setxattr(ctx context.Context, attr string, data []byte, flags uin // Getxattr gets an extended attribute from the file. func (f *File) Getxattr(ctx context.Context, attr string, dest []byte) (uint32, syscall.Errno) { ctx = detach(ctx) - log.Printf("[DEBUG] File.Getxattr: inode=%d attr=%s", f.inodeNum, attr) + logging.Debugf("File.Getxattr: inode=%d attr=%s", f.inodeNum, attr) switch attr { case "user.snapshots": @@ -3113,7 +3114,7 @@ func (f *File) Getxattr(ctx context.Context, attr string, dest []byte) (uint32, // Used for snapshot deletion via user.snapshot. pattern. func (f *File) Removexattr(ctx context.Context, attr string) syscall.Errno { ctx = detach(ctx) - log.Printf("[DEBUG] File.Removexattr: inode=%d attr=%s", f.inodeNum, attr) + logging.Debugf("File.Removexattr: inode=%d attr=%s", f.inodeNum, attr) // Handle user.snapshot. for snapshot deletion const snapshotPrefix = "user.snapshot." @@ -3310,7 +3311,7 @@ func (f *File) sharableInode(ctx context.Context) (*inode.Inode, error) { // hidden inode that shares the file's blocks as they are now. func (f *File) createSnapshot(ctx context.Context, name string) syscall.Errno { if f.fs.snapshotReg == nil || f.fs.snapshotWkr == nil { - log.Printf("[DEBUG] createSnapshot: snapshot subsystem not initialized") + logging.Debugf("createSnapshot: snapshot subsystem not initialized") return syscall.ENOTSUP } if len(name) >= snapshot.MaxSnapshotNameLen { diff --git a/internal/lock/lock.go b/internal/lock/lock.go index 74ff1ed3..6f796bdc 100644 --- a/internal/lock/lock.go +++ b/internal/lock/lock.go @@ -43,6 +43,7 @@ import ( "sync/atomic" "time" + "github.com/supporttools/GoClusterFS/internal/logging" "github.com/supporttools/GoClusterFS/internal/storage" ) @@ -430,11 +431,11 @@ func (m *Manager) Acquire(ctx context.Context, lockType LockType, resourceID uin tryStart := time.Now() held, added, by, err := m.attemptAcquire(key, recordID, mode, gid, time.Since(startTime)) if err != nil { - log.Printf("[DEBUG] lock.Acquire: type=%d res=%d - attempt error: %v", lockType, resourceID, err) + logging.Debugf("lock.Acquire: type=%d res=%d - attempt error: %v", lockType, resourceID, err) return err } if held { - log.Printf("[DEBUG] lock.Acquire: type=%d res=%d - acquired (retries=%d, dur=%v, gid=%d)", lockType, resourceID, retryCount, time.Since(startTime), gid) + logging.Debugf("lock.Acquire: type=%d res=%d - acquired (retries=%d, dur=%v, gid=%d)", lockType, resourceID, retryCount, time.Since(startTime), gid) // Trigger cache invalidation callback for exclusive locks. // We do this before returning so the caller reads fresh data. @@ -659,7 +660,7 @@ func (m *Manager) Release(lockType LockType, resourceID uint64) error { m.mu.Lock() held, ok := m.heldLocks[key] if !ok { - log.Printf("[DEBUG] lock.Release: type=%d res=%d gid=%d - NOT FOUND in heldLocks!", lockType, resourceID, gid) + logging.Debugf("lock.Release: type=%d res=%d gid=%d - NOT FOUND in heldLocks!", lockType, resourceID, gid) m.mu.Unlock() return ErrNotLockOwner } @@ -682,7 +683,7 @@ func (m *Manager) Release(lockType LockType, resourceID uint64) error { err := m.dropRecordKey(recordID) if err != nil { - log.Printf("[DEBUG] lock.Release: type=%d res=%d - release error: %v", lockType, resourceID, err) + logging.Debugf("lock.Release: type=%d res=%d - release error: %v", lockType, resourceID, err) } return err } @@ -734,7 +735,7 @@ func (m *Manager) expireCached(recordID uint64, gen uint64) { m.mu.Unlock() if err := m.releaseRecord(recordID); err != nil { - log.Printf("[DEBUG] lock: releasing cached record %d: %v", recordID, err) + logging.Debugf("lock: releasing cached record %d: %v", recordID, err) } } diff --git a/internal/logging/guard_test.go b/internal/logging/guard_test.go new file mode 100644 index 00000000..07f20657 --- /dev/null +++ b/internal/logging/guard_test.go @@ -0,0 +1,40 @@ +package logging + +import ( + "io/fs" + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +var ungatedDebug = regexp.MustCompile(`log\.Print(f|ln)?\("\[DEBUG\]`) + +func TestDebugLinesGoThroughDebugf(t *testing.T) { + root := filepath.Join("..", "..") + err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() && (d.Name() == ".git" || d.Name() == "node_modules") { + return filepath.SkipDir + } + if d.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") { + return nil + } + data, err := os.ReadFile(path) + if err != nil { + return err + } + for i, line := range strings.Split(string(data), "\n") { + if ungatedDebug.MatchString(line) { + t.Errorf("%s:%d logs [DEBUG] without -debug; use logging.Debugf", path, i+1) + } + } + return nil + }) + if err != nil { + t.Fatal(err) + } +} diff --git a/internal/logging/logging.go b/internal/logging/logging.go new file mode 100644 index 00000000..d02dfeba --- /dev/null +++ b/internal/logging/logging.go @@ -0,0 +1,21 @@ +// Package logging holds the -debug switch that goclusterfs-mount, +// goclusterfs-fsck and gfsctl share. +package logging + +import ( + "fmt" + "log" + "sync/atomic" +) + +var debug atomic.Bool + +// SetDebug turns Debugf output on or off. +func SetDebug(on bool) { debug.Store(on) } + +// Debugf logs like log.Printf with a [DEBUG] prefix, and only after SetDebug(true). +func Debugf(format string, args ...any) { + if debug.Load() { + _ = log.Output(2, "[DEBUG] "+fmt.Sprintf(format, args...)) + } +} diff --git a/internal/logging/logging_test.go b/internal/logging/logging_test.go new file mode 100644 index 00000000..89b6ebb7 --- /dev/null +++ b/internal/logging/logging_test.go @@ -0,0 +1,23 @@ +package logging + +import ( + "bytes" + "log" + "strings" + "testing" +) + +func TestDebugfPrintsOnlyWhenOn(t *testing.T) { + var out bytes.Buffer + log.SetOutput(&out) + defer log.SetOutput(log.Writer()) + defer SetDebug(false) + + Debugf("hidden %d", 1) + SetDebug(true) + Debugf("shown %d", 2) + + if got := out.String(); strings.Contains(got, "hidden") || !strings.Contains(got, "[DEBUG] shown 2") { + t.Errorf("log output = %q", got) + } +} diff --git a/internal/node/registry.go b/internal/node/registry.go index 4a56ce06..025c7544 100644 --- a/internal/node/registry.go +++ b/internal/node/registry.go @@ -15,6 +15,7 @@ import ( "fmt" "log" "math" + "math/bits" "os" "sync" "sync/atomic" @@ -129,6 +130,7 @@ type Registry struct { recoveryHandler RecoveryHandler failureCheckInterval time.Duration // How often to scan for failed nodes recoveryInProgress map[int]bool // Slots currently being recovered + replaying atomic.Uint64 // Bit i: this node is replaying slot i's journal } // Errors @@ -859,7 +861,7 @@ func (r *Registry) detectAndRecoverFailedNodes() { continue // Another node beat us to it } done := r.ownerDead(info.UUID, i) - err = handler(info) + err = r.replay(handler, i, info) done(err == nil) if err != nil { log.Printf("[Recovery] Replaying slot %d (node %x) failed: %v; any node retries after %v", @@ -1057,6 +1059,22 @@ func (r *Registry) GetFailedNodes() ([]*FailedNodeInfo, error) { return failed, nil } +func (r *Registry) replay(handler RecoveryHandler, slotIdx int, info *FailedNodeInfo) error { + bit := uint64(1) << slotIdx + r.replaying.Or(bit) + defer r.replaying.And(^bit) + return handler(info) +} + +// Replaying returns the slots whose journal this node is replaying now. +func (r *Registry) Replaying() []int { + var slots []int + for mask := r.replaying.Load(); mask != 0; mask &= mask - 1 { + slots = append(slots, bits.TrailingZeros64(mask)) + } + return slots +} + // RecoverFailedNode manually triggers recovery for a specific node. // Returns error if the node is not in a failed state or recovery fails. func (r *Registry) RecoverFailedNode(slotIdx int) error { @@ -1098,7 +1116,7 @@ func (r *Registry) RecoverFailedNode(slotIdx int) error { return err } - if err := handler(info); err != nil { + if err := r.replay(handler, slotIdx, info); err != nil { return fmt.Errorf("recovery handler failed: %w", err) } diff --git a/internal/node/replaying_test.go b/internal/node/replaying_test.go new file mode 100644 index 00000000..1eccd7f6 --- /dev/null +++ b/internal/node/replaying_test.go @@ -0,0 +1,26 @@ +package node + +import ( + "errors" + "reflect" + "testing" +) + +func TestReplayingListsSlotsWhileTheirHandlerRuns(t *testing.T) { + var r Registry + var during []int + handler := func(*FailedNodeInfo) error { + during = r.Replaying() + return errors.New("replay failed") + } + r.replaying.Or(1 << 63) + if err := r.replay(handler, 3, &FailedNodeInfo{}); err == nil { + t.Fatal("replay hid the handler's error") + } + if want := []int{3, 63}; !reflect.DeepEqual(during, want) { + t.Errorf("Replaying during the handler = %v, want %v", during, want) + } + if got := r.Replaying(); !reflect.DeepEqual(got, []int{63}) { + t.Errorf("Replaying after the handler = %v, want [63]", got) + } +} diff --git a/internal/storage/device.go b/internal/storage/device.go index b1b4e6c8..3a2d8647 100644 --- a/internal/storage/device.go +++ b/internal/storage/device.go @@ -7,6 +7,7 @@ package storage import ( + "errors" "fmt" "io" "os" @@ -50,6 +51,7 @@ type Device struct { file *os.File path string size int64 + lastErr atomic.Pointer[error] } // shardIndex returns the shard index for the given offset. @@ -200,7 +202,9 @@ func (d *Device) readAt(buf []byte, offset int64, sharded bool) (int, error) { defer d.unlockIORange(startShard, endShard) } - return d.file.ReadAt(buf, offset) + n, err := d.file.ReadAt(buf, offset) + d.record(err) + return n, err } // WriteAt writes data to the device at the specified offset. @@ -236,7 +240,9 @@ func (d *Device) writeAt(buf []byte, offset int64, sharded bool) (int, error) { defer d.unlockIORange(startShard, endShard) } - return d.file.WriteAt(buf, offset) + n, err := d.file.WriteAt(buf, offset) + d.record(err) + return n, err } // SetWriteGate makes every write call gate with its offset first, and fail @@ -261,7 +267,29 @@ func (d *Device) Sync() error { d.closeMu.RLock() defer d.closeMu.RUnlock() - return d.file.Sync() + err := d.file.Sync() + d.record(err) + return err +} + +// LastError returns the error of the last read, write or sync that failed, +// or nil once a later one succeeds. +func (d *Device) LastError() error { + if err := d.lastErr.Load(); err != nil { + return *err + } + return nil +} + +func (d *Device) record(err error) { + switch { + case err == nil: + if d.lastErr.Load() != nil { + d.lastErr.Store(nil) + } + case !errors.Is(err, io.EOF): + d.lastErr.Store(&err) + } } // Fd returns the file descriptor for use with syscalls. diff --git a/internal/storage/device_test.go b/internal/storage/device_test.go index 8bf5b57b..1ab81752 100644 --- a/internal/storage/device_test.go +++ b/internal/storage/device_test.go @@ -3,10 +3,12 @@ package storage import ( "bytes" "errors" + "io" "os" "path/filepath" "sync" "sync/atomic" + "syscall" "testing" ) @@ -410,3 +412,38 @@ func TestWriteGateRefusesWrites(t *testing.T) { t.Error("a refused write reached the device") } } + +func TestLastErrorHoldsTheLatestFailureUntilASuccess(t *testing.T) { + var d Device + if d.LastError() != nil { + t.Fatal("a new device reports an error") + } + d.record(syscall.EIO) + d.record(io.EOF) + if !errors.Is(d.LastError(), syscall.EIO) { + t.Fatalf("LastError = %v, want EIO; EOF is not a device failure", d.LastError()) + } + d.record(nil) + if d.LastError() != nil { + t.Errorf("LastError after a successful I/O = %v, want nil", d.LastError()) + } +} + +func TestFailedReadIsRecorded(t *testing.T) { + path := filepath.Join(t.TempDir(), "dev.img") + if err := os.WriteFile(path, make([]byte, 2*SectorSize), 0o600); err != nil { + t.Fatal(err) + } + dev, err := Open(path) + if err != nil { + t.Skipf("O_DIRECT not supported: %v", err) + } + buf := NewAlignedBuffer(SectorSize).Bytes() + if _, err := dev.ReadAt(buf, 0); err != nil || dev.LastError() != nil { + t.Fatalf("ReadAt = %v, LastError = %v", err, dev.LastError()) + } + _ = dev.file.Close() + if _, err := dev.ReadAt(buf, 0); err == nil || dev.LastError() == nil { + t.Errorf("ReadAt on a closed file = %v, LastError = %v, want both set", err, dev.LastError()) + } +} diff --git a/tests/integration/shutdown_test.go b/tests/integration/shutdown_test.go new file mode 100644 index 00000000..bc4f20fc --- /dev/null +++ b/tests/integration/shutdown_test.go @@ -0,0 +1,134 @@ +package integration + +import ( + "errors" + "net/http" + "os" + "os/exec" + "path/filepath" + "strings" + "syscall" + "testing" + "time" + + "github.com/supporttools/GoClusterFS/internal/api" +) + +func exitCode(t *testing.T, binary string, args ...string) (int, string) { + t.Helper() + out, err := exec.Command(binary, args...).CombinedOutput() + var exit *exec.ExitError + switch { + case err == nil: + return 0, string(out) + case errors.As(err, &exit): + return exit.ExitCode(), string(out) + } + t.Fatalf("run %s %v: %v", binary, args, err) + return 0, "" +} + +func TestHealthAndSignalShutdown(t *testing.T) { + m := startAPIMount(t, "", 20, "-broadcast=false") + base := "http://" + m.addr + + var health api.HealthResponse + if code := getAPI(t, base+"/health", &health); code != http.StatusOK || health.Status != api.HealthStatusOK { + t.Fatalf("GET /health = %d %+v", code, health) + } + for _, name := range []string{"device", "journal", "recovery", "broadcast"} { + if check, ok := health.Checks[name]; !ok || check.Status != api.HealthStatusOK { + t.Errorf("check %s = %+v (present %v), want ok", name, check, ok) + } + } + if msg := health.Checks["broadcast"].Message; !strings.Contains(msg, "-broadcast=false") { + t.Errorf("broadcast check message = %q, want it to say broadcast is off", msg) + } + + if err := m.WriteFile("held", []byte("data")); err != nil { + t.Fatal(err) + } + held, err := os.Open(m.Path("held")) + if err != nil { + t.Fatal(err) + } + if err := m.MountCmd.Process.Signal(syscall.SIGTERM); err != nil { + t.Fatal(err) + } + exited := make(chan error, 1) + go func() { exited <- m.MountCmd.Wait() }() + if _, ok := waitFor(15*time.Second, func() bool { return strings.Contains(m.log.String(), "Close the files and shells that use it") }); !ok { + held.Close() + t.Fatalf("SIGTERM with an open file did not report a busy unmount:\n%s", m.log.String()) + } + select { + case err := <-exited: + held.Close() + t.Fatalf("mount exited (%v) while a file under it was open", err) + case <-time.After(time.Second): + } + if !m.IsMounted() { + t.Fatal("the busy mount was unmounted") + } + if code := getAPI(t, base+"/health", nil); code != 0 { + t.Errorf("GET /health after SIGTERM = %d, want a refused connection", code) + } + + held.Close() + if err := m.MountCmd.Process.Signal(syscall.SIGTERM); err != nil { + t.Fatal(err) + } + select { + case err := <-exited: + if err != nil { + t.Fatalf("mount exited with %v after the second SIGTERM, want 0\n%s", err, m.log.String()) + } + case <-time.After(30 * time.Second): + t.Fatalf("mount did not exit within 30 s of the second SIGTERM\n%s", m.log.String()) + } + if m.IsMounted() { + t.Error("mount point still mounted after the mount exited") + } + if code, out := runFsck(t, m.ImagePath); code != 0 { + t.Errorf("fsck after the signal shutdown exited %d:\n%s", code, out) + } +} + +func TestCommandsShareExitCodes(t *testing.T) { + mount, err := findBinary() + if err != nil { + t.Fatal(err) + } + fsck := findFsckBinary(t) + gfsctl := filepath.Join(filepath.Dir(mount), "gfsctl") + if _, err := os.Stat(gfsctl); err != nil { + t.Fatalf("gfsctl not found next to %s; run 'make build' first", mount) + } + dir := t.TempDir() + missing := filepath.Join(dir, "missing.img") + + tests := []struct { + binary string + args []string + code int + says string + }{ + {mount, []string{"-h"}, 0, "Exit codes"}, + {mount, []string{"-no-such-flag"}, 16, "flag provided but not defined"}, + {mount, []string{"-device", missing}, 16, "-device and -mount are required"}, + {mount, []string{"-device", missing, "-mount", filepath.Join(dir, "none"), "-broadcast=false", "-api=false"}, 8, "mkdir -p"}, + {mount, []string{"-device", missing, "-mount", dir, "-broadcast=false", "-api=false"}, 8, "check the path"}, + {fsck, []string{"-h"}, 0, "Exit codes"}, + {fsck, []string{"-no-such-flag"}, 16, "flag provided but not defined"}, + {fsck, nil, 16, "give exactly one device"}, + {fsck, []string{missing}, 8, "check the path"}, + {gfsctl, []string{"--no-such-flag"}, 16, "--help"}, + {gfsctl, []string{"--port", "1", "health"}, 8, "is goclusterfs-mount running"}, + } + for _, tt := range tests { + code, out := exitCode(t, tt.binary, tt.args...) + if code != tt.code || !strings.Contains(out, tt.says) { + t.Errorf("%s %q exited %d, want %d with %q:\n%s", filepath.Base(tt.binary), tt.args, code, tt.code, tt.says, out) + } + } +} diff --git a/tests/test-manifest.yaml b/tests/test-manifest.yaml index fe6bed47..6fbccd0d 100644 --- a/tests/test-manifest.yaml +++ b/tests/test-manifest.yaml @@ -3560,6 +3560,99 @@ scenarios: environments: go: internal/api/prometheus_test.go::TestMetricsEndpoint_ExportsCapacity + - id: api-health-signal-shutdown + name: /health checks and SIGTERM shutdown + description: A live mount reports device, journal, recovery and broadcast checks on /health. SIGTERM with a file open under the mount stops the API, reports the busy unmount and keeps the mount; a second SIGTERM after the file is closed unmounts, exits 0 and leaves a clean fsck + priority: high + environments: + go: shutdown_test.go::TestHealthAndSignalShutdown + docker: TODO - same reason as api-health-loopback + vm: TODO - same reason as api-health-loopback + + - id: commands-share-exit-codes + name: mount, fsck and gfsctl share exit codes + description: The three binaries exit 0 after -h, 16 on a bad flag or missing argument, and 8 when the device, mount point or API cannot be reached, and say what to do next + priority: go_only + environments: + go: shutdown_test.go::TestCommandsShareExitCodes + + - id: api-stop-waits-for-requests + name: API Stop lets requests in flight finish + description: Stop closes the listener at once, waits for a request in flight, and leaves no server goroutines + priority: go_only + environments: + go: internal/api/lifecycle_test.go::TestStopLetsRequestsInFlightFinish + + - id: api-stop-cancels-after-timeout + name: API Stop cancels requests after the shutdown timeout + description: A request that outlasts ShutdownTimeout has its context cancelled, Stop returns ErrShutdownTimeout only after the handler returns, and no server goroutines are left + priority: go_only + environments: + go: internal/api/lifecycle_test.go::TestStopCancelsRequestsThatOutlastTheTimeout + + - id: api-stop-idle + name: API Stop on an idle server + description: Stop is idempotent, refuses new connections and leaves no server goroutines + priority: go_only + environments: + go: internal/api/lifecycle_test.go::TestStopIdleServer + + - id: mount-health-degraded-vs-down + name: /health separates degraded from down + description: A peer journal replay or broadcast trouble reports degraded with 200; a failing device or corrupt journal reports unhealthy with 503; -journal=false and -broadcast=false stay ok + priority: go_only + environments: + go: cmd/mount/health_test.go::TestHealthChecksSeparateDegradedFromDown + + - id: storage-last-error + name: Device remembers its last I/O failure + description: LastError holds the latest failed read, write or sync until one succeeds, and ignores EOF + priority: go_only + environments: + go: internal/storage/device_test.go::TestLastErrorHoldsTheLatestFailureUntilASuccess + + - id: node-replaying-slots + name: Registry lists the journals it replays + description: Replaying lists a slot while its recovery handler runs and drops it afterwards, even when the replay fails + priority: go_only + environments: + go: internal/node/replaying_test.go::TestReplayingListsSlotsWhileTheirHandlerRuns + + - id: broadcast-problem + name: Broadcast reports why peers miss invalidations + description: Problem names a stopped transport, a failed send, and missing multicast without unicast peers, and is empty when broadcast is off or working + priority: go_only + environments: + go: internal/cache/broadcast/problem_test.go::TestProblemSaysWhyPeersMissInvalidations + + - id: cli-parse-flags + name: Shared flag parsing exit codes + description: -h exits 0 and a bad flag or value exits 16; -debug and --debug both parse + priority: go_only + environments: + go: internal/cli/cli_test.go::TestParseFlagsExitCodes + + - id: logging-debug-switch + name: Debug lines only with -debug + description: Debugf prints [DEBUG] lines only after SetDebug(true) + priority: go_only + environments: + go: internal/logging/logging_test.go::TestDebugfPrintsOnlyWhenOn + + - id: gfsctl-exit-codes + name: gfsctl exit codes and hints + description: health exits 0, 1 or 4 for ok, degraded and unhealthy (decoding the 503 body), API and connection errors exit 8 with a hint, usage errors exit 16 with a --help hint + priority: go_only + environments: + go: cmd/gfsctl/commands/errors_test.go::TestExitCodesAndHints + + - id: gfsctl-debug + name: gfsctl --debug logs requests + description: --debug logs each API request with its status; without it nothing is logged + priority: go_only + environments: + go: cmd/gfsctl/commands/errors_test.go::TestDebugLogsEachRequest + - id: dir-setattr-keeps-mkdir name: Directory chmod keeps another node's mkdir description: Node A changes a directory's mode while node B makes a directory in it; the link count and mtime from node B survive @@ -3573,3 +3666,10 @@ scenarios: priority: go_only environments: go: internal/goclusterfs/setattr_race_test.go::TestFileSetattrKeepsOtherNodesSize + + - id: logging-no-ungated-debug + name: No [DEBUG] line bypasses -debug + description: No non-test Go file calls log.Printf or log.Println with a [DEBUG] prefix; they go through logging.Debugf + priority: go_only + environments: + go: internal/logging/guard_test.go::TestDebugLinesGoThroughDebugf