diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 994e493..544b36d 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -22,4 +22,4 @@ jobs: run: go build ./... - name: Test - run: go test ./... + run: go test -race ./... diff --git a/README.md b/README.md index 55a3213..d053313 100644 --- a/README.md +++ b/README.md @@ -191,7 +191,6 @@ gn metric list --from-exchange binance --to-exchange coinbase | `--assets` | | Filter by multiple assets (repeat for each, e.g. `--assets BTC --assets ETH`) | | `--currency` | `-c` | Filter by currency (e.g. `native`, `usd`) | | `--exchange` | `-e` | Filter by exchange (e.g. `binance`, `coinbase`) | -| `--format` | `-f` | Filter by response format (e.g. `json`, `csv`) | | `--interval` | `-i` | Filter by time interval (e.g. `1h`, `24h`) | | `--from-exchange` | | Source exchange for inter-exchange metrics | | `--to-exchange` | | Destination exchange for inter-exchange metrics | @@ -219,8 +218,10 @@ gn metric get market/price_usd_close --asset BTC --interval 24h gn metric get market/price_usd_close --asset BTC --since 2024-01-01 --until 2024-02-01 gn metric get indicators/sopr --asset BTC --interval 24h --since 30d gn metric get distribution/balance_exchanges --asset BTC --exchange binance --currency usd -# Bulk: append /bulk to the path; use -a '*' for all assets or multiple -a for specific ones +# Bulk: append /bulk to the path; use -a '*' for all assets or multiple -a for specific ones. +# --since is required; longer ranges than the API allows per request need --split gn metric get market/marketcap_usd/bulk -a '*' --since 30d +gn metric get market/marketcap_usd/bulk -a BTC -a ETH --since 2024-01-01 --split ``` | Flag | Short | Description | @@ -232,9 +233,14 @@ gn metric get market/marketcap_usd/bulk -a '*' --since 30d | `--currency` | `-c` | Currency for the metric (`usd`, `native`) | | `--exchange` | `-e` | Exchange filter (repeatable for bulk) | | `--network` | `-n` | Network filter | +| `--split` | | Bulk only: fetch a range longer than the API allows per request in several requests | For **bulk metrics**, append `/bulk` to the path (e.g. `market/marketcap_usd/bulk`). To pass multiple assets, repeat the `-a` (or `--asset`) flag for each: `-a BTC -a ETH -a SOL`. Use `-a '*'` to request all assets. +Point-in-time metrics (`*_pit`) include `computed_at`, the time the value was computed, as a field in JSON and as a trailing column in CSV and table output. + +The API requires `--since` for bulk metrics and accepts at most 10 days per request at `10m`/`1h`, 31 days at `24h` and 93 days at `1w`/`1month`, counted from the start of the interval containing `--since` (weeks start on Monday, months on the 1st, UTC). A longer range is rejected before any request is sent unless you pass `--split`: the CLI then fetches it window by window, prints how many requests that takes on stderr, and merges the points. Every request is billed per asset, so narrow `-a` for long ranges. The merged result is assembled in memory before it is printed. + ### `gn user credits` Show the current API usage for your account. @@ -257,8 +263,13 @@ Example output: All four fields describe the same window. `creditsLimit` is the allowance granted per `creditsPeriod`, both read from your account's API addon, and `creditsUsed` and `creditsLeft` are the usage and remainder within that window. The period -depends on your plan: the advanced product is metered **daily**, every other -product **monthly**. +depends on your plan: the advanced product is metered **daily**, in requests, +every other product **monthly**, in credits. + +If the API reports a period the CLI does not know, or add-ons of different +periods, `creditsPeriod` carries what the API sent (or `mixed`) and +`creditsUsed` and `creditsLeft` are omitted rather than measured against a +counter of another period. ### `gn config set key=value` @@ -286,7 +297,7 @@ OAuth session fields (`oauth-access-token`, `oauth-refresh-token`, `oauth-expire |------|-------|-------------| | `--api-key` | | Glassnode API key | | `--output` | `-o` | Output format: `json` (default), `csv`, `table` | -| `--dry-run` | | Print the request URL without executing | +| `--dry-run` | | Print the endpoint and parameters of the request without sending it. The API key is sent in the `X-Api-Key` header and is not part of the URL. The URL is built from the flags, not by the SDK: metric requests are sent with `f=json` appended, and parameter validation (empty values, `--since` after `--until`) happens only when the request is sent | | `--timestamp-format` | | Timestamp format in output | ## Output Formats @@ -326,8 +337,9 @@ gn metric get market/price_usd_close --asset BTC --since 30d --dry-run Append `/bulk` to the metric path. Specify multiple assets by repeating `-a` for each, or use `-a '*'` for all: ```bash -# Multiple specific assets (repeat -a for each) -gn metric get market/marketcap_usd/bulk -a BTC -a ETH -a SOL -s 2024-01-01 +# Multiple specific assets (repeat -a for each); a range over the API's +# per-request limit is fetched in several requests with --split +gn metric get market/marketcap_usd/bulk -a BTC -a ETH -a SOL -s 2024-01-01 -u 2024-04-01 --split # All assets (wildcard) gn metric get market/marketcap_usd/bulk -a '*' --interval 24h --since 30d diff --git a/cmd/asset_describe.go b/cmd/asset_describe.go index c0b8317..b71f8e3 100644 --- a/cmd/asset_describe.go +++ b/cmd/asset_describe.go @@ -33,8 +33,7 @@ var assetDescribeCmd = &cobra.Command{ if err != nil { return err } - redacted, _ := api.RedactAPIKeyFromURL(u) - _, _ = fmt.Println(redacted) + printDryRun(cmd, client, u) return nil } diff --git a/cmd/asset_list.go b/cmd/asset_list.go index 90e2679..7e427c8 100644 --- a/cmd/asset_list.go +++ b/cmd/asset_list.go @@ -1,7 +1,6 @@ package cmd import ( - "fmt" "regexp" "strings" @@ -51,8 +50,7 @@ var assetListCmd = &cobra.Command{ if err != nil { return err } - redacted, _ := api.RedactAPIKeyFromURL(u) - fmt.Println(redacted) + printDryRun(cmd, client, u) return nil } diff --git a/cmd/credits.go b/cmd/credits.go index fe1eeec..bcbbb18 100644 --- a/cmd/credits.go +++ b/cmd/credits.go @@ -1,8 +1,6 @@ package cmd import ( - "fmt" - "github.com/glassnode/glassnode-cli/internal/api" "github.com/glassnode/glassnode-cli/internal/output" "github.com/spf13/cobra" @@ -26,8 +24,7 @@ var creditsCmd = &cobra.Command{ if err != nil { return err } - redacted, _ := api.RedactAPIKeyFromURL(u) - fmt.Println(redacted) + printDryRun(cmd, client, u) return nil } diff --git a/cmd/credits_test.go b/cmd/credits_test.go index 489b661..775d604 100644 --- a/cmd/credits_test.go +++ b/cmd/credits_test.go @@ -19,7 +19,10 @@ func TestCredits_DryRun_RedactsKey(t *testing.T) { if !strings.Contains(stdout, "/v1/user/api_usage") { t.Errorf("stdout should contain path: %s", stdout) } - if !strings.Contains(stdout, "api_key=***") && !strings.Contains(stdout, "api_key=%2A%2A%2A") { - t.Errorf("stdout should contain redacted api_key: %s", stdout) + if strings.Contains(stdout, "api_key") { + t.Errorf("stdout must not contain an api_key parameter: %s", stdout) + } + if !strings.Contains(stderr, "X-Api-Key") { + t.Errorf("stderr should say the key is sent in the X-Api-Key header: %s", stderr) } } diff --git a/cmd/dryrun.go b/cmd/dryrun.go new file mode 100644 index 0000000..71433aa --- /dev/null +++ b/cmd/dryrun.go @@ -0,0 +1,21 @@ +package cmd + +import ( + "fmt" + + "github.com/glassnode/glassnode-cli/internal/api" + "github.com/spf13/cobra" +) + +// printDryRun prints the endpoint and parameters --dry-run would send. The URL +// is assembled from the flags rather than by the SDK, so it is a preview, not +// the request byte for byte: the SDK appends f=json to metric requests and +// validates parameters only when sending. The credential is sent in a header +// and is not part of the URL, which the note on stderr says, so the URL on +// stdout stays usable in scripts. +func printDryRun(cmd *cobra.Command, client *api.Client, urls ...string) { + for _, url := range urls { + fmt.Fprintln(cmd.OutOrStdout(), url) + } + fmt.Fprintf(cmd.ErrOrStderr(), "# credentials are sent in the %s, not in the URL\n", client.AuthHeader()) +} diff --git a/cmd/metric_describe.go b/cmd/metric_describe.go index 7ebe0f7..9a14491 100644 --- a/cmd/metric_describe.go +++ b/cmd/metric_describe.go @@ -1,8 +1,6 @@ package cmd import ( - "fmt" - "github.com/glassnode/glassnode-cli/internal/api" "github.com/glassnode/glassnode-cli/internal/output" "github.com/spf13/cobra" @@ -34,8 +32,7 @@ var metricDescribeCmd = &cobra.Command{ if err != nil { return err } - redacted, _ := api.RedactAPIKeyFromURL(u) - fmt.Println(redacted) + printDryRun(cmd, client, u) return nil } diff --git a/cmd/metric_get.go b/cmd/metric_get.go index 86c7947..4c0dea2 100644 --- a/cmd/metric_get.go +++ b/cmd/metric_get.go @@ -2,6 +2,7 @@ package cmd import ( "fmt" + "time" "github.com/glassnode/glassnode-cli/internal/api" "github.com/glassnode/glassnode-cli/internal/output" @@ -37,15 +38,18 @@ var metricGetCmd = &cobra.Command{ params := map[string]string{} + // One clock for the whole command: --since 31d then ends exactly 31 + // days before the planned range's end, not a fraction of a second more. + now := time.Now().Truncate(time.Second) if since != "" { - ts, err := timeparse.Parse(since) + ts, err := timeparse.ParseAt(since, now) if err != nil { return fmt.Errorf("parsing --since: %w", err) } params["s"] = fmt.Sprintf("%d", ts) } if until != "" { - ts, err := timeparse.Parse(until) + ts, err := timeparse.ParseAt(until, now) if err != nil { return fmt.Errorf("parsing --until: %w", err) } @@ -70,18 +74,50 @@ var metricGetCmd = &cobra.Command{ repeatedParams["e"] = exchanges } + // The API requires s for bulk requests and limits the range per + // request; longer ranges are fetched in several requests with --split. + plan, err := api.PlanBulk(params, now) + if err != nil { + return err + } + split, _ := cmd.Flags().GetBool("split") + windows := plan.Windows() + if len(windows) > 1 && !split { + return plan.RangeError(len(windows)) + } dryRun, _ := cmd.Flags().GetBool("dry-run") if dryRun { - u, err := client.BuildURL("/v1/metrics"+path+"/bulk", params, repeatedParams) - if err != nil { - return err + // One URL per window and the header note once; nothing is + // fetched, so the split summary does not apply. + urls := make([]string, 0, len(windows)) + for _, w := range windows { + u, err := client.BuildURL("/v1/metrics"+path+"/bulk", api.WindowParams(params, w), repeatedParams) + if err != nil { + return err + } + urls = append(urls, u) } - redacted, _ := api.RedactAPIKeyFromURL(u) - fmt.Println(redacted) + printDryRun(cmd, client, urls...) return nil } + if len(windows) > 1 { + fmt.Fprintln(cmd.ErrOrStderr(), plan.Describe(len(windows))) + } - resp, err := client.GetMetricBulk(cmd.Context(), path, params, repeatedParams) + // --split is the only consent to more than one request: without it + // the single request goes out as planned, with the bounds PlanBulk + // validated and --dry-run prints, and a rejection of its range is + // reported rather than answered with halved windows. + var resp *api.BulkResponse + if split { + resp, err = client.GetMetricBulkSplit(cmd.Context(), path, params, repeatedParams, plan, cmd.ErrOrStderr()) + } else { + resp, err = client.GetMetricBulk(cmd.Context(), path, api.WindowParams(params, windows[0]), repeatedParams) + if api.IsRangeTooLarge(err) { + err = fmt.Errorf("%w; the range is within the documented limit of %s at %s, add --split to retry it in shorter windows", + err, plan.WindowDays(), plan.Interval) + } + } if err != nil { return err } @@ -90,6 +126,11 @@ var metricGetCmd = &cobra.Command{ return output.Print(output.Options{Format: format, Data: resp, TimestampFormat: tsFmt}) } + if split, _ := cmd.Flags().GetBool("split"); split { + // Base metrics have no per-request range limit, so there is nothing + // to split; say so rather than fail a script that shares its flags. + fmt.Fprintln(cmd.ErrOrStderr(), "--split applies to bulk paths only and was ignored") + } if len(assets) > 0 { params["a"] = assets[0] } @@ -103,8 +144,7 @@ var metricGetCmd = &cobra.Command{ if err != nil { return err } - redacted, _ := api.RedactAPIKeyFromURL(u) - fmt.Println(redacted) + printDryRun(cmd, client, u) return nil } @@ -127,4 +167,5 @@ func init() { metricGetCmd.Flags().StringP("currency", "c", "", "native or usd") metricGetCmd.Flags().StringArrayP("exchange", "e", nil, "exchange filter (repeatable for bulk)") metricGetCmd.Flags().StringP("network", "n", "", "network filter") + metricGetCmd.Flags().Bool("split", false, "bulk only: fetch a range longer than the API allows per request in several requests") } diff --git a/cmd/metric_get_test.go b/cmd/metric_get_test.go index 55dbc08..ecf58cb 100644 --- a/cmd/metric_get_test.go +++ b/cmd/metric_get_test.go @@ -2,13 +2,19 @@ package cmd import ( "encoding/json" + "fmt" + "net/http" + "net/http/httptest" "os" "os/exec" "path/filepath" "runtime" + "strconv" "strings" "sync" + "sync/atomic" "testing" + "time" ) // moduleRoot returns the path to the repository root (parent of cmd/). @@ -75,7 +81,7 @@ func runCLI(t *testing.T, env []string, args ...string) (string, string, error) return outBuf.String(), errBuf.String(), runErr } -func TestMetricGet_DryRun_PrintsURLWithRedactedKey(t *testing.T) { +func TestMetricGet_DryRun_PrintsURLWithoutKey(t *testing.T) { stdout, stderr, err := runCLI(t, []string{"HOME=" + t.TempDir()}, "metric", "get", "/market/price_usd_close", "--api-key", "secret-key", @@ -87,8 +93,11 @@ func TestMetricGet_DryRun_PrintsURLWithRedactedKey(t *testing.T) { if strings.Contains(stdout, "secret-key") { t.Errorf("stdout must not contain API key: %s", stdout) } - if !strings.Contains(stdout, "api_key=***") && !strings.Contains(stdout, "api_key=%2A%2A%2A") { - t.Errorf("stdout should contain redacted api_key: %s", stdout) + if strings.Contains(stdout, "api_key") { + t.Errorf("stdout must not contain an api_key parameter: %s", stdout) + } + if !strings.Contains(stderr, "X-Api-Key") { + t.Errorf("stderr should say the key is sent in the X-Api-Key header: %s", stderr) } if !strings.Contains(stdout, "/v1/metrics") || !strings.Contains(stdout, "market/price_usd_close") { t.Errorf("stdout should contain metric path: %s", stdout) @@ -164,3 +173,86 @@ func TestMetricGet_InvalidUntil_Fails(t *testing.T) { t.Errorf("stderr should mention until/parsing: %q", stderr) } } + +// Without --split the CLI sends the one planned request and reports the API's +// rejection; it does not halve the window and multiply billed requests. +func TestMetricGet_Bulk_WithoutSplitSendsOneRequest(t *testing.T) { + var calls atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + w.WriteHeader(http.StatusBadRequest) + fmt.Fprint(w, `{"message":"too large time range (744h0m0s) requested (maximum allowed is 240h0m0s)"}`) + })) + t.Cleanup(server.Close) + _, stderr, err := runCLI(t, []string{"HOME=" + t.TempDir(), "GLASSNODE_BASE_URL=" + server.URL}, + "metric", "get", "/market/price_usd_close/bulk", + "--api-key", "k", "--asset", "BTC", "--since", "31d", + ) + if err == nil { + t.Fatal("expected non-zero exit when the API rejects the range") + } + if calls.Load() != 1 { + t.Errorf("requests = %d, want exactly 1 without --split", calls.Load()) + } + for _, want := range []string{"too large time range", "add --split"} { + if !strings.Contains(stderr, want) { + t.Errorf("stderr %q lacks %q", stderr, want) + } + } +} + +// The one request sent without --split carries the planned bounds, the same +// ones --dry-run prints: a future --until is capped at now, so the API never +// sees a range the plan did not validate. +func TestMetricGet_Bulk_WithoutSplitSendsPlannedBounds(t *testing.T) { + start := time.Now() + var calls atomic.Int32 + var until string + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + until = r.URL.Query().Get("u") + fmt.Fprint(w, `{"data":[]}`) + })) + t.Cleanup(server.Close) + _, stderr, err := runCLI(t, []string{"HOME=" + t.TempDir(), "GLASSNODE_BASE_URL=" + server.URL}, + "metric", "get", "/market/price_usd_close/bulk", + "--api-key", "k", "--asset", "BTC", "--since", "20d", "--until", "2999-01-01", + ) + if err != nil { + t.Fatalf("run CLI: %v\nstderr: %s", err, stderr) + } + u, convErr := strconv.ParseInt(until, 10, 64) + if calls.Load() != 1 || convErr != nil || u < start.Unix() || u > time.Now().Unix() { + t.Errorf("calls=%d u=%q, want one request with u capped at now", calls.Load(), until) + } +} + +func TestMetricGet_SplitOnBaseMetricIsIgnoredWithANote(t *testing.T) { + stdout, stderr, err := runCLI(t, []string{"HOME=" + t.TempDir()}, + "metric", "get", "/market/price_usd_close", "--api-key", "k", "--since", "400d", "--split", "--dry-run", + ) + if err != nil { + t.Fatalf("run CLI: %v\nstderr: %s", err, stderr) + } + if !strings.Contains(stderr, "--split applies to bulk paths only") || strings.Contains(stdout, "split") { + t.Errorf("stderr=%q stdout=%q, want the note on stderr and a plain URL on stdout", stderr, stdout) + } +} + +// A bulk dry run prints one URL per window, the header note once, and no +// split summary, since nothing is fetched. +func TestMetricGet_BulkDryRun_NoteOnce(t *testing.T) { + stdout, stderr, err := runCLI(t, []string{"HOME=" + t.TempDir()}, + "metric", "get", "/market/price_usd_close/bulk", "--api-key", "k", "--asset", "BTC", + "--since", "2024-01-01", "--until", "2024-03-01", "--split", "--dry-run", + ) + if err != nil { + t.Fatalf("run CLI: %v\nstderr: %s", err, stderr) + } + if strings.Count(stdout, "/v1/metrics/market/price_usd_close/bulk?") != 2 { + t.Errorf("stdout should hold two window URLs:\n%s", stdout) + } + if strings.Count(stderr, "# credentials") != 1 || strings.Contains(stderr, "splitting") { + t.Errorf("stderr should carry the note once and no split summary:\n%s", stderr) + } +} diff --git a/cmd/metric_list.go b/cmd/metric_list.go index b6ae1fa..088af6a 100644 --- a/cmd/metric_list.go +++ b/cmd/metric_list.go @@ -1,7 +1,7 @@ package cmd import ( - "fmt" + "errors" "github.com/glassnode/glassnode-cli/internal/api" "github.com/glassnode/glassnode-cli/internal/output" @@ -12,6 +12,9 @@ var metricListCmd = &cobra.Command{ Use: "list", Short: "List available metrics", RunE: func(cmd *cobra.Command, args []string) error { + if cmd.Flags().Changed("format") { + return errors.New(formatRemoved) + } apiKeyFlag, _ := cmd.Flags().GetString("api-key") apiKey, bearer, err := api.RequireAuth(cmd.Context(), apiKeyFlag) if err != nil { @@ -27,8 +30,7 @@ var metricListCmd = &cobra.Command{ if err != nil { return err } - redacted, _ := api.RedactAPIKeyFromURL(u) - fmt.Println(redacted) + printDryRun(cmd, client, u) return nil } @@ -64,9 +66,6 @@ func metricListParamsFromFlags(cmd *cobra.Command) (map[string]string, map[strin if v, _ := cmd.Flags().GetString("exchange"); v != "" { params["e"] = v } - if v, _ := cmd.Flags().GetString("format"); v != "" { - params["f"] = v - } if v, _ := cmd.Flags().GetString("interval"); v != "" { params["i"] = v } @@ -95,12 +94,18 @@ func metricListParamsFromFlags(cmd *cobra.Command) (map[string]string, map[strin return params, repeatedParams } +// formatRemoved explains the removed --format filter to scripts that still +// pass it; the hidden flag and this message go away in the next release. +const formatRemoved = `--format was removed: the API's response-format filter on the metric list returns no metrics for any value; ` + + `for metric data in CSV use "gn metric get ... -o csv"` + func init() { + metricListCmd.Flags().StringP("format", "f", "", "removed") + _ = metricListCmd.Flags().MarkHidden("format") metricListCmd.Flags().StringP("asset", "a", "", "filter by asset (single)") metricListCmd.Flags().StringArray("assets", nil, "filter by assets (multiple, e.g. --assets BTC --assets ETH)") metricListCmd.Flags().StringP("currency", "c", "", "filter by currency (e.g. native, usd)") metricListCmd.Flags().StringP("exchange", "e", "", "filter by exchange (e.g. binance, coinbase)") - metricListCmd.Flags().StringP("format", "f", "", "filter by response format (e.g. json, csv)") metricListCmd.Flags().StringP("interval", "i", "", "filter by time interval (e.g. 1h, 24h)") metricListCmd.Flags().String("from-exchange", "", "source exchange for inter-exchange metrics") metricListCmd.Flags().String("to-exchange", "", "destination exchange for inter-exchange metrics") diff --git a/cmd/metric_list_test.go b/cmd/metric_list_test.go new file mode 100644 index 0000000..2edaa1b --- /dev/null +++ b/cmd/metric_list_test.go @@ -0,0 +1,20 @@ +package cmd + +import ( + "strings" + "testing" +) + +// The removed --format filter fails with the reason and the replacement for +// one release, instead of cobra's "unknown shorthand flag". +func TestMetricList_RemovedFormatFlagExplains(t *testing.T) { + for _, args := range [][]string{{"-f", "csv"}, {"--format", "json"}} { + _, stderr, err := runCLI(t, []string{"HOME=" + t.TempDir()}, append([]string{"metric", "list", "--api-key", "k"}, args...)...) + if err == nil { + t.Fatalf("%v: expected a failure", args) + } + if !strings.Contains(stderr, "--format was removed") || !strings.Contains(stderr, "-o csv") || strings.Contains(stderr, "unknown shorthand") { + t.Errorf("%v: stderr = %s", args, stderr) + } + } +} diff --git a/cmd/names_list.go b/cmd/names_list.go index c49c830..3a0bd28 100644 --- a/cmd/names_list.go +++ b/cmd/names_list.go @@ -1,8 +1,6 @@ package cmd import ( - "fmt" - "github.com/glassnode/glassnode-cli/internal/api" "github.com/glassnode/glassnode-cli/internal/output" "github.com/spf13/cobra" @@ -34,8 +32,7 @@ func runNamesListCommand(cmd *cobra.Command, endpoint string, withFilter bool) e if err != nil { return err } - redacted, _ := api.RedactAPIKeyFromURL(u) - fmt.Println(redacted) + printDryRun(cmd, client, u) return nil } diff --git a/cmd/root.go b/cmd/root.go index f931b66..f35ef5a 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -20,7 +20,7 @@ func init() { rootCmd.Version = "dev" rootCmd.PersistentFlags().String("api-key", "", "API key override") rootCmd.PersistentFlags().StringP("output", "o", "json", "output format: json, csv, table") - rootCmd.PersistentFlags().Bool("dry-run", false, "print the request URL without executing") + rootCmd.PersistentFlags().Bool("dry-run", false, "print the endpoint and parameters of the request without sending it (the SDK adds f=json to metric requests and validates parameters only when sending)") rootCmd.PersistentFlags().String("timestamp-format", "humanized", "timestamp format: unix, humanized, or Go layout (e.g. 2006-01-02 15:04:05); applies to table/csv output") rootCmd.SetVersionTemplate("gn version {{.Version}}\n") diff --git a/go.mod b/go.mod index 3413a50..6a90d5d 100644 --- a/go.mod +++ b/go.mod @@ -3,6 +3,7 @@ module github.com/glassnode/glassnode-cli go 1.24.0 require ( + github.com/glassnode/glassnode-api-go-client v0.2.0 github.com/olekukonko/tablewriter v0.0.5 github.com/spf13/cobra v1.10.2 gopkg.in/yaml.v3 v3.0.1 diff --git a/go.sum b/go.sum index f15cc87..5cba138 100644 --- a/go.sum +++ b/go.sum @@ -4,6 +4,10 @@ github.com/clipperhouse/uax29/v2 v2.3.0 h1:SNdx9DVUqMoBuBoW3iLOj4FQv3dN5mDtuqwuh github.com/clipperhouse/uax29/v2 v2.3.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g= github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/glassnode/glassnode-api-go-client v0.1.0 h1:qm0XGrdAZJsGYbn7KVoiEjUkGS8YUA+6f0t6CD7C3xM= +github.com/glassnode/glassnode-api-go-client v0.1.0/go.mod h1:Jwk/v/xgvMtTXLMV/WnnVa+3ycFVZiBR4EgU8/JdcbQ= +github.com/glassnode/glassnode-api-go-client v0.2.0 h1:mNS8YOEgdCwi/fOHndtD54M5qfQ249OPM29+bxyKTsw= +github.com/glassnode/glassnode-api-go-client v0.2.0/go.mod h1:Jwk/v/xgvMtTXLMV/WnnVa+3ycFVZiBR4EgU8/JdcbQ= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= diff --git a/internal/api/api_usage.go b/internal/api/api_usage.go index e95cf46..da63820 100644 --- a/internal/api/api_usage.go +++ b/internal/api/api_usage.go @@ -2,137 +2,80 @@ package api import ( "context" - "encoding/json" "fmt" - "strings" + + glassnode "github.com/glassnode/glassnode-api-go-client" ) -// CreditsPeriod is the period over which an account's API credit allowance is -// granted. The values are the ones /v1/user/api_usage reports on the addon, so -// they reach the CLI output unchanged rather than being renamed on the way. -type CreditsPeriod string +type APIAddon = glassnode.APIAddon + +// CreditsPeriod is the period over which an account's API allowance is +// granted, as /v1/user/api_usage reports it on the add-on. +type CreditsPeriod = glassnode.AllowancePeriod const ( - CreditsPeriodDaily CreditsPeriod = "daily" - CreditsPeriodMonthly CreditsPeriod = "monthly" + CreditsPeriodDaily = glassnode.AllowanceDaily + CreditsPeriodMonthly = glassnode.AllowanceMonthly ) -type APIAddon struct { - Value int `json:"value"` - Period string `json:"period"` - RPM int `json:"rpm"` - Version string `json:"version"` -} - -// CreditsPeriod returns the addon's period as a CreditsPeriod. Period is a -// closed enum of CreditsPeriodDaily or CreditsPeriodMonthly, matched -// case-insensitively so a change in the server's casing or padding cannot -// silently route a daily allowance onto the month-to-date counter. The default -// keeps the summary readable rather than failing if the enum is ever widened, -// at the cost of labelling a new period as monthly. -func (a APIAddon) CreditsPeriod() CreditsPeriod { - switch strings.ToLower(strings.TrimSpace(a.Period)) { - case string(CreditsPeriodDaily): - return CreditsPeriodDaily - case string(CreditsPeriodMonthly): - return CreditsPeriodMonthly - default: - return CreditsPeriodMonthly - } -} - +// APIUsageResponse is the SDK's usage report; the CLI adds the summary it +// prints. CreditsUsed is month-to-date and DailyRequestsUsed the current day. type APIUsageResponse struct { - CustomerID int `json:"customerId"` - // CreditsUsed is month-to-date and DailyRequestsUsed is the current day, - // so each pairs with the addon period of the same name. - CreditsUsed int `json:"creditsUsed"` - DailyRequestsUsed int `json:"dailyRequestsUsed"` - APIAddons []APIAddon `json:"apiAddons"` -} - -// primaryAddon returns the addon carrying the largest credit allowance, or nil -// when the account has no addons. An account cannot hold both a monthly and a -// daily API addon, so this never has to choose between periods: the max only -// guards against ordering assumptions, it is not a cross-period comparison. -func (a *APIUsageResponse) primaryAddon() *APIAddon { - var primary *APIAddon - for i := range a.APIAddons { - if primary == nil || a.APIAddons[i].Value > primary.Value { - primary = &a.APIAddons[i] - } - } - - return primary -} - -// CreditsLimit returns the largest addon credit value, or 0 when there are no addons -func (a *APIUsageResponse) CreditsLimit() int { - if primary := a.primaryAddon(); primary != nil { - return primary.Value - } - - return 0 -} - -// CreditsPeriod returns the period the allowance reported by CreditsLimit is -// granted over, taken from the addon that allowance comes from. -func (a *APIUsageResponse) CreditsPeriod() CreditsPeriod { - if primary := a.primaryAddon(); primary != nil { - return primary.CreditsPeriod() - } - - return CreditsPeriodMonthly -} - -// CreditsUsedInPeriod returns the usage that counts against CreditsLimit: the -// current day's requests on a daily allowance, the month-to-date total on a -// monthly one. Both counters reset with their own period, so each only lines up -// with the allowance of the same period. -func (a *APIUsageResponse) CreditsUsedInPeriod() int { - if a.CreditsPeriod() == CreditsPeriodDaily { - return a.DailyRequestsUsed - } - - return a.CreditsUsed + glassnode.APIUsage } // CreditsSummary is the CLI response to the end user. Every field describes the // same window: CreditsLimit is the allowance granted per CreditsPeriod, and -// CreditsUsed and CreditsLeft are the usage and remainder within it. +// CreditsUsed and CreditsLeft are the usage and remainder within it. When the +// period is not one the CLI knows, CreditsPeriod carries it as reported and +// the usage fields are omitted, since no counter is known to match it. type CreditsSummary struct { - CreditsLeft int `json:"creditsLeft"` + CreditsLeft *int `json:"creditsLeft,omitempty"` CreditsLimit int `json:"creditsLimit"` CreditsPeriod CreditsPeriod `json:"creditsPeriod"` - CreditsUsed int `json:"creditsUsed"` + CreditsUsed *int `json:"creditsUsed,omitempty"` } func (a *APIUsageResponse) Summary() CreditsSummary { - limit := a.CreditsLimit() - used := a.CreditsUsedInPeriod() - left := limit - used - if left < 0 { - left = 0 + if allowance, ok := a.Allowance(); ok { + left, used := allowance.Remaining(), allowance.Used + return CreditsSummary{CreditsLeft: &left, CreditsLimit: allowance.Limit, CreditsPeriod: allowance.Period, CreditsUsed: &used} } + if len(a.APIAddons) == 0 { + // No allowance: nothing left of the month-to-date usage. + left, used := 0, a.CreditsUsed + return CreditsSummary{CreditsLeft: &left, CreditsPeriod: CreditsPeriodMonthly, CreditsUsed: &used} + } + // An unknown or mixed period: report what the API sent rather than + // measuring it against a counter of another period. + summary := CreditsSummary{CreditsPeriod: addonPeriod(a.APIAddons[0])} + for _, addon := range a.APIAddons { + summary.CreditsLimit = max(summary.CreditsLimit, addon.Value) + if addonPeriod(addon) != summary.CreditsPeriod { + summary.CreditsPeriod = "mixed" + } + } + return summary +} - return CreditsSummary{ - CreditsUsed: used, - CreditsLimit: limit, - CreditsPeriod: a.CreditsPeriod(), - CreditsLeft: left, +// addonPeriod is the add-on's period with the API's default, an absent +// period, read as monthly, as the SDK does. +func addonPeriod(addon APIAddon) CreditsPeriod { + if addon.Period == "" { + return CreditsPeriodMonthly } + return addon.Period } // GetAPIUsage fetches the current API usage for the authenticated user func (c *Client) GetAPIUsage(ctx context.Context) (*APIUsageResponse, error) { - body, err := c.Do(ctx, "GET", "/v1/user/api_usage", nil) + client, err := c.sdk() if err != nil { - return nil, fmt.Errorf("fetching API usage: %w", err) + return nil, err } - - var out APIUsageResponse - if err := json.Unmarshal(body, &out); err != nil { - return nil, fmt.Errorf("decoding API usage response: %w", err) + usage, err := client.GetAPIUsage(ctx) + if err != nil { + return nil, fmt.Errorf("fetching API usage response: %w", err) } - - return &out, nil + return &APIUsageResponse{APIUsage: *usage}, nil } diff --git a/internal/api/api_usage_test.go b/internal/api/api_usage_test.go index 4efd7f8..53f0795 100644 --- a/internal/api/api_usage_test.go +++ b/internal/api/api_usage_test.go @@ -3,11 +3,14 @@ package api import ( "context" "encoding/json" + "errors" "net/http" "net/http/httptest" "os" "strings" "testing" + + glassnode "github.com/glassnode/glassnode-api-go-client" ) func TestGetAPIUsage(t *testing.T) { @@ -19,7 +22,7 @@ func TestGetAPIUsage(t *testing.T) { var gotPath, gotAPIKey string server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotPath = r.URL.Path - gotAPIKey = r.URL.Query().Get("api_key") + gotAPIKey = r.Header.Get("X-Api-Key") w.Header().Set("Content-Type", "application/json") _, _ = w.Write(fixture) })) @@ -38,35 +41,27 @@ func TestGetAPIUsage(t *testing.T) { t.Errorf("path = %q, want /v1/user/api_usage", gotPath) } if gotAPIKey != "my-key" { - t.Errorf("api_key = %q, want my-key", gotAPIKey) + t.Errorf("X-Api-Key = %q, want my-key", gotAPIKey) } - if out.CreditsUsed != 6 { - t.Errorf("CreditsUsed = %d, want 6", out.CreditsUsed) + if out.CreditsUsed != 6 || out.DailyRequestsUsed != 2 { + t.Errorf("CreditsUsed = %d, DailyRequestsUsed = %d, want 6 and 2", out.CreditsUsed, out.DailyRequestsUsed) } if len(out.APIAddons) != 1 { - t.Errorf("len(APIAddons) = %d, want 1", len(out.APIAddons)) - } - if out.CustomerID != 4568 { - t.Errorf("CustomerID = %d, want 4568", out.CustomerID) - } - if out.DailyRequestsUsed != 2 { - t.Errorf("DailyRequestsUsed = %d, want 2", out.DailyRequestsUsed) + t.Fatalf("len(APIAddons) = %d, want 1", len(out.APIAddons)) } - if addon := out.APIAddons[0]; addon.Period != "monthly" || addon.RPM != 100 || addon.Version != "v3" { - t.Errorf("APIAddons[0] = %+v, want period=monthly rpm=100 version=v3", addon) - } - if out.CreditsLimit() != 1500000 { - t.Errorf("CreditsLimit() = %d, want 1500000 (max addon value)", out.CreditsLimit()) + if addon := out.APIAddons[0]; addon.Period != CreditsPeriodMonthly || addon.RPM != 100 { + t.Errorf("APIAddons[0] = %+v, want period=monthly rpm=100", addon) } sum := out.Summary() - if sum.CreditsUsed != 6 || sum.CreditsLimit != 1500000 || sum.CreditsLeft != 1500000-6 { - t.Errorf("Summary() = %+v, want creditsUsed=6 creditsLimit=1500000 creditsLeft=%d", sum, 1500000-6) - } - if sum.CreditsPeriod != CreditsPeriodMonthly { - t.Errorf("Summary().CreditsPeriod = %q, want %q", sum.CreditsPeriod, CreditsPeriodMonthly) + if *sum.CreditsUsed != 6 || sum.CreditsLimit != 1500000 || *sum.CreditsLeft != 1500000-6 || sum.CreditsPeriod != CreditsPeriodMonthly { + t.Errorf("Summary() = %+v, want creditsUsed=6 creditsLimit=1500000 creditsLeft=%d monthly", sum, 1500000-6) } } +func usage(creditsUsed, dailyRequestsUsed int, addons ...APIAddon) *APIUsageResponse { + return &APIUsageResponse{glassnode.APIUsage{CreditsUsed: creditsUsed, DailyRequestsUsed: dailyRequestsUsed, APIAddons: addons}} +} + func TestCreditsPeriod_ComesFromTheAddon(t *testing.T) { tests := []struct { name string @@ -75,20 +70,17 @@ func TestCreditsPeriod_ComesFromTheAddon(t *testing.T) { }{ {"daily", []APIAddon{{Value: 50, Period: "daily"}}, CreditsPeriodDaily}, {"monthly", []APIAddon{{Value: 1500000, Period: "monthly"}}, CreditsPeriodMonthly}, - {"an absent period falls back to monthly", []APIAddon{{Value: 50}}, CreditsPeriodMonthly}, - {"a period outside the enum falls back to monthly", []APIAddon{{Value: 50, Period: "weekly"}}, CreditsPeriodMonthly}, - {"no addons falls back to monthly", nil, CreditsPeriodMonthly}, - {"a mixed-case period still maps", []APIAddon{{Value: 50, Period: "Daily"}}, CreditsPeriodDaily}, - {"a padded period still maps", []APIAddon{{Value: 50, Period: " monthly "}}, CreditsPeriodMonthly}, + {"an absent period is monthly", []APIAddon{{Value: 50}}, CreditsPeriodMonthly}, + {"no addons is monthly", nil, CreditsPeriodMonthly}, + {"a period outside the enum is reported as sent", []APIAddon{{Value: 50, Period: "weekly"}}, "weekly"}, + {"mixed periods are reported as mixed", []APIAddon{{Value: 50, Period: "daily"}, {Value: 5000000, Period: "monthly"}}, "mixed"}, + {"an absent period counts as monthly when mixed", []APIAddon{{Value: 50}, {Value: 5000000, Period: "monthly"}, {Value: 7, Period: "weekly"}}, "mixed"}, + {"an absent period never surfaces as empty", []APIAddon{{Value: 50}, {Value: 60, Period: "weekly"}}, "mixed"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - usage := &APIUsageResponse{APIAddons: tt.addons} - if got := usage.CreditsPeriod(); got != tt.want { - t.Errorf("CreditsPeriod() = %q, want %q", got, tt.want) - } - if got := usage.Summary().CreditsPeriod; got != tt.want { + if got := usage(0, 0, tt.addons...).Summary().CreditsPeriod; got != tt.want { t.Errorf("Summary().CreditsPeriod = %q, want %q", got, tt.want) } }) @@ -100,70 +92,51 @@ func TestCreditsPeriod_ComesFromTheAddon(t *testing.T) { func TestSummary_UsageIsScopedToThePeriod(t *testing.T) { tests := []struct { name string - period string + addon APIAddon wantUsed int wantLeft int }{ - {"daily counts the day's requests", "daily", 2, 50 - 2}, - {"monthly counts the running total", "monthly", 4000, 50000 - 4000}, + {"daily counts the day's requests", APIAddon{Value: 50, Period: "daily"}, 2, 50 - 2}, + {"monthly counts the running total", APIAddon{Value: 50000, Period: "monthly"}, 4000, 50000 - 4000}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - limit := 50 - if tt.period == string(CreditsPeriodMonthly) { - limit = 50000 - } - usage := &APIUsageResponse{ - CreditsUsed: 4000, - DailyRequestsUsed: 2, - APIAddons: []APIAddon{{Value: limit, Period: tt.period}}, - } - - if got := usage.CreditsUsedInPeriod(); got != tt.wantUsed { - t.Errorf("CreditsUsedInPeriod() = %d, want %d", got, tt.wantUsed) - } - sum := usage.Summary() - if sum.CreditsUsed != tt.wantUsed { - t.Errorf("CreditsUsed = %d, want %d", sum.CreditsUsed, tt.wantUsed) - } - if sum.CreditsLeft != tt.wantLeft { - t.Errorf("CreditsLeft = %d, want %d", sum.CreditsLeft, tt.wantLeft) + sum := usage(4000, 2, tt.addon).Summary() + if *sum.CreditsUsed != tt.wantUsed || *sum.CreditsLeft != tt.wantLeft { + t.Errorf("Summary() used=%d left=%d, want %d and %d", *sum.CreditsUsed, *sum.CreditsLeft, tt.wantUsed, tt.wantLeft) } }) } } -func TestCreditsLimit_NoAddons(t *testing.T) { - usage := &APIUsageResponse{CreditsUsed: 3} - - if got := usage.CreditsLimit(); got != 0 { - t.Errorf("CreditsLimit() = %d, want 0 without addons", got) - } - if got := usage.Summary().CreditsLeft; got != 0 { - t.Errorf("CreditsLeft = %d, want 0 without addons", got) +func TestSummary_NoAddons(t *testing.T) { + sum := usage(3, 0).Summary() + if sum.CreditsLimit != 0 || *sum.CreditsLeft != 0 || *sum.CreditsUsed != 3 { + t.Errorf("Summary() = %+v, want no limit, nothing left, 3 used", sum) } } func TestSummary_CreditsLeftNeverNegative(t *testing.T) { - usage := &APIUsageResponse{ - DailyRequestsUsed: 500, - APIAddons: []APIAddon{{Value: 100, Period: "daily"}}, - } - - if got := usage.Summary().CreditsLeft; got != 0 { + if got := *usage(0, 500, APIAddon{Value: 100, Period: "daily"}).Summary().CreditsLeft; got != 0 { t.Errorf("CreditsLeft = %d, want 0 when usage exceeds the limit", got) } } -func TestSummary_MarshalsCreditsLimitAndPeriod(t *testing.T) { - usage := &APIUsageResponse{ - CreditsUsed: 9000, - DailyRequestsUsed: 6, - APIAddons: []APIAddon{{Value: 50, Period: "daily"}}, +// An unknown period must not be measured against a counter of another period: +// the limit and the period are reported, the usage fields are left out. +func TestSummary_UnknownPeriodOmitsUsage(t *testing.T) { + body, err := json.Marshal(usage(9000, 6, APIAddon{Value: 300, Period: "weekly"}).Summary()) + if err != nil { + t.Fatal(err) + } + if got, want := string(body), `{"creditsLimit":300,"creditsPeriod":"weekly"}`; got != want { + t.Errorf("summary JSON %s, want %s", got, want) } +} - body, err := json.Marshal(usage.Summary()) +func TestSummary_MarshalsCreditsLimitAndPeriod(t *testing.T) { + body, err := json.Marshal(usage(9000, 6, APIAddon{Value: 50, Period: "daily"}).Summary()) if err != nil { t.Fatalf("marshal summary: %v", err) } @@ -199,7 +172,8 @@ func TestGetAPIUsage_InvalidJSONReturnsError(t *testing.T) { t.Fatal("expected error for invalid JSON") } - if !strings.Contains(err.Error(), "decoding API usage response") { - t.Errorf("error = %v, want wrapping decode message", err) + var decode *glassnode.DecodeError + if !errors.As(err, &decode) || decode.Endpoint != "/v1/user/api_usage" { + t.Errorf("error = %v, want typed API usage decode error", err) } } diff --git a/internal/api/bulk_split.go b/internal/api/bulk_split.go new file mode 100644 index 0000000..7c3d6b9 --- /dev/null +++ b/internal/api/bulk_split.go @@ -0,0 +1,281 @@ +package api + +import ( + "context" + "errors" + "fmt" + "io" + "strconv" + "strings" + "time" + + glassnode "github.com/glassnode/glassnode-api-go-client" +) + +// Bulk requests are limited to a time range that depends on the resolution; +// see https://docs.glassnode.com/basic-api/bulk-metrics#timerange-constraints. +const ( + bulkWindowFine = 10 * 24 * time.Hour // 10m and 1h + bulkWindowDaily = 31 * 24 * time.Hour // 24h, the default resolution + bulkWindowCoarse = 93 * 24 * time.Hour // 1w and 1month + defaultResolution = "24h" + + // When the API still rejects a window as too large, the window is halved + // this many times before giving up. + maxWindowShrinks = 3 +) + +// intervalDuration is the length of one data interval at the resolution; a +// window shorter than it returns no points, so halving stops there. +func intervalDuration(interval string) time.Duration { + switch interval { + case "10m": + return 10 * time.Minute + case "1h": + return time.Hour + case "1w": + return 7 * 24 * time.Hour + case "1month": + return 31 * 24 * time.Hour + default: + return 24 * time.Hour + } +} + +// intervalStart returns the start of the interval containing t, in UTC: weeks +// start on Monday, months on the 1st. The API rounds s and u down this way and +// measures the range between the rounded bounds. +func intervalStart(t time.Time, interval string) time.Time { + t = t.UTC() + switch interval { + case "10m": + return t.Truncate(10 * time.Minute) + case "1h": + return t.Truncate(time.Hour) + case "1w": + midnight := t.Truncate(24 * time.Hour) + return midnight.AddDate(0, 0, -(int(midnight.Weekday())+6)%7) + case "1month": + return time.Date(t.Year(), t.Month(), 1, 0, 0, 0, 0, time.UTC) + default: + return t.Truncate(24 * time.Hour) + } +} + +// knownInterval reports whether the API offers the resolution; the window +// helpers fall back to 24h, so an unknown one must be refused before planning. +func knownInterval(interval string) bool { + switch interval { + case "10m", "1h", "24h", "1w", "1month": + return true + } + return false +} + +// BulkWindow returns the longest time range the API accepts in one bulk +// request at the given resolution. +func BulkWindow(interval string) time.Duration { + switch interval { + case "10m", "1h": + return bulkWindowFine + case "1w", "1month": + return bulkWindowCoarse + default: + return bulkWindowDaily + } +} + +// BulkPlan is the time range of a bulk request and the window it is fetched in. +type BulkPlan struct { + Since, Until time.Time + Interval string + Window time.Duration +} + +// Window is one request's time range: Since inclusive, Until exclusive. +type Window struct{ Since, Until time.Time } + +// PlanBulk reads the s, u and i parameters of a bulk request. s is required +// by the API; u defaults to now and is capped at now, since the API returns no +// future points. now is truncated to seconds, the API's resolution, so that a +// range ending now is never a fraction of a second over the window. +func PlanBulk(params map[string]string, now time.Time) (BulkPlan, error) { + plan := BulkPlan{Interval: params["i"], Until: now.Truncate(time.Second)} + if plan.Interval == "" { + plan.Interval = defaultResolution + } + if !knownInterval(plan.Interval) { + return plan, fmt.Errorf("unknown --interval %q; bulk metrics accept 10m, 1h, 24h, 1w or 1month", plan.Interval) + } + plan.Window = BulkWindow(plan.Interval) + raw, ok := params["s"] + if !ok { + return plan, fmt.Errorf("bulk metrics require --since (at most %s per request at %s; --split fetches a longer range in several requests)", + plan.WindowDays(), plan.Interval) + } + since, err := strconv.ParseInt(raw, 10, 64) + if err != nil { + return plan, fmt.Errorf("parameter s: expected Unix seconds, got %q", raw) + } + // The API counts the range from the start of the interval containing s, + // so the plan does too: at 1w, --since 93d reaches back up to 99 days. + plan.Since = intervalStart(time.Unix(since, 0), plan.Interval) + // boundedByNow: the range ends at now, either because --until was not + // given or because it lay in the future; the error then talks about now, + // not about an --until the user never saw. + boundedByNow := true + if raw, ok := params["u"]; ok { + until, err := strconv.ParseInt(raw, 10, 64) + if err != nil { + return plan, fmt.Errorf("parameter u: expected Unix seconds, got %q", raw) + } + if t := time.Unix(until, 0).UTC(); t.Before(plan.Until) { + plan.Until = t + boundedByNow = false + } + } + plan.Until = plan.Until.UTC() + if !plan.Since.Before(plan.Until) { + if boundedByNow { + return plan, fmt.Errorf("--since (%s) is not in the past", plan.Since.Format(time.RFC3339)) + } + return plan, fmt.Errorf("--since (%s) must be before --until (%s)", plan.Since.Format(time.RFC3339), plan.Until.Format(time.RFC3339)) + } + return plan, nil +} + +// Windows splits the range into consecutive windows the API accepts: from the +// start of the interval containing s to the start of the one containing u is +// at most plan.Window. Inner bounds are interval starts, and each window's +// Until is the next window's Since: the API includes the interval containing s +// and only intervals that end by u, so consecutive windows neither overlap nor +// leave gaps. +func (plan BulkPlan) Windows() []Window { + var windows []Window + for since := plan.Since; since.Before(plan.Until); { + limit := since.Add(plan.Window) + until := plan.Until + if intervalStart(until, plan.Interval).After(limit) { + until = intervalStart(limit, plan.Interval) + if !until.After(since) { + // A window shorter than one interval: no bound to round to. + until = limit + } + } + windows = append(windows, Window{since, until}) + since = until + } + return windows +} + +// WindowDays is the window's length as shown to the user, such as "31 days". +func (plan BulkPlan) WindowDays() string { return days(plan.Window) } + +// RangeError explains why the range needs --split; requests is len(Windows()). +func (plan BulkPlan) RangeError(requests int) error { + return fmt.Errorf("range %s..%s exceeds the API limit of %s per bulk request at %s; shorten it or add --split to fetch it in %d requests", + formatBound(plan.Since), formatBound(plan.Until), plan.WindowDays(), plan.Interval, requests) +} + +// Describe summarises the split for a warning to the user; requests is len(Windows()). +func (plan BulkPlan) Describe(requests int) string { + return fmt.Sprintf("splitting %s..%s into %d requests of up to %s (%s); the result is assembled in memory", + formatBound(plan.Since), formatBound(plan.Until), requests, plan.WindowDays(), plan.Interval) +} + +// WindowParams returns params with s and u set to the window's bounds. +func WindowParams(params map[string]string, w Window) map[string]string { + out := make(map[string]string, len(params)+2) + for key, value := range params { + out[key] = value + } + out["s"] = strconv.FormatInt(w.Since.Unix(), 10) + out["u"] = strconv.FormatInt(w.Until.Unix(), 10) + return out +} + +// GetMetricBulkSplit fetches a bulk metric window by window and merges the +// points. If the API rejects a window as too large, the window is halved for +// the rest of the range, up to maxWindowShrinks times. Any other failure +// aborts the whole fetch: a partial result would look complete to a script. +func (c *Client) GetMetricBulkSplit(ctx context.Context, path string, params map[string]string, repeated map[string][]string, plan BulkPlan, progress io.Writer) (*BulkResponse, error) { + if progress == nil { + progress = io.Discard + } + merged := &BulkResponse{} + // Adjacent windows share no timestamp (the API rounds s down and includes + // only intervals ending by u), so index is a guard: should a timestamp + // repeat, the later response, which is at least as fresh, replaces it. + index := map[int64]int{} + shrinks, fetched := 0, 0 + // The windows are laid out once and again only after a halving, which + // changes every window that remains. + windows := plan.Windows() + for len(windows) > 0 { + w := windows[0] + // One line per window: each goes through the SDK's retries and may + // wait on a rate limit, so a long split fetch should not fall silent. + fmt.Fprintf(progress, "window %d/%d %s..%s\n", fetched+1, fetched+len(windows), formatBound(w.Since), formatBound(w.Until)) + resp, err := c.GetMetricBulk(ctx, path, WindowParams(params, w), repeated) + if err != nil { + // Halve the range the API actually rejected: near the end of the + // range it is shorter than plan.Window, and halving the nominal + // window could repeat the same request. A window shorter than one + // interval would return no points and turn the rest of the fetch + // into silent gaps, so halving stops there. + if halved := w.Until.Sub(w.Since) / 2; IsRangeTooLarge(err) && shrinks < maxWindowShrinks && halved >= intervalDuration(plan.Interval) { + shrinks++ + plan.Window = halved + windows = BulkPlan{Since: w.Since, Until: plan.Until, Interval: plan.Interval, Window: plan.Window}.Windows() + continue + } + // Nothing has been printed, so the whole command is what to retry. + // The total is an estimate: a halved window changes how many remain. + remedy := "retry the command" + if IsRangeTooLarge(err) { + remedy = "shorten the range or use a coarser --interval" + } + return nil, fmt.Errorf("window %d of about %d (%s..%s): %w; the fetch is aborted and no data was written (%d windows had succeeded), %s", + fetched+1, fetched+len(windows), formatBound(w.Since), formatBound(w.Until), err, fetched, remedy) + } + fetched++ + for _, point := range resp.Data { + if i, ok := index[point.T]; ok { + merged.Data[i] = point + continue + } + index[point.T] = len(merged.Data) + merged.Data = append(merged.Data, point) + } + windows = windows[1:] + } + if merged.Data == nil { + merged.Data = []BulkDataPoint{} + } + return merged, nil +} + +// IsRangeTooLarge reports the API's rejection of a window as too long ("too +// large time range (...) requested"). Other 400s, including other complaints +// about the range, are not treated as a reason to halve or to suggest --split: +// that would spend requests and replace the API's message with one about +// window size. +func IsRangeTooLarge(err error) bool { + var apiErr *glassnode.APIError + return errors.As(err, &apiErr) && apiErr.StatusCode == 400 && strings.Contains(strings.ToLower(apiErr.Detail), "too large") +} + +// formatBound shows a window bound as a date when it is at midnight UTC and +// as an RFC 3339 timestamp otherwise, both of which --since accepts, so a +// retry of an hourly or 10-minute fetch resumes exactly at the bound. +func formatBound(t time.Time) string { + t = t.UTC() + if t.Equal(t.Truncate(24 * time.Hour)) { + return t.Format(time.DateOnly) + } + return t.Format(time.RFC3339) +} + +func days(d time.Duration) string { + return fmt.Sprintf("%d days", int(d.Hours()/24)) +} diff --git a/internal/api/bulk_split_test.go b/internal/api/bulk_split_test.go new file mode 100644 index 0000000..e14240a --- /dev/null +++ b/internal/api/bulk_split_test.go @@ -0,0 +1,374 @@ +package api + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "strconv" + "strings" + "sync/atomic" + "testing" + "time" + + glassnode "github.com/glassnode/glassnode-api-go-client" +) + +var ( + day = 24 * time.Hour + planNow = time.Date(2026, 10, 8, 12, 0, 0, 0, time.UTC) + planSince = time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC) +) + +func unix(t time.Time) string { return strconv.FormatInt(t.Unix(), 10) } + +func TestPlanBulk(t *testing.T) { + t.Run("requires since", func(t *testing.T) { + _, err := PlanBulk(map[string]string{}, planNow) + if err == nil || !strings.Contains(err.Error(), "--since") || !strings.Contains(err.Error(), "31 days") || !strings.Contains(err.Error(), "--split fetches") { + t.Fatalf("err = %v", err) + } + }) + t.Run("an unknown interval is refused before planning", func(t *testing.T) { + for _, interval := range []string{"1d", "24H", "weekly"} { + _, err := PlanBulk(map[string]string{"s": unix(planSince), "i": interval}, planNow) + if err == nil || !strings.Contains(err.Error(), "unknown --interval") || !strings.Contains(err.Error(), "1month") { + t.Errorf("%q: err = %v", interval, err) + } + } + }) + t.Run("defaults and window by resolution", func(t *testing.T) { + for interval, want := range map[string]time.Duration{"": 31 * day, "24h": 31 * day, "10m": 10 * day, "1h": 10 * day, "1w": 93 * day, "1month": 93 * day} { + params := map[string]string{"s": unix(planSince)} + if interval != "" { + params["i"] = interval + } + plan, err := PlanBulk(params, planNow) + if err != nil || plan.Window != want || !plan.Until.Equal(planNow) { + t.Errorf("%q: plan=%+v err=%v", interval, plan, err) + } + } + }) + t.Run("a range exactly one window long is one request", func(t *testing.T) { + for interval, window := range map[string]time.Duration{"24h": 31 * day, "1h": 10 * day, "1w": 93 * day, "1month": 93 * day} { + now := planNow.Add(123456789 * time.Nanosecond) // a clock read after --since was parsed + plan, err := PlanBulk(map[string]string{"s": unix(planNow.Add(-window)), "i": interval}, now) + if err != nil || len(plan.Windows()) != 1 || !plan.Until.Equal(planNow) { + t.Errorf("%s: windows=%v err=%v, want one window ending at %v", interval, plan.Windows(), err, planNow) + } + } + }) + // The API rounds s down to the start of its interval: at 1w, --since 93d + // on a Wednesday counts from the Monday before and needs two requests. + t.Run("since is rounded down to the start of its interval", func(t *testing.T) { + now := time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC) // a Tuesday + plan, err := PlanBulk(map[string]string{"s": unix(now.Add(-93 * day)), "i": "1w"}, now) + if err != nil { + t.Fatal(err) + } + monday := time.Date(2026, 7, 6, 0, 0, 0, 0, time.UTC) + windows := plan.Windows() + if !plan.Since.Equal(monday) || len(windows) != 2 || !windows[0].Until.Equal(monday.Add(91*day)) || !windows[1].Until.Equal(now) { + t.Fatalf("since=%v windows=%v, want 2026-07-06 and a 91-day first window", plan.Since, windows) + } + }) + t.Run("until in the future is capped at now", func(t *testing.T) { + plan, err := PlanBulk(map[string]string{"s": unix(planSince), "u": unix(planNow.Add(400 * day))}, planNow) + if err != nil || !plan.Until.Equal(planNow) { + t.Fatalf("plan=%+v err=%v", plan, err) + } + }) + t.Run("since must precede until", func(t *testing.T) { + if _, err := PlanBulk(map[string]string{"s": unix(planNow), "u": unix(planSince)}, planNow); err == nil || !strings.Contains(err.Error(), "must be before --until") { + t.Fatalf("since >= until: %v", err) + } + // Without --until the bound is now, and the message must not name --until. + _, err := PlanBulk(map[string]string{"s": unix(planNow.Add(day))}, planNow) + if err == nil || !strings.Contains(err.Error(), "is not in the past") || strings.Contains(err.Error(), "--until") { + t.Fatalf("since in the future: %v", err) + } + // A future --until is capped at now; the error must not report the + // capped value as if the user had given it. + _, err = PlanBulk(map[string]string{"s": unix(planNow.Add(day)), "u": unix(planNow.Add(2 * day))}, planNow) + if err == nil || !strings.Contains(err.Error(), "is not in the past") || strings.Contains(err.Error(), "--until") { + t.Fatalf("since and until in the future: %v", err) + } + }) +} + +// When the rejected window is the shorter tail of the range, halving the +// nominal window would repeat the identical request; the rejected range +// itself is halved instead. +func TestGetMetricBulkSplitHalvesTheRejectedRange(t *testing.T) { + plan := BulkPlan{Since: planSince, Until: planSince.Add(12 * day), Interval: "24h", Window: 31 * day} + server, calls, windows := bulkServer(t, 10*day, 0) + resp, err := newTestClient(server).GetMetricBulkSplit(context.Background(), "/market/price", map[string]string{"s": unix(plan.Since)}, nil, plan, nil) + if err != nil { + t.Fatal(err) + } + if calls.Load() != 3 || len(resp.Data) != 12 { + t.Fatalf("calls=%d points=%d, want 3 (12d rejected, then 6d + 6d) and 12", calls.Load(), len(resp.Data)) + } + if (*windows)[1].Until.Sub((*windows)[1].Since) != 6*day { + t.Errorf("windows=%v, want the second one to be 6 days", *windows) + } +} + +func TestBulkPlanWindows(t *testing.T) { + plan := BulkPlan{Since: planSince, Until: planSince.Add(70 * day), Interval: "24h", Window: 31 * day} + windows := plan.Windows() + if len(windows) != 3 { + t.Fatalf("windows = %v", windows) + } + for i := 1; i < len(windows); i++ { + if !windows[i].Since.Equal(windows[i-1].Until) { + t.Errorf("gap or overlap between %v and %v", windows[i-1], windows[i]) + } + } + if !windows[0].Since.Equal(planSince) || !windows[2].Until.Equal(plan.Until) || windows[2].Until.Sub(windows[2].Since) != 8*day { + t.Errorf("bounds: %v", windows) + } + if !strings.Contains(plan.RangeError(len(plan.Windows())).Error(), "3 requests") || !strings.Contains(plan.Describe(len(plan.Windows())), "3 requests of up to 31 days (24h)") { + t.Errorf("messages: %v / %s", plan.RangeError(len(plan.Windows())), plan.Describe(len(plan.Windows()))) + } + if one := (BulkPlan{Since: planSince, Until: planSince.Add(day), Window: 31 * day}).Windows(); len(one) != 1 { + t.Errorf("one-day range should be one window, got %d", len(one)) + } +} + +// bulkServer answers one daily BTC point per day of the requested window and +// records the windows it was asked for. Ranges above maxRange get the API's +// "too large time range" error. +func bulkServer(t *testing.T, maxRange time.Duration, failAt int) (*httptest.Server, *atomic.Int32, *[]Window) { + t.Helper() + var calls atomic.Int32 + var windows []Window + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + n := int(calls.Add(1)) + s, _ := strconv.ParseInt(r.URL.Query().Get("s"), 10, 64) + u, _ := strconv.ParseInt(r.URL.Query().Get("u"), 10, 64) + since, until := time.Unix(s, 0).UTC(), time.Unix(u, 0).UTC() + windows = append(windows, Window{since, until}) + if until.Sub(since) > maxRange { + w.WriteHeader(http.StatusBadRequest) + fmt.Fprintf(w, `{"message":"too large time range (%s) requested (maximum allowed is %s)"}`, until.Sub(since), maxRange) + return + } + if n == failAt { + w.WriteHeader(http.StatusForbidden) + fmt.Fprint(w, `{"message":"metric not in your plan"}`) + return + } + var points []string + for day := since.Truncate(24 * time.Hour); day.Add(24*time.Hour).Compare(until) <= 0; day = day.Add(24 * time.Hour) { + points = append(points, fmt.Sprintf(`{"t":%d,"bulk":[{"a":"BTC","v":1}]}`, day.Unix())) + } + fmt.Fprintf(w, `{"data":[%s]}`, strings.Join(points, ",")) + })) + t.Cleanup(server.Close) + return server, &calls, &windows +} + +func newTestClient(server *httptest.Server) *Client { + client := NewClient("key", "") + client.baseURL = server.URL + client.httpClient = server.Client() + return client +} + +func TestGetMetricBulkSplit(t *testing.T) { + plan := BulkPlan{Since: planSince, Until: planSince.Add(70 * day), Interval: "24h", Window: 31 * day} + server, calls, _ := bulkServer(t, 31*day, 0) + var progress strings.Builder + resp, err := newTestClient(server).GetMetricBulkSplit(context.Background(), "/market/price", map[string]string{"s": unix(plan.Since)}, nil, plan, &progress) + if err != nil { + t.Fatal(err) + } + if got := progress.String(); !strings.Contains(got, "window 1/3 2024-01-01..2024-02-01") || !strings.Contains(got, "window 3/3 ") { + t.Errorf("progress:\n%s", got) + } + if calls.Load() != 3 || len(resp.Data) != 70 { + t.Fatalf("calls=%d points=%d, want 3 and 70", calls.Load(), len(resp.Data)) + } + for i := 1; i < len(resp.Data); i++ { + if resp.Data[i].T-resp.Data[i-1].T != int64(day.Seconds()) { + t.Fatalf("gap or duplicate at %d: %d -> %d", i, resp.Data[i-1].T, resp.Data[i].T) + } + } +} + +func TestGetMetricBulkSplitShrinksWindow(t *testing.T) { + plan := BulkPlan{Since: planSince, Until: planSince.Add(31 * day), Interval: "24h", Window: 31 * day} + // The server allows only 10 days: 31 -> 15.5 -> 7.75 days, two shrinks. + server, calls, windows := bulkServer(t, 10*day, 0) + resp, err := newTestClient(server).GetMetricBulkSplit(context.Background(), "/market/price", map[string]string{"s": unix(plan.Since)}, nil, plan, nil) + if err != nil { + t.Fatal(err) + } + if len(resp.Data) != 31 { + t.Fatalf("points=%d, want 31", len(resp.Data)) + } + if (*windows)[0].Until.Sub((*windows)[0].Since) != 31*day || (*windows)[2].Until.Sub((*windows)[2].Since) > 10*day { + t.Errorf("window sizes: %v", *windows) + } + t.Logf("calls=%d", calls.Load()) + + // A limit below what three halvings reach is an error. + tiny, _, _ := bulkServer(t, 2*day, 0) + if _, err := newTestClient(tiny).GetMetricBulkSplit(context.Background(), "/market/price", map[string]string{"s": unix(plan.Since)}, nil, plan, nil); err == nil || !strings.Contains(err.Error(), "too large time range") { + t.Fatalf("err = %v", err) + } +} + +func TestGetMetricBulkSplitAbortsOnFailure(t *testing.T) { + plan := BulkPlan{Since: planSince, Until: planSince.Add(70 * day), Interval: "24h", Window: 31 * day} + server, calls, _ := bulkServer(t, 31*day, 2) + resp, err := newTestClient(server).GetMetricBulkSplit(context.Background(), "/market/price", map[string]string{"s": unix(plan.Since)}, nil, plan, nil) + if err == nil || resp != nil { + t.Fatalf("expected an error and no data, got %v %v", resp, err) + } + for _, want := range []string{"window 2 of about 3", "2024-02-01..2024-03-03", "metric not in your plan", "no data was written (1 windows had succeeded), retry the command"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error %q lacks %q", err, want) + } + } + if calls.Load() != 2 { + t.Errorf("calls=%d, want 2 (no requests after the failure)", calls.Load()) + } +} + +func TestWindowParams(t *testing.T) { + params := map[string]string{"s": "1", "u": "2", "i": "24h"} + w := Window{planSince, planSince.Add(day)} + out := WindowParams(params, w) + if out["s"] != unix(planSince) || out["u"] != unix(planSince.Add(day)) || out["i"] != "24h" || params["s"] != "1" { + t.Errorf("out=%v params=%v", out, params) + } +} + +// Should windows ever overlap, the later window's point replaces the earlier +// one instead of being dropped. +func TestGetMetricBulkSplitKeepsLaterPointOnOverlap(t *testing.T) { + plan := BulkPlan{Since: planSince, Until: planSince.Add(2 * day), Interval: "24h", Window: day} + var calls atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + boundary := planSince.Add(day).Unix() + if calls.Add(1) == 1 { + // The first window wrongly repeats the boundary point with one entry. + fmt.Fprintf(w, `{"data":[{"t":%d,"bulk":[{"a":"BTC","v":1}]},{"t":%d,"bulk":[{"a":"BTC","v":2}]}]}`, planSince.Unix(), boundary) + return + } + fmt.Fprintf(w, `{"data":[{"t":%d,"bulk":[{"a":"BTC","v":2},{"a":"ETH","v":3}]}]}`, boundary) + })) + t.Cleanup(server.Close) + resp, err := newTestClient(server).GetMetricBulkSplit(context.Background(), "/market/price", map[string]string{"s": unix(plan.Since)}, nil, plan, nil) + if err != nil { + t.Fatal(err) + } + if len(resp.Data) != 2 || len(resp.Data[1].Bulk) != 2 { + t.Fatalf("points=%d boundary entries=%d, want 2 and 2 (from the later window)", len(resp.Data), len(resp.Data[1].Bulk)) + } +} + +func TestIsRangeTooLarge(t *testing.T) { + for _, tt := range []struct { + status int + detail string + want bool + }{ + {400, "too large time range (155664h0m0s) requested (maximum allowed is 744h0m0s)", true}, + {400, "Requested Time Range is too large", true}, + {400, "parameter a=NOPE is invalid", false}, + {400, "invalid time range: s must be before u", false}, + {403, "time range not in your plan", false}, + } { + if got := IsRangeTooLarge(&glassnode.APIError{StatusCode: tt.status, Detail: tt.detail}); got != tt.want { + t.Errorf("IsRangeTooLarge(%d %q) = %v, want %v", tt.status, tt.detail, got, tt.want) + } + } +} + +func TestGetMetricBulkSplitErrorAfterHalving(t *testing.T) { + // Hourly data from 06:00: 20 days need two 10-day windows. The server + // accepts only 5 days, so the window is halved once, then the third + // request fails. + since := planSince.Add(6 * time.Hour) + plan := BulkPlan{Since: since, Until: since.Add(20 * day), Interval: "1h", Window: 10 * day} + server, _, _ := bulkServer(t, 5*day, 4) // call 1 too large; calls 2-3 succeed; call 4 fails + _, err := newTestClient(server).GetMetricBulkSplit(context.Background(), "/market/price", map[string]string{"s": unix(plan.Since)}, nil, plan, nil) + if err == nil { + t.Fatal("expected an error") + } + for _, want := range []string{"window 3 of about 4", "2024-01-11T06:00:00Z..2024-01-16T06:00:00Z", "retry the command"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error %q lacks %q", err, want) + } + } +} + +func TestFormatBound(t *testing.T) { + if got := formatBound(planSince); got != "2024-01-01" { + t.Errorf("midnight: %s", got) + } + if got := formatBound(planSince.Add(90 * time.Minute)); got != "2024-01-01T01:30:00Z" { + t.Errorf("intraday: %s", got) + } +} + +// At monthly resolution a 93-day window may be halved once (46.5 days, which +// rounds down to one month) but not twice (15.5 days would return no points). +func TestGetMetricBulkSplitDoesNotHalveBelowAnInterval(t *testing.T) { + plan := BulkPlan{Since: planSince, Until: planSince.Add(93 * day), Interval: "1month", Window: 93 * day} + server, calls, windows := bulkServer(t, 20*day, 0) // even one month is rejected + _, err := newTestClient(server).GetMetricBulkSplit(context.Background(), "/market/price", map[string]string{"s": unix(plan.Since)}, nil, plan, nil) + if err == nil || !strings.Contains(err.Error(), "time range") || !strings.Contains(err.Error(), "coarser --interval") || strings.Contains(err.Error(), "retry") { + t.Fatalf("err = %v, want the API's too-large error and a remedy other than retrying", err) + } + if calls.Load() != 2 || !(*windows)[1].Until.Equal(time.Date(2024, 2, 1, 0, 0, 0, 0, time.UTC)) { + t.Errorf("calls=%d windows=%v, want 93d then January and no shorter window", calls.Load(), *windows) + } +} + +func TestIntervalStart(t *testing.T) { + at := time.Date(2026, 10, 15, 13, 47, 5, 0, time.UTC) // a Thursday + for interval, want := range map[string]time.Time{ + "10m": time.Date(2026, 10, 15, 13, 40, 0, 0, time.UTC), + "1h": time.Date(2026, 10, 15, 13, 0, 0, 0, time.UTC), + "24h": time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC), + "1w": time.Date(2026, 10, 12, 0, 0, 0, 0, time.UTC), + "1month": time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC), + } { + if got := intervalStart(at, interval); !got.Equal(want) { + t.Errorf("%s: %v, want %v", interval, got, want) + } + } + if monday := time.Date(2026, 10, 12, 0, 0, 0, 0, time.UTC); !intervalStart(monday, "1w").Equal(monday) { + t.Errorf("a Monday is its own week start") + } +} + +// Inner window bounds fall on interval starts, so the API rounds neither of +// them and each window is exactly the range the plan checked. +func TestBulkPlanWindowsAlignToIntervals(t *testing.T) { + monday := time.Date(2026, 1, 5, 0, 0, 0, 0, time.UTC) + plan := BulkPlan{Since: monday, Until: monday.Add(300 * day), Interval: "1w", Window: 93 * day} + windows := plan.Windows() + for i, w := range windows { + if i < len(windows)-1 && (w.Until.Weekday() != time.Monday || w.Until.Sub(w.Since) > 93*day) { + t.Errorf("window %d = %v, want a Monday bound at most 93 days on", i, w) + } + if i > 0 && !w.Since.Equal(windows[i-1].Until) { + t.Errorf("gap or overlap between %v and %v", windows[i-1], w) + } + } +} + +func TestRangeMessagesKeepIntradayBounds(t *testing.T) { + since := planSince.Add(6 * time.Hour) + plan := BulkPlan{Since: since, Until: since.Add(19 * day), Interval: "1h", Window: 10 * day} + for _, msg := range []string{plan.RangeError(len(plan.Windows())).Error(), plan.Describe(len(plan.Windows()))} { + if !strings.Contains(msg, "2024-01-01T06:00:00Z..2024-01-20T06:00:00Z") { + t.Errorf("message drops the time of day: %s", msg) + } + } +} diff --git a/internal/api/client.go b/internal/api/client.go index 9bd8589..c616b15 100644 --- a/internal/api/client.go +++ b/internal/api/client.go @@ -2,50 +2,115 @@ package api import ( "context" - "errors" "fmt" "io" "net/http" "net/url" "os" + "sync" "time" + glassnode "github.com/glassnode/glassnode-api-go-client" "github.com/glassnode/glassnode-cli/internal/config" "github.com/glassnode/glassnode-cli/internal/oauth" "github.com/glassnode/glassnode-cli/internal/version" ) -const ( - defaultHTTPTimeout = 1 * time.Minute -) - -var ( - baseURL = "https://api.glassnode.com" - - // stderr is where warning messages are written. Overridable in tests. - stderr io.Writer = os.Stderr -) +var stderr io.Writer = os.Stderr +// Client adapts the public SDK to CLI authentication, configuration and output. +// HTTP requests, retries, errors and endpoint decoding are owned by the SDK. type Client struct { baseURL string apiKey string bearerToken string httpClient *http.Client + init sync.Once + sdkClient *glassnode.Client + initErr error } func NewClient(apiKey, bearerToken string) *Client { - if u := os.Getenv("GLASSNODE_BASE_URL"); u != "" { - baseURL = u + baseURL := os.Getenv("GLASSNODE_BASE_URL") + if baseURL == "" { + baseURL = glassnode.DefaultBaseURL + } + return &Client{baseURL: baseURL, apiKey: apiKey, bearerToken: bearerToken, + httpClient: &http.Client{Timeout: time.Minute}} +} + +func (c *Client) sdk() (*glassnode.Client, error) { + c.init.Do(func() { + options := []glassnode.Option{ + glassnode.WithBaseURL(c.baseURL), + glassnode.WithUserAgent("glassnode-cli-" + version.Version), + } + key := c.apiKey + if c.bearerToken != "" { + options = append(options, glassnode.WithTokenSource(&sessionTokens{token: c.bearerToken})) + key = "" + } + options = append(options, glassnode.WithHTTPClient(c.httpClient)) + c.sdkClient, c.initErr = glassnode.NewClient(key, options...) + }) + return c.sdkClient, c.initErr +} + +// AuthHeader describes the header that carries the credential, for the +// dry-run note. The SDK sends the API key in X-Api-Key rather than in the +// URL, so it stays out of proxy and access logs; dry-run output therefore +// shows the URL without it. +func (c *Client) AuthHeader() string { + if c.bearerToken != "" { + return "Authorization header (Bearer scheme)" + } + return "X-Api-Key header" +} + +// sessionTokens hands the CLI's OAuth session token to the SDK and refreshes +// it when the API rejects it, so a token revoked before its expiry is replaced +// once without teaching the SDK about config files or OAuth persistence. +type sessionTokens struct { + mu sync.Mutex + token string +} + +// Token returns the current session token. +func (s *sessionTokens) Token(ctx context.Context) (string, error) { + if err := ctx.Err(); err != nil { + return "", err } + s.mu.Lock() + defer s.mu.Unlock() + return s.token, nil +} - return &Client{ - baseURL: baseURL, - apiKey: apiKey, - bearerToken: bearerToken, - httpClient: &http.Client{ - Timeout: defaultHTTPTimeout, - }, +// Refresh exchanges the stored refresh token for a new access token after a +// 401 and keeps it for the following requests. A failure, including an +// expired session, reaches the user with its cause through the SDK's +// AuthError, so nothing is printed here. +func (s *sessionTokens) Refresh(ctx context.Context) (string, error) { + fresh, err := oauth.ForceRefreshAccessToken(ctx) + if err != nil { + return "", err } + s.mu.Lock() + s.token = fresh + s.mu.Unlock() + return fresh, nil +} + +func queryValues(params map[string]string, repeated map[string][]string) url.Values { + q := url.Values{} + for key, value := range params { + q.Set(key, value) + } + for key, values := range repeated { + for _, value := range values { + q.Add(key, value) + } + } + return q } // ResolveAPIKey returns the first non-empty value from: @@ -112,86 +177,3 @@ func RequireAPIKey(flagValue string) (string, error) { } return key, nil } - -func (c *Client) Do(ctx context.Context, method, path string, params map[string]string) ([]byte, error) { - return c.DoWithRepeatedParams(ctx, method, path, params, nil) -} - -func (c *Client) DoWithRepeatedParams(ctx context.Context, method, path string, params map[string]string, repeatedParams map[string][]string) ([]byte, error) { - u, err := c.buildURL(path, params, repeatedParams) - if err != nil { - return nil, err - } - - body, status, err := c.sendOnce(ctx, method, u) - if err != nil { - return nil, err - } - - // If we were using an OAuth bearer and the server returned Unauthorized, try exactly one refresh+retry. - // This handles the case where the access token was revoked before its expiry - if status == http.StatusUnauthorized && c.bearerToken != "" { - fresh, rerr := oauth.ForceRefreshAccessToken(ctx) - if rerr == nil && fresh != "" && fresh != c.bearerToken { - c.bearerToken = fresh - body, status, err = c.sendOnce(ctx, method, u) - if err != nil { - return nil, err - } - } else if rerr != nil && !errors.Is(rerr, oauth.ErrSessionExpired) { - _, _ = fmt.Fprintf(stderr, "warning: refresh after HTTP %d failed: %v\n", http.StatusUnauthorized, rerr) - } - } - - if status < http.StatusOK || status >= http.StatusMultipleChoices { - if status == http.StatusTooManyRequests { - return nil, fmt.Errorf("HTTP %d: rate limit exceeded. %s", status, string(body)) - } - return nil, fmt.Errorf("HTTP %d: %s", status, string(body)) - } - return body, nil -} - -func (c *Client) buildURL(path string, params map[string]string, repeatedParams map[string][]string) (*url.URL, error) { - u, err := url.Parse(c.baseURL + path) - if err != nil { - return nil, fmt.Errorf("parsing URL: %w", err) - } - q := u.Query() - if c.bearerToken == "" { - q.Set("api_key", c.apiKey) - } - for k, v := range params { - q.Set(k, v) - } - for k, vals := range repeatedParams { - for _, v := range vals { - q.Add(k, v) - } - } - u.RawQuery = q.Encode() - return u, nil -} - -func (c *Client) sendOnce(ctx context.Context, method string, u *url.URL) ([]byte, int, error) { - req, err := http.NewRequestWithContext(ctx, method, u.String(), nil) - if err != nil { - return nil, 0, fmt.Errorf("creating request: %w", err) - } - req.Header.Set("User-Agent", "glassnode-cli-"+version.Version) - if c.bearerToken != "" { - req.Header.Set("Authorization", "Bearer "+c.bearerToken) - } - - resp, err := c.httpClient.Do(req) - if err != nil { - return nil, 0, fmt.Errorf("executing request: %w", err) - } - defer func() { _ = resp.Body.Close() }() - - body, err := io.ReadAll(resp.Body) - if err != nil { - return nil, resp.StatusCode, fmt.Errorf("reading response: %w", err) - } - return body, resp.StatusCode, nil -} diff --git a/internal/api/client_test.go b/internal/api/client_test.go index d7127a2..d2b51dc 100644 --- a/internal/api/client_test.go +++ b/internal/api/client_test.go @@ -3,14 +3,18 @@ package api import ( "context" _ "embed" + "encoding/json" + "errors" "net/http" "net/http/httptest" + "net/url" "os" "strings" "sync" "testing" "time" + glassnode "github.com/glassnode/glassnode-api-go-client" "github.com/glassnode/glassnode-cli/internal/config" "github.com/glassnode/glassnode-cli/internal/oauth" "github.com/glassnode/glassnode-cli/internal/testhelper" @@ -172,12 +176,13 @@ func TestResolveAuth_RefreshesExpiredAccessToken(t *testing.T) { }) } -func TestDo_SendsCorrectURL(t *testing.T) { - var capturedURL string +func TestRequest_SendsCorrectURL(t *testing.T) { + var capturedURL, capturedKey string server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { capturedURL = r.URL.String() + capturedKey = r.Header.Get("X-Api-Key") w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte("{}")) + _, _ = w.Write([]byte("[]")) })) defer server.Close() @@ -185,25 +190,25 @@ func TestDo_SendsCorrectURL(t *testing.T) { client.baseURL = server.URL client.httpClient = server.Client() - _, err := client.Do(context.Background(), "GET", "/v1/test", map[string]string{"a": "b"}) + _, err := client.GetMetric(context.Background(), "/test", map[string]string{"a": "b"}) if err != nil { - t.Fatalf("Do: %v", err) + t.Fatalf("GetMetric: %v", err) } - if !strings.Contains(capturedURL, "api_key=my-api-key") { - t.Errorf("URL %q missing api_key param", capturedURL) + if capturedKey != "my-api-key" || strings.Contains(capturedURL, "api_key") { + t.Errorf("URL %q header %q: the key must travel in X-Api-Key only", capturedURL, capturedKey) } if !strings.Contains(capturedURL, "a=b") { t.Errorf("URL %q missing a=b param", capturedURL) } } -func TestDo_SendsBearerAuthorization(t *testing.T) { +func TestRequest_SendsBearerAuthorization(t *testing.T) { var authHdr, capturedURL string server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { authHdr = r.Header.Get("Authorization") capturedURL = r.URL.String() w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte("{}")) + _, _ = w.Write([]byte("[]")) })) defer server.Close() @@ -211,9 +216,9 @@ func TestDo_SendsBearerAuthorization(t *testing.T) { client.baseURL = server.URL client.httpClient = server.Client() - _, err := client.Do(context.Background(), "GET", "/v1/test", map[string]string{"a": "b"}) + _, err := client.GetMetric(context.Background(), "/test", map[string]string{"a": "b"}) if err != nil { - t.Fatalf("Do: %v", err) + t.Fatalf("GetMetric: %v", err) } if authHdr != "Bearer secret-token" { t.Errorf("Authorization header %q, want Bearer secret-token", authHdr) @@ -223,7 +228,7 @@ func TestDo_SendsBearerAuthorization(t *testing.T) { } } -func TestDo_401TriggersRefreshAndRetry(t *testing.T) { +func TestRequest_401TriggersRefreshAndRetry(t *testing.T) { withTempHome(t, func() { var mu sync.Mutex var authHdrs []string @@ -240,7 +245,7 @@ func TestDo_401TriggersRefreshAndRetry(t *testing.T) { return } w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte(`{"ok":true}`)) + _, _ = w.Write([]byte(`[{"t":1,"v":1}]`)) })) defer api.Close() @@ -264,12 +269,12 @@ func TestDo_401TriggersRefreshAndRetry(t *testing.T) { client.baseURL = api.URL client.httpClient = api.Client() - body, err := client.Do(context.Background(), "GET", "/v1/anything", nil) + points, err := client.GetMetric(context.Background(), "/anything", nil) if err != nil { - t.Fatalf("Do: %v", err) + t.Fatalf("GetMetric: %v", err) } - if !strings.Contains(string(body), "ok") { - t.Errorf("body %q, want contents including 'ok'", string(body)) + if len(points) != 1 { + t.Errorf("points = %v, want the retried response", points) } if len(authHdrs) != 2 { t.Fatalf("expected 2 API attempts, got %d", len(authHdrs)) @@ -277,10 +282,16 @@ func TestDo_401TriggersRefreshAndRetry(t *testing.T) { if authHdrs[0] != "Bearer stale" || authHdrs[1] != "Bearer fresh" { t.Errorf("auth headers %v", authHdrs) } + if _, err := client.GetMetric(context.Background(), "/anything", nil); err != nil { + t.Fatal(err) + } + if len(authHdrs) != 3 || authHdrs[2] != "Bearer fresh" { + t.Errorf("refreshed token was not reused: %v", authHdrs) + } }) } -func TestDo_401WithAPIKey_NoRetry(t *testing.T) { +func TestRequest_401WithAPIKey_NoRetry(t *testing.T) { // When using an API key (not a bearer), a 401 must NOT trigger a refresh attempt; we just // surface the error. This guards against silently attempting OAuth refreshes for users who // never ran gn login. @@ -300,7 +311,7 @@ func TestDo_401WithAPIKey_NoRetry(t *testing.T) { client.baseURL = server.URL client.httpClient = server.Client() - _, err := client.Do(context.Background(), "GET", "/v1/x", nil) + _, err := client.GetMetric(context.Background(), "/x", nil) if err == nil { t.Fatal("expected error") } @@ -313,7 +324,7 @@ func TestDo_401WithAPIKey_NoRetry(t *testing.T) { }) } -func TestDo_Non2xxReturnsError(t *testing.T) { +func TestRequest_Non2xxReturnsError(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusInternalServerError) _, _ = w.Write([]byte("server error")) @@ -324,18 +335,18 @@ func TestDo_Non2xxReturnsError(t *testing.T) { client.baseURL = server.URL client.httpClient = server.Client() - _, err := client.Do(context.Background(), "GET", "/v1/test", nil) + _, err := client.GetMetric(context.Background(), "/test", nil) if err == nil { t.Error("expected error for 500 status") } } -func TestDoWithRepeatedParams(t *testing.T) { +func TestRequest_RepeatedParams(t *testing.T) { var capturedURL string server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { capturedURL = r.URL.String() w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte("{}")) + _, _ = w.Write([]byte(`{"data":[]}`)) })) defer server.Close() @@ -344,9 +355,9 @@ func TestDoWithRepeatedParams(t *testing.T) { client.httpClient = server.Client() repeated := map[string][]string{"a": {"x", "y"}} - _, err := client.DoWithRepeatedParams(context.Background(), "GET", "/v1/test", nil, repeated) + _, err := client.GetMetricBulk(context.Background(), "/test", map[string]string{"s": "1"}, repeated) if err != nil { - t.Fatalf("DoWithRepeatedParams: %v", err) + t.Fatalf("GetMetricBulk: %v", err) } if !strings.Contains(capturedURL, "a=x") || !strings.Contains(capturedURL, "a=y") { t.Errorf("URL %q should contain a=x and a=y", capturedURL) @@ -360,8 +371,8 @@ func TestBuildURL(t *testing.T) { if err != nil { t.Fatalf("BuildURL: %v", err) } - if !strings.Contains(got, "api_key=test-key") { - t.Errorf("URL %q missing api_key", got) + if strings.Contains(got, "api_key") || strings.Contains(got, "test-key") { + t.Errorf("URL %q must not contain the API key", got) } if !strings.Contains(got, "p=v") { t.Errorf("URL %q missing p=v", got) @@ -371,21 +382,18 @@ func TestBuildURL(t *testing.T) { } } -func TestRedactAPIKeyFromURL(t *testing.T) { - raw := "https://api.example.com/v1/path?api_key=secret123&a=b" - redacted, err := RedactAPIKeyFromURL(raw) - if err != nil { - t.Fatalf("RedactAPIKeyFromURL: %v", err) - } - if strings.Contains(redacted, "secret123") { - t.Errorf("redacted URL should not contain secret: %q", redacted) - } - // Placeholder may be URL-encoded as %2A%2A%2A - if !strings.Contains(redacted, "api_key=***") && !strings.Contains(redacted, "api_key=%2A%2A%2A") { - t.Errorf("redacted URL should contain api_key redaction: %q", redacted) +func TestBuildURL_BaseURLAsTheSDKSeesIt(t *testing.T) { + client := NewClient("test-key", "") + client.baseURL = "https://api.example.com/" + got, err := client.BuildURL("/v1/path", nil, nil) + if err != nil || got != "https://api.example.com/v1/path" { + t.Errorf("trailing slash: got %q, %v", got, err) } - if !strings.Contains(redacted, "a=b") { - t.Errorf("redacted URL should preserve other params: %q", redacted) + bad := NewClient("test-key", "") + bad.baseURL = "api.example.com" + var inputErr *glassnode.InputError + if _, err := bad.BuildURL("/v1/path", nil, nil); !errors.As(err, &inputErr) { + t.Errorf("a base URL the SDK rejects must fail the dry run too, got %v", err) } } @@ -566,8 +574,8 @@ func TestGetMetric(t *testing.T) { if points[0].T != 1230940800 { t.Errorf("got T %d, want 1230940800", points[0].T) } - if points[0].V != 2.4755000000000003 { - t.Errorf("got V %v, want 2.4755000000000003", points[0].V) + if points[0].V != json.Number("2.4755000000000003") { + t.Errorf("got V %v, want the API's number 2.4755000000000003 unchanged", points[0].V) } } @@ -582,7 +590,7 @@ func TestGetMetricBulk(t *testing.T) { client.baseURL = server.URL client.httpClient = server.Client() - resp, err := client.GetMetricBulk(context.Background(), "/market/price_usd_close", nil, nil) + resp, err := client.GetMetricBulk(context.Background(), "/market/price_usd_close", map[string]string{"s": "1770076800"}, nil) if err != nil { t.Fatalf("GetMetricBulk: %v", err) } @@ -598,7 +606,7 @@ func TestGetMetricBulk(t *testing.T) { if resp.Data[0].Bulk[0]["a"] != "BTC" { t.Errorf("got a=%v", resp.Data[0].Bulk[0]["a"]) } - if resp.Data[0].Bulk[0]["v"] != 1513125006987.5164 { + if resp.Data[0].Bulk[0]["v"] != json.Number("1513125006987.5164") { t.Errorf("got v=%v", resp.Data[0].Bulk[0]["v"]) } } @@ -667,7 +675,7 @@ func TestGetMetricBulk_InvalidJSONReturnsError(t *testing.T) { client.baseURL = server.URL client.httpClient = server.Client() - _, err := client.GetMetricBulk(context.Background(), "/market/price", nil, nil) + _, err := client.GetMetricBulk(context.Background(), "/market/price", map[string]string{"s": "1"}, nil) if err == nil { t.Error("expected error for invalid JSON") } @@ -746,7 +754,7 @@ func TestGetMetricBulk_EmptyData(t *testing.T) { client.baseURL = server.URL client.httpClient = server.Client() - resp, err := client.GetMetricBulk(context.Background(), "/market/price", nil, nil) + resp, err := client.GetMetricBulk(context.Background(), "/market/price", map[string]string{"s": "1"}, nil) if err != nil { t.Fatalf("GetMetricBulk: %v", err) } @@ -757,18 +765,18 @@ func TestGetMetricBulk_EmptyData(t *testing.T) { // 4xx with body (point 7) -func TestDo_4xxReturnsErrorWithBody(t *testing.T) { +func TestRequest_4xxReturnsErrorWithBody(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusUnauthorized) _, _ = w.Write([]byte(`{"error":"Invalid API key"}`)) })) defer server.Close() - client := NewClient("key", "") + client := NewClient("test-api-secret", "") client.baseURL = server.URL client.httpClient = server.Client() - _, err := client.Do(context.Background(), "GET", "/v1/test", nil) + _, err := client.GetMetric(context.Background(), "/test", nil) if err == nil { t.Fatal("expected error for 401 status") } @@ -780,36 +788,113 @@ func TestDo_4xxReturnsErrorWithBody(t *testing.T) { } } -// RedactAPIKeyFromURL edge cases (point 7) +func TestGetMetricBulk_KeepsAllSelectors(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"data":[{"t":1,"bulk":[ + {"a":"BTC","e":"binance","v":1}, + {"a":"ETH","category":"more_10y","v":null}]}]}`)) + })) + defer server.Close() + + client := NewClient("key", "") + client.baseURL = server.URL + client.httpClient = server.Client() -func TestRedactAPIKeyFromURL_NoAPIKeyParam(t *testing.T) { - raw := "https://api.example.com/v1/path?a=b" - redacted, err := RedactAPIKeyFromURL(raw) + resp, err := client.GetMetricBulk(context.Background(), "/distribution/balance_exchanges", map[string]string{"s": "1"}, nil) if err != nil { - t.Fatalf("RedactAPIKeyFromURL: %v", err) + t.Fatalf("GetMetricBulk: %v", err) + } + rows := resp.Data[0].Bulk + if rows[0]["e"] != "binance" || rows[0]["a"] != "BTC" || rows[0]["v"] != json.Number("1") { + t.Errorf("exchange selector lost: %v", rows[0]) } - if redacted != raw { - t.Errorf("URL without api_key should be unchanged: got %q", redacted) + if rows[1]["category"] != "more_10y" || rows[1]["v"] != nil { + t.Errorf("category selector lost: %v", rows[1]) } } -func TestRedactAPIKeyFromURL_EmptyAPIKey(t *testing.T) { - raw := "https://api.example.com/v1/path?api_key=&a=b" - redacted, err := RedactAPIKeyFromURL(raw) +// Selectors reach the request as given; the SDK validates them, so a +// malformed s is rejected before anything is sent. +func TestGetMetric_SelectorsReachTheRequest(t *testing.T) { + var query url.Values + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + query = r.URL.Query() + _, _ = w.Write([]byte(`{"data":[]}`)) + })) + defer server.Close() + client := NewClient("key", "") + client.baseURL = server.URL + client.httpClient = server.Client() + + _, err := client.GetMetricBulk(context.Background(), "/market/price", map[string]string{"s": "100", "u": "200", "i": "24h", "c": "USD", "network": "eth"}, map[string][]string{"a": {"BTC", "ETH"}, "e": {"binance", "coinbase"}}) if err != nil { - t.Fatalf("RedactAPIKeyFromURL: %v", err) + t.Fatal(err) } - if !strings.Contains(redacted, "a=b") { - t.Errorf("should preserve other params: %q", redacted) + if query.Get("s") != "100" || query.Get("u") != "200" || query.Get("i") != "24h" || query.Get("c") != "USD" || query.Get("network") != "eth" || len(query["a"]) != 2 || len(query["e"]) != 2 { + t.Errorf("query: %v", query) } - if !strings.Contains(redacted, "api_key=") { - t.Errorf("should still have api_key param: %q", redacted) + calls := 0 + server.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { calls++ }) + var inputErr *glassnode.InputError + if _, err := client.GetMetric(context.Background(), "/market/price", map[string]string{"s": "yesterday"}); !errors.As(err, &inputErr) || calls != 0 { + t.Errorf("err = %v, calls = %d; want an InputError and no request", err, calls) } } -func TestRedactAPIKeyFromURL_InvalidURL(t *testing.T) { - _, err := RedactAPIKeyFromURL("://invalid") - if err == nil { - t.Error("expected error for invalid URL") +func TestGetMetricBulk_PassesEntriesThrough(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/metrics/market/price/bulk" { + t.Errorf("path %s", r.URL.Path) + } + _, _ = w.Write([]byte(`{"data":[{"t":1,"bulk":[{"a":"BTC","period":7,"v":12345678901234567890},{"a":"ETH","v":{"o":1,"c":2}}]}]}`)) + })) + defer server.Close() + + client := NewClient("key", "") + client.baseURL = server.URL + client.httpClient = server.Client() + + resp, err := client.GetMetricBulk(context.Background(), "/market/price", map[string]string{"s": "1"}, nil) + if err != nil { + t.Fatalf("GetMetricBulk: %v", err) + } + encoded, err := json.Marshal(resp.Data[0].Bulk) + if err != nil { + t.Fatal(err) + } + if got, want := string(encoded), `[{"a":"BTC","period":7,"v":12345678901234567890},{"a":"ETH","v":{"c":2,"o":1}}]`; got != want { + t.Errorf("bulk entries\n got %s\nwant %s", got, want) } } + +// A failed refresh after a 401 surfaces its cause in the returned error. +func TestRequest_401RefreshFailureReportsCause(t *testing.T) { + withTempHome(t, func() { + api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusUnauthorized) + _, _ = w.Write([]byte(`{"error":"unauthorized"}`)) + })) + defer api.Close() + idp := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusBadRequest) + _, _ = w.Write([]byte(`{"error":"invalid_grant","error_description":"refresh token revoked"}`)) + })) + defer idp.Close() + restore := oauth.OverrideDefaultsForTesting(idp.URL, "https://api.example", "cid") + defer restore() + if err := config.SaveOAuthSession(config.OAuthSession{AccessToken: "stale", RefreshToken: "rt1", ExpiresAt: time.Now().UTC().Add(time.Hour)}); err != nil { + t.Fatal(err) + } + + client := NewClient("", "stale") + client.baseURL = api.URL + client.httpClient = api.Client() + + _, err := client.GetMetric(context.Background(), "/anything", nil) + var authErr *glassnode.AuthError + if !errors.As(err, &authErr) || !strings.Contains(err.Error(), "invalid_grant") || !strings.Contains(err.Error(), "refresh token revoked") { + t.Fatalf("err = %v, want an AuthError carrying the IdP's cause", err) + } + }) +} diff --git a/internal/api/metadata.go b/internal/api/metadata.go index b9e65cc..a2ff8a4 100644 --- a/internal/api/metadata.go +++ b/internal/api/metadata.go @@ -2,85 +2,22 @@ package api import ( "context" - "encoding/json" - "fmt" "net/url" "strings" -) - -type Asset struct { - ID string `json:"id"` - ExternalIDs map[string]string `json:"external_ids,omitempty"` - Symbol string `json:"symbol"` - Name string `json:"name"` - AssetType string `json:"asset_type"` - Blockchains []Blockchain `json:"blockchains"` - Categories []string `json:"categories,omitempty"` - LogoURL string `json:"logo_url,omitempty"` - SemanticTags []string `json:"semantic_tags,omitempty"` - DefaultNetwork string `json:"default_network,omitempty"` -} - -type Blockchain struct { - Blockchain string `json:"blockchain"` - Address string `json:"address,omitempty"` - Decimals int `json:"decimals,omitempty"` - OnChainSupport bool `json:"on_chain_support"` -} - -type AssetsResponse struct { - Data []Asset `json:"data"` -} - -// NamesResponse is the response of the metadata list endpoints -// (/v1/metadata/tags, /v1/metadata/assets/tags, ...): {"data":[{"name":"..."},...]}. -type NamesResponse struct { - Data []NameEntry `json:"data"` -} -// NameEntry is a single named value in a names response. -type NameEntry struct { - Name string `json:"name"` -} - -type MetricMetadata struct { - Path string `json:"path,omitempty"` - Tier float64 `json:"tier,omitempty"` - IsPit bool `json:"is_pit,omitempty"` - Parameters map[string][]string `json:"parameters"` - Queried map[string]string `json:"queried,omitempty"` - Refs Refs `json:"refs,omitempty"` - BulkSupported bool `json:"bulk_supported"` - Timerange *Timerange `json:"timerange,omitempty"` - Modified int64 `json:"modified,omitempty"` - Descriptors *MetricDescriptors `json:"descriptors,omitempty"` -} - -type MetricVariant struct { - Base *string `json:"base,omitempty"` - Bulk *string `json:"bulk,omitempty"` - Pit *string `json:"pit,omitempty"` -} - -type MetricDescriptors struct { - Name string `json:"name,omitempty"` - ShortName string `json:"short_name,omitempty"` - Group string `json:"group,omitempty"` - Tags []string `json:"tags,omitempty"` - Description map[string]string `json:"description,omitempty"` - DataSharingGroup string `json:"data_sharing_group,omitempty"` -} - -type Timerange struct { - Min int64 `json:"min,omitempty"` - Max int64 `json:"max,omitempty"` -} + glassnode "github.com/glassnode/glassnode-api-go-client" +) -type Refs struct { - Doc string `json:"docs,omitempty"` - Studio string `json:"studio,omitempty"` - MetricVariant *MetricVariant `json:"metric_variant,omitempty"` -} +type Asset = glassnode.Asset +type Blockchain = glassnode.Blockchain +type AssetsResponse = glassnode.AssetsResponse +type NamesResponse = glassnode.NamesResponse +type NameEntry = glassnode.NameEntry +type MetricMetadata = glassnode.MetricMetadata +type MetricVariant = glassnode.MetricVariant +type MetricDescriptors = glassnode.MetricDescriptors +type Timerange = glassnode.TimeRange +type Refs = glassnode.Refs // assetToMap returns a map of JSON field names to values for the given asset. // Used by PruneAssets to build objects with only requested fields. @@ -123,103 +60,50 @@ func PruneAssets(assets []Asset, fields []string) []map[string]interface{} { } func (c *Client) ListAssets(ctx context.Context, filter string) ([]Asset, error) { - params := map[string]string{} - if filter != "" { - params["filter"] = filter - } - body, err := c.Do(ctx, "GET", "/v1/metadata/assets", params) + client, err := c.sdk() if err != nil { - return nil, fmt.Errorf("listing assets: %w", err) + return nil, err } - var resp AssetsResponse - if err := json.Unmarshal(body, &resp); err != nil { - return nil, fmt.Errorf("decoding assets response: %w", err) - } - return resp.Data, nil + return client.ListAssets(ctx, filter) } -// ListNames lists the named values from a metadata list endpoint such as -// /v1/metadata/tags or /v1/metadata/assets/categories, optionally filtered by a CEL expression. -func (c *Client) ListNames(ctx context.Context, path string, filter string) ([]string, error) { - params := map[string]string{} - if filter != "" { - params["filter"] = filter - } - body, err := c.Do(ctx, "GET", path, params) +func (c *Client) ListNames(ctx context.Context, path, filter string) ([]string, error) { + client, err := c.sdk() if err != nil { - return nil, fmt.Errorf("listing %s: %w", path, err) + return nil, err } - var resp NamesResponse - if err := json.Unmarshal(body, &resp); err != nil { - return nil, fmt.Errorf("decoding %s response: %w", path, err) - } - names := make([]string, 0, len(resp.Data)) - for _, e := range resp.Data { - names = append(names, e.Name) - } - return names, nil + return client.ListNames(ctx, path, filter) } -// ListMetrics lists available metric paths, optionally filtered by query parameters. -// See https://docs.glassnode.com/basic-api/metadata#query-parameters-1 -// params: a (asset), c (currency), e (exchange), f (format), i (interval), -// from_exchange, to_exchange, miner, maturity, network, period, quote_symbol. -// repeatedParams: use key "a" for multiple assets (e.g. a=BTC&a=ETH). -func (c *Client) ListMetrics(ctx context.Context, params map[string]string, repeatedParams map[string][]string) ([]string, error) { - if params == nil { - params = map[string]string{} - } - if repeatedParams == nil { - repeatedParams = map[string][]string{} - } - body, err := c.DoWithRepeatedParams(ctx, "GET", "/v1/metadata/metrics", params, repeatedParams) +func (c *Client) ListMetrics(ctx context.Context, params map[string]string, repeated map[string][]string) ([]string, error) { + client, err := c.sdk() if err != nil { - return nil, fmt.Errorf("listing metrics: %w", err) + return nil, err } - var metrics []string - if err := json.Unmarshal(body, &metrics); err != nil { - return nil, fmt.Errorf("decoding metrics response: %w", err) - } - return metrics, nil + return client.ListMetrics(ctx, &glassnode.MetricParams{Extra: queryValues(params, repeated)}) } func (c *Client) DescribeMetric(ctx context.Context, path, asset string) (*MetricMetadata, error) { - params := map[string]string{"path": path} - if asset != "" { - params["a"] = asset - } - body, err := c.Do(ctx, "GET", "/v1/metadata/metric", params) + client, err := c.sdk() if err != nil { - return nil, fmt.Errorf("describing metric: %w", err) + return nil, err } - var meta MetricMetadata - if err := json.Unmarshal(body, &meta); err != nil { - return nil, fmt.Errorf("decoding metric metadata: %w", err) - } - return &meta, nil + return client.GetMetricMetadata(ctx, path, &glassnode.MetricParams{Asset: asset}) } -// BuildURL constructs the full request URL without executing the request. +// BuildURL constructs the request URL without executing the request. The +// credential is not part of it: it travels in the header named by AuthHeader. +// The base URL is validated as the SDK validates it, so a dry run fails where +// the request would, and trailing slashes are dropped as the SDK drops them. func (c *Client) BuildURL(path string, params map[string]string, repeatedParams map[string][]string) (string, error) { - u, err := c.buildURL(path, params, repeatedParams) - if err != nil { + if _, err := c.sdk(); err != nil { return "", err } - return u.String(), nil -} - -// RedactAPIKeyFromURL returns a copy of the URL with the api_key query parameter -// replaced by a placeholder, for safe display (e.g. dry-run output). -func RedactAPIKeyFromURL(raw string) (string, error) { - u, err := url.Parse(raw) + u, err := url.Parse(strings.TrimRight(c.baseURL, "/") + path) if err != nil { return "", err } - q := u.Query() - if q.Has("api_key") { - q.Set("api_key", "***") - u.RawQuery = q.Encode() - } + u.RawQuery = queryValues(params, repeatedParams).Encode() return u.String(), nil } diff --git a/internal/api/metrics.go b/internal/api/metrics.go index d4c4dda..cba3837 100644 --- a/internal/api/metrics.go +++ b/internal/api/metrics.go @@ -2,14 +2,21 @@ package api import ( "context" - "encoding/json" "fmt" + "strings" + + glassnode "github.com/glassnode/glassnode-api-go-client" ) +// DataPoint is a metric point as the API sent it. Numbers are json.Number, +// since the SDK decodes with UseNumber: -o json prints them unchanged and +// integers above 2^53 keep their precision. type DataPoint struct { T int64 `json:"t"` V interface{} `json:"v,omitempty"` O map[string]interface{} `json:"o,omitempty"` + // ComputedAt is set by point-in-time metrics: when the value was computed. + ComputedAt *int64 `json:"computed_at,omitempty"` } type BulkDataPoint struct { @@ -22,27 +29,36 @@ type BulkResponse struct { } func (c *Client) GetMetric(ctx context.Context, path string, params map[string]string) ([]DataPoint, error) { - normalized := NormalizePath(path) - body, err := c.Do(ctx, "GET", "/v1/metrics"+normalized, params) + client, err := c.sdk() if err != nil { - return nil, fmt.Errorf("getting metric: %w", err) + return nil, err } + // The SDK reads s, u, i, c, a and e from Extra and validates them. + p := &glassnode.MetricParams{Extra: queryValues(params, nil)} var points []DataPoint - if err := json.Unmarshal(body, &points); err != nil { - return nil, fmt.Errorf("decoding metric response: %w", err) + if err := client.GetMetric(ctx, path, p, &points); err != nil { + return nil, fmt.Errorf("getting metric: %w", err) } return points, nil } -func (c *Client) GetMetricBulk(ctx context.Context, path string, params map[string]string, repeatedParams map[string][]string) (*BulkResponse, error) { - normalized := NormalizePath(path) - body, err := c.DoWithRepeatedParams(ctx, "GET", "/v1/metrics"+normalized+"/bulk", params, repeatedParams) +// GetMetricBulk fetches the bulk variant of a metric and returns its entries as +// the API sent them. The CLI only prints them, so it decodes into its own rows +// rather than the SDK's typed BulkEntry: numbers stay numbers (the SDK decodes +// with UseNumber), large integers keep their precision and selectors or values +// of any type pass through. +func (c *Client) GetMetricBulk(ctx context.Context, path string, params map[string]string, repeated map[string][]string) (*BulkResponse, error) { + client, err := c.sdk() if err != nil { + return nil, err + } + p := &glassnode.MetricParams{Extra: queryValues(params, repeated)} + var response BulkResponse + if err := client.GetMetric(ctx, strings.TrimSuffix(NormalizePath(path), "/")+"/bulk", p, &response); err != nil { return nil, fmt.Errorf("getting bulk metric: %w", err) } - var resp BulkResponse - if err := json.Unmarshal(body, &resp); err != nil { - return nil, fmt.Errorf("decoding bulk metric response: %w", err) + if response.Data == nil { + response.Data = []BulkDataPoint{} } - return &resp, nil + return &response, nil } diff --git a/internal/api/testdata/api_usage.json b/internal/api/testdata/api_usage.json index e9720e6..35fc96b 100644 --- a/internal/api/testdata/api_usage.json +++ b/internal/api/testdata/api_usage.json @@ -1 +1 @@ -{"customerId":4568,"creditsUsed":6,"dailyRequestsUsed":2,"apiAddons":[{"rpm":100,"value":1500000,"version":"v3","period":"monthly"}],"dataPackages":[],"allDataPackages":[{"name":"On-Chain Data","dataSharingGroups":["onchain"]},{"name":"Market Data","dataSharingGroups":["market"]}]} +{"customerId":10001,"creditsUsed":6,"dailyRequestsUsed":2,"apiAddons":[{"rpm":100,"value":1500000,"version":"v3","period":"monthly"}],"dataPackages":[],"allDataPackages":[{"name":"On-Chain Data","dataSharingGroups":["onchain"]},{"name":"Market Data","dataSharingGroups":["market"]}]} diff --git a/internal/output/output.go b/internal/output/output.go index 7c1b363..90e96e7 100644 --- a/internal/output/output.go +++ b/internal/output/output.go @@ -7,6 +7,7 @@ import ( "io" "os" "sort" + "strconv" "strings" "time" @@ -76,6 +77,13 @@ func PrintCSV(w io.Writer, data interface{}, timestampFormat string) error { return err } } + case api.CreditsSummary: + if err := write([]string{"limit", "period", "used", "left"}); err != nil { + return err + } + if err := write([]string{strconv.Itoa(v.CreditsLimit), string(v.CreditsPeriod), optionalInt(v.CreditsUsed), optionalInt(v.CreditsLeft)}); err != nil { + return err + } case []api.Asset: if err := write([]string{"id", "symbol", "name", "asset_type", "categories"}); err != nil { return err @@ -110,27 +118,42 @@ func PrintCSV(w io.Writer, data interface{}, timestampFormat string) error { cw.Flush() return cw.Error() } + computed := hasComputedAt(v) if v[0].O != nil { keys := sortedKeys(v[0].O) header := append([]string{"t"}, keys...) + if computed { + header = append(header, "computed_at") + } if err := write(header); err != nil { return err } for _, dp := range v { row := []string{formatTimestamp(dp.T, timestampFormat)} for _, k := range keys { - row = append(row, fmt.Sprintf("%v", dp.O[k])) + row = append(row, valueStr(dp.O[k])) + } + if computed { + row = append(row, computedAtStr(dp, timestampFormat)) } if err := write(row); err != nil { return err } } } else { - if err := write([]string{"t", "v"}); err != nil { + header := []string{"t", "v"} + if computed { + header = append(header, "computed_at") + } + if err := write(header); err != nil { return err } for _, dp := range v { - if err := write([]string{formatTimestamp(dp.T, timestampFormat), fmt.Sprintf("%v", dp.V)}); err != nil { + row := []string{formatTimestamp(dp.T, timestampFormat), valueStr(dp.V)} + if computed { + row = append(row, computedAtStr(dp, timestampFormat)) + } + if err := write(row); err != nil { return err } } @@ -152,7 +175,7 @@ func PrintCSV(w io.Writer, data interface{}, timestampFormat string) error { for _, entry := range dp.Bulk { row := []string{formatTimestamp(dp.T, timestampFormat)} for _, k := range keys { - row = append(row, fmt.Sprintf("%v", entry[k])) + row = append(row, valueStr(entry[k])) } if err := write(row); err != nil { return err @@ -177,6 +200,8 @@ func PrintTable(w io.Writer, data interface{}, timestampFormat string) error { for _, s := range v { table.Append([]string{s}) } + case api.CreditsSummary: + return printCreditsTable(w, v) case []api.Asset: table.SetHeader([]string{"ID", "Symbol", "Name", "Type", "Categories"}) for _, a := range v { @@ -201,21 +226,36 @@ func PrintTable(w io.Writer, data interface{}, timestampFormat string) error { if len(v) == 0 { return nil } + computed := hasComputedAt(v) if v[0].O != nil { keys := sortedKeys(v[0].O) header := append([]string{"TIME"}, upperAll(keys)...) + if computed { + header = append(header, "COMPUTED AT") + } table.SetHeader(header) for _, dp := range v { row := []string{formatTimestamp(dp.T, timestampFormat)} for _, k := range keys { - row = append(row, fmt.Sprintf("%v", dp.O[k])) + row = append(row, valueStr(dp.O[k])) + } + if computed { + row = append(row, computedAtStr(dp, timestampFormat)) } table.Append(row) } } else { - table.SetHeader([]string{"TIME", "VALUE"}) + header := []string{"TIME", "VALUE"} + if computed { + header = append(header, "COMPUTED AT") + } + table.SetHeader(header) for _, dp := range v { - table.Append([]string{formatTimestamp(dp.T, timestampFormat), fmt.Sprintf("%v", dp.V)}) + row := []string{formatTimestamp(dp.T, timestampFormat), valueStr(dp.V)} + if computed { + row = append(row, computedAtStr(dp, timestampFormat)) + } + table.Append(row) } } case *api.BulkResponse: @@ -232,7 +272,7 @@ func PrintTable(w io.Writer, data interface{}, timestampFormat string) error { for _, entry := range dp.Bulk { row := []string{formatTimestamp(dp.T, timestampFormat)} for _, k := range keys { - row = append(row, fmt.Sprintf("%v", entry[k])) + row = append(row, valueStr(entry[k])) } table.Append(row) } @@ -245,11 +285,41 @@ func PrintTable(w io.Writer, data interface{}, timestampFormat string) error { return nil } +// printCreditsTable lays the allowance out as labelled lines, like the metric +// metadata table. A daily allowance counts requests, a monthly one credits; +// usage is unknown when the period is not one the CLI knows. +func printCreditsTable(w io.Writer, s api.CreditsSummary) error { + unit := "credits" + switch s.CreditsPeriod { + case api.CreditsPeriodDaily: + unit = "requests" + case api.CreditsPeriodMonthly: + default: + unit = "" + } + fmt.Fprintf(w, "Period: %s\n", s.CreditsPeriod) + fmt.Fprintf(w, "Limit: %s\n", strings.TrimSpace(fmt.Sprintf("%d %s", s.CreditsLimit, unit))) + if s.CreditsUsed == nil || s.CreditsLeft == nil { + fmt.Fprintln(w, "Used: unknown for this period") + return nil + } + fmt.Fprintf(w, "Used: %d\n", *s.CreditsUsed) + fmt.Fprintf(w, "Left: %d\n", *s.CreditsLeft) + return nil +} + +func optionalInt(n *int) string { + if n == nil { + return "" + } + return strconv.Itoa(*n) +} + func printMetricMetadataTable(w io.Writer, m *api.MetricMetadata) error { fmt.Fprintf(w, "Path: %s\n", m.Path) - fmt.Fprintf(w, "Tier: %.0f\n", m.Tier) + fmt.Fprintf(w, "Tier: %d\n", m.Tier) fmt.Fprintf(w, "Bulk Supported: %t\n", m.BulkSupported) - fmt.Fprintf(w, "PIT: %t\n", m.IsPit) + fmt.Fprintf(w, "PIT: %t\n", m.IsPIT) if m.Descriptors != nil { if m.Descriptors.Name != "" { fmt.Fprintf(w, "Name: %s\n", m.Descriptors.Name) @@ -261,8 +331,8 @@ func printMetricMetadataTable(w io.Writer, m *api.MetricMetadata) error { fmt.Fprintf(w, "Tags: %s\n", strings.Join(m.Descriptors.Tags, ", ")) } } - if m.Timerange != nil { - fmt.Fprintf(w, "Timerange: %d - %d\n", m.Timerange.Min, m.Timerange.Max) + if m.TimeRange != nil { + fmt.Fprintf(w, "Timerange: %d - %d\n", m.TimeRange.Min, m.TimeRange.Max) } if len(m.Parameters) > 0 { fmt.Fprintln(w, "Parameters:") @@ -282,15 +352,42 @@ func sortedKeys(m map[string]interface{}) []string { return keys } -// valueStr formats a value for CSV/table. Slices of strings are joined with ";". +// valueStr formats a value for CSV/table. Numbers are written in plain +// decimal notation (1738241421021.0708, not 1.7382414210210708e+12), so they +// load into spreadsheets as numbers; slices of strings are joined with ";". func valueStr(v interface{}) string { - if v == nil { + switch x := v.(type) { + case nil: return "" + case []string: + return strings.Join(x, ";") + case float64: + return strconv.FormatFloat(x, 'f', -1, 64) + case float32: + return strconv.FormatFloat(float64(x), 'f', -1, 32) + case json.Number: + return x.String() + default: + return fmt.Sprintf("%v", v) + } +} + +// hasComputedAt reports whether any point carries computed_at, in which case +// the column is shown for all of them. +func hasComputedAt(points []api.DataPoint) bool { + for _, dp := range points { + if dp.ComputedAt != nil { + return true + } } - if ss, ok := v.([]string); ok { - return strings.Join(ss, ";") + return false +} + +func computedAtStr(dp api.DataPoint, timestampFormat string) string { + if dp.ComputedAt == nil { + return "" } - return fmt.Sprintf("%v", v) + return formatTimestamp(*dp.ComputedAt, timestampFormat) } func upperAll(ss []string) []string { diff --git a/internal/output/output_test.go b/internal/output/output_test.go index 5b1a635..351414c 100644 --- a/internal/output/output_test.go +++ b/internal/output/output_test.go @@ -305,7 +305,7 @@ func TestPrintTable_MetricMetadataWithOptionalFields(t *testing.T) { Group: "Market", Tags: []string{"price", "market"}, }, - Timerange: &api.Timerange{Min: 1609459200, Max: 1735689600}, + TimeRange: &api.Timerange{Min: 1609459200, Max: 1735689600}, Parameters: map[string][]string{"a": {"BTC", "ETH"}, "i": {"24h"}}, } if err := PrintTable(&buf, meta, ""); err != nil { @@ -325,3 +325,106 @@ func TestPrintTable_MetricMetadataWithOptionalFields(t *testing.T) { t.Errorf("table should include Parameters: %q", out) } } + +func TestPrintCSV_NumbersArePlainDecimal(t *testing.T) { + var buf bytes.Buffer + data := []api.DataPoint{{T: 1, V: 1738241421021.0708}, {T: 2, V: 0.000001}} + if err := PrintCSV(&buf, data, "unix"); err != nil { + t.Fatal(err) + } + want := "t,v\n1,1738241421021.0708\n2,0.000001\n" + if got := buf.String(); got != want { + t.Errorf("got %q, want %q", got, want) + } + var table bytes.Buffer + if err := PrintTable(&table, data, "unix"); err != nil { + t.Fatal(err) + } + if strings.Contains(table.String(), "e+") { + t.Errorf("table uses scientific notation: %s", table.String()) + } +} + +func TestPrintCSV_ComputedAtColumn(t *testing.T) { + computed := int64(1791244932) + var buf bytes.Buffer + data := []api.DataPoint{{T: 1791158400, V: 664280.0, ComputedAt: &computed}, {T: 1791244800, V: 640332.0}} + if err := PrintCSV(&buf, data, "unix"); err != nil { + t.Fatal(err) + } + want := "t,v,computed_at\n1791158400,664280,1791244932\n1791244800,640332,\n" + if got := buf.String(); got != want { + t.Errorf("got %q, want %q", got, want) + } + var table bytes.Buffer + if err := PrintTable(&table, data, "unix"); err != nil { + t.Fatal(err) + } + if !strings.Contains(table.String(), "COMPUTED AT") || !strings.Contains(table.String(), "1791244932") { + t.Errorf("table lacks computed_at: %s", table.String()) + } + var plain bytes.Buffer + if err := PrintCSV(&plain, []api.DataPoint{{T: 1, V: 2.0}}, "unix"); err != nil { + t.Fatal(err) + } + if strings.Contains(plain.String(), "computed_at") { + t.Errorf("computed_at column shown without PIT data: %q", plain.String()) + } +} + +// Values arrive as json.Number from the SDK; neither JSON nor CSV output may +// round them through float64. +func TestLargeIntegersKeepPrecision(t *testing.T) { + data := []api.DataPoint{{T: 1, V: json.Number("12345678901234567890")}} + var jsonOut bytes.Buffer + if err := PrintJSON(&jsonOut, data); err != nil { + t.Fatal(err) + } + if !strings.Contains(jsonOut.String(), `"v": 12345678901234567890`) { + t.Errorf("JSON rounded the value: %s", jsonOut.String()) + } + var csvOut bytes.Buffer + if err := PrintCSV(&csvOut, data, "unix"); err != nil { + t.Fatal(err) + } + if got, want := csvOut.String(), "t,v\n1,12345678901234567890\n"; got != want { + t.Errorf("CSV got %q, want %q", got, want) + } +} + +func TestCreditsSummary_TableAndCSV(t *testing.T) { + used, left := 2, 48 + daily := api.CreditsSummary{CreditsLimit: 50, CreditsPeriod: api.CreditsPeriodDaily, CreditsUsed: &used, CreditsLeft: &left} + var table bytes.Buffer + if err := PrintTable(&table, daily, ""); err != nil { + t.Fatal(err) + } + for _, want := range []string{"Period: daily", "Limit: 50 requests", "Used: 2", "Left: 48"} { + if !strings.Contains(table.String(), want) { + t.Errorf("table lacks %q:\n%s", want, table.String()) + } + } + var csvOut bytes.Buffer + if err := PrintCSV(&csvOut, daily, ""); err != nil { + t.Fatal(err) + } + if got, want := csvOut.String(), "limit,period,used,left\n50,daily,2,48\n"; got != want { + t.Errorf("csv got %q want %q", got, want) + } + + unknown := api.CreditsSummary{CreditsLimit: 300, CreditsPeriod: "weekly"} + table.Reset() + if err := PrintTable(&table, unknown, ""); err != nil { + t.Fatal(err) + } + if !strings.Contains(table.String(), "Limit: 300\n") || !strings.Contains(table.String(), "unknown for this period") { + t.Errorf("unknown period table:\n%s", table.String()) + } + csvOut.Reset() + if err := PrintCSV(&csvOut, unknown, ""); err != nil { + t.Fatal(err) + } + if got, want := csvOut.String(), "limit,period,used,left\n300,weekly,,\n"; got != want { + t.Errorf("unknown period csv got %q want %q", got, want) + } +} diff --git a/internal/timeparse/timeparse.go b/internal/timeparse/timeparse.go index 15d706a..a791840 100644 --- a/internal/timeparse/timeparse.go +++ b/internal/timeparse/timeparse.go @@ -8,13 +8,22 @@ import ( "unicode" ) +// Parse reads a time as Unix seconds, a date, an RFC 3339 timestamp or a +// duration back from now, such as 30d or 1h. func Parse(input string) (int64, error) { + return ParseAt(input, time.Now()) +} + +// ParseAt is Parse with relative durations measured back from now. A command +// that parses several times and plans requests from them should use one now, +// so that a range such as --since 31d is exactly 31 days long. +func ParseAt(input string, now time.Time) (int64, error) { if isAllDigits(input) { return strconv.ParseInt(input, 10, 64) } if dur, ok := parseRelativeDuration(input); ok { - return time.Now().Add(-dur).Unix(), nil + return now.Add(-dur).Unix(), nil } if t, err := time.Parse("2006-01-02", input); err == nil { diff --git a/internal/timeparse/timeparse_test.go b/internal/timeparse/timeparse_test.go index 577a5c1..2430ce8 100644 --- a/internal/timeparse/timeparse_test.go +++ b/internal/timeparse/timeparse_test.go @@ -85,3 +85,14 @@ func TestParse(t *testing.T) { } }) } + +func TestParseAt(t *testing.T) { + now := time.Date(2026, 10, 8, 12, 0, 0, 500, time.UTC) + got, err := ParseAt("31d", now) + if err != nil || got != now.Add(-31*24*time.Hour).Unix() { + t.Fatalf("ParseAt(31d) = %d, %v; want exactly 31 days before now", got, err) + } + if got, _ := ParseAt("2024-01-01", now); got != 1704067200 { + t.Errorf("absolute dates ignore now: %d", got) + } +} diff --git a/skills/glassnode-cli/SKILL.md b/skills/glassnode-cli/SKILL.md index f9d27fc..9c36418 100644 --- a/skills/glassnode-cli/SKILL.md +++ b/skills/glassnode-cli/SKILL.md @@ -60,10 +60,17 @@ gn metric get distribution/balance_exchanges --asset BTC --exchange binance --cu ``` ### Bulk fetch (multiple assets) -Append `/bulk` to the metric path. Repeat `-a` (or `--asset`) for each asset, or use `-a '*'` for all: +Append `/bulk` to the metric path. Repeat `-a` (or `--asset`) for each asset, or use `-a '*'` for all. +`--since` is required: the API caps one bulk request at 10 days for `10m`/`1h`, 31 days for `24h` +(the default) and 93 days for `1w`/`1month`, counted from the start of the interval containing +`--since` (weeks start on Monday, months on the 1st, UTC). A longer range is rejected before any request is sent +unless `--split` is passed; the CLI then fetches consecutive windows, prints the number of requests +on stderr, and aborts on any error naming the window and the `--since` to retry from. Each window +is a billed request. `--split` on a non-bulk path is ignored with a note on stderr. ```bash -gn metric get market/marketcap_usd/bulk -a BTC -a ETH -a SOL -s 1d +gn metric get market/marketcap_usd/bulk -a BTC -a ETH -a SOL --since 7d gn metric get market/marketcap_usd/bulk -a '*' --interval 24h --since 30d +gn metric get market/marketcap_usd/bulk -a BTC -a ETH --since 2024-01-01 --until 2024-04-01 --split ``` ### API credit usage (`gn user credits`) @@ -87,8 +94,11 @@ Example JSON output: All four fields describe the same window. `creditsLimit` is the allowance per `creditsPeriod` (`daily` on the advanced product, `monthly` on every other one), -and `creditsUsed` and `creditsLeft` are the usage and remainder within it. Never -report the limit or the usage as monthly without reading `creditsPeriod`. +and `creditsUsed` and `creditsLeft` are the usage and remainder within it; on a +daily allowance the unit is requests. Never report the limit or the usage as +monthly without reading `creditsPeriod`. If `creditsPeriod` is anything else +(or `mixed`), `creditsUsed` and `creditsLeft` are absent: say the usage is +unknown instead of computing it. ### Config set / get ```bash @@ -104,15 +114,15 @@ gn config get all |------|-------|-------------| | `--api-key` | | Glassnode API key | | `--output` | `-o` | Output format: `json` (default), `csv`, `table` | -| `--dry-run` | | Print the request URL without executing | +| `--dry-run` | | Print the endpoint and parameters without sending the request; the API key goes in the `X-Api-Key` header, not the URL. A preview built from the flags: the SDK appends `f=json` to metric requests and validates parameters only when sending | | `--timestamp-format` | | Timestamp format in output | ## Best Practices - Always call `gn metric describe ` before `gn metric get` to discover valid parameters, assets, exchanges, and intervals for a metric. - Use `--output json` (default) for structured output suitable for piping to `jq`. -- Use `--dry-run` to preview the API request URL without executing: `gn metric get market/price_usd_close --asset BTC --since 30d --dry-run`. -- For bulk metrics (path ending in `/bulk`), use `-a` with specific assets or `-a '*'` for all. +- Use `--dry-run` to preview the request without sending it: `gn metric get market/price_usd_close --asset BTC --since 30d --dry-run`. The URL has no API key; it is sent in the `X-Api-Key` header. +- For bulk metrics (path ending in `/bulk`), use `-a` with specific assets or `-a '*'` for all. `--since` is required; pass `--split` to fetch a range longer than the per-request cap in several billed requests. - Relative time values are supported for `--since` and `--until`: e.g. `30d`, `7d`, `1h`. ## Output Formats