7.3.0: rules deploy messages, quant serve --run background dev servers, local KV secrets, serve.json defaults

README.md

@@ -263,15 +263,15 @@ Flags:
 - `--host` - Address to listen on (default: `127.0.0.1`). Any other address makes the site and its KV data reachable on the network.
 - `--functions` - Functions manifest (default: `quant/edge-functions.json`).
 - `--rules` - Rules manifest (default: `quant/edge-rules.json`).
-- `--meta` - Template metadata with the KV stores (default: `quant/meta.json`).
+- `--meta` - Template metadata with the KV stores (default: `meta` in `quant/serve.json`, then `quant/meta.json`).
 - `--fixtures` - Directory of KV fixtures (default: `quant/kv-fixtures`).
-- `--org` - Organisation (default: the `quant.json` clientid, then `local`).
-- `--project` - Project (default: the `quant.json` project, then `dev`).
+- `--org` - Organisation (default: `org` in `quant/serve.json`, the `quant.json` clientid, then `local`).
+- `--project` - Project (default: `project` in `quant/serve.json`, the `quant.json` project, then `dev`). `quant serve` has no `--project` option of its own: it uses the global `--project` option of the CLI, so `quant serve --help` lists it with the global options.
 - `--domain` - Host for the domain match of rules (default: the Host of the request).
 - `--user-name` - Value for `{user_name}` in KV seed values (default: `git config user.name`, then `Owner`).
 - `--build` - Command to run once before the server starts.
 - `--watch` - Reload when files change. With `--build`, run the build again when files under `src/` or `public/` change.
-- `--run` - Long-running command, for example a dev server, to start beside the server. Quant serve stops it when it stops.
+- `--run` - Long-running command, for example a dev server, to start beside the server. Quant serve stops it when it stops. On macOS and Linux the command runs in its own process group, and quant serve stops the whole group. Some dev servers move to the background and the command exits 0 at once. If the process group is still alive, quant serve prints one notice, keeps serving and stops the group when it stops. If only the `--proxy` URL still answers (the dev server left the process group), quant serve says that it cannot stop that dev server. On Windows there is no process group: quant serve stops only the command, and it cannot stop a dev server that moved to the background.
 - `--proxy` - Dev server URL. Requests that no rule ends go to this URL. WebSocket upgrades (for example, HMR) also go to this URL, after the auth rules of the path. Function rules do not run for an upgrade.
 - `--geo` - Geolocation fields for functions, as a JSON object.
 - `--client-ip` - Client address for functions.
@@ -299,6 +299,7 @@ quant kv list notes --local --index by_author --equals author=ana
 quant kv get notes post:1 --local
 quant kv set notes post:1 '{"title":"Hello"}' --local
 quant kv set notes post:1 '{"title":"New"}' --local --if-version 3
+quant kv set notes api:token '"abc123"' --local --secret
 quant kv delete notes post:1 --local
 ```
 
@@ -310,9 +311,12 @@ Flags:
 - `--index` - `list`: index name.
 - `--equals` - `list`: `field=value` for the index. Repeat the flag for more fields.
 - `--if-version` - `set`: write only if the key has this version.
-- `--org` - Organisation (default: the `quant.json` clientid, then `local`).
-- `--project` - Project (default: the `quant.json` project, then `dev`).
-- `--meta` - Template metadata with the KV stores (default: `quant/meta.json`).
+- `--secret` - `set`: store the value encrypted, as `set(key, value, { secret: true })` in a function does. A function reads the value as usual. `list` shows `encrypted: true` and the value `[ENCRYPTED]`. `set` prints the entry with the value `[ENCRYPTED]`, so the value does not go into the terminal output or CI logs. `get --local` shows the value, because the local store is yours. The value is a command argument, so it goes into your shell history; the interactive menu asks for a secret value with hidden input. With `get`, `list` or `delete`, `--secret` is ignored and the command prints one notice. A key in an indexed collection cannot hold a secret (`INDEX_FIELD_ENCRYPTED`). The local store uses local keys only: do not put production secrets in it.
+- `--org` - Organisation (default: `org` in `quant/serve.json`, the `quant.json` clientid, then `local`).
+- `--project` - Project (default: `project` in `quant/serve.json`, the `quant.json` project, then `dev`).
+- `--meta` - Template metadata with the KV stores (default: `meta` in `quant/serve.json`, then `quant/meta.json`).
+
+To set these defaults one time for a project, put them in `quant/serve.json` (see [Local project settings](#local-project-settings-quantservejson)).
 
 ## Local ledgers
 
@@ -446,10 +450,20 @@ Each field is optional.
 - `globalHeaders` - Headers for every response from the site: pages, files, function responses, the 404 page and the basic authentication 401. A header that a function sets replaces the global header of the same name. `quant-server` and `age` cannot be set.
 - `basicAuth` - With a username and a password, every path needs these credentials. A request without them gets 401 with `WWW-Authenticate: Basic`. The check applies before the rules, so functions and redirects need the credentials too. WebSocket upgrades also need them. The `/_quant_serve/` control routes do not need them: they are local only and need the `x-quant-serve: 1` header (see [Safety](#safety)).
 - `trailingSlash` - `"redirect"`: a path that ends with `/` (except `/`) gets a 301 to the path without the trailing slashes, with the query string. A path that a rule answers is not redirected. `"none"` (the default): no redirect.
+- `org`, `project` - The organisation and the project for the local KV store ids, for `quant serve` and `quant kv --local`. Use them when the project has no `quant.json`. Each value is 1 to 100 letters, digits, `.`, `_` or `-`.
+- `meta` - The template metadata file with the KV stores (default: `quant/meta.json`). The path is relative to the project directory and must stay in it.
+
+For `org`, `project` and `meta`, the order is: the flag, then `quant/serve.json`, then `quant.json` (org and project only), then the default (`local`, `dev`, `quant/meta.json`). `quant serve` reads these three fields only at start-up, also with `--watch`. If `quant/serve.json` and `quant.json` set a different org (or project) and no flag chooses, `quant serve` and `quant kv --local` print one warning line that names the value they use.
+
+The local store ids contain the org and the project (for example `local-dev-orbit`). If you add or change `org` or `project` in `quant/serve.json`, the store ids change (for example to `acme-dev-orbit`), and the local data that you wrote before seems to be missing. The data is still in `.quant/kv/`. Set the old values again to see it.
+
+```json
+{ "org": "acme", "project": "marketing-site" }
+```
 
 The order is the order of the edge: global headers, basic authentication, rules, the trailing-slash redirect, then static content.
 
-`quant/serve.json` is local only. No deploy command reads it, and `quant deploy` never uploads it. `quant serve` never prints the password. Do not put a production password in the file. A file with the wrong shape stops `quant serve` at start-up, with a message that names the field. With `--watch`, `quant serve` reads the file again when it changes. If the new file has the wrong shape, `quant serve` prints the message and keeps the last valid settings.
+`quant/serve.json` is local only. No deploy command reads it, and `quant deploy` never uploads it. `quant serve` never prints the password. Do not put a production password in the file. A file with the wrong shape stops `quant serve` at start-up, with a message that names the field. It also stops `quant kv --local` (exit code 1), because that command reads the `org`, `project` and `meta` defaults from the same file. With `--watch`, `quant serve` reads the file again when it changes. If the new file has the wrong shape, `quant serve` prints the message and keeps the last valid settings.
 
 ## Edge libraries in quant serve
 
@@ -597,6 +611,14 @@ exits non-zero and lists each changed field (values only for matchers and weight
 Re-run with `--force` to overwrite those rules with the manifest. `--force` never
 takes over a rule this manifest did not create (a rule ID collision).
 
+When the rules API refuses a deploy with 409 or 422 and gives a message, `quant rules`
+prints that message and exits non-zero. Examples:
+
+- 409: the change may already be live. Reload the rules and check them. A retry is safe.
+- 422: the requested rule order cannot be kept. A retry does not help. Contact support.
+
+Without a message, a 409 means another request is changing the rules (retry shortly).
+
 Portal-issued template tokens expire after one year; rotate the token and update
 `QUANT_API_TOKEN` before expiry.
 

docs/adopting-quant-serve.md

@@ -49,6 +49,13 @@ bundles function sources into the committed function bodies.
    Give the dev server `--host 127.0.0.1` when it listens only on IPv6 by
    default, or the proxy cannot reach it.
 
+   In some environments the dev server moves to the background and the `--run`
+   command exits 0 at once. quant serve then prints that the command moved to the
+   background and keeps serving. On macOS and Linux, it stops the dev server when
+   it stops, but only while the dev server stays in the process group of the
+   `--run` command. On Windows, quant serve cannot stop a dev server that moved to
+   the background. See the `--run` flag in the README.
+
 3. Move the seed data into fixtures. Each store gets one file,
    `quant/kv-fixtures/<store-name>.json`, with a JSON array of entries:
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@quantcdn/quant-cli",
-  "version": "7.2.0",
+  "version": "7.3.0",
   "description": "Deploy tools for QuantCDN",
   "type": "module",
   "main": "cli.js",

src/commands/kv.js

@@ -6,7 +6,7 @@
 
 import fs from 'node:fs';
 import path from 'node:path';
-import { select, text, isCancel } from '@clack/prompts';
+import { select, text, password, confirm, isCancel } from '@clack/prompts';
 import { openLocalKv } from '../serve/kv/wire.js';
 import { pendingLedgerWarnings } from '../serve/kv/ledgers.js';
 import { DEFAULT_FILES } from '../serve/project.js';
@@ -16,6 +16,8 @@ import { resolveOptions } from './serve.js';
 const REMOTE_MESSAGE = 'Remote KV commands are not available yet. Use --local.';
 const ACTIONS = ['list', 'get', 'set', 'delete'];
 const MAX_PAGES = 100;
+// The mask of an encrypted value, the same as in a list.
+const ENCRYPTED = '[ENCRYPTED]';
 
 /** A failure with a message for the user and no stack. */
 class UsageError extends Error {}
@@ -91,14 +93,24 @@ function need(value, label) {
   return value;
 }
 
+// The set options: ifVersion, and secret (the value is stored encrypted, as
+// with `secret: true` in a function: a function reads it, a list shows it masked).
+function setOptions(args) {
+  const options = {};
+  if (args['if-version'] !== undefined) options.ifVersion = args['if-version'];
+  if (args.secret === true) options.secret = true;
+  return Object.keys(options).length > 0 ? options : undefined;
+}
+
 const OPERATIONS = {
   list: (store, args) => listAll(store, args),
   get: (store, args) => store.getEntry(need(args.key, 'The key')),
   async set(store, args) {
     const value = parseJson(need(args.value, 'The JSON value'));
-    const ifVersion = args['if-version'];
-    await store.set(need(args.key, 'The key'), value, ifVersion === undefined ? undefined : { ifVersion });
-    return store.getEntry(args.key);
+    await store.set(need(args.key, 'The key'), value, setOptions(args));
+    const entry = await store.getEntry(args.key);
+    // Never print a secret value after a write: it would go to logs and scrollback.
+    return entry && entry.encrypted ? { ...entry, value: ENCRYPTED } : entry;
   },
   async delete(store, args) {
     await store.delete(need(args.key, 'The key'));
@@ -113,14 +125,16 @@ function describeError(err) {
 
 async function execute(args, root, error) {
   if (!ACTIONS.includes(args.action)) throw new UsageError(`The action must be one of: ${ACTIONS.join(', ')}`);
+  if (args.secret === true && args.action !== 'set') error(`--secret applies only to set. It is ignored for ${args.action}.`);
   const options = resolveOptions({ org: args.org, project: args.project, meta: args.meta }, { root, gitUserName: () => null });
   const id = idOf(options, args.store);
   if (id === null) throw new UsageError(`FORBIDDEN: Access to KV store ${args.store} is not allowed`);
   // kv only creates missing stores: it declares no index and holds no index state.
   const stores = readStores(root, options.meta).map((store) => ({ ...store, indexes: [] }));
   const local = await openLocalKv({ root, org: options.org, project: options.project, stores });
   // The warnings come before the operation, so a failed operation keeps them.
-  for (const warning of [...local.warnings, ...pendingLedgerWarnings(local.backend, local.backend.storeIds())]) error(warning);
+  const warnings = [...options.configWarnings, ...local.warnings, ...pendingLedgerWarnings(local.backend, local.backend.storeIds())];
+  for (const warning of warnings) error(warning);
   return OPERATIONS[args.action](local.kvStore.open(id), args);
 }
 
@@ -144,18 +158,35 @@ export async function runKv(args, { root = process.cwd(), print = console.log, e
   }
 }
 
-// Ask for the action, the store and what the action needs. Null on cancel.
-async function promptArgs() {
-  const action = await select({ message: 'Operation', options: ACTIONS.map((value) => ({ value, label: value })) });
-  if (isCancel(action)) return null;
-  const store = await text({ message: 'Store name (from quant/meta.json)' });
-  if (isCancel(store)) return null;
+const PROMPTS = { select, text, password, confirm, isCancel };
+
+// set: the secret question (default no), then the JSON value. A secret
+// value is typed with hidden input. Null on cancel.
+async function askValue(prompts) {
+  const secret = await prompts.confirm({ message: 'Store the value as a secret (encrypted)?', initialValue: false });
+  if (prompts.isCancel(secret)) return null;
+  const ask = secret === true ? prompts.password : prompts.text;
+  const value = await ask({ message: 'JSON value' });
+  return prompts.isCancel(value) ? null : { value, secret: secret === true };
+}
+
+/**
+ * The menu: ask for the action, the store and what the action needs.
+ * @param {{ select, text, password, confirm, isCancel }} [prompts] The prompt functions (for tests).
+ * @returns {Promise<Object|null>} The args, or null on cancel.
+ */
+export async function askKv(prompts = PROMPTS) {
+  const action = await prompts.select({ message: 'Operation', options: ACTIONS.map((value) => ({ value, label: value })) });
+  if (prompts.isCancel(action)) return null;
+  const store = await prompts.text({ message: 'Store name (from quant/meta.json)' });
+  if (prompts.isCancel(store)) return null;
   const args = { local: true, action, store };
   if (action === 'list') return args;
-  const key = await text({ message: 'Key' });
-  if (isCancel(key)) return null;
-  const value = action === 'set' ? await text({ message: 'JSON value' }) : undefined;
-  return isCancel(value) ? null : { ...args, key, value };
+  const key = await prompts.text({ message: 'Key' });
+  if (prompts.isCancel(key)) return null;
+  if (action !== 'set') return { ...args, key, value: undefined };
+  const answers = await askValue(prompts);
+  return answers === null ? null : { ...args, key, ...answers };
 }
 
 export default {
@@ -172,10 +203,11 @@ export default {
     .option('index', { type: 'string', describe: 'list: index name' })
     .option('equals', { type: 'string', array: true, describe: 'list: field=value for the index (repeat)' })
     .option('if-version', { type: 'string', describe: 'set: write only if the key has this version' })
-    .option('org', { type: 'string', describe: 'Organisation (default: quant.json clientid, then local)' })
-    .option('project', { type: 'string', describe: 'Project (default: quant.json project, then dev)' })
-    .option('meta', { type: 'string', default: DEFAULT_FILES.meta, describe: 'Template metadata with KV stores' }),
-  promptArgs,
+    .option('secret', { type: 'boolean', describe: 'set: store the value encrypted (a list shows it masked)' })
+    .option('org', { type: 'string', describe: 'Organisation (default: quant/serve.json org, quant.json clientid, then local)' })
+    .option('project', { type: 'string', describe: 'Project (default: quant/serve.json project, quant.json project, then dev)' })
+    .option('meta', { type: 'string', describe: `Template metadata with KV stores (default: quant/serve.json meta, then ${DEFAULT_FILES.meta})` }),
+  promptArgs: () => askKv(),
   async handler(args) {
     const code = await runKv(args);
     if (code !== 0) process.exit(code);

src/commands/rules.js

@@ -59,12 +59,38 @@ function refusalError(rules) {
   return new Error('Managed rules were changed outside code; nothing was written. Re-run with --force to overwrite them with this manifest.');
 }
 
+const MESSAGE_STATUSES = [409, 422];
+const MAX_MESSAGE = 300;
+
+// A string as one short line, or null when it is not text.
+function oneLine(value) {
+  if (typeof value !== 'string' || value.trim() === '') return null;
+  const line = value.replace(/[\u0000-\u001f\u007f-\u009f]+/g, ' ').replace(/\s+/g, ' ').trim();
+  return line.length > MAX_MESSAGE ? `${line.slice(0, MAX_MESSAGE - 3)}...` : line;
+}
+
+function adviceText(status, code) {
+  if (status === 409) return 'Rules are being changed by another request (409). Retry shortly.';
+  return `Rules deployment failed (${status || code || 'network error'}). Check portal URL, token scopes and project access.`;
+}
+
+// A 409 or 422 `message` replaces the advice: the API writes it for the user
+// (for example, that the change may already be live, or that the rule order
+// cannot be kept). A string `error` alone is printed above the advice.
+function statusText(status, data, code) {
+  const advice = adviceText(status, code);
+  if (!MESSAGE_STATUSES.includes(status)) return advice;
+  const message = oneLine(data?.message);
+  if (message) return `Rules deployment failed (${status}): ${message}`;
+  const errorText = oneLine(data?.error);
+  return errorText ? `Rules deployment failed (${status}): ${errorText}\n${advice}` : advice;
+}
+
 // Never print Axios config/headers or an arbitrary upstream body containing credentials.
 function deployError(error) {
-  const status = error.response?.status;
-  if (status === 409 && Array.isArray(error.response?.data?.rules)) return refusalError(error.response.data.rules);
-  if (status === 409) return new Error('Rules are being changed by another request (409). Retry shortly.');
-  return new Error(`Rules deployment failed (${status || error.code || 'network error'}). Check portal URL, token scopes and project access.`);
+  const { status, data } = error.response || {};
+  if (status === 409 && Array.isArray(data?.rules)) return refusalError(data.rules);
+  return new Error(statusText(status, data, error.code));
 }
 
 export default {