feat(api)!: standardize list RPCs on opaque page tokens #3047

User Story

As an API or SDK client, I want every list operation to expose consistent continuation tokens, so that I can enumerate resources completely and safely while the underlying collection changes.

Problem Statement

OpenShell list RPCs currently use inconsistent pagination. Several accept limit and offset but return no total, continuation token, or truncation signal. Other public list RPCs have no pagination fields at all. A client cannot reliably distinguish a complete result from a truncated page, and offset-based paging is unstable when records are inserted or removed between requests.

Impact / Why This Matters

Clients must guess that a full-sized response implies another page, manually advance offsets, and risk omissions or duplicates under concurrent mutation. SDK authors repeat this logic differently, and callers cannot build dependable inventory, cleanup, or reconciliation workflows.

Proposed Design

Adopt one public list contract across gateway resources:

• Requests use page_size and an opaque page_token.
• Responses include next_page_token, empty only when enumeration is complete.
• Tokens bind the query scope and ordering needed to continue safely.
• Each resource defines a stable deterministic order.
• SDK iterators follow tokens until completion while still allowing callers to fetch one page.
• Invalid, expired, or query-mismatched tokens return documented errors.

The token encoding and persistence implementation remain internal.

Acceptance Criteria

• Every public List RPC either implements the standard pagination contract or explicitly documents why its result set is bounded.
• Paginated requests use consistent field names and validation limits.
• Responses expose next_page_token; clients never infer completion from page length.
• Ordering and concurrent insertion/deletion semantics are documented.
• Rust, Python, TypeScript, and Go SDKs expose consistent one-page and full-iteration behavior.
• Regression tests enumerate more than one page without omission or duplication.
• Offset-based fields are removed or migrated with reserved names/tags and documented breaking-change guidance.

Alternatives Considered

Keep offset pagination and add total_size. This signals truncation but remains unstable during concurrent mutation and makes totals potentially expensive. Add only a truncation boolean. This still leaves clients without a safe continuation mechanism. Leave smaller lists unpaginated. This creates another permanent API exception and unbounded growth risk.

Agent Investigation

Current public list requests and responses in proto/openshell.proto use several combinations of limit, offset, and no pagination. Internal persistence pagination and full-table iteration are tracked separately in #2802.

Related: #2565, #2802. Source audit: https://gist.github.com/mrunalp/e80942c1544a0225ee588796a41ab30b.