Repository navigation
Description
We have a schema, and it is open for extension.
It should be possible to filter an OST to show more or less detail.
Less depth for a simplified view of just the top layers
Less breadth to focus in on an area
Less density to exclude ideas that aren't in focus (eg leave out status: identified
I'm hoping we can define a simple filter language or find an existing one that's human friendly and works with json. Todoist filters are a good reference - they support basic boolean logic(| and & and parens), some keywords that translate to specific filters (eg "today", "@Label"). Another source of inspiration might be something akin to CSS or XPath selectors.
Let's do a top down and bottom up investigation of options - top down for good filter approaches, bottom up for existing low level json filtering - then see where they meet.
We'll also need to reconcile how relationships and hierarchies are selected. I don't necessarily want to go down a graphql path. A simple rubric might be that nodes that match the filter will also always include all of their parent nodes (even if parent nodes don't match a criteria; or possibly unless parent nodes don't match a specific filter criteria related to traversal). We might need a separate "where clause" (ie filter) and "select/include clause" (ie what to include from the matching objects and across relationships).
Example use cases to be fleshed out further, but off the top of my head:
- High level selected opportunities (no solutions or experiments) - select
opportunitywhere( wondering | exploring ) & opportunity(all layers above implied) - Detail view of a goal - select
goalwhere"Increase MAU"(title match) includegoaltoexperiment - All nodes reachable by parent links up and/or child links down from a given node
Activity
mindsocket commentedon Mar 9, 2026
Regarding filtering - since we already use JSONata for expressions throughout the codebase, it would make sense to explore using JSONata for filtering as well. This would provide consistency and could leverage our existing infrastructure. Views could potentially be saved in config or schema files (to be decided later) as a convenience feature.
mindsocket commentedon Mar 24, 2026
@claude research options for filtering from various angles (prior art, existing libraries, common patterns etc) based on the current state of this project and the thinking so far on this issue. Be sure to include assumptions or open questions that would need to be considered before deciding, and recommendations.
Respond with your findings here. Don't proceed if you can't access web resources needed to properly research.
claude commentedon Mar 24, 2026
Filtering Research for OST Tools
Executive Summary
Based on codebase analysis, JSONata is the recommended approach for implementing filtering in ost-tools. It's already integrated for rule validation, well-suited for tree/graph structures, and provides powerful query capabilities.
Proposed CLI:
Implementation effort: ~1-2 days for core --filter flag functionality
Current Filtering Landscape
Existing JSONata Integration
JSONata is already used in the codebase for rule validation (see
src/schema/evaluate-rule.ts):Evaluation Context (
EvalContext):nodes- All nodes in the space$/current- Current node being evaluatedparent- First parent nodeparents- All parent nodesExample Rules from Production (
schemas/_ost_strict.json):JSONata Capabilities for OST Filtering
resolvedType='opportunity'status='active'resolvedParentTitle='Some Parent'resolvedType='solution' and status='active'not(status='archived')$count(nodes[status='active'])customField.value > 5Proposed Implementation
Phase 1: CLI Flag (Priority)
Add
--filterflag to existing commands:Implementation in
src/validate.ts:Phase 2: Interactive Filter Builder (Future)
Design Considerations
Assumptions
Open Questions
Alternatives Considered
Prior Art & Patterns (Pending Web Research)
Still investigating:
Recommendation
Proceed with JSONata-based filtering for the following reasons:
Implementation Priority
--filtertovalidatecommand (1 day)--filtertodumpcommand (0.5 day)--filterto other commands as needed (0.5 day)Research in progress - awaiting web search access for comprehensive analysis of prior art, npm packages, and UX patterns from similar tools.
mindsocket commentedon Mar 24, 2026
Follow-up research: tree traversal and graph structure
Picking up from the earlier JSONata recommendation — it's sound as a direction but the previous analysis didn't fully work through what tree-based queries require. This comment digs into that specifically.
Two separate problems
The issue description actually contains two distinct concerns that need different solutions:
status: active")These need different mechanisms. Mixing them into one expression makes the language harder to use.
The critical gap in the current representation
flattenNode()currently exposesresolvedParentTitleandresolvedParentTitles— one hop up. Multi-hop queries (grandparent, any ancestor) require cross-node lookups in JSONata.In theory JSONata can do this — but with nested
$map/$filterlambdas that must explicitly capture outer variables (because$is shadowed inside nested predicates). For example, "solutions where grandparent goal hasstatus: active" would look something like:That's not user-hostile for schema rule authors, but it's verbose and brittle (hardcodes two hops). It's also not the right model to expose for filtering — JSONata's strength is JSON projection, not graph walking.
Two paths forward for tree queries
Option A: Augmented flat nodes (ancestors array)
Pre-compute ancestry chains in
flattenNode(). Add anancestorsarray to each node, ordered from immediate parent to root:Then the grandparent query becomes:
Or more robustly (any depth):
This keeps JSONata as the engine and makes multi-hop queries clean. The tradeoff: ancestry must be computed at evaluation time, and in a DAG (multiple parents),
ancestorswould need to be an array-of-arrays or flattened with deduplication.Option B: Build an actual graph structure
Pass a richer structure to JSONata rather than a flat array. For a tree (or forest), a nested representation:
{ "roots": [ { "title": "Goal A", "resolvedType": "goal", "status": "active", "children": [ { "title": "Opportunity X", "resolvedType": "opportunity", "children": [ { "title": "Solution Y", "resolvedType": "solution", ... } ] } ] } ], "nodes": [ /* flat array kept for global lookups */ ] }JSONata's
**(recursive descent) operator can then walk the tree:The problem: this codebase's hierarchy is a DAG (a node can have multiple parents), so a purely nested representation requires duplication or is impossible for multi-parent nodes. A nested structure works cleanly only for strict trees.
Recommendation: Option A with a depth-limited ancestors array. It avoids duplication issues with multi-parent nodes and keeps the flat
nodes[]array as the primary query target. Option B could work as a complementary structure for display/rendering but not as the canonical query model.The result shape / inclusion problem
This is where the tree structure matters most — and it's a post-filter operation, not part of the filter predicate. Patterns from the issue:
resolvedParentsfrom each match, add to result setchildrenmap (already built byclassifyNodesingraph-helpers.ts) from each matchThe existing
classifyNodes/buildHierarchyNodeSetinfrastructure already provides what's needed for this. The design question is how to expose it at the CLI level — probably something like:Or a more compact inline form TBD.
Prior art summary (web research)
ancestor::,descendant::) are the best semantic precedent for what the operations mean.//solution[ancestor::goal[@status='active']]directly expresses the grandparent query. The axis vocabulary is precise but unfamiliar to non-technical users.>, space,~) are the most learnable for parent/child relationships. The:has()pseudo-class (CSS4) partially addresses ancestor queries but still can't express "select B whose ancestor has property X" cleanly.[:PARENT_OF*]variable-length paths) is the clearest way to express "ancestor at any depth".(y:Goal {status:'active'})-[:PARENT_OF*]->(x:Solution)is readable and explicit about direction.LET ancestors = (FOR a IN 1..10 INBOUND x edge RETURN a)— closest in concept to what we'd compute, but verbose. The pattern of "capture ancestors into a variable, then filter on the array" maps well to the augmented flat nodes approach.Practical synthesis: a readable user-facing filter language might borrow:
type=solutionfor simple property predicates (Todoist-style)[ancestor: type=goal, status=active]orancestor(type=goal, status=active)for ancestor predicates (XPath concept, function-call syntax)include: parents/include: subtreemodifiers for result shape (no query language handles this elegantly inline)Open questions before designing further
--filter? If schema authors (power users), JSONata with augmented nodes may be fine. If product managers in a future UI, a simpler syntax is worth the investment.mindsocket commentedon Mar 24, 2026
Design decisions and theoretical spec (current state)
Summarising the thinking so far into a more complete picture. This supersedes the earlier research comments.
Two-expression model
Filtering has two distinct concerns handled by separate expressions:
1. Filter expression — determines which nodes are in the matched set. Evaluated per node. Returns boolean.
2. Include expression — expands the result set by walking the graph from matched nodes. Determines what gets pulled in alongside matches (ancestors, descendants, relationship-linked nodes). Separate mini-language, not JSONata.
These map roughly to SQL's
WHERE(filter) and the implicitJOIN/INCLUDEshape of a result.Node augmentation
To make multi-hop tree queries expressible, each node in the evaluation context will be augmented with:
resolvedParents(_field,_source: 'hierarchy'|'relationship',_selfRef)ancestors[]This makes ancestor queries expressible without cross-node lookups:
Filter expression language: JSONata
JSONata is confirmed as the filter language (user-facing). JMESPath was considered and rejected — it cannot express aggregations inside filter predicates (e.g.
length(ancestors[?status=='active']) > 0is not valid in standard JMESPath) and does not support custom function binding. JSONata handles both.The augmented
ancestors/descendantsarrays mean most realistic filter expressions don't require advanced JSONata features:Include expression language (sketch)
A separate mini-language for the include clause. Not JSONata. Combines:
Convenience functions:
ancestors— all ancestors of matched nodesdescendants— all descendants of matched nodessiblings— nodes sharing a parent with matched nodesrelationships— all relationship-linked nodes (non-hierarchy edges)Type names — include nodes of a specific type reachable from matches:
opportunity— direct parents of type opportunityproblem_statement— relationship children of that typeHierarchy level ranges — a slice of the hierarchy:
opportunity..goal— from opportunity level up to goal levelsolution..experiment— matched node's subtree down to experimentRelationship specs — by child type name (relationships have no standalone name, only
parent+type):problem_statement,assumption,riskInclude and exclude directions — both supported:
include: ancestors/exclude: descendants[resolvedType='experiment']The exact grammar is not yet finalised. The goal is something compact enough for a CLI flag and storable in config/schema files — same format in both contexts, SQL-query-level familiarity.
Storage / invocation
Both expressions are plain strings. No distinction in richness between ad-hoc CLI use and stored views:
Stored views (in config or schema, design TBD) would use the same strings.
Open questions
_field/_sourceprefix or something else)$metadata, or separate?)descendantstraversal follows hierarchy edges only or also relationship edges (or is this controlled per query via_sourcefiltering?)mindsocket commentedon Mar 24, 2026
Relationship reference syntax in include expressions
Relationships have no standalone name — they're identified by
parent+type+field. The include expression uses progressive specification to keep common cases terse:child_typeparent_type:child_typeparent_type:field:child_typeThe third form is needed when a parent type has two relationships to the same child type via different fields. Example:
{ "parent": "Activities", "type": "Data", "field": "data_required", ... }, { "parent": "Activities", "type": "Data", "field": "data_produced", ... }These would be referenced as
Activities:data_required:DataandActivities:data_produced:Datarespectively.Activities:Dataalone would be an error (ambiguous).mindsocket commentedon Mar 25, 2026
Addressed in #66