Skip to content

Support filtering #4

Closed
Closed

Description

@mindsocket

We have a schema, and it is open for extension.

It should be possible to filter an OST to show more or less detail.

Less depth for a simplified view of just the top layers
Less breadth to focus in on an area
Less density to exclude ideas that aren't in focus (eg leave out status: identified

I'm hoping we can define a simple filter language or find an existing one that's human friendly and works with json. Todoist filters are a good reference - they support basic boolean logic(| and & and parens), some keywords that translate to specific filters (eg "today", "@Label"). Another source of inspiration might be something akin to CSS or XPath selectors.

Let's do a top down and bottom up investigation of options - top down for good filter approaches, bottom up for existing low level json filtering - then see where they meet.

We'll also need to reconcile how relationships and hierarchies are selected. I don't necessarily want to go down a graphql path. A simple rubric might be that nodes that match the filter will also always include all of their parent nodes (even if parent nodes don't match a criteria; or possibly unless parent nodes don't match a specific filter criteria related to traversal). We might need a separate "where clause" (ie filter) and "select/include clause" (ie what to include from the matching objects and across relationships).

Example use cases to be fleshed out further, but off the top of my head:

  • High level selected opportunities (no solutions or experiments) - select opportunity where ( wondering | exploring ) & opportunity (all layers above implied)
  • Detail view of a goal - select goal where "Increase MAU" (title match) include goal to experiment
  • All nodes reachable by parent links up and/or child links down from a given node

Activity

  1. mindsocket commented on Mar 9, 2026

    @mindsocket
    OwnerAuthor

    Regarding filtering - since we already use JSONata for expressions throughout the codebase, it would make sense to explore using JSONata for filtering as well. This would provide consistency and could leverage our existing infrastructure. Views could potentially be saved in config or schema files (to be decided later) as a convenience feature.

  2. mindsocket commented on Mar 24, 2026

    @mindsocket
    OwnerAuthor

    @claude research options for filtering from various angles (prior art, existing libraries, common patterns etc) based on the current state of this project and the thinking so far on this issue. Be sure to include assumptions or open questions that would need to be considered before deciding, and recommendations.

    Respond with your findings here. Don't proceed if you can't access web resources needed to properly research.

  3. claude commented on Mar 24, 2026

    @claude

    Filtering Research for OST Tools

    Executive Summary

    Based on codebase analysis, JSONata is the recommended approach for implementing filtering in ost-tools. It's already integrated for rule validation, well-suited for tree/graph structures, and provides powerful query capabilities.

    Proposed CLI:

    ost-tools validate ProductX --filter "resolvedType='solution' and status='active'"
    ost-tools dump ProductX --filter "resolvedParentTitle='Increase user retention'"

    Implementation effort: ~1-2 days for core --filter flag functionality


    Current Filtering Landscape

    Existing JSONata Integration

    JSONata is already used in the codebase for rule validation (see src/schema/evaluate-rule.ts):

    Evaluation Context (EvalContext):

    • nodes - All nodes in the space
    • $ / current - Current node being evaluated
    • parent - First parent node
    • parents - All parent nodes

    Example Rules from Production (schemas/_ost_strict.json):

    // Count nodes by type and status
    "$count(nodes[resolvedType='outcome' and status='active']) <= 1"
    
    // Parent-child relationships
    "$count(nodes[resolvedParentTitle=$.current.title and resolvedType='solution']) >= 3"
    
    // Status propagation
    "current.status != 'active' or $exists(parent) = false or parent.status = 'active'"

    JSONata Capabilities for OST Filtering

    Use Case JSONata Expression
    Filter by type resolvedType='opportunity'
    Filter by status status='active'
    Parent relationship resolvedParentTitle='Some Parent'
    Multiple conditions resolvedType='solution' and status='active'
    Negation not(status='archived')
    Aggregations $count(nodes[status='active'])
    Custom fields customField.value > 5

    Proposed Implementation

    Phase 1: CLI Flag (Priority)

    Add --filter flag to existing commands:

    // src/index.ts
    program
      .command('validate <space>')
      .option('--filter <expr>', 'JSONata filter expression')
      .action(async (space, options) => {
        const { diagnostics } = await validate(space, {
          filter: options.filter
        });
      });

    Implementation in src/validate.ts:

    export async function validate(spaceName: string, options: { filter?: string }): Promise<ValidationResult> {
      const context = await loadSpace(spaceName);
      let nodes = context.nodes;
      
      if (options.filter) {
        const expression = jsonata(options.filter);
        const filteredResults = await expression.evaluate({
          nodes: context.nodes,
          current: null, // No current node in this context
        });
        nodes = filteredResults || [];
      }
      
      // Continue with validation on filtered nodes...
    }

    Phase 2: Interactive Filter Builder (Future)

    • Auto-complete for field names
    • Filter templates for common patterns
    • Visual filter composition

    Design Considerations

    Assumptions

    1. Tree-aware filtering: Users want to filter by tree position (parent/child relationships)
    2. Status-based filtering: Active/archived/draft filtering is common
    3. Type-based filtering: Filter by outcome/opportunity/solution/etc.
    4. Aggregation queries: Count nodes matching criteria

    Open Questions

    1. Output format for dump command: Should filtered output preserve tree structure or return flat list?
    2. Filter validation: Should invalid filter expressions fail fast or warn?
    3. Performance: For large spaces (1000+ nodes), should filters be pre-compiled?
    4. Filter persistence: Should users be able to save named filters in config?

    Alternatives Considered

    Approach Pros Cons Verdict
    JSONata Already integrated, powerful, tree-aware Learning curve for users ✅ Recommended
    jq Familiar to many, fast Not tree-aware, harder parent queries ❌ Not suitable
    JSONPath Simple, standard Limited expressiveness ❌ Too limited
    Custom DSL Full control Maintenance burden, re-inventing wheel ❌ YAGNI

    Prior Art & Patterns (Pending Web Research)

    Still investigating:

    • npm packages for tree/graph filtering
    • GitHub/Gmail/Todoist filter syntax patterns
    • CLI precedents for tree-structured data filtering
    • Query language patterns for hierarchical data

    Recommendation

    Proceed with JSONata-based filtering for the following reasons:

    1. ✅ Zero new dependencies - already in codebase
    2. ✅ Tree-aware - can query parent/child relationships naturally
    3. ✅ Powerful - supports aggregations, functions, composition
    4. ✅ Validated - proven in production for rule checking
    5. ✅ Extensible - can add custom functions if needed

    Implementation Priority

    1. Sprint 1: Add --filter to validate command (1 day)
    2. Sprint 2: Add --filter to dump command (0.5 day)
    3. Sprint 3: Add --filter to other commands as needed (0.5 day)
    4. Future: Interactive filter builder, saved filters, etc.

    Research in progress - awaiting web search access for comprehensive analysis of prior art, npm packages, and UX patterns from similar tools.

  4. mindsocket commented on Mar 24, 2026

    @mindsocket
    OwnerAuthor

    Follow-up research: tree traversal and graph structure

    Picking up from the earlier JSONata recommendation — it's sound as a direction but the previous analysis didn't fully work through what tree-based queries require. This comment digs into that specifically.


    Two separate problems

    The issue description actually contains two distinct concerns that need different solutions:

    1. Filter predicate — which nodes match (e.g. "solutions whose grandparent goal has status: active")
    2. Result shape / inclusion — what to include alongside matches (e.g. "also include the parent opportunity even if it doesn't independently match")

    These need different mechanisms. Mixing them into one expression makes the language harder to use.


    The critical gap in the current representation

    flattenNode() currently exposes resolvedParentTitle and resolvedParentTitles — one hop up. Multi-hop queries (grandparent, any ancestor) require cross-node lookups in JSONata.

    In theory JSONata can do this — but with nested $map/$filter lambdas that must explicitly capture outer variables (because $ is shadowed inside nested predicates). For example, "solutions where grandparent goal has status: active" would look something like:

    $filter(nodes[resolvedType='solution'], function($sol) {
      $exists(
        $filter(
          nodes[resolvedType='opportunity' and title=$sol.resolvedParentTitle],
          function($opp) {
            $exists(nodes[resolvedType='goal' and status='active' and title=$opp.resolvedParentTitle])
          }
        )
      )
    })
    

    That's not user-hostile for schema rule authors, but it's verbose and brittle (hardcodes two hops). It's also not the right model to expose for filtering — JSONata's strength is JSON projection, not graph walking.


    Two paths forward for tree queries

    Option A: Augmented flat nodes (ancestors array)

    Pre-compute ancestry chains in flattenNode(). Add an ancestors array to each node, ordered from immediate parent to root:

    {
      ...schemaData,
      resolvedType,
      resolvedParentTitle,   // existing — first parent
      resolvedParentTitles,  // existing — all parents
      ancestors: [           // NEW — ordered parent → grandparent → … → root
        { title: 'Opportunity X', resolvedType: 'opportunity', status: 'active' },
        { title: 'Goal A', resolvedType: 'goal', status: 'active' },
      ]
    }

    Then the grandparent query becomes:

    nodes[resolvedType='solution' and ancestors[1].status='active']
    

    Or more robustly (any depth):

    nodes[resolvedType='solution' and $exists(ancestors[resolvedType='goal' and status='active'])]
    

    This keeps JSONata as the engine and makes multi-hop queries clean. The tradeoff: ancestry must be computed at evaluation time, and in a DAG (multiple parents), ancestors would need to be an array-of-arrays or flattened with deduplication.

    Option B: Build an actual graph structure

    Pass a richer structure to JSONata rather than a flat array. For a tree (or forest), a nested representation:

    {
      "roots": [
        {
          "title": "Goal A", "resolvedType": "goal", "status": "active",
          "children": [
            {
              "title": "Opportunity X", "resolvedType": "opportunity",
              "children": [
                { "title": "Solution Y", "resolvedType": "solution", ... }
              ]
            }
          ]
        }
      ],
      "nodes": [ /* flat array kept for global lookups */ ]
    }

    JSONata's ** (recursive descent) operator can then walk the tree:

    roots.**.children[resolvedType='solution']
    

    The problem: this codebase's hierarchy is a DAG (a node can have multiple parents), so a purely nested representation requires duplication or is impossible for multi-parent nodes. A nested structure works cleanly only for strict trees.

    Recommendation: Option A with a depth-limited ancestors array. It avoids duplication issues with multi-parent nodes and keeps the flat nodes[] array as the primary query target. Option B could work as a complementary structure for display/rendering but not as the canonical query model.


    The result shape / inclusion problem

    This is where the tree structure matters most — and it's a post-filter operation, not part of the filter predicate. Patterns from the issue:

    • "include parent nodes of matched nodes" → walk resolvedParents from each match, add to result set
    • "include subtree from match down to depth N" → walk children map (already built by classifyNodes in graph-helpers.ts) from each match
    • "top layers only" → depth limit on hierarchy levels

    The existing classifyNodes / buildHierarchyNodeSet infrastructure already provides what's needed for this. The design question is how to expose it at the CLI level — probably something like:

    --filter "resolvedType='solution'"  # which nodes match
    --include ancestors                 # also include parents of matches
    --include subtree:2                 # also include up to 2 levels of children
    

    Or a more compact inline form TBD.


    Prior art summary (web research)

    • XPath axes (ancestor::, descendant::) are the best semantic precedent for what the operations mean. //solution[ancestor::goal[@status='active']] directly expresses the grandparent query. The axis vocabulary is precise but unfamiliar to non-technical users.
    • CSS combinators (>, space, ~) are the most learnable for parent/child relationships. The :has() pseudo-class (CSS4) partially addresses ancestor queries but still can't express "select B whose ancestor has property X" cleanly.
    • Cypher ([:PARENT_OF*] variable-length paths) is the clearest way to express "ancestor at any depth". (y:Goal {status:'active'})-[:PARENT_OF*]->(x:Solution) is readable and explicit about direction.
    • No existing JS/TS library was found that provides a declarative query language for flat node arrays with ancestor-aware filtering. Most tools work on nested tree objects (d3-hierarchy) or are full graph databases. This is a genuine gap — building something here would be novel.
    • AQL (ArangoDB) uses LET ancestors = (FOR a IN 1..10 INBOUND x edge RETURN a) — closest in concept to what we'd compute, but verbose. The pattern of "capture ancestors into a variable, then filter on the array" maps well to the augmented flat nodes approach.

    Practical synthesis: a readable user-facing filter language might borrow:

    • type=solution for simple property predicates (Todoist-style)
    • [ancestor: type=goal, status=active] or ancestor(type=goal, status=active) for ancestor predicates (XPath concept, function-call syntax)
    • Separate include: parents / include: subtree modifiers for result shape (no query language handles this elegantly inline)

    Open questions before designing further

    1. Should the user-facing filter syntax compile to JSONata, or should JSONata only power schema rules while filtering uses a different (simpler) evaluator?
    2. How should multi-parent nodes be handled in ancestry chains — do we include all paths, or just hierarchy-sourced parents?
    3. Is the "include" clause a CLI flag, a separate part of a filter expression, or out of scope for an initial implementation?
    4. What's the target user for --filter? If schema authors (power users), JSONata with augmented nodes may be fine. If product managers in a future UI, a simpler syntax is worth the investment.
  5. mindsocket commented on Mar 24, 2026

    @mindsocket
    OwnerAuthor

    Design decisions and theoretical spec (current state)

    Summarising the thinking so far into a more complete picture. This supersedes the earlier research comments.


    Two-expression model

    Filtering has two distinct concerns handled by separate expressions:

    1. Filter expression — determines which nodes are in the matched set. Evaluated per node. Returns boolean.

    2. Include expression — expands the result set by walking the graph from matched nodes. Determines what gets pulled in alongside matches (ancestors, descendants, relationship-linked nodes). Separate mini-language, not JSONata.

    These map roughly to SQL's WHERE (filter) and the implicit JOIN/INCLUDE shape of a result.


    Node augmentation

    To make multi-hop tree queries expressible, each node in the evaluation context will be augmented with:

    ancestors: [{ title, resolvedType, status, ..., _field, _source, _selfRef }, ...]
    descendants: [{ title, resolvedType, status, ..., _field, _source, _selfRef }, ...]
    
    • Flat arrays (not nested, not per-path) — duplicates removed where multiple paths exist (DAG case)
    • Each entry merges the node's data with edge metadata from resolvedParents (_field, _source: 'hierarchy'|'relationship', _selfRef)
    • Ordered nearest-first (ancestors: parent before grandparent; descendants: children before grandchildren)
    • Self-referential hops are traversed through — so a solution with two self-referential solution ancestors before reaching an opportunity would have all three in ancestors[]

    This makes ancestor queries expressible without cross-node lookups:

    resolvedType = 'solution' and $exists(ancestors[resolvedType = 'goal' and status = 'active'])
    
    resolvedType = 'solution' and $exists(ancestors[_source = 'hierarchy' and resolvedType = 'goal'])
    

    Filter expression language: JSONata

    JSONata is confirmed as the filter language (user-facing). JMESPath was considered and rejected — it cannot express aggregations inside filter predicates (e.g. length(ancestors[?status=='active']) > 0 is not valid in standard JMESPath) and does not support custom function binding. JSONata handles both.

    The augmented ancestors/descendants arrays mean most realistic filter expressions don't require advanced JSONata features:

    resolvedType = 'solution' and status = 'active'
    
    resolvedType = 'opportunity' and $exists(ancestors[resolvedType = 'goal' and status = 'active'])
    
    resolvedType = 'solution' and $count(descendants[resolvedType = 'experiment']) >= 1
    

    Include expression language (sketch)

    A separate mini-language for the include clause. Not JSONata. Combines:

    Convenience functions:

    • ancestors — all ancestors of matched nodes
    • descendants — all descendants of matched nodes
    • siblings — nodes sharing a parent with matched nodes
    • relationships — all relationship-linked nodes (non-hierarchy edges)

    Type names — include nodes of a specific type reachable from matches:

    • opportunity — direct parents of type opportunity
    • problem_statement — relationship children of that type

    Hierarchy level ranges — a slice of the hierarchy:

    • opportunity..goal — from opportunity level up to goal level
    • solution..experiment — matched node's subtree down to experiment

    Relationship specs — by child type name (relationships have no standalone name, only parent + type):

    • problem_statement, assumption, risk

    Include and exclude directions — both supported:

    • include: ancestors / exclude: descendants[resolvedType='experiment']

    The exact grammar is not yet finalised. The goal is something compact enough for a CLI flag and storable in config/schema files — same format in both contexts, SQL-query-level familiarity.


    Storage / invocation

    Both expressions are plain strings. No distinction in richness between ad-hoc CLI use and stored views:

    ost-tools show myspace --filter "resolvedType='solution' and status='active'" --include "ancestors"

    Stored views (in config or schema, design TBD) would use the same strings.


    Open questions

    1. Exact grammar for the include expression (sketch above, not finalised)
    2. Edge metadata field naming convention in augmented nodes (_field/_source prefix or something else)
    3. How stored views are declared and named (config file, schema $metadata, or separate?)
    4. Whether descendants traversal follows hierarchy edges only or also relationship edges (or is this controlled per query via _source filtering?)
  6. mindsocket commented on Mar 24, 2026

    @mindsocket
    OwnerAuthor

    Relationship reference syntax in include expressions

    Relationships have no standalone name — they're identified by parent + type + field. The include expression uses progressive specification to keep common cases terse:

    Form When to use
    child_type Child type is unambiguous across all relationships in the schema
    parent_type:child_type Child type alone is ambiguous (appears under multiple parents)
    parent_type:field:child_type Same parent+child pair exists with multiple fields

    The third form is needed when a parent type has two relationships to the same child type via different fields. Example:

    { "parent": "Activities", "type": "Data", "field": "data_required", ... },
    { "parent": "Activities", "type": "Data", "field": "data_produced", ... }

    These would be referenced as Activities:data_required:Data and Activities:data_produced:Data respectively. Activities:Data alone would be an error (ambiguous).

  7. mindsocket commented on Mar 25, 2026

    @mindsocket
    OwnerAuthor

    Addressed in #66

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions