Skip to content

Lambda and Variable Expressions and BoundExpressions - #9257

Closed
mhk197 wants to merge 38 commits into
developfrom
mk/lambda-variables
Closed

mhk197 wants to merge 38 commits into
developfrom
mk/lambda-variables

Conversation

@mhk197

@mhk197 mhk197 commented Aug 7, 2026 •

Copy link
Copy Markdown
Contributor

Add Variable expressions and standalone lambda syntax for future higher-order functions.

  • Add named Variable references and Scope bindings (name -> dtype), rejecting duplicate names.
  • Bind variables into typed BoundExpression::Variable nodes.
  • Add Lambda and BoundLambda as callable syntax/types owned by higher-order functions.
    Lambdas are deliberately not Expression variants: they do not have an array dtype or generic
    array-execution semantics.
  • Keep generic Expression machinery focused on array values: scalar functions, root, and
    variables.
  • Add protobuf round-tripping for variables and standalone lambdas, plus display, validity, and
    binding coverage.
  • Update expression consumers to reject detached variables when no runtime binding environment is
    available.

This establishes the type/binding representation; a future HOF vtable will supply parameter dtypes,
bind and optimize the lambda body, and evaluate it against its parameter arrays.

@codspeed

codspeed Bot commented Aug 7, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 14.01%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 28 regressed benchmarks
✅ 2140 untouched benchmarks
⏩ 106 skipped benchmarks1
🗄️ 4 archived benchmarks run2

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation case_when_nary_10_conditions[1000] 289.9 µs 382 µs -24.1%
❌ WallTime mul_u64_nonnull_neon 15.1 µs 19.7 µs -23.51%
❌ Simulation optimize_lookup_predicate[ids=192, shape=balanced_or] 415.5 µs 512.2 µs -18.89%
❌ Simulation optimize_lookup_predicate[ids=256, shape=balanced_or] 541.4 µs 667.2 µs -18.86%
❌ Simulation optimize_lookup_predicate[ids=128, shape=balanced_or] 294.7 µs 360.5 µs -18.26%
❌ WallTime mul_i64_nonnull_neon 17.1 µs 20.8 µs -17.84%
❌ Simulation optimize_or_chain[200] 485.8 µs 588.6 µs -17.47%
❌ WallTime multiply_shapes_neon[(16384, PerRowPerRow)] 17.3 µs 20.9 µs -17.1%
❌ WallTime arrow_checked_add_u32_avx2[16384] 17.7 µs 21.3 µs -16.74%
❌ Simulation optimize_lookup_predicate[ids=64, shape=balanced_or] 170.8 µs 205.1 µs -16.7%
❌ Simulation case_when_nary_equality_lookup[1000] 250 µs 294.2 µs -15.04%
❌ Simulation case_when_nary_early_dominant[1000] 175.8 µs 206.4 µs -14.84%
❌ WallTime deferred_i64_avx2[PerRowPerRow] 9.9 µs 11.7 µs -14.8%
❌ Simulation optimize_lookup_predicate[ids=64, shape=in_list] 39.3 µs 45.6 µs -13.89%
❌ Simulation case_when_all_true[1000] 102.1 µs 118.3 µs -13.71%
❌ Simulation case_when_nary_10_conditions[10000] 522.8 µs 605.2 µs -13.62%
❌ Simulation case_when_all_false[1000] 100 µs 115.6 µs -13.5%
❌ Simulation case_when_nary_3_conditions[1000] 177.4 µs 204.9 µs -13.46%
❌ Simulation cold_misaligned[(16, 64)] 346.1 µs 395.8 µs -12.55%
❌ Simulation optimize_lookup_predicate[ids=16, shape=balanced_or] 79.1 µs 90.4 µs -12.48%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/lambda-variables (511643b) with develop (01f147a)

Open in CodSpeed

Footnotes

  1. 106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

@mhk197
mhk197 force-pushed the mk/lambda-variables branch from e41b484 to c077107 Compare August 7, 2026 03:06
@mhk197 mhk197 added the changelog/feature A new feature label Aug 7, 2026
@mhk197 mhk197 changed the title lambdas and variables Lambda and Variable Expressions and BoundExpressions Aug 7, 2026
@mhk197
mhk197 marked this pull request as ready for review August 10, 2026 20:56
@mhk197
mhk197 force-pushed the mk/lambda-variables branch from 752dc14 to 1ce48c3 Compare August 12, 2026 00:50
@mhk197
mhk197 force-pushed the mk/lambda-variables branch from 893ab28 to 1fb9ad6 Compare August 12, 2026 17:06

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should expand this doc str its all about well typed expressions

return if expr.is_root() {
scope.names().iter().cloned().collect()
} else {
vec![]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about this?

Comment thread vortex-array/src/expr/traversal/mod.rs Outdated
match self {
BoundExpression::Scalar { children, .. } => f(&mut children.iter()),
BoundExpression::Root { .. } => f(&mut std::iter::empty()),
_ => f(&mut std::iter::empty()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you leave this enum as closed?

Comment on lines +72 to +81
pub struct BoundLambda {
/// The parameters and their dtypes: the argument side of the function type.
params: Box<[Variable]>,
param_dtypes: Box<[DType]>,
body: Arc<BoundExpression>,
}

impl BoundLambda {
/// The variables this lambda binds, in declaration order.
pub fn params(&self) -> &[Variable] {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do I use this it its not a BoundExpression?


/// The name of a value bound in a [`Scope`](crate::expr::Scope).
#[derive(Clone, Debug)]
pub struct Variable(Arc<str>);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think need to be able to go from a series of FieldNames to a series of Variable efficently. Does this allow that?

Base automatically changed from mk/bound-expression-enum to develop August 13, 2026 10:25
@mhk197
mhk197 force-pushed the mk/lambda-variables branch from a6f38fd to 27a031f Compare August 19, 2026 17:51
mhk197 added 10 commits August 28, 2026 15:42
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…ialization docs

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Variable and Lambda get reserved expression ids alongside Root, so the
unbound tree stays fully serializable. Variable carries its name in
VariableOpts; Lambda carries its params in LambdaOpts and its body as its
single proto child. All three reserved ids resolve before the registry
lookup.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
mhk197 added 25 commits August 28, 2026 15:44
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…les for now

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
mhk197 force-pushed the mk/lambda-variables branch from 691ae5d to 511643b Compare August 28, 2026 20:06
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Sep 15, 2026
@mhk197 mhk197 closed this Sep 17, 2026
@robert3005
robert3005 deleted the mk/lambda-variables branch November 9, 2026 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants