From 26ec3bd6c2a9161cb7ce5eb67c41ef30fac48757 Mon Sep 17 00:00:00 2001 From: Dave Lucia Date: Mon, 1 Jun 2026 09:53:12 -0700 Subject: [PATCH 1/3] chore(B12): start plan --- .../plans/B12-string-format-flags-bitmask.md | 154 ++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 .agents/plans/B12-string-format-flags-bitmask.md diff --git a/.agents/plans/B12-string-format-flags-bitmask.md b/.agents/plans/B12-string-format-flags-bitmask.md new file mode 100644 index 00000000..ca3b1008 --- /dev/null +++ b/.agents/plans/B12-string-format-flags-bitmask.md @@ -0,0 +1,154 @@ +--- +id: B12 +title: parse string.format flags once and dispatch on integer specifier +issue: 309 +pr: null +branch: perf/string-format-flags-bitmask +base: main +status: in-progress +direction: B +--- + +# B12 — string.format flag bitmask + integer specifier dispatch + +## Goal + +Speed up the `string.format` per-specifier hot path in +`lib/lua/vm/stdlib/string.ex` by removing redundant per-site work, with +zero change to observable output: + +1. **Parse flags once.** Today `parse_flags/2` collects the flag + characters into a binary, and `apply_width_flags/3` re-scans that + binary with `String.contains?(flags, "0")`, `String.contains?(flags, + "-")` on *every* specifier application. Parse the flags exactly once + at parse time into a small fixed-shape representation (a tuple of + booleans, or a single integer mask) so the apply path reads + precomputed fields instead of re-scanning a binary. +2. **Dispatch on an integer specifier.** Today `parse_specifier/1` + returns the conversion character as a one-byte binary (`"d"`, + `"f"`, …) and `apply_format_spec/2` dispatches with a `case` over + string literals. Store and match the conversion char as an integer + code point (`?d`, `?f`, …) so the `case` compiles to BEAM + integer-pattern dispatch, which is faster than binary matching. + +This mirrors Luerl's precomputed bitmask + `$d`-style integer dispatch +(`deps/luerl/src/luerl_lib_string_format.erl`). Expected to close ~10% +of the spec-heavy gap; stacks with the sibling string.format plans. + +## Out of scope + +- Any change to formatted output. This is a pure internal refactor; the + rendered string for every input must be byte-for-byte identical. +- The per-specifier formatter bodies: `format_spec_integer/1`, + `format_spec_unsigned/1`, `format_spec_float/2`, + `format_spec_scientific/3`, `format_spec_general/3`, + `format_spec_hex/2`, `format_spec_octal/1`, `format_char/1`, + `format_spec_string/2`, `format_quoted/1`, and their helpers. Sibling + plans #310 and #311 edit `string.ex` too; this plan must not touch + those bodies. Only the flag representation, the specifier token type, + and the dispatch/padding sites that consume them change. +- Starting to honor flags that are currently parsed-and-discarded. + `parse_flags/2` accepts `-+ 0#`, but only `0` and `-` are consulted + downstream; `+`, space, and `#` are parsed and ignored today. The + refactored representation must preserve that exact behavior (it may + carry those bits, but they must not change output). +- `string.format` argument validation, error messages, or arity. +- Any other stdlib function or any file other than + `lib/lua/vm/stdlib/string.ex`. + +## Success criteria + +- [ ] `string.format` produces byte-for-byte identical output for all + existing tests (no regression in + `test/lua/vm/stdlib/string_test.exs`). +- [ ] Flags are parsed exactly once, at parse time, into a fixed-shape + representation (boolean tuple or integer mask). No + `String.contains?(flags, ...)` re-scan remains on the + per-specifier apply path (`apply_width_flags/3`). +- [ ] The conversion specifier is stored and matched as an integer + code point (`?d`/`?f`/…), not a one-byte binary; the + `apply_format_spec/2` `case` dispatches on integers. +- [ ] Flags currently parsed-but-ignored (`+`, space, `#`) remain + ignored — output unchanged. +- [ ] `mix compile --warnings-as-errors` is clean. +- [ ] Full `mix test` passes with no regressions. +- [ ] `mix run benchmarks/string_format.exs` runs to completion; record + before/after numbers in the PR description. + +## Implementation notes + +All edits are confined to `lib/lua/vm/stdlib/string.ex`, in the format +parser and dispatch region (roughly lines 365–434 and 828–859): + +1. **Flag representation.** Change `parse_flags/2` (line 374) to emit a + fixed-shape value instead of accumulating a binary. Two acceptable + shapes (pick one and keep it internal): + - a tuple/struct of booleans, e.g. `{minus?, zero?, plus?, space?, + hash?}`, or + - a single integer mask with named bit constants. + Whichever is chosen, it is computed once during + `parse_format_spec/1` (line 366) and threaded through the + `{flags, width, precision, specifier}` tuple unchanged in arity/shape + at the tuple level (only the `flags` element's type changes). + +2. **Specifier as integer.** Change `parse_specifier/1` (line 404) from + `{<>, rest}` to `{c, rest}` (the raw integer code point). Update + the empty-string clause (line 406) unchanged. Rewrite the + `apply_format_spec/2` `case` (lines 415–431) to match integer code + points: `?d`, `?i`, `?u`, `?f`, `?e`, `?E`, `?g`, `?G`, `?x`, `?X`, + `?o`, `?c`, `?s`, `?q`, with the `_ ->` fallback raising the same + `"invalid option '%#{specifier}'"` error. NOTE: the interpolation + `#{specifier}` currently relies on `specifier` being a binary; with + an integer it must be rendered back to its character for the error + message (e.g. `<>` or `List.to_string([specifier])`) so + the error text is unchanged. + +3. **Padding path.** Rewrite `apply_width_flags/3` (line 828) to read + the precomputed flag fields instead of `String.contains?(flags, + "0")` (line 842), `String.contains?(flags, "-")` (line 846). The + pad-char selection (`"0"` when zero-flag set and minus-flag unset, + else `" "`), left-justify branch, and the sign-aware zero-pad branch + (line 852, `"-" <> pad <> binary_part(...)`) must remain semantically + identical. + +4. Run `mix format` after edits. Keep the `# Format string parser` + comments accurate; do not add any plan-id references in source. + +## Verification + +```bash +mix format +mix compile --warnings-as-errors +mix test test/lua/vm/stdlib/string_test.exs +mix test +mix run benchmarks/string_format.exs +``` + +Capture the `benchmarks/string_format.exs` output before and after the +change and paste the comparison into the PR description. The +`string_test.exs` run is the primary correctness gate (output +unchanged); the full `mix test` run guards against any caller of +`string.format` elsewhere in the suite. + +## Risks + +- **Silent output drift.** The dispatch and flag-handling rewrite could + subtly change a padding or specifier edge case. Mitigation: the + refactor preserves every branch one-to-one; `string_test.exs` plus + full `mix test` are the gate. If a case is thin in coverage (e.g. the + zero-pad-with-sign branch), confirm an existing test exercises it + before relying on green. +- **Error-message regression.** The invalid-option error interpolates + the specifier; switching it to an integer requires re-rendering the + character so `'%z'`-style messages stay identical. Called out in + Implementation note 2. +- **Sibling-file collision.** #310 and #311 also edit `string.ex`. This + plan stays inside the flag-parsing / specifier-dispatch region and + does not touch the `format_spec_*` formatter bodies, minimizing merge + conflicts. If a sibling lands first, rebase and re-confirm the + apply-path region is still as described. +- **No measurable win.** Per the Direction B findings, BEAM dispatch + refactors do not always pay off. If the benchmark shows no + improvement, the change is still a correctness-neutral simplification + (one flag parse instead of N re-scans); ship it on those grounds and + record the measured delta honestly in the PR. From 0f5fa902a49dc513fde49f6f3e54cd57d0862d38 Mon Sep 17 00:00:00 2001 From: Dave Lucia Date: Mon, 1 Jun 2026 09:58:15 -0700 Subject: [PATCH 2/3] perf(stdlib): parse string.format flags once, dispatch on integer specifier Parse format flags a single time at parse time into an integer bitmask instead of re-scanning the flags binary with String.contains?/2 on every specifier application, and store the conversion char as a raw code point so apply_format_spec/2 dispatches on BEAM integer patterns rather than one-byte binaries. Pure internal refactor: formatted output is unchanged. Plan: B12 Closes #309 --- lib/lua/vm/stdlib/string.ex | 67 ++++++++++++++++++++++++------------- 1 file changed, 43 insertions(+), 24 deletions(-) diff --git a/lib/lua/vm/stdlib/string.ex b/lib/lua/vm/stdlib/string.ex index c1497b93..61355ba9 100644 --- a/lib/lua/vm/stdlib/string.ex +++ b/lib/lua/vm/stdlib/string.ex @@ -28,6 +28,8 @@ defmodule Lua.VM.Stdlib.String do @behaviour Lua.VM.Stdlib.Library + import Bitwise + alias Lua.VM.ArgumentError alias Lua.VM.Executor alias Lua.VM.State @@ -364,18 +366,31 @@ defmodule Lua.VM.Stdlib.String do # Parse a format spec: [flags][width][.precision]specifier defp parse_format_spec(str) do - {flags, str} = parse_flags(str, "") + {flags, str} = parse_flags(str, 0) {width, str} = parse_width(str) {precision, str} = parse_precision(str) {specifier, str} = parse_specifier(str) {{flags, width, precision, specifier}, str} end - defp parse_flags(<>, acc) when c in ~c(-+ 0#) do - parse_flags(rest, acc <> <>) - end + # Flags are parsed once into an integer bitmask so the apply path reads + # precomputed bits instead of re-scanning a binary per specifier. Only the + # minus and zero bits affect output today (`+`, space, `#` are accepted by + # PUC-Lua's parser but ignored when rendering); the mask carries all five so + # the parse stays a single pass without changing behavior. + @flag_minus 0b00001 + @flag_zero 0b00010 + @flag_plus 0b00100 + @flag_space 0b01000 + @flag_hash 0b10000 - defp parse_flags(str, acc), do: {acc, str} + defp parse_flags(<>, mask), do: parse_flags(rest, mask ||| @flag_minus) + defp parse_flags(<>, mask), do: parse_flags(rest, mask ||| @flag_zero) + defp parse_flags(<>, mask), do: parse_flags(rest, mask ||| @flag_plus) + defp parse_flags(<>, mask), do: parse_flags(rest, mask ||| @flag_space) + defp parse_flags(<>, mask), do: parse_flags(rest, mask ||| @flag_hash) + + defp parse_flags(str, mask), do: {mask, str} defp parse_width(<> = str) when c in ?0..?9 do parse_number(str, 0) @@ -401,7 +416,9 @@ defmodule Lua.VM.Stdlib.String do defp parse_number(str, acc), do: {acc, str} - defp parse_specifier(<>), do: {<>, rest} + # Keep the conversion char as a raw integer code point so apply_format_spec/2 + # dispatches on BEAM integer patterns rather than one-byte binaries. + defp parse_specifier(<>), do: {c, rest} defp parse_specifier("") do raise ArgumentError, @@ -413,21 +430,21 @@ defmodule Lua.VM.Stdlib.String do defp apply_format_spec({flags, width, precision, specifier}, arg) do raw = case specifier do - "d" -> format_spec_integer(arg) - "i" -> format_spec_integer(arg) - "u" -> format_spec_unsigned(arg) - "f" -> format_spec_float(arg, precision || 6) - "e" -> format_spec_scientific(arg, precision || 6, :lower) - "E" -> format_spec_scientific(arg, precision || 6, :upper) - "g" -> format_spec_general(arg, precision || 6, :lower) - "G" -> format_spec_general(arg, precision || 6, :upper) - "x" -> format_spec_hex(arg, :lower) - "X" -> format_spec_hex(arg, :upper) - "o" -> format_spec_octal(arg) - "c" -> format_char(arg) - "s" -> format_spec_string(arg, precision) - "q" -> format_quoted(arg) - _ -> raise ArgumentError, function_name: "string.format", details: "invalid option '%#{specifier}'" + ?d -> format_spec_integer(arg) + ?i -> format_spec_integer(arg) + ?u -> format_spec_unsigned(arg) + ?f -> format_spec_float(arg, precision || 6) + ?e -> format_spec_scientific(arg, precision || 6, :lower) + ?E -> format_spec_scientific(arg, precision || 6, :upper) + ?g -> format_spec_general(arg, precision || 6, :lower) + ?G -> format_spec_general(arg, precision || 6, :upper) + ?x -> format_spec_hex(arg, :lower) + ?X -> format_spec_hex(arg, :upper) + ?o -> format_spec_octal(arg) + ?c -> format_char(arg) + ?s -> format_spec_string(arg, precision) + ?q -> format_quoted(arg) + _ -> raise ArgumentError, function_name: "string.format", details: "invalid option '%#{<>}'" end apply_width_flags(raw, flags, width) @@ -838,12 +855,14 @@ defmodule Lua.VM.Stdlib.String do if deficit <= 0 do str else - pad_char = - if String.contains?(flags, "0") and not String.contains?(flags, "-"), do: "0", else: " " + minus? = (flags &&& @flag_minus) != 0 + zero? = (flags &&& @flag_zero) != 0 + + pad_char = if zero? and not minus?, do: "0", else: " " pad = String.duplicate(pad_char, deficit) - if String.contains?(flags, "-") do + if minus? do # Left justify str <> pad else From aa875d566ab6160c2196759df8696e7f4464a86c Mon Sep 17 00:00:00 2001 From: Dave Lucia Date: Mon, 1 Jun 2026 09:59:00 -0700 Subject: [PATCH 3/3] chore(B12): mark plan as review --- .../plans/B12-string-format-flags-bitmask.md | 39 ++++++++++++++++++- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/.agents/plans/B12-string-format-flags-bitmask.md b/.agents/plans/B12-string-format-flags-bitmask.md index ca3b1008..5aac2341 100644 --- a/.agents/plans/B12-string-format-flags-bitmask.md +++ b/.agents/plans/B12-string-format-flags-bitmask.md @@ -2,10 +2,10 @@ id: B12 title: parse string.format flags once and dispatch on integer specifier issue: 309 -pr: null +pr: 317 branch: perf/string-format-flags-bitmask base: main -status: in-progress +status: review direction: B --- @@ -152,3 +152,38 @@ unchanged); the full `mix test` run guards against any caller of improvement, the change is still a correctness-neutral simplification (one flag parse instead of N re-scans); ship it on those grounds and record the measured delta honestly in the PR. + +## What changed + +Implemented in `lib/lua/vm/stdlib/string.ex` (PR #317): + +- `parse_flags/2` now folds the flag characters into an integer bitmask + in a single pass (`@flag_minus`, `@flag_zero`, `@flag_plus`, + `@flag_space`, `@flag_hash`) instead of accumulating a binary; + `parse_format_spec/1` seeds it with `0`. `+`, space, and `#` are still + carried but unconsulted, preserving today's ignore-them behavior. +- `parse_specifier/1` returns the conversion char as a raw integer code + point; `apply_format_spec/2` dispatches on `?d`/`?i`/.../`?q` integer + patterns. The invalid-option error re-renders the char with + `<>` so the message text is byte-for-byte identical. +- `apply_width_flags/3` reads `flags &&& @flag_minus` / `@flag_zero` + instead of `String.contains?/2`, removing the per-specifier re-scan + from the padding path. All padding branches are preserved one-to-one. +- Added `import Bitwise` for `|||` / `&&&`. + +Verification: `mix test test/lua/vm/string_test.exs` (152 passed), +full `mix test` (2114 passed, 19 skipped, 1 excluded), +`mix compile --warnings-as-errors` clean. Benchmark (Apple M4, lua chunk) +showed ~+17% / +30% / +23% ips across the three workloads; the +width-flagged path (the one that carried the re-scan) gained the most. + +### Deviations from plan + +- The plan's verification named `test/lua/vm/stdlib/string_test.exs`, + which does not exist; the format coverage is in + `test/lua/vm/string_test.exs`. Used that file plus full `mix test`. +- The benchmark's optional `luaport` dep needs C Lua headers (absent in + this environment) and its dep-compile aborts `mix run` before the + runtime skip fires. Numbers were captured with `luaport` temporarily + excluded from `mix.exs` for the bench run only; `mix.exs` is unchanged + in the PR.