Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 189 additions & 0 deletions .agents/plans/B12-string-format-flags-bitmask.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
---
id: B12
title: parse string.format flags once and dispatch on integer specifier
issue: 309
pr: 317
branch: perf/string-format-flags-bitmask
base: main
status: review
direction: B
---

# B12 — string.format flag bitmask + integer specifier dispatch

## Goal

Speed up the `string.format` per-specifier hot path in
`lib/lua/vm/stdlib/string.ex` by removing redundant per-site work, with
zero change to observable output:

1. **Parse flags once.** Today `parse_flags/2` collects the flag
characters into a binary, and `apply_width_flags/3` re-scans that
binary with `String.contains?(flags, "0")`, `String.contains?(flags,
"-")` on *every* specifier application. Parse the flags exactly once
at parse time into a small fixed-shape representation (a tuple of
booleans, or a single integer mask) so the apply path reads
precomputed fields instead of re-scanning a binary.
2. **Dispatch on an integer specifier.** Today `parse_specifier/1`
returns the conversion character as a one-byte binary (`"d"`,
`"f"`, …) and `apply_format_spec/2` dispatches with a `case` over
string literals. Store and match the conversion char as an integer
code point (`?d`, `?f`, …) so the `case` compiles to BEAM
integer-pattern dispatch, which is faster than binary matching.

This mirrors Luerl's precomputed bitmask + `$d`-style integer dispatch
(`deps/luerl/src/luerl_lib_string_format.erl`). Expected to close ~10%
of the spec-heavy gap; stacks with the sibling string.format plans.

## Out of scope

- Any change to formatted output. This is a pure internal refactor; the
rendered string for every input must be byte-for-byte identical.
- The per-specifier formatter bodies: `format_spec_integer/1`,
`format_spec_unsigned/1`, `format_spec_float/2`,
`format_spec_scientific/3`, `format_spec_general/3`,
`format_spec_hex/2`, `format_spec_octal/1`, `format_char/1`,
`format_spec_string/2`, `format_quoted/1`, and their helpers. Sibling
plans #310 and #311 edit `string.ex` too; this plan must not touch
those bodies. Only the flag representation, the specifier token type,
and the dispatch/padding sites that consume them change.
- Starting to honor flags that are currently parsed-and-discarded.
`parse_flags/2` accepts `-+ 0#`, but only `0` and `-` are consulted
downstream; `+`, space, and `#` are parsed and ignored today. The
refactored representation must preserve that exact behavior (it may
carry those bits, but they must not change output).
- `string.format` argument validation, error messages, or arity.
- Any other stdlib function or any file other than
`lib/lua/vm/stdlib/string.ex`.

## Success criteria

- [ ] `string.format` produces byte-for-byte identical output for all
existing tests (no regression in
`test/lua/vm/stdlib/string_test.exs`).
- [ ] Flags are parsed exactly once, at parse time, into a fixed-shape
representation (boolean tuple or integer mask). No
`String.contains?(flags, ...)` re-scan remains on the
per-specifier apply path (`apply_width_flags/3`).
- [ ] The conversion specifier is stored and matched as an integer
code point (`?d`/`?f`/…), not a one-byte binary; the
`apply_format_spec/2` `case` dispatches on integers.
- [ ] Flags currently parsed-but-ignored (`+`, space, `#`) remain
ignored — output unchanged.
- [ ] `mix compile --warnings-as-errors` is clean.
- [ ] Full `mix test` passes with no regressions.
- [ ] `mix run benchmarks/string_format.exs` runs to completion; record
before/after numbers in the PR description.

## Implementation notes

All edits are confined to `lib/lua/vm/stdlib/string.ex`, in the format
parser and dispatch region (roughly lines 365–434 and 828–859):

1. **Flag representation.** Change `parse_flags/2` (line 374) to emit a
fixed-shape value instead of accumulating a binary. Two acceptable
shapes (pick one and keep it internal):
- a tuple/struct of booleans, e.g. `{minus?, zero?, plus?, space?,
hash?}`, or
- a single integer mask with named bit constants.
Whichever is chosen, it is computed once during
`parse_format_spec/1` (line 366) and threaded through the
`{flags, width, precision, specifier}` tuple unchanged in arity/shape
at the tuple level (only the `flags` element's type changes).

2. **Specifier as integer.** Change `parse_specifier/1` (line 404) from
`{<<c>>, rest}` to `{c, rest}` (the raw integer code point). Update
the empty-string clause (line 406) unchanged. Rewrite the
`apply_format_spec/2` `case` (lines 415–431) to match integer code
points: `?d`, `?i`, `?u`, `?f`, `?e`, `?E`, `?g`, `?G`, `?x`, `?X`,
`?o`, `?c`, `?s`, `?q`, with the `_ ->` fallback raising the same
`"invalid option '%#{specifier}'"` error. NOTE: the interpolation
`#{specifier}` currently relies on `specifier` being a binary; with
an integer it must be rendered back to its character for the error
message (e.g. `<<specifier>>` or `List.to_string([specifier])`) so
the error text is unchanged.

3. **Padding path.** Rewrite `apply_width_flags/3` (line 828) to read
the precomputed flag fields instead of `String.contains?(flags,
"0")` (line 842), `String.contains?(flags, "-")` (line 846). The
pad-char selection (`"0"` when zero-flag set and minus-flag unset,
else `" "`), left-justify branch, and the sign-aware zero-pad branch
(line 852, `"-" <> pad <> binary_part(...)`) must remain semantically
identical.

4. Run `mix format` after edits. Keep the `# Format string parser`
comments accurate; do not add any plan-id references in source.

## Verification

```bash
mix format
mix compile --warnings-as-errors
mix test test/lua/vm/stdlib/string_test.exs
mix test
mix run benchmarks/string_format.exs
```

Capture the `benchmarks/string_format.exs` output before and after the
change and paste the comparison into the PR description. The
`string_test.exs` run is the primary correctness gate (output
unchanged); the full `mix test` run guards against any caller of
`string.format` elsewhere in the suite.

## Risks

- **Silent output drift.** The dispatch and flag-handling rewrite could
subtly change a padding or specifier edge case. Mitigation: the
refactor preserves every branch one-to-one; `string_test.exs` plus
full `mix test` are the gate. If a case is thin in coverage (e.g. the
zero-pad-with-sign branch), confirm an existing test exercises it
before relying on green.
- **Error-message regression.** The invalid-option error interpolates
the specifier; switching it to an integer requires re-rendering the
character so `'%z'`-style messages stay identical. Called out in
Implementation note 2.
- **Sibling-file collision.** #310 and #311 also edit `string.ex`. This
plan stays inside the flag-parsing / specifier-dispatch region and
does not touch the `format_spec_*` formatter bodies, minimizing merge
conflicts. If a sibling lands first, rebase and re-confirm the
apply-path region is still as described.
- **No measurable win.** Per the Direction B findings, BEAM dispatch
refactors do not always pay off. If the benchmark shows no
improvement, the change is still a correctness-neutral simplification
(one flag parse instead of N re-scans); ship it on those grounds and
record the measured delta honestly in the PR.

## What changed

Implemented in `lib/lua/vm/stdlib/string.ex` (PR #317):

- `parse_flags/2` now folds the flag characters into an integer bitmask
in a single pass (`@flag_minus`, `@flag_zero`, `@flag_plus`,
`@flag_space`, `@flag_hash`) instead of accumulating a binary;
`parse_format_spec/1` seeds it with `0`. `+`, space, and `#` are still
carried but unconsulted, preserving today's ignore-them behavior.
- `parse_specifier/1` returns the conversion char as a raw integer code
point; `apply_format_spec/2` dispatches on `?d`/`?i`/.../`?q` integer
patterns. The invalid-option error re-renders the char with
`<<specifier>>` so the message text is byte-for-byte identical.
- `apply_width_flags/3` reads `flags &&& @flag_minus` / `@flag_zero`
instead of `String.contains?/2`, removing the per-specifier re-scan
from the padding path. All padding branches are preserved one-to-one.
- Added `import Bitwise` for `|||` / `&&&`.

Verification: `mix test test/lua/vm/string_test.exs` (152 passed),
full `mix test` (2114 passed, 19 skipped, 1 excluded),
`mix compile --warnings-as-errors` clean. Benchmark (Apple M4, lua chunk)
showed ~+17% / +30% / +23% ips across the three workloads; the
width-flagged path (the one that carried the re-scan) gained the most.

### Deviations from plan

- The plan's verification named `test/lua/vm/stdlib/string_test.exs`,
which does not exist; the format coverage is in
`test/lua/vm/string_test.exs`. Used that file plus full `mix test`.
- The benchmark's optional `luaport` dep needs C Lua headers (absent in
this environment) and its dep-compile aborts `mix run` before the
runtime skip fires. Numbers were captured with `luaport` temporarily
excluded from `mix.exs` for the bench run only; `mix.exs` is unchanged
in the PR.
67 changes: 43 additions & 24 deletions lib/lua/vm/stdlib/string.ex
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ defmodule Lua.VM.Stdlib.String do

@behaviour Lua.VM.Stdlib.Library

import Bitwise

alias Lua.VM.ArgumentError
alias Lua.VM.Executor
alias Lua.VM.State
Expand Down Expand Up @@ -364,18 +366,31 @@ defmodule Lua.VM.Stdlib.String do

# Parse a format spec: [flags][width][.precision]specifier
defp parse_format_spec(str) do
{flags, str} = parse_flags(str, "")
{flags, str} = parse_flags(str, 0)
{width, str} = parse_width(str)
{precision, str} = parse_precision(str)
{specifier, str} = parse_specifier(str)
{{flags, width, precision, specifier}, str}
end

defp parse_flags(<<c, rest::binary>>, acc) when c in ~c(-+ 0#) do
parse_flags(rest, acc <> <<c>>)
end
# Flags are parsed once into an integer bitmask so the apply path reads
# precomputed bits instead of re-scanning a binary per specifier. Only the
# minus and zero bits affect output today (`+`, space, `#` are accepted by
# PUC-Lua's parser but ignored when rendering); the mask carries all five so
# the parse stays a single pass without changing behavior.
@flag_minus 0b00001
@flag_zero 0b00010
@flag_plus 0b00100
@flag_space 0b01000
@flag_hash 0b10000

defp parse_flags(str, acc), do: {acc, str}
defp parse_flags(<<?-, rest::binary>>, mask), do: parse_flags(rest, mask ||| @flag_minus)
defp parse_flags(<<?0, rest::binary>>, mask), do: parse_flags(rest, mask ||| @flag_zero)
defp parse_flags(<<?+, rest::binary>>, mask), do: parse_flags(rest, mask ||| @flag_plus)
defp parse_flags(<<?\s, rest::binary>>, mask), do: parse_flags(rest, mask ||| @flag_space)
defp parse_flags(<<?#, rest::binary>>, mask), do: parse_flags(rest, mask ||| @flag_hash)

defp parse_flags(str, mask), do: {mask, str}

defp parse_width(<<c, _::binary>> = str) when c in ?0..?9 do
parse_number(str, 0)
Expand All @@ -401,7 +416,9 @@ defmodule Lua.VM.Stdlib.String do

defp parse_number(str, acc), do: {acc, str}

defp parse_specifier(<<c, rest::binary>>), do: {<<c>>, rest}
# Keep the conversion char as a raw integer code point so apply_format_spec/2
# dispatches on BEAM integer patterns rather than one-byte binaries.
defp parse_specifier(<<c, rest::binary>>), do: {c, rest}

defp parse_specifier("") do
raise ArgumentError,
Expand All @@ -413,21 +430,21 @@ defmodule Lua.VM.Stdlib.String do
defp apply_format_spec({flags, width, precision, specifier}, arg) do
raw =
case specifier do
"d" -> format_spec_integer(arg)
"i" -> format_spec_integer(arg)
"u" -> format_spec_unsigned(arg)
"f" -> format_spec_float(arg, precision || 6)
"e" -> format_spec_scientific(arg, precision || 6, :lower)
"E" -> format_spec_scientific(arg, precision || 6, :upper)
"g" -> format_spec_general(arg, precision || 6, :lower)
"G" -> format_spec_general(arg, precision || 6, :upper)
"x" -> format_spec_hex(arg, :lower)
"X" -> format_spec_hex(arg, :upper)
"o" -> format_spec_octal(arg)
"c" -> format_char(arg)
"s" -> format_spec_string(arg, precision)
"q" -> format_quoted(arg)
_ -> raise ArgumentError, function_name: "string.format", details: "invalid option '%#{specifier}'"
?d -> format_spec_integer(arg)
?i -> format_spec_integer(arg)
?u -> format_spec_unsigned(arg)
?f -> format_spec_float(arg, precision || 6)
?e -> format_spec_scientific(arg, precision || 6, :lower)
?E -> format_spec_scientific(arg, precision || 6, :upper)
?g -> format_spec_general(arg, precision || 6, :lower)
?G -> format_spec_general(arg, precision || 6, :upper)
?x -> format_spec_hex(arg, :lower)
?X -> format_spec_hex(arg, :upper)
?o -> format_spec_octal(arg)
?c -> format_char(arg)
?s -> format_spec_string(arg, precision)
?q -> format_quoted(arg)
_ -> raise ArgumentError, function_name: "string.format", details: "invalid option '%#{<<specifier>>}'"
end

apply_width_flags(raw, flags, width)
Expand Down Expand Up @@ -838,12 +855,14 @@ defmodule Lua.VM.Stdlib.String do
if deficit <= 0 do
str
else
pad_char =
if String.contains?(flags, "0") and not String.contains?(flags, "-"), do: "0", else: " "
minus? = (flags &&& @flag_minus) != 0
zero? = (flags &&& @flag_zero) != 0

pad_char = if zero? and not minus?, do: "0", else: " "

pad = String.duplicate(pad_char, deficit)

if String.contains?(flags, "-") do
if minus? do
# Left justify
str <> pad
else
Expand Down
Loading