Skip to content

Add a native benchmark suite, recorded baselines and performance regression checks #151

Closed

Description

@dimensionscape

There is no dedicated benchmark suite, checked-in performance baseline, or automated performance regression gate in the repository. Correctness tests and GC stress tests do not tell us whether a change made allocation, collection, or ordinary native code slower. We also have no recorded numbers to support performance claims or compare proposed GC improvements.

I'd like a repeatable native benchmark suite first, then recorded results and regression checks built on it.

Initial coverage:

  • GC allocation throughput for small objects, strings, arrays and large buffers.
  • Collection pause distributions, including p50, p95, p99 and maximum, with collection counts and time spent collecting. Separate collection work from time waiting for mutators where practical.
  • Memory behavior: live data, occupied lines, reserved heap, large allocations and process peak memory, with each metric clearly defined.
  • Short-lived allocations, retained object graphs, fragmentation, allocation bursts followed by a small steady workload, and several allocating threads.
  • Native runtime operations such as ObjectMap, IntMap, StringMap, string construction and array sorting.

The suite should run locally with one documented command and validate its results so an optimization cannot look faster by doing incorrect or less work. Keep workload setup outside the timed section unless startup is what the benchmark measures. Use warmups, repeated samples and fixed workload seeds, and retain raw samples rather than only an average.

Record the commit, Haxe version, runtime configuration, compiler and optimization flags, OS, CPU, architecture and thread count with every result. Separate normal, moving, generational and big-block configurations. Verifier and sanitizer builds need separate results because their overhead makes direct comparisons misleading.

Start with a recorded baseline and a controlled comparison against the base commit on the same machine. Shared CI runners can produce useful trend reports, but hard regression gates need measured noise tolerances or a controlled runner. Flag changes in throughput, tail pauses and memory separately; improving one can worsen another.

A first deliverable should include the benchmark sources, run instructions, a baseline report with raw results, and a comparison command. Then add CI result retention and regression reporting, with blocking thresholds only after we understand normal variation. Keep a small PR subset and a broader scheduled suite if full coverage is too expensive.

This would give performance investigations such as #77, #83, #177 and #179 a common way to establish a benefit and catch regressions. It also covers performance-only fixes such as #141, which currently have no repeatable performance check.

Comparison with stock hxcpp (for 6.0.0)

The suite also runs against stock hxcpp, as a separate performance regression suite, so we can show how libhxcpp compares:

  • Build each benchmark twice from the same Haxe sources and the same Haxe version: once with libhxcpp, once with the latest stock hxcpp release (record its version and commit). Same machine, same compiler and flags, same runtime configuration where both support it.
  • Pick the metrics that matter most for users and keep them stable between releases: allocation throughput, collection pause p50/p99/max, peak memory, map operations, string operations and sorting, and startup time if it's measurable.
  • Results go in a table in README.md with libhxcpp, stock hxcpp and the ratio, plus the machine, OS, compiler, Haxe version and commits used. The table states how it was produced and links the command to reproduce it.
  • Report regressions as well as improvements. If libhxcpp is slower on a metric, the table says so.
  • The comparison is a separate command from the PR regression check, since it needs a stock hxcpp install and doesn't block merges.

Activity

  1. changed the title [-]Track runtime performance in CI[/-] [+]Add a native benchmark suite, recorded baselines and performance regression checks[/+] on Oct 17, 2026
  2. dimensionscape commented on Oct 21, 2026

    @dimensionscape
    MemberAuthor

    Done: the native benchmark suite in bench/ covers allocation, collection pauses, memory, maps, strings, sorting and cppia, with warmups, repeated samples and validated results; it compares libhxcpp with stock hxcpp on the same machine and writes the README and BENCHMARKS.md tables with the machine, compiler, Haxe version and commits. Recording results in CI and reporting regressions is now #258.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions