Skip to content

Add TinyStories to string-bench - #11306

Merged
robert3005 merged 2 commits into
developfrom
rk/string-bench-tinystories
Nov 12, 2026
Merged

robert3005 merged 2 commits into
developfrom
rk/string-bench-tinystories

Conversation

@robert3005

@robert3005 robert3005 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Add tinystories/text to the string-bench input catalog: the first 33,121 rows of the TinyStories train split (about 28 MiB). This is the size of one file that the NeMo Curator text IO benchmark writes.

Add the first 33,121 rows of the TinyStories train split as
`tinystories/text`. This is the size of one file that the NeMo Curator text
IO benchmark writes.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 force-pushed the rk/string-bench-tinystories branch from f10523e to ebeda56 Compare November 7, 2026 00:01
@robert3005 robert3005 added the changelog/chore A trivial change label Nov 7, 2026
@codspeed

codspeed Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 1917 untouched benchmarks
⏩ 47 skipped benchmarks1


Comparing rk/string-bench-tinystories (04669aa) with develop (ce2b6d0)

Open in CodSpeed

Footnotes

  1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@joseph-isaacs joseph-isaacs added the action/bench-string Run only the string encoding benchmark on this PR label Nov 9, 2026
@github-actions github-actions Bot removed the action/bench-string Run only the string encoding benchmark on this PR label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done ebeda56 string-bench 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: String Encoding 📖

Commits: PR ebeda56a vs base 0fee7df8


vortex / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/strings/amazon/book_reviews/fsst 554189079 / 554287534 / -0.0% 554333660 / 554924624 / -0.1% 1.00 / 1.00 / +0.1%
compress time/strings/amazon/book_reviews/onpair-12 985898532 / 984844800 / +0.1% 985865380 / 984379054 / +0.2% 1.00 / 1.00 / -0.0%
compress time/strings/amazon/book_reviews/onpair-16 996048220 / 997758440 / -0.2% 996045969 / 997803913 / -0.2% 1.00 / 1.00 / +0.0%
compress time/strings/amazon/book_titles/fsst 57818293 / 57986909 / -0.3% 57824712 / 57969291 / -0.2% 1.00 / 1.00 / -0.0%
compress time/strings/amazon/book_titles/onpair-12 89562427 / 89607872 / -0.1% 89537714 / 89588440 / -0.1% 1.00 / 1.00 / +0.0%
compress time/strings/amazon/book_titles/onpair-16 92361560 / 92188118 / +0.2% 92156046 / 92382694 / -0.2% 1.00 / 1.00 / +0.4%
compress time/strings/clickbench/url/shard-0/fsst 426951575 / 427596047 / -0.2% 426670536 / 427263924 / -0.1% 1.00 / 1.00 / -0.0%
compress time/strings/clickbench/url/shard-0/onpair-12 602686295 / 603337978 / -0.1% 602922752 / 603189125 / -0.0% 1.00 / 1.00 / -0.1%
compress time/strings/clickbench/url/shard-0/onpair-16 610084403 / 609558272 / +0.1% 610149361 / 609426327 / +0.1% 1.00 / 1.00 / -0.0%
compress time/strings/fineweb/text/fsst 213256355 / 213340261 / -0.0% 213146113 / 213486417 / -0.2% 1.00 / 1.00 / +0.1%
compress time/strings/fineweb/text/onpair-12 512543273 / 512183438 / +0.1% 512080229 / 511939310 / +0.0% 1.00 / 1.00 / +0.0%
compress time/strings/fineweb/text/onpair-16 525641485 / 525537156 / +0.0% 525465544 / 527018913 / -0.3% 1.00 / 1.00 / +0.3%
compress time/strings/news/headlines/fsst 332757970 / 333970186 / -0.4% 332673416 / 334059938 / -0.4% 1.00 / 1.00 / +0.1%
compress time/strings/news/headlines/onpair-12 469938522 / 470135961 / -0.0% 470042832 / 470253630 / -0.0% 1.00 / 1.00 / +0.0%
compress time/strings/news/headlines/onpair-16 480963856 / 481192724 / -0.0% 480467601 / 481080953 / -0.1% 1.00 / 1.00 / +0.1%
compress time/strings/tpch/l_comment/fsst 1060390203 / 1062961893 / -0.2% 1060455407 / 1063297594 / -0.3% 1.00 / 1.00 / +0.0%
compress time/strings/tpch/l_comment/onpair-12 1035173500 / 1035915162 / -0.1% 1035101262 / 1036062616 / -0.1% 1.00 / 1.00 / +0.0%
compress time/strings/tpch/l_comment/onpair-16 1048464631 / 1048914309 / -0.0% 1047901801 / 1048048606 / -0.0% 1.00 / 1.00 / -0.0%
compress time/strings/twitter/sentiment140/fsst 555227052 / 556330442 / -0.2% 555143611 / 556319385 / -0.2% 1.00 / 1.00 / +0.0%
compress time/strings/twitter/sentiment140/onpair-12 953395946 / 953512841 / -0.0% 953167566 / 953671623 / -0.1% 1.00 / 1.00 / +0.0%
compress time/strings/twitter/sentiment140/onpair-16 1040086139 / 1040777039 / -0.1% 1039733702 / 1041528894 / -0.2% 1.00 / 1.00 / +0.1%
decompress time/strings/amazon/book_reviews/fsst 41393974 / 41497179 / -0.2% 41418251 / 41498193 / -0.2% 1.00 / 1.00 / -0.1%
decompress time/strings/amazon/book_reviews/onpair-12 30546459 / 30524329 / +0.1% 30535796 / 30484652 / +0.2% 1.00 / 1.00 / -0.1%
decompress time/strings/amazon/book_reviews/onpair-16 33916898 / 34115516 / -0.6% 34032999 / 33993107 / +0.1% 1.00 / 1.00 / -0.7%
decompress time/strings/amazon/book_titles/fsst 3431662 / 3439138 / -0.2% 3423715 / 3444264 / -0.6% 1.00 / 1.00 / +0.4%
decompress time/strings/amazon/book_titles/onpair-12 3149038 / 3146134 / +0.1% 3154714 / 3151074 / +0.1% 1.00 / 1.00 / -0.0%
decompress time/strings/amazon/book_titles/onpair-16 3006646 / 3008589 / -0.1% 3007339 / 3052464 / -1.5% 1.00 / 0.99 / +1.4%
decompress time/strings/clickbench/url/shard-0/fsst 31819168 / 31804358 / +0.0% 31843866 / 31835154 / +0.0% 1.00 / 1.00 / +0.0%
decompress time/strings/clickbench/url/shard-0/onpair-12 18812903 / 18924676 / -0.6% 18736816 / 18916068 / -0.9% 1.00 / 1.00 / +0.4%
decompress time/strings/clickbench/url/shard-0/onpair-16 18107853 / 18205350 / -0.5% 17942917 / 17898952 / +0.2% 1.01 / 1.02 / -0.8%
decompress time/strings/fineweb/text/fsst 10363805 / 10353496 / +0.1% 10361119 / 10346716 / +0.1% 1.00 / 1.00 / -0.0%
decompress time/strings/fineweb/text/onpair-12 9628740 / 9547964 / +0.8% 9467365 / 9523932 / -0.6% 1.02 / 1.00 / +1.4%
decompress time/strings/fineweb/text/onpair-16 8434883 / 8457915 / -0.3% 8424982 / 8465941 / -0.5% 1.00 / 1.00 / +0.2%
decompress time/strings/news/headlines/fsst 17658657 / 17671997 / -0.1% 17621296 / 17733782 / -0.6% 1.00 / 1.00 / +0.6%
decompress time/strings/news/headlines/onpair-12 17058798 / 17066649 / -0.0% 17001590 / 17052343 / -0.3% 1.00 / 1.00 / +0.3%
decompress time/strings/news/headlines/onpair-16 16057159 / 16137846 / -0.5% 16093655 / 16164676 / -0.4% 1.00 / 1.00 / -0.1%
decompress time/strings/tpch/l_comment/fsst 50827467 / 50673215 / +0.3% 51106511 / 50539847 / +1.1% 0.99 / 1.00 / -0.8%
decompress time/strings/tpch/l_comment/onpair-12 42643652 / 42915978 / -0.6% 42927960 / 42933388 / -0.0% 0.99 / 1.00 / -0.6%
decompress time/strings/tpch/l_comment/onpair-16 42344376 / 42910005 / -1.3% 42299118 / 42643980 / -0.8% 1.00 / 1.01 / -0.5%
decompress time/strings/twitter/sentiment140/fsst 36886089 / 36931046 / -0.1% 36962199 / 36925842 / +0.1% 1.00 / 1.00 / -0.2%
decompress time/strings/twitter/sentiment140/onpair-12 33814673 / 33850371 / -0.1% 33826853 / 33824777 / +0.0% 1.00 / 1.00 / -0.1%
decompress time/strings/twitter/sentiment140/onpair-16 33699527 / 33852167 / -0.5% 33732367 / 33876198 / -0.4% 1.00 / 1.00 / -0.0%
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
vortex-file-compressed size/strings/amazon/book_reviews/fsst 66249788 / 66249788 / +0.0%
vortex-file-compressed size/strings/amazon/book_reviews/onpair-12 48618844 / 48618844 / +0.0%
vortex-file-compressed size/strings/amazon/book_reviews/onpair-16 48243532 / 48243532 / +0.0%
vortex-file-compressed size/strings/amazon/book_titles/fsst 5434916 / 5434916 / +0.0%
vortex-file-compressed size/strings/amazon/book_titles/onpair-12 4765620 / 4765620 / +0.0%
vortex-file-compressed size/strings/amazon/book_titles/onpair-16 4892932 / 4892932 / +0.0%
vortex-file-compressed size/strings/clickbench/url/shard-0/fsst 49403932 / 49403932 / +0.0%
vortex-file-compressed size/strings/clickbench/url/shard-0/onpair-12 29634060 / 29634060 / +0.0%
vortex-file-compressed size/strings/clickbench/url/shard-0/onpair-16 30002068 / 30002068 / +0.0%
vortex-file-compressed size/strings/fineweb/text/fsst 17098836 / 17098836 / +0.0%
vortex-file-compressed size/strings/fineweb/text/onpair-12 13554932 / 13554932 / +0.0%
vortex-file-compressed size/strings/fineweb/text/onpair-16 12265644 / 12265644 / +0.0%
vortex-file-compressed size/strings/news/headlines/fsst 30686308 / 30686308 / +0.0%
vortex-file-compressed size/strings/news/headlines/onpair-12 26439252 / 26439252 / +0.0%
vortex-file-compressed size/strings/news/headlines/onpair-16 26916196 / 26916196 / +0.0%
vortex-file-compressed size/strings/tpch/l_comment/fsst 75050292 / 75050292 / +0.0%
vortex-file-compressed size/strings/tpch/l_comment/onpair-12 57417388 / 57417388 / +0.0%
vortex-file-compressed size/strings/tpch/l_comment/onpair-16 57453556 / 57453556 / +0.0%
vortex-file-compressed size/strings/twitter/sentiment140/fsst 63507044 / 63507044 / +0.0%
vortex-file-compressed size/strings/twitter/sentiment140/onpair-12 54111996 / 54111996 / +0.0%
vortex-file-compressed size/strings/twitter/sentiment140/onpair-16 56477060 / 56477060 / +0.0%

@joseph-isaacs joseph-isaacs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we hook it up in ci?

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 enabled auto-merge (squash) November 9, 2026 18:55
@robert3005
robert3005 merged commit 0734d99 into develop Nov 12, 2026
72 checks passed
@robert3005
robert3005 deleted the rk/string-bench-tinystories branch November 12, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants