ext/standard: Add php_next_utf8_char_inline()

[ArtUkrainskiy]

json_encode(), utf8_decode() and the xml extension decode one character at a time with php_next_utf8_char(). This adds php_next_utf8_char_inline(), which decodes valid UTF-8 at the call site and keeps the existing function as the fallback for invalid input and as the exported API.

Non-ASCII text in json_encode() gets faster across all six compiler/CPU configurations tested: Cyrillic takes 0.70–0.91 of master's cycles, CJK 0.68–0.84 and emoji 0.82–0.89. With GCC 13 and 14 on the Intel machine some ASCII-heavy inputs get slower at the same instruction count, from code placement; see Trade-offs.

Before. php_next_utf8_char() lives in html.c, so it cannot be inlined into ext/json. Inside it, get_next_char() handles 14 charsets and was not inlined either, so each character paid two calls and a switch on the charset, and the value, the new position and the status came back through memory: about 43 instructions per non-ASCII character in callgrind (36 in get_next_char(), 7 in the wrapper).

After. php_next_utf8_char_inline() in ext/standard/html.h has the same contract as php_next_utf8_char(). It decodes ASCII and valid two-, three- and four-byte sequences without a call; every invalid or truncated sequence goes to php_next_utf8_char(), which is unchanged and still exported.

Why the shortcut is exact. A lead byte C2..DF followed by a continuation byte 80..BF is always a valid shortest-form character: C0 and C1 are the only overlong two-byte leads. For three bytes the inline path checks the same two conditions as get_next_char(): the result is at least U+0800 (not overlong) and outside U+D800..U+DFFF (not a surrogate). For four bytes it takes lead bytes F0..F4 only and checks that the result is in U+10000..U+10FFFF, as get_next_char() does. Any other input runs the old code, so which bytes are rejected or skipped and the returned status stay exactly as before.

EXPECTED(c < 0xe0) keeps the two-byte case on the straight path. Without it GCC 13 placed that case behind two extra taken jumps: Cyrillic ran the same instructions but took 20% more cycles.

Trade-offs

• Code placement with GCC 13 and 14 on the Intel machine. Some inputs run the same instructions as master but take more cycles: a newline every 40 bytes (1.20 and 1.18), URLs (1.09 and 1.15), ASCII with an occasional é escaped as \u00e9 (1.10 and 1.13), and with GCC 13 also 1,000 arrays (1.10), 1,000 objects (1.07) and English text with an emoji every few words (1.05, at 0.92 of the instructions). With clang 18 and 20 and on AMD these inputs are at 0.91–1.02. The generated loop over plain bytes is identical to master's and at the same address (objdump, GCC 13). php_json_escape_string() grows by 487 bytes, so php_json_encode_zval() and php_json_encode_array(), which follow it, move by 192 bytes; on the 1,000 arrays branch mispredictions go from 1.0 to 3.6–3.9 million per 3,000 calls, at every hot branch, including code this PR does not touch. Hinting branches in the source moved slowdowns of this kind to other inputs when I tried it on an earlier version. Splitting the scan out of the loop and using a byte table removed them for the two- and three-byte version (see the appendix of the report).
• An é exactly every 31 bytes measured between 0.92 and 1.27 on the earlier version depending on compiler and process, consistent with the branch predictor learning the periodic pattern. The table uses é at random positions with the same density.
• ext/standard: Add a fast path for ASCII in htmlspecialchars() #24145 makes get_next_char() always inline, which also speeds up the out-of-line php_next_utf8_char(). Measured with the two- and three-byte version: on top of ext/standard: Add a fast path for ASCII in htmlspecialchars() #24145 this PR still removes 14–20% of the instructions on non-ASCII text, but with GCC 13 the cycles change less (Cyrillic 0.98, Cyrillic with JSON_UNESCAPED_UNICODE 0.79, CJK 0.88 of ext/standard: Add a fast path for ASCII in htmlspecialchars() #24145's cycles). The two merge without conflicts.
• API. php_next_utf8_char() and its exported symbol are unchanged; php_next_utf8_char_inline() is new (UPGRADING.INTERNALS).
• Other callers. utf8_decode() (deprecated, still shipped) runs 1.7–2.8× fewer instructions on 4 KiB of Latin-1, Cyrillic or ASCII text (GCC 13). The xml extension's decoder uses the same path; not measured.
• Size. +1,296 bytes of code with GCC 13: php_json_escape_string() +487, utf8_decode() +384, the xml decoder +425.

Results

Relative to master, lower is better. Release builds (-O2, x86-64, --enable-werror): GCC 13.3, GCC 14.2, clang 18.1 and clang 20.1 on an i7-13700H pinned to one P-core; GCC 15.2 and clang 21.1 on a Ryzen 9 5950X pinned to one core. Cycles and instructions per call from perf stat over N calls minus the same process with 0 calls; cycles are the median of alternating runs from one series per machine, 5 per row on the Intel machine and 15 on AMD, where other load was running on other cores (master against itself: medians within 2%). Instructions are given as a range over the six builds.

input (4 KiB unless noted)	instructions	GCC 13.3	GCC 14.2	clang 18.1	clang 20.1	GCC 15.2 (AMD)	clang 21.1 (AMD)
Cyrillic	0.74–0.77	0.72	0.79	0.76	0.91	0.70	0.80
Cyrillic, JSON_UNESCAPED_UNICODE	0.69–0.79	0.64	0.70	0.69	0.70	0.82	0.71
CJK	0.77–0.81	0.78	0.78	0.72	0.84	0.68	0.83
CJK, JSON_UNESCAPED_UNICODE	0.79–0.82	0.80	0.79	0.80	0.83	0.81	0.73
French	0.83–0.86	0.95	0.94	0.79	0.86	0.82	0.87
French, JSON_UNESCAPED_UNICODE	0.80–0.87	0.85	0.84	0.78	0.79	0.88	0.83
emoji	0.82–0.86	0.82	0.87	0.83	0.83	0.89	0.84
emoji, JSON_UNESCAPED_UNICODE	0.75–0.83	0.75	0.76	0.83	0.85	0.75	0.72
English text with an emoji every few words	0.92–0.94	1.05	0.96	0.91	0.91	0.97	0.94
ASCII with an é every 31 bytes on average	0.93–0.94	1.10	1.13	0.91	0.93	0.96	0.95
ASCII with an é every 31 bytes on average, JSON_UNESCAPED_UNICODE	0.92–0.94	0.96	0.96	0.93	0.90	0.94	0.92
ASCII, 16 bytes	1.00	1.00	1.00	1.01	1.00	1.00	1.04
ASCII, 4 KiB	1.00	1.01	1.00	1.00	0.99	1.01	0.99
HTML	0.99–1.00	1.00	1.05	0.98	0.99	1.05	1.00
URLs	0.99–1.00	1.09	1.15	0.99	1.00	0.99	1.00
a quote every 16 bytes	1.00	0.83	0.82	1.00	1.01	1.01	1.02
a newline every 40 bytes	1.00	1.20	1.18	1.00	1.00	1.00	1.02
a quote in every byte	1.00–1.02	1.00	1.00	1.00	1.00	0.92	1.00
1,000 arrays of 10 fields	1.00	1.10	1.00	1.01	1.00	0.99	0.99
1,000 objects of 10 declared properties	1.00	1.07	0.97	1.02	1.00	1.02	0.99

All inputs, with the spread between runs, are in the report linked below.

Verification

Output and json_last_error() are identical to master with all six compiler and CPU combinations on:

• every pair of bytes, alone and inside ASCII, and every three-byte sequence with an E0..EF lead byte, alone and inside ASCII, under six flag sets each;
• every four-byte sequence with an F0..FF lead byte, for every value of the second and third byte and the fourth byte at the continuation boundaries, alone, inside ASCII and truncated, under three flag sets;
• every byte value at every position of ASCII strings 1–24 bytes long;
• random strings of 0–300 bytes (200,000 per Intel build, 400,000 per AMD build, 200,000 on the sanitizer build), as values and as keys, mixing escapes, two-, three- and four-byte characters, U+2028/2029, overlong forms and invalid sequences, under 11 flag sets;
• literal keys through arrays, stdClass and declared properties;
• utf8_decode() on the same pairs and triples.

ext/json, ext/xml and ext/standard/tests/strings pass with the four Intel builds and on a GCC debug build with ASan and UBSan, where the exhaustive checks above report nothing either. The new test json_encode_utf8_sequences.phpt covers the edges of the inline path and the fallback (lone continuation bytes, truncated and overlong sequences, surrogates, four-byte edges, F5) under the JSON_INVALID_UTF8_*, JSON_UNESCAPED_* and JSON_PARTIAL_OUTPUT_ON_ERROR flags; it fails if the overlong checks of the two- and three-byte inline paths are removed. Separately, the inline decoder was compared with the UTF-8 case of get_next_char(), extracted from html.c, on every sequence of one to four bytes with a lead byte 80..FF at every prefix length, at cursor 0 and after one ASCII byte, and on every ASCII byte (4.3 billion calls): no difference in value, cursor or status. The same comparison reports differences when either range check of the four-byte path is removed.

Not tested: aarch64, 32-bit and MSVC builds (the change has no width-dependent arithmetic).

Full benchmarks, raw runs and reproduction:
https://github.com/ArtUkrainskiy/php-src-bench/tree/main/reports/json-encode-utf8-inline

Broader context and comparison with the other open json_encode() optimization PRs:
https://artukrainskiy.github.io/posts/json-encode-utf8-autumn-2026/

[adapik]

I went through the implementation and I find the approach reasonable: the inline path only takes sequences that are valid by construction, and everything else falls back to the unchanged php_next_utf8_char(), so the error semantics can't diverge. I also ran it through my own benchmarks and the results are solid.

Regarding overlap with my other PRs (24283 and 24282):

• The dirty table is a small, self-contained change to the clean/dirty classification of a byte. I think it stands on its own and combines well with this PR, since this PR changes what happens after a byte is classified as dirty (non-ASCII), not the classification itself.
• I don't see any overlap with the SIMD work either. The two optimizations operate on different branches of php_json_escape_string() (the SIMD one on runs of clean ASCII, this one on the non-ASCII decode), so they complement each other.

Depending on which one is merged first, I'm happy to rebase mine, that's not a problem.

[adapik]

The new test only covers json_encode(). Since the validation logic is now duplicated between html.c and html.h, a small phpt exercising the same boundary sequences through utf8_decode() would stop the two copies from drifting apart silently.

[adapik]

This is not a blocker, only my thoughts.

php_next_utf8_char() lives in html.c because it is the UTF-8 branch of get_next_char(), which exists for htmlentities() and its 14 charsets, and it was exported in 2010 only so that utf8_decode() could reuse it. Today three of its four callers (json, xml, utf8_decode()) have nothing to do with HTML entities.

This PR adds a new public declaration to ext/standard/html.h. It is not an ABI change, but it is new source-level API. Since we are creating that surface anyway, perhaps it's a good moment to give it a neutral home (e.g. a small dedicated header) rather than cementing html.h as the place for a generic UTF-8 decoder. php_next_utf8_char() can stay exported from html.c unchanged, so nothing breaks.

[ArtUkrainskiy]

@adapik Thanks for reviewing it and for running it through your benchmarks.

Test: I added ext/standard/tests/strings/utf8_decode_sequences.phpt with the same boundary sequences through utf8_decode(), plus overlong encodings of é, which utf8_decode() makes visible as é instead of "?". The expected output comes from master. To check that the two tests catch drift between the copies, I mutated the inline decoder one condition at a time:

• accepting C0/C1 leads, three- and four-byte overlong forms, skipping the third-byte check, advancing the cursor by 2 instead of 3: both tests fail;
• accepting surrogates or code points above U+10FFFF: only the json_encode() test fails, because utf8_decode() turns both a valid character above U+00FF and an invalid sequence into a single "?".

Header: I put the function in html.h next to php_next_utf8_char() because of the earlier review of #18126, where the suggestion was to optimize the existing decoder rather than add a new one elsewhere (#18126 (comment)). A small dedicated header, with php_next_utf8_char() staying exported from html.c, is a change of a few lines here.

@bukka, as the owner of ext/standard and ext/json: do you prefer the inline decoder in html.h, as it is now, or in a separate header (for example ext/standard/php_utf8.h)? I'll update the PR either way.