Validate Unicode scalar values before native UTF-8 encoding #407

cpp.encoding.Utf8.encode(codepoint, buffer) accepts surrogate values and values outside the Unicode range, and writes invalid UTF-8 bytes to the output. The corresponding UTF-16 encoder rejects the same inputs.

Reproduced against current main at 5f7cfef66ed4d9543c6c1a2763bdc2148433435c, using Haxe nightly 5.0.0-preview.1+fe63b35 with runtime API 500, MSVC 19.44.35213 on Windows x64 and GCC 13.3 on Linux x86-64 under WSL. Builds use HXCPP_M64, default GC settings and smart strings.

The test passes 0xD800, 0xDFFF, 0x110000 and -1 through cpp.Char32, using a four-byte output buffer filled with 0x55. Every UTF-8 call succeeds and changes the buffer. Every UTF-16 call rejects the same value. Valid boundary values from U+0000 through U+10FFFF still round-trip.

In Utf8::encode(char32_t, View<uint8_t>), the branch selection only checks the ranges needed for one, two, three or four bytes. There is no scalar-value validation before writing. In particular, surrogate values enter the three-byte branch, and all larger values enter the four-byte branch.

Reject surrogate and out-of-range values before writing the output, consistently with the UTF-16 encoder. Audit the scalar byte-count helpers for the same validation contract. No compiler or standard library change is needed. This concerns encoding supplied codepoints, separately from the malformed-input decoding work in #71.

A conventional regression is prepared in test/regression/Utf8InvalidScalars/{Main.hx,build.hxml,stdout.txt}. Expected output has all four checks true. Actual output:

invalid scalars are rejected by UTF8 encoding: false
invalid scalars leave the output buffer unchanged: false
UTF16 encoding rejects the same invalid scalars: true
valid boundary scalars round trip: true

#if (haxe_ver >= 5 && !haxe_5_preview_1)
using cpp.marshal.ViewExtensions;
#end
class Main {
	static function main() {
		var rejects = true;
		var untouched = true;
		var utf16Control = true;
		var valid = true;
		#if (haxe_ver >= 5 && !haxe_5_preview_1)
		for (point in [0xD800, 0xDFFF, 0x110000, -1]) {
			var bytes = haxe.io.Bytes.alloc(4);
			bytes.fill(0, 4, 0x55);
			var refused = false;
			try cpp.encoding.Utf8.encode((point:cpp.Char32), bytes.asView()) catch (_:Dynamic) refused = true;
			if (!refused) rejects = false;
			if (bytes.toHex() != "55555555") untouched = false;
			refused = false;
			try cpp.encoding.Utf16.encode((point:cpp.Char32), bytes.asView()) catch (_:Dynamic) refused = true;
			if (!refused) utf16Control = false;
		}
		for (point in [0, 0x7F, 0x80, 0xD7FF, 0xE000, 0x10FFFF]) {
			var bytes = haxe.io.Bytes.alloc(4);
			var length = cpp.encoding.Utf8.encode((point:cpp.Char32), bytes.asView());
			if (cpp.encoding.Utf8.codepoint(bytes.asView().slice(0, length)) != point) valid = false;
		}
		#end
		Sys.println("invalid scalars are rejected by UTF8 encoding: " + rejects);
		Sys.println("invalid scalars leave the output buffer unchanged: " + untouched);
		Sys.println("UTF16 encoding rejects the same invalid scalars: " + utf16Control);
		Sys.println("valid boundary scalars round trip: " + valid);
	}
}

Build with the Haxe nightly using haxe -main Main -cpp bin -D HXCPP_M64, then run bin/Main (bin/Main.exe on Windows). This exercises API 500; older compiler configurations skip the new encoding API.