Repository navigation
Improper truncation when reading from UTF-8 CHAR(n) fields containing characters outside of the Basic Multilingual Plane #1213
Description
Activity
That's an interesting one. :) I'll look at it.
I had a similar issue in Jaybird (see FirebirdSQL/jaybird#760 and FirebirdSQL/jaybird@45ad2eb)
- added a commit that references this issue
on Apr 3, 2025 Thank you for fixing this. However, there is still an edge case that causes issues, namely mixing characters from both within an without the BMP. As an example, here is the
HighLowSurrogatePassingTest, modified by adding an exclamation mark after the emoji:await using (var cmd = Connection.CreateCommand()) { const string Value = "😊!"; cmd.CommandText = "select cast(@value1 as varchar(2) character set utf8), cast(@value2 as char(2) character set utf8) from rdb$database"; cmd.Parameters.Add("value1", Value); cmd.Parameters.Add("value2", Value); await using (var reader = await cmd.ExecuteReaderAsync()) { await reader.ReadAsync(); Assert.AreEqual(Value, reader[0]); Assert.AreEqual(Value, reader[1]); } }
This modified test fails on the second assert, because the exclamation mark has been removed during reading. The issue seems to be that the removal of padding characters is still done using the regular
string.Substringmethod, which will likely need a rune-based alternative as well.Good catch. Let me look at that.
- added a commit that references this issue
on Apr 13, 2025 Fixed in bd0db88. Let me know whether you see another some missing edge case.
Fixed in bd0db88. Let me know whether you see another some missing edge case.
Thanks, looks good to me! I think there is a small difference in how the new
EnumerateRunesExmethod handles invalid strings (i.e. incomplete surrogate pairs) between .NET and .NET Standard/Framework, but as far as I can tell and have been able to test, it should not affect things. The resulting rune count is still the same in both versions, and strings converted from UTF-8 (e.g. read from the database) should not be able to contain incomplete surrogate pairs.Is there an estimated release date for the next version?
Is there an estimated release date for the next version?
This week.
When using the ADO.NET provider to read a UTF-8 CHAR(n) field containing at least one character outside of the Basic Multilingual Plane (e.g. any emoji), the result will be improperly truncated. As an example, reading a CHAR(1) field containing the character '😊' (code point 0x1F60A) will result in a string value containing only the high surrogate (0xD83D). If this same character is stored in a VARCHAR(1) field, reading it works as expected.
I believe the cause of this issue can be found in
GdsStatement.ReadRawValue:NETProvider/src/FirebirdSql.Data.FirebirdClient/Client/Managed/Version10/GdsStatement.cs
Lines 1534 to 1539 in 4230c1e
After reading the string value from the
IXdrReader, that value is truncated to remove the extra characters that were present in the buffer as padding. However, this truncation combines usage of theDbField.CharCountproperty (the number of Unicode code points stored in the field) with the .NETstring.Lengthproperty andstring.Substringmethod (which are based on the number of UTF-16 code units), leading to incorrect behavior when a single code point is encoded using multiple code units.