Repository navigation
Description
Bug Report
1. Describe the bug
On row format V2, TableCodec.decodeObjects returns a String (with silent
U+FFFD replacement) for BINARY/VARBINARY columns, instead of byte[].
Any non-UTF-8 byte value (e.g. 0xFF) is irreversibly corrupted. The V1 path
returns byte[] correctly for the same column, so V1 and V2 disagree.
Root cause:
DataTypeFactory.convertTyperemapsTypeString → TypeBlobfor
binary-charset string columns — but this only selects the DataType
subclass (BytesType); it does not change the instance's type field.DataType(InternalTypeHolder holder)re-derives the field from the raw
code:this.tp = MySQLType.fromTypeCode(holder.getTp())(DataType.java),
so the created instance is of classBytesTypebut carries
getType() == TypeStringfor aBINARYcolumn.RowDecoderV2.decodeColswitches ontp.getType()(the field, not the
class) → falls intocase TypeString:→new String(colData, UTF_8)
(RowDecoderV2.java:54-57), silently replacing invalid sequences.- The V1 path (
TableCodecV1) dispatches polymorphically on the class
(DataType.decode(...)→BytesType.decodeNotNull→byte[]), hence the
V1/V2 divergence.
convertType exists so the class handles binary content, but the V2 switch
ignores the class entirely: BINARY (field=TypeString) and VARBINARY
(field=TypeVarchar) both land in the string branch. BLOB family is unaffected
(field stays a blob code → case TypeBlob: return colData).
2. Minimal reproduce step (Required)
-- TiDB, row format V2 (default for new installs)
CREATE TABLE t (id INT PRIMARY KEY, c BINARY(1));
INSERT INTO t VALUES (1, 0xFF);// value = TableCodec-encoded row value bytes (e.g. from a CDC event / TiKV read),
// handle = row handle, tableInfo = session.getCatalog().getTable("db", "t")
Object[] vals = TableCodec.decodeObjects(value, handle, tableInfo);
System.out.println(vals[1].getClass()); // class java.lang.String
System.out.println(vals[1]); // "\uFFFD" (0xEFBFBD) — original 0xFF lost3. What did you see instead (Required)
The BINARY(1) column value is a java.lang.String containing U+FFFD
(EF BF BD). The original byte 0xFF is irreversibly lost.
4. What did you expect to see? (Required)
byte[] { (byte) 0xFF } — raw column bytes, as the V1 path
(BytesType.decodeNotNull) already returns for the same column.
5. What are your Java Client and TiKV versions? (Required)
- Client Java: 3.2.0, 3.3.5 and master (all affected —
RowDecoderV2.java:54-57
identical in all three; both files last touched 2021-12-30) - TiKV: TiKV 8.5.x (observed via TiDB 8.5.6,
tidb_row_format_version=2)
6. Suggested fix
Check charset before the string branch in RowDecoderV2.decodeCol:
case TypeString:
case TypeVarString:
case TypeVarchar:
if (Charset.CharsetBin.equals(tp.getCharset())) {
return colData;
}
return new String(colData, StandardCharsets.UTF_8);
Activity
ti-chi-bot commentedon Nov 3, 2026
Welcome @juefengyi! It looks like this is your first issue to tikv/client-java 🎉