Skip to content

RowDecoderV2 decodes BINARY/VARBINARY as UTF-8 String, silently corrupting non-UTF-8 bytes (V1/V2 behavior diverge) #799

Open

Description

@juefengyi

Bug Report

1. Describe the bug

On row format V2, TableCodec.decodeObjects returns a String (with silent
U+FFFD replacement) for BINARY/VARBINARY columns, instead of byte[].
Any non-UTF-8 byte value (e.g. 0xFF) is irreversibly corrupted. The V1 path
returns byte[] correctly for the same column, so V1 and V2 disagree.

Root cause:

  1. DataTypeFactory.convertType remaps TypeString → TypeBlob for
    binary-charset string columns — but this only selects the DataType
    subclass
    (BytesType); it does not change the instance's type field.
  2. DataType(InternalTypeHolder holder) re-derives the field from the raw
    code: this.tp = MySQLType.fromTypeCode(holder.getTp()) (DataType.java),
    so the created instance is of class BytesType but carries
    getType() == TypeString for a BINARY column.
  3. RowDecoderV2.decodeCol switches on tp.getType() (the field, not the
    class) → falls into case TypeString: → new String(colData, UTF_8)
    (RowDecoderV2.java:54-57), silently replacing invalid sequences.
  4. The V1 path (TableCodecV1) dispatches polymorphically on the class
    (DataType.decode(...) → BytesType.decodeNotNull → byte[]), hence the
    V1/V2 divergence.

convertType exists so the class handles binary content, but the V2 switch
ignores the class entirely: BINARY (field=TypeString) and VARBINARY
(field=TypeVarchar) both land in the string branch. BLOB family is unaffected
(field stays a blob code → case TypeBlob: return colData).

2. Minimal reproduce step (Required)

-- TiDB, row format V2 (default for new installs)
CREATE TABLE t (id INT PRIMARY KEY, c BINARY(1));
INSERT INTO t VALUES (1, 0xFF);
// value = TableCodec-encoded row value bytes (e.g. from a CDC event / TiKV read),
// handle = row handle, tableInfo = session.getCatalog().getTable("db", "t")
Object[] vals = TableCodec.decodeObjects(value, handle, tableInfo);
System.out.println(vals[1].getClass());   // class java.lang.String
System.out.println(vals[1]);              // "\uFFFD" (0xEFBFBD) — original 0xFF lost

3. What did you see instead (Required)

The BINARY(1) column value is a java.lang.String containing U+FFFD
(EF BF BD). The original byte 0xFF is irreversibly lost.

4. What did you expect to see? (Required)

byte[] { (byte) 0xFF } — raw column bytes, as the V1 path
(BytesType.decodeNotNull) already returns for the same column.

5. What are your Java Client and TiKV versions? (Required)

  • Client Java: 3.2.0, 3.3.5 and master (all affected — RowDecoderV2.java:54-57
    identical in all three; both files last touched 2021-12-30)
  • TiKV: TiKV 8.5.x (observed via TiDB 8.5.6, tidb_row_format_version=2)

6. Suggested fix

Check charset before the string branch in RowDecoderV2.decodeCol:

case TypeString:
case TypeVarString:
case TypeVarchar:
    if (Charset.CharsetBin.equals(tp.getCharset())) {
        return colData;
    }
    return new String(colData, StandardCharsets.UTF_8);

Activity

  1. ti-chi-bot commented on Nov 3, 2026

    @ti-chi-bot

    Welcome @juefengyi! It looks like this is your first issue to tikv/client-java 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions