Skip to content

Marshal: a class path that is not ASCII carries its encoding, as 4.0 dumps it - #893

Merged
tannevaled merged 2 commits into
mainfrom
marshal-multibyte-classpath
Nov 5, 2026
Merged

tannevaled merged 2 commits into
mainfrom
marshal-multibyte-classpath

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Three more of the 27 examples that went red on the move to Ruby 4.0 (#799),
again one cause. core/marshal/dump_spec.rb goes 191 → 194, fail 18 → 15.

The reference

marshal.c at v4.0.9 brackets a class path with the same encoding machinery
a String goes through:

VALUE encname = w_encivar(path, arg);   /* emits 'I' when path is non-ASCII */
w_byte(TYPE_CLASS, arg);
w_bytes(RSTRING_PTR(path), RSTRING_LEN(path), arg);
w_encname(encname, arg);                /* 1 ivar: :E => true               */

w_encivar returns nil for an ASCII-only path — that single line is what keeps
every ordinary class name dumping exactly as before. encoding_name maps UTF-8
to Qtrue, so the trailer is the short :E => true form and never the long
:encoding => "…" one: a constant name here is a Go string, hence UTF-8, and
there is no way to name a constant in another encoding.

Measured, not read

rubydiff against MRI 4.0.7 on this host: AGREE, byte for byte, on all
eleven probes. The three the specs assert:

"\x04\bIc,MarshalSpec::Multibyteぁあぃい Class\x06:\x06ET"
"\x04\bIm-MarshalSpec::Multibyteけげこご Module\x06:\x06ET"
"\x04\bIc+MarshalSpec::Multibyteぁあぃい Time\x06:\x06ET"

⛔ The controls matter more than the three rows

control why
Marshal.dump(String) is still "\x04\bc\vString" a wrapper emitted unconditionally passes all three spec rows and breaks every existing dump
all three round-trip to the identical object the reader had never been handed an 'I' in front of a 'c' — and the specs do not assert this: their second line has no .should
Marshal.dump([K, K]) still ends @\x06, loading to [true, true] on .equal? the link must point at the start of the wrapped form, not at the 'c' inside it

specdelta, provenanced, over core/marshal:

control   rbgo-rmaxC  — built from fdf0939840e8
treatment rbgo-mshlT  — built from 6a7944b6bc40
MOVE none declared: both engines read the same corpus, so every delta is the ENGINE

core/marshal/dump_spec.rb    191   194   +3
core/marshal/load_spec.rb    230   230   +0      <- the reader side did not move
6 file(s): control 442 passing, treatment 445 passing, +3
1 file(s) moved, 0 regressed

Coverage: writeClassRef and classPathEncIvar at 100.0%.

Independent of #892 — different file, different cause, same issue.

🤖 Generated with Claude Code

…dumps it

Three more of the 27 examples that went red on the move to Ruby 4.0 (#799),
again one cause. `core/marshal/dump_spec.rb` goes 191 → 194, fail 18 → 15.

## The reference

marshal.c at v4.0.9 brackets the class path with the same encoding machinery a
String goes through:

    VALUE encname = w_encivar(path, arg);   /* emits 'I' when path is non-ASCII */
    w_byte(TYPE_CLASS, arg);
    w_bytes(RSTRING_PTR(path), RSTRING_LEN(path), arg);
    w_encname(encname, arg);                /* 1 ivar: :E => true               */

`w_encivar` returns nil when the path is ASCII-only, which is what keeps every
ordinary class name dumping exactly as before:

    static VALUE
    w_encivar(VALUE str, struct dump_arg *arg)
    {
        VALUE encname = encoding_name(str, arg);
        if (NIL_P(encname) || is_ascii_string(str)) return Qnil;
        w_byte(TYPE_IVAR, arg);
        return encname;
    }

`encoding_name` maps UTF-8 to Qtrue, so the trailer is the short `:E => true`
form and never the long `:encoding => "..."` one — a constant name in rbgo is a
Go string and therefore UTF-8, and there is no way to name a constant in
another encoding.

## Measured, not read

rubydiff against MRI 4.0.7 on this host: AGREE, byte for byte, on all eleven
probes. The three dumps the specs assert:

    "\x04\bIc,MarshalSpec::Multibyte\xE3\x81\x81...Class\x06:\x06ET"
    "\x04\bIm-MarshalSpec::Multibyte\xE3\x81\x91...Module\x06:\x06ET"
    "\x04\bIc+MarshalSpec::Multibyte\xE3\x81\x81...Time\x06:\x06ET"

## The controls matter more than the three rows

* An ASCII name must still dump unwrapped — `Marshal.dump(String)` is
  `"\x04\bc\vString"`. A wrapper emitted unconditionally would pass all three
  spec rows and break every existing dump.
* The reader had never been handed an 'I' in front of a 'c'. All three
  round-trip to the identical object, which the specs do NOT assert — their
  second line has no `.should`.
* Object links still resolve to the start of the wrapped form:
  `Marshal.dump([K, K])` ends `@\x06`, and loading it gives `[true, true]` on
  `.equal?`.

specdelta, provenanced, over core/marshal: control 442, treatment 445, **+3, 1
file moved, 0 regressed**, with `load_spec.rb` unchanged at 230.

Coverage: writeClassRef and classPathEncIvar at 100.0%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 6e86fac into main Nov 5, 2026
26 checks passed
@tannevaled
tannevaled deleted the marshal-multibyte-classpath branch November 5, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant