Encoding error with bad raise message in StringIO #8710

Activity

1. headius commented on Mar 24, 2025

   

   headius

   on Mar 24, 2025

   MemberAuthor

   More actions

   Seems related to this change: ruby/stringio@f75f90c

   The encoding check here doesn't fail but it ends up writing MBC into the StringIO and switching it to US-ASCII. That causes a subsequent write to blow up, because the bytes aren't valid anymore.
2. headius commented on Mar 24, 2025

   

   headius

   on Mar 24, 2025

   Last edited by headius

   MemberAuthor

   More actions

   Reduced:

   begin
     raise
   rescue => ex
   end
   
   require 'stringio'
   err = StringIO.new
   err.print("El Niño is coming!".dup.force_encoding("US-ASCII"))
   err.print(ex.backtrace[0])
3. headius commented on Mar 24, 2025

   

   headius

   on Mar 24, 2025

   MemberAuthor

   More actions

   This is not directly a fix but it eliminates the internal Java RuntimeException:

   diff --git a/ext/java/org/jruby/ext/stringio/StringIO.java b/ext/java/org/jruby/ext/stringio/StringIO.java
   index 2eb0588..83bd60f 100644
   --- a/ext/java/org/jruby/ext/stringio/StringIO.java
   +++ b/ext/java/org/jruby/ext/stringio/StringIO.java
   @@ -1662,7 +1662,7 @@ public class StringIO extends RubyObject implements EncodingCapable, DataType {
            try {
                RubyString unused = (RubyString) CAT_WITH_CODE_RANGE.invokeExact(myString, str);
            } catch (Throwable t) {
   -            throw new RuntimeException(t);
   +            Helpers.throwException(t);
            }
        }

   There's some very sensitive combination of encoding, code range, and valid or invalid content at play here. If CRuby runs the repro above and calls the final print with a MBC string, it raises the same error. It seems that when the string is 7-bit ASCII it avoids the encoding incompatibility... and possibly that is a bug in CRuby.
4. headius commented on Mar 24, 2025

   

   headius

   on Mar 24, 2025

   MemberAuthor

   More actions

   Ok, I'm pretty sure this is just a lucky combination of things that caused us to fall into a more strict encoding path.

   This is as reduced as I can get the case, using JRuby internal APIs to force the code range to VALID:

   repro_without_exception_jruby_valid.rb:

   require 'jruby'
   require 'stringio'
   
   err = StringIO.new
   err.print("ñ".dup.force_encoding("US-ASCII"))
   str = "x"
   JRuby.ref(str).setCodeRange(32)
   err.print(str)

   When the second print string is VALID, it causes additional encoding checks to fire in the rb_enc_str_buf_cat logic used by StringIO#write. If the string is not VALID (UNKNOWN or 7BIT) it does not follow that path and blindly adds the additional string content to the target.

   In CRuby, I do not know how to force a non-MBC string to become VALID, but if we use a UTF-8 MBC string for the second print (which should be VALID), CRuby errors the same way:

   repro_without_exception_with_mbc.rb:

   require 'stringio'
   
   err = StringIO.new
   err.print("ñ".dup.force_encoding("US-ASCII"))
   err.print("å")

   Output:

   $ cx 3.4 ruby blah.rb
   blah.rb:5:in 'StringIO#write': incompatible character encodings: US-ASCII and UTF-8 (Encoding::CompatibilityError)
   	from blah.rb:5:in 'IO::generic_writable#print'
   	from blah.rb:5:in '<main>'
   

   I can also trigger it on CRuby if there are any MBC components in the stack trace, such as by having MBC in the filename:

   $ cat reprø_without_exception_mbc_filename.rb
   require 'stringio'
   
   begin
     raise
   rescue => ex
   end
   
   err = StringIO.new
   err.print("ñ".dup.force_encoding("US-ASCII"))
   err.print(ex.backtrace.join)
   $ cx 3.4 ruby reprø_without_exception_mbc_filename.rb
   blåh.rb:10:in 'StringIO#write': incompatible character encodings: US-ASCII and UTF-8 (Encoding::CompatibilityError)
   	from blåh.rb:10:in 'IO::generic_writable#print'
   	from blåh.rb:10:in '<main>'
   

   The "with exception" example from previous comments fails because the backtrace elements are VALID. This is because when we construct the Ruby backtrace, we turn Java exception stack trace strings into Ruby strings as VALID UTF-8, which then follows the VALID path (even if they might be 7-bit). The resulting string triggers the encoding checks as in the "without exception" case above.

   I can "fix" this by converting Java strings into Ruby strings as either UNKNOWN or by re-scanning for code range (which makes the backtrace elements 7BIT in this case. However I think this test case in Rake probably just managed to get lucky with the right combination of encodings, bytes, and code ranges.

   @hsbt @nobu I would like to discuss these findings with you when you have an opportunity. I will fix JRuby and push a new jruby-head build that resolves the failure in rake, but this is a very fragile test case that perhaps is not testing what it should be testing.
5. headius

   added a commit that references this issue on Mar 24, 2025

   Less eager coderange when converting Java String

   ...

   22d8720
6. headius

   added this to the JRuby 10.0.0.0 milestone on Mar 24, 2025
7. headius commented on Mar 24, 2025

   

   headius

   on Mar 24, 2025

   MemberAuthor

   More actions

   Fixed by #8711
8. headius

   closed this as completedon Mar 24, 2025
9. headius

   linked a pull request that will close this issueLess eager coderange when converting Java String #8711on Mar 24, 2025