Skip to content

Follow redirects whose Location holds non-ASCII bytes - #95

Open
ihamzatarar wants to merge 1 commit into
ruby:masterfrom
ihamzatarar:follow-non-ascii-redirect-location
Open

ihamzatarar wants to merge 1 commit into
ruby:masterfrom
ihamzatarar:follow-non-ascii-redirect-location

Conversation

@ihamzatarar

Copy link
Copy Markdown

Fixes #46.

A Location field holding raw non-ASCII bytes cannot be parsed by URI.parse, so the redirect is abandoned:

URI.open("https://old.reddit.com/comments/1i8cnq3")
# OpenURI::HTTPError: 301 Moved Permanently (Invalid Location URI)
# caused by URI::InvalidURIError: URI must be ascii only

Servers that do this are not following RFC 9110, which requires the field to be a URI reference, but they are common, and browsers percent encode the bytes and follow the redirect rather than fail.

This does the same: if URI.parse rejects the field, the non-ASCII bytes are percent encoded and parsed once more. The first attempt is unchanged, so a Location that parses today takes exactly the path it takes now, and one that is malformed for any other reason still raises. test_redirect_invalid, which sends Location: ::, still passes.

Tests

test_redirect_non_ascii_location in test/open-uri/test_open-uri.rb, modelled on test_redirect_relative: the server answers 301 with Location: /caf\xC3\xA9, and the test asserts the follow-up request is GET /caf%C3%A9.

On master it fails with:

URI::InvalidURIError: URI must be ascii only "/caf\xC3\xA9"

Suite on Ruby 3.4.6, run standalone:

master   40 tests, 146 assertions, 0 failures, 30 errors
patched  41 tests, 149 assertions, 0 failures, 30 errors

The 30 errors are the same before and after: they need assert_join_threads from ruby/ruby's test helpers, which is not available when the suite runs outside the Ruby tree.

URI.parse rejects a Location field containing raw non-ASCII bytes, so
open-uri gave up on the redirect even though browsers percent encode
those bytes and follow it. Servers that send them are not following
RFC 9110, but they are common enough to matter.

Escape the non-ASCII bytes and parse once more when the first attempt
fails. A Location that parses today is unaffected, and one that is
malformed for any other reason still raises.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

301 redirects do not escape non-ASCII characters in the redirect URLs

1 participant