Starting a connection hangs indefinitely sometimes

We've been seeing some of our background jobs hang on jobs that use net-http. Here's 2 backtraces:

This is from using S3's put_object call:

/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect'
/usr/local/bundle/ruby/3.2.0/gems/timeout-0.4.0/lib/timeout.rb:186:in `block in timeout'
/usr/local/bundle/ruby/3.2.0/gems/timeout-0.4.0/lib/timeout.rb:193:in `timeout'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1243:in `start'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:87:in `method_missing'
/usr/local/bundle/ruby/3.2.0/gems/aws-sdk-core-3.181.0/lib/seahorse/client/net_http/connection_pool.rb:307:in `start_session'
/usr/local/bundle/ruby/3.2.0/gems/aws-sdk-core-3.181.0/lib/seahorse/client/net_http/connection_pool.rb:100:in `session_for'
/usr/local/bundle/ruby/3.2.0/gems/aws-sdk-core-3.181.0/lib/seahorse/client/net_http/handler.rb:128:in `session'
/usr/local/bundle/ruby/3.2.0/gems/aws-sdk-core-3.181.0/lib/seahorse/client/net_http/handler.rb:76:in `transmit'
/usr/local/bundle/ruby/3.2.0/gems/aws-sdk-core-3.181.0/lib/seahorse/client/net_http/handler.rb:50:in `call'
/usr/local/bundle/ruby/3.2.0/gems/aws-sdk-core-3.181.0/lib/seahorse/client/plugins/content_length.rb:24:in `call'
/usr/local/bundle/ruby/3.2.0/gems/aws-sdk-core-3.181.0/lib/seahorse/client/plugins/request_callback.rb:118:in `call'
...  [ aws code ] ...
/usr/local/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.133.0/lib/aws-sdk-s3/bucket.rb:682:in `put_object'
... [ our code ] ...

(full backtrace here)

This is from using Faraday:

/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect'
/usr/local/bundle/ruby/3.2.0/gems/timeout-0.4.0/lib/timeout.rb:186:in `block in timeout'
/usr/local/bundle/ruby/3.2.0/gems/timeout-0.4.0/lib/timeout.rb:193:in `timeout'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start'
/usr/lib/fullstaq-ruby/versions/3.2.2/lib/ruby/3.2.0/net/http.rb:1237:in `start'
/usr/local/bundle/ruby/3.2.0/gems/faraday-net_http-1.0.1/lib/faraday/adapter/net_http.rb:146:in `request_via_request_method'
/usr/local/bundle/ruby/3.2.0/gems/faraday-net_http-1.0.1/lib/faraday/adapter/net_http.rb:131:in `request_with_wrapped_block'
/usr/local/bundle/ruby/3.2.0/gems/faraday-net_http-1.0.1/lib/faraday/adapter/net_http.rb:122:in `perform_request'
/usr/local/bundle/ruby/3.2.0/gems/faraday-net_http-1.0.1/lib/faraday/adapter/net_http.rb:66:in `block in call'
... [ truncated ] ...

It looks to be hanging in Ruby's net/http's TCPSocket.open call.

Some things to note, in case it matters:

We're noticing this happen in background jobs. We're using Sidekiq.
We use the latest version of Ruby's timeout gem (0.4.0), which has this notable change: Raise exception instead of throw/catch for timeouts timeout#30

Downgrading to timeout 0.3.2 did not help.

Went looking for anything around timeouts, and found #6 ... not sure if that is relevant though.

Hello,
Recently, we tried migrating to Ruby 3.4 and faced a similar issue.
Unfortunately, I could not reproduce the issue on some of the public domains, only on one of our production servers.
Here is what we do (this code runs in AWS Lambda, and it works as expected using Ruby 3.2 Lambda runtime):

def init
  @request_uri = URI('https://some-url')
  # we are reusing the same session for all the requests
  @http = Net::HTTP.start(@request_uri.host, @request_uri.port, :use_ssl => @request_uri.scheme == 'https', :read_timeout => 1, :open_timeout => 1)
end

def do_request(endpoint:, query_params: nil)
  url = URI("#{@request_uri}#{endpoint}")
  url.query = URI.encode_www_form(query_params) if query_params

  req = Net::HTTP::Get.new(url)
  req['User-Agent']    = 'test'
  req['Authorization'] = 'Bearer some-token'

  response = @http.request(req) or raise 'no response'

  unless response.code == '200'
    raise "Error getting #{endpoint}: unsuccessful request"
  end

  puts("#{req.uri}: #{response.code} - #{response.body}")

  JSON.parse(response.body)
end

The code above hangs forever on the second request (ignoring any specified timeout options) even though it is implemented similarly to what the documentation suggests https://docs.ruby-lang.org/en/3.4/Net/HTTP.html#class-Net::HTTPSession-label-Queries.
Interrupting the waiting gives the following stacktrace:

/usr/share/rvm/rubies/ruby-3.4.8/lib/ruby/3.4.0/openssl/buffering.rb:80:in 'OpenSSL::SSL::SSLSocket#sysread': Interrupt
        from /usr/share/rvm/rubies/ruby-3.4.8/lib/ruby/3.4.0/openssl/buffering.rb:80:in 'OpenSSL::Buffering#fill_rbuff'
        from /usr/share/rvm/rubies/ruby-3.4.8/lib/ruby/3.4.0/openssl/buffering.rb:339:in 'OpenSSL::Buffering#eof?'
        from /usr/share/rvm/rubies/ruby-3.4.8/lib/ruby/3.4.0/net/protocol.rb:138:in 'Net::BufferedIO#eof?'
        from /usr/share/rvm/rubies/ruby-3.4.8/lib/ruby/3.4.0/net/http.rb:2464:in 'Net::HTTP#begin_transport'
        from /usr/share/rvm/rubies/ruby-3.4.8/lib/ruby/3.4.0/net/http.rb:2404:in 'Net::HTTP#transport_request'
        from /usr/share/rvm/rubies/ruby-3.4.8/lib/ruby/3.4.0/net/http.rb:2378:in 'Net::HTTP#request'

However, if I change the order of request creation and query setting like the following - everything works (but no query parameters applied):

  req = Net::HTTP::Get.new(url)
  url.query = URI.encode_www_form(query_params) if query_params

Does anyone have any ideas what goes wrong?

#233 looks related

Navigation Menu

Repository navigation

Starting a connection hangs indefinitely sometimes #149

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Starting a connection hangs indefinitely sometimes #149

Description

Activity

ngan commented on Aug 28, 2023

technicalpickles commented on Sep 12, 2023

tory-kk commented on Mar 6, 2026

tory-kk commented on Mar 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions