ApacheHttp2Transport: send() hangs forever when an Error kills the I/O dispatcher thread (Error is never surfaced) #1253

[REQUIRED] Step 2: Describe your environment

• Operating System version: Linux (Ubuntu 24.04), OpenJDK 21
• Firebase SDK version: reproduced on 9.11.0 (latest) and 9.5.0
• Library version: httpclient5 5.6.4 / httpcore5 5.4.3 (9.11.0 defaults); 9.5.0 with httpcore5 + httpcore5-h2 5.3.6
• Firebase Product: messaging (affects every API using the default ApacheHttp2Transport)

[REQUIRED] Step 3: Describe the problem

Since 9.4.0 (#979), the default transport is ApacheHttp2Transport, which runs network I/O on httpclient5's httpclient-dispatch-N threads. If an Error (not an Exception) is thrown on one of those threads, for example a LinkageError from a mismatched transitive dependency, an OutOfMemoryError, or a StackOverflowError:

1. httpcore5's IOReactorWorker rethrows it and the dispatcher thread dies. The error is only printed to stderr by the JVM's default uncaught-exception handler. The SDK never observes it. The FutureCallback passed in ApacheHttp2Request.execute() is empty, and it isn't invoked in this case anyway.
2. The in-flight request's future is never completed or failed.
3. ApacheHttp2Request.execute() calls responseFuture.get() with no bound when readTimeout == 0. That is the FirebaseOptions default: TimeoutInitializer always applies the options' timeouts, which overrides google-http-client's 20s defaults. The connect and response timeouts passed to httpclient5 are 0 (disabled), and they would be enforced by the dead dispatcher thread anyway.

Result: FirebaseMessaging.send() blocks the calling thread forever, with no exception and no SDK log.

With a non-zero read timeout (9.11.0 only, via #1239), the call fails after the timeout with Unknown error while making a remote service call: Stream exception in request, caused by TimeoutValueException. The real error isn't in the cause chain, so there is still no way to tell from the SDK what went wrong. Before the HTTP/2 transport, the same Error would have been thrown synchronously on the caller's thread.

Real-world impact. We hit this in production on 9.5.0. Our build resolved httpcore5/httpcore5-h2 5.3.6 next to httpclient5 (firebase-admin only declares httpclient5). httpcore5-h2 5.3.4 changed the ClientH2UpgradeHandler/ClientH2PrefaceHandler constructors that httpclient5 5.3.x/5.4.x call (apache/httpcomponents-core@048bcc1). Each new connection threw NoSuchMethodError on the dispatcher thread. The version skew was on our side, and Apache marks those classes @Internal. But every thread calling send() hung indefinitely, and push delivery silently stopped for about 48 hours with no exception, metric, or SDK log to point at the cause.

Steps to reproduce:

Repro.java uses only the public API plus the SDK's own ApacheHttp2Transport.defaultHttpAsyncClientBuilder(). It sends with a fake token, so a healthy client fails fast with a 401.

cs fetch --classpath com.google.firebase:firebase-admin:9.11.0 > cp.txt
java -cp "$(cat cp.txt)" Repro.java                      # control: 401 in ~330ms
java -cp "$(cat cp.txt)" Repro.java inject-error         # hangs (still blocked after 60s)
java -cp "$(cat cp.txt)" Repro.java inject-error 5000    # "Stream exception in request" after 5s, real Error not in cause chain

# Natural trigger on 9.5.0 (no injection): hangs exactly the same way
cs fetch --classpath com.google.firebase:firebase-admin:9.5.0 \
  --force-version org.apache.httpcomponents.core5:httpcore5:5.3.6 \
  --force-version org.apache.httpcomponents.core5:httpcore5-h2:5.3.6 > cp-950.txt
java -cp "$(cat cp-950.txt)" Repro.java

Output with inject-error on 9.11.0:

Exception in thread "httpclient-dispatch-1" java.lang.LinkageError: simulated Error on the I/O dispatch thread
	at Repro$ThrowingListener.connected(Repro.java:70)
	at org.apache.hc.core5.reactor.InternalDataChannel.onTLSSessionStart(InternalDataChannel.java:184)
	...
	at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
RESULT: send() still blocked after 60s. fcm-caller stack:
    at org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:97)
    at com.google.firebase.internal.ApacheHttp2Request.execute(ApacheHttp2Request.java:134)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
    at com.google.firebase.internal.ErrorHandlingHttpClient.send(ErrorHandlingHttpClient.java:97)
    ...
    at com.google.firebase.messaging.FirebaseMessaging.send(FirebaseMessaging.java:91)

On 9.5.0 with httpcore5 5.3.6, the dispatcher dies with java.lang.NoSuchMethodError: 'void org.apache.hc.core5.http2.impl.nio.ClientH2UpgradeHandler.<init>(org.apache.hc.core5.http2.impl.nio.ClientH2StreamMultiplexerFactory)' at HttpAsyncClientProtocolNegotiationStarter.createHandler. send() blocks at ApacheHttp2Request.java:123.

Relevant Code:

public class Repro {
  public static void main(String[] args) throws Exception {
    FirebaseOptions.Builder options = FirebaseOptions.builder()
        .setProjectId("repro-project")
        .setCredentials(new FakeCredentials());
    if (args.length > 1) {
      options.setReadTimeout(Integer.parseInt(args[1]));
    }
    if (args.length > 0 && args[0].equals("inject-error")) {
      options.setHttpTransport(new ApacheHttp2Transport(
          ApacheHttp2Transport.defaultHttpAsyncClientBuilder()
              .setIOSessionListener(new ThrowingListener())
              .build()));
    }
    FirebaseApp app = FirebaseApp.initializeApp(options.build());

    Future<String> result = Executors.newSingleThreadExecutor().submit(() ->
        FirebaseMessaging.getInstance(app).send(Message.builder().setToken("fake-device-token").build()));
    // Healthy client: FirebaseMessagingException (401) in well under a second.
    // Broken dispatcher with default options: never returns.
    System.out.println(result.get(60, TimeUnit.SECONDS));
  }

  static final class FakeCredentials extends GoogleCredentials {
    @Override
    public AccessToken refreshAccessToken() {
      return new AccessToken("fake-token", new Date(System.currentTimeMillis() + 3_600_000));
    }
  }

  static final class ThrowingListener implements IOSessionListener {
    @Override public void connected(IOSession s) { throw new LinkageError("simulated Error on the I/O dispatch thread"); }
    @Override public void startTls(IOSession s) {}
    @Override public void inputReady(IOSession s) {}
    @Override public void outputReady(IOSession s) {}
    @Override public void timeout(IOSession s) {}
    @Override public void exception(IOSession s, Exception ex) {}
    @Override public void disconnected(IOSession s) {}
  }
}

Suggested fixes

1. Never wait without a bound in ApacheHttp2Request.execute(). Either ship finite default timeouts in FirebaseOptions, or bound the client-side wait even when readTimeout == 0. Today the SDK's liveness depends on a background thread it doesn't control.
2. Surface dispatcher death to callers. For example, build the default client with HttpAsyncClientBuilder.setThreadFactory(...) so dispatcher threads have an UncaughtExceptionHandler. That handler can record the Throwable and fail outstanding and future requests with an IOException whose cause is the Error. Alternatively, check httpAsyncClient.getStatus() while waiting. httpcore5's I/O reactor exception callback only receives Exceptions, so it doesn't cover this case.
3. Log it. The SDK already depends on SLF4J; logging dispatcher death there would make the root cause visible in applications' normal logs.
4. Pin httpcore5 and httpcore5-h2 explicitly in the POM, at the version matching the declared httpclient5. Dependency resolution in consuming builds is then less likely to pair httpclient5 with an incompatible core. Apache doesn't guarantee compatibility for the @Internal classes httpclient5 uses.

Workarounds for other users

• Set FirebaseOptions.Builder#setConnectTimeout and setReadTimeout. On 9.11.0 and later, this turns the hang into a timeout error, though the root cause is still hidden.
• Keep httpcore5/httpcore5-h2 aligned with the httpclient5 version the SDK declares.