Skip to content

Robots.txt Edge case (doesn't exist?) #432

Open

Description

@R-Sandor

When I tried scrape https://pnpm.io/benchmarks

The backend produced this error message. I attempted to visit where the site thought the robots.txt should live:
"https://pnpm.io/robots.txt and it is a 404.

server-1      | 2026-09-27T09:28:21.433Z ERROR 1 --- [mcat-handler-38] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://pnpm.io/robots.txt": pnpm.io] with root cause
server-1      |
server-1      | java.net.UnknownHostException: pnpm.io
server-1      |         at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:574) ~[na:na]
server-1      |         at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:284) ~[na:na]
server-1      |         at java.base/java.net.Socket.connect(Socket.java:668) ~[na:na]
server-1      |         at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[na:na]
server-1      |         at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:181) ~[na:na]
server-1      |         at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:166) ~[na:na]
server-1      |         at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:516) ~[na:na]
server-1      |         at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:604) ~[na:na]
server-1      |         at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:206) ~[na:na]
server-1      |         at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:288) ~[na:na]
server-1      |         at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[na:na]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Activity

  1. Abhishek84313 commented on Oct 5, 2026

    @Abhishek84313
    Contributor

    @R-Sandor can i work on this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions