You cant run multiple spiders #249

Activity

1. blackhood5678

   added

   bugSomething isn't workingSomething isn't working

   on May 24, 2024
2. blackhood5678

   assigned

   ksassnowski

   on May 24, 2024
3. claytongray commented on Jun 27, 2024

   

   claytongray

   on Jun 27, 2024

   Last edited by claytongray

   More actions

   I have the same issue. I'm running 1 spider inside of a foreach loop. Because of that, I see multiple, duplicate requests being made. So by the 10th loop, I have 10 spiders running. Those 10 spiders seem to have the corresponding index amount of startUrls. So on the 10th loop, I have 10 spiders, spider 1 of 10 requests the link once, spider 2 of 10 requests the link twice, the third spider requests the link 3 times, etc etc. Even the RequestDeduplicationMiddleware doesn't seem to do anything.

   I noticed if I start 2 different Spider classes, even with 2 separate set of URLs, multiple requests are made. So it seems every time a Roach::startSpider() is called, a new spider is created, but will listen to any overrides, such as startUrls.
4. joelmellon commented on Sep 1, 2024

   

   joelmellon

   on Sep 1, 2024

   Last edited by joelmellon

   More actions

   Duplicate/related: #36

   Work around is to run spider jobs separately via Laravel queues or another way of "forking" into different PHP processes.
5. mattheobjornson commented on Jan 21, 2025

   

   mattheobjornson

   on Jan 21, 2025

   More actions

   Quick&dirty workaround/hack I've used was:

   1. patch Roach.php (add function to Roach class):

      public static function killSpider(): void
      {
          self::$container = null;
      }

   2. Then after each spider run, call it:

      Roach::startSpider(FirstSpider::class);
      Roach::killSpider();
      Roach::startSpider(SecondSpider::class);
      Roach::killSpider();