Repository navigation
Requests with ExecuteJavascriptMiddleware are sent twice #239
Description
Describe the bug
When using the ExecuteJavascriptMiddleware, two requests are sent. One by Browsershot, the other by Guzzle.
Reproduction
My spider only has the ExecuteJavascriptMiddleware registered as downloader middleware.
I placed dumps with the request object id in the following places, just before requests are sent:
RoachPHP\Http\Client
RoachPHP\Downloader\Middleware\ExecuteJavascriptMiddleware
When the spider runs, both dumps are shown:
Expected behavior
I was expecting only one request being sent: one from Browsershot, not also one from Guzzle.
Package versions (please complete the following information):
- core: v3.0.1
Activity
fadarrizz commentedon Apr 9, 2024
Since the
ExecuteJavascriptMiddlewarehandles a response, it's called once a response is received. So, the http client has already sent the request before it reaches the middleware.My feeling is that the middleware should be able to handle the request before it is handled by something else.
Looking at how Scrapy does this, a downloader middleware can process requests and can return a response. When a response is returned by the middleware, no other request processing is done, only response processing.
https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#scrapy.downloadermiddlewares.DownloaderMiddleware.process_request
@ksassnowski What do you think is a suitable solution for this?