Repository navigation
Unable to mark scans as failed #54
Description
Maybe I am missing the documentation but when you send the spider out to say https://1013polebeauty.com it would be nice for me to be able to mark this domain as invalid so we do not crawl it again.
I looked into the downloader middleware and spider middleware however the response already seems to be processed at this point, is there any way that you can look at the response code and if there is an issue $yield or be able to update a Laravel model?
Activity
AkioSarkiz commentedon Sep 12, 2022
I think you need to check the domain before submitting. And cache the result. This way you will not send a request if the domain is not valid