gh-79638: Treat an unreachable robots.txt as "disallow all" - #138555

gh-79638: Treat an unreachable robots.txt as "disallow all"#138555

serhiy-storchaka merged 5 commits into

python:mainpython/cpython:mainfrom

serhiy-storchaka:robotparser-unreachableserhiy-storchaka/cpython:robotparser-unreachableCopy head branch name to clipboard