SEO Toolbox

Robots.txt Tester

Read a site robots.txt and test whether a given URL is crawlable.

Also in Server & Headers

HTTP Headers CheckerEvery response header, with the security ones graded.Needs a page URLHTTP Status Code CheckerCheck many URLs at once and see status, redirects and timing.Needs a list of URLsRedirect CheckerTrace the full redirect chain hop by hop, and spot loops.Needs a page URLXML Sitemap CheckerFind a sitemap, count its URLs and spot-check that they resolve.Needs a domain name
Robots.txt Tester — SEO Toolbox