What is robots.txt?
robots.txt is a small file at the root of your website (e.g. example.com/robots.txt) that tells search-engine crawlers which pages they may or may not access. A single wrong Disallow line can hide important pages from Google.
How to use this robots.txt tester
Paste your robots.txt content, type the path you want to check (for example /blog/my-post/), pick a crawler (or leave it as * for all), and click Test. You will instantly see whether that path is Allowed or Blocked, and which rule decided it.
Common robots.txt mistakes
Blocking your whole site with Disallow: /, blocking CSS/JS that Google needs to render the page, or forgetting to add your sitemap. Talk to our SEO team for a free technical audit.