robots.txt Checker
A single wrong line in robots.txt can make entire sections of your website disappear from Google. Our checker reads the file the way Googlebot does and shows you within seconds where the problems are.
Checking
Tip: enter a subpage (e.g. your-domain.com/shop/) to also check whether exactly this page may be crawled.
How to read the result
Each check gets a traffic light colour. Green is fine, yellow is worth a look, red is probably costing you visibility. Points marked blue are for information only.
The traffic light
- OK
This point is implemented correctly.
- Review
Not an acute error, but room for improvement.
- Problem
Likely has a negative effect and should be fixed.
- Info
For information, without rating.
- 01
Reachability
The file must be located at your-domain.com/robots.txt and respond with status code 200. If it is missing (404), crawlers may access everything. That is not an error, but you give up control. A server error (5xx) is critical: Google then pauses crawling of the entire website.
- 02
Syntax
We check every line. Crawlers silently skip lines without a colon, unknown directives and rules placed before the first User-agent. That is what makes these errors tricky: the restriction you meant to set simply does not exist for Google.
- 03
Googlebot and Bingbot
We evaluate the rules according to the official standard RFC 9309, which Google follows as well. The group matching the crawler name applies, otherwise the group for all (*). Within a group, the longest matching rule wins; in a tie, Allow wins.
- 04
CSS, JavaScript and images
Google renders pages like a browser and needs access to style and script files to do so. If they are blocked, Google sees an incomplete page, which can affect how your mobile layout is assessed. We test the files your homepage actually loads.
- 05
Sitemap
A sitemap entry in robots.txt is the easiest way to tell every search engine about your site structure. We check whether the referenced sitemaps can actually be retrieved and contain valid XML.
- 06
Blocked areas
Finally, you see all Disallow rules in one overview. Go through the list and ask yourself for each entry: should this area really stay out of search engines?
Why robots.txt has so much influence
robots.txt is the first file a search engine crawler requests on your website. It defines which areas the crawler may visit. That is useful for keeping admin areas, internal search results or shopping cart pages out of crawling and for focusing the search engine on the pages that really matter.
At the same time, it is a common cause of sudden ranking losses. A relaunch is the classic case: during development, "Disallow: /" blocks the staging site, and at go-live the line is forgotten. The new website is then invisible to Google, often for weeks, until someone notices the drop.
Good to know: robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other websites link to it, just without a description. If you want to reliably keep a page out of Google, you need a noindex tag. For Google to read it, the page must not be blocked in robots.txt.
Also: robots.txt is not access protection. Anyone can read it, and it is a request to well-behaved crawlers, not a technical barrier. Confidential content belongs behind a password.
A solid starting point for WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://your-domain.com/sitemap.xmlFor a typical WordPress website this is a good basis. Adjust the paths to your system and then verify the result with the checker.
Frequently asked questions about robots.txt
Where does robots.txt have to be located?
How quickly does Google pick up changes?
Is Disallow enough to remove a page from Google?
What do the asterisk (*) and dollar sign ($) mean?
Does capitalisation matter?
What does Crawl-delay do?
Is my input stored?
Not sure your settings are right?
We review your website’s robots.txt, sitemap and indexing together with you and tell you exactly what to do. As a web agency in Vienna, we look after websites technically and editorially, from the first concept to ongoing operation.
Related tools
- AI AI Visibility Check Shows whether ChatGPT, Claude, Perplexity and Google AI Overviews may fetch and cite your website.
- SEO SEO Check Checks the most important on-page factors of a page, from title and description to headings and alt texts.
- 301 Redirect & Relaunch Check Checks up to 50 old addresses: do visitors and Google end up on the right page after the move?
















