Start a projectEnquiry

robots.txt Checker

A single wrong line in robots.txt can make entire sections of your website disappear from Google. Our checker reads the file the way Googlebot does and shows you within seconds where the problems are.

Tip: enter a subpage (e.g. your-domain.com/shop/) to also check whether exactly this page may be crawled.

How to read the result

Each check gets a traffic light colour. Green is fine, yellow is worth a look, red is probably costing you visibility. Points marked blue are for information only.

The traffic light

  • OK

    This point is implemented correctly.

  • Review

    Not an acute error, but room for improvement.

  • Problem

    Likely has a negative effect and should be fixed.

  • Info

    For information, without rating.

  1. 01

    Reachability

    The file must be located at your-domain.com/robots.txt and respond with status code 200. If it is missing (404), crawlers may access everything. That is not an error, but you give up control. A server error (5xx) is critical: Google then pauses crawling of the entire website.

  2. 02

    Syntax

    We check every line. Crawlers silently skip lines without a colon, unknown directives and rules placed before the first User-agent. That is what makes these errors tricky: the restriction you meant to set simply does not exist for Google.

  3. 03

    Googlebot and Bingbot

    We evaluate the rules according to the official standard RFC 9309, which Google follows as well. The group matching the crawler name applies, otherwise the group for all (*). Within a group, the longest matching rule wins; in a tie, Allow wins.

  4. 04

    CSS, JavaScript and images

    Google renders pages like a browser and needs access to style and script files to do so. If they are blocked, Google sees an incomplete page, which can affect how your mobile layout is assessed. We test the files your homepage actually loads.

  5. 05

    Sitemap

    A sitemap entry in robots.txt is the easiest way to tell every search engine about your site structure. We check whether the referenced sitemaps can actually be retrieved and contain valid XML.

  6. 06

    Blocked areas

    Finally, you see all Disallow rules in one overview. Go through the list and ask yourself for each entry: should this area really stay out of search engines?

Why robots.txt has so much influence

robots.txt is the first file a search engine crawler requests on your website. It defines which areas the crawler may visit. That is useful for keeping admin areas, internal search results or shopping cart pages out of crawling and for focusing the search engine on the pages that really matter.

At the same time, it is a common cause of sudden ranking losses. A relaunch is the classic case: during development, "Disallow: /" blocks the staging site, and at go-live the line is forgotten. The new website is then invisible to Google, often for weeks, until someone notices the drop.

Good to know: robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other websites link to it, just without a description. If you want to reliably keep a page out of Google, you need a noindex tag. For Google to read it, the page must not be blocked in robots.txt.

Also: robots.txt is not access protection. Anyone can read it, and it is a request to well-behaved crawlers, not a technical barrier. Confidential content belongs behind a password.

A solid starting point for WordPress

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://your-domain.com/sitemap.xml

For a typical WordPress website this is a good basis. Adjust the paths to your system and then verify the result with the checker.

Frequently asked questions about robots.txt

Where does robots.txt have to be located?

Always in the root directory of the domain, i.e. at https://your-domain.com/robots.txt. Every subdomain such as shop.your-domain.com needs its own file. Crawlers ignore a robots.txt in a subfolder.

How quickly does Google pick up changes?

Google usually caches robots.txt for up to 24 hours. In Google Search Console, you can request a recrawl in the robots.txt report.

Is Disallow enough to remove a page from Google?

No. Disallow only prevents the crawler from fetching the page. If it is already known or linked, it can remain in the index. Use the noindex meta tag instead and keep the page accessible to crawlers so Google can read the tag.

What do the asterisk (*) and dollar sign ($) mean?

The asterisk stands for any number of characters, the dollar sign for the end of the address. "Disallow: /*.pdf$" therefore blocks all PDF files, but not pages whose address merely contains ".pdf" somewhere in the middle.

Does capitalisation matter?

Not for the directives themselves (User-agent, Disallow, Allow), but it does for paths. "Disallow: /Internal/" does not block "/internal/".

What does Crawl-delay do?

Google ignores it and manages the crawl rate itself. Bing and many other crawlers respect it. Values above 10 seconds slow these search engines down so much that new content is found very late.

Is my input stored?

We store the checked address and the result anonymously for statistical purposes. Your IP address is not stored. More in our privacy policy.

Not sure your settings are right?

We review your website’s robots.txt, sitemap and indexing together with you and tell you exactly what to do. As a web agency in Vienna, we look after websites technically and editorially, from the first concept to ongoing operation.

See all 35 tools

Ready for
Unlimited?

This feature loads the following service. Data (e.g. your IP address) will be transmitted to the provider.

Privacy Policy