Robots.txt is a text file that provides instructions to search engine crawlers and other automated bots about which areas of a website they are allowed or asked not to crawl.
The file is part of a website's technical setup and is usually placed in the root directory of a domain.
Robots.txt can be useful for managing crawler access, but it is important to understand what it can and cannot do. It should not be treated as a general tool for removing webpages from search results.
Robots.txt provides instructions to automated crawlers using a set of rules.
For example, a website might use robots.txt to tell certain crawlers not to request URLs under a particular directory.
A very simple example is:
User-agent: *
Disallow: /private/
This rule tells crawlers represented by * not to crawl URLs under the /private/ path.
The exact behaviour can depend on the crawler and the rules provided.
Robots.txt is normally located at the root of a domain.
For example:
https://example.com/robots.txt
It is not normally placed inside a subdirectory such as:
https://example.com/blog/robots.txt
For a standard website, the root-level robots.txt file is the one crawlers look for when accessing the domain.
A basic robots.txt file might look like this:
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
In this example, the User-agent identifies the crawler group, while the empty Disallow means no paths are being disallowed by that rule.
The Sitemap line provides the location of the website's XML sitemap.
A User-agent identifies the crawler or group of crawlers to which a robots.txt rule applies.
The wildcard:
User-agent: *
is commonly used when a rule is intended for all crawlers that follow the robots.txt standard.
A website can also provide rules for specific crawlers where appropriate.
The Disallow directive tells a crawler which URL path it should not crawl.
For example:
User-agent: *
Disallow: /admin/
This indicates that the crawler should not crawl URLs under the /admin/ path.
Be careful when using broad Disallow rules because an incorrectly configured rule can prevent crawlers from accessing pages or resources that you actually want them to discover.
The Allow directive can be used to indicate that a particular path may be crawled when other rules would otherwise disallow it.
For example:
User-agent: *
Disallow: /private/
Allow: /private/public-page.html
The exact handling of conflicting rules can vary depending on the crawler, so robots.txt rules should be kept as clear and simple as possible.
Not necessarily.
This is one of the most important things to understand about robots.txt.
Robots.txt primarily controls whether crawlers are instructed to crawl particular URLs. It is not a reliable method for removing a URL from search results.
A URL that is blocked from crawling can potentially still be known to a search engine through links or other signals.
If you need a page to be removed from search results, use the appropriate indexing and removal methods rather than relying on robots.txt alone.
These two concepts are often confused.
Robots.txt is primarily about crawler access.
Noindex is an instruction related to whether a page should be included in a search engine's index.
If a crawler cannot access a page because robots.txt blocks it, it may not be able to see a noindex instruction placed on that page.
Therefore, blocking a URL in robots.txt is not simply another way of adding noindex.
There can be legitimate reasons to control crawler access to certain areas of a website.
For example, you may want to reduce unnecessary crawling of certain resource-heavy or non-public areas.
However, avoid blocking important pages simply because you do not want them to appear in search results.
Before adding a Disallow rule, make sure you understand exactly what URLs it will affect.
WordPress installations commonly contain administrative areas that are not intended to be publicly indexed.
However, WordPress itself and many plugins may already provide appropriate handling for these areas.
Do not blindly copy robots.txt rules from another WordPress website.
Review your actual website configuration and make changes only when there is a clear reason.
Generally, you should be careful about blocking resources that are necessary for search engines to understand or render your webpages.
Modern websites can rely on CSS and JavaScript to display important content and functionality.
Blocking important resources can make it harder for crawlers to properly process a page.
Before disallowing a resource, confirm that it is not required for the page to function or be properly understood.
Robots.txt and XML sitemaps serve different purposes, but they can work together.
An XML sitemap provides search engines with a list of important URLs that you want them to be able to discover.
Robots.txt provides crawler access instructions.
You can include your sitemap location in robots.txt, for example:
Sitemap: https://example.com/sitemap.xml
Learn more in our guide to What Is an XML Sitemap?.
A robots.txt file is a simple text file, but its rules need to be written carefully.
A basic file could be:
User-agent: *
Disallow:
This indicates that there are no paths being disallowed by this rule.
If you want to block a specific directory:
User-agent: *
Disallow: /private/
If you want to provide your sitemap as well:
User-agent: *
Disallow: /private/
Sitemap: https://example.com/sitemap.xml
Keep the file simple and avoid adding rules unless you understand the URLs they affect.
Open your website's robots.txt URL directly in a browser.
For example:
https://example.com/robots.txt
Review the file and check:
For larger websites, use the relevant webmaster tools to investigate crawling and indexing issues as well.
Incorrect robots.txt rules can cause significant SEO problems.
Common mistakes include:
Always test important changes carefully.
A rule such as:
User-agent: *
Disallow: /
instructs compliant crawlers not to crawl the entire website.
This can have serious consequences if used accidentally on a live website.
Before publishing a robots.txt file, check that broad rules are not unintentionally preventing access to important content.
Robots.txt deserves special attention when moving a website from development or staging to production.
A staging website may intentionally block crawlers while it is being developed.
When the website goes live, those blocking rules need to be reviewed.
An accidental production-wide Disallow rule can prevent search engines from crawling important pages.
Robots.txt is part of technical SEO because it influences how crawlers interact with your website.
However, a correct robots.txt file does not guarantee rankings or indexing.
It should be considered alongside:
Technical SEO works best when these different elements are configured consistently.
Before considering your robots.txt configuration complete, check that:
Robots.txt is a text file that provides crawler instructions about which areas of a website should or should not be crawled.
It is normally located at the root of a domain, such as https://example.com/robots.txt.
No. Robots.txt primarily controls crawler access and should not be relied upon as a page removal method.
Disallow tells a crawler not to crawl a specified path when the crawler follows the relevant robots.txt rules.
User-agent identifies the crawler or group of crawlers to which a robots.txt rule applies.
Only when there is a specific and intentional reason, such as a private or temporary environment. Blocking the entire site on a live website can prevent search engines from crawling important content.
A correctly configured robots.txt file can help manage crawler access, but it is not a direct ranking factor that guarantees better search performance.
Including the sitemap URL in robots.txt is a useful way to make the sitemap location discoverable to crawlers.
It can contain rules affecting image URLs, but blocking resources should be done carefully because images may be important to your website and search visibility.
They serve different purposes. Robots.txt controls crawler access, while noindex is used to indicate that a page should not be included in a search engine's index. Choose the appropriate method based on what you are trying to achieve.
Robots.txt is one part of a larger technical SEO setup.
Learn more about What Is an XML Sitemap? or read our How to Improve Website Speed guide.
You can also explore How to do Technical SEO or browse the complete SEO Guides & Resources collection.
Use our free SEO tools to investigate your website and identify technical SEO issues that may need attention.
Explore Free SEO Tools →