---
title: "Robots.txt: A Plain Guide With Examples, AI Crawlers and Fixes"
url: https://lucidbrands.agency/resources/seo/robots-txt
description: "What robots.txt is and how it works, the directives, copyable templates for common sites, how to handle AI crawlers like GPTBot, robots.txt vs noindex, how to fix \"Blocked by robots.txt\" in Search Console and common mistakes."
lang: en
---

Image: Lucid Brands Home (https://lucidbrands.agency/img/logo.svg)

©Lucid Brands 2026

Close article: https://lucidbrands.agency/resources/seo

Image: Website SEO resource article: Robots.txt: A Plain Guide (https://lucidbrands.agency/media/resources/art-robots-txt.jpg)

# Robots.txt: A Plain Guide

Robots.txt is a plain text file at the root of your website (yoursite.com/robots.txt) that tells search engine and AI crawlers which parts of your site they may or may not crawl. It uses simple rules like `User-agent` (which bot) and `Disallow` (which paths to skip). Robots.txt controls crawling, not indexing: a blocked page can still appear in Google if other sites link to it, so use a noindex tag to keep a page out of search results. Most small business sites need only a short robots.txt that allows everything important and points to the sitemap.

## What is robots.txt?

Robots.txt is part of the Robots Exclusion Protocol, a standard for telling automated crawlers how to behave on your site. Content Whale notes the protocol was first proposed in 1994 and became an official IETF standard, RFC 9309, in September 2022.

It is useful for:

- Keeping crawlers out of admin areas, internal search results and endless filter pages
- Saving crawl effort on large sites so important pages get crawled more
- Pointing crawlers to your XML sitemap
- Setting rules for AI crawlers

It is not useful for:

- Hiding private information (the file is public, and bad bots ignore it)
- Removing pages from Google search results (use noindex)
- Security of any kind

## How robots.txt works

1. A crawler arrives at your site and requests yoursite.com/robots.txt.
2. It reads the rules for its own name (user agent), or the general rules for all bots.
3. It skips any paths it is told not to crawl and crawls the rest.

Some technical details from Content Whale's summary of RFC 9309: crawlers must be able to read at least 500 KiB of the file, and they may cache robots.txt for up to 24 hours, so changes are not always picked up instantly.

Well behaved crawlers such as Googlebot and Bingbot follow robots.txt. Scrapers and malicious bots often do not.

## Robots.txt directives explained

| Directive | What it does | Example |
| --- | --- | --- |
| `User-agent` | Names the crawler the rules apply to; `*` means all | `User-agent: *` |
| `Disallow` | Paths the crawler should not crawl | `Disallow: /wp-admin/` |
| `Allow` | Exceptions inside a disallowed path | `Allow: /wp-admin/admin-ajax.php` |
| `Sitemap` | The full URL of your XML sitemap | `Sitemap: https://yoursite.com/sitemap.xml` |
| `Crawl-delay` | Asks bots to wait between requests | Google ignores it, and Content Whale notes Yandex stopped supporting it on February 22, 2018 |

**Pattern rules:**

- `*` matches any sequence of characters: `Disallow: /*?sort=` blocks URLs with a sort parameter.
- `$` marks the end of a URL: `Disallow: /*.pdf$` blocks PDF files.
- Paths are case sensitive: `/Blog/` and `/blog/` are different.
- When rules conflict, Google follows the most specific (longest) matching rule.

Ambiguous rules are more common than you might think. Content Whale cites a study of 7,593 sites that found 282 robots.txt files with ambiguous rules and 18 with conflicting rules.

## Robots.txt examples and templates

These are starting points. Adjust paths to match your site and test before publishing.

**Allow everything (most small business sites):**

```
User-agent: *
Disallow:

Sitemap: https://yoursite.com/sitemap.xml
```

**WordPress site:**

```
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://yoursite.com/sitemap_index.xml
```

**Ecommerce site with filters:**

```
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /*?sort=
Disallow: /*?filter=

Sitemap: https://yoursite.com/sitemap.xml
```

**Staging or development site (block all crawlers):**

```
User-agent: *
Disallow: /
```

Use password protection on staging sites too. Robots.txt alone does not keep a site private, and forgetting to remove `Disallow: /` when a site goes live is one of the most damaging SEO mistakes.

**Platform notes:**

- **WordPress:** creates a virtual robots.txt by default. SEO plugins let you edit it. See WordPress SEO (https://lucidbrands.agency/resources/seo/wordpress-seo-guide).
- **Shopify:** provides a default robots.txt that you can customize through a theme template file.
- **Wix and Squarespace:** generate robots.txt automatically, with limited or setting based editing.

Check your platform's current documentation, since these options change.

## Robots.txt and AI crawlers

AI companies run crawlers that collect content for training models and for answering questions in real time. You can set separate rules for each one by name. Common examples (names as published by each company; check their documentation for current lists):

| User agent | Company | Purpose |
| --- | --- | --- |
| `GPTBot` | OpenAI | Collecting content for model training |
| `OAI-SearchBot` | OpenAI | Search features in ChatGPT |
| `ChatGPT-User` | OpenAI | Fetching pages when a user asks ChatGPT to |
| `Google-Extended` | Google | A control for use in Gemini model training (does not affect Google Search) |
| `ClaudeBot` | Anthropic | Collecting content for model training |
| `PerplexityBot` | Perplexity | Indexing for Perplexity answers |
| `CCBot` | Common Crawl | An open web archive used by many AI projects |

**A common approach for businesses that want to be cited by AI tools but not used for training:**

```
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /wp-admin/

Sitemap: https://yoursite.com/sitemap.xml
```

Think carefully before blocking. For most small businesses, being visible and cited in AI answers is more valuable than keeping public marketing pages out of training data. Blocking search related AI crawlers can reduce how often tools like ChatGPT or Perplexity mention you. Learn more in LLM SEO (https://lucidbrands.agency/resources/ai-search-and-automation/llm-seo) and Google AI Overviews (https://lucidbrands.agency/resources/ai-search-and-automation/google-ai-overviews).

**Is ChatGPT a web crawler?** ChatGPT itself is an AI assistant, but OpenAI runs crawlers (such as GPTBot and OAI-SearchBot) and fetches pages when users ask ChatGPT to browse (ChatGPT-User). Each can be controlled separately in robots.txt.

## Robots.txt vs noindex vs other options

| Goal | Use | Why |
| --- | --- | --- |
| Keep a page out of Google results | `noindex` meta tag or `X-Robots-Tag` header | Robots.txt only stops crawling; the page can still be indexed from links |
| Stop crawlers wasting time on filter or search URLs | Robots.txt `Disallow` | Saves crawl effort |
| Keep content private | Password protection or login | Robots.txt is public and not enforced |
| Remove a page quickly from Google | Search Console Removals tool plus noindex or deletion | Temporary removal while the permanent fix takes effect |
| Point crawlers to important pages | XML sitemap. See XML sitemap (https://lucidbrands.agency/resources/seo/xml-sitemap) | Helps discovery |
| Offer AI tools a summary of your site | llms.txt (an emerging, unofficial proposal) | Not a standard; support varies |

**Important:** do not block a page in robots.txt and add noindex at the same time. If Google cannot crawl the page, it never sees the noindex tag.

## How to create and edit robots.txt

1. **Check what you have:** visit yoursite.com/robots.txt.
2. **Write the file** in a plain text editor (not a word processor). Name it exactly `robots.txt`.
3. **Use UTF-8 encoding** and one directive per line.
4. **Upload it to the root** of your domain. It only works at the root, not in a subfolder.
5. **Each subdomain needs its own** (shop.yoursite.com/robots.txt is separate).
6. **Test it** before and after publishing.

## How to test robots.txt

- **Google Search Console robots.txt report:** shows which robots.txt files Google found, when it last crawled them, and any errors or warnings.
- **URL Inspection tool:** enter a URL to see if Google can crawl it, and why not.
- **Manual check:** open yoursite.com/robots.txt in a browser and read every rule.
- **Third party testers:** many SEO tools include robots.txt checkers.

Robots.txt checks are a standard part of any SEO audit (https://lucidbrands.agency/resources/seo/seo-audit).

## How to fix "Blocked by robots.txt" errors

Search Console shows two common robots.txt messages in the Pages report.

**"Blocked by robots.txt"** (page not indexed because Google cannot crawl it)

1. Open the Pages report, click the issue and review the listed URLs.
2. Decide: should these pages be in Google?
3. **If no:** leave them blocked. This is not an error, just information.
4. **If yes:** find the rule blocking them using URL Inspection, edit robots.txt to remove or narrow the rule, publish, then click Validate fix.

**"Indexed, though blocked by robots.txt"** (Google indexed the URL from links but cannot crawl it)

1. **If the page should be in Google:** remove the blocking rule so Google can crawl it properly.
2. **If it should not be in Google:** remove the robots.txt block, add a noindex tag, and let Google crawl the page so it sees the noindex. Once it drops out of the index, you can block it again if needed.

## Common mistakes

- Leaving `Disallow: /` from a staging site on the live site
- Blocking CSS and JavaScript files Google needs to render pages
- Using robots.txt to hide private information
- Blocking pages and adding noindex at the same time
- Wrong capitalization in paths
- Placing the file in a subfolder instead of the root
- Forgetting the sitemap line
- Blocking AI search crawlers without meaning to
- Writing rules so broad they block important sections (`Disallow: /p` blocks /products and /pricing)

Robots.txt is one small part of technical SEO (https://lucidbrands.agency/resources/seo/technical-seo), but a single wrong line can remove a whole site from search.

## Robots.txt and crawl budget

Crawl budget is the number of pages a search engine is willing and able to crawl on your site in a given period. For most small business sites with a few hundred pages, it is not a concern: Google will crawl everything important. It matters on large sites, such as ecommerce stores with thousands of products and filter combinations.

Robots.txt helps large sites by blocking low value URLs:

- Internal search result pages
- Sort and filter parameter combinations
- Session IDs and tracking parameters
- Cart, checkout and account pages
- Endless calendar or archive pages

Do not use robots.txt to block pages that are important but thin or duplicate. Improve them, consolidate them with redirects, or use canonical tags instead.

## Robots.txt checklist

- The file is at the root: yoursite.com/robots.txt
- It returns a normal page (not an error) when you visit it
- No `Disallow: /` left over from a staging site
- CSS, JavaScript and image files are not blocked
- Admin, cart, internal search and filter URLs are blocked where appropriate
- The sitemap URL is listed
- AI crawler rules reflect a deliberate decision
- No page is both blocked and set to noindex
- Search Console's robots.txt report shows no errors
- Someone reviews the file after every redesign or platform change

## FAQ

## Is robots.txt mandatory?

No. If you do not have one, crawlers assume they can crawl everything. Most sites still benefit from a simple file that points to the sitemap and blocks admin or duplicate areas.

## Is robots.txt good for SEO?

Yes, when used carefully. It helps search engines focus on your important pages and avoid wasting effort on duplicates and filters. Used carelessly, it can block pages you want ranked.

## Is ignoring robots.txt illegal?

Robots.txt is a voluntary standard, not a law in itself. Reputable crawlers follow it. Whether ignoring it creates legal problems depends on other laws, terms of service and the situation, so ask a lawyer if this affects you.

## How do I fix a blocked by robots.txt error?

Check whether the page should be indexed. If it should, find the blocking rule with Search Console's URL Inspection tool, remove or narrow it, then validate the fix. If it should not be indexed, the message is not a problem.

## Make sure search engines see your best pages

Lucid Brands' SEO and Content Management service, from $500 a month, covers technical SEO, including robots.txt, sitemaps, indexing and AI crawler settings, alongside content and local SEO. Get a free website audit (https://lucidbrands.agency/website-audit) to check what Google and AI tools can and cannot see on your site.

Your path to lucidity

## Get Started With a Free Website Audit

Not sure where to begin? Start with a free website audit. Our team analyzes your website's SEO, speed, content and user experience to identify what to improve first.

Explore More

## Related Resources

### What Is SEO? A Plain English Guide

What SEO is and how it works: how Google crawls, indexes and ranks pages, the three types of SEO, how AI search changes it, costs, timelines and first steps.

Read More
https://lucidbrands.agency/resources/seo/what-is-seo

### Ecommerce SEO: A Practical Guide

A practical ecommerce SEO guide: keyword research for categories and products, site architecture, product and category page templates, faceted navigation, out of stock handling, product schema, content, links, AI search and a checklist.

Read More
https://lucidbrands.agency/resources/seo/ecommerce-seo

### Law Firm SEO Guide

How law firm SEO works: practice area pages, Google Business Profile, reviews, legal directories, ethics rules, AI search, costs, timelines and an ROI example.

Read More
https://lucidbrands.agency/resources/seo/law-firm-seo-guide

All Website SEO Resources (https://lucidbrands.agency/resources/seo)

## Privacy Choices

We measure visits and personalize ads as you browse. Your choice below decides whether that keeps happening, and you can change it any time. Our website provider counts visits without cookies.

Privacy policy (https://lucidbrands.agency/privacy-policy)

## Structured data

```json
[
  {
    "@context": "https://schema.org",
    "@type": "BlogPosting",
    "@id": "https://lucidbrands.agency/resources/seo/robots-txt#article",
    "url": "https://lucidbrands.agency/resources/seo/robots-txt",
    "headline": "Robots.txt: A Plain Guide",
    "description": "What robots.txt is and how it works, the directives, copyable templates for common sites, how to handle AI crawlers like GPTBot, robots.txt vs noindex, how to fix \"Blocked by robots.txt\" in Search Console and common mistakes.",
    "keywords": "robots txt",
    "datePublished": "2026-11-14",
    "dateModified": "2026-11-14",
    "mainEntityOfPage": "https://lucidbrands.agency/resources/seo/robots-txt",
    "articleSection": "Website SEO",
    "author": {
      "@type": "Organization",
      "name": "Lucid Brands Team",
      "url": "https://lucidbrands.agency/about"
    },
    "publisher": {
      "@id": "https://lucidbrands.agency/#organization"
    },
    "isPartOf": {
      "@id": "https://lucidbrands.agency/resources/seo#webpage"
    }
  },
  {
    "@context": "https://schema.org",
    "@type": "FAQPage",
    "mainEntity": [
      {
        "@type": "Question",
        "name": "Is robots.txt mandatory?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "No. If you do not have one, crawlers assume they can crawl everything. Most sites still benefit from a simple file that points to the sitemap and blocks admin or duplicate areas."
        }
      },
      {
        "@type": "Question",
        "name": "Is robots.txt good for SEO?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "Yes, when used carefully. It helps search engines focus on your important pages and avoid wasting effort on duplicates and filters. Used carelessly, it can block pages you want ranked."
        }
      },
      {
        "@type": "Question",
        "name": "Is ignoring robots.txt illegal?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "Robots.txt is a voluntary standard, not a law in itself. Reputable crawlers follow it. Whether ignoring it creates legal problems depends on other laws, terms of service and the situation, so ask a lawyer if this affects you."
        }
      },
      {
        "@type": "Question",
        "name": "How do I fix a blocked by robots.txt error?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "Check whether the page should be indexed. If it should, find the blocking rule with Search Console's URL Inspection tool, remove or narrow it, then validate the fix. If it should not be indexed, the message is not a problem."
        }
      }
    ]
  },
  {
    "@context": "https://schema.org",
    "@type": "BreadcrumbList",
    "itemListElement": [
      {
        "@type": "ListItem",
        "position": 1,
        "name": "Home",
        "item": "https://lucidbrands.agency/"
      },
      {
        "@type": "ListItem",
        "position": 2,
        "name": "Resources",
        "item": "https://lucidbrands.agency/resources"
      },
      {
        "@type": "ListItem",
        "position": 3,
        "name": "Website SEO",
        "item": "https://lucidbrands.agency/resources/seo"
      },
      {
        "@type": "ListItem",
        "position": 4,
        "name": "Robots.txt: A Plain Guide",
        "item": "https://lucidbrands.agency/resources/seo/robots-txt"
      }
    ]
  }
]
```