Your robots.txt might be cutting you out of ChatGPT.

What each bot does, per the companies that run them

Straight from each company's documentation, read today:

• OpenAI: GPTBot gathers content to train its models; OAI-SearchBot surfaces sites in ChatGPT search. OpenAI is blunt about it: sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links.
• Anthropic: ClaudeBot is for training; Claude-SearchBot improves search results and Claude-User fetches pages when someone asks Claude to. Anthropic says blocking either of the last two may reduce your visibility in its users' searches.
• Perplexity: PerplexityBot surfaces and links sites in its results and, per its docs, isn't used to train models.
• Google: Google-Extended controls whether your content trains Gemini. Google states it doesn't affect your inclusion in Search and isn't a ranking signal.

There's a third kind robots.txt doesn't really govern: the fetch that happens when a person asks the AI to open your page. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them because a user requested the visit.

A robots.txt that keeps the two apart

If you don't want your site training models but do want to show up when people ask, put this at yoursite.com/robots.txt:

# Training: out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Search and everything else: in
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

If you don't mind training, the last three lines are enough. What I'd avoid is User-agent: * with Disallow: / plus hand-picked exceptions: it's easy to lock out a new search bot without noticing. Since October 30, 2026, Anthropic also honors a machine-readable no-training preference signal; if your robots.txt names ClaudeBot, that rule wins.

Where the block sneaks in

• Your builder. Lovable's docs say it creates a robots.txt with every new project, but they don't show what's in it. Don't assume: open it.
• Cloudflare. Turn on Bot Preference Sync and Cloudflare prepends its own rules to your robots.txt; the example in its docs disallows GPTBot, ClaudeBot and Google-Extended, among others, and lets everyone else through. And if you set the Search policy to Block, the block happens at the network edge and shows up in no robots.txt at all. New domains start with all three policies on Allow.
• The wrong domain. Google notes the file only covers the protocol, host and port it's served from. The robots.txt on yoursite.lovable.app isn't the one on yoursite.com: check the domain you actually share.

A two-minute check

Open https://yoursite.com/robots.txt in your browser and look for OAI-SearchBot, Claude-SearchBot, PerplexityBot, and any Disallow: / under User-agent: *. If you change it, give it time: OpenAI says about 24 hours for its systems to pick it up, Perplexity up to 24.

Slop Score Pro reads your robots.txt and splits the result in two: «You're blocking the AI tools that search», which is serious, and «You're blocking the bots that train AI», which is a fair choice as long as it's on purpose. Also make sure each page's HTML arrives with text in it, since a bot allowed into an empty page has nothing to quote. To start with the free version, measure your site with the Slop Score; if your link also shows up without an image in chats, there's the post on link previews.