Yes. You can ask OpenAI and Anthropic to keep new crawls of your public pages out of AI training while allowing their search bots to visit. Add rules for the training crawlers, GPTBot and ClaudeBot, to your site’s robots.txt file.
Before you start
You’ll need access to the tool that manages your site’s robots.txt. This public file tells crawlers which pages they can request. Open it by adding /robots.txt to your website address, for example https://example.com/robots.txt.
Block the training crawlers
- Open your existing
robots.txtfile. Copy its contents somewhere safe before making changes. - Open the tool that manages the file. If you have a file in your site’s public folder, open File Manager in cPanel. If your website or WordPress SEO plugin generates the file, use that tool’s settings.
- Add the two groups below. If either bot already has a group, review that bot’s existing rules instead of adding a second group. To block the entire site for that bot, replace its existing
AllowandDisallowrules withDisallow: /. - Save the changes.
- Reopen
/robots.txtin your browser and check that both groups appear.
Add these rules to block the two training crawlers across the site.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Keep search access available
Review the existing rules for OAI-SearchBot, Claude-SearchBot and Claude-User. If one of these bots has no named group, check the general User-agent: * group. A Disallow: / rule in an applicable group can prevent it from requesting any page on that site.
Keep rules that apply to your other crawlers and any paths you intend to exclude. Also check your security plugin or service, such as Cloudflare, for rules that block these bots independently of robots.txt.
Troubleshoot your rules
- The old file still appears. Check whether your website or a plugin generates the file, and whether your caching service still serves the previous version.
- The rule works on one address. Check each address that serves content, including a separate subdomain such as
blog.example.com. - A search bot still can’t read the page. Look for security blocks or rate limits in your site’s logs. OpenAI says its search systems can take about 24 hours to adjust to changes in
robots.txt.
Frequently asked questions
Will these rules remove content already collected?
These settings describe how the providers should treat future crawls. Check each provider’s documentation for details about its data practices.
Can I apply the rules to just one folder?
Yes. For example, use Disallow: /news/ under the training bot’s name to exclude that folder instead of the whole site.
Will my site definitely appear in AI answers?
The services decide which pages to show or cite. Allowing access lets them read eligible public pages.
Can I use robots.txt to protect private pages?
Use a login or password for private pages. The robots.txt file itself is publicly readable.
Related articles
- Do Ecenica website hosting plans support ChatGPT and Claude crawlers?
- How to use File Manager in cPanel
For details about how the providers use the rules, see OpenAI’s crawler documentation and Anthropic’s crawler documentation.