Do Ecenica website hosting plans support ChatGPT and Claude crawlers?

Search another help topic

Yes. OAI-SearchBot, Claude-SearchBot and Claude-User can access public pages on Ecenica website hosting when your site and security settings allow them. There’s no hosting add-on to install.

What does each bot do?

  • OAI-SearchBot helps ChatGPT find pages for its web search results.
  • Claude-SearchBot crawls pages for Claude’s web search results.
  • Claude-User fetches a page when Claude needs it to answer someone’s request.

Check which pages the bots can access

  1. Open the page in a private browser window to check whether it loads without a login or password.
  2. Add /robots.txt to the site’s address, such as https://example.com/robots.txt. This public file tells crawlers which paths they can request. If your site uses www, use that address here too.
  3. Look for rules under the bot’s name or under User-agent: *, which can apply when a bot has no rules of its own. A Disallow: / rule can block the whole site when it applies to the bot and no more specific rule allows the page.
  4. Check the security log in your website’s security plugin or service, such as Cloudflare. Look for requests that were blocked, challenged or rate limited.

If you need to change robots.txt, edit it where it’s managed. WordPress or an SEO plugin may generate it for you. If it’s a file in your website folder, you can edit it in File Manager in cPanel. Open the address again after saving to confirm that it shows your new rules.

Troubleshoot blocked requests

  • A 403 Forbidden response or a browser challenge can mean a security rule stopped the request. Check the log for the rule that matched it.
  • A 429 Too Many Requests response means the request hit a limit. Check whether the response came from your website, hosting or a service in front of your site.

If you need help, contact Ecenica support with the page address, the bot name and the time and time zone of the request. Include any error code or matching log entry you have.

Frequently asked questions

Can I allow search while opting out of AI training?

Yes. OpenAI and Anthropic have separate rules for their training crawlers, GPTBot and ClaudeBot. You can manage those rules separately from the three bots covered here.

How soon will ChatGPT pick up a change to robots.txt?

OpenAI says its search systems can take about 24 hours to adjust after you change robots.txt. See the OpenAI crawler documentation.

Can I use robots.txt to keep a page private?

Protect private pages with a login or password. Anyone can read your robots.txt file, and crawlers that ignore it can still request pages that are otherwise public.

What if I don’t have a robots.txt file?

If /robots.txt returns 404 Not Found, there’s no file at that address. Check your website and security settings to see whether they allow the bots to reach the pages you want to share.

Will allowing these bots make my website appear in answers?

Allowing access lets ChatGPT and Claude read eligible pages. Each service decides which pages to show or cite.

For Anthropic’s access rules, see the Anthropic crawler documentation.

Related articles

Need help with this?

Open a ticket and link to this guide so our team can see what you have tried.

View support tickets

Power your business with Ecenica Hosting

Built for WordPress and serious websites. Fast, secure and supported by real people in the UK.

Ecenica hosting services represented by a connected red route across London