How can I tell whether AI bots are visiting my website?

Search another help topic

Check your website’s access logs to see whether requests name AI bots such as OAI-SearchBot, Claude-SearchBot and Claude-User. The logs can also show which pages they requested and how your website responded.

Before you start

You’ll need access to cPanel for the website you want to check. If you use a service such as Cloudflare in front of your site, you may also need to check its security events. Requests blocked there may never reach your hosting server.

Find the requests in cPanel

  1. Open cPanel for your hosting service.
  2. Click Raw Access under Metrics.
  3. Download the current log for your website’s domain. Choose the HTTPS or SSL log if cPanel lists it separately and your site uses HTTPS.
  4. Extract the downloaded .gz archive.
  5. Open the extracted log in a plain-text editor.
  6. Search for one bot name at a time, such as OAI-SearchBot or Claude-SearchBot.

Read a matching line

Access logs typically show the request time, page path, response code and user agent. The user agent is the name and other details the visiting software sends with the request. For example, a line containing GET /about/ HTTP/1.1, 200 and OAI-SearchBot records a request for /about/ that the server answered with a success code.

  • 200 OK means the server returned a response successfully. Check the page if you suspect it serves a maintenance screen or challenge.
  • 301 or 302 means the server redirected the request. Check the destination if you expected the bot to reach a different page.
  • 403 Forbidden means a rule or permission denied access.
  • 429 Too Many Requests means a request limit was reached.

Troubleshoot missing or blocked requests

  • There are no matching lines. Check the date range and confirm that you downloaded the right domain and log. If you’ve enabled archiving, check the archived logs too.
  • The request is blocked. Look for the same time and page path in your security plugin or CDN’s security events. A CDN is a service that can serve pages and filter traffic in front of your hosting.
  • You can’t find Raw Access. Ask Ecenica support whether access logs are available on your plan. Include the affected website address.

Frequently asked questions

Does the name in the log prove who sent the request?

No. Software can use any user-agent name. If you have the visitor’s original IP address, compare it with the ranges in the provider’s crawler documentation. If a proxy sits in front of your site, check the logs that record the original address.

Is a 200 response proof that the page appears in AI answers?

No. The response code shows what your site returned. The AI service decides whether to use or cite the page.

Can I see visits from last month?

Check Archived Raw Logs in cPanel if archiving was enabled for the period you need. Retention and access vary by service.

What should I send to support?

Share the page address, bot name, response code, and time with its time zone. Remove any private information, such as access tokens in URLs, before sharing log entries.

Related articles

See the cPanel Raw Access documentation and the OpenAI and Anthropic crawler documentation for published IP ranges.

Filed under
AI

Need help with this?

Open a ticket and link to this guide so our team can see what you have tried.

View support tickets

Power your business with Ecenica Hosting

Built for WordPress and serious websites. Fast, secure and supported by real people in the UK.

Ecenica hosting services represented by a connected red route across London