See which AI bots read your site, in 10 minutes: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the robots.txt that keeps you cited

OpenAI, Anthropic and Perplexity each send at least two different bots to your site, and they don't do the same job. One collects pages that may train the next model. Another indexes pages so the chatbot's search can cite them. A third only shows up when a person asks the chatbot about you. If your robots.txt treats them as one thing called "AI", you're probably blocking the wrong one.

This guide is the list of those bots from each vendor's own documentation, read on 9 October 2026, then three ways to see which ones actually hit your site in about ten minutes, a robots.txt decision table, and an honest note on llms.txt.

I tried to pull the real bot split for jordanhongtai.com for this guide. I'll show you what I got and what I didn't, the same way I did in the page-types guide, because a guide about logs that invents a log would be a bad joke.

What you'll have when you're done

• A list of the AI user agents that matter, what each one is for, and whether robots.txt stops it
• The bot traffic on your own site, from Vercel, Cloudflare or a raw access log
• A way to tell a real GPTBot from a scraper pretending to be one
• A robots.txt that blocks training if you want, and keeps you in AI search
• A clear view on whether llms.txt is worth your afternoon
• [SCREENSHOT: Vercel Observability, Edge Requests broken down by bot name, next to the robots.txt]

Before you start

• Access to your site's hosting: a Vercel or Cloudflare dashboard, or a server where you can read the access log.
• The ability to edit robots.txt. On Next.js that's app/robots.ts. On WordPress, an SEO plugin or a file in the site root.
• Ten minutes for the first look, then a week before you change anything.
• Nothing paid. Everything here works on free plans, with one exception noted in step 3.

Step 1: Know which bot does what

Here's every user agent worth knowing, in the vendors' own words as of 9 October 2026.

Two lines in that table are easy to miss. Google-Extended never shows up in a log. Google crawls with its usual user agents and reads the token as an instruction, so you control it in robots.txt and never see it. And blocking Google-Extended doesn't take you out of Google Search or its ranking, per Google's own crawler page.

So the bots sort into three jobs. Training (GPTBot, ClaudeBot, Google-Extended). Search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot). And a fetch a person started (ChatGPT-User, Claude-User, Perplexity-User). Keep those three in your head for step 5.

Step 2: Look at what you have now

Before you read a single log, open your own robots.txt in a browser at yoursite.com/robots.txt. Here's mine, read on 9 October 2026.

User-Agent: *
Allow: /
Disallow: /api/
Disallow: /admin

Host: https://www.jordanhongtai.com
Sitemap: https://www.jordanhongtai.com/sitemap.xml

There's no rule naming any AI bot, so every one of them is allowed everywhere except the API and the admin area. I think that's the right call for a site whose whole job is to be cited, and it's also the default most sites have without ever deciding it. I also checked jordanhongtai.com/llms.txt. It returns a 404. More on whether that matters in step 6.

Step 3: See the bots on Vercel

Vercel gives you three places to look, and one of them didn't work for me.

1. Observability, Edge Requests. Vercel added a breakdown of edge requests by bot name and bot category, including AI crawlers, in May 2025. Every plan can see the dashboard. Filtering by category and breaking it down in the query builder needs Observability Plus, the one paid piece in this guide.
2. Firewall, AI Bots ruleset. In the project's Firewall, under Bot Management, the AI Bots managed ruleset is off by default. Set it to Log, not Deny. Log records the AI bot traffic without blocking anything, and the Firewall view then shows top user agents and top paths.
3. The CLI. vercel logs pulls request logs from the terminal. Here's the honest part. I ran it against jordanhongtai.com's production logs, and each row carries the path, the status code, the cache result and the source, with no user agent field. Filtering by user agent didn't work either. So the CLI can't tell you which bot asked for a page. The dashboard can, and in an unattended run I didn't have a dashboard session, so I don't have my own bot split to show you this week.

What the CLI did show me was one request in the last week for /guides/llms.txt, which returned a 404. Something was looking for an llms.txt file in the wrong folder. I can't tell you what it was without the user agent.

vercel logs --environment production --since 7d --status-code 404 --json

Step 4: See the bots on Cloudflare or in a raw log

Cloudflare. If your domain runs through Cloudflare, open AI Crawl Control, which used to be called AI Audit. Cloudflare's docs say it's available on all plans and works with no setup. It shows which AI services are crawling, their request patterns, and which ones follow your robots.txt, and it lets you allow or block each crawler one by one.

A raw access log. On your own server, the user agent is the last quoted field on each line of a standard nginx or Apache log. This counts hits per AI bot.

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log | sort | uniq -c | sort -rn

And this shows which pages one bot asked for, most requested first.

grep "GPTBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Now the catch. A user agent is just text, and anyone can send it. A scraper can call itself GPTBot. Every vendor publishes the IP ranges its real bots use: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, claude.com/crawling/bots.json, perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. Check a few of the IPs from your log against those lists before you trust the count.

Step 5: Decide your robots.txt, one job at a time

This is where the three jobs from step 1 matter. Here's how I'd decide.

The middle row is what most businesses actually want, and it's a few lines.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

The bottom row has a hole in it. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. If you really need those out, it's a firewall rule by user agent and IP, which is what Cloudflare's per-crawler block and Vercel's Deny action do.

One thing I'd never do: block the search bots to stop training. That's the mistake this whole guide is about. Block GPTBot and you've opted out of training. Block OAI-SearchBot and you've opted out of being found in ChatGPT search.

Step 6: llms.txt, honestly

llms.txt is a proposed file that gives AI tools a plain-text map of your site. It's cheap to make, which is why so many sites added one. The evidence that it does anything for citations is thin.

SE Ranking studied nearly 300,000 domains in November 2025. About 10% had an llms.txt, and its analysis found the file doesn't affect how AI systems cite content today. None of the vendor pages in step 1 mention it either. My own site doesn't have one, and the only request for one in my logs was a 404 in the wrong folder.

I'd spend the afternoon on robots.txt and on the pages themselves first. If you want an llms.txt anyway, it costs you twenty minutes and no harm, as long as you don't expect it to move a number.

What I'd do differently

Split the bots by job before you touch robots.txt. One rule for "AI" is how sites end up invisible in ChatGPT search.

Log before you block. Set Vercel's AI Bots ruleset or Cloudflare's crawler view to watch for a week first.

Check IPs, not just names. A user agent is a claim, and the vendor IP lists are the receipt.

Don't spend a day on llms.txt. Spend it on the pages you want cited.