Guides / Traffic

Your newest users are not people. Decide what they are allowed to do.

In June 2026, Cloudflare's own measurements crossed a line: automated clients now generate 57.5% of requests for web pages, against 42.5% from humans, a crossover the company's CEO had predicted for late 2027. The share is of requests, not of attention, and it arrived early because of agents acting on a person's behalf rather than classic scrapers. Your vibe-coded app is part of that denominator. Some of the machines are collecting training data, some are indexing you for AI search answers, some are a paying customer's assistant doing their admin for them, and platforms are now formalising the last group: Lovable can publish your app's functions as actions an assistant calls, and OpenAI's Sign in with ChatGPT beta went live with six developer platforms in early August. Each kind of visitor deserves a different answer. This guide gives you the exact user agents and IP lists to identify them, the robots.txt lines and what they do and do not enforce, rate limiting that still works when the caller is a cloud IP pool, and the checks to make before you hand your app's actions to assistants.

Four kinds of machine visitor, four different answers.

The mistake is treating this as one problem called bots. The vendors themselves split their automated traffic by purpose, and publish the split, because they want site owners to make different decisions for each. Reading OpenAI's and Anthropic's own crawler documentation, four categories fall out:

  1. Training crawlers. OpenAI's GPTBot and Anthropic's ClaudeBot collect content that may be used to train foundation models. Blocking them costs you nothing operationally; OpenAI states plainly that disallowing GPTBot indicates a site's content should not be used in training, and Anthropic says the same of ClaudeBot.
  2. Search and answer bots. OAI-SearchBot and Claude-SearchBot index you so assistants can cite and link you. OpenAI notes that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. For most products, this is traffic you want: assistant answers are how a growing share of buyers first find software.
  3. User-triggered fetchers. ChatGPT-User and Claude-User fire when a person asks their assistant to open your page or use your service. This is a human customer wearing a machine face, and OpenAI's documentation carries the sentence that should reset your expectations: because these actions are initiated by a user, robots.txt rules may not apply.
  4. Agents you invited. If you enable Lovable's agent integrations, your published app's functionality becomes actions an assistant can call: Lovable proposes the actions, you choose who can call them, and you get an MCP link your users add to ChatGPT or Claude. At that point agent traffic is not an anomaly to filter; it is your API surface, and it needs the same care as any API you would ship deliberately.

The policy follows the category. Training crawlers are a licensing decision. Search bots are a marketing decision. User-triggered fetchers and invited agents are product traffic, and the question stops being how to block them and becomes how to authenticate, limit, and bill them, the same questions our load guide asks about human traffic, at machine speed. Note this guide is about agents using your app; for the agent that builds your app deleting your database, see the agent guardrails guide.

Identify who is actually calling.

Start with your access logs, because the answer is already in them. Every vendor above declares itself in the user-agent string. This one command gives you the count per agent for an nginx or Apache combined-format log:

awk -F'"' '{print $6}' access.log | \
  grep -iE 'gptbot|chatgpt-user|oai-searchbot|claudebot|claude-user|claude-searchbot|perplexitybot|bytespider' | \
  sort | uniq -c | sort -rn

The strings to know, from the vendors' own documentation: OpenAI sends GPTBot, ChatGPT-User, and OAI-SearchBot (each with a version suffix and a link back to openai.com), and Anthropic sends ClaudeBot, Claude-User, and Claude-SearchBot. A user-agent string is a claim, not proof: anyone can send GPTBot in a header. OpenAI publishes the IP ranges each of its agents calls from as JSON at openai.com/gptbot.json, openai.com/chatgpt-user.json, and openai.com/searchbot.json, so a claimed GPTBot hit from an address outside those ranges is an impostor. The reverse also happens: in August 2025 Cloudflare publicly accused Perplexity of crawling sites that had blocked it by rotating user agents and network routes. Declared identity is a courtesy that most large vendors extend and some do not, which is why enforcement (the next two sections) cannot live in robots.txt alone.

Then split your dashboard's traffic numbers by those agents before you celebrate or panic. A growth spike that is ClaudeBot re-crawling your marketing pages is not product-market fit, and a conversion-rate collapse can be nothing more than thousands of machine sessions that will never sign up diluting the denominator. If your analytics runs client-side JavaScript, most crawlers never execute it, so your server logs and your analytics tool will disagree; the logs are the ones telling the truth about load and cost.

Write the policy down in robots.txt, then enforce it elsewhere.

robots.txt is where you declare intent. A sensible starting policy for a product whose content is its value, but which wants to be found by assistant users:

# Training crawlers: our content is not free training data
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Search and answer bots: index us, cite us, send us users
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Flip the training-crawler lines to Allow if being in training data suits you; for most small products it is a giveaway with no return path. Anthropic additionally honours the non-standard Crawl-delay directive if you want ClaudeBot slower rather than gone. Two honest caveats. First, robots.txt only binds the well-behaved: it changed nothing about the Perplexity behaviour Cloudflare described. Second, the user-triggered fetchers are explicitly outside it: OpenAI says robots.txt rules may not apply to ChatGPT-User because a human initiated the request. Enforcement therefore lives at the edge and in your app: CDN-level managed rules that verify declared bots against their published IP ranges (Cloudflare and its competitors ship these), web application firewall rules for the impostors, and rate limits, which is the part you control completely and the part vibe-coded apps ship without.

Rate limit identities, not IP addresses.

The rate limiting most tutorials teach keys on the caller's IP address, and agent traffic breaks both of its assumptions at once. Requests from one assistant vendor arrive from a large, shifting pool of cloud addresses, so a per-IP limit set for a human with one browser never triggers. Meanwhile hundreds of real mobile users can share one carrier NAT address, so the same limit set aggressively enough to catch agents blocks paying humans. The unit that behaves consistently is the identity: the API key, the session, the signed-in user. Key your limits on that, and fall back to IP only for anonymous routes.

In nginx, that is a two-line change from the tutorial version:

# Key on the Authorization header when present, IP otherwise
map $http_authorization $rl_key {
    ""      $binary_remote_addr;
    default $http_authorization;
}

limit_req_zone $rl_key zone=per_client:10m rate=5r/s;

server {
    location /api/ {
        limit_req zone=per_client burst=20 nodelay;
        limit_req_status 429;
    }
}

If your app is on Supabase or another serverless stack with no nginx in front, do it in the database, where every request already lands. A fixed-window counter is one table and one function:

create table if not exists request_counts (
  identity text not null,
  window_start timestamptz not null,
  requests int not null default 0,
  primary key (identity, window_start)
);

create or replace function take_token(p_identity text, p_limit int)
returns boolean
language plpgsql
security definer
as $$
declare
  current_count int;
begin
  insert into request_counts (identity, window_start, requests)
  values (p_identity, date_trunc('minute', now()), 1)
  on conflict (identity, window_start)
  do update set requests = request_counts.requests + 1
  returning requests into current_count;
  return current_count <= p_limit;
end;
$$;

Call take_token(auth.uid()::text, 60) at the top of your RPC or edge function and return HTTP 429 when it comes back false; agents, unlike some humans, read the status code and back off. Delete rows older than an hour on a schedule so the table stays small. Return the limit decision before you do the expensive work, not after: on serverless, every request a crawler makes is a function invocation you pay for, and if the route calls a language model the meter spins faster still, which is the failure mode our runaway API bills guide covers, with spending caps to match. Machine traffic does not get tired, does not get bored, and does not sleep in your timezone, so the limits, not your attention, have to be the thing that holds.

Exposing actions to assistants makes you an API vendor. Act like one.

The invited-agent category is the newest, and the platforms are making it a checkbox. Since mid-July 2026, Lovable's agent integrations can turn a published app's functionality into actions assistants call: you enable it from the editor, Lovable proposes the actions, you choose who can call them, and your users add the resulting MCP link to ChatGPT or Claude. The identity rails are being laid at the same time: Supabase became a launch partner for OpenAI's Sign in with ChatGPT beta in late July, alongside five other developer platforms, so the account your user holds with their assistant is starting to double as an identity elsewhere. None of this is speculative traffic anymore; it is a distribution channel with a settings page.

Before you enable it, apply the same production questions you would to any API, because that is what you are shipping:

  1. Expose the minimum. Review the proposed action list and remove anything destructive or bulk: an assistant that can look up availability and book one slot is a feature, an assistant that can cancel all bookings is an incident waiting for a mis-parsed sentence.
  2. Authenticate the person, not just the agent. Actions should run as the end user with that user's permissions, never as a shared service account. If your data layer is Supabase, this is exactly the job of Row Level Security, per our RLS guide: policies hold no matter how confused the caller is.
  3. Rate limit per user identity, as above. An assistant retrying a failed booking in a loop looks exactly like abuse, because operationally it is.
  4. Log agent actions distinctly. Record which actions were invoked by an assistant and for whom, so the support ticket that starts with "I never did that" has an answer. Your users are accountable for their agents, but only if you can show them what their agents did.
  5. Keep a kill switch. Know the path to disable the integration or revoke one user's access before you need it, and test it. Unpublishing is the blunt version and per-action control in the editor is the precise one. Note that the scope widened on 10 August 2026: agent integrations were limited to publicly published apps at launch, and Business and Enterprise workspaces can now put one on an app published only to the workspace, where the tools always require sign-in.

The honest summary: agent traffic is not an invasion, it is a shift in who holds the browser. The businesses that do well out of it will be the ones that can tell machine visitors apart, charge or limit them accordingly, and expose exactly the actions they mean to, nothing more. All of that is ordinary production engineering, which is precisely the layer vibe-coded apps skip; the rest of the guides library exists for the same reason.

Questions

Questions founders ask.

How do I tell whether traffic is an AI agent or a human?

Check the user-agent string in your server logs first: OpenAI declares GPTBot, ChatGPT-User, and OAI-SearchBot, and Anthropic declares ClaudeBot, Claude-User, and Claude-SearchBot. Then verify the claim against the vendor's published IP ranges, which OpenAI serves as JSON at openai.com/gptbot.json, openai.com/chatgpt-user.json, and openai.com/searchbot.json; a declared bot calling from outside its published ranges is an impostor. Also compare server logs against your analytics tool: most crawlers never execute client-side JavaScript, so a gap between the two is usually machine traffic. What you cannot do is rely on user agents alone, since undeclared crawlers exist and have been publicly documented.

Should I block AI crawlers from my app or site?

Split the question by purpose before answering. Training crawlers such as GPTBot and ClaudeBot collect content for model training; blocking them in robots.txt costs you nothing operationally and both vendors document that a disallow signals your content should be excluded from training. Search and answer bots such as OAI-SearchBot and Claude-SearchBot are different: OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, so blocking them removes you from a channel where buyers increasingly ask for software recommendations. User-triggered fetchers like ChatGPT-User are your own customers acting through an assistant, and blocking them breaks real usage. Most products should block trainers, allow search bots, and manage the rest with authentication and rate limits.

Does robots.txt actually stop AI agents?

Only the well-behaved ones, and only the categories it was meant for. Declared training and search crawlers from the major vendors do honour robots.txt, and Anthropic additionally supports the Crawl-delay directive. But OpenAI's own documentation says robots.txt rules may not apply to ChatGPT-User because those requests are initiated by a user, and undeclared crawling happens: in August 2025 Cloudflare publicly accused Perplexity of reaching blocked sites by rotating user agents and network routes. Treat robots.txt as your published policy, and put the enforcement where it cannot be ignored: CDN or WAF rules that verify declared bots against published IP ranges, and rate limits keyed on identity inside your app.

Why is per-IP rate limiting not enough for agent traffic?

Because both of its assumptions fail at once. Agent requests arrive from large, rotating pools of cloud addresses, so no single IP crosses a per-IP threshold even while the vendor as a whole hammers you. Meanwhile many real users share one address behind carrier NAT, so tightening the per-IP limit far enough to catch agents starts returning 429 to paying humans. Key your limits on stable identities instead: the API key, session, or signed-in user, with IP-based limits only as a fallback for anonymous routes. And enforce the limit before the expensive work runs, because on serverless every machine request is an invocation you pay for, and on model-backed routes it is tokens too.

Are Lovable's agent integrations safe to turn on?

They are as safe as the API design you bring to them. The mechanism itself is deliberate: Lovable proposes the actions, you choose who can call them, and users add an MCP link to their assistant. The scope was publicly published apps only at launch in July 2026, and from 10 August 2026 a Business or Enterprise workspace can also expose an app published only to that workspace, where the tools always require sign-in. The risk is in what you expose and as whom it runs. Keep the action list minimal and non-destructive, make actions run with the end user's own permissions backed by Row Level Security rather than a shared service role, rate limit per user, log assistant-invoked actions separately so disputes are answerable, and know how to disable the integration quickly. If those five are in place, an assistant booking a slot for your customer is just a new client shape on an API you already control.

Before the machines find you

Get your traffic policy reviewed with the rest of your production gaps.

A production-readiness review covers how your app meets the traffic actually arriving in 2026: whether you can tell agents from humans in your logs, whether rate limits hold when the caller is a cloud IP pool, whether exposed actions run with the right permissions, and what a machine-speed mistake would cost you. Ranked by the engineers who would do the work.