Bots & AI crawlers

New projects default to Detect and report separately: recognized bots go only to this report and do not inflate the human overview.

ConsoleWeb analyticsBots

What the report shows

There are three tabs: AI & agents, AI visibility, and All bots.

AI & agents has cards for AI requests, AI agents, AI crawlers, and visits sent back. Below them, the AI content ledger lists, per operator, reads (training crawlers and answer engines in separate columns), human visitors sent back, and a return ratio. The ratio is that operator’s reads divided by the human visitors it sent back. Further down are a trend of reads against visits back, and the most-read pages.

AI visibility puts fetches, visits sent back, and the ratio of the two in one place. See AI traffic loop.

All bots lists name, operator, purpose, requests, pages visited, the most common status code, last seen, trust level, detection layers matched, and source channel. Purposes include search, AI training, AI answer engines, and monitoring. Search engines also have a small health table: fetches, success rate, 404s, and 5xx. If a row is wrong, report a false positive for a platform admin to review.

How to use it

Start with the ledger: who is reading, and who sent people back. Then open All bots and check one crawler’s pages and status codes. Whether fetches obeyed your rules file is in Crawl health & compliance.

Limits

With only the page script, you see only bots that execute JavaScript. The top of the page says so. Status codes, response time, and crawlers that do not run JavaScript need server collection or a log import. See Server events & log import. Trust is verified, suspected, or unknown. Unknown does not by itself mean the crawler is spoofed.

FAQ

Are bots still inside the human reports? In the default mode, recognized bots are excluded from human visitors, sessions, and pageviews. Only Do not detect in settings mixes them in. That mode is not recommended.

Is the return ratio the same as the ratio on AI visibility? No. The return ratio is reads divided by visitors sent back. A larger number means many reads and few people. Crawl→visit on AI visibility divides visits sent back by crawls, so the direction is reversed.

AI traffic loop

It does not predict rank, and it does not replace click data from a search account.

ConsoleWeb analyticsBotsConsoleWeb analyticsSources

On Bots, open the AI visibility tab. On Sources, open the AI sources tab. That tab counts people, not crawlers.

What the report shows

AI visibility has four cards: AI crawls, AI-referred visits (sessions whose source is an AI assistant), Crawl→visit (visits divided by crawls), and Sitemap coverage (when a sitemap is set, the share of its URLs an AI crawler has fetched). The trend draws crawls and visits together. The By AI brand table has crawls, pages, visits, and crawl→visit.

Below that are Cited pages (landing pages that received AI-referred visits) and coverage by crawler. AI sources on Sources uses the same definition as AI-referred visits here: human sessions, not request counts.

How to use it

Read the ratio and the brand table first, and find brands with many crawls and few visits. Then check Cited pages for the article being cited. To change channel classification, use channel rules on Sources. See Sources & channels.

Limits

With only the page script, crawl counts are incomplete. Crawlers that do not run JavaScript need Server events & log import. With no sitemap, coverage is empty, not zero percent. Query-level data is on search performance. See SEO & content decay.

FAQ

Sources already has AI. Why open Bots? Sources only counts people who were sent back. Bots is where you see how many times a crawler read the page. You need both to see whether reads and return visits are in proportion.

Is the ratio a click-through rate? No. It is visits divided by crawl requests, not clicks divided by impressions in search results. The return ratio in the AI content ledger on Bots runs the other way (crawls divided by visitors). See Bots & AI crawlers.