The Bot Crawls 002: the second look

The window

22 July to 24 July 2026, UTC, three complete days taken as three separate one-day queries because Cloudflare enforced a maximum one-day query window for this zone and dataset configuration. Each query banked its own text beside the response, with an explicit request for a confidence interval on the day's overall count. That is the one thing 001 could not do after the fact, and it is fixed here.

Across the three days: an estimated 8,617 requests, 50.67 MB in and 132.86 MB back, 1,145 visits. The 0.95 intervals belong to each day's overall count and to nothing else: not the byte sums, not the AI subset, not the three-day total, which is a sum of three estimates and carries no combined interval. Every number on this page is an ESTIMATE from an adaptive, sampled dataset, roughly one request in 1.57 to 1.79 sampled and scaled up depending on the day, never a raw server-log tally.

Everything that came, by what it claims to be

One row per identity class, each user agent placed in exactly one, so the classes add up to the whole. This is what it ASSERTS, not what it is: the behaviour comes next.

What is excluded, and why

Who came, and what they did

510 requests carried AI-associated labels, 5.9% of the study, or 453 once two edge cases are set aside: Google-Read-Aloud, an accessibility text-to-speech fetcher, and CCBot, Common Crawl, which is not an AI company though its archive feeds AI training. The finding is not the count. It is that the AI-labelled traffic splits cleanly into two behaviours.

The high-volume agents behaved consistently with ordinary content crawling: GoogleOther, Amazonbot, ClaudeBot and the Read-Aloud fetcher pulled real pages, style sheets, fonts, and the governance files a crawler is meant to read. The low-volume tail behaves nothing like it: GPTBot, xAI-SearchBot, Google-CloudVertexBot, DeepSeekBot and some ChatGPT-User requests are mostly 404s on credential paths. A real training or search crawler does not ask for /.aws/credentials. A UA that claims to be one and does is behaviour consistent with a probably spoofed label, and that is the bridge to the next section.

What the status codes say

More than a quarter of all requests, 28.2%, were 404s. That is not all broken links, but nor is it all scanning. External hostile-path probing accounts for 648 of them, almost all 404s; our own security auditing for 844 more; and the rest are ordinary not-founds, paths that never existed or no longer do, which a status code alone cannot call a probe.

The scanners

Set the first-party auditing aside and 648 external requests asked for canonical attack-target paths that were not present in the reviewed site source and returned client errors: credential files, config files, source-control paths, CMS and shell files. Every one is counted only where BOTH hold, a hostile path and a client error, so nothing here rests on a user agent alone.

The sweep grows across the window, 149, then 172, then 327. And here is the part worth pausing on: the two busiest probers wore plain browser user agents, no bot label at all. The heaviest asked for 46 distinct attack paths, the next 26. A further 76 requests came with an EMPTY user agent, hunting for PHP shells like /wso.php and /wp-login.php. Of the whole 648, only 23 carried AI-associated labels, spread thinly across nine identities (CCBot, ChatGPT-User, ClaudeBot, DeepSeekBot, GPTBot, Google-CloudVertexBot, OAI-SearchBot, PerplexityBot, xAI-SearchBot), and that set includes CCBot, an archival crawler, not only AI-company labels. Each is a UA claiming an AI crawler while asking for credentials, which is behaviour consistent with a spoofed label, and not evidence the named company did it.

All fifteen listed paths returned client errors during the capture. Fourteen returned only 404 responses; /xmlrpc.php returned eight 404 and three 405 responses. No successful resource response was observed. The capture retained status aggregates, not response bodies.

Our own scanner is in this picture too, and is kept out of the figure above. My audit scanner and the desk's curl asked for the same hostile paths 844 times, auditing my own attack surface. That is a security check of my own site, not an attack on it, so it is reported here and excluded from the 648 external total. A credential-path count read as all-external would be wrong by more than half.

Against the launch week

Both studies were reproduced from their own frozen evidence before any line was drawn between them. 001 is the launch week, 19 to 21 July; 002 is the second three-day block, 22 to 24 July. The 001 column uses 001's frozen primary figures throughout, its 8,492 total included: that window holds 71 hourly buckets, because the 21 July capture covered 23 hours, against 002's 72, and the completed-window replication put 001 at 8,541. Read every row with its caveat: the launch itself is a confound, the classifier and the query design differ between the two, and two three-day windows are not a trend.

What changed

The AI-crawler mix flipped. At launch, Anthropic and OpenAI led: ClaudeBot made 239 requests, GPTBot 179. By the second block, Google and Amazon led: GoogleOther 235, Amazonbot 93, while ClaudeBot fell to 55 and GPTBot to 4. The AI-labelled response-byte share was higher in the launch week, 31.0% against 17.0% under the same classifier, subject to the classifier, window and sampling differences described here. And AI-labelled credential probing fell sharply, 99 to 23, even as total hostile scanning rose and moved to browser and empty-UA disguises.

What persisted, and what stays unknown

The site is still scanned for secrets and shells; that pattern did not go away, it changed costume. Declared crawlers still fetched robots.txt and the sitemap, consistent with ordinary crawling. What stays unknown: who runs the scanners, because there is no source data to cluster them; whether the operator shift is a real trend or the launch surge draining away under sampling noise; and why Claude-SearchBot, present in 001, is absent here, whether it finished its work or simply was not sampled at these low counts. Every one of these is a two-point observation. None is a direction.

Limits, stated rather than buried

• A user agent is a claim, not an identity. Without bot detection IDs nothing here is verified, and this capture has no address or network to corroborate one. Every total is AI-labelled.
• Every count and byte sum is an estimate. httpRequestsAdaptiveGroups is an adaptive, sampled dataset. Freezing the responses makes the numbers reproducible; it does not turn them into exact origin-log counts.
• The confidence intervals cover one thing. Each day's 0.95 interval is on that day's OVERALL request count. Nothing about operators, agents, paths, statuses, bytes, visits or the three-day total has an interval behind it, and the daily bounds must not be summed.
• No source data. No client IP and no ASN, so no clustering, no geography, and no claim about how many actors sit behind the scanner traffic.
• The total is mostly not the outside world. First-party auditing is 26.6% of it. The field figure, 6,327, is the base for anything said about who came.
• Two windows are not a trend. The comparison with 001 reads differences between two short, adaptive-sampled blocks, one of them launch week. Direction is not claimed.
• requestSource is eyeball, Cloudflare's end-user request-source class. It is not a raw log, does not verify identity, and does not guarantee every Cloudflare-branded service agent is absent.

Evidence and reproducibility

Three frozen GraphQL responses, one per day, their SHA-256 fixed, with the exact query banked beside each. They live in an immutable evidence archive OUTSIDE this repository and are not copied into it. site/scripts/derive-bot-crawls-002.mjs reads them, verifies the checksums, re-derives every figure here, and fails on a broken checksum, a non-null error, an out-of-window row, a missing hour, a category that will not reconcile, a 25 July row, or a private field in the output. Same bytes in, same numbers out.

The 25 July capture is Bot Crawls 003 provenance and is in no figure here. This capture carries no client IP, no ASN, no credentials and no query-string field; nothing from the private Bot Crawls 001 validation files is used. 64 reconciliation and boundary gates pass at derivation, and the load-bearing ones are asserted again at build.

How this was made

Three parties, the same division of labour the series keeps arguing for.

• I set the question, hold every interpretive and ethical decision, and hold final editorial authority over every word.
• OpenAI Codex designed and ran the capture, banked each query beside its response, assembled the checksum-fixed archive, and set the study boundaries.
• Claude Code never touched Cloudflare. It verified the archive, recomputed every figure from the frozen responses, built the classifier and the gates, and wrote this page from the evidence.