The Bot Crawls 001: what actually found a new website

The window

19 July to 21 July 2026, UTC, taken as three separate one-day queries because Cloudflare caps this dataset at a day per query. The site went live at 00:07Z on 19 July (01:07 BST), so the window opens on launch day and closes 23 hours into 21 July. Each query asked for a whole UTC day, half open from midnight to midnight, but the last one ran before the day's final hour had been aggregated, so the export stops after the 22:00Z bucket. The last hour is drawn hatched on every board below to mark a day that stops early, not an hour that was measured short.

The share climbs across the three days while the absolute traffic falls, which is the launch noise draining away and leaving the crawlers behind. It is the one trend in this study I would not yet call a finding: three days is three points.

What is excluded, and why

20.8% of all traffic in the window was our own. It is removed from every AI figure above and named here rather than quietly dropped, because a study that trims its own footprint without saying so is not a study.

Who came

One row per operator, one bar per hour, across all 71 hours of the window. Each row is scaled to its own busiest hour, and that peak is printed at the end of the row, so the rows are comparable in SHAPE and never in height: a full bar on Diffbot is two requests and a full bar on Anthropic is dozens. Sharing one scale would flatten every small operator into a flat line, which would be tidier and would hide the finding.

Two operators account for most of it, and they behave differently: Anthropic's crawlers made the most requests, OpenAI's took the most bytes. The crawler-level cut says why, and it is the first thing on this page I did not expect.

What the status codes say

The 302s are the Cloudflare Access gate working: 34 redirects to a login page, every one of them a request for a private /lab/ route. The 404s are a different story: 99 of the 101 fall in a single hour on launch day, and the other 2 are ordinary crawlers meeting a missing page on the two days after.

One crawler, tested against the gate

A focused raw-event extraction for 20 July, run later and separately, shows what those redirects look like from the crawler's side. It returned 57 rows for /lab/ across 39 distinct paths. 42 rows carried status 302 and 15 carried status 200. Rows, not requests: a raw-event query returns sampled events and cannot measure volume, which is why no total on this page comes from one.

35 rows presented the Claude-SearchBot user agent, across 35 different /lab/ paths: one row each, a methodical sweep rather than a retry loop. 34 were redirected by Cloudflare Access. One returned 200, /lab/thesis/, which is the page deliberately excepted from the gate. All 35 came through one source address on the same Amazon.com, Inc. network, AS16509. The remaining 22 rows in the extract, including 14 of the 200s, are first party: me and the desk reading the benches from UK consumer networks, signed in through Access.

The source network, the one-request-per-path pattern and the /robots.txt behaviour are all consistent with documented Claude-SearchBot behaviour: the file allowed search crawlers into /lab/ at the time, and ClaudeBot, which the same file disallowed, never tried. They do not independently verify the crawler's identity. Amazon.com, Inc. is shared infrastructure that anyone can rent, and Anthropic states that it does not publish stable crawler IP ranges because its crawlers run on service-provider addresses. Consistent behaviour from a plausible network is the most this evidence supports. Note also what a 302 is: Access answered with a redirect to a login page, so the protected pages behind it were not served.

The probe hour

Every one of the 99 launch-day 404s falls inside a single hour, 07:00Z on 19 July, which is nearly seven hours after the site went live at 00:07Z. They are requests for credentials and configuration: /.env.backup, /.ssh/id_dsa, /.config/anthropic/credentials/default.json, /.codex/config.toml.

They arrive wearing 13 different AI crawler identities belonging to 9 companies: Amazon, Apple, ByteDance, Cohere, Diffbot, Google, Mistral, OpenAI, Perplexity. Requests for credential and private-key paths are inconsistent with the documented purposes of those crawlers, and strongly suggest spoofed user-agent labels.

A later raw-event query, narrowed to that hour, adds source-level evidence to the pattern. It returned 26 rows, every one a 404, across 24 paths and 19 user-agent strings from 4 sources. 23 of the 26 shared one source label. That source presented 17 distinct user-agent strings, thirteen AI crawler labels and four ordinary browser strings, and requested 23 paths in about 21 seconds.

That clustering is consistent with coordinated scanning, or with shared network egress. It does not identify one machine or one actor, and the record says why: the network behind that source is Cloudflare, Inc., AS13335, and a proxy or network egress of that kind aggregates many clients behind one address. The page claims a coordinated burst, not a culprit.

The response sizes corroborate rather than prove. Every probe the primary path cut measured was served the same 404 page, between 6,226 and 6,275 bytes apiece, which is what one 404 template looks like from any number of clients. Identical response sizes would look the same whether this was one scanner or several working from the same wordlist, so on their own they show a shared template, never a shared actor.

No Anthropic-labelled user agent appeared among the 99 credential-path probes. This is enforced by a build check: if a future change to the banked evidence or classifier introduces a Claude-labelled request into the burst, the study will fail to build until the discrepancy is reviewed. During the same 07:00Z hour, ClaudeBot made two successful requests consistent with its wider crawling pattern. The burst is therefore defined by behaviour, requests for credential and configuration paths returning 404, not merely by their timing.

Corrections, kept where they happened. The plan said a scanner impersonated thirteen AI companies; it was thirteen identities across nine companies. The first build of this page then reported the burst as 79 requests across seven companies, because its classifier was copied from that same ten-pattern query and so could not see MistralAI-User, plain Applebot or GoogleOther: twenty requests, every one a 404 inside this hour. A sweep of all 179 unmatched agent strings confirms those three were the only ones missing. Found by OpenAI Codex reviewing the built page, which is the second desk doing what it is for.

What they took

With the probes set aside, the rest reads like ordinary crawling: robots.txt, the sitemap, real pages, the Markdown companions, the PDFs.

The top four files are 70.7% of the response bytes sent to AI-labelled traffic once the probe burst is set aside, which is the denominator that matters for crawler cost. The press headshot alone accounts for 15.24 MB across 3 requests, which is the measured cost of one image of 5.08 MB (4.84 MiB) that was never resized. Every byte figure on this page is decimal MB, a million bytes, which is the unit the raw sums are in; the binary equivalent is given here once because the two conventions differ by five per cent and mixing them is how a file grows or shrinks in the retelling.

Limits, stated rather than buried

• A user agent is a claim, not an identity. Without botDetectionIds nothing here is verified. Every total is AI-labelled.
• Crawling is not citing. This page says what was fetched. Whether any of it reached an answer is the Field Tests' question, and on 20 July their answer was that none of the seven sessions cited this site.
• The window stops an hour short of the third day. The primary capture ends after the 22:00Z bucket on 21 July, and that last hour is hatched on every board. A completion capture the next day showed the hour it missed held 49 requests, 1 of them AI-labelled.
• Three days is three days. The rising AI share is a shape, not a trend, and the window is short enough that one scanner moved it.
• requestSource is eyeball, which restricts the query to Cloudflare's end-user request-source class. That is not a raw log, it does not verify a crawler's identity, and it is not a guarantee that every Cloudflare-branded service user agent is absent from these totals.
• Every count and byte sum on this page is an estimate, not a tally. httpRequestsAdaptiveGroups is an adaptive dataset: Cloudflare samples it and returns estimates, and it publishes confidence intervals for quantifying the uncertainty. Freezing the response files makes those API outputs reproducible; it does not turn them into exact origin-log counts, and the two completed days replicating byte for byte shows the API answering consistently rather than the numbers being exact.
• What the intervals do and do not cover. The original nine queries did not capture confidence intervals, so the figures published here cannot be given one after the fact. The later control queries did capture them, but for overall request counts only, not AI-labelled subsets or byte sums: nothing on this page about operators, paths, percentages or the AI-labelled subset has an interval behind it. Where the later window matches the frozen one, the intervals were 19 July, 3,378 to 4,162 around 3,770; 20 July, 2,707 to 2,927 around 2,817. The 21 July control is not comparable and is not quoted: it covers the completed 24-hour day around 1,954, while the figure frozen here is 1,905 over 23 hours, so the interval belongs to a different quantity. Read the two that are quoted as the order of the sampling uncertainty, not as a bound on every number here. The confidence level is not stated because it is not on record: Cloudflare returns the estimate, the bounds and the sample size with no level field, and the query text was not banked beside the response. Study 002 banks the query with the response, which is the only thing that makes it knowable.
• The raw-event queries answer a narrower question than the grouped ones. sampleInterval is not available as a field on the individual httpRequestsAdaptive events used here, and rayName was not authorised for this zone and token. Those rows are good for attribution and behaviour, and they can never stand in for the grouped totals, which are where the adaptive counts, the sums and the confidence bounds live. Nothing on this page counts volume from a raw-event query.

Evidence

Fifteen files sit behind this page and they are not one kind of thing. Naming them separately is the point: the moment replication, completion and validation evidence are allowed to blur into the primary dataset, no figure can be traced to the file it came from.

The primary nine are banked at source-materials/crawler-telemetry/ and the later six beside them in replication-2026-07-22/, all recomputed by site/scripts/derive-bot-crawls.mjs. The queries were restricted to a single Cloudflare zone using its private zone identifier, retained in the archived research record but omitted here. Query design, extraction and the first analysis: OpenAI Codex (GPT-5.6 Sol) with Kish, 21 to 22 July 2026. Recomputed from the raw responses by this script.

Those paths are inside a private repository, so they are a provenance record rather than something a reader can open. The banked files are privately archived and available on request, and any figure here can be checked against them by re-running the derivation script on the frozen responses.

The two raw-event files carry client IP addresses and are not in the repository. The originals stay private. What is banked is a derivative with the address removed and replaced by a study-local label (source-A and so on), which keeps the only thing the analysis needs, that certain events shared one source, and publishes no address. A plain hash was considered and rejected: an unkeyed hash of an IPv4 address is not anonymisation, because the whole space can be enumerated in minutes. Network and country are kept, since they carry the reading without carrying the address.

Six reconciliation gates pass on the primary figures at derivation: per-day totals sum to the whole, operator and crawler totals sum to the AI total, the status cut agrees with the hourly cut, and every excluded agent is accounted for in the table above. A further fifteen gates check the later evidence, including that the two completed days replicate exactly, that every hour present in both 21 July captures agrees, that the replication only ADDED paths and never changed one, and that no client address survives into the banked derivatives.

How this was made

Three parties, and the division of labour is worth stating because it is the argument this site keeps making about working with these tools.

• I set the question, ran and banked the original Cloudflare extraction by hand, took the interpretive and ethical decisions, and hold final editorial authority over every word of it.
• OpenAI Codex designed the GraphQL query approach and the first analysis, later ran the controlled recapture and the focused raw-event queries, reconciled the evidence, designed the extraction protocol, and did the privacy and acceptance review.
• Claude Code never touched Cloudflare. It implemented and tested the study from the evidence supplied and the brief agreed, and wrote the derivation that recomputes every figure here.

What that arrangement bought was not agreement. Two systems agreeing proves nothing, and where they disagreed the disagreement had to be settled against the data, which is how the classifier error and the sampling misreading were both caught. What it bought was procedure: evidence banked before it was interpreted, assumptions challenged by someone who had not formed them, different classes of error found by different readers, implementation kept separate from evidence review, and a person accountable at the end.