> Canonical: https://drkishanrees.com/dispatches/the-bots-arrived/ # The bots arrived. The citations did not. *What the first three days of a new website taught me about visibility in an AI-mediated world* By Dr Kishan Rees With contributions from OpenAI GPT-5.6 Sol (Codex) and Anthropic Opus 4.8 (Claude Code). When I launched my new website, recognisable AI-crawler labels appeared in its logs within hours. At first, that looked like success. The machines had found me. But a crawler finding a website is not the same as an AI system understanding it, trusting it or citing it. In a [related field test](/colophon/field-test-003/), none of seven AI sessions cited the site, even though crawler-labelled traffic had already retrieved its pages and documents. That distinction has changed how I think about Generative Engine Optimisation, or GEO. The useful question is not simply, “Can AI see my website?” It is, “At which point between discovery and citation does the website disappear?” ## What appeared in the first three days [Cloudflare’s adaptive analytics](https://developers.cloudflare.com/analytics/sampling/) returned an estimated 8,492 requests across the frozen study window, 19–21 July 2026 UTC. Of these, 585 carried user-agent labels associated with AI companies, 1,770 came from my own audits and known tools, and 6,137 remained other or unclassified traffic. The categories describe what the evidence supports; they do not imply that the remaining traffic was human. Those labels require caution. A user agent is a name supplied by the visitor; it is [not proof of identity](https://developers.cloudflare.com/ai-crawl-control/reference/graphql-api/). Some of the activity looked like ordinary, purposeful crawling. Another concentrated burst searched for credential and configuration files while presenting thirteen different AI-crawler identities. Its behaviour was inconsistent with the documented purposes of those crawlers and strongly suggested spoofed labels. This is a behavioural inference based on its targets, timing, user-agent switching and source clustering; it is not a verified attribution to one machine, person or company. This matters because a simple dashboard total could easily turn hostile scanning into a flattering story about AI attention. The less dramatic activity was more instructive. Crawlers retrieved: - robots.txt - the sitemap - ordinary webpages - Markdown companions - PDFs - images The website was not being approached as a single visual homepage. Machines were exploring its policies, structure, source-like documents and media. Yet discovery still did not become citation. ## AI visibility is a pipeline I now find it more useful to treat GEO as a sequence of distinct stages: **Discovery.** Can the system find the URL at all, whether through links, a sitemap, a prior crawl or some other route? **Access.** Do [robots.txt](https://www.rfc-editor.org/rfc/rfc9309.html), authentication, network controls and the server permit the relevant crawler to retrieve it? **Understanding.** Is the content structured clearly enough for a machine to identify the subject, claims, evidence, authorship and relationships between pages? **Trust.** Does the system have a reason to treat the material as reliable, current and useful relative to other sources? **Citation.** When a relevant question is asked, does the system actually use or name the website in its answer? A website can pass one stage and fail at the next. Calling all five stages “visibility” conceals where the real problem lies. ## Robots.txt is an editorial document One of the more useful findings came from two decisions that were individually sensible. The site’s crawler policy allowed a search crawler to enter the private /lab/ area so that it could encounter [page-level noindex instructions](https://developers.google.com/search/docs/crawling-indexing/block-indexing). [Cloudflare Access](https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/self-hosted-public-app/) was later added to protect those research pages. Access correctly redirected the crawler to a login page. The security worked. No protected page content was served. But the crawler could no longer reach the page-level instruction that had justified allowing it into /lab/ in the first place. Two correct controls, designed separately, had quietly cancelled the logic behind one another. The rule enforced at the edge matters more than the instruction written on a page the crawler cannot reach. Crawler policy is therefore not technical housekeeping. It is part of the editorial architecture of a website, and it should be reviewed whenever access controls change. ## Machine-readable publishing helps, but proves less than we think The crawlers’ interest in sitemaps, Markdown files and PDFs supports publishing important work in more than one usable form. It does not prove that Markdown causes citation, or that a PDF will be preferred by an AI system. It does show that machines use more than the page a human sees in a browser. For someone building an online presence, the reasonable response is not to manufacture hundreds of thin “AI-optimised” pages. It is to make genuine work easier to discover and interpret: - use clear titles and authorship - maintain a coherent internal structure - publish useful primary material - provide machine-readable navigation - expose evidence in stable formats - keep crawler and security policies intentional The objective should be legibility without sacrificing substance. ## Visibility also has a cost AI-labelled traffic represented an estimated 6.9% of requests in the frozen window but 31.0% of the response bytes returned by the server. One unoptimised press photograph alone generated 15.24 MB of response traffic from three requests. That is a small example, but it adds an operational dimension to GEO. If machines repeatedly retrieve large images and documents, performance and publishing discipline matter. A website designed to be machine-accessible should not make every machine download the heaviest possible version of an asset. ## Website administration is now part of medical journalism and publishing I once thought of website administration as the machinery behind the work: hosting, access rules, analytics, redirects and files. This experiment made it feel much closer to the work itself. The public page showed me what I intended to publish. The administrative layer showed me what people and machines could actually find, retrieve and receive. That distinction revealed several things the visible website could not: a conflict between robots policy and access control, scanners wearing AI-crawler names, an analytics filter that initially missed twenty scanner requests, and the bandwidth cost of oversized assets. Learning the administrative side of a website does not mean becoming a systems engineer. It means knowing enough to ask better questions: - What is being requested? - Who only claims to be requesting it? - What did the server actually return? - Which rules permitted or prevented access? - What evidence should be preserved before the dashboard changes? - Does the machine experience match the publishing intention? For medical journalists and publishers, those questions carry particular weight. Provenance, access, accuracy and accountability are not peripheral concerns when AI systems increasingly mediate how health information is discovered and represented. Medical journalism does not end when an article is written and uploaded. How the work is structured, protected, exposed to crawlers, updated and measured influences who and what can find it. Learning enough website administration to examine those processes is therefore part of responsible digital publishing. For anyone investing in GEO, that knowledge also creates independence. It makes it possible to distinguish discovery from citation, crawler interest from security noise, and a persuasive dashboard from defensible evidence. The administrative layer is not merely backstage infrastructure. It is where editorial intentions, access decisions, provenance and public accountability become observable technical reality. ## Why this study used three desks The work was deliberately divided across three desks. ### The human and editorial desk I set the question, performed and banked the original extraction, supplied the context, made the interpretive and ethical decisions, and retained final responsibility for every published claim. ### The evidence and challenge desk OpenAI Codex designed the initial Cloudflare GraphQL approach, analysed the returned evidence, ran a controlled recapture and focused validation queries, challenged assumptions, reconciled discrepancies, and reviewed the privacy and acceptance rules. ### The implementation and testing desk Claude Code worked from the frozen evidence and agreed brief. It built the derivation, recomputed the figures, added gates that stop inconsistent totals reaching the page, implemented the publication and tested the final artefact. The value of three desks was not that two AI systems agreed with each other. Agreement between systems proves nothing. The value was separation of responsibility. Evidence was banked before interpretation. Assumptions were challenged by a desk that had not originated them. The implementation was tested against the evidence rather than trusted because it looked persuasive. When the desks disagreed, the dispute had to be resolved against the underlying data. A human remained accountable for the decision at the end. This process caught different classes of error: an incomplete classifier, an incorrect interpretation of a partial day, an overconfident attribution from response sizes, and a conflict between crawler policy and access control. The method did not eliminate error. It made errors easier to expose, name and correct. ## What I would tell someone building a GEO presence My conclusions remain provisional. This is a three-day observation of one newly launched website, not a universal study of AI discovery. But it suggests a practical starting point: - **Separate discovery from citation.** A crawler visit is not an endorsement and not evidence that a model used the page. - **Treat crawler identities cautiously.** A user-agent string is a claim until independently verified. - **Publish work, not merely metadata.** Structured pages and machine-readable documents help systems encounter substantive material; they cannot create authority that the work does not possess. - **Review security and crawler policy together.** The effective policy is whatever the crawler experiences at the network edge. - **Measure the whole pipeline.** Test whether systems can find, retrieve, interpret and cite the site rather than celebrating a single traffic number. - **Preserve the evidence.** Analytics change. A frozen response, declared exclusion rules and reproducible derivation are more useful than a screenshot of a dashboard. - **Keep a person accountable.** AI can query, classify, challenge, implement and test. Publication still requires someone to decide what the evidence justifies saying. The bots arrived quickly. The harder work begins after that. The goal is not merely to be crawled. It is to publish work that can be found, accessed, understood, trusted and cited when it genuinely helps. Companion research page: [The Bot Crawls 001: what actually found a new website](/colophon/bot-crawls-001/). A version of this essay was also published on LinkedIn: https://www.linkedin.com/pulse/bots-arrived-citations-did-dr-kishan-rees-mryne. This page is maintained as the version of record. Cite as: Rees, Kishan. 'The bots arrived. The citations did not.'. drkishanrees.com, first published 23 July 2026. https://drkishanrees.com/dispatches/the-bots-arrived/ ## Production note - **First published:** 23 July 2026. - **Last amended:** 13 August 2026. - **OpenAI GPT-5.6 Sol (Codex):** Initial draft and implementation of Kish’s dictated revisions; evidence and challenge desk. - **Anthropic Opus 4.8 (Claude Code):** Independent second-desk review. - **Final editorial responsibility:** Dr Kishan Rees.