Research note
Small pilot: what do local-government robots.txt files say about AI crawlers?
A method note from ValleyFill. Run on . This is a small pilot, not a survey of all local governments.
The short version
We read the public robots.txt file of 400 randomly sampled local-government .gov domains. Of the 200 files we could read, 72 mention at least one AI crawler, 16 fully disallow at least one AI crawler, and 17 disallow all crawlers. Mentioning a crawler is not the same as blocking AI. We name no agency, and we make no claim about anyone's accessibility, readiness or compliance.
What we did
- Population: domains of type City, County or Special district in CISA's public .gov domain list (cisagov/dotgov-data,
current-full.csv, downloaded 26 October 2026). - Sample: 400 domains chosen at random with a fixed seed (20261026), so the same draw can be repeated.
- Requests: a GET of
https://<domain>/robots.txt, retried on thewww.host only if the first request failed (redirects were followed). 8-second timeout, no other pages fetched, nothing stored beyond the aggregate counts. Our user agent identified us:ValleyFill-research/0.1 (+https://valleyfill.com; hello@valleyfill.com) robots.txt aggregate study. The pilot ran twice on 26 October 2026, so each site received at most a few small requests. - Reading the files: we checked 13 known AI crawler names (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Claude-SearchBot, CCBot, Google-Extended, PerplexityBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Amazonbot). We counted whether a file mentions any of them anywhere (including comments), and whether a rule group that names one of them contains
Disallow: /. - Publishing: aggregate counts only. We do not publish or share per-domain results.
Results
| Outcome | Domains |
|---|---|
| Readable robots.txt | 200 |
| No robots.txt (404 or 410, or a web page returned instead) | 77 |
| Could not connect from our server (timeouts, TLS or DNS errors) | 99 |
| HTTP 403 | 23 |
| Other (HTTP 400) | 1 |
Of the 200 readable files:
- 72 (36%) mention at least one AI crawler. 61 of those contain no Cloudflare managed-robots.txt or Content-Signal markers; 15 of the readable files contain such markers.
- 16 (8%) have a rule group naming an AI crawler with
Disallow: /. - 17 (8.5%) disallow all crawlers (
User-agent: *withDisallow: /).
An earlier run on the same sample the same day found 203 readable files and the same 72 and 17.
Limits to keep in mind
- Small, single vantage point: 200 readable files, requested from one server. Some of the 99 sites we could not reach may block data-center addresses.
- robots.txt only: firewalls, bot management and other controls can also block AI crawlers. We did not test them.
- Mentioning a crawler is not blocking it: a file may name a crawler to allow it, to limit some paths, or only in a comment.
- Domains are not governments: one government can hold several domains, and many small governments use .org, .us or .com, which this sample does not cover.
- A snapshot: robots.txt files change. This is the state on 26 October 2026.
- Not peer reviewed. No readiness, accessibility or compliance claim is made or implied.
Reproduce it
The scripts are plain Python (standard library only): one for sampling and requests, one for counting. We will share them on request at hello@valleyfill.com. Please don't scale this up against government servers without care: we kept it to a handful of small requests per domain.
Why we did it
The Georgetown Beeck Center report Making Government a Trusted Source in AI Systems (14 October 2026), covered by StateScoop on 16 October 2026, names access as one of three factors governments control: making sure legitimate AI crawlers can reach public information. This pilot measures one small piece of that: what public robots.txt files say.