Original research, September 2026

28% of marketing agencies cannot be read by AI crawlers

We took the 110 search terms people use to hire an agency, collected the top 20 organic results for each one, and sent every company in them the same requests that ChatGPT, Claude and Perplexity send when they gather sources. 917 agencies returned a clear answer. 257 of them could not be fully read.

These are the firms companies hire to be found. Most of them do not know this is happening, because nothing about it shows up in Google Analytics or in their rankings.

917

agencies measured

257

not fully readable

28%

of the field

Ranking first on Google does not protect you

The result we did not expect. Agencies sitting in the top three organic positions were the least readable group in the study. 31.9% of top-three results were unreachable, against 29.7% out on page two. Whatever earns a first position in Google search buys nothing with the crawlers that feed AI answers. They are separate systems with separate gatekeepers, and a site can pass one while failing the other.

Google position	Results	Unreachable	Share
1 to 3	229	73	31.9%
4 to 10	516	133	25.8%
11 to 20	502	149	29.7%

Counted per search result rather than per agency, because one agency holds different positions on different terms.

Almost nobody did this on purpose

There is a version of this finding where agencies have decided to keep AI out. That is not what the data shows. The largest failure mode by a wide margin, 160 of 917 agencies, is a server that answers a browser and then refuses the identical request once it says it is a crawler. That is a filter in a security plugin or a CDN bot rule, switched on to stop scrapers, catching the crawlers that build AI answers on the way past.

Turns crawlers away by name

160 agencies, 17.4%

The server answers a browser normally, then refuses the same request once it announces itself as an AI crawler. This is a rule someone wrote, usually in a security plugin or a CDN bot filter, and it is almost always unintentional.

Refuses anything that is not a browser

70 agencies, 7.6%

The connection completes and the server returns nothing usable to a non-browser client. The site loads perfectly in Chrome and is closed to every crawler at once.

Returns too little text to use

11 agencies, 1.2%

The page is reachable but carries so little readable text that there is nothing for a model to quote.

Answers with a bot challenge

8 agencies, 0.9%

A bot-protection layer serves an interstitial instead of the content. The crawler gets a challenge page and indexes that.

Ships an empty page and builds it in JavaScript

7 agencies, 0.8%

The server returns a near-empty document that JavaScript fills in. Crawlers that do not run JavaScript, which includes several of the ones that feed AI answers, see a blank page.

Rate-limits crawlers until they give up

1 agencies, 0.1%

Requests are throttled hard enough that a crawler abandons the fetch.

ClaudeBot gets turned away most often

Where a site refused a crawler by name, this is which names it refused. Most sites that block one block several, so an agency can appear on more than one row. The ordering tends to follow whichever bot list a plugin shipped with rather than any decision about a specific AI company.

Crawler	Feeds	Agencies refusing it
ClaudeBot	Claude	133
GPTBot	ChatGPT	120
OAI-SearchBot	ChatGPT	55
PerplexityBot	Perplexity	51
Google-Extended	Gemini and AI Overviews	43

What it looks like on a real site

The most severe verdicts in the set, quoted as the probe recorded them. Every one of these sites loads normally in a browser, which is why the problem survives so long.

echo-factory.com

echo-factory.com completed a TLS handshake but returned no usable response to a non-browser client (403). If the page loads in a browser, this is bot protection rather than downtime.

cardinaldigitalmarketing.com

cardinaldigitalmarketing.com completed a TLS handshake but returned no usable response to a non-browser client (403). If the page loads in a browser, this is bot protection rather than downtime.

advancemediany.com

advancemediany.com completed a TLS handshake but returned no usable response to a non-browser client (timeout). If the page loads in a browser, this is bot protection rather than downtime.

seoinla.com

seoinla.com completed a TLS handshake but returned no usable response to a non-browser client (TypeError: fetch failed). If the page loads in a browser, this is bot protection rather than downtime.

50pros.com

50pros.com completed a TLS handshake but returned no usable response to a non-browser client (429). If the page loads in a browser, this is bot protection rather than downtime.

coalitiontechnologies.com

coalitiontechnologies.com completed a TLS handshake but returned no usable response to a non-browser client (403). If the page loads in a browser, this is bot protection rather than downtime.

How we measured

We started from 110 search terms that people actually use to hire an agency, split across SEO firms by city, digital marketing firms by city, and marketing firms by industry. For each term we recorded the companies in the top 20 organic results, then probed each domain with the live user agents used by GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, alongside a control request that identifies as an ordinary browser.

An agency counts once, no matter how many terms it ranks for, and it is scored on its worst result. Serving one page cleanly while refusing another is not reachable. Where a probe could not reach a conclusion we report it as inconclusive and leave it out of the denominator rather than recording it as a failure, which is why the base here is 917 agencies and not 919. Social profiles and agency directories that rank for these terms were removed, because the question is about agencies.

One limit worth stating plainly. This measures whether a crawler can fetch and read the page. It does not prove any AI system chose to crawl the site or used it in an answer. A site that fails here cannot be cited. A site that passes here is eligible to be, which is not the same as being cited.

The full results

Every term we captured, with the agencies ranking for it and the verdict on each one. Ordered by how many agencies in that market came back unreadable.