We Blocked AI Crawlers for Five Months

A one-week audit of presscart.com: a scary traffic chart that was mostly one zero-click query, crawler rules that changed inside unrelated commits, and data that cannot tell us what either one cost.

From 8 May to 7 October 2026, presscart.com's robots.txt (the file that tells crawlers which pages they may fetch) blocked AI training crawlers plus Google-Extended. The training crawlers were GPTBot, CCBot, ClaudeBot and Omgilibot from 8 May, joined by anthropic-ai on 20 May. Google-Extended, which Google documents as governing grounding in Gemini as well as training, was blocked from 20 May. The search and retrieval crawlers were never blocked: OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot and PerplexityBot were allowed the whole time.

This is our write-up of a week spent auditing presscart.com, the site of Presscart, the marketplace whose team built Lectern. It is the "before" post, and it reports no result. The first fair reading of the fix comes after 21 October 2026.

Key Takeaways

• From 8 May to 7 October 2026, presscart.com blocked AI training crawlers and Google-Extended, while search and retrieval crawlers, including those behind ChatGPT search and Perplexity, stayed allowed.
• Ahrefs showed traffic falling, but Search Console clicks were flat, and about 79 percent of the impression drop came from one zero-click navigational query on one page.
• The crawler rules were changed inside commits about other work and were never reviewed as a crawler policy. Nobody noticed for about five months.
• Every mention figure in this audit was measured while the block was in place, so the data cannot separate the block's effect from anything else.
• The rules were reversed on 6 and 7 October 2026, and a dated follow-up will report what we see after 21 October 2026.

The Alarming Number Was Mostly Noise

The audit started with a third-party alarm. Ahrefs showed presscart.com's organic traffic, keywords and AI citations all heading down. We have written separately about what Ahrefs measures for AI visibility, so this post sticks to what happened when we checked its numbers against Google's.

Google Search Console, Google's own report of how a site performs in search, told a different story. Clicks were flat: 683 in the last eight weeks against 680 in the eight weeks before. Impressions did fall, by about 39 percent between those two periods. An impression counts each time a page shows up in results, whether or not anyone clicks.

When we broke the impression drop down by query and page, about 79 percent of it came from one comparison post. That post had been collecting zero-click impressions for "businesswire," a navigational query, meaning someone typing a brand name to reach that brand's own site. The post logged 26,818 impressions for it in one 28-day window and 2 in the next.

Those searchers were looking for another company, and the impressions brought no clicks. Most of the site-wide "decline" was one page losing a query it never served.

How the Crawler Rules Changed

The second finding came from the site's git history, the record of every change to its code. The crawler rules moved inside commits about other work.

Date	robots.txt change	What the commit was about
Before 8 May 2026	Every AI crawler named in the file allowed	n/a
8 May 2026	GPTBot, CCBot, ClaudeBot and Omgilibot set to Disallow	Metadata and structured data
20 May 2026	Google-Extended and anthropic-ai added as Disallow	Search visibility claims
Throughout	OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot and PerplexityBot allowed	Never changed

Nobody reviewed those lines as a crawler policy, and nobody noticed them for about five months.

The split between training and search crawlers matters, and the providers draw it themselves. OpenAI's crawler overview says each setting is independent: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep its content out of training. Blocking GPTBot does not block ChatGPT search, and on presscart.com OAI-SearchBot was allowed throughout.

Google-Extended is the one rule that reached past training. It is not a crawler but a robots.txt token, and Google's crawler documentation says it controls whether Google may use a site's content for training future Gemini models and for grounding in Gemini Apps, which Google describes as providing content from the Google Search index to the model at prompt time. The same page says it has no effect on a site's inclusion or ranking in Google Search. So a single line added in a commit about search visibility claims also covered whether Gemini could ground answers on the site.

We have one observation here, and it stays an observation. In the two weekly panels where all five providers answered the same questions, on 28 September and 5 October 2026, Gemini cited presscart.com in 1 of 12 branded answers, against 11 of 12 for ChatGPT and 12 of 12 each for Claude and Perplexity. On unbranded questions, Gemini cited the site in none of 89 answers across our stored checks, which is not unusual: ChatGPT's 512 unbranded answers did not cite it either. We cannot say the block caused that.

What the Data Cannot Separate

Lectern runs a weekly panel of unbranded buyer questions, the kind a buyer asks without naming a vendor. On 5 October 2026, AI answers named Presscart in 4 of 132 checks, or 3 percent. The panel began on 1 September 2026. Across its six runs, Presscart was named in 3 to 6 unbranded answers per run, out of 73 to 132 answers, and the panel grew from three providers to five on 28 September, so the runs are not like for like.

Every one of those numbers was measured while the block was in place. With no clean reading from before or after it, this data cannot separate the block's effect from anything else.

One of those other things is a weakness we already knew about: the link profile. Ahrefs puts presscart.com's Domain Rating at 18. Moz's crawl of July 2026 counted 192 linking domains.

The profile has a stranger problem too. Its top anchor texts, the clickable words in links pointing at the site, are spam text. We do not know who placed the links.

We make no causal claim here. The block, the link profile and the 3 percent mention rate sit side by side in the same period, and nothing in this data ranks them.

What Changed Afterwards

On 6 October 2026, Google-Extended was allowed again, because Google documents it as governing grounding in Gemini as well as training. On 7 October 2026, the remaining training crawlers were allowed. That was a founder decision.

Two process changes followed. Lectern now checks AI crawler access every day. Lectern also compares a shipped change with the rest of the site before judging it, so a movement that the whole site shares is not counted as a result of one change.

Neither change has produced a measured result yet. They are what we do now.

A Checklist to Run on Your Own Site This Week

1. Read every AI crawler rule in your robots.txt, then read its history. Open yourdomain.com/robots.txt. Sort the rules into training crawlers (GPTBot, ClaudeBot, CCBot) and search and retrieval crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot), and check Google-Extended on its own because of its grounding role. Then look up which commits last changed those lines. Ours sat inside commits about metadata and search visibility claims.

2. Break any traffic drop down by query and page before reacting. Our 39 percent impression fall was mostly one page and one query. Search Console lets you filter by both, and that breakdown turned our alarm into a footnote.

3. Check what a third-party tool counts as traffic before trusting its decline. Ahrefs showed traffic down while Search Console showed 683 clicks against 680. A third-party estimate and your own click data can disagree, and your own clicks are the record of what happened.

What We Cannot Claim Yet

This audit covers one site over one week. It makes no causal claims about the block, the links or the mention rate, and it promises no outcome from the change. The first fair reading comes after 21 October 2026, once the reopened crawlers have had time to fetch the site and our weekly panel has run against the new rules. We will publish a dated follow-up with those numbers, whatever they show.

That leaves one thing to do before then: open your own robots.txt and read it as a policy, not as a file someone else last touched.

FAQ

How do I check whether my robots.txt blocks AI crawlers?

Open yourdomain.com/robots.txt and read each AI user-agent rule. Separate training crawlers like GPTBot, ClaudeBot and CCBot from search and retrieval crawlers like OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot, and check Google-Extended separately, since Google documents it as governing grounding in Gemini as well as training. Then find the commits that last changed those lines.

When will you report whether the change made a difference?

After 21 October 2026. The rules were reversed on 6 and 7 October 2026, and a dated follow-up post will report what Lectern's weekly panel shows.