Ranking on Google no longer means AI tools can see you. Google is closing its results pages to scrapers, so ChatGPT, Copilot and Perplexity are finding the web their own way. The result is 2 internets, and your business needs to show up in both.
I spotted the first sign myself. Searching Google in an incognito window, I now get a bot check before I see results. I’m not alone. Search Engine Roundtable has reported Google testing a “Sign in to verify you’re a human” prompt in Chrome incognito (August 2026) and a “verify your request” screen (September 2026). Google hasn’t announced either. They look like tests, not a new rule. But they fit a pattern.
Key points
- Google’s results are getting harder for bots to read. JavaScript required since January 2025, the 100 results setting gone since September 2025, and a lawsuit against a company that resells Google results.
- Your Google rankings have not changed. Google has not wiped or rebuilt its index.
- AI tools use other sources. Copilot runs on Bing. ChatGPT search uses its own crawler plus outside providers, Bing among them.
- Bing has a second job now. In Ireland it had 4.2% of searches in September 2026, and it also feeds AI answers.
- Your own website decides the rest. Your robots.txt, host and firewall control which AI crawlers get in, and they can only read text that is in the HTML.
Google is locking its results away
Over the last 2 years Google has made its results pages much harder for bots to read.
- January 2025: Google Search started requiring JavaScript. Google said it was to fight bots and spam.
- September 2025: Google stopped supporting the &num=100 setting. Rank trackers used it to pull 100 results in 1 request.
- December 2025: Google sued SerpApi, a company that sells Google results to other tools. Google says SerpApi gets around “SearchGuard”, its bot blocking system. A judge dismissed the first complaint in July 2026. Google filed a narrower one in August, and the judge dismissed that too on 9 November 2026. So far the courts have not backed Google, and the case may not be over.
- 1 January 2027: Google’s Custom Search JSON API closes to existing users. It is already closed to new ones.
Here is what has not happened. Google has not wiped or rebuilt its index. Your Google rankings are where they were. What is changing is who else gets to read Google’s results.
That matters because AI tools have used Google’s results. In August 2025, The Information reported that OpenAI was getting Google data through SerpApi. Neither company confirmed it. Either way, scraping Google is getting harder, slower and more legally risky. So the AI tools lean on other sources.
There are now 2 internets
Google’s internet. Google Search, AI Overviews, AI Mode and Gemini all run on Google’s own index.
Everyone else’s. The other AI tools build answers from a mix of sources:
- Microsoft Copilot is grounded on Bing.
- ChatGPT search uses OpenAI’s own crawler (OAI-SearchBot) plus outside providers. Microsoft says its Web IQ service, powered by Bing, is used by ChatGPT. Research by Peec AI (September 2026) found ChatGPT uses at least 8 outside providers, including Bing and Google results bought from data companies.
- Perplexity and Claude each run their own search crawlers.
These indexes drift apart. A page Google has indexed can be missing from Bing. A site that ranks well on Google can be absent from a Copilot answer. Being visible in one internet doesn’t guarantee the other. At Pixelated Bubble we check our clients in both.
We told clients to ignore Bing. That advice is out of date.
For years I told clients not to worry about Bing. In Ireland, Google had 93.5% of searches in September 2026 and Bing had 4.2% (StatCounter). Google still matters most.
But Bing now does a second job. It feeds Copilot, and it is one of the sources behind ChatGPT search. Bing Webmaster Tools even has an AI Performance report (public preview since February 2026) that shows how often your pages are cited in Copilot and Bing’s AI answers.
Here’s what to do. It takes about 30 minutes.
- Compare your index on both. Search
site:yourdomain.ieon Google and on Bing. The counts are rough. If Bing shows far fewer pages, you have a gap. For a firm answer, use URL Inspection in each tool. - Check Google Search Console. Confirm your sitemap is submitted and look at the Pages report for anything not indexed.
- Set up Bing Webmaster Tools. Sign in and choose Import from Google Search Console. Your sites are verified automatically and your sitemaps come across. Data can take up to 48 hours to appear.
- Turn on IndexNow. It tells Bing (and Yandex, Naver, Seznam, Yep and Amazon) the moment a page changes. Google doesn’t use it. On WordPress, Rank Math includes it free in the Instant Indexing module. In Yoast it’s a Premium feature.
- Test the answers. Ask ChatGPT, Copilot and Perplexity the questions your customers ask, such as “best plumber in Letterkenny”. Note who gets named and which pages get cited. Repeat monthly.
Google’s lockdown doesn’t stop AI reading your website
This is the part people mix up. Google’s blocking only affects bots scraping Google’s results pages. It does nothing to stop AI crawlers visiting your own website directly. Your robots.txt file decides which bots are welcome. So do your host and your firewall.
Let the search bots in. Keep the heavy crawlers in check.
Each AI company runs separate bots for separate jobs. That gives you control. OpenAI says each setting is independent, so blocking GPTBot does not remove you from ChatGPT search.
| Bot | Company | Job | Our advice |
|---|---|---|---|
| Bingbot | Microsoft | Bing index, Copilot | Allow |
| OAI-SearchBot | OpenAI | ChatGPT search | Allow |
| Claude-SearchBot | Anthropic | Claude search | Allow |
| PerplexityBot | Perplexity | Perplexity search | Allow |
| GPTBot | OpenAI | Model training | Your choice |
| ClaudeBot | Anthropic | Model training | Your choice |
| CCBot | Common Crawl | Training datasets | Your choice |
| Bytespider | ByteDance | Model training | Consider blocking |
Watch the names. ClaudeBot is Anthropic’s training crawler. The one that powers Claude’s search is Claude-SearchBot. Blocking training bots is a business decision, not a must. It saves server load, but it may also keep you out of what future AI models know.
A starting robots.txt for a WordPress site:
# Search and citation bots: welcome
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# Training crawlers: blocked (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
# Everyone else
User-agent: *
Crawl-delay: 10
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.ie/sitemap_index.xml3 things to know:
- Crawl-delay is a hint. Google ignores it. Bing supports it. Anthropic says it respects it “where appropriate”. OpenAI and Perplexity don’t say.
- robots.txt is a request, not a lock. Bots fetching a page because a user asked (ChatGPT-User, Perplexity-User) may not follow it. In 2025 Cloudflare caught Perplexity crawling with undeclared bots. To enforce limits, use Cloudflare’s AI Crawl Control to allow or block each crawler, and add a rate limiting rule.
- Your host or firewall may overrule you. Some managed WordPress hosts block AI bots by default. Since 15 September 2026, new Cloudflare domains block training and agent bots by default on pages that show ads. Check your settings so your “allow” actually works.
Then cache everything. A cached page costs your server almost nothing to serve, whoever asks for it.
Why a cached WordPress site has the edge
A December 2024 study by Vercel and MERJ found GPTBot, ClaudeBot and PerplexityBot don’t run JavaScript. Googlebot does. We’ve seen nothing newer that changes this.
That matters. WordPress with LiteSpeed Cache sends every visitor a complete HTML page. Any crawler can read it, and it’s cheap to serve.
Many apps built with tools like Lovable work differently. The page arrives nearly empty and JavaScript fills it in. A crawler that doesn’t run JavaScript can see almost nothing. Lovable has improved this: apps created since May 2026 render on the server. Plenty of older ones have not caught up. When we fetched 271 AI built Irish websites without JavaScript on 15 November 2026, 111 sent back fewer than 25 words. The background is in our State of Irish Websites 2026 report and in what Google and ChatGPT actually see.
If your site was built this way, check it. Open the page source (not the inspector) and see if your text is there.
There’s a cost side too. On Vercel, the AI bot rules are switched off by default. In 2026, people on Vercel’s community forum reported free Hobby accounts paused after heavy GPTBot and ClaudeBot traffic. Crawling isn’t free when you pay per request.
Show up in both internets
Pixelated Bubble is a Google Partner with 12 years in search. We check where your site appears on Google, on Bing and in the AI answers your customers read. Then we fix the gaps. Book a free 15 minute call.
FAQ
Does ChatGPT use Google or Bing?
Neither on its own. ChatGPT search uses OpenAI’s own crawler plus outside providers. Microsoft says its Bing powered Web IQ service is one of them. Research by Peec AI in September 2026 found at least 8 providers.
Should I block GPTBot?
It is your choice. GPTBot collects training data. Blocking it does not remove you from ChatGPT search, which uses a separate bot called OAI-SearchBot.
Do I need Bing Webmaster Tools in Ireland?
Yes, it is worth 30 minutes. Bing had 4.2% of Irish searches in September 2026, but it also feeds Copilot and is one of the sources behind ChatGPT search.
Has Google changed my rankings?
No. Google has not wiped or rebuilt its index. What has changed is how easily other tools can read Google’s results pages.
Can AI crawlers read a JavaScript website?
Often not. A 2024 study by Vercel and MERJ found the main AI crawlers don’t run JavaScript. If your text is not in the page source, they can’t read it.
Last updated: 15 November 2026.
Sources
- Google Search tests sign in to verify searches are human, Search Engine Roundtable, 3 August 2026
- Google: verifying your request to continue to search results, Search Engine Roundtable, September 2026
- Google amends SerpApi suit with content licensing terms, Search Engine Journal
- Court dismisses Google’s amended complaint against SerpApi, SerpApi (the defendant’s own account), 10 November 2026
- Search engine market share, Ireland, StatCounter
- Overview of OpenAI crawlers, OpenAI
- The rise of the AI crawler, Vercel and MERJ
- IndexNow and Bing Webmaster Tools