Google-Extended: what blocking it changes, and what it leaves alone

A token, not a crawler

Google-Extended is unusual among the names in the AI crawler list: it never appears in a server log. There is no Google-Extended user-agent making requests. Googlebot fetches your page exactly as it always has, and then Google consults your robots.txt for a Google-Extended group to decide one thing: whether that page may be used to train Gemini models and to ground Gemini responses served through Google’s developer APIs.

That is the whole scope. Google-Extended does not decide whether the page is indexed, how it ranks, whether it appears in AI Overviews or AI Mode, or whether its snippet is shown. Those are all Search decisions, governed by Googlebot and by the noindex, nosnippet and max-snippet directives.

The AI visibility check lists Google-Extended in the training group for this reason, and lists Googlebot separately in the search-index group, where blocking it removes the page from Google entirely.

The rule

Opt out of Gemini training, stay in Search

User-agent: Google-Extended
Disallow: /

# Googlebot keeps crawling and indexing as before
User-agent: *
Allow: /
Disallow: /admin/

Opt out for part of the site

User-agent: Google-Extended
Allow: /
Disallow: /research/
Disallow: /members/

Stay opted in, explicitly

User-agent: Google-Extended
Allow: /

With no Google-Extended group at all, the wildcard group applies, so a site whose * group allows everything is opted in by default. Writing the group down is mostly documentation — useful when several people edit the file.

What people expect it to do, and what actually does that

Most of the questions about Google-Extended are really questions about AI Overviews. Here is the mapping.

• “I don’t want my page summarised in AI Overviews.” Google-Extended does nothing here. AI Overviews use the Search index. The control is <meta name="robots" content="nosnippet">, which withholds the page’s text from every result feature, snippets included; or max-snippet:N to cap how much text may be used. Both apply to the whole of Search, so weigh the cost: a page with no snippet is a page with a weaker result.
• “I want to leave Google entirely.” Block Googlebot, or noindex. Not Google-Extended.
• “I want to stop Gemini being trained on my writing.” This is what Google-Extended is for. Disallow it.
• “I want Gemini to know about my product.” Allow Google-Extended, keep Googlebot allowed, and add structured data that says what the page is. Being in the training set is a weak, slow effect; being clearly labelled in the index Google actually answers from is the one that matters.

The controls that do affect AI Overviews

Because this is the question people actually arrive with, here are the three directives that govern how much of a page Google may show in any Search feature, AI Overviews and AI Mode included. All three go in the page, not in robots.txt.

<!-- Withhold all text from snippets and AI features; the page still ranks, with a bare title -->
<meta name="robots" content="nosnippet">

<!-- Allow at most 100 characters to be quoted -->
<meta name="robots" content="max-snippet:100">

<!-- Lift the cap so a whole answer can be quoted (what this site does) -->
<meta name="robots" content="max-snippet:-1, max-image-preview:large">

Google also supports a data-nosnippet attribute on individual elements, for withholding one paragraph rather than the page. Each of these is a trade: text Google may not quote is text that cannot appear in a snippet either, and a result with no snippet gets fewer clicks. Most sites that want to be cited in AI answers should do the opposite — lift the cap — and use Google-Extended only for the training decision it was built for.

How it interacts with other Google tokens

Google runs several crawlers with their own tokens — Googlebot, Googlebot-Image, AdsBot-Google, Google-InspectionTool and so on — and the usual rule applies: each uses the most specific group that names it. Google-Extended is checked in addition to, not instead of, the Googlebot rules. A page Googlebot cannot fetch is never a candidate for Gemini training regardless of the Google-Extended group, because there is nothing to train on.

One consequence: you cannot use Google-Extended to allow Gemini training on a page that is noindex. Google will not use content it has been asked not to index.

Where the mistakes are

Putting Googlebot in the training block

# This removes the site from Google Search. Do not do this.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Googlebot
Disallow: /

The first three lines are a training opt-out. The fourth deletes you from Google. Block lists copied from forums contain this more often than you would think, usually because someone added “Google” for completeness.

Expecting a log entry

Site owners sometimes add the rule, see no “Google-Extended” requests, and conclude it is being ignored. There are never any such requests. The evidence that it is honoured is Google’s documentation, not your access log.

Assuming the CDN’s managed robots.txt covered it

Cloudflare’s managed robots.txt adds training-crawler rules and a Content-Signal line, and its list does include Google-Extended in some configurations. Check the live file rather than assuming; the check reports the token as the internet sees it.

Content-Signal and Google

The Content-Signal convention lets a robots.txt state, in one line, whether the site consents to search, ai-input (being quoted in answers) and ai-train. Google has not said it reads it; Google-Extended remains the control Google documents. Adding a Content-Signal line does no harm and states a policy for crawlers that do read it. It is not a substitute for the group above.

Check it worked

Re-run the check. Google-Extended reads “Blocked (fine)” if you opted out, and Googlebot must read “Allowed” regardless — if it does not, fix that first, because it is the only rule on the page that affects Google Search. The equivalent training tokens for the other providers are covered in the GPTBot guide and the ClaudeBot guide.