Allowing and Directing AI Crawlers: robots.txt and llms.txt for Your Therapy Practice
AI search engines will only read your site if you allow their crawlers. This article explains what GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Grok do, how to allow them via robots.txt, and how llms.txt can improve how your content is cited.
Dit artikel hoort bij: Becoming Findable as a Therapist: The Complete SEO Handbook
- +Therapists with a functional website who want to be visible in AI search engines
- +Practices that have not yet set up their robots.txt or llms.txt
- +Anyone who wants to understand which crawler does what, without leaving the technical aspects to a developer
- -Those who want to protect their content from AI training (then consciously choose the opposite route)
- -Therapists without their own website (you do not have robots.txt control over a Therapieland profile)
Many websites block AI crawlers without realizing it. A standard WordPress installation, a Squarespace theme, or an old robots.txt from 2022 can simply exclude ChatGPT, Claude, and Perplexity. The result: your site still exists for classic Google, but you are never cited in an AI answer. This article shows you how to check and fix that.
Which Crawler Does What
Not every AI bot does the same thing. Some crawl for training (your content is incorporated into a future model), while others crawl live for search answers. You usually want to allow both, but it's good to know the difference.
OpenAI (ChatGPT)
GPTBot crawls for model training. OAI-SearchBot crawls live when someone in ChatGPT asks a question that requires web search. ChatGPT-User crawls when a user explicitly shares a URL or requests a visit. For visibility in ChatGPT answers, OAI-SearchBot is the most important.
Anthropic (Claude)
ClaudeBot crawls for model training. anthropic-ai is the broader bot category. Claude-SearchBot and Claude-User crawl live for web search and direct URL requests. Since 2025, Claude actively uses web search for relevant queries.
Perplexity
PerplexityBot crawls for both indexing and live answers. Perplexity-User crawls when a user clicks something. Perplexity is the most transparent of all engines regarding which sources they use.
Googlebot performs classic indexing and also powers AI Overviews. Google-Extended is the separate opt-in for use in Bard, Gemini, and future AI products. If you want to be found in Gemini, explicitly allow Google-Extended.
xAI (Grok)
Grok is different from the rest: it is intertwined with X (Twitter) and uses Live Search to retrieve the current web for every query. The crawler is called Grok (some sources also mention GrokBot). For therapy practices, this means two things: explicitly allow Grok in your robots.txt and ensure your content reflects freshness (recent date, regular updates). An active and honest X profile also helps, as Grok weighs social signals from X more heavily than other engines.
Step 1: Check Your Current robots.txt
In your browser, type: yourdomain.com/robots.txt. If you get a 404 (not found), your site is not blocking anything and everyone is allowed. If you get a file with Disallow rules, read them carefully. A rule like 'User-agent: GPTBot' followed by 'Disallow: /' completely blocks ChatGPT. Look for rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and anthropic-ai.
Step 2: Explicitly Allow AI Crawlers
Add the following block at the top of your robots.txt (replace or add to, but do not leave conflicting Disallow rules):
User-agent: GPTBot - Allow: / | User-agent: OAI-SearchBot - Allow: / | User-agent: ChatGPT-User - Allow: / | User-agent: ClaudeBot - Allow: / | User-agent: anthropic-ai - Allow: / | User-agent: PerplexityBot - Allow: / | User-agent: Google-Extended - Allow: / | User-agent: Grok - Allow: / | User-agent: GrokBot - Allow: /. Conclude with a Sitemap rule that points to your sitemap.xml.
A note: these allow rules are a polite request, not a technical barrier. Those with malicious intent can ignore them. The real goal is for the official crawlers from OpenAI, Anthropic, Perplexity, Google, and xAI to see that they are welcome and do not drop out beforehand.
Step 3: Add an llms.txt
llms.txt is a proposed standard format (see llmstxt.org) that provides a readable map of your site to AI models. Unlike sitemap.xml (pure URLs for crawlers), llms.txt contains short descriptions per page, grouped by theme. An AI model can read it in half a second and immediately knows where to look for what.
Build llms.txt as follows: H1 with your practice name, blockquote with a one-liner about what you do, a short introductory paragraph, then H2 sections (for example, Services, Approach, FAQs, Knowledge Base) with a bulleted list of markdown links with a short explanation for each item. Place the file at yourdomain.com/llms.txt.
Step 4: Test if You Are Truly Discoverable
Wait a week (crawlers need time to recognize the new rules) and then test a search query that your clients would also ask in Perplexity, Claude, ChatGPT, and Grok (on X). Are you cited? If not, first check if the crawlers mentioned above are blocked elsewhere on your site (CDN settings, Cloudflare firewall, Vercel configuration can still block despite a correct robots.txt).
When You Explicitly DO NOT Want AI to Use Your Content
An honest caveat: some therapists fundamentally do not want their work to be used in AI training. For example, because they value human-to-human contact, or because they protect paid content. In that case, you block GPTBot, ClaudeBot, Google-Extended, and Grok (training bots) but allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot. This way, you remain discoverable in AI answers, but your content is not integrated into future models. Note: Grok primarily retrieves content live, so a complete block also means no citations in Grok answers.
Further Reading
This article is part of the 'Becoming Discoverable as a Therapist' pillar. For the substantive aspect of discoverability, read E-E-A-T for Healthcare Content. If you then want to measure if it works, look at Measuring AI Citations. And make sure you are not accidentally penalized by reading how to avoid GEO errors.
Wil je zien welke zoekwoorden cli毛nten in jouw stad gebruiken?
Het Cli毛nt Kompas geeft je in 5 minuten een live overzicht van zoekvolumes en drukte op zoektermen voor jouw praktijk.
Start het Cli毛nt KompasGerelateerde pagina's
Verder lezen in deze pilaar
Veelgestelde vragen
Do I have to edit my robots.txt myself or does my web developer do that?
On WordPress, you can usually do it yourself via an SEO plugin (Yoast, Rank Math). On Squarespace and Wix, robots.txt can be partially configured via settings. For a custom site or if you are in doubt: ask your web developer to add the three blocks above. It is a one-minute change.
Is an llms.txt mandatory?
No. It is a proposed standard format that more and more AI models are using, but it is not an official W3C standard. We see that sites with a well-structured llms.txt are measurably cited more often, so the effort is low and the return positive.
What happens if I do nothing?
If your robots.txt currently does not contain Disallow rules for AI crawlers, you are probably already accessible. But without llms.txt and without explicit Allow rules, you miss the signal that your content is welcome to be cited. No drama in the short term, a missed opportunity in the medium term.
Can I change my mind later?
Yes. robots.txt and llms.txt are live editable. Add a Disallow and most crawlers will respect it within a week. Note: content that has already been incorporated into an AI model will remain there until the model is retrained. That is why it is better to make the choice consciously now.
Why is Grok different from ChatGPT, Claude, or Perplexity?
Grok from xAI is intertwined with X (Twitter) and uses Live Search to retrieve the current web for each query. Two things are therefore extra important: the freshness of your content (recent date, regular updates) and your presence on X. For therapy practices that are not active on X, the benefit remains limited; for those who do share calmly and honestly on X, Grok can become a growing channel.