Somebody on your board, or in your engineering channel, has told you to add an llms.txt file so that AI assistants will start citing your product. Before it goes on the roadmap: no major AI system currently reads it, and Google has said so publicly. There is a real version of this problem worth solving, and it is not a text file.
What llms.txt claims to be
It is a proposed file that sits at the root of your domain, in the same place as robots.txt, listing the pages you would most like a large language model (LLM) to read, laid out in clean markdown. The pitch is that AI assistants would fetch it, understand your product from it, and cite you more often as a result.
That is a proposal, not a standard. No search engine or assistant vendor has committed to reading it, and it is worth holding that distinction clearly, because the amount of confident advice about llms.txt bears no relationship to the amount of evidence behind it.
Google's actual position
John Mueller of Google described the file as "purely speculative for now" in a comment reported on 2 June 2026, adding that the file has existed for years and that none of the AI systems use it (sourced from Search Engine Journal). Google has separately pointed to WebMCP as the direction it prefers, an approach aimed at letting AI agents perform tasks on a site rather than read a curated list of pages.
Google's own documentation backs this up twice, once by omission and once by statement. Its guidance on AI features does not mention llms.txt anywhere, and it says plainly that "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary" (sourced from Google Search Central).
None of that means llms.txt will never be adopted. It means shipping one today buys you nothing you can measure, and there is currently no published evidence that adding or changing one produces a single citation.
The problem that is actually real
While everyone argues about a file nobody reads, a much more common failure is quietly costing software companies visibility: they are blocking the crawlers they want quoting them, and usually by accident.
It happens two ways. Somebody adds a blanket AI bot block to robots.txt during a scraping scare, and nobody revisits it once the panic passes. Or a content delivery network or web application firewall has a "block AI bots" toggle that got switched on eighteen months ago, which does not appear in robots.txt at all, so nobody finds it by reading the file.
The distinctions that matter
The crawlers are not interchangeable, and blocking one does not do what most people assume it does.
Google documents Google-Extended as "a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models", and states that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (sourced from Google Search Central). So blocking it is a training decision, not a search decision.
OpenAI documents three separate agents, each controlled independently (sourced from OpenAI):
GPTBot, "used to crawl content that may be used in training our generative AI foundation models".
OAI-SearchBot, "used to surface websites in search results in ChatGPT's search features".
ChatGPT-User, used when somebody asks ChatGPT a question and it visits a page in response.
The practical consequence is worth reading twice. Blocking GPTBot keeps your content out of training while leaving you visible in ChatGPT's search. Blocking OAI-SearchBot takes you out of the results your buyers are actually looking at. Plenty of teams mean to do the first and do the second, and never connect it to the traffic that quietly stopped arriving.
The fifteen-minute check
Open yourdomain.com/robots.txt and read it end to end. Look for GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, PerplexityBot, ClaudeBot, and any blanket disallow that catches everything.
Separate the two decisions before you change anything. Whether your content trains a model, and whether your product is findable in an assistant's search, are not the same question and do not have to have the same answer.
Check your content delivery network or firewall for a bot management rule, because a block set there will never show up in robots.txt.
Confirm your staging protections have not leaked onto production, which is a surprisingly common cause of a site being invisible to everything.
Make sure your public documentation and pricing pages are genuinely crawlable, since those are the pages that actually get quoted back to a buyer.
That is the whole exercise. No new file, no roadmap ticket, no vendor.
What to do with the roadmap slot instead
If what you want is the outcome llms.txt promises, being the source an assistant reaches for when somebody asks about your category, the levers are unglamorous and familiar: clear public documentation, a pricing page that answers questions rather than deferring them to a form, and third-party coverage in the places buyers already trust. None of that is an engineering ticket. It is the SEO and content work, which is slower and duller than shipping a file, and that is exactly why most of your competitors have not done it either.
That is the same groundwork as any proper marketing audit, which is where I would start rather than with a file. And if you want the wider picture of how AI search is reshaping organic traffic generally, that is its own conversation.
Adding llms.txt costs you an afternoon and buys nothing measurable today. Checking whether you are blocking OAI-SearchBot costs you fifteen minutes and might explain why you are invisible. Which of those is on your sprint?