Elevator Talk: Is llms.txt Actually Doing Anything, or Just Sitting There?

September 4, 2026

Is one markdown file enough to feed every LLM what it needs to know about your brand? Brandon Stauffer posed that question in an internal Slack thread, and the answer came back fast and with receipts: crawl data from two clients, a note about what bots were actually built to parse, and a reframe of what llms.txt is even for.

The short version is that nobody on the team thinks a single file does the job, but the reasons are more interesting than the verdict. Some of it is technical — bots parse HTML, not markdown. Some of it is strategic — one file can't manufacture the consensus LLMs are actually looking for. And the crawl data ends up confirming something we published back in June: llms.txt was never built for the job most people are hiring it to do.

Here's how the conversation went.

Not a Magic Bullet, But Probably Worth Having

Alyssa FelixSearch Marketer

Did you read my blog?

Why llms.txt isn't a magic bullet for AI, but you should probably have one anyway

The short version, for anyone who hasn't:

It's a table of contents, not a pitch. The comparison to robots.txt is fair — the standard was inspired by it — but they run in opposite directions. Robots.txt tells crawlers what they can't touch. llms.txt points AI agents toward what's worth going to. And like robots.txt, nothing about it forces compliance. An agent has to choose to use the map.
The spec is deliberately minimal. A title is the only hard requirement. Past that, it's a short description of what your site or product does plus links to documentation, APIs, and other machine-readable resources an agent would actually need to take action.
It is not a place to stuff brand messaging. That's where most of the confusion comes from. Your website and the quality of the content on it still do that job.
It was built for agentic browsing, meaning AI systems that complete tasks rather than just answer questions. Chrome's Lighthouse documentation for agentic browsing calls it out as a forward-looking signal, but Google has not said it treats llms.txt as a ranking signal. The value is functional, not algorithmic.
So it's low-effort infrastructure, not a growth lever. Less like installing a new engine, more like labeling your filing cabinet. It doesn't change what's inside, but when an agent shows up needing to find something fast, you'll be glad it's there.

A File Can't Replace the Ecosystem

Alondra MeloSenior Search Marketer

I agree with Alyssa's blog on this being helpful but not a replacement for the broader GEO/SEO ecosystem. We can't guarantee that every LLM will crawl it the same way, or at all, so it's important to still focus on the other supporting components of an AI strategy.

A centralized markdown or LLM reference file is useful for clearly stating core facts: company description, products, audiences, differentiators, locations, pricing approach, FAQs, terminology, important URLs.

The actual website is still the most important foundation. Those same facts should exist on crawlable, authoritative pages rather than only inside the markdown file.

Structured data and schema reinforce entities, products, organization details, FAQs, authors, and locations.

Third-party corroboration matters because LLMs often rely on sources beyond the brand itself: reviews, directories, press, industry publications, Wikipedia and Wikidata where appropriate, partner sites, and other mentions.

And consistent information everywhere is probably more important than making an enormous markdown file. You want every source telling essentially the same story.

Bots Were Built to Parse HTML

Mike WithamHead of Search

Alondra and Alyssa are spot on, especially Alondra's last point about consistent info everywhere. LLM bots in particular do not care about the traditional guardrails and parameters designed to direct bots. They're going to consume anything and everything.

The theory behind llms.txt is sound. You give a crawler a simplified version of a page in a language that's easier to crawl with less noise, and the bot prefers that page and comes back to it. It gets compared to schema markup, which is also a different, easier format for a bot to parse.

The problem is that bots were designed to crawl and parse HTML, not markdown files. So they're not going to look for .md files, and when they do come across them, they'll often ignore them. The major difference between schema and markdown is that structured data is part of the HTML version of the page. The bot knows it's there and will easily find and use it.

We know this because of multiple tests we ran with clients where the .md files got hardly any traction and appeared nowhere in the logs. So even when a hot new idea has sound theory behind it, always test it.

If you were going to publish brand information in that simplified form, I'd recommend one .txt page with important brand info, similar to a robots.txt file. An instruction page in that format is normal and much more likely for a bot to find and crawl.

7 Visits and 4% of Crawls

Britni DillardSenior Search Strategist

I love Alyssa's article and agree it's not a magic solution but can be helpful. Alondra's point about having consistent information everywhere is key. If a brand wants to be presented accurately and consistently in LLMs, it needs consistent information across the entire web: schema, on-page content, third-party sources like Wikipedia, industry publications, and review sites, llms.txt, markdown files, about us pages, all of it.

To Mike's point about always testing a theory, here's what we've seen.

We have a client who listed a landing page inside their llms.txt to see how often that page would be crawled and visited if it was listed only there. It's received 7 visits in the last month. I don't know how many times it was crawled since I haven't seen log files, but 7 visits across all channels in a month is really low for how often this site gets visited and crawled in general.

Another client launched markdown versions of many pages on their site. Looking at the crawl percentage of those markdown files versus the HTML versions of the same pages, markdown crawls accounted for only 4% of verified AI bot crawls, with "verified" meaning the user agent matches the published IP ranges for that agent. 4%.

So while llms.txt and other .md files could be helpful in informing LLMs about a brand or website, they're not likely to be the solution that changes the game.

The Crawl Data Is What Alyssa Predicted

Taylee SnyderSearch Marketer

Britni's crawl numbers make total sense, and they line up with what Alyssa's article said back in June: this thing was built for agentic browsing. If llms.txt isn't meant for the bots powering AI search answers, but for agentic tools — coding assistants, task agents, API integrations — that only fetch it when they need it for a specific job, then that's closer to sitemap.xml behavior. Low steady traffic isn't failure, it's just how a reference file gets used rather than browsed.

So maybe the real test isn't "how often is it crawled," it's "when an agent is actually trying to do something with our site, does it find the file and use it?" That's a different job than AI visibility entirely. It's infrastructure, not content strategy, so it was never going to show up in GEO metrics the same way.

The Takeaway

If llms.txt is on your roadmap because a competitor shipped one, the honest answer from this thread is: publish it, keep it accurate, and stop expecting it to move visibility. The theory is reasonable — hand the crawler a cleaner version of your content and it should prefer that version — but the crawlers reading your site were built to parse HTML, and the numbers reflect that. A page listed only in llms.txt pulled 7 visits in a month. Markdown versions of real pages took 4% of verified AI bot crawls against their HTML counterparts. That's not a rollout problem you can optimize your way out of. It's what the format is currently worth.

The more useful reframe is that a markdown file is a single source saying something about you, and LLMs don't build confidence from a single source. They build it from agreement. The same facts — what you sell, who you sell to, where you operate, what makes you different — need to exist in crawlable on-page content, in schema that lives inside the HTML, and in third-party sources you don't control: reviews, directories, press, industry publications, partner sites. Consistency across all of those beats comprehensiveness in any one of them, which means the biggest lever usually isn't writing a longer file, it's fixing the places where your own sources contradict each other. If you're going to publish a simplified reference anyway, a .txt file in the robots.txt tradition is a more familiar object for a bot to find than a directory of .md files.

Which leads to the most useful thing in this thread: the crawl data didn't contradict the original argument, it confirmed it. Alyssa's June article said llms.txt was built for agentic browsing — AI systems that complete tasks rather than answer questions — and that its value is functional, not algorithmic. Two months of client testing produced exactly the numbers you'd expect if that's true. A file whose audience is agents fetching a reference on demand should show low, steady traffic, the same way nobody judges sitemap.xml by pageviews. That means the honest test was never "how often was it crawled," it's "when an agent tried to do something with our site, did it find the file and use it?" Decide which of those two jobs you're hiring the file for before you decide whether it's working, because grading infrastructure on a visibility scorecard will tell you to rip out something that's doing its job fine. If you want the fuller picture of where AI search leverage actually sits, How brands win in AI search is a recent episode of The Campaign on exactly that.

Three things worth doing this quarter: check your own logs for verified AI bot crawls on any markdown or llms.txt assets you've already shipped, rather than trusting anyone's benchmark including ours. Audit whether the core facts in that file also exist in on-page content and schema, since that's where bots are actually looking. And run the third-party pass — reviews, directories, press, Wikidata — to find where the web is telling a different story about you than your own site is.

Getting every source to agree on who a brand is, and proving it with crawl data instead of theory, is the core of our GEO/AEO work.

If you've shipped an llms.txt and have no idea whether it's doing anything, let's talk about testing it properly.