Contents
On 27/07/2026, I set up a file called llms.txt for marketing365.vn, along with a full version, llms-full.txt. The idea was simple: instead of forcing language models to read every HTML page along with the interface, ads, and menus, I packaged the clean content into a text file with a table of contents, descriptions, and outlines for each article. As of today, the full file weighs 183,880 bytes, contains 262 articles across 12 sections, and is regenerated every week by an automated command at 5 a.m. every Monday.
Today, 14/08/2026, I opened the server logs to count how many AI bots had downloaded it. The number was 0. Not few. None at all.
Counting from the server’s own logs, not guessing
There was a trap here that I almost fell into. The server is running 13 different sites, and the shared nginx log does not record which domain received a request, so reading the shared log would give the combined total for all 13 sites. To count marketing365.vn separately, I had to read the log for the exact container running this site. I merged all logs, including compressed rotated copies, and ended up with 4,138,335 lines spanning from 06/06/2026 to today.
From that pile, filtering by the user-agent strings of AI crawlers gave me 78,421 downloads, 7,310 MB of data, and 34,758 different paths touched. They are very active readers. Amazonbot accounted for 30,628 requests, GPTBot 16,317, meta-externalagent 12,768, ClaudeBot 12,333, PerplexityBot 3,909, and the other nine crawlers together 2,466.
Then I filtered specifically for the two paths /llms.txt and /llms-full.txt. In total, there were exactly 100 opens of these two files from the day they were created until now. Of those, 59 were my own command-line checks, 36 came from a browser, 2 from a web-scanning service, and the rest were a few miscellaneous hits. None were from AI bots.
My first count came back positive, and it was wrong
On the first run, I wrote the filter as a substring match: any path containing llms would count. The result showed GPTBot 10 times, Amazonbot 6, meta 6, ClaudeBot 4, and Perplexity 2. It looked plausible, and I almost wrote the article as “the file is being read, but only a little.”
Fortunately, that number conflicted with another count I was running in parallel. When I opened it up and checked, it turned out the string llms matched two ordinary articles:
/4-xu-huong-seo-moi-nhat-tin-hieu-ai-llmstxt-nhan-gia-tot/
/en/openai-broadcom-jalapeno-chip-llms/These two articles have llms in their path names, and bots read them like any other article. After correcting it to an exact-path match — counting only when the path is exactly /llms.txt or /llms-full.txt — the result dropped to 0, matching the parallel count. This is exactly the kind of error this whole series is documenting: the four times I guessed the wrong source of layout shifts also started with a loose measurement like this.
No one was told this file exists
Now that we know the answer is 0, the next question is why. I checked the three places where a bot could have learned about the file:
- Downloading
robots.txtto read it: no line mentioned llms. Counted 0 occurrences. - Downloading the homepage to look for a link tag: also 0 occurrences in the page source.
- Reading the beginning of the server response: it disclosed the sitemap, the news source, and the API directory — but not llms.
In other words, I made a file for machines to read and then hid it somewhere no one knew how to reach. Bots do not guess paths; they follow what is explicitly disclosed.
This check is more reliable than speculation, because on the same site I already have three other sources to compare against. The sitemap is disclosed both in robots.txt and in the response headers: AI bots made 425 requests. The source for each article is linked directly in the page: 652 requests. The API directory is disclosed only in the response headers, with no in-page link: 0 requests. And llms.txt, disclosed nowhere: 0 requests. Four sources, four disclosure levels, and the number of requests followed the disclosure level exactly.
The most obvious fix only reaches one quarter
The first instinct is to add a file declaration to robots.txt. Before doing that, I counted how many bots actually downloaded robots.txt to read it. This is where I was surprised.
Across the entire log period, the three heaviest crawlers — Amazonbot, GPTBot, and meta-externalagent — together accounted for 59,713 requests, or 76% of AI bot traffic, and never once downloaded robots.txt. The total number of robots.txt downloads by the AI bot group was 517, almost all of them from ClaudeBot (248), OAI-SearchBot (157), Bytespider (73), and PerplexityBot (37).
So the idea that “just put it in robots.txt and you’re done” is not true for this site. By download count, it only reaches about one quarter. If you want the three biggest crawlers to know the file exists, you have to disclose it where they are guaranteed to pass, meaning directly in the page — a link tag in the document head, or in the server response headers, or both.
One thing needs to be said plainly so I do not fool myself: this log starts on 06/06/2026, the day the server was rebuilt. If Amazonbot or GPTBot downloaded robots.txt before that and remembered it, I cannot see it. Their zero is only true within the period I measured, not as an eternal truth.
Cutting it to exactly 18 days for a fair comparison
Comparing 78,421 requests for the full period with 0 opens of llms.txt is a bit unfair, because the file only existed from 27/07. So I narrowed it to the exact period from 27/07 to 14/08, meaning the 18 days the file had already been on the server:
- AI bots made 37,435 requests, transferred 3,357 MB, and touched 18,905 different paths.
- Of those, 235 were robots.txt downloads and 186 were sitemap requests.
- Number of opens for llms.txt or llms-full.txt: 0.
They came into the house, walked through all 18,905 rooms, stopped to read the signboard 235 times, and yet no one opened the filing cabinet I had prepared for them, because nothing on the signboard mentioned that cabinet.
Three things I still cannot conclude
I do not know whether anyone will read it once it is disclosed. This article only proves one direction: if you do not disclose it, no one will read it. The reverse direction requires disclosing it and counting again after a few weeks. I deliberately did not change anything while writing this article, so the next measurement still has a baseline to compare against.
I do not know whether bots may read it through another address. Some services aggregate content through third parties, in which case the user-agent string does not carry any bot name. My counting method filters by user-agent string, so it would miss that type. For llms.txt, that does not affect the conclusion, because the total number of opens for the two files is only 100 and I checked each one individually.
I do not know whether the file is worth keeping. 183 KB regenerated every week takes space and an automated command. If it is properly disclosed and still gets 0 requests three months later, that is evidence to remove it, not a reason to make it bigger. The principle I followed when fixing 613 articles with broken links was the same: if you cannot measure the effect, do not rush to replicate it.
For anyone planning to create llms.txt for their own site, the short conclusion is this: the file is easy to make, the hard part is disclosure. And before trusting any guide that says “adding it to robots.txt is enough,” open your logs and count whether the biggest bots on your site even bother to download robots.txt. On this site, the answer is no.
