Bryan Needs a Title for His Home Page
The challenges of delivering standards based work
I configured my server by the book. Ten AI models still couldn’t read my documents.
I set up a technical documentation bundle: eight markdown files, a PDF, a diagram source, and an llms.txt manifest with absolute URLs, token estimates, and a reading order. Correct MIME types. Validated headers. Clean UTF-8. Every URL resolves.
Then I asked ten commercial AI models, including frontier-class, to fetch the manifest and read the documents. Same server, same files, same day and none of them succeeded cleanly. Believe it or not, this was a step up from having them ingest from an HTML landing page with direct file links!
But, two couldn’t reach the server at all (blocked by their own infrastructure, not mine, although they both blamed my server). Two rejected every markdown file because their fetchers don’t accept text/markdown. The correct configuration makes content less accessible than serving the wrong type.
Four silently truncated the largest document. One stopped at 32 KB (13%). Another at 120K characters (49%). None knew they had partial copies with one exception: Claude Opus 5.1 (High) told me its parser only retained the first and last thirds of my document and asked me to resubmit. That’s the behavior I want to see but doesn’t help much in an unattended mode.
Four served cached versions despite headers requiring revalidation. Three models returned three different stale versions of the same file on the same day. Every one blamed my server. There was definitely a trend to blame me or my infrastructure when things failed and I’m not pleased with that. Either the models don’t have full visibility to their failure modes or they’re just hallucinating. The latter is probably the more dangerous though.
One decoded UTF-8 as Latin-1. Every diacritic destroyed. Not critical but still irritating. I’ve fixed that on my server to enforce decoding in the future.
One fabricated a finding… reported a file that doesn’t exist, cited footers that don’t reference it, and presented invented observations alongside real ones at identical confidence. This set off a scramble to figure out if a stray error had gotten into the text somehow, but it ended up being a confabulation based on a phrasing expanded to imagined content.
The dangerous failures were silent. Truncation, caching, encoding corruption, and fabrication produced no errors. The model answers confidently from whatever it retrieved, and nothing in the output tells you retrieval failed.
The industry sells “give it a URL” as a core capability. For most models on most documents, it isn’t as reliable as opening the link in a browser. And you can’t tell from inside the conversation unless you’re actively probing secondary to whatever it is you’re actually trying to get done.
If you’re building an llms.txt: the format works. The implementations don’t. Plan for rejected MIME types, ignored headers, silent truncation, and stale caches. Verify independently and across models.
I set up a technical documentation bundle: eight markdown files, a PDF, a diagram source, and an llms.txt manifest with absolute URLs, token estimates, and a reading order. Correct MIME types. Validated headers. Clean UTF-8. Every URL resolves.
Then I asked ten commercial AI models, including frontier-class, to fetch the manifest and read the documents. Same server, same files, same day and none of them succeeded cleanly. Believe it or not, this was a step up from having them ingest from an HTML landing page with direct file links!
But, two couldn’t reach the server at all (blocked by their own infrastructure, not mine, although they both blamed my server). Two rejected every markdown file because their fetchers don’t accept text/markdown. The correct configuration makes content less accessible than serving the wrong type.
Four silently truncated the largest document. One stopped at 32 KB (13%). Another at 120K characters (49%). None knew they had partial copies with one exception: Claude Opus 5.1 (High) told me its parser only retained the first and last thirds of my document and asked me to resubmit. That’s the behavior I want to see but doesn’t help much in an unattended mode.
Four served cached versions despite headers requiring revalidation. Three models returned three different stale versions of the same file on the same day. Every one blamed my server. There was definitely a trend to blame me or my infrastructure when things failed and I’m not pleased with that. Either the models don’t have full visibility to their failure modes or they’re just hallucinating. The latter is probably the more dangerous though.
One decoded UTF-8 as Latin-1. Every diacritic destroyed. Not critical but still irritating. I’ve fixed that on my server to enforce decoding in the future.
One fabricated a finding… reported a file that doesn’t exist, cited footers that don’t reference it, and presented invented observations alongside real ones at identical confidence. This set off a scramble to figure out if a stray error had gotten into the text somehow, but it ended up being a confabulation based on a phrasing expanded to imagined content.
The dangerous failures were silent. Truncation, caching, encoding corruption, and fabrication produced no errors. The model answers confidently from whatever it retrieved, and nothing in the output tells you retrieval failed.
The industry sells “give it a URL” as a core capability. For most models on most documents, it isn’t as reliable as opening the link in a browser. And you can’t tell from inside the conversation unless you’re actively probing secondary to whatever it is you’re actually trying to get done.
If you’re building an llms.txt: the format works. The implementations don’t. Plan for rejected MIME types, ignored headers, silent truncation, and stale caches. Verify independently and across models.