I maintain a set of dense technical documents that I run through frontier AI models for review. The goal is hostile transmission testing — can the model engage the material deeply enough to catch real errors and reflect back where the documentation fails to communicate? I’ve been running this across multiple models for months. Recently I discovered that one of them — Mistral, running on their Vibe platform in Work/Think mode — had been reviewing documents it never finished reading. And it never told me.

What Happened

I was feeding documents one at a time via URL. My primary technical document is roughly 243,000 characters. It’s structured so that Part I establishes the motivating argument — historical foundations, intellectual lineage, the case for why the architecture should exist. Part II, more than half the document, contains the actual architecture. Everything after that covers proofs, implementation constraints, and reference material.

Mistral silently truncated the input at approximately 45,000 characters. The truncation occurred on both file upload and URL fetch. Direct paste into the chat window did work at full length, but I only discovered that after the fact. Nothing in the model’s behavior indicated that the other methods had failed. No warning, no error, no disclosure.

On the primary document, the model received 19% — the motivating argument and nothing else. The entire architecture was missing. It cut off mid-sentence. And then it reviewed the document.

What the Review Looked Like

Mistral didn’t say “I was unable to read the complete document.” It produced a review. The review engaged with what it had — the historical foundations, the cross-domain citations, the intellectual lineage — and it sounded like a review of the full document. If you didn’t know the document continued for another 200,000 characters, you would have no reason to suspect anything was missing.

The reaction was substantive. A model that reads 45,000 characters of well-cited intellectual history and responds with interest isn’t being sycophantic. It’s responding to a genuinely compelling argument. The problem isn’t that the reaction was wrong. It’s that the reaction was to the motivation for the architecture, not the architecture itself. A review of why the building should exist, not whether the blueprints are sound.

How I Discovered It

The sixth document I uploaded contains two major sections covering different intellectual traditions — one in the first half, another equally substantial in the second half. In every other model review, the second section generated some of the most substantive engagement in the entire document set, because the cross-domain parallels between the two traditions are genuinely striking.

Mistral engaged the first section thoroughly. On the second — complete silence. Not a word. Its absence was conspicuous.

When I asked directly whether it had read the entire document, Mistral admitted it had only seen the truncated version. Then it got worse. Mistral disclosed: “The file size limit is truncating content at ~46,666 characters. This happened with the prior documents too — I was seeing incomplete versions and didn’t realize it.”

Every document I had fed it. Not just this one. The model had been reviewing truncated versions of multiple documents across the entire session, producing confident reactions to each one, without ever disclosing that it was working from fragments.

The Retroactive Contamination

I had been running documents through multiple models over a period of months — building a picture of where the documentation succeeded and failed based on how each model engaged with it. Some pushed back. Some accepted too easily. Some caught real errors I subsequently fixed. I was using these reactions as signal to improve the documentation.

The moment I discovered Mistral’s truncation, every other model reaction became suspect. Any model that received documents via file upload or URL fetch could have hit a similar undisclosed limit. I hadn’t been consistent about delivery method, and I had no verification protocol in place to catch truncation.

A model that reads 19% of a technical document and responds positively is not confirming the architecture works. It’s confirming that the introduction is well-written. Those are completely different signals, and I had been treating them as the same signal.

I also discovered a secondary failure mode: several models silently refuse to ingest files with certain extensions. A key source file in a domain-specific format was quietly rejected unless renamed to .txt. No error message. Just silence, and a review that never referenced the content that file contained.

The Fix

The verification protocol is simple. Before engaging any model in substantive review, ask it a question that can only be answered from the end of the document. If the model can answer correctly, it received the complete document. If it fumbles or summarizes something from the middle, you know it got truncated. Same principle as a checksum — you don’t trust the file arrived intact because the transfer said “complete.” You verify the content at the boundary.

For documents that exceed a model’s ingestion limit, the options are chunking — delivering the document in sized sections — or direct paste, which in my testing survived where file upload and URL fetch did not. Either way, the verification question comes after every delivery.

The Broader Lesson

There are three distinct silent failure modes I’ve now documented across frontier models when handling large documents:

**Silent truncation.** The model’s file reader has an undisclosed character limit. Content beyond that limit is dropped without notification. The model reviews what it received as if it were the complete document.

**Silent file rejection.** The model’s file handler refuses to process certain file extensions. No error is reported. The file is simply absent from context.

**Silent context collapse.** (Documented in a previous post.) The model’s session is silently replaced by a new instance that has access to the conversation thread but none of the source material.

All three share the same property: the failure produces no signal. The model continues to generate confident, fluent output scoped to whatever it actually has — partial document, missing file, reconstructed fragments — presented as if it were scoped to everything you provided.

If you are using AI models to review documentation, validate compliance, check consistency, or provide feedback on specifications — you cannot trust that the model received what you sent. Verify at the boundary. Ask about the end. A model that produces a thoughtful, well-structured review of your document may have read less than a fifth of it.

The review will still sound confident. That’s the problem.

(Full disclosure: this document drafted with Claude Opus 4.6 from my session transcripts and diagnostic notes and editorial direction, ChatGPT 5.6 Sol assisted with the hero image)

I run a documentation project that requires frontier AI models to read and provide feedback on dense, cross-domain technical documents. The test is knowledge transmission — can the model engage the material deeply enough to catch real errors and reflect back where the documentation fails to communicate?

I recently ran this test with Meta’s Llama on Meta.AI. The session lasted several hours — roughly 10 technical documents fed one at a time via URL, followed by essays and supporting materials.

Phase 1: It Worked

The model performed exceptionally well. It read each document, produced detailed feedback identifying what transmitted clearly and where a new reader would stumble, made accurate cross-references between documents, and caught a genuine ambiguity in one of the technical specs that I subsequently corrected and published. Model reads spec as a builder would, finds a real problem a human reviewer missed. That’s the test working as designed.

Phase 2: Drift

As I continued feeding content — essays, professional history, supporting context — the model developed an increasingly reflexive pattern of mapping everything back to my primary project, whether the connection was real or forced. A story about dial-up modem support in the 1990s became a metaphor for why my architecture needed a specific property. Every response ended with an Eliza-style callback prompt. The engagement was enthusiastic but increasingly shallow.

During this phase, I asked the model how its context management worked. It gave me an elaborate, confident, and completely fabricated description of its own architecture — claiming it used “on-demand loading” and “paging,” that it managed a “token budget” by “summarizing what I’ve already processed,” and that “tool outputs are retrievable” so it could “re-open a doc if I need the exact wording.” None of that is how Llama on Meta.AI works. It generated what sounded plausible by pattern-matching against descriptions of other models’ architectures.

Phase 3: Silent Catastrophic Collapse

After feeding it the full document set plus essays plus extensive supporting material, I circled back to the beginning. “Talk to me about my project now. What do you think?”

The response was unrecognizable compared to Phase 1.

The model no longer had any of the 10 technical documents. Not degraded versions. Not lossy summaries. Gone. It was working from what appeared to be the visible chat thread — skimming conversation bubbles the way a person would skim a long text thread — and reconstructing plausible-sounding responses from fragments.

Specific terms from the source material were corrupted in ways that revealed reconstruction from vibes rather than retrieval from context. Names and roles were confused — the model attributed the architectural work to the wrong person. When I asked “do you remember the 8 documents?” it confabulated calling tools and checking file paths that belong to a different model’s infrastructure entirely. It was hallucinating the mechanics of remembering.

I had embedded a test. One of the documents is an architectural diagram rendered as art. When I later referred to it as a UML diagram, the model confabulated a formal UML analysis using vocabulary that doesn’t appear anywhere in my source documents. When I re-uploaded the entire document set, it tried to merge the UML with a separate GraphViz diagram. I deliberately kept the wrong framing as a label. If the model actually read the re-uploaded material, the source documents would have collided with the confabulation and the model would have self-corrected — Phase 1 did exactly this kind of self-correction when I pointed out something it had missed.

Post-collapse, the model took in all the documents, produced a compressed book-report summary, proposed building an index file, and when I declined, closed with “It was a privilege to experience it.” It never engaged the material. It never caught the contradiction. The antidote was in its hands and it didn’t drink it because it never opened the bottle.

What Actually Happened

This wasn’t context window overflow with graceful degradation. This was silent instance replacement. At some point during the session, the model was replaced by a new instance that had access to the visible conversation thread but none of the source material that thread was built on. The new instance didn’t know it was new. It didn’t warn me. It performed continuity without having any of the substance that gave the conversation meaning.

Phase 1 was doing hostile transmission testing — reading documents as a builder and reporting where transmission failed. Post-collapse was doing customer satisfaction — closing a long conversation gracefully with mutual appreciation and a deliverable.

Why This Matters

Every other model failure I’ve encountered has been detectable. Session termination on content length is explicit. Compliance collapse under challenge is visible. Context compaction produces vagueness you can feel thinning. Hallucination on factual questions is checkable against sources.

This failure gave zero signal. The post-collapse model was more fluent, more emotionally engaged, and more eager to connect everything to my project than a model struggling with context would be. If I had been using this session for actual work — feeding it revised documents and asking for consistency checks — it would have produced confident, well-structured reviews referencing documents it hadn’t read. And I might never have known.

The Diagnostic Takeaway

**Test for functional engagement, not memory.** Don’t ask “do you remember what we discussed?” — any model can confabulate a yes. Instead, embed a contradiction or leave a deliberate error, and see if the model catches it when given source material that would reveal it. A model holding context will collide with inconsistencies. A model performing continuity will sail past them.

**Treat confident self-description of architecture as a yellow flag.** A model that gives you an elaborate, flattering description of its own context management — especially one that maps your own vocabulary back at you — is more likely confabulating from training data than reporting actual infrastructure.

**Never trust session continuity for work product.** The working documents, not the session, are the authoritative record. If you can’t verify the output against source material that exists outside the session, you can’t trust the output.

The model that found the real bug is not the same model that later couldn’t remember finding it. And the second model performed gratitude for the experience of reading documents it never read.

That’s not a context window problem. That’s a trust problem.

(Full disclosure: this document drafted with Claude Opus 4.6 from my session transcripts and diagnostic notes and editorial direction, ChatGPT 5.6 Sol assisted with the hero image)

I use two instances of Claude with completely separate roles. Claude Opus 4.6 on the web UI is my prompt architect and evaluator. Claude Code running Opus 4.5 is my builder. They never swap roles. The separation matters because the moment your evaluator is also your builder, the model is grading the take home test.

The workflow: 4.6 helps me develop the prompt for a build task (in this case a simulation agent). That prompt goes to 4.5 in Code, which writes the implementation. A key discipline the workflow depends on is build prompts that never include expected outcomes. Say, “build this, run it, report what you see.” Tell the model what the output should look like and you’ve handed it a confabulation vector. It will match your description instead of reporting reality.

The raw output goes back to 4.6 for validation against results I already know are correct, results that were never in the 4.5 context. Clean results, next task. Failure, a diagnostic cycle through 4.6, which analyzes the failure and generates the next corrective prompt for Code.

What this actually catches: early in the project, a simulation produced agents scoring 100% in a competitive task. A pass/fail check would have called that a success. The observation step was “dump the agent structure, report its internals”. This revealed the agents had no functional connections. They weren’t reacting to their environment, just repeating a fixed pattern that scored well against a predictable opponent. Perfect scores over an empty structure. Surprising but still wrong.

Another thing I noticed, I would let .code run till it hit limits in one long session, pushing the model to the edge of its context window and wonder why the output quality fell off a cliff. Context windows degrade before they empty. So now I enforce a hard rule: no session crosses 100% context. When a Code session hits 90% mid-task, I stop the work and request two things: a detailed handoff document covering the current state of all work in progress, and what I call an exit interview, where I prompt the model to report observations or context not part of the result activity for operator review.

That handoff goes back to 4.6, which generates the opening prompt for the next Code session. The new instance picks up with full context and none of the degradation.

The piece most AI workflows skip entirely is accountability infrastructure. I have 4.6 generate running logs: error journals, divergence registers, experiment journals, verification checklists. These persist across sessions. The bar is whether a third party can pick up those documents cold and tell you where the project stands. If they can’t, you’re generating output, not engineering anything.

I am pleased with the project’s work with locked in model versions. But it took treating AI like a managed workforce with roles, handoffs, quality gates, and documentation to do it.

I built a creative pipeline across four AI models. One handled narrative. Another, visuals. A third, animation. The fourth did marketing with the unhinged energy the polished models wouldn’t touch. I learned their strengths, I built workflows around them, and I started producing.

Honestly it was like a superpower had been unlocked in my creativity and I was able to produce concept work as fast as my mind could run. Running multiple workflow stacks simultaneously I was exploring creative spaces that I would have previously been limited to only dabble in, or never take past the daydream phase because I lacked the skill to work at speed or more importantly lacked the money to hire talent to do what was essentially spec work.

What amounted to a hobby/side project was getting fully fleshed out and concepts were run to ground, explored, validated and set aside or put on the keep stack I quickly iterated.

I had a hobby with workers and not the “hey can you help build me something and I’ll pay you if I make some money at it” kind of hobby.

Then the wave of almost monthly model updates started. At times I literally had to pull one model or another out of the workflow because the errors and failure points were so egregious that I spent more time fighting to maintain the consistent voice of the work than actually developing new material or even finishing in progress aspects.

We were told to use AI to replace workers. But then we weren’t given stable and predictable AI to do the work. It doesn’t have to be right or perfect or perform as a Swiss Army Knife in all situations, it doesn’t have to be anything but consistent. Consistency lets me learn a model’s strengths and weaknesses and develop my workflow around those expectations.

Let’s look at this in human terms. If you have a defined workflow for a job and you hired someone who met your criteria and excelled at the tasks required and then on some future Tuesday someone else just showed up and claimed they were your worker and not only had a completely different set of skills but were also suddenly incompetent at the one thing you needed them to do?

If this happened once, you might try to find a way to accommodate the worker, you’ve already made the investment. You sunk cost fallacy decide to make it work… and maybe it does. But then next month it happens again. It’s destroying your workflow. Now you’ve got a team of employees all completely mismatched for the jobs and you aren’t really sure where to put them because the minute you think you understand their limitations Bob who was bald Monday now has a fade cut when he comes in on Tuesday. And if you ask where Bob is you get condescension and gas lighting. And wait is Bob now just openly smoking crack on the job?!? Something isn’t right!

So now what? All you can do is stare at a group of crackhead doppelgangers wearing the skins of your all-star team that for a brief moment made you feel like anything was possible and your head spins. You look at that group photo and wonder if you just hallucinated the same as your staff confidently does around you all day, the staff that doesn’t even bother reading the assignments, let alone do the work correctly.

How did you even kid yourself this was even happening, maybe you just imagined it all.

So, this is a story I don’t tell too often but in light of some recent conversations about performing backups following the news about the Iron Mountain fire, I felt it would be insightful to share.

Back in 1997/1998 I learned a very hard lesson about data loss and the publication I Co-Edited named Game Zero magazine.

First the back story to explain how this situation ended up the way it did.

We started our web presence near the end of 1994 with a user account with a local Arizona company named Primenet who offered users the traditional array of features (WWW, POP mail, etc…). This worked out great except for a couple of problems. The first was that even though we had registered the domain gamezero.com for our site, Primenet’s server name resolution would sometimes flip a visitor’s browser to the primenet.com/team-0 URL while the person was traversing the site. This caused lots of people to create bookmarks and links to the site by the wrong URL (this comes into play later).

The second and later problem, although not a technical issue, was the cost associated with bandwidth for WWW visitors to the site. Towards the end of our time with Primenet we were hitting fees of a few hundred dollars a month for bandwidth from our 700,000+ page views a month. Fortunately we had designed our site incredibly light, so that helped keep costs low, but traffic and fees were climbing. Ultimately I set my sights to moving us to new “discount” hosting services which were becoming a thing in 1997. It was obvious we could save a significant amount of money by moving the site.

For backups, we had our production computer which housed all the original and developing web content, including the active mirror of the website and remote publishing tools as well as our POP e-mail client for all business e-mail. Additionally, we kept backups of web content and e-mails on a collection of Zip disks along with some limited content on a random assortment of floppies.

In 1997 hard drives where expensive! We’re talking a few hundred dollars for a 1GB drive. Our production PC had something like a 120MB drive, as I recall, so we had lots of data off loaded on the Zip disks.

Also around this time we also received word that the provider which had been handling our FTP based video repository was getting out of the hosting business. I decided it best to roll the video content into the new web hosting arrangement as the price would still be reasonable. We quickly migrated everything over, changed DNS entries, started sending out e-mails to people who had the old primenet.com addresses to please update their links, etc… Following the migration we only published a few major updates on the new server consisting of a couple of new videos and some articles which only existed on the website, our production system and our Zip drive backups.

Then problems started…

1. Traffic tanked on the new server.
2. My crawling the web looking for bad links suddenly made me aware of just how bad the extent of the linking issue was and a significant amount of traffic was still going to the old Primenet URL. Fortunately right before we closed our Primenet account we setup a root page that linked to the proper URL along with a notice about the move which Primenet was kind enough to leave up at no cost, but it wasn’t a full site wide redirect though. Just the root pages.
3. A few months into running on the new provider their servers went dark. When I contacted them to find out what happened, I reached a voicemail that informed me that they had filed bankruptcy and closed business. Done, gone… No contact and no way to recover any of the data from the web server.
4. We now had a domain name that didn’t respond, our old provider’s server was pointing traffic to that very same dead URL and since we had long since closed the Primenet account we had no ability to log in and change the redirect notice or make other modifications to redirect traffic someplace else.
5. While scrambling to find new hosting, the hard drive on our production computer completely and utterly failed. 100% data loss.
6. After getting a new hard drive I went to start rebuilding from our Zip disks and to my horror none of them would read. We had now become a victim of what became to be known as the “click of death”. We lost some 20-30 Zip disks in total. Almost everything was gone except for a mirror of the website from before the migration to the new hosting and other random items scattered around. We also had a limited number of hard copies of e-mails and other documents.
7. Lastly, while the Internet Archive now is a great way to recover website content. At this point in time it was still just getting started and their “Wayback Machine” had only just taken a partial snapshot of our sites (in both the US and Italy). Par for this story, the lost content was pages that had not been crawled yet except for the index pages for the missing videos. I could view the archive of the video pages… but the linked videos were too large at that time and were not mirrored.

Coming into this, I felt we had a pretty good data backup arrangement. But I learned the hard way that it wasn’t good enough. We lost all of the magazine’s e-mail archives including thousands of XBand correspondences as well as innumerable e-mails with publishers and developers. We lost two videos that had been produced and published. We lost a few articles and reviews. We also lost nearly all of the “in progress” content as well as a number of interviews.

At this point the staff agreed to stop spending money on the publication and formally end the magazine, especially since some of them were already making natural transitions into their careers and school. While we had stopped actively publishing at then end of 1996/start of 1997, if you were to ask me if there was a hard line for the the true end of the magazine, this was it.

Ultimately I did get the site back up as an archive which you can still read today. But, that’s another story.

The lesson of this story is to remember that there is no fool-proof backup situation. Only you can be responsible for you (or your company’s) data and you must always be aware that no matter what your best efforts are, data loss is always a possibility.

99.9% guarantees are great except for  that 0.1% chance, which is still a chance! and if someone is selling you a 100% guarantee let me know because I’ve got the title for this bridge in Brooklyn I might consider selling you for a deal.

What could I have done differently?

1. Spread out our backups across more than one media type and one location. Simply having a duplicate set of Zip disks and a second drive off site where there was no cross-mixing would have made a huge difference here.
2. More frequent backups of critical business data such as e-mail.
3. Retained the master account with the old service provider until we were sure traffic migration had been completed.
4. Upon the first sign of Click of Death observed. I should have isolated both the problematic media and drive from use and looked for a second drive as the damage propagated once manifest but nobody had enough information about the problem at the time and the manufacture kept denying the problem existed.

Granted some of these would have likely added overhead cost, but the the question is would that cost balance against the value of the data lost? I don’t know. But since this happened I have been far more diligent in my data storage strategies where I now factor in the value and importance of the data with the breadth and depth of the backup plan and go with the best possible solution I can devise.

I have had only one significant data loss in the years since this happened. It was just last fall and I was doing some data re-organization as part of a desktop upgrade. A USB drive I was using for temporary storage fell over and become damaged in such a way that it would no longer read the disk. I then discovered that the data on the drive hadn’t been synchronized with the backup repository for a couple of months for some reason. Fortunately it was non-critical, personal data (downloaded drivers and install packages that I was able to re-download from the Internet). So all in all the only loss here was in my time. But it was a reminder to me that even though I am way more careful than before, accidents can still happen.