This is a Plain English Papers summary of a research paper called How Cross-Model Compatibility Lets Attackers Extract Proprietary LLM Reasoning Traces.
If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter.
The illusion of safety Major AI companies now show users their models' step-by-step reasoning as a feature.
OpenAI offers it through o1, Anthropic through extended thinking, Google through its reasoning-focused variants.
But this reasoning is a double-edged sword.
It's intellectually valuable to share, showing users why a model reached a conclusion.
But it's also intellectually valuable to steal.
Competitors want to understand how frontier models think.
Researchers want to study their reasoning patterns.
Attackers want to extract proprietary algorithms.
So the companies made a choice: hide the reasoning from users by encrypting it.
The idea sounds straightforward enough.
Return the reasoning to the user's device in an encrypted, unreadable form.
The user can't see it, competitors can't see it, but they can pass it back to the server in future requests if they need continuity with previous reasoning.
The server alone holds the decryption keys.
Problem solved.
Except it wasn't.
Researchers discovered that this encryption doesn't actually hide reasoning.
It just makes it look hidden.
The encrypted blocks are designed to work everywhere within a company's ecosystem, across different sessions and different models.
That universal compatibility is a feature for convenience.
But it's also an architectural vulnerability that anyone can exploit.
The architectural gamble To understand where this went wrong, you need to see how the system actually works.
When a user sends a request to a frontier model like GPT-4, the model internally generates a reasoning trace, the raw thought process behind its answer.
Instead of returning this reasoning in plaintext, the company encrypts it on the server before sending it to the client.
The encrypted block, which we can call a reasoning token, travels to the user's device alongside the final answer.
The user's device can't open it.
It's encrypted.
But the device can store it, and when the user makes a follow-up request that needs the previous reasoning, they send the encrypted token back to the server.
The server decrypts it, uses the reasoning to inform the next response, and hands back another encrypted token.
The security assumption embedded in this design seems reasonable: only the company's servers hold the decryption keys.
Tokens are cryptographic black boxes from the user's perspective.
But there's a hidden assumption underneath, one that silently shaped the architecture.
The assumption is that isolation between users matters less than convenience within the company's ecosystem.
So instead of using different encryption keys for different users or different security tiers, the system uses one logical encryption scheme across all models, all sessions, and all users within a provider.
Your encrypted reasoning tokens are cryptographically compatible with mine.
A token from ChatGPT is readable by GPT-3.5.
A token from Claude Instant is readable by Claude
3.
This interchangeability is intentional.
It simplifies the system.
It lets reasoning flow freely within the family of models.
It feels secure because the tokens look like random garbage.
And that's where the vulnerability lives.
Not in broken encryption, but in this choice to make encrypted blocks universally interchangeable.
The fatal flaw Here's the core problem: if an encrypted reasoning token from GPT-4 is structurally compatible with GPT-3.5, then someone with access to both can perform a decryption attack.
They extract the encrypted token from GPT-4, inject it into GPT-3.5, and ask GPT-3.5 to output its contents.
The weaker model, having decrypted it as part of normal processing, will output it in plaintext.
The stronger model never gets jailbroken directly.
The weaker model does it for th