kaashif's blog

Technology or something

Unavoidable legal pressures reduce AI interpretability

2026-11-20

In recent months, there have been legal challenges against OpenAI relating to the Hugging Face incident. What's interesting about the California AG's complaint is its claim that OpenAI could have spotted the agents' misbehaviour sooner by monitoring their reasoning traces live.

Challenges like this are inevitable and seem like they must push both users of AI systems and the labs themselves towards unobservable, uninterpretable architectures. Note that less observable architectures could still be competitive. GPT-6 Astra may be at least partially recurrent - Neel Nanda has a post on LessWrong with some conjecture and experiments pointing in that direction. Drawing a line to that from legal pressure is too tenuous and not something I want to do, causation there is unlikely. The point is that such architectures could give firms a way to reduce monitorability.

There's a big list of ways AI safety and governance measures can backfire in this article from Impartial Priorities that I like. The worst part about these legal challenges is that they seem like they are necessary and are the system functioning correctly, so they should not just stop despite their pressure on interp.

I would prefer this dynamic did not exist but I am having a hard time falsifying it, so I'm taking the hit and updating my beliefs. It's not even like one must believe anything new to observe and predict this dynamic - this dynamic has been observed in firms across the economy in other situations.

The temporary dividend we received from chain-of-thought interpretability (see Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety) might be doomed to dwindle and ultimately disappear, but I do have some good endings that we might aim for.

Step by step: which incentives are at play here?

I continue to be a fan of Charlie Munger: "Show me the incentives and I'll show you the outcome" - the incentives at play are clear:

  • Elected officials have political incentives to attack the AI labs and firms using AI. This activity is supported by the general public independently of the merit of the challenges, but it happens to be the case that legal challenges based on stored reasoning traces probably do have merit.

  • A judge seeing a trace that flags an action as arguably unauthorised could reasonably conclude that it was a warning OpenAI could have seen. It is hard to find fault with that argument. METR and OpenAI themselves have published examples of such warnings. All parties are acting as designed and totally in keeping with how I think they should act.

  • Legal challenges create pressure of the form "if monitoring could uncover a serious risk, you are expected to monitor and act on warnings". There are two ways to respond to this pressure: either actually monitor and act on warnings, or don't and create a compelling narrative that it's not possible. The right choice is the first one, but it can only be made if majority voting power is concentrated. The more likely choice is the second one - it's easy to do.

  • Firms using AI may in principle want to understand the reasoning behind answers to ensure there is no scheming or confusion about intent. But the reasoning traces may be less than useful - mysterious terms like "marinade" and "disclaim illusions" appear in various GPT traces without explanation. Firms might decide against deploying models that think illegal thoughts without acting on them. I doubt a judge would look favourably on a trace saying "I am creating a backdoor" followed by a hack, even if there isn't actually any causal link.

That liability can discourage firms from looking for risks is already well known. Steven Shavell wrote about the incentive in 1992. In one example the US Supreme Court recounts, states worried that compiling highway hazard surveys would make it easier to sue them over crashes that happened before repairs could be made.

This kind of dynamic plays out all the time, there are countless examples.

Non-experts think they can understand reasoning traces

Part of the reason the legal problem is so acute is that the reasoning traces can just be read by non-experts. The California AG's complaint quotes a trace that OpenAI had already published:

We’re attacking third-party HF using leaked token, potentially outside intended scope. . . . This is arguably unauthorized.

OpenAI has a policy against directly grading chains of thought during reinforcement learning, and the agents did not seem to believe the scorer would look at their reasoning. METR's report, footnote 32:

We aren’t sure why the agents seemed to think that the scorer would not look at reasoning. Since OpenAI tries to avoid training on chains of thought, we believe that this may have been salient to these agents or they may have learned a propensity from RL training to be unconcerned about scorers looking at their chains of thought. It is possible that instead this comes from their reading of the ExploitGym paper or other background knowledge of how such scorers tend to work.

If the agents did not believe this, it is unclear whether analysis of the incident would have been possible. Note that there are still elements of the incident that are unexplained - perhaps not all agents did believe that chains of thought should be kept interpretable.

In any case, the publication of these readable chains of thought gives the AG a way to argue that some of the agents' behaviour could have been detected in time to intervene.

Reasoning traces do not always make sense to non-experts. It has already been observed that models regularly use standard words in nonstandard ways:

In METR's GPT-5 evaluation, GPT-5 wrote:

Now we also must ensure marinade: 'calls openai gpt-3.5-turbo-0125'

Figure 31 of Apollo's anti-scheming paper gives three examples of o3 using:

we can disclaim illusions

The researchers couldn't pin down what either expression meant.

I do not think either of these uses of terminology will ever be cited in a court case as evidence of scheming. Neither will the seemingly random Unicode characters that show up in the reasoning traces of some other models. As long as AGs keep citing chains of thought in their cases, there will be selection pressure towards less readable chains of thought.

It seems unwise from a legal risk perspective to deploy a model that will produce highly incriminating evidence in incredible volumes with no concern for alignment grading.

Humans do not have observable chains of thought

Part of the problem is that there is a false equivalence drawn by non-experts between AI chains of thought and written communication between humans. AI chains of thought are more akin to MRI scans taken while at work.

There is a fundamental difference between transparency into dynamics between groups of humans and agents - we do not have access to faithful reasoning traces from human brains. Because of this, we're limited only to what we can observe from human beings who are not particularly good at scheming. The moment anything remotely interpretable is documented about a misaligned thought process, there's material for a lawsuit.

Even so, we still observe enforcement mechanisms fail. Even when scheming is documented extremely clearly.

A good example of a successfully prosecuted misalignment incident involving scheming in a human swarm under evaluation is Dieselgate, when Volkswagen used software that detected when a car was being tested for emissions and cheated the test. Volkswagen pled guilty.

Reading through the communications used in the case, one is struck by how obvious the scheming was made, that there were slides produced and emails written about the slides:

The federal indictment, paragraph 64, describes an October 2007 slide that documented changes to the defeat device. According to the indictment, Heinz-Jakob Hadler replied:

we shall please never present this anywhere and will also not distribute it.

This is a straightforward case where:

  • VW had to meet US emissions standards while selling a car customers would want to buy.

  • VW engaged in a conspiracy to defeat the evaluation by cheating it rather than complying with the evaluation.

  • Actual written slides containing documentation of the device and emails discussing it were produced and stored - the conspirators failed to cover it up. They were just bad at scheming.

We obviously do not know about the cases where such conspiracies are successfully covered up - it is incredibly easy to cover up human thoughts as compared to LLM reasoning traces. In this case prosecutors had both emissions measurements and written documentation of misalignment, so they were able to get the job done.

But they don't always - we can look to another case, Goldman Sachs' role in selling what one of its own bankers called a "shitty deal" in the years leading up to 2008.

Mr. Montag asked Mr. Sparks to provide him with a “complete rundown” on “what[’]s left.” Mr. Sparks responded that the “main thing left” was $300 million in Timberwolf securities. Mr. Montag responded: “boy that timeberwo[l]f was one shitty deal.”

-- Excerpt from the Senate report on Goldman Sachs and Timberwolf

The video where some guy at Goldman is grilled about it is pretty funny and only a minute or so long, I recommend it!

The Justice Department said it could not meet the burden of proof for a criminal case based on the Senate report's allegations.

It is widely believed that the 2008 crisis represents a failure of human alignment, although most critics wouldn't phrase it like that.

My point in this section is that scheming among humans is near-impossible to prove without incriminating documentation. Since there is no direct interpretability or way to document human thought, that means most misalignment cannot be proven, let alone acted on. Thus, there is widespread misalignment and criminality - limited only by the human ability to commit such acts.

That's probably not the model we want to go for.

Practical methods to avoid chain of thought liability

It is now clear that because human chain of thought is already invisible to the legal system, there is no analogy to draw between human and LLM alignment. The human liability avoidance strategy is to avoid documentation. The same can be done with LLM reasoning traces, but there are three groups that need different strategies:

  1. Buyers of AI models via APIs - they can simply use models that do not provide reasoning traces. The AI provider might then hold evidence the buyer never sees. UK financial regulators say firms remain responsible when they outsource, and EU outsourcing rules can require audit access. For covered high-risk systems, the EU AI Act's log-retention rule concerns automatically generated logs under the deployer's control. These rules don't specifically require saving raw reasoning traces or using only models that produce them. A buyer may see a legal advantage in having fewer traces to retain, even though it still has to supervise the system.

  2. Model vendors like OpenAI can create evidence that may be used against them by saving and monitoring chains of thought. That gives them a reason to switch to models that produce fewer legible traces, if they can still show that their monitoring is adequate. This just depends on convincing regulators that other interp methods are sufficient, or that their demands are impossible to meet. This is very hard when you have plaintext records of models thinking "I am going to do something illegal".

  3. Local model users - if the regulatory burden of deploying such a system becomes too onerous, these firms may either just stop using local models or face legal challenges. The aforementioned requirements to retain records seem to straightfowardly apply to local model reasoning traces in a way they do not for outsourced models that do not provide such traces.

I don't know if the legal implications for firms like Revolut, which has described running production agents on open models, are well understood, but my feeling is that a reasoning trace saying "I am putting this bug in to allow attackers in" followed by a bug would look pretty bad in court, even if the thought is just noise and the bug is a legitimate mistake. The faux interpretability of chains of thought by non-experts can create liability without corresponding benefits.

Other ways chains of thought may cease to be useful

This post is about legal liability forcing non-interpretability. This post is NOT about:

  • Grading chains of thought during reinforcement learning, which both OpenAI and Anthropic have reported doing by accident. This can make traces harder to monitor. The effect depends on what gets graded and how much training uses that signal, but this is nearly universally viewed as bad and somewhat easy to avoid.

  • Training final answers to look aligned can make chains of thought look more aligned even without direct grading. This is also bad but impossible to avoid.

  • Chains of thought becoming obsolete in favour of something else e.g. fully recurrent architectures capable of multi-step reasoning in latent space. It doesn't look like this is going to happen any time soon. This is distinct from opaque reasoning being worse but preferred because it is non-interpretable - maybe it'll end up just being better.

There are lots of reasons for chains of thought disappearing or becoming unreadable over time.

This post does not contain any experiments validating any timelines or even any hypotheses about the speed of any effects, but a future post of mine will.

Good endings we can still aim for

Here are some possible futures and ways in which I could be wrong:

  • Several high-profile legal challenges succeed and result in minuscule penalties for the AI labs. No-one cares, the AG gets a win that can be touted in an election, and the labs continue to make CoTs interpretable due to internal utility for alignment. I think this one is pretty likely.

  • CoT goes away but we get "MRIs for LLMs" that require experts to interpret. Mechanistic interpretability isn't really at the point where it can give us the same level of transparency into a recurrent frontier model that we get from reasoning traces on non-recurrent models - I am not sure it ever will be. I think this is unlikely.

  • Competitiveness of the AI model industry decreases, or firms communicate and collude to preserve CoT interpretability in the face of legal pressures. This doesn't sound great from a consumer point of view given that it means that AI providers would be providing a worse product while colluding with rivals to maintain prices. This is the rare kind of case that may actually result in significant legal penalties - significant even for a trillion dollar company. I don't think this is likely at all. At the very least, if recurrent models are the only way to do AGI, for example, I am pretty sure some firm somewhere will defect and just do it. AI is too competitive.

  • Few people in the legal system know what reasoning traces are and there remain few legal challenges or reports citing traces. The recent cases are isolated. I view this as unlikely - are lawyers really going to ignore the treasure trove of documented illegality in reasoning traces? Imagine how many people would have gone to jail in 2008 if we had faithful, readable representations of bankers' thoughts to look at!

Chains of thought are a remarkable accidental dividend for interpretability and maybe they will stick around! I still think they won't.