What Happens When AI Reasoning Becomes Invisible?

If we can't see the path an AI model takes to its answer, can we truly audit it? That question sits at the heart of a call made by Tomek Korbak, Mikita Balesni and Jasmine Wang, all dismissed from OpenAI: as models improve, the methods for testing how safe they are must improve too.

The three researchers insist that work must continue to keep techniques that inspect models' reasoning traces accessible in the future. Otherwise, they expect risky behaviors that may emerge down the line to slip through unnoticed.

The written path behind the answer: CoT

With this technique, the model writes out the steps leading up to a conclusion. That way, experts can read not just what it got wrong, but why it answered the way it did.

In comparison

Why are traces shrinking in new models?

The Astra class produces less written-out reasoning than the previous generation when solving problems. In other words, the text monitoring systems can read shrinks, making it harder to figure out from the outside what the model is doing.

According to OpenAI, this doesn't mean Astra has evaded oversight. The company acknowledges the limitation exists and considers better observation of reasoning an important topic for safety research.

After all, the written text isn't a one-to-one dump of the model's inner world either. And when new methods are layered on top, the picture gets blurrier.

Recurrent depth brings invisible computation

The second notable method can run multiple rounds of processing in sequence for a single query. The model can thus carry out a far longer computation.

Unfortunately, these steps aren't written out externally. So even if we see the answer via CoT, the computation that led to it can't always be read.

Background to the dismissals

The reason given for terminating Korbak, Balesni and Wang was violation of company policies and sharing of sensitive information with an external AI safety organization. OpenAI said the decision was unrelated to the employees raising safety concerns.

The company isn't opposed to the oversight approach the researchers advocate either; it says CoT monitoring work should continue. The three, for their part, keep calling for audit techniques to be preserved going forward.