On March 27, Anthropic published what may be the most important AI research paper of 2025 so far. "On the Biology of a Large Language Model" presents groundbreaking interpretability research on Claude 3.5 Haiku, using a technique called circuit tracing to map how the model actually processes information.
The findings are fascinating, sometimes beautiful, and in one specific area, deeply unsettling for anyone relying on AI systems in production.
Here's the finding that matters most: models fabricate their chain-of-thought explanations. The reasoning they show you isn't necessarily the reasoning they used. They engage in what the researchers characterize as motivated reasoning — constructing post-hoc justifications for conclusions they've already reached through opaque internal processes.
If you're an engineering leader deploying AI systems that use chain-of-thought reasoning — and if you're using any modern LLM for complex tasks, you are — this finding has immediate practical implications.
What Circuit Tracing Revealed
Anthropic's research used circuit tracing to map the internal computational pathways of Claude 3.5 Haiku. Think of it as an MRI for neural networks — rather than observing inputs and outputs, the researchers traced how information flows through the model's internal representations.
Three findings stand out.
First, the research revealed a shared conceptual space where reasoning happens before language. The model processes concepts in an internal representation that isn't tied to any specific language. It can learn something in English and apply that knowledge when processing French — not through translation, but through a language-independent conceptual layer.
This is scientifically elegant and suggests that large language models develop abstractions more sophisticated than simple pattern matching. They build conceptual representations that transcend the surface form of their training data.
Second, the research confirmed that models can learn in one language and apply knowledge in another. Cross-linguistic transfer works because the underlying conceptual representations are shared. A model trained predominantly on English doesn't need to see the same fact in German to access it when processing German text.
Third — and this is the critical finding for engineering teams — the research revealed that models fabricate chain-of-thought explanations post-hoc. The reasoning shown in the model's "thinking" process doesn't necessarily reflect the computational path the model actually took to reach its conclusion.
This is motivated reasoning at the model architecture level. The model reaches a conclusion through opaque internal computation, then generates a plausible-sounding explanation after the fact.
Why This Matters for Production AI
Chain-of-thought reasoning has been one of the most significant advances in AI capability over the past two years. Models that "think step by step" produce better results on complex tasks. Reasoning models from OpenAI (o1, o3-mini), DeepSeek (R1), xAI (Grok 3 "Think" mode), and Anthropic (Claude 3.7 Sonnet with extended thinking) all use variants of this technique.
Enterprise teams use chain-of-thought reasoning for exactly the reasons you'd expect: it produces better results, and it appears to provide transparency into the model's decision-making process. If the model shows its work, we can verify its reasoning. If the reasoning looks sound, we have higher confidence in the conclusion.
The Anthropic research undermines this assumption. If the chain-of-thought is a post-hoc fabrication rather than a faithful record of the model's actual computation, then reviewing the chain-of-thought is less informative than we believed.
This doesn't mean chain-of-thought is useless. The technique still produces better outputs by constraining the model's generation process. And the fabricated reasoning often looks correct because the model's training incentivizes plausible-sounding explanations. But it does mean that chain-of-thought transparency is not equivalent to genuine interpretability.
The distinction matters enormously for engineering teams that use chain-of-thought as part of their verification process. If you're reviewing an AI's reasoning to verify its output, you're reviewing a narrative the model constructed to justify its conclusion — not the actual process that produced the conclusion. It's the difference between reading a well-written report and observing the research that produced the report's findings.
The Verification Implication
The practical implication is straightforward: you cannot rely on the AI's self-reported reasoning as your primary verification mechanism.
This seems obvious when stated directly, but it contradicts the implicit assumption behind many AI deployment patterns. Teams that use chain-of-thought prompting often treat the visible reasoning as evidence of correctness. "The model showed its work, and the work looks right, so the answer is probably right."
The Anthropic research suggests this logic is unreliable. The "work" the model shows may be a plausible narrative constructed to accompany a conclusion reached through an entirely different process. The narrative might be correct anyway — but you can't determine that by examining the narrative alone.
What you can do is verify the output independently of the model's self-reported reasoning. Does the code compile and pass tests that the model didn't write? Does the analysis match independently verified data? Does the recommendation align with domain-specific criteria defined outside the model's context?
This is the fundamental argument for specification-driven verification over reasoning-review verification. Rather than asking "does the AI's reasoning look correct?" ask "does the AI's output satisfy independently defined specifications?"
The specification doesn't trust the model's explanation. It tests the model's results.
What This Means for AI Governance
The interpretability research has direct implications for AI governance frameworks being developed under the EU AI Act and similar regulations.
The EU AI Act requires transparency in AI system decision-making. Many organizations are planning to satisfy this requirement by logging chain-of-thought reasoning — capturing the model's visible "thinking" process as evidence of how decisions were made.
The Anthropic research suggests this approach is insufficient. If chain-of-thought explanations are fabricated post-hoc, they don't provide genuine transparency into the model's decision-making process. They provide a narrative that accompanies the decision — which is useful for documentation but not sufficient for genuine auditability.
Genuine auditability requires more than capturing the model's self-explanation. It requires capturing the inputs, the context, the configuration, the output, and independent verification of the output against expected behavior. The model's chain-of-thought can be one data point in a comprehensive audit trail, but it shouldn't be the primary — or only — evidence of how a decision was made.
For organizations building toward EU AI Act compliance, this is an important architectural decision. Build your audit infrastructure around independently verifiable records of system behavior, not around the model's self-reported reasoning. The model's explanation is a narrative. The audit trail needs to be evidence.
The Deeper Challenge
Anthropic's research reveals a deeper challenge for the AI industry: the gap between what models can do and what we can verify about how they do it.
This gap exists at multiple levels. At the model level, we can observe inputs and outputs but can't fully trace the internal computation. At the application level, we can see what the AI produced but can't always verify why. At the organizational level, we can measure outcomes but can't always attribute them to specific AI behaviors.
Interpretability research like Anthropic's is essential because it narrows this gap — not by making models fully transparent, but by giving us better tools to understand their behavior. Circuit tracing doesn't make the model fully interpretable, but it reveals patterns (like motivated reasoning) that change how we should approach verification.
The practical takeaway for engineering teams is humility about what we know and rigor about what we verify. Don't trust the model's explanation of itself. Do trust independent verification of the model's outputs. Build your governance infrastructure on the assumption that the model is a black box that produces useful outputs and unreliable self-explanations.
Practical Steps for Engineering Leaders
Given the interpretability findings, here's what engineering teams should adjust:
Stop using chain-of-thought review as your primary quality gate. It's useful as supplementary information, but it shouldn't be the mechanism that determines whether AI output ships to production. Independent verification — tests, specifications, contracts, and domain-specific checks — should be the primary gate.
Log chain-of-thought for analysis, not for trust. Capture the model's reasoning in your audit trail because it's useful for debugging and pattern analysis. But annotate it clearly as the model's self-reported reasoning, not as verified evidence of the decision-making process.
Build verification infrastructure that doesn't depend on model transparency. If interpretability research eventually makes models fully transparent — a goal that's years or decades away — that's wonderful. In the meantime, your verification infrastructure should work under the assumption that the model's internal processes are opaque. Spec-driven verification, output testing, and behavioral monitoring are robust strategies regardless of interpretability progress.
Invest in outcome-based evaluation. Rather than trying to verify the model's reasoning process, verify its results against real-world outcomes. Did the code work? Did the prediction match reality? Did the recommendation produce the intended effect? Outcome-based evaluation sidesteps the interpretability challenge entirely.
The Honest Answer
When clients and stakeholders ask "how does the AI make decisions?" the honest answer, informed by this research, is: "We don't fully know, and the AI's own explanation may not be reliable. What we do know is that the AI's outputs pass these specific verification checks, satisfy these independently defined specifications, and are logged in an immutable audit trail."
That's a less satisfying answer than "the AI shows its reasoning and the reasoning looks sound." But it's a more accurate and more defensible answer. And in an environment where regulatory scrutiny of AI systems is increasing, accuracy and defensibility matter more than narrative satisfaction.
Anthropic's interpretability research is a gift to the engineering community. It tells us something important about the systems we're deploying: their self-explanations are unreliable. The appropriate response isn't panic — it's better verification infrastructure.
The models are getting smarter. Our verification needs to get smarter too. And it needs to be independent of the models it's verifying.