Final Report: Formalizing, Probing, and Intervening on the Belief-Action Gap In Agents

Modern language-model agents are increasingly evaluated through their external behaviour: whether they complete a task, choose the correct action, or reach the intended goal. However, behavioural success or failure alone does not reveal whether a failure is caused by an absence of task-relevant information, an incorrect internal model of the environment, or a failure to use internally available information when selecting an action. This distinction is especially important for agentic safety because an agent that internally represents the correct state of the world or the task-optimal action, but nevertheless acts differently, presents a qualitatively different problem from an agent that simply lacks the relevant representation. We study this distinction through the belief–action gap: the mismatch between what an agent appears to represent internally and what it ultimately does. We mathematically formulate this mismatch for languagemodel agents interacting with Markov decision processes. In this formulation, belief is a distribution over latent environment states, action is the model’s emitted decision, and the gap is the excess expected cost of the emitted action relative to the best action under the agent’s inferred belief, transition, and value structure. To study this operationally, we separate the true simulator state, representational beliefs decoded from hidden activations, explicit behavioural beliefs elicited through natural-language probes, and the action actually emitted by the model. This separation allows us to ask whether a suboptimal action is best explained as a belief failure, a transition-model failure, a value/cost misalignment, or an action-selection failure. We instantiate the framework in text-rendered gridworld navigation tasks, where the environment state and optimalaction set can be computed exactly. The gridworld setting lets us compare, at the same state, the simulator’s optimal actions, actions decoded from model activations, explicit answers to behavioural state probes, and the model’s emitted action. We use this setting to develop a measurement pipeline combining representational probes, behavioural probes, inverse-reinforcement-learning-based cost recovery, optimal-action-set evaluation, and state-level agreement analysis.