Robust Policy Optimisation under Reward Misspecification via Support-Bounded Uncertainty

We propose a framework for safe policy improvement when the reward function is only a partially trustworthy proxy for the true objective. Safe Policy Improvement with Baseline Bootstrapping (SPIBB) constrains policy updates in poorly-covered regions, but assumes the proxy reward is accurate within its trusted support. We address its complementary case, bounded reward uncertainty within that support. Given a baseline $\pi_0$ and trusted support $K = \{(s,a) : \mu_{\pi_0}(s,a) \geq \tau\}$, we define a support-bounded uncertainty set $\mathcal{U}_K$ and bootstrapped policy class $\Pi_K$ and prove that the worst-case safety constraint reduces exactly to a penalised occupancy-shift condition: $\Delta\mu_K \cdot R'_K - \varepsilon_\text{in}\|\Delta\mu_K\|_1 \geq 0$. Lagrangian relaxation then converts this to tractable modified policy iteration. Experiments across 40 random MDP configurations show naive proxy optimisation causes Goodhart failure in 87.5\% of cases; our method recovers in 75\% of those with a 90\% reduction in Goodhart gap. The framework is formally certifiable for misspecification radii $\varepsilon_\text{in} \leq 0.15$ and generalises without modification to Gymnasium environments, where naive methods catastrophically fail.