Learning Interpretable Updates to LLMs

Standard post-training procedures alter language-model parameters in globally opaque ways, raising safety concerns about the introduction of hidden biases and spurious shortcuts. We explore whether sparse continuous masks over MLP neurons can serve as an inherently interpretable post-training mechanism: by freezing the base model and training only per-neuron scalar gates with an L0 sparsity penalty, we hoped that the modified neurons would correspond directly to the concepts the model relies on. We instantiate this idea on a synthetic sentiment classification task with a perfect "bad"↔negative spurious correlation, and run a comprehensive sweep across both Qwen-2.5-0.5B-Instruct and Llama-3.1-8B-Instruct, covering every single-layer mask choice, all-layers configurations with trainable and frozen classifier heads, and a complete ablation study with five intervention methods and a scaling sweep. The capacity-control side of the method works: per-layer mask restriction tightly bounds the spurious gap. The interpretability side does not. The training-modified neurons are essentially uncorrelated with the neurons that causally drive the classifier's decision, and ablating the most-modified neurons leaves accuracy unchanged. The neurons that actually matter are surfaced by behavioral measurements (activation contrast between has_bad and no_bad inputs), and direction ablation on the top-10 activation-contrast neurons closes 24% of the spurious gap without hurting overall accuracy. We diagnose this as a manifestation of a broader challenge for weight-space localization when the optimizer has multiple equivalent-loss solutions, and outline experimental conditions under which the mask method could be made interpretability-load-bearing