Catalog-Driven Supervised Sparse Autoencoders: Specifying Token-Level Features Before Training Recovers What Unsupervised Dictionaries Miss
Tsogt-Ochir Enkhbayar
Mentored by Georg Lange
Working report from the SPAR program. May not reflect the authors' current views.
Abstract
Modern SAE pipelines use LLMs as supervisors only after the unsupervised dictionary is fixed to interpret latents that have already been trained. We test whether the dictionary even admits clean upstream supervision and find it does not. On layer 9 of a GPT-2 Small SAE, of the top 300 unsupervised latents by ΔR², the best 100 pass through a strict catalog-quality cascade and only 2 survive. The rest are rejected by a frontier LLM as polysemantic bundles. We move the LLM upstream: a frontier LLM creates a catalog of token-level concepts as prefix-decidable yes/no questions; a local LLM labels every (token, feature) pair through a three-leveled prefix caching scheme; and a supervised SAE is trained with its decoder columns pinned to the mean directions of those concepts. The supervised alternative reaches a mean calibrated F1 of 0.604 on a 103-feature catalog and 0.481 on a 466-feature catalog, against 0.025 for real EleutherAI Delphi auto-interp on the unsupervised 24,576-latent gpt2-small-res-jb SAE. Both sides are scored identically: per-feature F1 of latent firing against the same Qwen3 annotator on the same held-out positions. The supervised dictionary matches reconstruction quality at 68× fewer latents (R² = 0.969) while producing directions that individually admit falsifiable description and causal intervention.