Signs of Life for Capability Asymmetric Debate

Could having more capable systems debate each other enable less capable judges to provide scalable oversight?  Prior work showed it works when debaters have privileged access to information (information asymmetric) but not when the only difference between debaters and judges is their capability on a target task (capability asymmetric).   We revisit this setting here and find that, with sufficiently capable debaters, less capable LLM judges are ~7% better at finding the correct answer to complex reasoning problems from GPQA-diamond when given access to debates.  Further, we find that these gains follow a tight linear relationship with debater strength.  Our human annotation of debates suggests that the debate protocol is working well, and that the limitation to larger gains is LLM judge quality. Dedicated judge training is a direction for future work.