Human Behavioral Phenomena in Language Models: Investigating Code-Switching and Recency Bias Across Multilingual Settings

As large language models (LLMs) are deployed to a global user base, understanding how they respond to multilingual and multicultural input is a pressing alignment concern. This paper investigates two related questions: (1) how do language models exhibit language or recency bias in multilingual settings, and (2) how does the grammatical structure of code-switched input shape the language in which frontier LLMs respond – including instruction following? We begin by building a simple multilingual dataset and evlauating the biases of ten LLMs in a controlled setting, observing that while LLMs do exhibit language bias, this is linked most strongly to query language rather than order. Then, using the CodeMixQA benchmark across four languages (Hindi, Simplified Chinese, Spanish, and Urdu), we evaluate a subset of the 10 models — Claude Opus 4.6, Gemini 2.5 Flash, GPT-4.1-mini, GPT-5.4, and Qwen3.6-plus — across approximately 7,100 API calls. Our central finding is that grammatical matrix language, rather than surface token frequency alone, is the dominant driver of model response language. This grammar-forcing effect is universal across all models and language pairs tested, yet factual accuracy is unaffected, pointing to a purely metalinguistic behavior rather than a comprehension limitation. Critically, models diverge sharply in how well they override this effect under explicit instruction, with significant alignment implications for multilingual deployment.