Towards understanding LLM behavior through fine-tuning

Recent work shows that it is still unclear when and how large language models (LLMs) generalize when trained on new data. To better understand this, we finetune LLMs on small datasets, and evaluate the effects on the LLMs' behavior. We investigate how two distinct finetuning interventions --- training on stated beliefs in multi-turn conversations and implanting factual information via synthetic document finetuning (SDF) --- impact related queries and downstream tasks. In the first setting, we finetune LLMs on liberal and conservative conversational data, and evaluate how their responses change on both directly political and broader worldview questions, and a gender-bias downstream task. The models adopt their fine-tuned political ideologies on out-of-distribution questions and become more/less biased on downstream tasks. In the second setting, we finetune models on synthetic documents about controversial topics, e.g. factory farming. The fine-tuned models express different sentiment on directly related queries about the topic, but do not act differently on downstream tasks (e.g. recipe recommendations) or even on tangentially related queries. Together, these results suggest that finetuning on stated preferences generalizes to downstream actions more readily than finetuning on beliefs implanted as facts through pretraining-style documents.