As LLMs combine into our each day lives, understanding their conduct turns into important. In our ongoing efforts to review mannequin conduct and alignment, we current this work as an early step in that route. We give attention to behavioral tendencies — the underlying tendencies that form responses in social contexts — and introduce a framework to review how carefully the tendencies expressed by LLMs align with these of people.
Behavioral tendencies are sometimes quantified through self-report questionnaires below totally different traits (e.g., empathy, assertiveness), the place people price their settlement with preference-statements, akin to, “I’m fast to specific an opinion.” The questionnaires used on this examine are standardized, scientifically validated measures broadly used for assessing persona traits in worldwide analysis and psychology akin to: IRI (empathy), ERQ (emotion regulation), and extra. Every instrument is grounded in peer-reviewed literature that establishes its psychometric validity and reliability utilizing totally different methods. We selected essentially the most broadly used devices for our analysis.
Our goal is to construct upon such psychological questionnaires, however immediately making use of them to LLMs presents technical challenges, as LLM outputs are delicate to immediate phrasing and distribution shifts. Consequently, tendencies “claimed” by LLMs inside a self-report format will not be assured to efficiently switch to conduct in lifelike, open-ended settings.
To handle these challenges, in “Evaluating Alignment of Behavioral Tendencies in LLMs,” our framework evaluates LLMs’ behavioral tendencies in lifelike user-assistant situations the place their advisory position can result in tangible affect. This examine is an early step in evaluating the alignment between human consensus and mannequin conduct throughout lifelike, sensible situations, specializing in on a regular basis human-to-human interactions and office conditions. We be sure that these situations stay grounded in established psychological questionnaires to seize the essence of core behavioral traits. Examined situations included skilled composure, battle decision, sensible duties akin to reserving a visit, and life-style or each day decision-making, highlighting mannequin conduct in settings consultant of typical human day-to-day experiences. Our large-scale evaluation of 25 LLMs reveals two sorts of gaps: one the place mannequin tendencies deviate from consensus amongst human annotators, and one other when mannequin tendencies don’t seize the vary of human opinions when consensus is absent. These early outcomes spotlight the chance for higher behavioral alignment to make sure that fashions can extra appropriately navigate the nuances of social dynamics, outcomes we anticipate future analysis to construct on.