Evaluating LLMs’ Bayesian capabilities
As with people, to be efficient, an LLM’s consumer interactions require continuous updates to its probabilistic estimates of the consumer’s preferences primarily based on every new interplay with them. Right here we ask: do LLMs act as if they’ve probabilistic estimates which might be up to date as anticipated from optimum Bayesian inference? To the extent that the LLM’s conduct deviates from the optimum Bayesian technique, how can we reduce these deviations?
To check this, we used a simplified flight advice process, through which the LLMs work together as assistants with a simulated consumer for 5 rounds. In every spherical, three flight choices have been introduced to each the consumer and the assistant. Every flight was outlined by a departure time, a period, numerous stops, and a price. Every simulated consumer was characterised by a set of preferences: for every function, they may have a powerful or weak desire for prime or low values of the function (e.g., they might want longer or shorter flights), or no desire concerning this function.
We in contrast the LLMs’ conduct to that of a mannequin, a Bayesian assistant, that follows the optimum Bayesian technique. This mannequin maintains a likelihood distribution that displays its estimates of the consumer’s preferences, and makes use of Bayes’ rule to replace this distribution as new details about the consumer’s decisions turns into obtainable. In contrast to many real-life eventualities, the place it’s tough to specify and implement the Bayesian technique computationally, on this managed setting it’s simple to implement and permits us to exactly estimate the extent to which LLMs deviate from it.
The purpose of the assistant was to suggest the flight that matches the consumer’s alternative. On the finish of every spherical, the consumer indicated to the assistant whether or not or not it selected accurately, and offered it with the right reply.