Abstract:
AI models are typically trained through supervised learning, using examples of desired responses, and through reinforcement learning, using rewards for attempted responses. When absolute rewards are unavailable, comparisons between responses can be used to learn a reward signal for reinforcement learning. Popular direct preference methods simplify this process by training on the comparisons themselves. This talk examines how mismatches between training assumptions and model capabilities can cause unexpected behaviour, e.g., ultimately favouring responses that are actually preferred less, and how to address these issues. It then turns to reinforcement learning with absolute rewards, showing why uncertainty in those rewards matters for the risk of making a model worse. Drawing on theory and experiments with language models, we will present methods that address model mismatch and account for reward uncertainty. Together, these findings highlight how careful treatment of model limitations and uncertain feedback can help make AI alignment more reliable, in the broader challenge of aligning AI systems reliably with human goals. (Joint work with Sayak Ray Chowdhury, Debangshu Banerjee, Kintan Saha)
Bio: Aditya Gopalan is a Professor of Electrical Communication Engineering at the Indian Institute of Science. His research interests include machine learning and statistical inference, sequential decision making, statistical modeling, and optimization.