This is a cross-post from my Substack to get some feedback from the alignment community. I have condensed some sections to get the message across.

It's rarely questioned in the mainstream why we need RLHF in the first place? If we can do supervised fine-tuning (SFT) sufficiently well, then can we just ditch reward modeling and RL at all? This seems to be a question that many people silently have (e.g., in this tweet).

To better understand the goal of RLHF, I reflected on the related paradigm of inverse RL (IRL), which learns a reward function from human demonstrations rather than comparisons, and I took a deep dive into the roles of reward and RL in both frameworks. As a preview, both IRL and RLHF can alleviate exposure bias, a mismatch between training and testing conditions that potentially underlies degenerate behavior in SFT models. However, the advantage of RLHF is the possibility of outperforming the dataset. Nevertheless, the effect of the latter is unclear (perhaps dubious?) in practice.

A brief recap of IRL

Before RLHF established the binary comparison approach to learning from human feedback, the basic approach was just to imitate human behavior (a.k.a., imitation learning). To ensure we learn good behavior, the humans who demonstrate the behavior must be experts at the desired task. Given a dataset of expert behavior $D = {(s, a)}$ , imitation learning searches for a policy $^π$ which maximizes the likelihood of the dataset:
${max}_{^π} E_{(s, a) \sim D} [log^π (a | s)]$
In the setting of language modeling, if we interpret the state $s$ as past tokens (a.k.a., context) and action $a$ as the next token, the imitation learning objective is equivalent to the maximum likelihood objective used for training autoregressive language models.

Inverse RL can be seen as a special class of imitation learning algorithm which makes an additional assumption on the structure of the learner policy $^π$ that it is an optimal policy with respect to a pair of reward function $R (s, a)$ and environment dynamics $P (s^{'} | s, a)$ , where the latter is often assumed to be given. IRL instead searches for a reward function under which the optimal policy maximizes the likelihood of the dataset:
$\begin{matrix} max R & E_{(s, a) \sim D} [log^π (a | s; R)] s.t. & ^π = arg max π E_{π, P} [\infty \sum t = 0 γ^{t} R (s_{t}, a_{t})] \end{matrix}$

SFT headache: Exposure bias and neural text degeneration

It is by now well documented that language models train...

Posts

Wikitag Contributions

Comments