This is the third post in a four-part series about post-training. If you missed them, read part 1 and part 2. The final post, on implementing your own pipeline, will be coming October 7.
Now that you understand the gist of reinforcement learning and supervised fine-tuning, it’s time to explore what post-training has actually accomplished in the frontier models you know and love.
Remember our prompt “Why do people like golden retrievers?” GPT-3 would often answer nonsensically. But that all changed in November 2022, with the launch of ChatGPT. Now “Why do people like golden retrievers?” actually returned a reasonable response like “Because they are affectionate, patient, and make excellent family pets” no matter who was typing (with no weird formatting tricks to consider). Anyone who could send a text message could get a response back on any topic.
I’ll cover some of those behavior changes below, then take you through ChatGPT’s training pipeline as described in OpenAI’s InstructGPT paper.
Conversational and helpful
The most visible impact of post-training is models that can chat with you and hold a relatively long conversation. This sounds simple, but it’s not.
Being conversational means more than responding to a question with an answer. The model needs to recognize when a question is ambiguous and ask for clarification or make the right assumptions in a quick response. It should adjust its tone and detail level to the context, for example being brief for a quick factual question, but thorough for a learning-oriented one. The model should be coherent across multiturn conversations without losing the thread. It also needs to handle messy real-world inputs: You attach a giant PDF and ask it to find one specific clause, and it should either find it or tell you it can’t, not hallucinate an answer.
Safety and alignment
Post-training is also the primary mechanism for making models safe. Safety in this context means a few things:
- Refusing to generate harmful content (like instructions for creating weapons, when asked)
- Avoiding biased or discriminatory outputs
- Not making up information when unsure (hallucination reduction)
- Respecting user privacy
However, you can define safety rules in whatever way you want and teach the model to abide by them, within the limits of what your reward signals can capture. If you think cats are unsafe, because you’re a dog person, you can teach the model that in post-training—as long as you can properly encode that into a reward signal.
Safety is often in tension with helpfulness. On one extreme, a model that’s too conservative will refuse reasonable requests, something that has frustrated many users. On the other extreme, a model that’s too permissive will comply with harmful ones. Navigating this trade-off is difficult. Ultimately, it comes down to determining where to draw the line, which is (as of today) a human decision within labs. Post-training is the tool to implement wherever the line is drawn.
Tool use and function calling
Tool use is one of the most practically important capabilities enabled by post-training. Tools include search engines, APIs, calculators, databases, and code interpreters. Being able to hit a search engine alone allows the model to not hallucinate, given its own knowledge cutoff. Tools are extremely useful ways for models to interact with the world, and are fundamental components in building agents.
Tool use is a set of new behaviors. The model needs to recognize when a user’s request would benefit from an external tool. It needs to know which tools are available to it, and not hallucinate a tool. It needs to formulate a correct API call with the right parameters. It needs to interpret the results that come back and incorporate them into a natural language response. It needs to do all of this seamlessly, without the user needing to know the details of the underlying tool.
This is taught almost entirely through SFT, at least initially. The training data includes many examples of conversations where the model correctly decides to invoke a tool that it has access to, constructs the right call, and processes the result. RL can further improve tool use by rewarding the model for correct tool invocations and penalizing unnecessary or incorrect ones.
As an example of tool use, let’s say you’re building a veterinary appointment scheduling assistant. A user asks: “My golden retriever has been limping since yesterday. Can I see Dr. Patel this afternoon?” A pretrained model might generate plausible but fictional appointment times. A post-trained model with tool use instead calls the clinic’s scheduling API, checks Dr. Patel’s availability, and responds: “Dr. Patel has an opening at 3:15pm today. I’ve tentatively held it for you. Should I confirm?” The model needed to decide if the user’s intent was urgent, select the right tool, construct the API call with the right veterinarian and time constraints, and present the result conversationally.
Tool use has expanded through the Model Context Protocol (MCP), a lightweight standard for connecting models to external services like Gmail, GitHub, or a company’s internal databases. Rather than building custom integrations for each tool, MCP provides a standard interface that any API can plug into, and different frontier models have now included learning MCP in their post-training recipes. Agentic frameworks take this further by allowing models to chain multiple tool calls together to accomplish common multistep tasks more easily.
Reasoning (“thinking”)
Reasoning models, or models that are trained to “think” before they answer, are an exciting result of post-training. Rather than producing an immediate response, these models generate an internal chain of thought, working through the problem step-by-step, before arriving at a final answer. As a result, their answers are more often correct than nonreasoning models that might guess at an answer.
This capability has an interesting relationship with pretraining and post-training. The raw ability to reason is latent in pretrained models; they’ve been trained on text that includes mathematical proofs, logical arguments, scientific analyses, and code with comments explaining the logic. But pretrained models don’t default to reasoning. They default to pattern-matching, which often produces plausible-looking but incorrect answers.
Reasoning models dramatically outperform standard models on tasks that require multistep logic: mathematical problem-solving, complex coding, scientific analysis, and planning. The improvements are not incremental. On the 2024 AIME exam, GPT-4o was only able to get 12% of problems correct on average. OpenAI’s o1 reasoning model solved 74% off the bat, with a single attempt. With 1,000 attempts and a learned scoring function to rerank the attempts, it reached 93%, a result placing it among the top 500 students who took the AIME math exam in the US.
More capable reasoning requires more compute, both during training and at inference time. Scaling laws meet post-training. Models that have learned to spend more inference (test-time) compute on reasoning tend to reach better answers and therefore exhibit higher intelligence. For some frontier reasoning models, the RL post-training phase uses as much compute as the entire pretraining phase.
The cost is not only in post-training compute but also in inference (test-time) tokens and latency. Reasoning takes up a lot of tokens and can result in a longer time to get a response back to the user. But the type of request matters. For a quick factual question, you don’t need reasoning. For a complex technical problem, the extra latency is well worth it. This is something that model providers can modulate during post-training.
The classic ChatGPT pipeline
As I mentioned above, the first post-training pipeline that captured global attention was ChatGPT’s, and it drew on the pipeline described in the InstructGPT paper. While modern systems use more advanced approaches today, this classic pipeline remains the conceptual foundation for nearly all alignment methods.
The pipeline has three stages, each building on the previous one:
- Supervised fine-tuning (SFT) on human demonstrations
- Training a reward model on human preference comparisons
- Reinforcement learning with human feedback (RLHF) to optimize the main model using the reward model
Stage 1: SFT on demonstrations
The first stage is straightforward and teaches the model to follow instructions and behave like an assistant.
OpenAI contracted ~40 human labelers to label their data, and they were careful to filter for people who were good at identifying harmful outputs. The labelers had to write ideal responses to prompts. But what’s interesting is that the prompts came from two sources: (1) prompts submitted by real users through the OpenAI API and (2) prompts that labelers wrote themselves. The users had to write prompts too, because these were the days before ChatGPT. There weren’t that many real users with instruction-like prompts through the API to collect.
The prompts were diverse and mostly in English. There’s also an extensive data cleaning pipeline to remove duplicates and remove sensitive PII (personally identifiable information). Importantly, they split the training, validation, and test sets by human labeler. This is to avoid data leakage that could happen within a single user’s data between training and validation/testing.
The resulting SFT dataset had ~13,000 prompts, all with human-labeled responses. The base model was GPT-3 at the time, a pretrained model without any post-training. Using SFT, they trained GPT-3 for 16 epochs, which was effective for the final RLHF model. This was interesting, because for the SFT stage alone, the model overfit after just 1 epoch, but ultimately SFT was an intermediate stage so they picked the best checkpoint for the final RLHF model. They also mixed in 10% pretraining data during this phase, because it would help the next RL phase.
At this point, this SFT model could already be pretty useful: It could have a conversation and follow instructions, which is leaps and bounds beyond the pretrained GPT-3 checkpoint.
Stage 2: Preference data and reward modeling
This next stage is training the reward model. The reward model needs to grade millions of responses during RL training. In the original method, OpenAI’s team mainly trained the reward model on responses from the SFT model. However, as the policy model is trained in the RL loop and generates new, and likely better, responses from its evolving checkpoints, the reward model needs to stay robust. As a result, they also continually updated the reward model using responses from new RL checkpoints over time.
To train the reward model in InstructGPT’s RLHF pipeline, OpenAI needed pairwise comparisons of two model responses from one prompt, and a label for which one is better. For example, given “What’s 2+2?” and the responses are “4” and “Yes,” the label should say “4” is better than “Yes.” Note again that these are responses from the SFT model (and later, the RL-ed models during the RL training loop), not the pretrained base model. So labeling can only happen after you’ve SFT-ed your model. If you need to retrain that model, you likely need to relabel to make sure the reward model is trained on the right distribution of data pairs.
Reward model training
The reward model was small at 6B parameters, for both efficiency and stability, and included a head that outputted a scalar reward. They had tried multiple sizes, but found this was more stable than using the original 175B main model. It was also more compute efficient, as the reward model would take up extra compute, for both inference and training, on top of training the main model itself. More recently, reward models have become a lot larger, but note that they don’t have to be the same model or same size model as the main model.
To train the model, the loss was a cross-entropy loss that represented the log odds that someone would prefer one option over the other, in the pairwise comparison. This was done by taking the difference between the rewards of the preferred and unpreferred options. So in the example “What’s 2+2?,” if the reward model correctly assigns “4” a high reward and “Yes” a small reward, then the difference would be high and positive, and the loss would be small. However, if the reward model were to incorrectly assign “Yes” a higher reward than “4,” the difference would be high and negative, and the loss would be huge—discouraging it from outputting this result again.
One of the big challenges in training the reward model was overfitting, and OpenAI found that training for only 1 epoch would help prevent that.
Reward model data
The simplest way to get preference pairs is to generate two responses per prompt and have a labeler tell you which one was better. To make more efficient use of each prompt, instead the model would generate not 2 but 4–9 different responses per prompt that human labelers would rank from best to worst.
Rankings can be transformed into pairwise comparisons, so it was an efficient way to collect those preference pairs. A ranking of N responses yields N-choose-2 pairs. For example, a ranking of 4 responses results in 6 pairs, a ranking of 9 results in 36 pairs. That means with 33K prompts and 4–9 responses ranked per prompt, there would be 200K–1.2M pairwise comparisons used to train a separate reward model. That’s a lot of data, from relatively efficient data labeling.
This is a relatively efficient use of human annotations. Just compare it to SFT. It’s easier, cheaper, faster, and more reliable (higher agreement between people) than writing good responses from scratch, so this stage of human labeling wasn’t as tedious as in SFT.
However, using the pairs from rankings wasn’t straightforward in training. The reward model would overfit if they mixed the pairs randomly, even in just 1 epoch, because the pairs for a single prompt were highly correlated with each other. So instead, they would train all the pairs from the same prompt as one element in a batch, and normalize it. This was also computationally more efficient to run and score all the N responses at once together, e.g., just score 9 times and reuse those calculations in this pass, rather than 36 times for each pair if mixed into the dataset.
A quick note on terminology. This data is often called preference data, because it’s about collecting human preferences. The reward model can also be called a preference model.
Stage 3: RLHF (reinforcement learning with human feedback)
At this point, you have an SFT model that can follow instructions, and a reward model that can score responses. The goal of RL is to continue training the SFT model to produce responses that the reward model scores highly. If the reward model is any good, the resulting model will produce responses that humans would prefer.
The RL algorithm used was PPO. As you learned previously about RL terminology, the SFT model is the “policy” that takes actions (generating tokens) in an environment (the conversation). The reward model provides the reward after the policy generates a complete response, and a critic model calculates the expected reward, a baseline estimate that offers a more stable overall reward signal in training.
Here are the critical steps. I’ve covered some of them before and will dive into others in detail later on in this section.
- Sample a prompt. The prompt comes from the dataset of 31,000 prompts that were gathered from users organically using the API. No need for human labels.
- Generate a response. Then, the current policy generates a response. The current policy is the SFT model in the beginning, but as the policy updates, it’s a new model that generates responses to be graded. A single prompt-response pair is called a “rollout.” In practice, this all happens in a batch of rollouts.
- Calculate the reward. The reward model grades the full response with a reward. For every token position in the full response, they subtract a per-token KL divergence penalty between the current policy and the original SFT model. The per-token KL penalty and the reward model score added at the final token make up the per-token reward signal.
- Calculate the advantage. The critic estimates the expected reward for the full response, at each token position in generation (so with partial knowledge of the full response). The critic’s expected rewards and the per-token reward signals are combined, using an algorithm called GAE (Generalized Advantage Estimation), to estimate how much better the reward was compared to expected. This is the advantage. In InstructGPT, the critic was initialized with the same weights as the reward model, giving it a head start on estimating expected reward.
- Update the policy. Then, PPO updates the policy model’s weights with the advantage, pushing it towards rollouts with higher advantage and away from ones with lower advantage.
- (Optional) Mix in pretraining data and the pretraining objective in the policy model’s loss function to prevent catastrophic forgetting.
- Update the critic to better predict expected future rewards at each token position, by using the actual per-token rewards (from the reward model and KL penalty) as the targets in training.
- (Optional) Update the reward model. Collect new ranking data on the current best policy and train a new reward model. In practice, OpenAI did collect some data from the PPO models, but most was from the original SFT model.
- Repeat! This RL loop repeats over many prompts and many updates, with the latest policy always generating its responses.
The KL penalty
If you just let the model maximize the reward model’s score with no constraints, it finds weird, degenerate outputs that exploit quirks in the reward model to get high scores without actually being good responses. This is called “reward hacking,” and it’s one of the central problems in RLHF.
To address this, OpenAI added a KL divergence penalty between the RL policy and the original SFT model (“reference policy”) in the reward calculation. In AI, KL divergence is a common method of measuring how different two probability distributions are. In this case, it would measure how different the RL policy is from the old policy and penalize being too far from it, essentially telling the model: You can optimize for higher reward, but you can’t drift too far from where you started. If the RL model starts producing outputs that look nothing like what the SFT model would produce, the penalty helps to pull it back by making the reward for those outputs lower.
The total reward for a response becomes the reward model’s score minus the KL divergence from the SFT model, with a coefficient term that weighs how much to care about the KL divergence. If the coefficient is too low, you’re saying that you don’t need to penalize drift from the reference policy, and you’ll get reward hacking. Too high and the model barely changes from the SFT checkpoint.
They also mixed in a significant amount of pretraining data during the RL phase, adding a pretraining loss alongside the RL objective. This was to prevent the model from degrading on general tasks from pretraining, like knowledge recall or coherent long-form creative text, as it optimized for reward. This is sometimes called the “alignment tax,” where you trade-off alignment for general capabilities, a type of “catastrophic forgetting.” This is an active area of research.
So the final RL objective combined three things: (1) maximize the reward model’s score on prompted responses, (2) stay close to the SFT model via the KL penalty, and (3) maintain performance on pretraining data. This means improving on the things humans care about without losing what the model already knew how to do from SFT.
Practical details
The critic reduces noise and makes training stable enough to make PPO work practically. In practice, OpenAI initialized the critic from the 6B-parameter reward model, since it’s already trained to predict reward and gives the critic a head start as it is further trained in the RL loop.
The RL training was computationally expensive and involved running several models simultaneously: the policy model (the main model being trained), the critic (estimating the reward as a baseline, also being trained), the reward model (grading responses), and a copy of the SFT model (for computing KL divergence).
That’s four models in memory at once. That’s a lot of GPU memory, especially when the policy and SFT models are 175B parameters! In addition to weights, the policy and value models also needed their gradients, optimizer states, and cached activations for backpropagation because they were being trained, which can actually multiply the per-model memory cost by 3-4 times. This is another reason the reward and critic models were kept at 6B.
PPO also requires generating fresh rollouts during training, which is much slower than SFT where you already have all the data upfront. Each PPO training step also requires grading each rollout with the reward model, computing advantages with the critic, and updating both the policy and the critic. All these moving parts make the system harder to tune and debug compared to SFT, and harder to parallelize than pretraining.
Hyperparameters like learning rate, the KL penalty coefficient term, the number of rollouts per batch, and the clipping ratio all matter and interact with each other. The whole system depends on the quality and representativeness of your human annotations. Many RL training runs fail or produce degenerate results.
Getting it right
Getting all this right takes significant engineering effort and experience, but the first step is deeply understanding the pieces. Modern methods have addressed many of these issues, but the ideas from InstructGPT remain the foundation of post-training today.
Ultimately, human evaluators compare all the models. People preferred the RLHF model’s outputs over the SFT model’s, and the SFT model’s over base GPT-3’s. Each stage of the pipeline added a large improvement in the model’s response quality.
RLHF was extremely effective. In experiments, human evaluators preferred even a tiny 1.3B parameter RLHF model over the 175B parameter SFT model, most of the time. That’s a model over 100x smaller, trained with RL, beating a much larger model trained only with SFT. This made a strong case that how you train matters as much as how big your model is. Overall, the largest RLHF model still beat the smaller RLHF model.
The RLHF model was also better at following explicit constraints in instructions, less likely to produce harmful outputs, and hallucinated less, though it didn’t eliminate hallucinations as you may remember when you first used ChatGPT (and even now).
One caveat worth noting: The labelers who evaluated the final model were the same population who created the training data. When they tested with held-out labelers who hadn’t been involved in data creation, preferences for the RLHF model were still positive but less dramatic. The model was, to some degree, optimized for the preferences of a specific group of people. This means if you create the data to follow your preferences, the model will optimize for those.
Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >
Log in
