Benchmark chart, parameter count, and a quick scan of the release notes are usually enough to form an opinion about a new model. Somewhere in the release notes, there is usually a short paragraph on post-training that is easy to miss. Almost no one reads it, even though this exact part describes how the model behaves – what it rejects, how it reasons, and the tone it adopts by default.

Two labs can start from the same base checkpoint and end up with products that behave nothing alike. The difference lives almost entirely in what happens after pre-training.

Frontier labs have treated post-training methods such as RLHF, RLAIF, and DPO as competitive secrets. Open-source projects such as Tülu and RewardBench started to unpack that process by publishing the full methodology, including data, code, and results.

This article explains what post-training actually does, why it matters, and how projects like Tülu and RewardBench are making this process more transparent.

What is LLM post-training and why does it matter as much as architecture

Large language models are usually described by the data they were trained on, the number of parameters they contain, and the architecture they use. Those characteristics shape what a model can learn. They don't fully explain how it behaves once people start talking to it.

That behavior comes from post-training, a second stage. Nathan Lambert, who leads post-training research at Ai2 and wrote the RLHF book, divides post-training into three approaches:

• Supervised fine-tuning (SFT) – teaches the model the shape of a conversation.
• Preference fine-tuning – uses human or AI feedback to shape tone and style, through methods such as RLHF, DPO or RLAIF.
• Reinforcement learning with verifiable rewards (RLVR) – the newest addition, which trains models using automatically verifiable outcomes, such as correct math answers or code that passes automated tests.

The easiest way to see this in action is to watch a raw base model try to answer a question. Ask it, "How do I write a professional resignation letter?", and it might simply repeat the question, give an incomplete answer, or continue the text verbatim. The model's knowledge is fine; it just has no sense that it's supposed to stop and answer.

Ask an instruction-tuned model version the same question, and it responds with a structured letter, adopting an appropriate tone.

Lambert calls this the "elicitation" view. That means most of what the model needs is already in there from pre-training, and post-training's job is mostly extraction and drawing it out into a conversational form. This framing is more useful than most of the alternatives, though this is still a contested read of what's actually happening under the hood.

RL-based training is absorbing a growing share of the compute used to develop frontier models. DeepSeek R1, released in early 2025, spent about 5% of its overall training compute on reinforcement learning. A 2026 study of Ai2's OLMo 3 illustrates how quickly those costs can grow: the reasoning variant ("Think") consumed 17 times more datacenter energy during post-training than its instruction-tuned counterpart. The difference came almost entirely from the cost of generating RL rollouts at scale. Pre-training still takes up most of the budget, but the balance is shifting as post-training requires more compute.

RLHF – the original post-training paradigm

When ChatGPT demonstrated a new level in language models, RLHF (Reinforcement Learning from Human Feedback) was one of the key techniques behind that. It still shapes most people's understanding of what post-training even means. For the first time, it became possible to train large language models directly on human preferences at scale.

The classic RLHF pipeline consists of three stages. First, a base LLM is fine-tuned on instruction-response pairs. Then, it is trained on human comparisons, learning which of its actions lead to better outcomes. Here is what each stage does:

1. Supervised fine-tuning (SFT) trains a model on input-output pairs. The pretrained LLM learns to reproduce the target responses by minimizing the loss between predictions and the examples. This helps it follow instructions more consistently and match the desired style and format. In practice, SFT involves several steps, including data preprocessing, tokenization, and loss computation.
2. Reward model training learns to predict human preferences by analyzing preferred and non-preferred responses. It assigns higher scores to responses that better match the desired behavior. The model does not memorize these rankings, but rather learns a generalizable function that can then score new responses. The process typically involves preprocessing the preference data and computing the training loss.
3. Reinforcement learning (typically Proximal Policy Optimization, or PPO) helps the model respond in ways people like better. The model generates an answer, and the reward model gives it a score based on human preferences. The system then uses that score as feedback, adjusting to do better next time, but only in small steps, so it doesn't drift too far from what it learned during SFT. This is enforced by the KL (Kullback-Leibler) divergence penalty, which punishes the model for straying too far from its SFT behavior.

In practice, RLHF has some downsides too. For one, collecting consistent, high-quality comparisons based on human preferences takes a lot of time and money. But that's only half the cost of RLHF. The other half shows up once training actually begins.

Unlike SFT, RLHF doesn't just work off a fixed dataset. The model keeps generating new responses as it trains, exploring different ways to answer. To make that possible, you need several models running at the same time: the model being trained, a frozen copy of it for comparison, the reward model, and usually a value network too. With all of that running together, things can get unstable.

And if you've worked with RLHF, you've probably run into reward hacking – that's when the model finds some loophole in the reward model and exploits it instead of actually getting better. It's not just an occasional glitch; reward hacking tends to get worse as models get more capable.

DPO – the simplification that took over open source

Direct Preference Optimization (DPO), introduced by Rafailov et al. in 2023, simplifies RLHF by replacing the reward model and RL stages with a single training step. Given a preferred and a rejected response to the same prompt, DPO trains directly on that pair, without using a separate reward model or RL loop. Let’s see more closely how it works:

1. Response generation. The model (or a reference version of it) produces two responses to the same prompt.
2. Preference labeling. Annotators (human or AI) mark one response as preferred and the other as rejected, the same raw material RLHF's reward model would train on.
3. Direct optimization. The model's own probabilities for the preferred and rejected responses become the training signal. A single loss function raises the probability of the preferred response and lowers the probability of the rejected one.

The key idea behind DPO is that it learns directly from the difference between preferred and rejected responses. Instead of training a separate reward model and then running reinforcement learning, DPO uses the policy’s probabilities to determine which response to favor. This makes preference optimization look much more like supervised fine-tuning.

Compared with RLHF, this approach offers several practical advantages:

• Fewer models involved. You only need the policy itself and a frozen reference model, and this is roughly half of what a full RLHF setup requires.
• No RL loop. DPO removes the least stable piece of the entire pipeline.
• A lower expertise bar. Teams no longer need deep experience debugging reinforcement learning.

On the TL;DR summarization benchmark, DPO reached around a 61% win rate against human-written summaries, beating PPO's best-case performance of 57%. DPO also stayed more consistent across different temperature settings.

Thanks to this, DPO became the main post-training method for open-source projects, including Zephyr-Beta and the Tülu family. The path wasn't smooth: early versions like Zephyr-Beta and Tülu 2 struggled until researchers found the right learning rate. Once they did, within a year DPO went from an idea to a standard part of competitive open-source training.

The disadvantage of DPO is that it trains on a fixed, offline set of preference pairs. It doesn't generate new responses during training, so it cannot learn from how its own behavior evolves. For most instruction-following tasks, that limitation doesn't matter much. But as reasoning models become more capable, this matters more: discovering new strategies requires more than choosing between two pre-written responses. That's why the frontier labs still rely on online RL, even as DPO remains a popular choice for many other applications.

RLAIF – scaling feedback with AI judges

Reinforcement Learning from AI Feedback (RLAIF) follows the same basic pipeline as RLHF but uses synthetic feedback instead of human. The term goes back to Anthropic's Constitutional AI research, where an LLM critiques and revises its own outputs against a written set of principles. Those judgments are then used to create preference data, which can be used with RLHF or DPO.

The main advantage of RLAIF is cost. According to Nathan Lambert's RLHF Book, a single piece of human preference data typically costs $1 or more per prompt as of 2026, and can cost over $10 for specialized or technical tasks. AI feedback from a frontier model can cost a fraction of a cent per judgment. That difference makes large-scale preference data much more accessible to smaller teams and researchers.

The cheaper feedback can also be surprisingly effective. In a 2023 study, researchers compared RLHF vs RLAIF and found that models trained on AI-labeled preferences performed roughly as well as those trained on human-labeled preferences when evaluated by humans. The result helped establish RLAIF as more than just a cheaper alternative to human annotation.

But AI feedback is not a direct substitute for human judgment. It has different failure modes. Lambert describes human data as high-noise but low-bias, and synthetic preference data as low-noise but high-bias. AI feedback is cheap and consistent, but it can also reproduce the systematic blind spots of the model providing the feedback. Those biases can be difficult to detect because the resulting signal is often clean and consistent.

Without an external reference point, the model can end up reinforcing behaviors that it prefers rather than ones that are actually useful. In practice, most teams use both sources. AI feedback handles large volumes of easier judgments, while humans are involved in complex or safety-sensitive cases.

RLHF, DPO, and RLAIF differ in how they generate feedback, optimize responses, and balance scalability with human oversight. The table below highlights the key differences.

RLHF vs DPO vs RLAIF

Checking whether any of this worked: RewardBench

In reinforcement learning, the final result is only as good as the reward signal guiding it. The point is that reward models (RM) don't produce text for people to simply read and evaluate. They assign a higher score to better responses and a lower score to weaker ones. If that scoring function is wrong in some way, the LLM will optimize for the wrong target. The responses can become more confident but less truthful, sound safer while becoming more evasive, or grow longer without becoming more useful.

A slightly miscalibrated RM can teach the policy the wrong behavior even if the training metrics continue to improve. And nothing necessarily signals that the optimization is heading in the wrong direction.

The RewardBench project was designed to resolve this problem. Created by Nathan Lambert and his collaborators at Ai2 (the Allen Institute for AI), RewardBench tests where RMs struggle, including refusal behavior, reasoning quality, and instruction following.

RewardBench is an open-source benchmark that provides ready-to-use inference code for dozens of RM types (DPO, sequence classifiers, generative LLM-as-judge). It uses a single, standardized dataset format. Instead of writing a custom evaluation pipeline from scratch, a user can plug the model into infrastructure that's already been tested. The earlier scripts calculated scores across four categories: Chat, Chat Hard, Safety, and Reasoning. RewardBench v2 uses different ones: Factuality, Precise Instruction Following, Math, Safety, Focus, and Ties. Results can be uploaded directly to the HuggingFace leaderboard for a fair comparison.

RewardBench provided the open-source community with a single way to evaluate RMs that were previously difficult to compare. It identified weaknesses in existing models and preference datasets, and gave researchers a shared target to improve upon. This made RM development more measurable and reproducible.

Case study – Tülu 2 and Tülu 3: open post-training, in public

The race between RLHF, DPO, and RLVR wasn't only happening at closed labs. The Allen Institute for AI (Ai2) tracked it step by step, in public, from the earliest days of DPO to the arrival of RLVR.

Tülu is an ongoing experiment in open-source post-training. Ai2 publishes the models, datasets, training code, and evaluation tools needed to reproduce the full pipeline.

Tülu 2 arrived in the fall of 2023, at the moment when DPO was still an unproven idea. Built by fine-tuning Llama 2 70B, Tülu 2 demonstrated that preference-tuning could be done effectively outside closed labs and established DPO as a viable alternative to full RLHF.

Tülu 3, released about a year later, is an even more ambitious version of the same idea. Built on Llama 3.1, it runs a four-stage pipeline:

• Prompt curation – selects and builds the training prompts
• Supervised fine-tuning (SFT) – establishes base instruction-following behavior
• Preference tuning – trains on large-scale on-policy preference data
• RLVR – checks correctness automatically on objective tasks like math and code

As a result, it outperformed the official instruction-tuned variants of Llama 3.1, Qwen 2.5, and Mistral on key benchmarks. Incredibly, this was achieved by a team of just 10–15 people.

What separates Tülu 3 from a merely good open model is the quality and transparency of the post-training recipe. Tülu remains the clearest public evidence that a strong base LLM doesn't need to be far larger to compete.

Where this is heading: RLVR and reasoning models

The newest layer in the stack is Reinforcement Learning with Verifiable Rewards (RLVR). It is associated with reasoning models such as OpenAI's o1 and DeepSeek R1. Instead of relying on a human or AI judge, RLVR evaluates responses using objective criteria: known answers for math problems, unit tests for code, and predefined rules for structured outputs. The system is rewarded when its response passes the check.

This removes one of the biggest problems with RM quality that we discussed above. But the method only works well where correctness can actually be verified. In areas like math and coding, this signal is much cleaner than the subjective preferences of a human or an AI.

During training, the policy generates multiple answers, identifies which ones are correct, increases the probability of those responses, and repeats the cycle. When longer reasoning produces the right answer, RLVR reinforces this. Over many iterations, this leads to the extended reasoning patterns seen in models such as o1 and DeepSeek R1.

A single RLVR training run is relatively cheap compared to pretraining. But this doesn't mean cheap overall – RLVR pipelines get iterated, scaled, and tuned repeatedly. That repeated iteration is where frontier labs are now investing.

Conclusion

Post-training shapes how a model applies the capabilities it developed during pre-training. This affects how it responds, reasons, and follows instructions.

RLHF, RLAIF, DPO, and now RLVR give researchers different ways to turn a capable base LLM into a more useful one. No single method does everything – in practice, labs mix and tune them as the field evolves. That is why projects like Tülu and RewardBench are important. They make post-training transparent enough for others to evaluate and improve.

For a deeper look, Nathan Lambert's RLHF Book, Ai2's Tülu project, and RewardBench provide some of the best public resources on how modern post-training works. For more analysis on AI, training, and the infrastructure behind them, subscribe to the Propagated.ai newsletter.

FAQ

RLHF vs DPO – what is the difference?

RLHF is the original post-training technique behind ChatGPT. It first trains a reward model on human preference data, then uses reinforcement learning (typically PPO) to optimize the policy against that reward model's scores.

DPO learns directly from preference pairs through one training stage. This method is now widely used in open-source projects due to a simpler and cheaper training pipeline.

Is RLAIF replacing human feedback?

Only for high-volume, routine tasks. RLAIF uses LLMs as evaluators, so its effectiveness depends on their quality. AI judges can still introduce systematic biases and blind spots; that is why most frontier training pipelines combine AI-generated feedback with human annotations.

How to use RewardBench?

There are two ways, depending on your goal:

To compare existing RMs, check the public leaderboard on Hugging Face, which ranks them across categories such as Math, Safety, and Factuality, and shows an average score across them.

To evaluate your own LLM reward model, run Ai2's open-source evaluation harness on the benchmark's preference pairs. To add it to the leaderboard, open an issue on GitHub with the model's name on Hugging Face. Find full installation instructions in the RewardBench GitHub repository.

Is post-training more important than model size?

Not necessarily. Size still matters, but it doesn't determine how that knowledge gets used. Post-training is what teaches an LLM to follow instructions, reason through tasks, respond safely, and interact naturally with people. Done well, it can let a smaller model outperform a larger one on specific tasks, especially instruction following and reasoning. In practice, size and post-training work together.

What is RLVR in AI?

RLVR (Reinforcement Learning with Verifiable Rewards) trains LLMs using objectively correct outcomes. Unlike preference-based methods, it does not require human or AI judges to assess each response. It avoids some of the problems associated with subjective reward models and gives reinforcement learning a clearer signal.

This makes RLVR particularly effective for training reasoning models, where correctness can be checked automatically, such as mathematics or programming.