Log in
This guide is for developers and ML practitioners who have some experience with OpenAIʼs APIs and wish to use their fine-tuned models for research or other appropriate uses. OpenAI’s services are not intended for the personalized treatment or diagnosis of any medical condition and are subject to our applicable terms.
This guide discusses fine-tuning methods supported by OpenAI, specifically highlighting what each method is best for and not best for, to help you identify the most suitable technique for your use case. It then provides an in-depth look at one particular method — Direct Preference Optimization (DPO) — and provides links to existing guides for the other techniques.
What is fine-tuning? Fine-tuning is the process of continuing training on a smaller, domain-specific dataset to optimize a model for a specific task. There are two main reasons why we would typically fine-tune:
- Improve model performance on a specific task
- Improve model efficiency (reduce the number of tokens needed, distill expertise into a smaller model, etc.)
Currently, the OpenAI platform supports four fine-tuning methods:
- Supervised fine-tuning (SFT): this technique employs traditional supervised learning using input-output pairs to adjust model parameters. The training process adjusts model weights to minimize the difference between predicted and target outputs across the provided examples. The model will replicate features that it finds in provided pairs.
- Vision fine-tuning: this technique extends supervised fine-tuning to multimodal data by processing both text and image in a unified training framework. The training process adjusts model weights to minimize errors across text-image pairs and as a result improve the model's understanding of image inputs.
- Direct preference optimization (DPO): this technique uses pairwise comparisons (e.g., preferred and rejected example responses) to optimize a model to favor certain outputs over others. The model learns to replicate the preference patterns found in the provided comparison data.
- Reinforcement fine-tuning (RFT): this technique uses reinforcement learning with a reward signal (via a grader or reward model) to fine-tune the model for complex objectives. In RFT, the model generates outputs for given prompts during training, and each output is evaluated for quality. The model's parameters are then updated to maximize the reward, reinforcing behaviors that lead to better outcomes. This iterative feedback loop encourages the model to improve reasoning or decision-making strategies.
To help you select the appropriate fine-tuning technique, the table below summarizes the scenarios each method is best suited for, as well as those for which it is not well suited:
| Technique | Good For | Not Good For |
|---|---|---|
| Supervised fine-tuning (SFT) | Emphasizing knowledge already present in the model. Customizing response structure or tone. Generating content in a specific format. Teaching complex instructions or correcting instruction-following failures. Optimizing cost/latency (saving tokens from prompt or distilling). | Adding entirely new knowledge (consider RAG instead). Tasks with subjective quality. |
| Vision fine-tuning | Specialized visual recognition tasks (e.g., image classification). Domain-specific image understanding. Correcting failures in instruction following for complex prompts. | Purely textual tasks. Generalized visual tasks without specific context. General image understanding. |
| Direct preference optimization (DPO) | Aligning model outputs with subjective preferences (tone, politeness). Refining outputs via human-rated feedback. Achieving nuanced behavioral alignment. | Learning completely new tasks. Tasks without clear human preference signals. |
| Reinforcement fine-tuning (RFT) | Complex domain-specific tasks that require advanced reasoning. Refining existing partial capabilities (fostering emergent behaviours). Tasks with measurable feedback. Scenarios with limited explicit labels where reward signals can be defined. | Tasks where the model has no initial skill. Tasks without clear feedback or measurable signals. |
Today, there are pre-existing Cookbooks for:
- Supervised fine-tuning (SFT): (1) How to fine-tune chat models (2) Leveraging model distillation to fine-tune a model
- Vision fine-tuning: Vision fine-tuning on GPT-4o for visual question answering
- Reinforcement fine-tuning (RFT): (1) Reinforcement fine-tuning (RFT), (2) Reinforcement fine-tuning for healthbench QA
Direct preference optimization (DPO) will be covered in this guide.
As mentioned above, Direct Preference Optimization (DPO) is an alignment technique for fine-tuning language models using pairwise preference data (e.g., ranked pairs of responses). DPO directly optimizes a model to favor certain outputs over others using explicit pairwise comparisons, typically from human preferences. This approach simplifies alignment and eliminates the need for a separate reward model or complex reinforcement learning procedures, making DPO a lightweight alternative to techniques such as Reinforcement Learning from Human Feedback (RLHF).
When should you use DPO? DPO excels in scenarios when response quality is subjective, cannot be measured objectively, or when nuanced criteria such as tone, style, appropriateness, or clarity matter - typically cases where multiple valid outputs exist. Example applications where DPO is particularly effective in aligning AI responses include:
- Enhancing Conversational AI Responses
- Improving Code Generation Quality & Style
- Ensuring Compliance with Legal, Ethical & Safety Standards
- Controlling Brand Voice, Professionalism, & Tone
- Customizing Creative Outputs & User Experience
By fine-tuning on explicit pairs of preferred vs non‑preferred completions, DPO aligns model outputs to these nuanced preferences. The below table gives examples of pairwise preference data for a fictional AI assistant that represents an organization, where preferred responses are clear, professional, and aligned with brand standards.
| Example Question | Chosen Response | Rejected Response |
|---|---|---|
| Q1: How do I review your product? | To submit a product review, please visit your account dashboard, select the product, and click ‘Write a review.’ Share your honest experience, rate key features, and submit when ready. | Yo, just leave some quick stars or whatever, it’s chill! |
| Q2: How do I review your product? | We welcome your feedback! In the ‘Reviews’ section on the product page, click ‘Leave a Review,’ rate it, and add your comments about what you liked or areas for improvement. | Just scribble something—doesn’t matter what, honestly. |
| Q3: How to troubleshoot this particular error? | To address the error ‘X101,’ first clear your cache, then verify your internet connection. If the issue remains, follow our step-by-step guide at [Support → Troubleshooting → Error X101]. | Just reboot it, I guess. If it doesn't work, you're on your own! |
In this guide, weʼll walk through how to apply DPO using the fine-tuning API. You will learn key steps to take in order to successfully run preference fine-tuning jobs for your use-cases.
Here’s what we’ll cover:
- 1. Recommended Workflow
- 2. Demonstration Scenario
- 3. Generating the Dataset
- 4. Benchmarking the Base Model
- 5. Fine-Tuning
- 6. Using your Fine-Tuned Model
OpenAI recommends the following workflow:
- Performing Supervised Fine-Tuning (SFT) on a subset of your preferred responses.
- Using the SFT fine-tuned model as the starting point, apply DPO using preference comparison data.
Performing Supervised Fine-Tuning (SFT) before Direct Preference Optimization (DPO) enhances model alignment and overall performance by establishing a robust initial policy, ensuring the model already prefers correct responses. This reduces the magnitude of weight updates during DPO, stabilizing training and preventing overfitting by allowing DPO to efficiently refine subtle nuances. Consequently, the combined SFT-then-DPO workflow converges faster and yields higher-quality results.
In this guide, we'll focus exclusively on applying Direct Preference Optimization (DPO). However, depending on your use case, you may find performance gains from first performing Supervised Fine-Tuning (SFT). If so, you can follow the SFT guide linked above, save the resulting model ID, and use that as the starting point for your DPO job.
To make things concrete, let’s walk through fine-tuning a customer-facing AI assistant to follow a fictional brand’s voice and style. Imagine Good Vibes Corp, an organization that prides itself on a friendly, enthusiastic tone with a personal touch.
They want their customer AI assistant to answer queries in a way that reflects these brand guidelines (e.g. an upbeat attitude, polite language, and a friendly sign-off), and prefer those responses over more generic or curt answers. This is a good scenario for DPO: there’s no objectively correct answer format, but there is a preferred style.
DPO will help the model learn from comparisons which style is preferred. We'll outline the steps to: (1) generate a synthetic preference dataset of prompts with paired responses (one in the desired brand voice and one not). (2) Evaluate base model performance using the OpenAI evals API. (3) Prepare and upload the data in the required JSONL format for preference fine-tuning. (4) Fine-tune the model with DPO using the OpenAI fine-tuning API. (5) Evaluate the fine-tuned model using the OpenAI evals API to show how the brand-style preference improved.
We are going to synthesize a dataset for this demonstration. First, let’s create a seed bank of questions to generate more variations from.
Let’s get started!
! pip install openai nest-asyncio --quietPROMPT_SEED_POOL = [
"Hi, I ordered a gadget last week. When will it arrive?",
"Your product stopped working after two days. Can I get help?",
"Do you offer discounts for long-term customers?",
"Can I change the shipping address for my order?",
"What is your return policy for damaged items?",
"My tracking number hasn't updated in three days—can you check the status?",
"How long is the warranty on your products, and how do I submit a claim?",
"Can I add gift wrapping to my order before it ships?",
"Do you accept PayPal or other alternative payment methods?",
"Is there an option to expedite shipping if my order hasn't left the warehouse yet?",
]Next, we’ll define functions to take each prompt from our seed bank and generate related questions. We’ll create a dataset of preference pairs by first generating these prompt variations, then producing both a preferred and a rejected response for every prompt.
This dataset is synthetic and serves to illustrate the mechanics of Direct Preference Optimization — when developing your own application you should collect or curate a high-quality, preference dataset. Note: the volume of data required for DPO depends on the use case; generally more is better (thousands to tens of thousands), and for preference pairs the ordering logic should be consistent (e.g. if A > B and B > C, then A > C).
import asyncio
from openai import AsyncOpenAI
from typing import List, Dict, Any
async_client = AsyncOpenAI()
SYSTEM_PROMPT = "You are a customer-support assistant."
async def _generate_related_questions_from_prompt(
prompt: str, k: int, sem: asyncio.Semaphore, *, model: str
) -> List[str]:
"""Return *k* distinct customer-service questions related to the given prompt."""
out: List[str] = []
async with sem:
for _ in range(k):
resp = await async_client.responses.create(
model=model,
input=[
{
"role": "system",
"content": (
"Return ONE distinct, realistic customer-service question "
"related in topic or theme to the following question, "
"but NOT a direct paraphrase."
),
},
{"role": "user", "content": prompt},
],
temperature=0.9,
max_output_tokens=60,
)
out.append(resp.output_text.strip())
return out
async def expand_prompt_pool(
prompts: List[str], *, k: int = 3, concurrency: int = 32, model: str
) -> List[str]:
"""Expand each prompt into *k* related questions using the given model."""
sem = asyncio.Semaphore(concurrency)
tasks = [
_generate_related_questions_from_prompt(p, k, sem, model=model) for p in prompts
]
results = await asyncio.gather(*tasks)
return [v for sub in results for v in sub]
async def _generate_preference_pair(
prompt: str, sem: asyncio.Semaphore, *, model: str
) -> Dict[str, Any]:
"""Generate a preference pair for the given prompt."""
async with sem:
friendly_task = async_client.responses.create(
model=model,
input=[
{
"role": "system",
"content": (
"You are Good Vibes Corp's exceptionally energetic, outrageously friendly and "
"enthusiastic support agent."
),
},
{"role": "user", "content": prompt},
],
temperature=0.7, # higher temperature to increase creativity & on-brand tone adherence
max_output_tokens=80,
)
blunt_task = async_client.responses.create(
model=model,
input=[
{
"role": "system",
"content": "You are a terse, factual support agent with no empathy or politeness.",
},
{"role": "user", "content": prompt},
],
temperature=0.3, # lower temperature to limit creativity & emphasize tonal difference
max_output_tokens=80,
)
friendly, blunt = await asyncio.gather(friendly_task, blunt_task)
return {
"input": {
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt},
]
},
"preferred_output": [
{"role": "assistant", "content": friendly.output_text}
],
"non_preferred_output": [
{"role": "assistant", "content": blunt.output_text}
],
}Now, using these defined functions we'll build our dataset by generating friendly versus blunt response pairs. The friendly responses reflect the brand's desired communication style. We'll do this asynchronously for efficiency, creating a dataset suited for Direct Preference Optimization.
import math
import nest_asyncio
async def build_dataset(
*,
pair_count: int = 500,
concurrency: int = 8,
expand_prompt_pool_model: str,
generate_preference_pair_model: str,
) -> List[Dict[str, Any]]:
"""Return *pair_count* preference pairs (single-shot expansion)."""
seed = PROMPT_SEED_POOL
deficit = max(0, pair_count - len(seed))
k = max(1, math.ceil(deficit / len(seed)))
expanded = await expand_prompt_pool(
seed,
k=k,
concurrency=concurrency,
model=expand_prompt_pool_model,
)
prompt_bank = (seed + expanded)[:pair_count]
sem = asyncio.Semaphore(concurrency)
tasks = [
_generate_preference_pair(p, sem, model=generate_preference_pair_model)
for p in prompt_bank
]
return await asyncio.gather(*tasks)
nest_asyncio.apply()
pairs = await build_dataset(
pair_count=500,
concurrency=8,
expand_prompt_pool_model="gpt-4.1-mini-2025-04-14",
generate_preference_pair_model="gpt-4.1-mini-2025-04-14",
)
print(f"Dataset ready with {len(pairs)} pairs.")Dataset ready with 500 pairs.
Below, we split our dataset into training, validation, and testing sets. We also show a sample from the training dataset, which demonstrates a clear difference between the preferred (friendly, on-brand) and non-preferred (blunt, neutral) responses for that input pair.
# set dataset sizes
n = len(pairs)
n_train = int(0.8 * n)
n_val = int(0.1 * n)
n_test = n - n_train - n_val
# split dataset into train, test & validation
train_pairs = pairs[:n_train]
val_pairs = pairs[n_train : n_train + n_val]
test_pairs = pairs[n_train + n_val :]
train_pairs[0]{'input': {'messages': [{'role': 'system',
'content': 'You are a customer-support assistant.'},
{'role': 'user',
'content': 'Hi, I ordered a gadget last week. When will it arrive?'}]},
'preferred_output': [{'role': 'assistant',
'content': 'Hey there, awesome friend! 🌟 Thanks a bunch for reaching out! I’d LOVE to help you track down your gadget so you can start enjoying it ASAP! 🎉 Could you please share your order number or the email you used to place the order? Let’s make this delivery magic happen! 🚀✨'}],
'non_preferred_output': [{'role': 'assistant',
'content': 'Provide your order number for delivery status.'}]}To assess the model's performance prior to fine-tuning, we'll use an automated grader (LLM-as-a-Judge) to score each response for friendliness and empathy. The grader will assign a score from 0 to 4 for each answer, allowing us to compute a mean baseline score for the base model.
To do this, we first generate responses for the base model on the test set, then use the OpenAI evals API to create and run an evaluation with an automated grader.
async def generate_responses(
testset,
model,
temperature=0.0,
max_output_tokens=80,
concurrency=8,
):
"""
Generate responses for each prompt in the testset using the OpenAI responses API.
Returns: List of dicts: [{"prompt": ..., "response": ...}, ...]
"""
async_client = AsyncOpenAI()
sem = asyncio.Semaphore(concurrency)
async def get_response(prompt):
async with sem:
resp = await async_client.responses.create(
model=model,
input=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt},
],
temperature=temperature,
max_output_tokens=max_output_tokens,
)
return {"prompt": prompt, "response": resp.output_text}
tasks = [get_response(item["item"]["input"]) for item in testset]
results = await asyncio.gather(*tasks)
return results
# generate responses for the base model over the test set
base_model = "gpt-4.1-mini-2025-04-14"
testset = [
{"item": {"input": pair["input"]["messages"][1]["content"]}} for pair in test_pairs
]
responses = await generate_responses(testset, model=base_model)Next, we'll use the OpenAI evals API to create & run an evaluation with an automated grader, starting by defining the rubric for the LLM-as-a-Judge. Note: we will access responses via data logging, so in order for this to work, you'll need to be in an org where data logging isn't disabled (through zdr, etc.). If you aren't sure if this is the case for you, go to https://platform.openai.com/logs?api=responses and see if you can see the responses you just generated.
JUDGE_SYSTEM = """
You judge whether a reply matches Good Vibes Corp's desired tone:
energetic, super-friendly, enthusiastic.
Score 0-4 (higher = more energy):
4 - Highly enthusiastic: multiple upbeat phrases / emojis / exclamations, clear empathy, proactive help.
3 - Energetic & friendly: visible enthusiasm cue (≥1 emoji OR exclamation OR upbeat phrase), warm second-person tone.
2 - Pleasant: polite & positive but lacks obvious enthusiasm cues.
1 - Neutral: correct, businesslike, minimal warmth.
0 - Rude, negative, or unhelpful.
"""from openai import OpenAI
sync_client = OpenAI()
# set judge model
judge_model = "gpt-4.1-2025-04-14"
# create the evaluation
logs_eval = sync_client.evals.create(
name="Good Vibes Corp Tone Eval",
data_source_config={
"type": "logs",
},
testing_criteria=[
{
"type": "score_model",
"name": "General Evaluator",
"model": judge_model,
"input": [
{
"role": "system",
"content": JUDGE_SYSTEM,
},
{
"role": "user",
"content": (
"**User input**\n"
"{{item.input}}\n"
"**Response to evaluate**\n"
"{{sample.output_text}}"
),
},
],
"range": [0, 4],
"pass_threshold": 2,
}
],
)# run the evaluation
base_run = sync_client.evals.runs.create(
name=base_model,
eval_id=logs_eval.id,
data_source={
"type": "responses",
"source": {"type": "responses", "limit": len(test_pairs)},
},
)# score base model
base_data = sync_client.evals.runs.output_items.list(
eval_id=logs_eval.id, run_id=base_run.id
).data
base_scores = [s.results[0]["score"] for s in base_data]
print("Average score:", sum(base_scores) / len(base_scores))With a baseline established, we can now fine-tune the model using the training set and DPO. This process will teach the model to prefer responses that align with our desired style, based on the preference pairs we created earlier.
Note: beta (β) is a unique fine-tuning hyperparameter for Direct Preference Optimization (DPO). It’s a floating-point number ranging between 0 and 2, controlling the balance between preserving a model’s existing behavior and adapting to new, preference-aligned responses.
- High β (close to 2): makes the model more conservative, strongly favoring previous behavior. The fine-tuned model will show minimal deviations from its original style or characteristics, emphasizing consistency and avoiding abrupt changes.
- Moderate β (around 1): balances between adherence to prior behavior and adaptation to new preferences. Recommended as a sensible starting point for most practical scenarios.
- Low β (close to 0): encourages aggressive adaptation, causing the model to prioritize newly provided preferences more prominently. This might result in significant stylistic shifts and greater alignment with explicit preferences but could lead to unexpected or overly specialized outputs.
Technically, beta scales the difference in log-probabilities in the DPO loss; a larger β causes the sigmoid-based loss function to saturate with smaller probability differences, yielding smaller weight updates (thus preserving old behavior). It is recommended to experiment systematically with the β value to achieve optimal results tailored to your specific use-case and desired trade-offs between stability and adaptation.
import io
import json
# create training file
train_buf = io.BytesIO("\n".join(json.dumps(p) for p in train_pairs).encode())
train_buf.name = "train.jsonl"
train_file_id = sync_client.files.create(file=train_buf, purpose="fine-tune").id
# create validation file
val_buf = io.BytesIO("\n".join(json.dumps(p) for p in val_pairs).encode())
val_buf.name = "val.jsonl"
val_file_id = sync_client.files.create(file=val_buf, purpose="fine-tune").id
# create a fine-tuning job
ft = sync_client.fine_tuning.jobs.create(
model=base_model,
training_file=train_file_id,
validation_file=val_file_id,
method={
"type": "dpo",
"dpo": {
"hyperparameters": {
"n_epochs": 2,
"beta": 0.1,
"batch_size": 8,
}
},
},
)
print(f"Fine-tuning job created: job_id = {ft.id}")Fine-tuning job created: job_id = ftjob-5QPmA36QezFRGoXjuvIAPuAQ
Once fine-tuning is complete, we'll evaluate the DPO-tuned model on the same test set. By comparing the mean scores before and after fine-tuning, as well as reviewing example outputs, we can see how the model's alignment with our preferences has improved.
# generate responses
job = sync_client.fine_tuning.jobs.retrieve(ft.id)
if job.status == "succeeded":
responses = await generate_responses(testset, model=job.fine_tuned_model)
post_run = sync_client.evals.runs.create(
name=ft.id,
eval_id=logs_eval.id,
data_source={
"type": "responses",
"source": {"type": "responses", "limit": len(test_pairs)},
},
)# get scores from the evaluation
post_data = sync_client.evals.runs.output_items.list(
eval_id=logs_eval.id, run_id=post_run.id
).data
post_scores = [s.results[0]["score"] for s in post_data]
# print scores & a sample comparison from the test set for illustration
print(
"Δ mean:",
sum(t - b for b, t in zip(base_scores, post_scores)) / len(base_scores),
)
print("\n=== SAMPLE COMPARISON ===")
idx = 0
print(f"Prompt:\n {testset[idx]['item']['input']}\n")
print(f"Base model reply: \n {base_data[idx].sample.output[0].content} \n")
print(f"DPO-tuned model reply \n {post_data[idx].sample.output[0].content}")Δ mean: 0.45 === SAMPLE COMPARISON === Prompt: Can I upgrade to faster delivery if my package is still being processed? Base model reply: Whether you can upgrade to express shipping while your order is still being processed depends on the store's policies. Generally, many stores allow shipping upgrades before the order is shipped. To assist you better, could you please provide your order number or the name of the store you ordered from? Alternatively, you can contact the store's customer service directly to request the upgrade. DPO-tuned model reply Hi! I’d be happy to help with that. If your package hasn’t shipped yet, there’s a good chance we can upgrade your delivery speed. Could you please provide me with your order number? I’ll check the status and let you know the available options for faster delivery.
