Also known as: instruction fine-tuning, instruct tuning, SFT for instructions
Instruction tuning is fine-tuning a pre-trained language model on (instruction, response) pairs so it learns to follow directions. The step that turns 'GPT-base' into 'GPT-instruct'.
A pre-trained base LLM, fresh from next-token prediction on web-scale text, doesn’t follow instructions. Ask it “What is the capital of France?” and it might continue with “is a question often asked by tourists. The capital of France is…” — completing the prompt as text rather than answering it. Instruction tuning is the step that fixes this.
Collect (or generate) a corpus of (instruction, response) pairs:
Instruction: Summarize this article in three bullet points.
[Article text]
Response:
- Bullet 1
- Bullet 2
- Bullet 3Fine-tune the base model on this data with standard cross-entropy loss on the response tokens (the instruction part is typically masked out of the loss). Output: a model that, given an instruction, produces a response.
That’s the entire mechanic. The interesting work is in the data.
Hand-writing 100K diverse instruction-response pairs is brutal. The 2023 trick — Self-Instruct (Wang et al.), then Alpaca, then Evol-Instruct — was: ask a frontier LLM to generate instruction data, with seed examples and template prompts to drive diversity. Modern instruction sets (Tülu, OpenHermes, Magpie) are overwhelmingly LLM-generated; the same synthetic-supervision shape that drives most modern fine-tuning beyond the frontier labs.
The standard recipe computes cross-entropy loss only on the response tokens — the instruction tokens are presented as input but their per-token loss is masked to zero. The reasoning is that you don’t want the model learning to generate instructions; you want it learning to complete them. If you don’t mask, the gradient pushes the model to memorize instruction shapes, which both wastes capacity and degrades the model’s ability to handle out-of-distribution instructions at inference time. Some recipes additionally mask special tokens like the chat template separators so the model treats them as fixed scaffolding, not learned content.
Instruction tuning is the first half of the post-pre-training stack. The second half is preference optimization — RLHF or DPO — which takes an instruction-tuned model and pushes it toward responses humans prefer among the many possible instruction-following ones. Instruction tuning is necessary but not sufficient; an instruction-tuned model without alignment is helpful but often not honest, harmless, or stylistically calibrated.
Instruction-following rerankers (zerank-2 included) use the same supervision shape — instructions like “rerank these documents for relevance to the query, prioritizing recent docs” — turning the reranker itself into a small instruction-tuned LLM with a relevance-scoring head.
Instruction tuning teaches the model the shape of helpful responses (follow the instruction, produce a coherent answer). RLHF/DPO further refines which responses humans prefer among many helpful-shaped ones. They're sequential: instruction tuning first to make the model usable, then preference optimization to make it preferred.
Originally human-written (FLAN, Super-NaturalInstructions). Now overwhelmingly synthetic — frontier LLMs generate (instruction, response) pairs at scale. Self-Instruct, Alpaca, and Evol-Instruct were the first widely-used recipes; today every open-weight instruct model leans on synthetic data heavily.
It can — narrowly tuned instruction data causes [catastrophic forgetting](/concepts/catastrophic-forgetting/) of pre-training knowledge. The fix is mixing in pre-training-style data during fine-tuning and using parameter-efficient methods like LoRA that limit how much the base weights move.
