Also known as: entropy bonus, max-entropy regularization, entropy term
Adding an entropy bonus to a training objective to keep the model's output distribution from collapsing too sharply. Used in policy-gradient RL (PPO, SAC, A3C) to encourage exploration.
Entropy regularization adds an explicit reward (or negative loss) for the model’s output distribution being spread out:
where
In on-policy methods (REINFORCE, A3C, PPO), the policy is a stochastic distribution over actions. The gradient of the expected return only flows through actions the policy takes. If the policy is already nearly deterministic, the agent never tries alternatives — the exploration problem.
Entropy reg keeps the policy stochastic during training:
loss = -mean(advantage * log_prob(actions)) # standard policy gradient
loss -= beta * entropy(policy_dist) # entropy bonusPPO’s canonical hyperparameter is entropy_coef = 0.01. Soft actor-critic (SAC) goes further: the entire reward formulation includes a α · H(π) term so the policy is learned to be high-entropy.
In language-model RL fine-tuning ( RLHF , DPO ), the dominant regularization is not entropy but KL-to-reference. That said, an explicit entropy bonus often appears alongside the KL term — the KL keeps the policy near the SFT reference, the entropy bonus prevents pathological collapse within that constraint. The two are complementary: KL anchors the location, entropy preserves the spread.
In pseudo-label-based methods (FixMatch, Mean Teacher, noisy student), the model trains on its own confident predictions. The pathology is that a model that gets a few examples slightly wrong amplifies that wrongness into confident misclassification. A small entropy regularization on unlabeled examples blocks the collapse.
Adjacent but distinct.
Temperature sampling is a decode-time intervention: train a model, then at inference, scale the logits by
Entropy regularization is a training-time intervention: add an entropy term to the loss so the model learns to prefer spread-out predictions over peaked ones. The weights themselves carry the regularization.
A model trained with no entropy reg and decoded with high temperature can mimic the output of a model trained with entropy reg and decoded at T=1 — but only in the immediate one-step sense. Over many decode steps, the trained-in regularization compounds (the model has learned not to be over-confident) in a way temperature scaling cannot reproduce.
In RL, this distinction matters: temperature scaling at decode does not change the policy’s value estimates or its training dynamics. Entropy regularization at training does. SAC’s success rests on this difference.
The β coefficient controls how much of the gradient comes from “do the task” vs “stay uncertain.” Three useful defaults:
The honest move is to start at the literature default for your setting, then sweep β on a log scale (10x steps) and pick by validation reward / NLL.
Entropy regularization is a one-hyperparameter cure for a specific pathology: a model that gets too confident too fast. It pays back enormously when that is your problem (RL exploration, pseudo-labeling) and is mostly a no-op when it is not (standard supervised classification, where the cross-entropy loss already provides the right signal). The technique is small but load-bearing — most published RL results have it, often invisibly inside the algorithm’s default config.
A policy that collapses to argmax stops exploring. Adding β·H(π(·|s)) to the reward (or subtracting from the loss) creates a gradient pressure toward higher-entropy policies, which by construction try more diverse actions. The agent gets penalized for being too sure of itself before the value estimates have stabilized. Once the value function is well-fit, β can decay so the policy can commit to good actions.
Entropy regularization keeps the policy spread out intrinsically — it does not care what the spread-out distribution looks like, just that it is not too peaked. KL regularization to a reference (the dominant approach in RLHF/PPO and DPO) anchors the trained policy near a specific reference distribution. Entropy reg encourages exploration; KL reg encourages conservatism. PPO uses both: KL to the previous policy AND an entropy bonus.
Sometimes. In standard cross-entropy SFT it is rarely useful — the loss already pushes toward the one-hot label, and entropy regularization fights that signal. In semi-supervised settings with pseudo-labels, adding a small entropy bonus to unlabeled examples prevents the model from collapsing to a degenerate confident-and-wrong solution. The technique shows up under different names (confidence penalty, label smoothing-ish, FixMatch-style consistency).
