One sampling interface for two learning regimes: preserve exploration while learning to exploit, or preserve a pretrained policy while learning to adapt.
Reward optimization can narrow a diffusion policy's behavior. Fine-tuning a pretrained generative policy can also disturb capabilities it already has. CFG-Policy addresses both by giving exploration and exploitation separate generative branches.
Keep one generative policy fixed, train the complementary policy, and combine their fields during action sampling.
Diffusion policies can represent multimodal actions, but reinforcement learning may reduce their diversity. In RL post-training, directly fine-tuning a pretrained policy may disrupt acquired capabilities. CFG-Policy separates exploration and exploitation into two generative policies and combines their score or velocity fields through classifier-free guidance during sampling. In online RL, a frozen exploration branch sustains diversity while an exploitation branch learns from rewards. In offline-to-online post-training, a pretrained exploitation branch stays frozen while a trainable branch steers behavior. The framework preserves the underlying RL objectives and is evaluated with QSM and QVPO on five MuJoCo tasks and with Flow-SDE+PPO on four RoboTwin manipulation tasks.
Both branches evaluate the same intermediate sample at each diffusion or flow step. The guidance weight controls how much each branch shapes the resulting action.
Preserve exploration. Learn exploitation.
Keep the exploration policy frozen to sustain behavioral diversity. Optimize the exploitation policy using the existing RL objective, then sample actions from their composed field.
Preserve capability. Learn adaptation.
Keep the pretrained policy frozen and train a second steering branch. Guidance adapts behavior without directly changing the original action generator.
Adaptive guidance. Instead of fixing the weight throughout training, CFG-Policy selects it to improve critic value while maintaining a minimum pathwise deviation from the pure exploitation sampler.
The paper evaluates the same policy composition principle in online MuJoCo control and RoboTwin RL post-training.
Final episodic return, mean ± sample standard deviation over five runs. Higher is better.
| Method | HalfCheetah | Walker2d | Ant | Humanoid | Hopper |
|---|---|---|---|---|---|
| QSM | 5,020 ± 455 | 2,041 ± 477 | −24 ± 3 | 5,922 ± 262 | 2,753 ± 307 |
| QSM + CFG-Policy | 6,218 ± 533 | 2,349 ± 583 | 1,425 ± 308 | 6,094 ± 169 | 3,101 ± 461 |
| QVPO | 11,166 ± 5,076 | 5,220 ± 689 | 7,129 ± 61 | 6,612 ± 214 | 3,727 ± 176 |
| QVPO + CFG-Policy | 12,990 ± 1,993 | 5,692 ± 413 | 7,374 ± 45 | 7,041 ± 385 | 3,871 ± 192 |
Task success rate when adapting pretrained π0 and π0.5 policies. Higher is better.
| Model | Method | Adjust Bottle | Pick Dual Bottles | Rotate QR Code | Grab Roller |
|---|---|---|---|---|---|
| π0 | Flow-SDE+PPO | 82.81% | 43.75% | 34.38% | 92.19% |
| CFG-Policy | 98.44% | 57.81% | 42.19% | 93.75% | |
| π0.5 | Flow-SDE+PPO | 89.06% | 12.50% | 7.81% | 100.00% |
| CFG-Policy | 92.19% | 15.62% | 10.94% | 100.00% |