Self-distillation

Technique · Training · 3 items since Jun 2026

A model trains on its own filtered or guided outputs instead of a separate teacher's.

Organizations: Weibo Zhejiang University
Techniques: Agentic reinforcement learning Curriculum learning Test-time training

Trends

Aug 28
Zhejiang University's label-free TTPO
A test-time training method that distills rollouts agreeing with a majority-vote pseudo-label while penalizing confident disagreements via grouped RL, matching label-supervised self-distillation and lifting Qwen3-1.7B from 38.0% to 45.2% on competition math.
Aug 28
AHEAD adds step-aware hints to agentic RL
A self-distillation method for multi-turn agentic RL that gives environment feedback on every step and LLM-generated corrective hints only on error steps, a minimal change to GRPO worth +13.3 on ALFWorld and +11.0 on WebShop at 7B with no inference overhead.
Jun 15
Weibo AI's VibeThinker-3B
A 3B dense open-weight reasoning model scoring 94.3 on AIME 2026 (97.1 with test-time scaling) and 80.2 Pass@1 on LiveCodeBench v6, matching far larger flagships such as DeepSeek V3.2 and Gemini 3 Pro via curriculum SFT, multi-domain RL, and self-distillation.