Context distillation
Technique · Training · 2 items since Jul 2026
Moving behavior learned from in-context examples or feedback into the model's weights.
Related
- Organizations
- Techniques
Trends
Items
- Jul 24
Microsoft Research's LLM-as-a-Coach
An LLM judge becomes a coach whose textual feedback is distilled into the policy through on-policy context distillation, beating rubric-based RL on non-verifiable tasks while mitigating reward hacking.
- Jul 24
Experience Distillation internalizes agent trial and error
In-context trial-and-error experience is distilled into weights with no further environment interaction, keeping at least 64.8% of in-context gains (versus 3.8% for supervised fine-tuning) and matching RL with at least 9.6x fewer environment samples.