Knowledge distillation

Technique · Training · 4 items since Apr 2026

Training a model to imitate the outputs of another, usually larger, teacher model.

Trends

Sep 25
Pistis pairs IDRL with auto-harnessing
A 9B/27B multimodal model family trained with Interleaved Distillation and Reinforcement Learning, paired with a system-level method that automatically improves the inference harness without updating model weights.
Sep 4
University of Waterloo's Program-as-Weights
Compiles a natural-language specification into a reusable neural function by distilling teacher-generated examples into a small adapter on a 0.6B interpreter, reaching 83.6% semantic accuracy on FuzzyBench-Hard while running locally without a remote provider.
May 27
ECNU's Skill0.5 for agentic RL
The framework internalizes general skills through privileged distillation on hard tasks and applies task-specific skills on easy ones via a difficulty-aware selector, beating memory- and skill-based RL baselines on ALFWorld and WebShop in and out of distribution.
Apr 19
Byte-level distillation bridges mismatched tokenizers
Converting teacher outputs to byte-level probabilities and distilling through a lightweight byte-level decoder head on the student resolves vocabulary mismatch without heuristic alignment, matching or beating more complex cross-tokenizer methods across 1B-8B model pairs.