Trends

Efficiency methods cut the compute and memory models need

Architecture and inference efficiency · Apr 10 – Jul 10

Organizations

NVIDIADeepSeekLehigh UniversityMITNational University of SingaporeStanford University

Updates

  1. Jul 10

    DeepSeek's DSpark speculative decoding sped per-user generation 60-85% at matched throughput in DeepSeek-V4's live serving, and NVIDIA's Nemotron-Labs-Diffusion decoded about 6x more tokens per forward pass than Qwen3-8B for roughly 4x the throughput on a GB200.

    DeepSeek's DSpark speculative decodingNVIDIA's Nemotron-Labs-Diffusion

  2. May 1

    Nemotron 3 Nano Omni reaches 9x the throughput of comparable open omni models through Conv3D temporal compression and token reduction, and RecursiveMAS cuts multi-agent token use 34.6-75.6% by passing latent state instead of text.

    NVIDIA's Nemotron 3 Nano OmniStanford's RecursiveMAS

  3. Apr 10New

    TriAttention cuts KV memory 10.7x and fits a 32B reasoning model on a 24GB RTX 4090, MegaTrain trains 120B-parameter models on a single H200, and DMax raises diffusion decoding from 2.04 to 5.47 tokens per forward pass.

    TriAttention compresses the KV cache 10.7xMegaTrain trains 100B+ models on one GPUDMax speeds up diffusion LLM decoding

Predictions

  1. Nov 2026

    Open-weight competition centers on KV-cache compression and modality token reduction rather than parameter scaling, with efficiency per context token replacing efficiency per parameter as the headline model-card metric. May 1

  2. Dec 2026

    Running 30B+ reasoning models on consumer-grade hardware becomes routine, lowering the cost curve for local agentic applications. Apr 10

  3. Jan 2027

    Hybrid and diffusion-based decoding schemes move from research into production serving stacks as labs chase throughput and latency gains beyond raw model scale. Jul 10