TokenGate prunes tokens to speed long-context prefill up to 11.3x

Stanford and UC Berkeley researchers released TokenGate, a drop-in learned token-pruning module for LLMs. It cuts time-to-first-token for long prompts by 1.4x to 11.3x and peak memory by 1.5x to 3.4x. The authors report no accuracy loss.

Why it matters: Prefill is the dominant latency and memory cost for agents that load large codebases or documents, so gains here directly lower serving cost. If the no-loss claim holds across models, it complements harness-level efficiency work by cutting cost at the attention layer itself.