Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits
Researchers introduced a new model-free controller called Decode-Latency Feedback Prefill (DLFP) to reduce interference in concurrent autoregressive inference. The controller adjusts prefilled chunks based on observed latency and achieves a 27.7% reduction in P99 inter-token latency on a 0.6B Qwen3 model. However, the mechanism does not generalize to larger models or multi-GPU configurations.
Save an API key to vote.