ORACLE is a concurrency-aware online routing mechanism that improves accuracy and throughput for large language models in heterogeneous workloads. It dynamically assigns a task-appropriate verifier and reduces verifier latency using a delayed feedback strategy.

FlexRouter is a routing framework for Large Language Models (LLMs) that optimizes for answer coverage by selecting models that complement each other, improving the probability of a correct response. It uses Determinantal Point Processes (DPPs) and a novel training objective to adaptively determine subset sizes without a predefined budget.

Researchers propose a market-aware routing approach for Large Language Models (LLMs) that considers not just the model's cost but also the quality, latency, and availability of different providers serving the model. They introduce a policy that routes requests to the cheapest provider that meets quality and health criteria, and a certification mechanism to ensure reliable routing. This work has implications for the development and operation of LLMs in production environments.

Researchers propose SaveRouter, a sparse-supervision framework for large language model (LLM) routing that selectively acquires model feedback and shares capability information across queries, reducing supervision expenditure and improving serving-time efficiency.

Researchers propose a foundation model for large language model routing, called RouteFM, which can learn to characterize candidate models and infer their capabilities from behavioral context, enabling pretraining once and routing anywhere. This approach outperforms baselines in experiments and demonstrates transfer across different environments.