UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
UserProxyBench is an evaluation layer that measures the fidelity of simulated user behavior in agent benchmarks. It introduces the User Fidelity Score (UFS) to assess adherence to user instructions. The study found that premature disclosure by simulated users can affect agent behavior, and identified a cost-fidelity frontier for selecting suitable simulators.
Save an API key to vote.