A study on agent evaluation reliability explores the impact of scaffolds and tasks on model rankings. The authors develop a Bayesian variance-decomposition framework to separate signal from noise in sparse, imbalanced leaderboards. They find that reliability depends on the measurement goal, scaffold choice can change conclusions, and more tasks may not resolve all uncertainty.