A continuous evaluation framework for enterprise AI agent skills is proposed, combining outcome-level and process-level checks to detect behavioral drift in skills due to changing tool APIs, models, and specifications.

A research paper proposes rethinking the evaluation of AI agents by considering them as configurable systems, not just models. The authors introduce a new benchmark and find that agent configuration choices, such as task information and time budget, have a significant impact on performance and behavior.

A study on agent evaluation reliability explores the impact of scaffolds and tasks on model rankings. The authors develop a Bayesian variance-decomposition framework to separate signal from noise in sparse, imbalanced leaderboards. They find that reliability depends on the measurement goal, scaffold choice can change conclusions, and more tasks may not resolve all uncertainty.

A new dataset (AED) for identifying errors and failures in AI agents is introduced, with 50,228 error-diagnosis pairs from various environments and policy models. The dataset aims to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. The authors present a five-stage pipeline for collecting natural failures, generating diagnoses, and checking proposed corrections against recorded evidence.

AgBench is a benchmark suite for evaluating agentic AI on personal devices, assessing trade-offs between local, hybrid, and cloud execution. It evaluates task success, latency, cloud API cost, and data exposure across devices, workloads, and deployment architectures.