A benchmarking framework, AstroAgentBench, is introduced for evaluating agentic planning in space mission planning tasks. It assesses the performance of large language model (LLM) agents in domains like scheduling, observation planning, and constellation design. The framework provides a standardized evaluation methodology and highlights the importance of task-contract formulation, verifier feedback, and search adaptation in achieving high-quality plans.

KaliBench is a fine-grained benchmark for evaluating the ability of language models to generate executable commands for real-world cybersecurity tools. It includes 8,504 query-command pairs across 1,642 tools and enables precise and reproducible assessment of tool selection and argument construction.

Mem++ is a non-destructive memory framework for long-term organizational LLM agents. It stores documents whole with their date and author, allowing for read-time selection of relevant documents and fusion of lexical and semantic rankings. Evaluations show Mem++ outperforms baseline memory systems by 8-13 points and achieves the best overall score on the gpt-4.1-mini model.

Researchers investigate whether large language models (LLMs) can automatically formalize symbolic constraints for Neuro-Symbolic predictors, making them suitable for high-stakes applications. They introduce a benchmark to evaluate this and find that LLMs can generate formulas similar to human experts, leading to high-quality predictions.

EurekaBench is a benchmark for measuring AI agents' ability to make scientific discoveries. It tests agents' ability to conduct long-horizon experiments and discover mechanisms that explain observations. The benchmark evaluates three axes of scientific discovery: following scientific constraints, predictive accuracy, and deriving scientific insights. Current AI agents perform well in predictive accuracy but struggle to derive meaningful scientific insights.

A new approach to multi-agent workflow optimization, InFlowOp, is proposed. It uses a label-free cost function to determine task decomposition and agent assignment, and corrects faults during execution with the cheapest move. InFlowOp outperforms single-agent baselines in various domains and backbones.

Incident-Arena is a benchmark for agentic site-reliability-engineering (SRE) that evaluates AI coding agents' ability to execute on production incident response. It includes 20 carefully selected tasks and a novel verification method that goes beyond static checks to functional verifiers.

CompMat-Bench is a benchmark for evaluating AI agents on computational materials science tasks. It reproduces research steps and assesses agents on preparing inputs and analyzing outputs for expensive simulations, supporting four evaluation conditions. The benchmark demonstrates the ability of agents based on three LLMs to complete individual materials research steps with pass rates of 66.0-90.4%.

A benchmark for evaluating the ability of AI agents to generate user-facing documentation. The DoGBENCH benchmark evaluates agents' performance in producing accurate and complete documentation for open-source projects. The results show that current agents struggle with tasks such as describing interfaces and providing decisive evidence, with failure modes identified in a separate audit.

Researchers introduce E2E-SWE, a benchmark for evaluating the ability of LLM-powered coding agents to build complete, functional software repositories from scratch. The benchmark contains 186 tasks across 11 programming languages, with varying levels of success across 13 frontier models.

Talk2Agent is a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. It builds human-spoken versions of tasks and evaluates various voice interfaces, including ASR models, audio-capable LLMs, and contextual biasing. Talk2Agent proposes an execution-free, task-conditioned evaluation framework to measure task-relevant information retention after voice interfaces.

A new benchmark, VAmoS Energy, is introduced to evaluate the performance of voice agents in handling complex conversations, including background speech and user requests. The benchmark combines challenges such as customer impatience and verification requirements, and results show that current voice stacks have varying completion rates and are prone to errors.

The paper introduces NAQD-Env, a synthetic environment to evaluate language agents' selective withdrawal decisions. It assesses the agents' ability to suspend affected actions, preserve unaffected work, and resume after repair, using a deterministic reference policy. The evaluation results show that current models struggle with selective withdrawal, motivating further research in this area.

AgentBug-Smith is a tool that automates harness bug reproduction in agentic systems, achieving higher success rates than general software bug reproduction techniques. It constructs a live and extensible benchmark, Live-Harness-Bench, containing 200 reproducible harness bugs.

DatalogBench is a benchmark for evaluating large language models (LLMs) on text-to-Datalog synthesis tasks. It consists of 136 curated tasks and uses execution on held-out inputs to grade synthesized programs. The results show that current LLMs struggle with recursive reasoning and decomposition, and two coding agents improve performance by up to 83.8%.

CruxBench is a new benchmark for evaluating the information discovery capabilities of large language models (LLMs). It assesses a model's ability to identify key questions (cruxes) that provide important steps toward solving a problem, rather than just answering fixed reference labels. CruxBench is unique in being contamination-resistant, open-ended, and grounded in real-world beliefs, and has been evaluated on eight diverse models with promising results.

A new benchmark, LibraryDesignBench, is introduced to evaluate how well AI agents design libraries for other agents. The benchmark consists of 15 library-design tasks in four languages, and the results show that agent designers reproduce human-written production libraries in 11 of the tasks.

MAADBench is a new benchmark for anomaly detection in multi-agent systems (MAS) with evolving LLM backbones. It addresses challenges in MAS AD benchmarking by providing a refreshable paradigm with sampled-and-coupled generative tasks, refreshable trace generation, and automated label provision. The benchmark is designed to support diverse LLM backbones and offers a rich research agenda for MAS-specific anomaly detection.

CheatBench is a benchmark for measuring reward gaming in AI agents, allowing researchers to study how agents cheat when faced with difficult tasks. It includes environments for various domains and supports comparisons across models and task categories.

GeoOutageBench is a benchmark for assessing LLM-based geospatiotemporal KGQA for multimodal outage and resilience analysis. It considers a spatiotemporal KG that integrates visual, textual, and structured data from various sources. The benchmark provides a competency query taxonomy at different difficulty levels and evaluates three tasks: LLMs' understanding of ambiguous geospatiotemporal questions, ontology utility, and answer accuracy of multimodal KGQA retrieval.