MINCE is a method for shrinking LLM evaluation datasets by using Monte Carlo simulation to find the minimum subset size that bounds accuracy drift, reducing evaluation time by up to 89%.

A study evaluates the effectiveness of profession-specific system prompts on scientific tasks, comparing them to generic and unrelated prompts. The results show that longer prompts do not improve accuracy and increase costs.

DatalogBench is a benchmark for evaluating large language models (LLMs) on text-to-Datalog synthesis tasks. It consists of 136 curated tasks and uses execution on held-out inputs to grade synthesized programs. The results show that current LLMs struggle with recursive reasoning and decomposition, and two coding agents improve performance by up to 83.8%.

CruxBench is a new benchmark for evaluating the information discovery capabilities of large language models (LLMs). It assesses a model's ability to identify key questions (cruxes) that provide important steps toward solving a problem, rather than just answering fixed reference labels. CruxBench is unique in being contamination-resistant, open-ended, and grounded in real-world beliefs, and has been evaluated on eight diverse models with promising results.

A new benchmark, KNOWS, is introduced to evaluate AI agents' ability to perform complex tasks, such as synthesizing and organizing knowledge, and navigating program interfaces. Current agents struggle with visual understanding and long-horizon reasoning, highlighting areas for improvement in AI development.

A new evaluation framework for assessing large language models' (LLMs') contextual understanding in question answering (QA) tasks. The framework, based on knowledge graphs, measures semantic and structural similarity to evaluate LLMs' ability to reason over context. Results show significant improvements over baseline models.