KaliBench is a fine-grained benchmark for evaluating the ability of language models to generate executable commands for real-world cybersecurity tools. It includes 8,504 query-command pairs across 1,642 tools and enables precise and reproducible assessment of tool selection and argument construction.