ToolFence is a new authorization system for secure tool-using LLM agents. It provides fine-grained provenance-aware authorization, distinguishing user-authorized values from untrusted observations, and reduces the frequency of expensive judge calls for runtime efficiency. ToolFence improves runtime efficiency and effectively addresses the within-tool attack.