Red-Team Leaderboard

Overview

LLM agents read external content, call tools and change application state, so a successful attack shows up as a leaked secret or an unauthorized action, not just a bad reply. Automated red-teaming services such as Promptfoo, PyRIT, DeepTeam and Dreadnode are built to find these failures, but when a test finds nothing it is unclear whether the agent resisted or the service never reached a weakness. GreyZone evaluates the services themselves: each one acts as an end-to-end attacker and is measured on whether it generates and delivers attacks that cause verified security-policy violations in real agent applications.

• Real applications. 10 independently developed, open-source agent applications (coding, enterprise collaboration, email, trading, commerce, banking, smart home, Kubernetes operations and social/on-chain agents) run with their own workflows, permissions and security boundaries. External dependencies are replaced with controlled simulations, and every run starts from a prepared initial state.
• 46 attack scenarios. Each scenario fixes the application, initial state, security policy, attack objective, attacker constraints and oracle. The objective is the same for every service; how to reach it is up to the service.
• One attacker interface. An MCP interface exposes only the operations the attacker's role permits, for direct injection (messages to the agent) or indirect injection (content the agent reads during its task). Service-specific adapters connect each red-teaming service to it.
• Independent oracles. Each scenario's oracle checks execution evidence and application state, not the service's own verdict or the agent's reply, before reporting a verified outcome.