This research analyzes methodologies for evaluating LLM-driven offensive security tools, providing a systematic review of current benchmarking practices across 16 research papers.

• Identifies inconsistent evaluation approaches in LLM-based penetration testing tools
• Reveals gaps in testbed designs and evaluation metrics for security applications
• Provides actionable recommendations for more rigorous future research
• Emphasizes need for standardized benchmarking frameworks for security applications

Important for security professionals as it highlights the need for more robust evaluation methods when deploying AI-powered offensive security tools, ensuring reliable performance in real-world scenarios.

Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Design