This research introduces a comprehensive benchmarking framework to evaluate how effectively large language models resist attempts to extract biased responses.

Key findings:

• Systematically assesses LLM vulnerabilities to bias elicitation using scalable automated testing
• Employs another LLM as an unbiased judge to evaluate responses, reducing evaluation costs
• Identifies specific weaknesses in current models' defensive mechanisms against adversarial prompting
• Provides a standardized methodology for comparing robustness across different model architectures

For security professionals, this framework offers a crucial tool to identify and address potential exploitation vectors before deploying LLMs in sensitive environments, helping prevent harmful biases from affecting critical decision systems.

Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge