This research quantifies the Data Compliance Gap (DCG) - the performance drop when LLMs respect web crawling opt-outs from content owners.

• Models trained on fully-compliant data show 5-15% performance degradation across tasks
• Effects are most severe in specialized domains (e.g., biomedical research)
• LLMs trained on opt-out-filtered datasets struggle with niche knowledge and specialized reasoning
• Ethical compliance creates real trade-offs between model performance and respecting content owners' rights

This research highlights critical security and privacy implications as AI companies must balance regulatory compliance with competitive performance demands.

Original Paper: Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs