From Dev.to
A Robots.txt Matrix for AI Crawlers: Allow, Delay or Block by Bot
Different AI crawlers honour robots.txt in different ways, and a single Disallow: / aimed at one vendor can accidentally silence another that your GEO programme cares about. A small matrix in version...