From Dev.to
What 4,700 bot visits taught me about AI crawlers (and a tool to check any site's robots.txt
A few weeks ago I wanted to answer a simple question: how many of the "visitors" on my small side projects are actually people? The answer turned out to be: not many. Most of the traffic was crawlers,...