 But when an AI tool scrapes your website, you're not really getting much in return. Which is why the robots.txt file has become a way of saying, no thanks, do not do that here. With the AI industry scraping away anyway, more and more corners of the internet may soon become harder to access for everyone. Bobby Allen, NPR News.