 And a robot.txt file then kind of puts a sign in front of that website to say, if you're a robot, sort of this visitor, you need to abide by the rules here. This is where you are or aren't welcome. This is what you can and can't do. That's Colleen Chen of UC Berkeley Law School, who teaches classes on how AI is changing the web. Over the years, the robots.txt page became something of a social contract for the entire internet. Tech giants like Google and Facebook adopted it. And even though it had no legal teeth, it was respected.