 for the entire internet tech giants like google and facebook adopted it and even though it had no legal teeth it was respected say there's a corporate or administrative page you don't want showing up on google put it in the file it helped hold the entire internet together says former google engineer jacob hoffman andrews that system has remarkably worked well for 30 years