Robots.txt is used as part of the Robots Exclusion Standard, a voluntary protocol the Internet Archive respects that disallows bots from indexing certain pages delineated by its creator as off-limits. As a result, the Internet Archive has rendered unavailable a number of web sites that now are inaccessible through the Wayback Machine. Currently, the Internet Archive applies robots.txt rules retroactively; if a site blocks the Internet Archive, such as Healthcare Advocates, any previously archived pages from the domain are rendered unavailable as well. In cases of blocked sites, only the robots.txt file is archived.
ఒక సైటు ఇంటర్నెట్ ఆర్కైవ్ను రికార్డ్ చేయకుండా నిరోధిస్తే, ఏ ఫైలు ఇంకా సేవ్ చేయబడుతుంది?
Robots.txt