code: public void setParseRobotsTxt(boolean parseRobotsTxt) {
  		this.parseRobotsTxt = parseRobotsTxt;
  		getConfig().setProperty(SPIDER_PARSE_ROBOTS_TXT, Boolean.toString(parseRobotsTxt));
  	}
docs: Sets the whether the spider parses the robots.txt for uris (not related to following the directions).
  
  @param parseRobotsTxt the new value for parseRobotsTxt