code: public void checkRobots(final long time) {
  		if (! RuntimeConfiguration.FETCH_ROBOTS) return;
  
  		/* Safeguard: if there's no robots filter, and I'm not fetching it, and it's not the first
  		 * path in the queue, then something's wrong. */
  		if (robotsFilter == null && lastRobotsFetch != Long.MAX_VALUE && (isEmpty() || firstPath() != ROBOTS_PATH)) {
  			LOGGER.error("No robots filter and no robots path for " + this);
  			lastRobotsFetch = Long.MAX_VALUE; // This inhibits further enqueueing until robots.txt is fetched.
  			enqueueRobots();
  			return;
  		}
  
  		if (lastExceptionClass == null) {
  			// If we are not retrying an exception, we check whether it is time to reload robots.txt
  			final long robotsExpiration = frontier.rc.robotsExpiration;
  			if (time - robotsExpiration >= lastRobotsFetch) {
  				if (robotsFilter == null) {
  					if (lastRobotsFetch == 0) LOGGER.info("Going to get robots for {} for the first time", Util.toString(schemeAuthority));
  					else LOGGER.info("Going to try again to get robots for {}", Util.toString(schemeAuthority));
  				}
  				else LOGGER.info("Going to get robots for {} because it has expired ({} >= {} ms have passed)", Util.toString(schemeAuthority), Long.valueOf(time - lastRobotsFetch), Long.valueOf(robotsExpiration));
  				lastRobotsFetch = Long.MAX_VALUE; // This inhibits further enqueueing until robots.txt is fetched.
  				enqueueRobots();
  			}
  		}
  	}
docs: Checks whether the current robots information has expired and, if necessary, schedules a new <code>robots.txt</code> download.
  
  @param time the current time.