code: public synchronized void enqueueRobots() {
  		if (! RuntimeConfiguration.FETCH_ROBOTS) return;
  		if (nextFetch == Long.MAX_VALUE) return;
  		synchronized(this) {
  			if (pathQueries.isEmpty()) {
  				pathQueries.enqueueFirst(ROBOTS_PATH);
  				putInEntryIfNotAcquired();
  			}
  			else {
  				final byte[] first = pathQueries.dequeue();
  				pathQueries.enqueueFirst(ROBOTS_PATH);
  				pathQueries.enqueueFirst(first);
  			}
  		}
  	}
docs: Enqueues the <code>/robots.txt</code> path as the first element of the queue, if the queue is
  empty, or as the second element otherwise, possibly putting this visit state in its
  entry.
  
  <p>This behaviour is necessary: is this visit state
  is {@linkplain #acquired}, the first path+query might be still need to be dequeued by a
  {@link ParsingThread}.