code: public static char[][] parseRobotsResponse(final URIResponse robotsResponse, final String userAgent) throws IOException {
  		final int status = robotsResponse.response().getStatusLine().getStatusCode();
  		if (status / 100 != 2) LOGGER.info("Got status " + status + " while fetching robots: URL was " + robotsResponse.uri());
  		if (status / 100 == 4 || status / 100 == 5) return EMPTY_ROBOTS_FILTER; // For status 4xx and 5xx, we consider everything allowed.
  		if (status / 100 != 2 && status / 100 != 3) return null; // For status 2xx and 3xx we parse the content. For the rest, we consider everything forbidden.
  		// See if BOM is present and compute its length
  		BOMInputStream bomInputStream = new BOMInputStream(robotsResponse.response().getEntity().getContent(), true);
  		int bomLength = bomInputStream.hasBOM()? bomInputStream.getBOM().length() : 0;
  		// Skip BOM, if necessary
  		bomInputStream.skip(bomLength);
  		// Parse robots (BOM is ignored, robots are UTF-8, as suggested by https://developers.google.com/search/reference/robots_txt
  		char[][] result = parseRobotsReader(new InputStreamReader(bomInputStream, Charsets.UTF_8), userAgent);
  		if (LOGGER.isDebugEnabled()) LOGGER.debug("Robots for {} successfully got with status {}: {}", robotsResponse.uri(), Integer.valueOf(status), toString(result));
  		return result;
  	}
docs: Parses a <code>robots.txt</code> file contained in a {@link FetchData} and
  returns the corresponding filter as an array of sorted prefixes. HTTP statuses
  different from 2xx are {@linkplain Logger#warn(String) logged}. HTTP statuses of class 4xx
  generate an empty filter. HTTP statuses 2xx/3xx cause the tentative parsing of the
  request content. In the remaining cases we return {@code null}.
  
  @param robotsResponse the response containing <code>robots.txt</code>.
  @param userAgent the string representing the user agent of interest.
  @return an array of character arrays, which are prefixes of the URLs not to follow, in sorted order,
  or {@code null}