code: public boolean isRobotPermitted(String url, String userAgent)
              throws IOException, RobotsUnavailableException {
          RobotRules rules = getRulesForUrl(url, userAgent);
          return !rules.blocksPathForUA(new LaxURI(url, false).getPath(),
                  userAgent);
      }
docs: Returns true if a robot with the given user-agent is allowed to access
  the given url.
  
  @param url
  @param userAgent
  @return
  @throws IOException
  @throws RobotsUnavailableException