code: def check_robotstxt(url, session):
      
      roboturl = get_roboturl(url)
      rp = get_robotstxt_parser(roboturl, session=session)
      if not rp.can_fetch(UserAgent, str(url)):
          raise IOError("%s is disallowed by %s" % (url, roboturl))
docs: Check if robots.txt allows our user agent for the given URL.
      @raises: IOError if URL is not allowed