code: def check_connection (self):
          
          self.session = self.aggregate.get_request_session()
          # set the proxy, so a 407 status after this is an error
          self.set_proxy(self.aggregate.config["proxy"].get(self.scheme))
          self.construct_auth()
          # check robots.txt
          if not self.allows_robots(self.url):
              self.add_info(_("Access denied by robots.txt, checked only syntax."))
              self.set_result(_("syntax OK"))
              self.do_check_content = False
              return
          # check the http connection
          request = self.build_request()
          self.send_request(request)
          self._add_response_info()
          self.follow_redirections(request)
          self.check_response()
          if self.allows_simple_recursion():
              self.parse_header_links()
docs: Check a URL with HTTP protocol.
          Here is an excerpt from RFC 1945 with common response codes:
          The first digit of the Status-Code defines the class of response. The
          last two digits do not have any categorization role. There are 5
          values for the first digit:
            - 1xx: Informational - Not used, but reserved for future use
            - 2xx: Success - The action was successfully received,
              understood, and accepted.
            - 3xx: Redirection - Further action must be taken in order to
              complete the request
            - 4xx: Client Error - The request contains bad syntax or cannot
              be fulfilled
            - 5xx: Server Error - The server failed to fulfill an apparently
              valid request