code: def start_element (self, tag, attrs):
          
          if tag == 'meta' and attrs.get('name') == 'robots':
              val = attrs.get_true('content', u'').lower().split(u',')
              self.follow = u'nofollow' not in val
              self.index = u'noindex' not in val
              raise StopParse("found <meta name=robots> tag")
          elif tag == 'body':
              raise StopParse("found <body> tag")
docs: Search for meta robots.txt "nofollow" and "noindex" flags.