code: def allows_url (self, url_data):
          
          roboturl = url_data.get_robots_txt_url()
          with self.get_lock(roboturl):
              return self._allows_url(url_data, roboturl)
docs: Ask robots.txt allowance.