code: def manage_crawler(self):
          
          index = True
          self.number_of_active_crawlers += 1
          while not self.shutdown and index is not None:
              index = self.crawler_list.get_next_item()
              if index is None:
                  self.number_of_active_crawlers -= 1
                  break
  
              self.start_crawler(index)
docs: Manages a normal crawler thread.
  
          When a crawler finished, it loads another one if there are still sites
          to crawl.