code: def _is_bot(user_agent):
      
      bot_list = [
                  'http://www.baidu.com/search/spider.html',
                  'python-requests',
                  'http://ltx71.com/',
                  'http://drupal.org/',
                  'www.sogou.com',
                  'http://search.msn.com/msnbot.htm',
                  'semantic-visions.com crawler',
                 ]
      for bot in bot_list:
          if re.search(re.escape(bot), user_agent):
              return True
      return False
docs: Check if user_agent is a known bot.