Ahrefs runs our own web crawler, or bot, that visits millions of websites to retrieve information and store them in our records. In other words, this is how Ahrefs gets all its data ;)
- Crawl priority: in general, the more “popular” a site is (ie. a higher Domain Rating (DR), more quality backlinks), the more likely a domain will be crawled. Domains that are less “popular” (ie. lower DR, less quality backlinks) will be partially crawled.
- Speed of crawling/updating: this depends on a page's URL Rating (UR). Pages with a high UR are more likely to be indexed and are updated more often than pages with a low UR.
When our bots have yet to crawl the target site, you'll see the following errors in our reports:
- No Data
- No Results
- Page is not currently in our index
- We don't currently have any pages for that target in our index
Since the target page has not been crawled yet, on-page data such as the website's title, content, HTML source code etc are not available in our database.
If the domain is new, please give our crawler some time to pick up data (1-2 weeks is generally a good bet.)
Note: Backlinks reports will be available if we crawled other sites that have backlinks to the target site.
It's also possible that the domain might be blocking us from crawling it. To find out if your domain is blocking our crawler, check the status of your robots.txt in our robots checker: https://ahrefs.com/robot
To fix that, please read this article: https://help.ahrefs.com/site-explorer/crawling/how-do-i-enable-ahrefs-bot-to-crawl-my-website-and-index-its-pages