Domain Crawling Lists

DNS | Dataset

Domain popularity lists provide a starting point for crawling domains with the most users. The most commonly used list for security research is the Alexa list.

  • Alexa
    The list is updated daily and contains one million websites. The ranking is based on page views, but very volatile.
  • CISCO Umbrella
    The list is updated daily and contains one million websites. The ranking is based on traffic seen on the OpenDNS resolvers.
  • Majestic
    The list is updated daily and contains one million websites. The ranking is based on backlinks from other websites.
  • Tranco
    A Research-Oriented Top Sites Ranking Hardened Against Manipulation
    The Tranco list aims to provide a better list for security research. The authors explain on their website and their paper what the flaws in the existing lists.
  • Quantcast
    The list is updated daily and contains around 500,000 websites. It is based on users visiting the site within the previous month and highly US focussed.
  • Cloudflare Radar Cloudflare uses their 1.1.1.1 DNS resolver to create a top 1 million list. The lists are also available on a per country level, e.g., https://radar.cloudflare.com/domains/de. More details are available in their announcement blog post.
  • CrUX Chrome Google Chrome collects the top 1 million visited website and published them as part of the Chrome UX Report. The repository captures the monthly data and provides access to older versions. In an Internet Measurement Conference (IMC) paper this list was shown to best correlate with the HTTP requests as seen by Cloudflare.