Domain Crawling Lists

DNS | Dataset

Domain popularity lists provide a starting point for crawling domains with the most users. The most commonly used list for security research is the Alexa list.

  • CISCO Umbrella
    The list is updated daily and contains one million websites. The ranking is based on traffic seen on the OpenDNS resolvers.
  • Majestic
    The list is updated daily and contains one million websites. The ranking is based on backlinks from other websites.
  • Tranco
    A Research-Oriented Top Sites Ranking Hardened Against Manipulation
    The Tranco list aims to provide a better list for security research. The authors explain on their website and their paper what the flaws in the existing lists.
  • Cloudflare Radar Cloudflare uses their 1.1.1.1 DNS resolver to create a top 1 million list. The lists are also available on a per country level, e.g., https://radar.cloudflare.com/domains/de. More details are available in their announcement blog post.
  • CrUX Chrome Google Chrome collects the top 1 million visited website and published them as part of the Chrome UX Report. The repository captures the monthly data and provides access to older versions. In an Internet Measurement Conference (IMC) paper this list was shown to best correlate with the HTTP requests as seen by Cloudflare.