Domain Crawling Lists

DNS | Dataset

Domain popularity lists provide a starting point for crawling domains with the most users. The most commonly used list for security research is the Alexa list.

  • Alexa
    The list is updated daily and contains one million websites. The ranking is based on page views, but very volatile.
  • CISCO Umbrella
    The list is updated daily and contains one million websites. The ranking is based on traffic seen on the OpenDNS resolvers.
  • Majestic
    The list is updated daily and contains one million websites. The ranking is based on backlinks from other websites.
  • Tranco
    A Research-Oriented Top Sites Ranking Hardened Against Manipulation
    The Tranco list aims to provide a better list for security research. The authors explain on their website and their paper what the flaws in the existing lists
  • Quantcast
    The list is updated daily and contains around 500,000 websites. It is based on users visiting the site within the previous month and highly US focussed.