All about DNS

.nl stats and data - SIDN Labs

 https://stats.sidnlabs.nl/en/

Datasets | DNSSEC | IP | Networks

Historic datasets (from 2014 onwards) for the .nl TLD. Datasets are available in JSON format.

Datasets cover information about:

  • DNS
    • Domain Names
    • Query Type
    • Resonse Codes
    • IPv6 Support
  • Resolvers
    • Location
    • Number of IP addresses
    • Validating Resolvers
    • Popular Networks
    • Port Randomness
  • DNSSEC
    • Validating Queries
    • DANE
    • Used Algorithms
  • Mail
    • Mail RRs
    • SPF Information

Active DNS

 https://www.activednsproject.org/

Datasets | IP | Networks

Historical DNS database. Access can be requested for academic use.

Activly queries many DNS records, e.g., .com zone. It can contain information not in DNSDB, if the information was never seen by a resolver. It does not contain all informatin, as some domains may be unknown to the project and thus cannot be crawled. It uses popular zones, domain lists (e.g., Alexa, blacklists) and other domain feeds.

They normally maintain a rolling 14-day window.

Copy files (for date 2017-10-05) (ddos@gladbeck):

sftp -B1024000 -C -rp "activedns@kokino.gtisc.gatech.edu:active-dns/20171005/" .

The data is encocded in AVRO format, which can also be parsed as JSONL. Python has a AVRO library. AVRO schema:

{
    "namespace": "astrolavos.avro",
    "type": "record",
    "name": "ActiveDns",
    "fields": [
        {"name": "date", "type": "string"},
        {"name": "qname", "type": "string"},
        {"name": "qtype", "type": "int"},
        {"name": "rdata", "type": ["string", "null"]},
        {"name": "ttl", "type": ["int", "null"]},
        {"name": "authority_ips", "type": "string"},
        {"name": "count", "type": "long"},
        {"name": "hours", "type": "int"},
        {"name": "source", "type": "string"},
        {"name": "sensor", "type": "string"}
    ]
}

Some more information about some fields that are unique to that schema. The IPs in Authority IP are the collection of the authority name server IPs that replied to our query. We gather all the IPs that gave us the same answer for an entire day and concatenate them on the same field, mostly in order to reduce the number of records that we have to keep. The only field that might be slightly confusing, is the "hours" field. This is a 24bit integer that encodes the time of day we saw this RR for date date (for example, 000000000000000001000010 = 18:00 and 23:00). Another important thing to keep in mind, is NXDOMAINs. A resolved QNAME does not exist when both the rdata and ttl fields are equal to null. If rdata exists but ttl is null then the record was part of the glue of the DNS packet and not in the answer section.

A similar active DNS project is Open INTEL which seems to be larger in scope and the data is publicly available.


 https://www.akamai.com/us/en/why-akamai/dns-trends-and-traffic.jsp

Datasets | IP

Akamai provides aggregated data about the root servers. They provide the overall amount of DNS queries, trends, and IPv4/IPv6 query ratios.


APNIC Labs Stats

 https://stats.labs.apnic.net/

Autonomous Systems | BGP | Datasets | DNSSEC | IP

APNIC gathers many statistics and offers them on their website. However, they provide way more data than it might initially look like, since many of the datasets are not linked from their main page.


Censys

 https://censys.io/

Certificates | Datasets | IP | Networks

Censys performs regular scans for common protocols (e.g., DNS, HTTP(S), SSH). Provides a search for TLS certificates.

Access is free, but requires registration. The website no longer provides free bulk access. Bulk access requires a commercial or a research license. The free access is limited to 1000 API calls per day.

@InProceedings{censys15,
    author = {Zakir Durumeric and David Adrian and Ariana Mirian and Michael Bailey and J. Alex Halderman},
    title = {A Search Engine Backed by {I}nternet-Wide Scanning},
    booktitle = {Proceedings of the 22nd {ACM} Conference on Computer and Communications Security},
    month = oct,
    year = 2015
}

Collection of "bad" packets in PCAPs

 https://github.com/DNS-OARC/bad-packets

Datasets | IP | Networks | PCAPs

Collection of "bad" packets in PCAPs that can be used for testing software.


Compacted-DNS (C-DNS): A Format for DNS Packet Capture

 https://tools.ietf.org/html/rfc8618

Tools

C-DNS is a space efficient file format for storing DNS traffic. It is based on CBOR with a fixed schema as defined in RFC 8618.

Some tools exists to work with C-DNS files. The dns-stats compactor can convert pcap-to-cdns and vice versa. It can be configured flexibly to only record those fields of the DNS message which are of interest, thus saving space in the compacted format.


Curated Lists of DNS Server Software

DNSSEC

https://www.statdns.com/resources/

Contains lists for DNS servers, libraries, tools, and other resources.

https://linuxmafia.com/kb/Network_Other/dns-servers.html

List of maintained and unmaintained DNS servers, including descriptions for each of them.

https://dnsinstitute.com/implementations/

Short overview over open source projects for authoritative and recursive servers and development libraries.


Cyber Threat Intelligence Feeds

 https://github.com/TW-NCERT/ctifeeds

IP | Networks | Spam

Provides an outdated list of different Cyber Thread Intelligence Feeds of other organizations.


CZ.NIC Statistics

 https://stats.adam.nic.cz/dashboard/en/index.html

Datasets

The website contains information about the cz. TLD operated by CZ.NIC. It contains information about the query volume, query type, round-trip time (RTT) and geographic location of the traffic sources. It also has information about the registry functions, such as registrar information, domain transfers or whois requests. Lastly, information about the mojeID accounts, a login provider operated by CZ.NIC are also available.


DMAP Domain Mapper by SIDN Labs

 https://dmap.sidnlabs.nl/

Datasets | Networks | Tools

DMAP is a scalable web scanning suit which supports DNS, HTTPS, TLS, and SMTP. It works based on domain names and crawls the domain for all supported protocols. The advantage over other tools is the unified SQL data model with 166 features and the easy scalability over many crawling machines.


DNS Authoritative Server Benchmarks

 https://www.knot-dns.cz/benchmark/

DNSSEC | Datasets

The website is an ongoing project by Knot DNS to measure the performance of various DNS servers. Four open source servers are tested, namely BIND, Knot DNS, NSD, and PowerDNS. The benchmark includes different zone configurations matching to root zones, TLD zones, or hosting zones as well as different DNSSEC configurations.


DNS Census 2013

 https://dnscensus2013.neocities.org/index.html

Datasets | IP | Networks

The DNS Census 2013 consist of about 2.5 billion DNS records collected in 2012/2013. The data is gathered from some available zone files and passive or active DNS collecting. The DNS records are written into CSV files containing one DNS record per line.


DNS Coffee

 https://dns.coffee/

Datasets | IP | Networks

DNS Coffee collects and archives stats from DNS Zone files in order to provide insights into the growth and changes in DNS over time.

The website includes information such as the size of different zones. It track over 1200 zone files.

It provides searching through the zones files based on domain names, name servers, or IP addresses. It can also visualize the relationship between a domain, the parent zones and the name server in what they call a "Trust Tree".


DNS Privacy Project

 https://dnsprivacy.org/wiki/

Datasets | DNSSEC | TLS

The DNS Privacy Project aims to improve privacy for users on the Internet.

The project is split into different groups working on DNS privacy:

The project focusses mostly on DNS over TLS. They provide overviews for the implementation status, configuration for test servers, and ongoing server monitoring which features they provide.


DNS Quality/Overview Tools

Datasets | DNSSEC | Networks | Tools

Check My DNS

Browser-based DNS resolver quality measurement tool. Uses the browser to generate many resolver queries and tests for features they should have, such as EDNS support, IPv6, QNAME Minimisation, etc.

This test is also available as a CLI tool: https://github.com/DNS-OARC/cmdns-cli

DNSSEC Debugger

Analyze DNSSEC deployment for a zone and show errors in the configuration.

DNSViz

Gives an overview over DNSSEC delegations, response sizes, and name servers.

GitHub: https://github.com/dnsviz/dnsviz

DNS X-Ray

The website has an online test, which performs DNS lookups. These DNS lookups test if certain resource records are overwritten in the cache. The tool can then determine what DNS software is used, where the server is located, how many caches there are, etc.

EDNS Compliance Tester

Test name server of zones for correct EDNS support.

The Transitive Trust and DNS Dependency Graph Portal

Shows the trust dependencies in DNS. Given a domain name it can show how zones delegate to each other and why. The delegation is done between IP addresses and zones.

Root Canary Project

The project used to monitor the first root KSK key rollover. Now it contains the paper "Roll, Roll, Roll your Root: A Comprehensive Analysis of the FirstEver DNSSEC Root KSK Rollover" describing the experiences of the first root KSK rollover

Additionally, it includes a tester for DNSSEC algorithm support, which shows the algorithms supported by the currently used recursive resolver. It provides statistics about support for DNSSEC algorithms. It has a web based test to test your own resolver and provides a live monitoring using the RIPA Atlas.


DNS Queries to Authoritative DNS Server at SURFnet by Google's Public DNS Resolver

 https://data.4tu.nl/articles/dataset/DNS_Queries_to_Authoritative_DNS_Server_at_SURFnet_by_Google_s_Public_DNS_Resolver/12682040

Datasets | Networks

This dataset covers approximately 3.5 billion DNS queries that were received at one of SURFnet's authoritative DNS servers from Google's Public DNS Resolver. The queries were collected during 2.5 years. The dataset contains only those queries that contained an EDNS Client Subnet.

The dataset covers data from 2015-06 through 2018-01.

DOI Identifier


DNS Replay Tool (drool)

 https://www.dns-oarc.net/tools/drool

IP | Networks | PCAPs | Tools

Tool to replay DNS queries captured in a pcap file with accurate timing between queries. Allows modifying the replay like changing IP addresses, speed up or slow down the queries.


DNSCAP

 https://www.dns-oarc.net/tools/dnscap

IP | Networks | PCAPs | Tools

DNS network capture utility. Similar in concept to tcpdump, but with specialized options for DNS.


DNSDB

 https://scout.dnsdb.info/

Datasets | Networks

Historical DNS database. Contains information recorded at recursive resolver about domain names, first/last seen, current bailiwick. Allows to see the lifetime of resource records and can be used as a large database.


dnsdumpster

 https://dnsdumpster.com/

Datasets | Tools

dnsdumpster.com fetches a lot of DNS information belonging to one domain. It checks the authorative name servers, which records exist, and where the servers are located.


DNSMON

 https://atlas.ripe.net/dnsmon/

Datasets | Networks

Historical information about the reachability of root and some TLD name servers.


dnsperf and resperf

 https://www.dns-oarc.net/tools/dnsperf

IP | Networks | PCAPs | Tools

DNS performance measurement tools.


DNSSEC Deployment Reports

 https://rick.eng.br/dnssecstat/

Datasets | DNSSEC | Networks

Regularly updated reports about current DNSSEC deployment. Contains information per TLD and global distribution.


DNSSEC Early Warning System

 https://www.dnssek.info/

Datasets | DNSSEC

The website keeps track of all DNSSEC keys in the top level domains (TLDs) and informs when the signatures are about to expire. The time before some RRSIGs expire is color coded. It also shows error which happened during validation.


dnsstream (Twitter)

 https://twitter.com/dnsstream

Datasets | Networks

@dnsstream is a Twitter bot, which sends out notifications for important DNS changes of domains.

  • Potential DDoS attacks
  • Domains which link to know malicious IPs
  • Name server changes for a domain

dnsteal DNS Exfiltration Tool

 https://github.com/m57/dnsteal

CTF | Tools

dnsteal provides a fake DNS server and encodes a file into a series of DNS requests. The fake DNS server then reassembles the file. This can be used to hide the file exfiltration as DNS traffic, however, since it doesn't use the default DNS server it is quite noisy.


dnsthought

 https://dnsthought.nlnetlabs.nl/

Datasets | DNSSEC | Networks

Dnsthought list many statistics about the resolvers visible to the .nl-authoritative name servers. The data is gathered from the RIPE Atlas probes. There is a dashboard which only works partially.

Raw data access is also available.


DNSTOP

 http://dns.measurement-factory.com/tools/dnstop/

Networks | Tools

Top-like utility showing information about captured DNS requests. It shows information about the domains queries, the types, and responses.


Domain Crawling Lists

Datasets

Domain popularity lists provide a starting point for crawling domains with the most users. The most commonly used list for security research is the Alexa list.

  • Alexa
    The list is updated daily and contains one million websites. The ranking is based on page views, but very volatile.
  • CISCO Umbrella
    The list is updated daily and contains one million websites. The ranking is based on traffic seen on the OpenDNS resolvers.
  • Majestic
    The list is updated daily and contains one million websites. The ranking is based on backlinks from other websites.
  • Tranco
    A Research-Oriented Top Sites Ranking Hardened Against Manipulation
    The Tranco list aims to provide a better list for security research. The authors explain on their website and their paper what the flaws in the existing lists
  • Quantcast
    The list is updated daily and contains around 500,000 websites. It is based on users visiting the site within the previous month and highly US focussed.

Flamethrower

 https://github.com/DNS-OARC/flamethrower

IP | Networks | Tools

Flamethrower is a small, fast, configurable tool for functional testing, benchmarking, and stress testing DNS servers and networks. It supports IPv4, IPv6, UDP, TCP, DoT, and DoH and has a modular system for generating queries used in the tests.


Forward DNS Rapid7

 https://opendata.rapid7.com/sonar.fdns_v2/

Datasets | IP | Networks

This dataset contains the responses to DNS requests for all forward DNS names known by Rapid7's Project Sonar. Until early November 2017, all of these were for the 'ANY' record with a fallback A and AAAA request if neccessary. After that, the ANY study represents only the responses to ANY requests, and dedicated studies were created for the A, AAAA, CNAME and TXT record lookups with appropriately named files.

The data is updated every month. Historic data can be downloaded after creating a free account.


Get your public IP using DNS

 https://docs.google.com/spreadsheets/d/1lkNJ0uQwbeC1ZTRrxdtuPLCIl7mlUreoKfSIgajnSyY/htmlview

IP

Some Open DNS operators provide a way to return the IP address of the request packet.

Google DNS:

dig o-o.myaddr.l.google.com txt @ns1.google.com +short

OpenDNS:

dig myip.opendns.com @resolver1.opendns.com +short

Akamai:

dig whoami.akamai.net. @ns1-1.akamaitech.net. +short

Source


Hello DNS

 https://powerdns.org/hello-dns/

DNSSEC | Tutorials

Hello DNS is a project to write a easy to read/understand summary of the DNS specification. It provides an entrypoint to understand DNS given that the full DNS specification is easily 2000 pages in size.


ICANN Managed Root Servers Statistics

 https://stats.dns.icann.org/

Datasets

The Grafana dashboard shows live statistics about query volume, query type, and geographic locations. The data is collected for ICANN Managed Root Servers (IMRS) which are the L-root servers.


Internet Maps (RIPE NCC)

 https://atlas.ripe.net/results/maps/

Datasets | Maps | Networks

Maps of measurements done with the RIPE Atlas.


iodine DNS Tunnel

 https://code.kryo.se/iodine/

Tools

iodine allows to tunnel IPv4 traffic through a DNS server. This can be used if network access is restricted, but DNS is unfiltered, for example in when a captive portal is deployed.


List of DNS related RFCs

 https://powerdns.org/dns-camel/

Networks

Contains information about the state of the RFC and what kind of information they contain.


Lists of DNS Blacklists

Datasets | IP | Networks | Spam | Tools

These projects either operate DNS based Real-time Blackhole Lists (RBL) or allow checking if an IP is contained. The Multi-RBL websites are helpful in finding a large quantity of RBLs.


NAT64 Testers

 https://www.nat64check.org

Networks | Tools

This website measures support for NAT64 in other websites.


netray.io Internet Observatory

 https://netray.io/

Certificates | Datasets | Networks

The Internet Observatory is a project by the RWTH Aachen University. It combines different scanning projects.

As of writing it contains information about:

  • DNS
  • HTTP2 and Server Push
  • QUIC
  • TCP Initial Window
  • Certificate Authority Aurthoization (CAA)

NextDNS

 https://nextdns.io/

Tools

A free and configurable DNS resolver. It provides customizable blocking, such as for ads, trackers, or malicious websites. Additionally, statistics can be shown, such as for most blocked website.

A similar self-hosted variant is Pi-hole.


Open INTEL

 https://www.openintel.nl/

Datasets | IP | Networks

Open INTEL is an active DNS database. It gathers information from public zone files, domain lists (Alexa, Umbrella), and reverse DNS entries. Once every 24 hours data is collected about a bunch of DNS RRsets (SOA, NS, A, AAAA, MX, TXT, DNSKEY, DS, NSEC3, CAA, CDS, CDNSKEY). The data is openly avaible as AVRO files and dates back until 2016.

The data can be freely downloaded. There is documentation on the layout of the AVRO files.

The project is similar to Active DNS but seems to be larger in scope.


OWASP Amass

 https://github.com/OWASP/Amass

CTF | Tools

The OWASP Amass tool suite obtains subdomain names by scraping data sources, recursive brute forcing, crawling web archives, permuting/altering names and reverse DNS sweeping. Additionally, Amass uses the IP addresses obtained during resolution to discover associated netblocks and ASNs. All the information is then used to build maps of the target networks.


Passive DNS (CIRCL)

 https://www.circl.lu/services/passive-dns/

Datasets | IP | Networks

Passive DNS dataset from circl.lu.


Passive DNS – Common Output Format

 https://github.com/adulau/pdns-qof

The Passive DNS Common Output Format describes a format used for querying passive DNS interfaces. The format is currently an IETF RFC draft. The format is used by CERT.at, Farsight, and CIRCL, as well as other projects.


Pi-hole

 https://pi-hole.net/

Tools

A free and configurable DNS stub-resolver. It provides customizable blocking, such as for ads, trackers, or malicious websites. Additionally, statistics can be shown, such as for most blocked website.

It can also function as a DHCP server for clients on the same network.

A similar service is NextDNS.


Public DNS Server List

 https://www.publicdns.xyz/

Datasets

The website provides a currated list of various public DNS resolver operators and the IP addresses of the DNS servers.


Public Suffix List

 https://publicsuffix.org/

Datasets | Networks

The public suffix list gives a way to easily determine the effective second level domain, i.e., the domain which a domain owner registered and which can be under different owners.


Resolver Testbed

 https://github.com/icann/resolver-testbed

Tools

This repo describes a testbed to test various DNS resolvers. The purpose of the testbed is to allow researchers to set up many resolvers and run tests on each. For example, a test might see what the resolver emits when it is priming, or when it is responding to a particular query while using DNSSEC validation.


respdiff

 https://gitlab.labs.nic.cz/knot/respdiff

Networks | Tools

DNS responses gathering and differences analysis toolchain.


RFC 8145 Root Trust Anchor Reports

 http://root-trust-anchor-reports.research.icann.org/

Datasets | DNSSEC

The root trust anchor reports show statistics how far the support for different root signing keys is in the resolver population. The data is collected using the trust anchor reporting specified in RFC 8145. There are graphs showing the distribution over time, combined for all root servers or split per letter, and a list of IP addresses which are only reporting support for outdated key signing keys (KSK).


RIPE Atlas

 https://atlas.ripe.net/

Certificates | Datasets | IP | Networks

RIPE operates a set of probes, which can be used to send pings or similar measurements. The probes are mainly placed in Europe but some are also in other continents.

All the collected measurements can be found in the RIPE Atlas Daily Archives. The blog post gives some more details.


Root Servers

 https://root-servers.org

Datasets | Tools

Overview page for the DNS root servers. It contains links to general news and all the supporting organizations.

The website features a map with all geographic locations. It contains information about locations, IPv4/IPv6 reachability and IP addresses.

Each root server has its own subdomain in the form of http://a.root-servers.org. It contains access to historical performance data like:

  • Size and time of zone updates
  • RCODE volume
  • query and response sizes for UDP and TCP
  • traffic volume (packets per time)
  • Unique sources

Routing Information Service (RIS)

 https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris

BGP | Datasets | Networks | Tools

Different information regarding reachability and connectiveness of ASs.


scans.io Internet-Wide Scan Data Repository

 https://scans.io/

Certificates | Datasets | IP | Networks

The website contains no usable data anymore and only links to empty pages on Censys.

The website used to host many free Internet scans of different kinds. It included historical data and activly maintained datasets.


Shadowserver Scanning Project

 https://scan.shadowserver.org/

Datasets | Networks

The Shadowserver Scanning projects performs regular Internet wide scans for many protocols. They scan for four main types of protocols:

  1. Amplification protocols, e.g., DNS or NTP
  2. Botnet protocols, e.g., Gameover Zeus or Sality
  3. Protocols that should not be exposed, e.g., Elastic Search, LDAP, or RDP
  4. Vulnerable Protocols, e.g., SSLv3

The website is a great resource to get general statistics about the protocols, like the number of hosts speaking the protocol, their geographic distribution, associated ASNs, and the historic information.


Shodan

 https://www.shodan.io/

Certificates | Datasets | IP | Networks

Shodan performs regular scan on common ports.

Access is free, but requires registration. More results can be gained with a paid account.


SIDN Labs DNS Workbench

 https://workbench.sidnlabs.nl/

DNSSEC

The DNS workbench is a testbed which allows experimentation how different authoritative DNS servers answer to queries.

It covers five open source authoritative servers, namely Bind9, Knot, NSD4, PowerDNS, and Yadifa. The workbench contains zones to test the support for many different resource record (RR) types, DNSSEC validation and how invalid zones are managed, delegations, zone transfers, and potentially more.

Find the project on GitHub.


WAND Active Measurement Project

 https://amp.wand.net.nz/

Autonomous Systems | Datasets | Networks | Tools

AMP is a system designed to continuously perform active network measurements between a mesh of specialist monitor machines, as well as to other targets of interest. These measurements are used to provide both a view of long-term network performance as well as to detect notable network events when they happen.

The project is run with a custom client and server software. The measurement results can be viewed on the website. It includes traceroutes, latencies (DNS, HTTP, ICMP, TCP), HTTP page sizes, and packet loss. The software is available as open source.


Wildcard DNS for IP Addresses

IP | Networks | Tools

These services allow you to create a domain name for any IP address. The IP address is encoded into the domain name. An overview over different services can be found here.

Online Services

  • https://nip.io/ provides IPv4 only

    • Supports both . and - separators.
    • 10.0.0.1.nip.io resolves to 10.0.0.1
    • 192-168-1-250.nip.io resolves to 192.168.1.250
    • customer1.app.10.0.0.1.nip.io resolves to 10.0.0.1
    • magic-127-0-0-1.nip.io resolves to 127.0.0.1
  • https://sslip.io/ provides IPv4 and IPv6

    • Supports both . and - separators.
    • Provides the ability to use the service with your own branding.
    • 192.168.0.1.sslip.io resolves to 192.168.0.1
    • 192-168-1-250.sslip.io resolves to 192.168.1.250
    • www.192-168-0-1.sslip.io resolves to 192.168.0.1
    • –1.sslip.io resolves to ::1
    • 2a01-4f8-c17-b8f--2.sslip.io resolves to 2a01:4f8:c17:b8f::2

Self-hosted Options

  • hipio is a Haskell service for IPv4.

ZMap Project

 https://zmap.io/

IP | Networks | Tools

Different utilities for network scanning. Most imporantly the zmap component, which is a packet scanner for different protocols. It also contains other tools like ways to iterate over the IPv4 address space and blacklist/whitelist management.