Historic datasets (from 2014 onwards) for the .nl TLD. Datasets are available in JSON format.
Datasets cover information about:
Historical DNS database. Access can be requested for academic use.
Activly queries many DNS records, e.g., .com zone.
It can contain information not in DNSDB, if the information was never seen by a resolver.
It does not contain all informatin, as some domains may be unknown to the project and thus cannot be crawled.
It uses popular zones, domain lists (e.g., Alexa, blacklists) and other domain feeds.
They normally maintain a rolling 14-day window.
Copy files (for date 2017-10-05) (ddos@gladbeck):
sftp -B1024000 -C -rp "activedns@kokino.gtisc.gatech.edu:active-dns/20171005/" .
The data is encocded in AVRO format, which can also be parsed as JSONL. Python has a AVRO library. AVRO schema:
{
"namespace": "astrolavos.avro",
"type": "record",
"name": "ActiveDns",
"fields": [
{"name": "date", "type": "string"},
{"name": "qname", "type": "string"},
{"name": "qtype", "type": "int"},
{"name": "rdata", "type": ["string", "null"]},
{"name": "ttl", "type": ["int", "null"]},
{"name": "authority_ips", "type": "string"},
{"name": "count", "type": "long"},
{"name": "hours", "type": "int"},
{"name": "source", "type": "string"},
{"name": "sensor", "type": "string"}
]
}
Some more information about some fields that are unique to that schema. The IPs in Authority IP are the collection of the authority name server IPs that replied to our query. We gather all the IPs that gave us the same answer for an entire day and concatenate them on the same field, mostly in order to reduce the number of records that we have to keep. The only field that might be slightly confusing, is the "hours" field. This is a 24bit integer that encodes the time of day we saw this RR for date date (for example, 000000000000000001000010 = 18:00 and 23:00). Another important thing to keep in mind, is NXDOMAINs. A resolved QNAME does not exist when both the rdata and ttl fields are equal to null. If rdata exists but ttl is null then the record was part of the glue of the DNS packet and not in the answer section.
Akamai provides aggregated data about the root servers. They provide the overall amount of DNS queries, trends, and IPv4/IPv6 query ratios.
APNIC gathers many different statistics and offers them on their website. However, they provide way more data then it might initially look like, since many of the datasets are not linked from their main page.
Censys performs regular scans for common protocols (e.g., DNS, HTTP(S), SSH). Provides a search for TLS certificates.
Access is free, but requires registration.
@InProceedings{censys15,
author = {Zakir Durumeric and David Adrian and Ariana Mirian and Michael Bailey and J. Alex Halderman},
title = {A Search Engine Backed by {I}nternet-Wide Scanning},
booktitle = {Proceedings of the 22nd {ACM} Conference on Computer and Communications Security},
month = oct,
year = 2015
}
Collection of "bad" packets in PCAPs that can be used for testing software.
Contains lists for DNS servers, libraries, tools, and other resources.
List of maintened and unmaintained DNS servers, including descriptions for each of them.
Provides an outdated list of different Cyber Thread Intelligence Feeds of other organizations.
DMAP is a scalable web scanning suit which supports DNS, HTTPS, TLS, and SMTP. It works based on domain names and crawls the domain for all supported protocols. The advantage over other tools is the unified SQL data model with 166 features and the easy scalability over many crawling machines.
The website is an ongoing project by Knot DNS to measure the performance of various DNS servers. Four open source servers are tested, namely BIND, Knot DNS, NSD, and PowerDNS. The benchmark includes different zone configurations matching to root zones, TLD zones, or hosting zones as well as different DNSSEC configurations.
The DNS Privacy Project aims to improve privacy for users on the Internet.
The project is split into different groups working on DNS privacy:
The project focusses mostly on DNS over TLS. They provide overviews for the implementation status, configuration for test servers, and ongoing server monitoring which features they provide.
Browser-based DNS resolver quality measurement tool. Uses the browser to generate many resolver queries and tests for features they should have, such as EDNS support, IPv6, QNAME Minimisation, etc.
This test is also available as a CLI tool: https://github.com/DNS-OARC/cmdns-cli
Analyze DNSSEC deployment for a zone and show errors in the configuration.
Gives an overview over DNSSEC delegations, response sizes, and name servers.
Github: https://github.com/dnsviz/dnsviz
The website has an online test, which performs DNS lookups. These DNS lookups test if certain resource records are overwritten in the cache. The tool can then determine what DNS software is used, where the server is located, how many caches there are, etc.
Test name server of zones for correct EDNS support.
Shows the trust dependencies in DNS. Given a domain name it can show how zones delegate to each other and why. The delegation is done between IP addresses and zones.
The project monitors the KSK rollover.
It provides statistics about support for DNSSEC algorithms. It has a web based test to test your own resolver and provides a live monitoring using the RIPA Atlas.
This dataset covers approximately 3.5 billion DNS queries that were received at one of SURFnet's authoritative DNS servers from Google's Public DNS Resolver. The queries were collected during 2.5 years. The dataset contains only those queries that contained an EDNS Client Subnet.
The dataset covers data from 2015-06 through 2018-01.
Tool to replay DNS queries captured in a pcap file with accurate timing between queries. Allows modifying the replay like changing IP addresses, speed up or slow down the queries.
DNS network capture utility. Similar in concept to tcpdump, but with specialized options for DNS.
Historical DNS database. Contains information recorded at recursive resolver about domain names, first/last seen, current bailiwick. Allows to see the lifetime of resource records and can be used as a large database.
Historical information about the reachability of root and some TLD name servers.
DNS performance measurement tools.
Regularly updated reports about current DNSSEC deployment. Contains information per TLD and global distribution.
The website keeps track of all DNSSEC keys in the top level domains (TLDs) and informs when the signatures are about to expire. The time before some RRSIGs expire is color coded. It also shows error which happened during validation.
@dnsstream is a Twitter bot, which sends out notifications for important DNS changes of domains.
Dnsthought list many statistics about the resolvers visible to the .nl-authoritative name servers.
Top-like utility showing information about captured DNS requests. It shows information about the domains queries, the types, and responses.
Domain popularity lists provide a starting point for crawling domains with the most users. The most commonly used list for security research is the Alexa list.
Hello DNS is a project to write a easy to read/understand summary of the DNS specification. It provides an entrypoint to understand DNS given that the full DNS specification is easily 2000 pages in size.
Maps of measurements done with the RIPE Atlas.
Contains information about the state of the RFC and what kind of information they contain.
These projects either operate DNS based Real-time Blackhole Lists (RBL) or allow checking if an IP is contained. The Multi-RBL websites are helpful in finding a large quantaty of RBLs.
These websites measure support for NAT64 in other websites.
The Internet Observatory is a project by the RWTH Aachen University. It combines different scanning projects.
As of writing it contains information about:
A free and configurable DNS resolver. It provides customizable blocking, such as for ads, trackers, or malicious websites. Additionally, statistics can be shown, such as for most blocked website.
A similar self-hosted variant is Pi-hole.
Open Resolver scanning project by the Shadowserver Foundation.
The OWASP Amass tool suite obtains subdomain names by scraping data sources, recursive brute forcing, crawling web archives, permuting/altering names and reverse DNS sweeping. Additionally, Amass uses the IP addresses obtained during resolution to discover associated netblocks and ASNs. All the information is then used to build maps of the target networks.
Passive DNS dataset from circl.lu.
A free and configurable DNS stub-resolver. It provides customizable blocking, such as for ads, trackers, or malicious websites. Additionally, statistics can be shown, such as for most blocked website.
It can also function as a DHCP server for clients on the same network.
A similar service is NextDNS.
The website provides a currated list of various public DNS resolver operators and the IP addresses of the DNS servers.
The public suffix list gives a way to easily determine the effective second level domain, i.e., the domain which a domain owner registered and which can be under different owners.
DNS responses gathering and differences analysis toolchain.
The root trust anchor reports show statistics how far the support for different root signing keys is in the resolver population. The data is collected using the trust anchor reporting specified in RFC 8145. There are graphs showing the distribution over time, combined for all root servers or split per letter, and a list of IP addresses which are only reporting support for outdated key signing keys (KSK).
RIPE operates a set of probes, which can be used to send pings or similar measurements. The probes are mainly placed in Europe but some are also in other continents.
All the collected measurements can be found in the RIPE Atlas Daily Archives. The blog post gives some more details.
Overview page for the DNS root servers. It contains links to general news and all the supporting organizations.
The website features a map with all geographic locations. It contains information about locations, IPv4/IPv6 reachability and IP addresses.
Each root server has its own subdomain in the form of http://a.root-servers.org. It contains access to historical performance data like:
RCODE volume
Different information regarding reachability and connectiveness of ASs.
A list of Internet scans for free to download. Some of the data is historical, some scans are still actively updated.
Links to a downloadable list of the Alexa top 1 million.
Shodan performs regular scan on common ports.
Access is free, but requires registration. More results can be gained with a paid account.
The DNS workbench is a testbed which allows experimentation how different authoritative DNS servers answer to queries.
It covers five open source authoritative servers, namely Bind9, Knot, NSD4, PowerDNS, and Yadifa. The workbench contains zones to test the support for many different resource record (RR) types, DNSSEC validation and how invalid zones are managed, delegations, zone transfers, and potentially more.
These services allow you to create a domain name for any IP address. The IP address is encoded into the domain name. An overview over different services can be found here.
http://xip.io/ provides IPv4 only
10.0.0.1.xip.io resolves to 10.0.0.1www.10.0.0.1.xip.io resolves to 10.0.0.1foo.bar.10.0.0.1.xip.io resolves to 10.0.0.1https://nip.io/ provides IPv4 only
. and - separators.10.0.0.1.nip.io resolves to 10.0.0.1192-168-1-250.nip.io resolves to 192.168.1.250customer1.app.10.0.0.1.nip.io resolves to 10.0.0.1magic-127-0-0-1.nip.io resolves to 127.0.0.1https://sslip.io/ provides IPv4 and IPv6
. and - separators.192.168.0.1.sslip.io resolves to 192.168.0.1192-168-1-250.sslip.io resolves to 192.168.1.250www.192-168-0-1.sslip.io resolves to 192.168.0.1–1.sslip.io resolves to ::12a01-4f8-c17-b8f--2.sslip.io resolves to 2a01:4f8:c17:b8f::2https://ip6.name/ provides IPv6 only
x replaces the :: in the IPv6 address.2001.db8.8000.0.0.0.0.1.ip6.name resolves to 2001:db8:8000::12001.db8.8000.x.1.ip6.name resolves to 2001:db8:8000::1x.1.ip6.name resolves to ::1
Different utilities for network scanning. Most imporantly the zmap component, which is a packet scanner for different protocols. It also contains other tools like ways to iterate over the IPv4 address space and blacklist/whitelist management.