All about Networks

.nl stats and data - SIDN Labs

Datasets | DNS | DNSSEC | IP

Historic datasets (from 2014 onwards) for the .nl TLD. Datasets are available in JSON format.

Datasets cover information about:

  • DNS
    • Domain Names
    • Query Type
    • Resonse Codes
    • IPv6 Support
  • Resolvers
    • Location
    • Number of IP addresses
    • Validating Resolvers
    • Popular Networks
    • Port Randomness
  • DNSSEC
    • Validating Queries
    • DANE
    • Used Algorithms
  • Mail
    • Mail RRs
    • SPF Information

Active DNS

Datasets | DNS | IP

Historical DNS database. Access can be requested for academic use.

Activly queries many DNS records, e.g., .com zone. It can contain information not in DNSDB, if the information was never seen by a resolver. It does not contain all informatin, as some domains may be unknown to the project and thus cannot be crawled. It uses popular zones, domain lists (e.g., Alexa, blacklists) and other domain feeds.

They normally maintain a rolling 14-day window.

Copy files (for date 2017-10-05) (ddos@gladbeck):

sftp -B1024000 -C -rp "activedns@kokino.gtisc.gatech.edu:active-dns/20171005/" .

The data is encocded in AVRO format, which can also be parsed as JSONL. Python has a AVRO library. AVRO schema:

{
    "namespace": "astrolavos.avro",
    "type": "record",
    "name": "ActiveDns",
    "fields": [
        {"name": "date", "type": "string"},
        {"name": "qname", "type": "string"},
        {"name": "qtype", "type": "int"},
        {"name": "rdata", "type": ["string", "null"]},
        {"name": "ttl", "type": ["int", "null"]},
        {"name": "authority_ips", "type": "string"},
        {"name": "count", "type": "long"},
        {"name": "hours", "type": "int"},
        {"name": "source", "type": "string"},
        {"name": "sensor", "type": "string"}
    ]
}

Some more information about some fields that are unique to that schema. The IPs in Authority IP are the collection of the authority name server IPs that replied to our query. We gather all the IPs that gave us the same answer for an entire day and concatenate them on the same field, mostly in order to reduce the number of records that we have to keep. The only field that might be slightly confusing, is the "hours" field. This is a 24bit integer that encodes the time of day we saw this RR for date date (for example, 000000000000000001000010 = 18:00 and 23:00). Another important thing to keep in mind, is NXDOMAINs. A resolved QNAME does not exist when both the rdata and ttl fields are equal to null. If rdata exists but ttl is null then the record was part of the glue of the DNS packet and not in the answer section.


BGPlay

BGP | Datasets | Tools

BGPlay shows a graph of the observed BGP routes. It allows to replay historical BGP announcements and displays route changes.

Documentation
Github


BGPmon Archive

BGP | Datasets

Downloadable dataset of historic BGP information from different vantage points.


BGPStream (CAIDA)

BGP | Datasets | Tools

An open-source software framework for live and historical BGP data analysis, supporting scientific research, operational monitoring, and post-event analysis.

BGP streams are freely accesible and provided by Route View, RIPE, and BGPmon.


BGPStream (OpenDNS)

BGP | Datasets

BGP Stream is a free resource for receiving alerts about hijacks, leaks, and outages in the Border Gateway Protocol.

BGP Steam provides real-time information about BGP events. It includes information about affected IPs, ASNs, and even a replay feature how the BGP announcements changed.

A live alert bot also exists on Twitter.


Caida Datasets Overview

BGP | Datasets | IP

Overview of datasets, monitors, and reports produced and organized by Caida. Also contains links to other datasets.


Censys

Certificates | Datasets | DNS | IP

Censys performs regular scans for common protocols (e.g., DNS, HTTP(S), SSH). Provides a search for TLS certificates.

Access is free, but requires registration.

@InProceedings{censys15,
    author = {Zakir Durumeric and David Adrian and Ariana Mirian and Michael Bailey and J. Alex Halderman},
    title = {A Search Engine Backed by {I}nternet-Wide Scanning},
    booktitle = {Proceedings of the 22nd {ACM} Conference on Computer and Communications Security},
    month = oct,
    year = 2015
}

Collection of "bad" packets in PCAPs

Datasets | DNS | IP | PCAPs

Collection of "bad" packets in PCAPs that can be used for testing software.


Common Crawl

Datasets

The Common Crawl project builds an openly accessible database of crawled websites. They index can be searched.


Cyber Threat Intelligence Feeds

DNS | IP | Spam

Provides an outdated list of different Cyber Thread Intelligence Feeds of other organizations.


DDoS Mon

Amplification | Datasets | Denial-of-Service

Provides a search interface to search for domain names and IP addresses under attacks. Shows results for the last 30 days. Provides an API, which requires special authorization.


DMAP Domain Mapper by SIDN Labs

Datasets | DNS | Tools

DMAP is a scalable web scanning suit which supports DNS, HTTPS, TLS, and SMTP. It works based on domain names and crawls the domain for all supported protocols. The advantage over other tools is the unified SQL data model with 166 features and the easy scalability over many crawling machines.


dn42

BGP | IP | VPN

dn42 is a big dynamic VPN. It employs various Internet technologies, such as BGP, whois, DNS, etc.

Users can experiment with technology, they normally would not use in a separated environment.

Mostly different hackerspaces participate in the dn42 network, such as different locations of the CCC.


DNS Quality/Overview Tools

Datasets | DNS | DNSSEC | Tools

Check My DNS

Browser-based DNS resolver quality measurement tool. Uses the browser to generate many resolver queries and tests for features they should have, such as EDNS support, IPv6, QNAME Minimisation, etc.

This test is also available as a CLI tool: https://github.com/DNS-OARC/cmdns-cli

DNSSEC Debugger

Analyze DNSSEC deployment for a zone and show errors in the configuration.

DNSViz

Gives an overview over DNSSEC delegations, response sizes, and name servers.

Github: https://github.com/dnsviz/dnsviz

DNS X-Ray

The website has an online test, which performs DNS lookups. These DNS lookups test if certain resource records are overwritten in the cache. The tool can then determine what DNS software is used, where the server is located, how many caches there are, etc.

EDNS Compliance Tester

Test name server of zones for correct EDNS support.

The Transitive Trust and DNS Dependency Graph Portal

Shows the trust dependencies in DNS. Given a domain name it can show how zones delegate to each other and why. The delegation is done between IP addresses and zones.

Root Canary Project

The project monitors the KSK rollover.

It provides statistics about support for DNSSEC algorithms. It has a web based test to test your own resolver and provides a live monitoring using the RIPA Atlas.


DNS Queries to Authoritative DNS Server at SURFnet by Google's Public DNS Resolver

Datasets | DNS

This dataset covers approximately 3.5 billion DNS queries that were received at one of SURFnet's authoritative DNS servers from Google's Public DNS Resolver. The queries were collected during 2.5 years. The dataset contains only those queries that contained an EDNS Client Subnet.

The dataset covers data from 2015-06 through 2018-01.


DNS Replay Tool (drool)

DNS | IP | PCAPs | Tools

Tool to replay DNS queries captured in a pcap file with accurate timing between queries. Allows modifying the replay like changing IP addresses, speed up or slow down the queries.


DNSCAP

DNS | IP | PCAPs | Tools

DNS network capture utility. Similar in concept to tcpdump, but with specialized options for DNS.


DNSDB

Datasets | DNS

Historical DNS database. Contains information recorded at recursive resolver about domain names, first/last seen, current bailiwick. Allows to see the lifetime of resource records and can be used as a large database.


DNSMON

Datasets | DNS

Historical information about the reachability of root and some TLD name servers.


dnsperf and resperf

DNS | IP | PCAPs | Tools

DNS performance measurement tools.


DNSSEC Deployment Reports

Datasets | DNS | DNSSEC

Regularly updated reports about current DNSSEC deployment. Contains information per TLD and global distribution.


dnsstream (Twitter)

Datasets | DNS

@dnsstream is a Twitter bot, which sends out notifications for important DNS changes of domains.

  • Potential DDoS attacks
  • Domains which link to know malicious IPs
  • Name server changes for a domain

dnsthought

Datasets | DNS | DNSSEC

Dnsthought list many statistics about the resolvers visible to the .nl-authoritative name servers.


DNSTOP

DNS | Tools

Top-like utility showing information about captured DNS requests. It shows information about the domains queries, the types, and responses.


Driftnet

CTF | Tools

Driftnet watches network traffic, and picks out and displays JPEG and GIF images for display.


Internet Maps (RIPE NCC)

Datasets | DNS

Maps of measurements done with the RIPE Atlas.


IP to ASN Mapping (CIRCL LU)

Autonomous Systems Number | Datasets | IP

Historical dataset about IP to ASN mappings.


IP to ASN Mapping (Cymru)

Autonomous Systems Number | Datasets | IP

Historical dataset about IP to ASN mappings.


IPv6 Deployment Reports

Datasets | IP

RIPE Report

Per continent, region, or country measurements of IPv6 deployment and preference. Allows to access historical data.

APNIC Report

Per continent, region, or country measurements of IPv6 deployment and preference.


IPv6 Hitlist Collection

Datasets | IP

A curated list of IPv6 hosts, gathered by crawling different lists. Includes:

  • Alexa domains
  • Cisco Umbrella
  • CAIDA DNS names
  • Rapis7 DNS ANY and rDNS
  • Various zone files

Access to the full list requires registration by email.

Based on the paper "Scanning the IPv6 Internet: Towards a Comprehensive Hitlist".


IXP Pricing Overview

BGP | Datasets

Contains a list of pricing information of different IXP.


List of Amplification Protocols

Amplification | Datasets | Denial-of-Service

Contains a list of UDP-based protocols, which can be used for amplification attacks.


List of BGP Routing Datasets

BGP | Datasets

Isolario

Isolario also provides historial routing data in MTR format for their route collectors. The data contains snapshots every two hours and updates with a granularity of five minutes.

Packet Clearing House (PCH)

The Packet Clearing House (PCH) publishes BGP data collected at more than 100 internet exchange points (IXP). The snapshot dataset contains the state of the routing tables in daily intervals.

PCH also provides raw routing data in MRT format. These contain all the update information in sorted by time.

Routing Information Service (RIS)

The RIS is the main resource from RIPE featuring all kinds of datasets about AS assignments and connectivity.

Routeviews

Routeviews is a project by the University of Oregon to provide live and historical BGP routing data.


List of DNS related RFCs

DNS

Contains information about the state of the RFC and what kind of information they contain.


List of Looking Glasses Providing Traceroutes

Datasets

The websites shows links to different looking glasses which provide either traceroute information or are usable as route servers.


Lists of DNS Blacklists

Datasets | DNS | IP | Spam | Tools

These projects either operate DNS based Real-time Blackhole Lists (RBL) or allow checking if an IP is contained. The Multi-RBL websites are helpful in finding a large quantaty of RBLs.


Multilevel MDA-Lite Paris **Traceroute**

IP | Tools

Multi-level MDA-Lite Paris Traceroute is a traceroute tool, which understands and learns more complex network topologies. Often times the network is not just a line, but multiple paths are possible and chosen at random.

A good description of the tool can be found in the RIPE Labs post or in the IMC 2018 paper.


NAT64 Testers

DNS | Tools

These websites measure support for NAT64 in other websites.


Netlab 360 OpenData Project

Amplification | Datasets | Tools

The Netlab of 360.com provides some open data streams.

One dataset concerns the number of abused reflectors per protocol.


netray.io Internet Observatory

Certificates | Datasets | DNS

The Internet Observatory is a project by the RWTH Aachen University. It combines different scanning projects.

As of writing it contains information about:

  • DNS
  • HTTP2 and Server Push
  • QUIC
  • TCP Initial Window
  • Certificate Authority Aurthoization (CAA)

NetworkScan Mon

Amplification | Datasets | Tools

Overview over IP addresses scanning the internet and which ports are scanned.


nmap Stylesheet

The nmap stylesheet converts the nmap XML output into a nice website. A sample report can be found under this link.


Open Resolver Scan

Datasets | DNS

Open Resolver scanning project by the Shadowserver Foundation.


OpenIPmap RIPE

BGP | Datasets | Tools

IP geolocation services feeding itself from geolocation databases, user provided locations, and most importantly active RTT measurements based on the RIPE Atlas system. It also provides a nice API to query the location. It provides a breakdown on where the results stem from and how much they contribute to the overall result.


Paris Traceroute

IP | Tools

This is an improvement on the traditional traceroute program. It is able to detect multiple distinct routes and display them accordingly. The classical traceroute would produce weird results on changing network routes.


Passive DNS (CIRCL)

Datasets | DNS | IP

Passive DNS dataset from circl.lu.


PEERING: The BGP Testbed

BGP | Tools

PEERING is an environment where researchers and educators can play with BGP announcements in a real but sandboxed environment.

Description from the website:

The long-term goal of the PEERING system is to enable on-demand, safe, and controlled access to the Internet routing ecosystem for researchers and educators:

  • PEERING for researchers. Today, it is hard for researchers to conduct Internet routing experiments. To perform a routing experiment, a research institution has to obtain Internet resources (IP addresses and ASNs) and establish relations with upstream networks. PEERING eliminates these obstacles and provides researchers controlled on-demand access to the routing ecosystem.
  • PEERING for educators. Educators can use the PEERING infrastructure in teaching students the Internet routing architecture. The students access to live BGP sessions to multiple ISPs.

PeeringDB

BGP | Datasets

Contains information for some networks about peering information. This includes peering partnes, transfer speeds, peering requirements and similar.

Documentation


Public Suffix List

Datasets | DNS

The public suffix list gives a way to easily determine the effective second level domain, i.e., the domain which a domain owner registered and which can be under different owners.


respdiff

DNS | Tools

DNS responses gathering and differences analysis toolchain.


RIPE Atlas

Certificates | Datasets | DNS | IP

RIPE operates a set of probes, which can be used to send pings or similar measurements. The probes are mainly placed in Europe but some are also in other continents.

All the collected measurements can be found in the RIPE Atlas Daily Archives. The blog post gives some more details.


Routing Information Service (RIS)

BGP | Datasets | DNS | Tools

Different information regarding reachability and connectiveness of ASs.


ROV Deployment Monitor

BGP | Datasets

The Route Origin Validation (ROV) Deployment Monitor measures how many AS have deployed ROV. It uses PEERING for BGP annoucements and uses BGP monitors to see in which ASs the wrong announcements are filtered. A blogpost at APNIC describes it in more detail.


RPKI Browsers

Datasets | RPKI | Tools

These websites allow you to browser the valid RPKI announcements. They show which address ranges are covered by RPKI and who the issuing authority is.


RPKI Tester

RPKI | Tools

Website which tests, if your provider filters invalid annoucements using RPKI.


scans.io Internet-Wide Scan Data Repository

Certificates | Datasets | DNS | IP

A list of Internet scans for free to download. Some of the data is historical, some scans are still actively updated.

Links to a downloadable list of the Alexa top 1 million.


Shodan

Certificates | Datasets | DNS | IP

Shodan performs regular scan on common ports.

Access is free, but requires registration. More results can be gained with a paid account.


snidump

CTF | Tools

This is a tcpdump-like program for printing TLS SNI and HTTP/1.1 Host fields in live or captured traffic.


Task-centered iproute2 user guide

Cheatsheet | Tutorials

Userguide for the newer ip command under Linux. The guide consists of different tasks one might want to perform and their corresponding ip commands.


Tcpdump advanced filters

Cheatsheet | Tools | Tutorials

The website contains different tcpdump filters. It starts with basic filters and then builds up ever more complex ones. This is a good source for looking up complicated filters, if one does not want to write them themself.


TeleGeography Map Gallery

Datasets

TeleGeography provides different maps about the Internet. They contain information about submarine cables, global traffic volume, latency, internet exchange points. The data for the Submarine Map and the Internet Exchange Map can also be found on Github in text format.


traIXroute

IP | Tools

A traceroute like tool, that detects where a path crosses an IXP.


vizAS

BGP | Tools | Datasets

vizAS by APNIC shows the connectiveness between different ASs split by countries. It is usefull to find the ASs which are most central in the graph.


Wildcard DNS for IP Addresses

DNS | IP | Tools

These services allow you to create a domain name for any IP address. The IP address is encoded into the domain name. An overview over different services can be found here.

Online Services

  • http://xip.io/ provides IPv4 only

    • 10.0.0.1.xip.io resolves to 10.0.0.1
    • www.10.0.0.1.xip.io resolves to 10.0.0.1
    • foo.bar.10.0.0.1.xip.io resolves to 10.0.0.1
  • https://nip.io/ provides IPv4 only

    • Supports both . and - separators.
    • 10.0.0.1.nip.io resolves to 10.0.0.1
    • 192-168-1-250.nip.io resolves to 192.168.1.250
    • customer1.app.10.0.0.1.nip.io resolves to 10.0.0.1
    • magic-127-0-0-1.nip.io resolves to 127.0.0.1
  • https://sslip.io/ provides IPv4 and IPv6

    • Supports both . and - separators.
    • Provides the ability to use the service with a your own branding.
    • 192.168.0.1.sslip.io resolves to 192.168.0.1
    • 192-168-1-250.sslip.io resolves to 192.168.1.250
    • www.192-168-0-1.sslip.io resolves to 192.168.0.1
    • –1.sslip.io resolves to ::1
    • 2a01-4f8-c17-b8f--2.sslip.io resolves to 2a01:4f8:c17:b8f::2
  • https://ip6.name/ provides IPv6 only

    • A x replaces the :: in the IPv6 address.
    • 2001.db8.8000.0.0.0.0.1.ip6.name resolves to 2001:db8:8000::1
    • 2001.db8.8000.x.1.ip6.name resolves to 2001:db8:8000::1
    • x.1.ip6.name resolves to ::1

Self-hosted Options

  • hipio is a Haskell service for IPv4.

Ziggy: the RPKI Wayback Machine

RPKI | Tools

Ziggy is a tool to inspect the RPKI ecosystem at arbitrary points in the past. It is developed by NlNetLabs. More details abouut the ziggy tool can be found in the announcement blogpost.


ZMap Project

DNS | IP | Tools

Different utilities for network scanning. Most imporantly the zmap component, which is a packet scanner for different protocols. It also contains other tools like ways to iterate over the IPv4 address space and blacklist/whitelist management.