All about IP

.nl stats and data - SIDN Labs

Datasets | DNS | DNSSEC | Networks

Historic datasets (from 2014 onwards) for the .nl TLD. Datasets are available in JSON format.

Datasets cover information about:

  • DNS
    • Domain Names
    • Query Type
    • Resonse Codes
    • IPv6 Support
  • Resolvers
    • Location
    • Number of IP addresses
    • Validating Resolvers
    • Popular Networks
    • Port Randomness
  • DNSSEC
    • Validating Queries
    • DANE
    • Used Algorithms
  • Mail
    • Mail RRs
    • SPF Information

Active DNS

Datasets | DNS | Networks

Historical DNS database. Access can be requested for academic use.

Activly queries many DNS records, e.g., .com zone. It can contain information not in DNSDB, if the information was never seen by a resolver. It does not contain all informatin, as some domains may be unknown to the project and thus cannot be crawled. It uses popular zones, domain lists (e.g., Alexa, blacklists) and other domain feeds.

They normally maintain a rolling 14-day window.

Copy files (for date 2017-10-05) (ddos@gladbeck):

sftp -B1024000 -C -rp "activedns@kokino.gtisc.gatech.edu:active-dns/20171005/" .

The data is encocded in AVRO format, which can also be parsed as JSONL. Python has a AVRO library. AVRO schema:

{
    "namespace": "astrolavos.avro",
    "type": "record",
    "name": "ActiveDns",
    "fields": [
        {"name": "date", "type": "string"},
        {"name": "qname", "type": "string"},
        {"name": "qtype", "type": "int"},
        {"name": "rdata", "type": ["string", "null"]},
        {"name": "ttl", "type": ["int", "null"]},
        {"name": "authority_ips", "type": "string"},
        {"name": "count", "type": "long"},
        {"name": "hours", "type": "int"},
        {"name": "source", "type": "string"},
        {"name": "sensor", "type": "string"}
    ]
}

Some more information about some fields that are unique to that schema. The IPs in Authority IP are the collection of the authority name server IPs that replied to our query. We gather all the IPs that gave us the same answer for an entire day and concatenate them on the same field, mostly in order to reduce the number of records that we have to keep. The only field that might be slightly confusing, is the "hours" field. This is a 24bit integer that encodes the time of day we saw this RR for date date (for example, 000000000000000001000010 = 18:00 and 23:00). Another important thing to keep in mind, is NXDOMAINs. A resolved QNAME does not exist when both the rdata and ttl fields are equal to null. If rdata exists but ttl is null then the record was part of the glue of the DNS packet and not in the answer section.


Caida Datasets Overview

BGP | Datasets | Networks

Overview of datasets, monitors, and reports produced and organized by Caida. Also contains links to other datasets.


Censys

Certificates | Datasets | DNS | Networks

Censys performs regular scans for common protocols (e.g., DNS, HTTP(S), SSH). Provides a search for TLS certificates.

Access is free, but requires registration.

@InProceedings{censys15,
    author = {Zakir Durumeric and David Adrian and Ariana Mirian and Michael Bailey and J. Alex Halderman},
    title = {A Search Engine Backed by {I}nternet-Wide Scanning},
    booktitle = {Proceedings of the 22nd {ACM} Conference on Computer and Communications Security},
    month = oct,
    year = 2015
}

Collection of "bad" packets in PCAPs

Datasets | DNS | Networks | PCAPs

Collection of "bad" packets in PCAPs that can be used for testing software.


Cyber Threat Intelligence Feeds

DNS | Networks | Spam

Provides an outdated list of different Cyber Thread Intelligence Feeds of other organizations.


dn42

BGP | Networks | VPN

dn42 is a big dynamic VPN. It employs various Internet technologies, such as BGP, whois, DNS, etc.

Users can experiment with technology, they normally would not use in a separated environment.

Mostly different hackerspaces participate in the dn42 network, such as different locations of the CCC.


DNS Replay Tool (drool)

DNS | Networks | PCAPs | Tools

Tool to replay DNS queries captured in a pcap file with accurate timing between queries. Allows modifying the replay like changing IP addresses, speed up or slow down the queries.


DNSCAP

DNS | Networks | PCAPs | Tools

DNS network capture utility. Similar in concept to tcpdump, but with specialized options for DNS.


dnsperf and resperf

DNS | Networks | PCAPs | Tools

DNS performance measurement tools.


IP to ASN Mapping (CIRCL LU)

Autonomous Systems Number | Datasets | Networks

Historical dataset about IP to ASN mappings.


IP to ASN Mapping (Cymru)

Autonomous Systems Number | Datasets | Networks

Historical dataset about IP to ASN mappings.


IPv6 Deployment Reports

Datasets | Networks

RIPE Report

Per continent, region, or country measurements of IPv6 deployment and preference. Allows to access historical data.

APNIC Report

Per continent, region, or country measurements of IPv6 deployment and preference.


IPv6 Hitlist Collection

Datasets | Networks

A curated list of IPv6 hosts, gathered by crawling different lists. Includes:

  • Alexa domains
  • Cisco Umbrella
  • CAIDA DNS names
  • Rapis7 DNS ANY and rDNS
  • Various zone files

Access to the full list requires registration by email.

Based on the paper "Scanning the IPv6 Internet: Towards a Comprehensive Hitlist".


Lists of DNS Blacklists

Datasets | DNS | Networks | Spam | Tools

These projects either operate DNS based Real-time Blackhole Lists (RBL) or allow checking if an IP is contained. The Multi-RBL websites are helpful in finding a large quantaty of RBLs.


Multilevel MDA-Lite Paris **Traceroute**

Networks | Tools

Multi-level MDA-Lite Paris Traceroute is a traceroute tool, which understands and learns more complex network topologies. Often times the network is not just a line, but multiple paths are possible and chosen at random.

A good description of the tool can be found in the RIPE Labs post or in the IMC 2018 paper.


Paris Traceroute

Networks | Tools

This is an improvement on the traditional traceroute program. It is able to detect multiple distinct routes and display them accordingly. The classical traceroute would produce weird results on changing network routes.


Passive DNS (CIRCL)

Datasets | DNS | Networks

Passive DNS dataset from circl.lu.


RIPE Atlas

Certificates | Datasets | DNS | Networks

RIPE operates a set of probes, which can be used to send pings or similar measurements. The probes are mainly placed in Europe but some are also in other continents.

All the collected measurements can be found in the RIPE Atlas Daily Archives. The blog post gives some more details.


scans.io Internet-Wide Scan Data Repository

Certificates | Datasets | DNS | Networks

A list of Internet scans for free to download. Some of the data is historical, some scans are still actively updated.

Links to a downloadable list of the Alexa top 1 million.


Shodan

Certificates | Datasets | DNS | Networks

Shodan performs regular scan on common ports.

Access is free, but requires registration. More results can be gained with a paid account.


traIXroute

Networks | Tools

A traceroute like tool, that detects where a path crosses an IXP.


Wildcard DNS for IP Addresses

DNS | Networks | Tools

These services allow you to create a domain name for any IP address. The IP address is encoded into the domain name. An overview over different services can be found here.

Online Services

  • http://xip.io/ provides IPv4 only

    • 10.0.0.1.xip.io resolves to 10.0.0.1
    • www.10.0.0.1.xip.io resolves to 10.0.0.1
    • foo.bar.10.0.0.1.xip.io resolves to 10.0.0.1
  • https://nip.io/ provides IPv4 only

    • Supports both . and - separators.
    • 10.0.0.1.nip.io resolves to 10.0.0.1
    • 192-168-1-250.nip.io resolves to 192.168.1.250
    • customer1.app.10.0.0.1.nip.io resolves to 10.0.0.1
    • magic-127-0-0-1.nip.io resolves to 127.0.0.1
  • https://sslip.io/ provides IPv4 and IPv6

    • Supports both . and - separators.
    • Provides the ability to use the service with a your own branding.
    • 192.168.0.1.sslip.io resolves to 192.168.0.1
    • 192-168-1-250.sslip.io resolves to 192.168.1.250
    • www.192-168-0-1.sslip.io resolves to 192.168.0.1
    • –1.sslip.io resolves to ::1
    • 2a01-4f8-c17-b8f--2.sslip.io resolves to 2a01:4f8:c17:b8f::2
  • https://ip6.name/ provides IPv6 only

    • A x replaces the :: in the IPv6 address.
    • 2001.db8.8000.0.0.0.0.1.ip6.name resolves to 2001:db8:8000::1
    • 2001.db8.8000.x.1.ip6.name resolves to 2001:db8:8000::1
    • x.1.ip6.name resolves to ::1

Self-hosted Options

  • hipio is a Haskell service for IPv4.

ZMap Project

DNS | Networks | Tools

Different utilities for network scanning. Most imporantly the zmap component, which is a packet scanner for different protocols. It also contains other tools like ways to iterate over the IPv4 address space and blacklist/whitelist management.