All about IP

.nl stats and data - SIDN Labs

 https://stats.sidnlabs.nl/en/

Datasets | DNS | DNSSEC | Networks

Historic datasets (from 2014 onwards) for the .nl TLD. Datasets are available in JSON format.

Datasets cover information about:

  • DNS
    • Domain Names
    • Query Type
    • Resonse Codes
    • IPv6 Support
  • Resolvers
    • Location
    • Number of IP addresses
    • Validating Resolvers
    • Popular Networks
    • Port Randomness
  • DNSSEC
    • Validating Queries
    • DANE
    • Used Algorithms
  • Mail
    • Mail RRs
    • SPF Information

Active DNS

 https://www.activednsproject.org/

Datasets | DNS | Networks

Historical DNS database. Access can be requested for academic use.

Activly queries many DNS records, e.g., .com zone. It can contain information not in DNSDB, if the information was never seen by a resolver. It does not contain all informatin, as some domains may be unknown to the project and thus cannot be crawled. It uses popular zones, domain lists (e.g., Alexa, blacklists) and other domain feeds.

They normally maintain a rolling 14-day window.

Copy files (for date 2017-10-05) (ddos@gladbeck):

sftp -B1024000 -C -rp "activedns@kokino.gtisc.gatech.edu:active-dns/20171005/" .

The data is encocded in AVRO format, which can also be parsed as JSONL. Python has a AVRO library. AVRO schema:

{
    "namespace": "astrolavos.avro",
    "type": "record",
    "name": "ActiveDns",
    "fields": [
        {"name": "date", "type": "string"},
        {"name": "qname", "type": "string"},
        {"name": "qtype", "type": "int"},
        {"name": "rdata", "type": ["string", "null"]},
        {"name": "ttl", "type": ["int", "null"]},
        {"name": "authority_ips", "type": "string"},
        {"name": "count", "type": "long"},
        {"name": "hours", "type": "int"},
        {"name": "source", "type": "string"},
        {"name": "sensor", "type": "string"}
    ]
}

Some more information about some fields that are unique to that schema. The IPs in Authority IP are the collection of the authority name server IPs that replied to our query. We gather all the IPs that gave us the same answer for an entire day and concatenate them on the same field, mostly in order to reduce the number of records that we have to keep. The only field that might be slightly confusing, is the "hours" field. This is a 24bit integer that encodes the time of day we saw this RR for date date (for example, 000000000000000001000010 = 18:00 and 23:00). Another important thing to keep in mind, is NXDOMAINs. A resolved QNAME does not exist when both the rdata and ttl fields are equal to null. If rdata exists but ttl is null then the record was part of the glue of the DNS packet and not in the answer section.

A similar active DNS project is Open INTEL which seems to be larger in scope and the data is publicly available.


 https://www.akamai.com/us/en/why-akamai/dns-trends-and-traffic.jsp

Datasets | DNS

Akamai provides aggregated data about the root servers. They provide the overall amount of DNS queries, trends, and IPv4/IPv6 query ratios.


APNIC Labs Stats

 https://stats.labs.apnic.net/

Autonomous Systems | BGP | Datasets | DNS | DNSSEC

APNIC gathers many statistics and offers them on their website. However, they provide way more data than it might initially look like, since many of the datasets are not linked from their main page.


Caida Datasets Overview

 https://www.caida.org/catalog/datasets/overview/

Autonomous Systems | BGP | Datasets | Networks

Overview of datasets, monitors, and reports produced and organized by Caida. Also contains links to other datasets.


Censys

 https://censys.io/

Certificates | Datasets | DNS | Networks

Censys performs regular scans for common protocols (e.g., DNS, HTTP(S), SSH). Provides a search for TLS certificates.

Access is free, but requires registration. The website no longer provides free bulk access. Bulk access requires a commercial or a research license. The free access is limited to 1000 API calls per day.

@InProceedings{censys15,
    author = {Zakir Durumeric and David Adrian and Ariana Mirian and Michael Bailey and J. Alex Halderman},
    title = {A Search Engine Backed by {I}nternet-Wide Scanning},
    booktitle = {Proceedings of the 22nd {ACM} Conference on Computer and Communications Security},
    month = oct,
    year = 2015
}

Cloudflare Radar

 https://radar.cloudflare.com/

Datasets | Networks

Cloudflare Radar is Cloudflares reporting website about internet trends and general traffic statistics. The website shows information about observed attacks and attack types and links to the DDoS report. General traffic statistics are reported, such as the used browser, fraction of human traffic, IP, HTTP, and TLS version.

The website also provides more detailed information for domains and IP addresses. Domains have information about age, popularity, and visitors. IP addresses have ASN and geolocation information.

More information about Cloudflare Radar are available in the introduction blogpost.


Collection of "bad" packets in PCAPs

 https://github.com/DNS-OARC/bad-packets

Datasets | DNS | Networks | PCAPs

Collection of "bad" packets in PCAPs that can be used for testing software.


Cyber Threat Intelligence Feeds

 https://github.com/TW-NCERT/ctifeeds

DNS | Networks | Spam

Provides an outdated list of different Cyber Thread Intelligence Feeds of other organizations.


dn42

 https://dn42.eu/Home

BGP | Networks | VPN

dn42 is a big dynamic VPN. It employs various Internet technologies, such as BGP, whois, DNS, etc.

Users can experiment with technology, they normally would not use in a separated environment.

Mostly different hackerspaces participate in the dn42 network, such as different locations of the CCC.


DNS Census 2013

 https://dnscensus2013.neocities.org/index.html

Datasets | DNS | Networks

The DNS Census 2013 consist of about 2.5 billion DNS records collected in 2012/2013. The data is gathered from some available zone files and passive or active DNS collecting. The DNS records are written into CSV files containing one DNS record per line.


DNS Coffee

 https://dns.coffee/

Datasets | DNS | Networks

DNS Coffee collects and archives stats from DNS Zone files in order to provide insights into the growth and changes in DNS over time.

The website includes information such as the size of different zones. It track over 1200 zone files.

It provides searching through the zones files based on domain names, name servers, or IP addresses. It can also visualize the relationship between a domain, the parent zones and the name server in what they call a "Trust Tree".


DNS Replay Tool (drool)

 https://www.dns-oarc.net/tools/drool

DNS | Networks | PCAPs | Tools

Tool to replay DNS queries captured in a pcap file with accurate timing between queries. Allows modifying the replay like changing IP addresses, speed up or slow down the queries.


DNSCAP

 https://www.dns-oarc.net/tools/dnscap

DNS | Networks | PCAPs | Tools

DNS network capture utility. Similar in concept to tcpdump, but with specialized options for DNS.


dnsperf and resperf

 https://www.dns-oarc.net/tools/dnsperf

DNS | Networks | PCAPs | Tools

DNS performance measurement tools.


Dublin Traceroute

 https://dublin-traceroute.net/README.md

Networks | Tools

This is an improvement on Paris traceroute and the classical traceroute. It can detect changing routes and detect NATs along the path.


Entropy/IP

 https://github.com/akamai/entropy-ip

Tools

The Entropy/IP algorithm allows for inspecting and generating IPv6 addresses. Entropy/IP can determine the entropy of different nibbles and relationship between different components. Based on this analysis it can also create valid-looking IPv6 addresses.


Flamethrower

 https://github.com/DNS-OARC/flamethrower

DNS | Networks | Tools

Flamethrower is a small, fast, configurable tool for functional testing, benchmarking, and stress testing DNS servers and networks. It supports IPv4, IPv6, UDP, TCP, DoT, and DoH and has a modular system for generating queries used in the tests.


Forward DNS Rapid7

 https://opendata.rapid7.com/sonar.fdns_v2/

Datasets | DNS | Networks

This dataset contains the responses to DNS requests for all forward DNS names known by Rapid7's Project Sonar. Until early November 2017, all of these were for the 'ANY' record with a fallback A and AAAA request if neccessary. After that, the ANY study represents only the responses to ANY requests, and dedicated studies were created for the A, AAAA, CNAME and TXT record lookups with appropriately named files.

The data is updated every month. Historic data can be downloaded after creating a free account.


Get your public IP using DNS

 https://docs.google.com/spreadsheets/d/1lkNJ0uQwbeC1ZTRrxdtuPLCIl7mlUreoKfSIgajnSyY/htmlview

DNS

Some Open DNS operators provide a way to return the IP address of the request packet.

Google DNS:

dig o-o.myaddr.l.google.com txt @ns1.google.com +short

OpenDNS:

dig myip.opendns.com @resolver1.opendns.com +short

Akamai:

dig whoami.akamai.net. @ns1-1.akamaitech.net. +short

Source


IP Abuse Lists

Datasets | Networks

These websites have lists of abusive IP addresses. They can be checked with a web form or some websites also provide a feed.


IP to ASN Mapping (CIRCL LU)

 https://www.circl.lu/services/ip-asn-history/

Autonomous Systems | Datasets | Networks

Historical dataset about IP to ASN mappings.


IP to ASN Mapping (Cymru)

 https://team-cymru.com/community-services/ip-asn-mapping/

Autonomous Systems | Datasets | Networks

Historical dataset about IP to ASN mappings.


IPv4 Heatmap

 https://github.com/measurement-factory/ipv4-heatmap

Tools

The IPv4 heatmap tool draws an image of active IPv4 addresses. The IP addresses are mapped to pixels useing a Hilbert curve or a Z-curve. The image can be extended with annotations about the address space, for example to show which regional internet registry is assigned to the address.


IPv6 Deployment Reports

Datasets | Networks

RIPE Report

Per continent, region, or country measurements of IPv6 deployment and preference. Allows to access historical data.

APNIC Report

Per continent, region, or country measurements of IPv6 deployment and preference.


IPv6 Hitlist Collection

Datasets | Networks

https://www.net.in.tum.de/projects/gino/ipv6-hitlist.html

A curated list of IPv6 hosts, gathered by crawling different lists. Includes:

  • Alexa domains
  • Cisco Umbrella
  • CAIDA DNS names
  • Rapis7 DNS ANY and rDNS
  • Various zone files

Access to the full list requires registration by email.

Based on the paper "Scanning the IPv6 Internet: Towards a Comprehensive Hitlist".

https://ipv6hitlist.github.io/

The website contains the additional material of the IMC paper Clusters in the Expanse: Understanding and Unbiasing IPv6 Hitlists. The IPv6 addresses can be downloaded from the website. The website has three lists, responsive IPv6 addresses, aliased prefixes, and non-aliased prefixes. Additionally, the website also has a list of tools used during the data creation.


IPv6 Security/Network Tools

Tools


Lists of DNS Blacklists

Datasets | DNS | Networks | Spam | Tools

These projects either operate DNS based Real-time Blackhole Lists (RBL) or allow checking if an IP is contained. The Multi-RBL websites are helpful in finding a large quantity of RBLs.


Mini Internet Project

 https://github.com/nsg-ethz/mini_internet_project

BGP | Networks | Tools

The mini internet project is part of the curiculum by the Networked Systems Group of ETH Zurich. It teaches the students the basic steps how to create a mini internet. It starts with the basics of intra-network routing, by setting up multiple L2 switches. Then the students have to configure L3 routers to connect multiple L2 sites together. Lastly, in a big hackathon style, the students need to connect their local network with the network of the other students, by properly configuring BGP routers and setting up routing policies.

The code and the tasks are all available in the GitHub repository.

The APNIC Blog has a nice introduction to the project too.


Multilevel MDA-Lite Paris **Traceroute**

 https://gitlab.planet-lab.eu/cartography/

Networks | Tools

Multi-level MDA-Lite Paris Traceroute is a traceroute tool, which understands and learns more complex network topologies. Often times the network is not just a line, but multiple paths are possible and chosen at random.

A good description of the tool can be found in the RIPE Labs post or in the IMC 2018 paper.


Open INTEL

 https://www.openintel.nl/

Datasets | DNS | Networks

Open INTEL is an active DNS database. It gathers information from public zone files, domain lists (Alexa, Umbrella), and reverse DNS entries. Once every 24 hours data is collected about a bunch of DNS RRsets (SOA, NS, A, AAAA, MX, TXT, DNSKEY, DS, NSEC3, CAA, CDS, CDNSKEY). The data is openly avaible as AVRO files and dates back until 2016.

The data can be freely downloaded. There is documentation on the layout of the AVRO files.

The project is similar to Active DNS but seems to be larger in scope.


Paris Traceroute

 https://paris-traceroute.net/

Networks | Tools

This is an improvement on the traditional traceroute program. It is able to detect multiple distinct routes and display them accordingly. The classical traceroute would produce weird results on changing network routes.

Another similar program is Dublin traceroute.


Passive DNS (CIRCL)

 https://www.circl.lu/services/passive-dns/

Datasets | DNS | Networks

Passive DNS dataset from circl.lu.


RIPE Atlas

 https://atlas.ripe.net/

Certificates | Datasets | DNS | Networks

RIPE operates a set of probes, which can be used to send pings or similar measurements. The probes are mainly placed in Europe but some are also in other continents.

All the collected measurements can be found in the RIPE Atlas Daily Archives. The blog post gives some more details.


scans.io Internet-Wide Scan Data Repository

 https://scans.io/

Certificates | Datasets | DNS | Networks

The website contains no usable data anymore and only links to empty pages on Censys.

The website used to host many free Internet scans of different kinds. It included historical data and activly maintained datasets.


Shodan

 https://www.shodan.io/

Certificates | Datasets | DNS | Networks

Shodan performs regular scan on common ports.

Access is free, but requires registration. More results can be gained with a paid account.


traIXroute

 http://www.inspire.edu.gr/traIXroute/

Networks | Tools

A traceroute like tool, that detects where a path crosses an IXP.


Wildcard DNS for IP Addresses

DNS | Networks | Tools

These services allow you to create a domain name for any IP address. The IP address is encoded into the domain name. An overview over different services can be found here.

Online Services

  • https://nip.io/ provides IPv4 only

    • Supports both . and - separators.
    • 10.0.0.1.nip.io resolves to 10.0.0.1
    • 192-168-1-250.nip.io resolves to 192.168.1.250
    • customer1.app.10.0.0.1.nip.io resolves to 10.0.0.1
    • magic-127-0-0-1.nip.io resolves to 127.0.0.1
  • https://sslip.io/ provides IPv4 and IPv6

    • Supports both . and - separators.
    • Provides the ability to use the service with your own branding.
    • 192.168.0.1.sslip.io resolves to 192.168.0.1
    • 192-168-1-250.sslip.io resolves to 192.168.1.250
    • www.192-168-0-1.sslip.io resolves to 192.168.0.1
    • –1.sslip.io resolves to ::1
    • 2a01-4f8-c17-b8f--2.sslip.io resolves to 2a01:4f8:c17:b8f::2

Self-hosted Options

  • hipio is a Haskell service for IPv4.

Yarrp: Yelling at Random Routers Progressively

 https://www.cmand.org/yarrp/

Networks | Tools

Yarrp is a active network topology discovery tool. It's goal is to identify router interfaces and interconnections on internet scale. Conceptually this is similar to running many traceroutes and stiching them together into one view. However, traceroutes are designed to understand the connection between two hosts and do not scale easily.


zesplot: IPv6 Visualisation

 https://github.com/zesplot/zesplot

Tools

zesplot is an IPv6 visualization tool. It turns a list of IP addresses into a picture, for example as a heatmap representation. It works based on squarified treemaps, since the IPv4 way of space filling curves works poorly for such a sparse space.


ZMap Project

 https://zmap.io/

DNS | Networks | Tools

Different utilities for network scanning. Most imporantly the zmap component, which is a packet scanner for different protocols. It also contains other tools like ways to iterate over the IPv4 address space and blacklist/whitelist management.