All about Network

Content

.nl stats and data - SIDN LabsAMP-Research: Amplification ResearchAPNIC RExBGPStream (CAIDA)BGPStream (CISCO)BGPlayCAIDA BGP Hijacking ObservatoryCAIDA Datasets OverviewCensysCloudflare RadarCollection of "bad" packets in PCAPsCommon CrawlCyber Threat Intelligence FeedsDMAP Domain Mapper by SIDN LabsDNS Census 2013DNS CoffeeDNS Core CensusDNS Quality/Overview ToolsDNS Queries to Authoritative DNS Server at SURFnet by Google's Public DNS ResolverDNS Replay Tool (drool)DNS open zone dataDNSCAPDNSDBDNSMONDNSSEC Deployment MapsDNSSEC Deployment ReportsDNSTOPDriftnetDublin TracerouteDuckDuckGo Tracker RadarFD.io: The World's Secure Networking Data PlaneFlamethrowerForward DNS Rapid7Hurricane Electric Submarine Cable MapICANN Indentifier Technologies Health IndicatorsIP Abuse ListsIP Command CheatsheetIP to ASN Mapping (CIRCL LU)IP to ASN Mapping (Cymru)IPmap RIPEIPv6 Deployment ReportsIPv6 Hitlist CollectionIPv6 RIPEnessIXP Pricing OverviewInternet Health ReportInternet Maps (RIPE NCC)Internet Society PulseIs BGP safe yet?List of Amplification ProtocolsList of BGP Routing DatasetsList of DNS related RFCsList of Looking Glasses Providing TraceroutesLists of DNS BlacklistsMANRS ObservatoryMeasurement Factory: DNS SurveyMetis: Atlas probe selectionMini Internet ProjectMultilevel MDA-Lite Paris **Traceroute**Multipath TCP Measurement ServiceNETSCOUT Cyber Threat HorizonNetlab 360 OpenData ProjectNetworkScan MonOpen INTELOpen Traffic Generator (OTG)PEERING: The BGP TestbedParis TraceroutePassive DNS (CIRCL)PathVis: Visualising TraceroutePeeringDBPublic Suffix ListRDAPRIPE AtlasRIPE Atlas: Probe FiltersRIPEstat: Providing open data and insights for Internet resourcesROV Deployment MonitorRPKI BrowsersRPKI TesterRegex to parse router hostnamesRelationship between DNS RFCsRouting Information Service (RIS)Shadowserver DashboardShadowserver Scanning ProjectShodanTLD Apex HistoryTask-centered iproute2 user guideTcpdump advanced filtersTeleGeography Map GalleryWildcard DNS for IP AddressesYarrp: Yelling at Random Routers ProgressivelyZMap ProjectZiggy: the RPKI Wayback MachineZonefiles: Domain Listsdn42dnsperf and resperfdnsthoughtioda: Internet Outage Detection and AnalysisnPrintnmap StylesheetpyNTM: Network Traffic ModelerrespdiffsnidumptraIXroutewirediff

.nl stats and data - SIDN Labs

https://stats.sidnlabs.nl/en/

DNS | DNSSEC | Dataset | IP | Network

Historic datasets (from 2014 onwards) for the .nl TLD. Datasets are available in JSON format.

Datasets cover information about:

  • DNS
    • Domain Names
    • Query Type
    • Response Codes
    • IPv6 Support
  • Resolvers
    • Location
    • Number of IP addresses
    • Validating Resolvers
    • Popular Networks
    • Port Randomness
  • DNSSEC
    • Validating Queries
    • DANE
    • Used Algorithms
  • Mail
    • Mail RRs
    • SPF Information

Censys

https://censys.io/

Certificate | DNS | Dataset | IP | Network

Censys performs regular scans for common protocols (e.g., DNS, HTTP(S), SSH). Provides a search for TLS certificates.

Access is free, but requires registration. The website no longer provides free bulk access. Bulk access requires a commercial or a research license. The free access is limited to 1000 API calls per day.

@InProceedings{censys15,
    author = {Zakir Durumeric and David Adrian and Ariana Mirian and Michael Bailey and J. Alex Halderman},
    title = {A Search Engine Backed by {I}nternet-Wide Scanning},
    booktitle = {Proceedings of the 22nd {ACM} Conference on Computer and Communications Security},
    month = oct,
    year = 2015
}

Cloudflare Radar

https://radar.cloudflare.com/

BGP | DDoS | DNS | Dataset | IP | Network

Cloudflare Radar is Cloudflares reporting website about internet trends and general traffic statistics. The website shows information about observed attacks and attack types and links to the DDoS report. General traffic statistics are reported, such as the used browser, fraction of human traffic, IP, HTTP, and TLS version.

The website also provides more detailed information on domains and IP addresses. Domains have information about age, popularity, and visitors. IP addresses have ASN and geolocation information.

More information about Cloudflare Radar is available in the introduction blogpost.

The Radar data is also available via API, for example the attack data: https://developers.cloudflare.com/api/operations/radar_get_AttacksLayer3Summary

DNS Coffee

https://dns.coffee/

DNS | Dataset | IP | Network | Search

DNS Coffee collects and archives stats from DNS Zone files in order to provide insights into the growth and changes in DNS over time.

The website includes information such as the size of different zones. It tracks over 1200 zone files.

It provides searching through the zones files based on domain names, name servers, or IP addresses. It can also visualize the relationship between a domain, the parent zones and the name server in what they call a "Trust Tree".

DNS Core Census

https://observatory.research.icann.org/dns-core-census

DNS | Dataset | IP | Network

The DNS Core Census is an ICANN project to gather information about top-level-domains (TLDs). This covers ccTLDs, gTLDs, effective TLDs (like co.uk), and entries in arpa. The census contains information about the zone, like metadata and contractual information, about the name servers, about addresses of the name servers, and the route origins. The data is kept for a 35-day rolling window.

Further information about the project can be found in this presentation and OCTO-019 from ICANNs Chief Technology Officer

DNS Quality/Overview Tools

DNS | DNSSEC | Dataset | Network | Tool

Check My DNS

Browser-based DNS resolver quality measurement tool. Uses the browser to generate many resolver queries and tests for features they should have, such as EDNS support, IPv6, QNAME Minimisation, etc.

This test is also available as a CLI tool: https://github.com/DNS-OARC/cmdns-cli

DNSSEC Debugger

Analyze DNSSEC deployment for a zone and show errors in the configuration.

DNSViz

Gives an overview of DNSSEC delegations, response sizes, and name servers.

GitHub: https://github.com/dnsviz/dnsviz

DNS X-Ray

The website has an online test, which performs DNS lookups. These DNS lookups test if certain resource records are overwritten in the cache. The tool can then determine what DNS software is used, where the server is located, how many caches there are, etc.

EDNS Compliance Tester

Test name server of zones for correct EDNS support.

The Transitive Trust and DNS Dependency Graph Portal

Shows the trust dependencies in DNS. Given a domain name it can show how zones delegate to each other and why. The delegation is done between IP addresses and zones.

Root Canary Project

The project used to monitor the first root KSK key rollover. Now it contains the paper "Roll, Roll, Roll your Root: A Comprehensive Analysis of the FirstEver DNSSEC Root KSK Rollover" describing the experiences of the first root KSK rollover

Additionally, it includes a tester for DNSSEC algorithm support, which shows the algorithms supported by the currently used recursive resolver. It provides statistics about support for DNSSEC algorithms. It has a web based test to test your own resolver and provides a live monitoring using the RIPA Atlas.

DNSSEC algorithms resolver test

DNS Queries to Authoritative DNS Server at SURFnet by Google's Public DNS Resolver

https://data.4tu.nl/articles/dataset/DNS_Queries_to_Authoritative_DNS_Server_at_SURFnet_by_Google_s_Public_DNS_Resolver/12682040

DNS | Dataset | Network

This dataset covers approximately 3.5 billion DNS queries that were received at one of SURFnet's authoritative DNS servers from Google's Public DNS Resolver. The queries were collected during 2.5 years. The dataset contains only those queries that contained an EDNS Client Subnet.

The dataset covers data from 2015-06 through 2018-01.

DOI Identifier

DNSDB

https://scout.dnsdb.info/

DNS | Dataset | Network

Historical DNS database. Contains information recorded at recursive resolver about domain names, first/last seen, current bailiwick. Allows to see the lifetime of resource records and can be used as a large database.

DuckDuckGo Tracker Radar

https://github.com/duckduckgo/tracker-radar

Dataset | Network

Tracker Radar collects common third party domains and rich metadata about them. The data is collected from the DuckDuckGo crawler. More details are in this blogpost.

This is not a block list, but a data set of the most common third party domains on the web with information about their behavior, classification and ownership. It allows for easy custom solutions with the significant metadata it has for each domain: parent entity, prevalence, use of fingerprinting, cookies, privacy policy, and performance. The data on individual domains can be found in the domains directory.

FD.io: The World's Secure Networking Data Plane

https://fd.io/

Network | Tool

FD.io is a very fast userspace networking library, which allows to create programs for packet processing. While DPDK allows fast read and write access to the NICs, FD.io is foccussed on processing the packets. Possible use cases are a packet forwarder, implementing a NAT, or a VPN.

More details also in this APNIC blogpost: https://blog.apnic.net/2020/04/17/kernel-bypass-networking-with-fd-io-and-vpp/

Forward DNS Rapid7

https://opendata.rapid7.com/sonar.fdns_v2/

DNS | Dataset | IP | Network

This dataset contains the responses to DNS requests for all forward DNS names known by Rapid7's Project Sonar. Until early November 2017, all of these were for the 'ANY' record with a fallback A and AAAA request if necessary. After that, the ANY study represents only the responses to ANY requests, and dedicated studies were created for the A, AAAA, CNAME and TXT record lookups with appropriately named files.

The data is updated every month. Historic data can be downloaded after creating a free account.

ICANN Indentifier Technologies Health Indicators

https://ithi.research.icann.org/metrics.html

DNS | DNSSEC | Dataset | Network

ICANN tracks the general health of the DNS ecosystem and related ecosystems. The data is updated irregularly, but historic data is available. The collected data covers eight major topics:

  1. M1: inaccuracy of Whois Data
  2. M2: Domain Name Abuse
  3. M3: DNS Root Traffic Analysis
  4. M4: DNS Recursive Server Analysis
  5. M5: Recursive Resolver Integrity
  6. M6: IANA registries for DNS parameters
  7. M7: DNSSEC Deployment.
  8. M8: DNS Authoritative Servers Analysis

Each topic has too many sub categories to list here.

IPmap RIPE

https://ipmap.ripe.net/

BGP | Dataset | Map | Network | Tool

IP geolocation services feeding itself from geolocation databases, user provided locations, and most importantly, active RTT measurements based on the RIPE Atlas system. It also provides a nice API to query the location. It provides a breakdown on where the results stem from and how much they contribute to the overall result.

IPv6 Hitlist Collection

Dataset | IP | Network

https://www.net.in.tum.de/projects/gino/ipv6-hitlist.html

A curated list of IPv6 hosts, gathered by crawling different lists. Includes:

  • Alexa domains
  • Cisco Umbrella
  • CAIDA DNS names
  • Rapis7 DNS ANY and rDNS
  • Various zone files

Access to the full list requires registration by email.

Based on the paper "Scanning the IPv6 Internet: Towards a Comprehensive Hitlist".

https://ipv6hitlist.github.io/

The website contains the additional material of the IMC paper Clusters in the Expanse: Understanding and Unbiasing IPv6 Hitlists. The IPv6 addresses can be downloaded from the website. The website has three lists, responsive IPv6 addresses, aliased prefixes, and non-aliased prefixes. Additionally, the website also has a list of tools used during the data creation.

IPv6 RIPEness

https://ripeness.ripe.net/

Dataset | IPv6 | Network

RIPE gathers data about the IPv6 deployments worldwide and publishes the information on their IPv6 RIPEness website. The deployments are judged on four points:

  • Having an IPv6 address space allocation or assignment from the RIPE NCC
  • Visibility in the Routing Information Service (RIS)
  • Having a route6 object in the RIPE Database
  • Having a reverse DNS delegation set up

Internet Health Report

https://ihr.iijlab.net/ihr/en-us/global_report

Dataset | Network

The Internet Health Report reports on significant disruption events between networks. They use BGP and traceroutes as their data sources. The report contains information about the connectives of ASes, such as the most common upstream networks and RPKI status of announcements. Link quality information is included, like historic network delay, forwarding anomalies, or network disconnects.

Internet Society Pulse

https://pulse.internetsociety.org/

Autonomous System | BGP | Dataset | Network | Tool

The Internet Society gathers data to show the general health and availability of the internet. They measure four categories: internet shutdowns, technology use, resilience, and concentration. Under internet shutdowns, they show which countries are performing what kind of disruption, e.g., regional or national. The technology sections lists basic statistics about HTTPS, IPv6, TLS, DNSSEC.

Is BGP safe yet?

https://isbgpsafeyet.com/

BGP | Dataset | Network | RPKI

"Is BGP safe yet?" is an effort by Cloudflare to track the deployment of RPKI filtering across different ISPs. They provide a tester on the website with which each user can test if the current ISP is filtering RPKI invalid announcements. The website includes a list of networks and if and how they use RPKI (signing and/or filtering).

More details for this project can be found in Cloudflare's blog or on the GitHub project.

List of BGP Routing Datasets

BGP | Dataset | Network

Packet Clearing House (PCH)

The Packet Clearing House (PCH) publishes BGP data collected at more than 100 internet exchange points (IXP). The snapshot dataset contains the state of the routing tables in daily intervals.

PCH also provides raw routing data in MRT format. These contain all the update information in sorted by time.

Routing Information Service (RIS)

The RIS is the main resource from RIPE featuring all kinds of datasets about AS assignments and connectivity.

Routeviews

Routeviews is a project by the University of Oregon to provide live and historical BGP routing data.

Lists of DNS Blacklists

DNS | Dataset | IP | Network | Spam | Tool

These projects either operate DNS based Real-time Blackhole Lists (RBL) or allow checking if an IP is contained. The Multi-RBL websites are helpful in finding a large quantity of RBLs.

MANRS Observatory

https://observatory.manrs.org/

BGP | Dataset | Network

Mutually Agreed Norms for Routing Security (MANRS) is an initiative to improve the state of routing security. The observatory shows what kind of incidents occurred and how prepared networks are, e.g., with filtering and coordination efforts. The data is available globally and comparisons between regions are available. Historic data is accessible on the website.

Metis: Atlas probe selection

https://ihr.iijlab.net/ihr/en-us/metis/selection

Network | Tool

The website provides a tool to select a list of autonomous systems with a fairer probe distribution. Probes are not distributed equally, but rather cluster based on population. This leads to large biases towards western locations and certain autonomous systems. The website offers different distance metrics. The output is a list of autonomous system numbers for use in the RIPE Atlas API.

Mini Internet Project

https://github.com/nsg-ethz/mini_internet_project

BGP | IP | Network | Tool

The mini internet project is part of the curriculum by the Networked Systems Group of ETH Zurich. It teaches the students the basic steps how to create a mini internet. It starts with the basics of intra-network routing, by setting up multiple L2 switches. Then the students have to configure L3 routers to connect multiple L2 sites together. Lastly, in a big hackathon style, the students need to connect their local network with the network of the other students, by properly configuring BGP routers and setting up routing policies.

The code and the tasks are all available in the GitHub repository.

The APNIC Blog has a nice introduction to the project too.

Open INTEL

https://www.openintel.nl/

DNS | Dataset | IP | Network

Open INTEL is an active DNS database. It gathers information from public zone files, domain lists (Alexa, Umbrella), and reverse DNS entries. Once every 24 hours data is collected about a bunch of DNS RRsets (SOA, NS, A, AAAA, MX, TXT, DNSKEY, DS, NSEC3, CAA, CDS, CDNSKEY). The data is openly avaible as AVRO files and dates back until 2016.

The data can be freely downloaded. There is documentation on the layout of the AVRO files.

The project is similar to Active DNS but seems to be larger in scope.

PEERING: The BGP Testbed

https://peering.ee.columbia.edu/

BGP | Network | Tool

PEERING is an environment where researchers and educators can play with BGP announcements in a real but sandboxed environment.

Description from the website:

The long-term goal of the PEERING system is to enable on-demand, safe, and controlled access to the Internet routing ecosystem for researchers and educators:

  • PEERING for researchers. Today, it is hard for researchers to conduct Internet routing experiments. To perform a routing experiment, a research institution has to obtain Internet resources (IP addresses and ASNs) and establish relations with upstream networks. PEERING eliminates these obstacles and provides researchers controlled on-demand access to the routing ecosystem.
  • PEERING for educators. Educators can use the PEERING infrastructure in teaching students the Internet routing architecture. The students access to live BGP sessions to multiple ISPs.

RIPE Atlas: Probe Filters

https://github.com/nicholaskernan/probe-filters

Network | Tool

The repository contains code for a better probe selection for the RIPE Atlas measurement system. Probes are not distributed equally, but rather cluster based on population. This leads to large biases towards western locations and certain autonomous systems. The goal of the repository is to find a more equal, thus fairer probe selection.

RIPEstat: Providing open data and insights for Internet resources

https://stat.ripe.net/

Autonomous System | BGP | DNS | Dataset | Network | Tool

RIPEstat is a network statistics platform by RIPE. The platform shows data for IP addresses, networks, ASNs, and DNS names. This includes information such as the registration information, abuse contacts, blocklist status, BGP information, geolocation lookups, or reverse DNS names. Additionally, the website links to many other useful tools, such as an address space hierarchy viewer, historical whois information, and routing consistency checks.

Shadowserver Dashboard

https://dashboard.shadowserver.org/

DNS | Dataset | Malware | Network

The Shadowserver Scanning projects performs regular Internet wide scans for many protocols. The dashboard shows the gathered data about botnet sinkholes, Internet scans, honeypots, DDoS, and IoT data. This includes information about the size of botnets, the number of IP addresses with open ports like MySQL, the botnets as seen by honeypots, or the used protocols for DDoS attacks.

The blog post provides an introduction to the new dashboard.

Shadowserver Scanning Project

https://scan.shadowserver.org/

DNS | Dataset | Malware | Network

The Shadowserver Scanning projects performs regular Internet wide scans for many protocols. They scan for four main types of protocols:

  1. Amplification protocols, e.g., DNS or NTP
  2. Botnet protocols, e.g., Gameover Zeus or Sality
  3. Protocols that should not be exposed, e.g., Elastic Search, LDAP, or RDP
  4. Vulnerable Protocols, e.g., SSLv3

The website is a great resource to get general statistics about the protocols, like the number of hosts speaking the protocol, their geographic distribution, associated ASNs, and the historic information.

Wildcard DNS for IP Addresses

DNS | IP | Network | Tool

These services allow you to create a domain name for any IP address. The IP address is encoded into the domain name. An overview of different services can be found here.

Online Services

  • https://nip.io/ provides IPv4 only

    • Supports both . and - separators.
    • 10.0.0.1.nip.io resolves to 10.0.0.1
    • 192-168-1-250.nip.io resolves to 192.168.1.250
    • customer1.app.10.0.0.1.nip.io resolves to 10.0.0.1
    • magic-127-0-0-1.nip.io resolves to 127.0.0.1

  • https://sslip.io/ provides IPv4 and IPv6

    • Supports both . and - separators.
    • Provides the ability to use the service with your own branding.
    • 192.168.0.1.sslip.io resolves to 192.168.0.1
    • 192-168-1-250.sslip.io resolves to 192.168.1.250
    • www.192-168-0-1.sslip.io resolves to 192.168.0.1
    • –1.sslip.io resolves to ::1
    • 2a01-4f8-c17-b8f--2.sslip.io resolves to 2a01:4f8:c17:b8f::2

Self-hosted Options

  • hipio is a Haskell service for IPv4.

ZMap Project

https://zmap.io/

DNS | IP | Network | Tool

Different utilities for network scanning. Most imporantly the zmap component, which is a packet scanner for different protocols. It also contains other tools like ways to iterate over the IPv4 address space and blacklist/whitelist management.

dn42

https://dn42.eu/Home

BGP | IP | Network | VPN

dn42 is a big dynamic VPN. It employs various Internet technologies, such as BGP, whois, DNS, etc.

Users can experiment with technology, they normally would not use in a separated environment.

Mostly different hackerspaces participate in the dn42 network, such as different locations of the CCC.

nPrint

https://nprint.github.io/

Network | Tool

The nPrint project is a collection of open-source software and benchmarks for network traffic analysis that aim to replace the built-to-task approach currently taken when examining traffic analysis tasks.