All about Datasets

Content

.nl stats and data - SIDN Labs 0day "In the Wild" Active DNS Akamai DNS Trends and Traffic Alexa Top 1 Million Security Analysis AMP-Research: Amplification Research APNIC Labs Stats AVR Instruction Set BGP Looking Glass BGPlay BGPmon Archive BGPStream (CAIDA) BGPStream (OpenDNS) Binary Hardening in IoT Products Bitcoin Monitoring CAIDA BGP Hijacking Observatory Caida Datasets Overview caniuse.rs Censored Planet Censys Certificate Search crt.sh CIRCL hashlookup Citizenlab Censorship Test Lists Cloudflare Radar Collection of "bad" packets in PCAPs Common Crawl Computer Security Conference Ranking and Statistic Corona Dashboards for Germany and Europa Corona Dashboards for Saarland Corona Vacine Dashboards CTF Challenge Search CZ.NIC Statistics DDoS Mon DEF CON CTF Archive Der Deutschlandatlas: Deutschland neu vermessen DMAP Domain Mapper by SIDN Labs DNS Authoritative Server Benchmarks DNS Census 2013 DNS Coffee DNS Privacy Project DNS Quality/Overview Tools DNS Queries to Authoritative DNS Server at SURFnet by Google's Public DNS Resolver DNSDB dnsdumpster DNSMON DNSSEC Deployment Reports DNSSEC Early Warning System dnsstream (Twitter) dnsthought Domain Crawling Lists DuckDuckGo Tracker Radar Forward DNS Rapid7 GitHub Advisory Database gitignore Templates Global Security Database (GSD) Google Transparency Report HTTP Status Codes Hurricane Electric Submarine Cable Map ICANN Managed Root Servers Statistics ICLab Data IETF Officiel RFC Bibtex Downloads Intel Management Engine Partitions Internet Maps (RIPE NCC) ioda: Internet Outage Detection and Analysis IP Abuse Lists IP to ASN Mapping (CIRCL LU) IP to ASN Mapping (Cymru) IPmap RIPE IPv6 Deployment Reports IPv6 Hitlist Collection Is BGP safe yet? IXP Pricing Overview Known Exploited Vulnerabilities Catalog libc Database Search Linux System Call Table List of Amplification Protocols List of BGP Routing Datasets List of Chrome CLI Switches List of Default Passwords List of Looking Glasses Providing Traceroutes List of Network Speed Tests Lists of DNS Blacklists Malware Bazaar Manchester Academic Phrasebank Netlab 360 OpenData Project netray.io Internet Observatory NetworkScan Mon NIST RPKI Monitor Online Hash Crackers Open INTEL Open Observatory of Network Interference (OONI) Open Source Vulnerabilities (OSV) Over The Wire: Wargames Passive DNS (CIRCL) Passive SSL (CIRCL) PeeringDB Privilege Escalation Cheatsheet (Vulnhub) Public DNS Server List Public Suffix List Regex to parse router hostnames RFC 8145 Root Trust Anchor Reports RIPE Atlas RIPEstat: Providing open data and insights for Internet resources Root Servers Routing Information Service (RIS) ROV Deployment Monitor RPKI Browsers RSSAC002 DNS Root Server Data scans.io Internet-Wide Scan Data Repository Shadowserver Scanning Project Shodan System Security Cirrcus TeleGeography Map Gallery Transient Execution Attacks vizAS Vulnerable (Docker) Containers WAND Active Measurement Project World Country Information x86 Instruction Set Zonefiles: Domain Lists

.nl stats and data - SIDN Labs

 https://stats.sidnlabs.nl/en/

DNS | DNSSEC | IP | Networks

Historic datasets (from 2014 onwards) for the .nl TLD. Datasets are available in JSON format.

Datasets cover information about:

  • DNS
    • Domain Names
    • Query Type
    • Resonse Codes
    • IPv6 Support
  • Resolvers
    • Location
    • Number of IP addresses
    • Validating Resolvers
    • Popular Networks
    • Port Randomness
  • DNSSEC
    • Validating Queries
    • DANE
    • Used Algorithms
  • Mail
    • Mail RRs
    • SPF Information

0day "In the Wild"

 https://docs.google.com/spreadsheets/d/1lkNJ0uQwbeC1ZTRrxdtuPLCIl7mlUreoKfSIgajnSyY/htmlview

0day

Google Project Zero tracks a list of zero day exploits discovered in the wild. They track public resources to find uses of zero days and collect them in this spreadsheet. The spreadsheet contains data since 2014. Their blog provides an introduction and explanation of the spreadsheet.


Active DNS

 https://www.activednsproject.org/

DNS | IP | Networks

Historical DNS database. Access can be requested for academic use.

Activly queries many DNS records, e.g., .com zone. It can contain information not in DNSDB, if the information was never seen by a resolver. It does not contain all informatin, as some domains may be unknown to the project and thus cannot be crawled. It uses popular zones, domain lists (e.g., Alexa, blacklists) and other domain feeds.

They normally maintain a rolling 14-day window.

Copy files (for date 2017-10-05) (ddos@gladbeck):

sftp -B1024000 -C -rp "activedns@kokino.gtisc.gatech.edu:active-dns/20171005/" .

The data is encocded in AVRO format, which can also be parsed as JSONL. Python has a AVRO library. AVRO schema:

{
    "namespace": "astrolavos.avro",
    "type": "record",
    "name": "ActiveDns",
    "fields": [
        {"name": "date", "type": "string"},
        {"name": "qname", "type": "string"},
        {"name": "qtype", "type": "int"},
        {"name": "rdata", "type": ["string", "null"]},
        {"name": "ttl", "type": ["int", "null"]},
        {"name": "authority_ips", "type": "string"},
        {"name": "count", "type": "long"},
        {"name": "hours", "type": "int"},
        {"name": "source", "type": "string"},
        {"name": "sensor", "type": "string"}
    ]
}

Some more information about some fields that are unique to that schema. The IPs in Authority IP are the collection of the authority name server IPs that replied to our query. We gather all the IPs that gave us the same answer for an entire day and concatenate them on the same field, mostly in order to reduce the number of records that we have to keep. The only field that might be slightly confusing, is the "hours" field. This is a 24bit integer that encodes the time of day we saw this RR for date date (for example, 000000000000000001000010 = 18:00 and 23:00). Another important thing to keep in mind, is NXDOMAINs. A resolved QNAME does not exist when both the rdata and ttl fields are equal to null. If rdata exists but ttl is null then the record was part of the glue of the DNS packet and not in the answer section.

A similar active DNS project is Open INTEL which seems to be larger in scope and the data is publicly available.


 https://www.akamai.com/us/en/why-akamai/dns-trends-and-traffic.jsp

DNS | IP

Akamai provides aggregated data about the root servers. They provide the overall amount of DNS queries, trends, and IPv4/IPv6 query ratios.


Alexa Top 1 Million Security Analysis

 https://crawler.ninja/

Daily scans of the Alexa 1 million list. The website contains some high level statistics. Every six month a more detailed crawl report is collected. Full raw data used to be available via scans.io.


AMP-Research: Amplification Research

 https://github.com/Phenomite/AMP-Research

Amplification | Denial-of-Service | Networks | Tools

The AMP-Research project collects information about amplification vectors in protocols including reproduction possibilities. For each vector the port and protocol are listed, as well as, the amplification factor. A scanning script or payload for scanning with zmap is included too.


APNIC Labs Stats

 https://stats.labs.apnic.net/

Autonomous Systems | BGP | DNS | DNSSEC | IP

APNIC gathers many statistics and offers them on their website. However, they provide way more data than it might initially look like, since many of the datasets are not linked from their main page.


AVR Instruction Set

 https://www.microchip.com/webdoc/avrassembler/avrassembler.wb_instruction_list.html

Cheatsheet | CTF

This websites provide reference documentation of the AVR instruction set, which is used for Arduino boards.


BGP Looking Glass

 http://www.bgplookingglass.com/

Autonomous Systems | BGP

The website contains a list of hundrets of looking glasses for various autonomous systems. All looking glasses are publicly accessible.


BGPlay

 https://stat.ripe.net/special/bgplay

BGP | Networks | Tools

BGPlay shows a graph of the observed BGP routes. It allows replaying historical BGP announcements and displays route changes.

Documentation
GitHub


BGPmon Archive

 http://bgpmon.io/archive/help

BGP | Networks

Downloadable dataset of historic BGP information from different vantage points.


BGPStream (CAIDA)

 https://bgpstream.caida.org/

Autonomous Systems | BGP | Networks | Tools

An open-source software framework for live and historical BGP data analysis, supporting scientific research, operational monitoring, and post-event analysis.

BGP streams are freely accesible and provided by Route View, RIPE, and BGPmon.


BGPStream (OpenDNS)

 https://bgpstream.com/

Autonomous Systems | BGP | Networks

BGP Stream is a free resource for receiving alerts about hijacks, leaks, and outages in the Border Gateway Protocol.

BGP Steam provides real-time information about BGP events. It includes information about affected IPs, ASNs, and even a replay feature how the BGP announcements changed.

A live alert bot also exists on Twitter.


Binary Hardening in IoT Products

 https://cyber-itl.org/2019/08/26/iot-data-writeup.html

Detailed analysis on a 10 year dataset of IoT binaries and their security features. The Cyber ITL focussed on which compiler and toolchain hardenings the vendors use.

CITL identified a number of important takeaways from this study:

  • On average, updates were more likely to remove hardening features than add them.
  • Within our 15 year data set, there have been no positive trends from any one vendor.
  • MIPS is both the most common CPU architecture and least hardened on average.
  • There are a large number of duplicate binaries across multiple vendors, indicating a common build system or toolchain.

Bitcoin Monitoring

 https://www.dsn.kastel.kit.edu/bitcoin/index.html

The Bitcoin monitoring project by the Karlsruher Institute for Technology measures the activity in the Bitcoin network. They report about the origin of peers (ASN, country), the software version used, and report about block and transaction propagation delay.


CAIDA BGP Hijacking Observatory

 https://dev.hicube.caida.org/feeds/hijacks/events

BGP | Networks

The BGP hijacking observatory lists potential BGP hijacks. It can observe different kinds of hijacks, e.g., shorter path or more specific prefix. It lists the hijacking time, potential victims and attackers, and the affected prefix.

More details about the different hijacking methods are in the AIMS-KISMET presentation.


Caida Datasets Overview

 https://www.caida.org/catalog/datasets/overview/

Autonomous Systems | BGP | IP | Networks

Overview of datasets, monitors, and reports produced and organized by Caida. Also contains links to other datasets.


caniuse.rs

 https://caniuse.rs/

Rust | Tools

caniuse.rs shows which library functions where stabilized in which Rust version.


Censored Planet

 https://data.censoredplanet.org/raw

Censored Planet is a censorship measurement platform that collects data using multiple remote measurement techniques in more than 200 countries.

The website provides access to many recent scans. The scans are performed using different techniques to find different censors.


Censys

 https://censys.io/

Certificates | DNS | IP | Networks

Censys performs regular scans for common protocols (e.g., DNS, HTTP(S), SSH). Provides a search for TLS certificates.

Access is free, but requires registration. The website no longer provides free bulk access. Bulk access requires a commercial or a research license. The free access is limited to 1000 API calls per day.

@InProceedings{censys15,
    author = {Zakir Durumeric and David Adrian and Ariana Mirian and Michael Bailey and J. Alex Halderman},
    title = {A Search Engine Backed by {I}nternet-Wide Scanning},
    booktitle = {Proceedings of the 22nd {ACM} Conference on Computer and Communications Security},
    month = oct,
    year = 2015
}

Certificate Search crt.sh

 https://crt.sh/

Certificates

Certificate search engine. crt.sh is based on the certificate transparency logs and provide wildcard search for domains.


CIRCL hashlookup

 https://hashlookup.circl.lu/

Lookup files by their md5 or sha1 hashes. The response contains information such as the filename, size or where the file was found, like a Linux package. On the website you have the API documentation which can be used directly from the browser.


Citizenlab Censorship Test Lists

 https://github.com/citizenlab/test-lists

The GitHub repository contains multiple lists for finding website censorship. The lists are organized by country and contain URLs specific for each of them. The URLs are also categoried and cover four broad themes:

  • Political, e.g., governmental views or human rights
  • Social, e.g., sexuality or gambling
  • Conflicts, e.g., armed conflicts or border displutes
  • Internet tools, e.g., hosting providers or circumvention methods.

Cloudflare Radar

 https://radar.cloudflare.com/

IP | Networks

Cloudflare Radar is Cloudflares reporting website about internet trends and general traffic statistics. The website shows information about observed attacks and attack types and links to the DDoS report. General traffic statistics are reported, such as the used browser, fraction of human traffic, IP, HTTP, and TLS version.

The website also provides more detailed information for domains and IP addresses. Domains have information about age, popularity, and visitors. IP addresses have ASN and geolocation information.

More information about Cloudflare Radar are available in the introduction blogpost.


Collection of "bad" packets in PCAPs

 https://github.com/DNS-OARC/bad-packets

DNS | IP | Networks | PCAPs

Collection of "bad" packets in PCAPs that can be used for testing software.


Common Crawl

 https://commoncrawl.org/

Networks

The Common Crawl project builds an openly accessible database of crawled websites. They index can be searched.


Computer Security Conference Ranking and Statistic

 https://people.engr.tamu.edu/guofei/sec_conf_stat.htm

Paper Writing | Security

This website offers a ranking of many computer security conferences. The ranking is accompanied by a yearly acceptance ratio statistic.


Corona Dashboards for Germany and Europa

Robert Koch-Institut Official German dashboard.

Robert Koch-Institut Lagebericht Daily situational report about the state in Germany. Contains additional information about the situation in Germany and additional statistics.

COVID Trends Germany Daily updated dashboard with many graphs for Germany.

Berliner Morgenpost Shows sub-country numbers for Europe and worldwide.

WHO European Region Country level information for Europe.

WHO European Region Subnational Explorer Subnation information for Europe with incidence rates over the last 7/14 days.

Johns Hopkins University Contains worldwide information.

ECDC COVID-19 Country Overviews Very detailed breakdown for countries worldwide.

ECDC Europe Weekly updated incidence and test positivity rates within Europe.

ECDC Worldwide Daily updated worldwide numbers with by-region breakdowns.

Reuters Provides per country and regionally aggregated information.


Corona Dashboards for Saarland

Corona Situation in Saarland

Corona Situation in Saarbrücken


Corona Vacine Dashboards

Vacine Status Germany Information about the amount of people vaccinated and information about what vacines.

Zeit Vacine Status Germany More detailed vacine status for Germany and some international information.

Bloomberg Covid Vacine Vacination status with US focus and worldwide information.


CTF Challenge Search

 https://ctf.courgettes.club/

CTF

Search for CTF challenges by language or topic of the challenge.


CZ.NIC Statistics

 https://stats.adam.nic.cz/dashboard/en/index.html

DNS

The website contains information about the cz. TLD operated by CZ.NIC. It contains information about the query volume, query type, round-trip time (RTT) and geographic location of the traffic sources. It also has information about the registry functions, such as registrar information, domain transfers or whois requests. Lastly, information about the mojeID accounts, a login provider operated by CZ.NIC are also available.


DDoS Mon

 https://ddosmon.net

Amplification | Denial-of-Service | Networks

Provides a search interface to search for domain names and IP addresses under attacks. Shows results for the last 30 days. Provides an API, which requires special authorization.


DEF CON CTF Archive

 https://archive.ooo/

CTF

The website provides a playable archive of old DEFCON challenges. The chanllenges are tagged by category and by original CTF.

The files are available on GitHub.


Der Deutschlandatlas: Deutschland neu vermessen

 https://www.deutschlandatlas.bund.de/DE/Service/Kartensuche/kartensuche_node.html

Maps

These 50 maps show the difference between regions in Germany. They show which regions are set up well and which are lacking behind. The maps show population, work, health care, infrastructure, and more.


DMAP Domain Mapper by SIDN Labs

 https://dmap.sidnlabs.nl/

DNS | Networks | Tools

DMAP is a scalable web scanning suit which supports DNS, HTTPS, TLS, and SMTP. It works based on domain names and crawls the domain for all supported protocols. The advantage over other tools is the unified SQL data model with 166 features and the easy scalability over many crawling machines.


DNS Authoritative Server Benchmarks

 https://www.knot-dns.cz/benchmark/

DNS | DNSSEC

The website is an ongoing project by Knot DNS to measure the performance of various DNS servers. Four open source servers are tested, namely BIND, Knot DNS, NSD, and PowerDNS. The benchmark includes different zone configurations matching to root zones, TLD zones, or hosting zones as well as different DNSSEC configurations.


DNS Census 2013

 https://dnscensus2013.neocities.org/index.html

DNS | IP | Networks

The DNS Census 2013 consist of about 2.5 billion DNS records collected in 2012/2013. The data is gathered from some available zone files and passive or active DNS collecting. The DNS records are written into CSV files containing one DNS record per line.


DNS Coffee

 https://dns.coffee/

DNS | IP | Networks

DNS Coffee collects and archives stats from DNS Zone files in order to provide insights into the growth and changes in DNS over time.

The website includes information such as the size of different zones. It track over 1200 zone files.

It provides searching through the zones files based on domain names, name servers, or IP addresses. It can also visualize the relationship between a domain, the parent zones and the name server in what they call a "Trust Tree".


DNS Privacy Project

 https://dnsprivacy.org/wiki/

DNS | DNSSEC | TLS

The DNS Privacy Project aims to improve privacy for users on the Internet.

The project is split into different groups working on DNS privacy:

The project focusses mostly on DNS over TLS. They provide overviews for the implementation status, configuration for test servers, and ongoing server monitoring which features they provide.


DNS Quality/Overview Tools

DNS | DNSSEC | Networks | Tools

Check My DNS

Browser-based DNS resolver quality measurement tool. Uses the browser to generate many resolver queries and tests for features they should have, such as EDNS support, IPv6, QNAME Minimisation, etc.

This test is also available as a CLI tool: https://github.com/DNS-OARC/cmdns-cli

DNSSEC Debugger

Analyze DNSSEC deployment for a zone and show errors in the configuration.

DNSViz

Gives an overview over DNSSEC delegations, response sizes, and name servers.

GitHub: https://github.com/dnsviz/dnsviz

DNS X-Ray

The website has an online test, which performs DNS lookups. These DNS lookups test if certain resource records are overwritten in the cache. The tool can then determine what DNS software is used, where the server is located, how many caches there are, etc.

EDNS Compliance Tester

Test name server of zones for correct EDNS support.

The Transitive Trust and DNS Dependency Graph Portal

Shows the trust dependencies in DNS. Given a domain name it can show how zones delegate to each other and why. The delegation is done between IP addresses and zones.

Root Canary Project

The project used to monitor the first root KSK key rollover. Now it contains the paper "Roll, Roll, Roll your Root: A Comprehensive Analysis of the FirstEver DNSSEC Root KSK Rollover" describing the experiences of the first root KSK rollover

Additionally, it includes a tester for DNSSEC algorithm support, which shows the algorithms supported by the currently used recursive resolver. It provides statistics about support for DNSSEC algorithms. It has a web based test to test your own resolver and provides a live monitoring using the RIPA Atlas.


DNS Queries to Authoritative DNS Server at SURFnet by Google's Public DNS Resolver

 https://data.4tu.nl/articles/dataset/DNS_Queries_to_Authoritative_DNS_Server_at_SURFnet_by_Google_s_Public_DNS_Resolver/12682040

DNS | Networks

This dataset covers approximately 3.5 billion DNS queries that were received at one of SURFnet's authoritative DNS servers from Google's Public DNS Resolver. The queries were collected during 2.5 years. The dataset contains only those queries that contained an EDNS Client Subnet.

The dataset covers data from 2015-06 through 2018-01.

DOI Identifier


DNSDB

 https://scout.dnsdb.info/

DNS | Networks

Historical DNS database. Contains information recorded at recursive resolver about domain names, first/last seen, current bailiwick. Allows to see the lifetime of resource records and can be used as a large database.


dnsdumpster

 https://dnsdumpster.com/

DNS | Tools

dnsdumpster.com fetches a lot of DNS information belonging to one domain. It checks the authorative name servers, which records exist, and where the servers are located.


DNSMON

 https://atlas.ripe.net/dnsmon/

DNS | Networks

Historical information about the reachability of root and some TLD name servers.


DNSSEC Deployment Reports

 https://rick.eng.br/dnssecstat/

DNS | DNSSEC | Networks

Regularly updated reports about current DNSSEC deployment. Contains information per TLD and global distribution.


DNSSEC Early Warning System

 https://www.dnssek.info/

DNS | DNSSEC

The website keeps track of all DNSSEC keys in the top level domains (TLDs) and informs when the signatures are about to expire. The time before some RRSIGs expire is color coded. It also shows error which happened during validation.


dnsstream (Twitter)

 https://twitter.com/dnsstream

DNS | Networks

@dnsstream is a Twitter bot, which sends out notifications for important DNS changes of domains.

  • Potential DDoS attacks
  • Domains which link to know malicious IPs
  • Name server changes for a domain

dnsthought

 https://dnsthought.nlnetlabs.nl/

DNS | DNSSEC | Networks

Dnsthought list many statistics about the resolvers visible to the .nl-authoritative name servers. The data is gathered from the RIPE Atlas probes. There is a dashboard which only works partially.

Raw data access is also available.


Domain Crawling Lists

DNS

Domain popularity lists provide a starting point for crawling domains with the most users. The most commonly used list for security research is the Alexa list.

  • Alexa
    The list is updated daily and contains one million websites. The ranking is based on page views, but very volatile.
  • CISCO Umbrella
    The list is updated daily and contains one million websites. The ranking is based on traffic seen on the OpenDNS resolvers.
  • Majestic
    The list is updated daily and contains one million websites. The ranking is based on backlinks from other websites.
  • Tranco
    A Research-Oriented Top Sites Ranking Hardened Against Manipulation
    The Tranco list aims to provide a better list for security research. The authors explain on their website and their paper what the flaws in the existing lists
  • Quantcast
    The list is updated daily and contains around 500,000 websites. It is based on users visiting the site within the previous month and highly US focussed.

DuckDuckGo Tracker Radar

 https://github.com/duckduckgo/tracker-radar

Networks

Tracker Radar collects common third party domains and rich metadata about them. The data is collected from the DuckDuckGo crawler. More details are in this blogpost.

This is not a block list, but a data set of the most common third party domains on the web with information about their behavior, classification and ownership. It allows for easy custom solutions with the significant metadata it has for each domain: parent entity, prevalence, use of fingerprinting, cookies, privacy policy, and performance. The data on individual domains can be found in the domains directory.


Forward DNS Rapid7

 https://opendata.rapid7.com/sonar.fdns_v2/

DNS | IP | Networks

This dataset contains the responses to DNS requests for all forward DNS names known by Rapid7's Project Sonar. Until early November 2017, all of these were for the 'ANY' record with a fallback A and AAAA request if neccessary. After that, the ANY study represents only the responses to ANY requests, and dedicated studies were created for the A, AAAA, CNAME and TXT record lookups with appropriately named files.

The data is updated every month. Historic data can be downloaded after creating a free account.


GitHub Advisory Database

 https://github.com/github/advisory-database

Security

GitHubs Advisory Database tracks CVEs and all GitHub advisories. Contributions to GitHub advisories are possible.


gitignore Templates

Git

These websites provide templates for good .gitignore files:


Global Security Database (GSD)

 https://github.com/cloudsecurityalliance/gsd-database/

Security

Global Security Database (GSD) is a project run by the Cloud Security Alliance to collect and track vulnerabilities. Conceptually it is similar to CVEs but with an open collaborative contribution process. Data from other vulnerability databases is imported and managed together.


Google Transparency Report

 https://transparencyreport.google.com/

Certificates

Google's Transparency Report contains various information. It provides information about email encryption, HTTPS encryption, information about potentially harmful applications in Android, and live reports of traffic disruptions, such as censorship.

It provides a certificate search based on the certificate transparency logs, similar to crt.sh: https://transparencyreport.google.com/https/certificates


HTTP Status Codes

 https://httpstatuses.io/

The website contains a list of HTTP status codes. Each code has an explanation and a reference to official documentation.

For example, https://httpstatuses.io/404.


Hurricane Electric Submarine Cable Map

 https://he.net/3d-map/

Maps | Networks

3D map showing submarine cables and the backbone network of Hurricane Electric.


ICANN Managed Root Servers Statistics

 https://stats.dns.icann.org/

DNS

The Grafana dashboard shows live statistics about query volume, query type, and geographic locations. The data is collected for ICANN Managed Root Servers (IMRS) which are the L-root servers.


ICLab Data

 https://iclab.gitlab.io/post/iclab_data/

The ICLab data provides longitudinal access for cesorship data. The data is collected from 2016 onwards. The analysis encompases differend censorship techniques such as DNS manipulation or packet injection.


IETF Officiel RFC Bibtex Downloads

Paper Writing | TeX

The IETF now provides official bibtexs to download. They work for RFCs, BCPs, and drafts.

The bibtex for BCPs work, but only, if the BCP consist of a single RFC. If the BCP consists of multiple RFCs, the bibtex will only show the first one.

For drafts, the draft version number, the last two digits, have to be removed from the URL.

Examples:

Available entries can be found in the RFC Index and the BCP Index.


Intel Management Engine Partitions

 https://docs.google.com/document/d/18ETZrjubqlDZumv3cZfT36WPw9sEAoHEOnhkmWLBvgY

The document lists and describes a large part of the Intel Management Engine Partitions. This is useful as a general resource to learn about the features of Intel ME.


Internet Maps (RIPE NCC)

 https://atlas.ripe.net/results/maps/

DNS | Maps | Networks

Maps of measurements done with the RIPE Atlas.


ioda: Internet Outage Detection and Analysis

 https://ioda.caida.org/

Networks

IODA is a project by CAIDA to use different data sources to detect macroscopic internet outages in realtime. It measures the internet activity using BGP, darknets, and active probing. The website provides a realtime feed and a historical view of outages.


IP Abuse Lists

IP | Networks

These websites have lists of abusive IP addresses. They can be checked with a web form or some websites also provide a feed.


IP to ASN Mapping (CIRCL LU)

 https://www.circl.lu/services/ip-asn-history/

Autonomous Systems | IP | Networks

Historical dataset about IP to ASN mappings.


IP to ASN Mapping (Cymru)

 https://team-cymru.com/community-services/ip-asn-mapping/

Autonomous Systems | IP | Networks

Historical dataset about IP to ASN mappings.


IPmap RIPE

 https://ipmap.ripe.net/

BGP | Maps | Networks | Tools

IP geolocation services feeding itself from geolocation databases, user provided locations, and most importantly active RTT measurements based on the RIPE Atlas system. It also provides a nice API to query the location. It provides a breakdown on where the results stem from and how much they contribute to the overall result.


IPv6 Deployment Reports

IP | Networks

RIPE Report

Per continent, region, or country measurements of IPv6 deployment and preference. Allows to access historical data.

APNIC Report

Per continent, region, or country measurements of IPv6 deployment and preference.


IPv6 Hitlist Collection

IP | Networks

https://www.net.in.tum.de/projects/gino/ipv6-hitlist.html

A curated list of IPv6 hosts, gathered by crawling different lists. Includes:

  • Alexa domains
  • Cisco Umbrella
  • CAIDA DNS names
  • Rapis7 DNS ANY and rDNS
  • Various zone files

Access to the full list requires registration by email.

Based on the paper "Scanning the IPv6 Internet: Towards a Comprehensive Hitlist".

https://ipv6hitlist.github.io/

The website contains the additional material of the IMC paper Clusters in the Expanse: Understanding and Unbiasing IPv6 Hitlists. The IPv6 addresses can be downloaded from the website. The website has three lists, responsive IPv6 addresses, aliased prefixes, and non-aliased prefixes. Additionally, the website also has a list of tools used during the data creation.


Is BGP safe yet?

 https://isbgpsafeyet.com/

BGP | Networks | RPKI

"Is BGP safe yet?" is an effort by Cloudflare to track the deployment of RPKI filtering accross different ISPs. They provide a tester on the website with which each user can test if the current ISP is filtering RPKI invalid announcements. The website includes a list of networks and if and how they use RPKI (signing and/or filtering).

More details for this project can be found in Cloudflare's blog or on the GitHub project.


IXP Pricing Overview

 https://peering.exposed/

BGP | Networks

Contains a list of pricing information of different IXP.


Known Exploited Vulnerabilities Catalog

 https://www.cisa.gov/known-exploited-vulnerabilities-catalog

Security

The Known Exploited Vulnerabilities Catalog is a project by the Cybersecurity & Infrastructure Security Agency (CISA) tracking actively exploitet vulnerabilities. The data includes the vulnerability, the affected product, short description of the vulnerabililty, and remediation actions. The data is available in machine readable format.


libc Database Search

CTF | Reverse Engineering

Online interface to find a libc database by function offsets. They are powered by the libc-database repository.


Linux System Call Table

Cheatsheet | CTF | x86

These websites provided an overview over the Linux systemcall interface by listing the syscall numbers, their meanings, and their arguments.


List of Amplification Protocols

 https://www.us-cert.gov/ncas/alerts/TA14-017A

Amplification | Denial-of-Service | Networks

Contains a list of UDP-based protocols, which can be used for amplification attacks.


List of BGP Routing Datasets

BGP | Networks

Isolario

Isolario also provides historial routing data in MTR format for their route collectors. The data contains snapshots every two hours and updates with a granularity of five minutes.

Packet Clearing House (PCH)

The Packet Clearing House (PCH) publishes BGP data collected at more than 100 internet exchange points (IXP). The snapshot dataset contains the state of the routing tables in daily intervals.

PCH also provides raw routing data in MRT format. These contain all the update information in sorted by time.

Routing Information Service (RIS)

The RIS is the main resource from RIPE featuring all kinds of datasets about AS assignments and connectivity.

Routeviews

Routeviews is a project by the University of Oregon to provide live and historical BGP routing data.


List of Chrome CLI Switches

 https://peter.sh/experiments/chromium-command-line-switches/#single-process

Cheatsheet

Most command line switches of Google Chrome are totally undocumented in the offical documentations. This website offers a list of all known switches with a single sentence description of what they are doing.


List of Default Passwords

 https://default-password.info/

CTF | Passwords

The website features a large list of default passwords found in routers and IoT devices. The data is sorted by manufacturer and can be searched.


List of Looking Glasses Providing Traceroutes

 http://www.traceroute.org/#Looking%20Glass

Networks

The websites shows links to different looking glasses which provide either traceroute information or are usable as route servers.


List of Network Speed Tests

Autonomous Systems | BGP


Lists of DNS Blacklists

DNS | IP | Networks | Spam | Tools

These projects either operate DNS based Real-time Blackhole Lists (RBL) or allow checking if an IP is contained. The Multi-RBL websites are helpful in finding a large quantity of RBLs.


Malware Bazaar

 https://bazaar.abuse.ch/

The Malware Bazaar is a project by abuse.ch to create an open repository with malware samples. The repository is small in size, but it can be freely downloaded and contributed by everyone. It only contains malicious files, which is in contrast to common malware feeds like Virustotal.


Manchester Academic Phrasebank

 https://www.phrasebank.manchester.ac.uk/

The Academic Phrasebank is a general resource for academic writers. It aims to provide you with examples of some of the phraseological ‘nuts and bolts’ of writing organised according to the main sections of a research paper or dissertation.

The data bank contains the categories “Introducing Work”, “Referring to Sources”, “Describing Methods”, “Reporting Results”, “Discussing Findings”, and “Writing Conclusions”.


Netlab 360 OpenData Project

 https://data.netlab.360.com/

Amplification | Networks | Tools

The Netlab of 360.com provides some open data streams.

One dataset concerns the number of abused reflectors per protocol.


netray.io Internet Observatory

 https://netray.io/

Certificates | DNS | Networks

The Internet Observatory is a project by the RWTH Aachen University. It combines different scanning projects.

As of writing it contains information about:

  • DNS
  • HTTP2 and Server Push
  • QUIC
  • TCP Initial Window
  • Certificate Authority Aurthoization (CAA)

NetworkScan Mon

 https://scan.netlab.360.com/

Amplification | Networks | Tools

Overview over IP addresses scanning the internet and which ports are scanned.


NIST RPKI Monitor

 https://rpki-monitor.antd.nist.gov/

BGP | RPKI

The NIST RPKI Monitor shows different statistics about RPKI adoption and about the validation status. It shows the number of validating prefixes, their history, the autonomous systems with the most VALID and INVALID prefixes and how validation changes over time.


Online Hash Crackers

CTF | Hashes | Passwords

These websites have access to large rainbow tables and allow quick access to known weak hashes.


Open INTEL

 https://www.openintel.nl/

DNS | IP | Networks

Open INTEL is an active DNS database. It gathers information from public zone files, domain lists (Alexa, Umbrella), and reverse DNS entries. Once every 24 hours data is collected about a bunch of DNS RRsets (SOA, NS, A, AAAA, MX, TXT, DNSKEY, DS, NSEC3, CAA, CDS, CDNSKEY). The data is openly avaible as AVRO files and dates back until 2016.

The data can be freely downloaded. There is documentation on the layout of the AVRO files.

The project is similar to Active DNS but seems to be larger in scope.


Open Observatory of Network Interference (OONI)

 https://ooni.org/

Open Observatory of Network Interference (OONI) is an active measurement platform for censorship measurements. Many different measurements are run, such as blocking of messengers, Tor and VPN blocking, or middleboxes.


Open Source Vulnerabilities (OSV)

 https://osv.dev/

Security

osv.dev is a vulnerability database or open source projects. It mainly acts as an aggregator for multiple other databases. For example data from Google's OSS-Fuzz project and various language specific vulnerability databases is combined together.


Over The Wire: Wargames

 https://overthewire.org/wargames/

CTF | Tools

Over The Wire provides with the wargames many different challenges, to learn exploitations of different things. There are different wargames based on skill and required tooling. In each level the user has to retrieve a flag to procede to the next level.


Passive DNS (CIRCL)

 https://www.circl.lu/services/passive-dns/

DNS | IP | Networks

Passive DNS dataset from circl.lu.


Passive SSL (CIRCL)

 https://www.circl.lu/services/passive-ssl/

Certificates

Historical certificate dataset. Allows querying based on IP address or certificate.


PeeringDB

 https://www.peeringdb.com/

Autonomous Systems | BGP | Networks

Contains information for some networks about peering information. This includes peering partnes, transfer speeds, peering requirements and similar.

Documentation


Privilege Escalation Cheatsheet (Vulnhub)

 https://github.com/Ignitetechnologies/Privilege-Escalation

CTF

The repo contains a curated list of various ways to perform privilege escalation. It is sorted by different attack vectors.


Public DNS Server List

 https://www.publicdns.xyz/

DNS

The website provides a currated list of various public DNS resolver operators and the IP addresses of the DNS servers.


Public Suffix List

 https://publicsuffix.org/

DNS | Networks

The public suffix list gives a way to easily determine the effective second level domain, i.e., the domain which a domain owner registered and which can be under different owners.


Regex to parse router hostnames

Networks

https://catalog.caida.org/details/paper/2019_learning_regexes_extract_router
https://catalog.caida.org/details/paper/2020_learning_extract_use_asns

These two papers focus on how to extract information from the hostname of routers. These hostnames occur when performing traceroutes. The Regexs can be use to extract identifiers and AS numbers. The generated datasets of the papers are openly accessible.


RFC 8145 Root Trust Anchor Reports

 http://root-trust-anchor-reports.research.icann.org/

DNS | DNSSEC

The root trust anchor reports show statistics how far the support for different root signing keys is in the resolver population. The data is collected using the trust anchor reporting specified in RFC 8145. There are graphs showing the distribution over time, combined for all root servers or split per letter, and a list of IP addresses which are only reporting support for outdated key signing keys (KSK).


RIPE Atlas

 https://atlas.ripe.net/

Certificates | DNS | IP | Networks

RIPE operates a set of probes, which can be used to send pings or similar measurements. The probes are mainly placed in Europe but some are also in other continents.

All the collected measurements can be found in the RIPE Atlas Daily Archives. The blog post gives some more details.


RIPEstat: Providing open data and insights for Internet resources

 https://stat.ripe.net/

Autonomous Systems | BGP | DNS | Networks | Tools

RIPEstat is a network statistics platform by RIPE. The platform shows data for IP addresses, networks, ASNs, and DNS names. This includes information such as the registration information, abuse contacts, blocklist status, BGP information, geolocation lookups, or reverse DNS names. Additionally, the website links to many other useful tools, such as an address space hierarchy viewer, historical whois information, and routing consistency checks.


Root Servers

 https://root-servers.org

DNS | Tools

Overview page for the DNS root servers. It contains links to general news and all the supporting organizations.

The website features a map with all geographic locations. It contains information about locations, IPv4/IPv6 reachability and IP addresses.

Each root server has its own subdomain in the form of http://a.root-servers.org. It contains access to historical performance data like:

  • Size and time of zone updates
  • RCODE volume
  • query and response sizes for UDP and TCP
  • traffic volume (packets per time)
  • Unique sources

Routing Information Service (RIS)

 https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris

BGP | DNS | Networks | Tools

Different information regarding reachability and connectiveness of ASs.


ROV Deployment Monitor

 https://rov.rpki.net/

BGP | Networks | RPKI

The Route Origin Validation (ROV) Deployment Monitor measures how many AS have deployed ROV. It uses PEERING for BGP annoucements and uses BGP monitors to see in which ASs the wrong announcements are filtered. A blogpost at APNIC describes it in more detail.


RPKI Browsers

Networks | RPKI | Tools

These websites allow you to browser the valid RPKI announcements. They show which address ranges are covered by RPKI and who the issuing authority is.


RSSAC002 DNS Root Server Data

 https://www.dns.icann.org/rssac/rssac002/

DNS

RSSAC002 describes measurements for DNS root servers. It collects data, such as the load time, rcode volumes, traffic volume, and unique sources. The data is collected daily and goes back to 2013.

The data is also available in a git repository, which is not always up-to-date. https://github.com/rssac-caucus/RSSAC002-data


scans.io Internet-Wide Scan Data Repository

 https://scans.io/

Certificates | DNS | IP | Networks

The website contains no usable data anymore and only links to empty pages on Censys.

The website used to host many free Internet scans of different kinds. It included historical data and activly maintained datasets.


Shadowserver Scanning Project

 https://scan.shadowserver.org/

Networks | DNS

The Shadowserver Scanning projects performs regular Internet wide scans for many protocols. They scan for four main types of protocols:

  1. Amplification protocols, e.g., DNS or NTP
  2. Botnet protocols, e.g., Gameover Zeus or Sality
  3. Protocols that should not be exposed, e.g., Elastic Search, LDAP, or RDP
  4. Vulnerable Protocols, e.g., SSLv3

The website is a great resource to get general statistics about the protocols, like the number of hosts speaking the protocol, their geographic distribution, associated ASNs, and the historic information.


Shodan

 https://www.shodan.io/

Certificates | DNS | IP | Networks

Shodan performs regular scan on common ports.

Access is free, but requires registration. More results can be gained with a paid account.


System Security Cirrcus

 http://s3.eurecom.fr/~balzarot/notes/top4_2018/

Paper Writing | Security

The System Security Cirrcus by Davide Balzarotti presents many statistics about the Top-4 security conferences, such as authors and affiliations.


TeleGeography Map Gallery

Maps | Networks

TeleGeography provides different maps about the Internet. They contain information about submarine cables, global traffic volume, latency, internet exchange points. The data for the Submarine Map and the Internet Exchange Map can also be found on GitHub in text format.


Transient Execution Attacks

 https://transient.fail/

Security

The website lists all known speculation side channel attacks. Each attack contains information about the attacked buffer, the affected vendors, and working state. They are sorted into a hierarchy. Each attack is also linked to proof-of-concepts and the academic papers.


vizAS

 https://stats.apnic.net/vizas/

Autonomous Systems | BGP | Networks | Tools

vizAS by APNIC shows the connectiveness between different ASs split by countries. It is usefull to find the ASs which are most central in the graph.


Vulnerable (Docker) Containers

 https://vulnerablecontainers.org/

CTF | Docker

The website lists docker containers from Docker Hub with known vulnerabilities in it. The top 1000 docker containers from Docker Hub are regularly scanned with Trivy and the results reported here.

A similar tool to scan for vulnerable containers is Clair scanner.


WAND Active Measurement Project

 https://amp.wand.net.nz/

Autonomous Systems | DNS | Networks | Tools

AMP is a system designed to continuously perform active network measurements between a mesh of specialist monitor machines, as well as to other targets of interest. These measurements are used to provide both a view of long-term network performance as well as to detect notable network events when they happen.

The project is run with a custom client and server software. The measurement results can be viewed on the website. It includes traceroutes, latencies (DNS, HTTP, ICMP, TCP), HTTP page sizes, and packet loss. The software is available as open source.


World Country Information

 https://mledoze.github.io/countries/

Many different metadata about countries, such as name, country code, languages. It also has a geojson of the country outline and the flags.


x86 Instruction Set

Cheatsheet | x86 | CTF

These websites provide reference documentation of the x86 instruction set:


Zonefiles: Domain Lists

 https://zonefiles.io/detailed-domain-lists/

Networks

The website provides download access to domains in many TLDs. Most lists are updated daily. However, not all of the lists seem complete. For example, DENIC reports that they manage over 17 million domains, whereas zonefiles.io only reports over 6 million domains.