Developing and running a testbed for the DDoS Clearing House
Demonstrating the DDoS Clearing House in a representative simulated environment
Authors: Thijs van den Hout (1), Remco Poortinga – van Wijnen (2), Cristian Hesselman (1, 3), Christos Papachristos (4), Karin Vink (1)
- SIDN, the Netherlands
- SURF, the Netherlands
- University of Twente, the Netherlands
- FORTH, Greece
We have created a distributed testbed that enables us to realistically test the DDoS Clearing House: a system that enables organisations to handle DDoS attacks more proactively by automatically sharing measurements of the DDoS attacks they handle. Our testbed allows us to temporarily skip typically time-consuming organisational processes such as setting up data sharing agreements and deploying software in production systems, which helps to advance the system towards a pilot and a production version. We discuss the motivation for developing our testbed, its requirements, implementation and our lessons learnt. We’re developing the Clearing House and the testbed as part of the CONCORDIA project, and we’ll be using both in the Dutch Anti-DDoS Coalition.
DDoS Clearing House recap
The DDoS Clearing House is a system that enables organisations to continually and automatically share measurements of the DDoS attacks they handle in the form of “DDoS fingerprints”. The Clearing House thus widens these organisations’ view on the DDoS attack landscape and enables them to proactively prepare their networks for a particular DDoS attack before it might actually hit them, which reduces the probability of system outages and increases the availability of services for customers and users. The DDoS Clearing House is an additional layer of security that complements DDoS mitigation services such as NBIP’s NaWas, which organisations need to have in place to handle actual DDoS traffic.
The DDoS Clearing House is one of the core services of a so-called “Anti-DDoS Coalition”, a group of organisations that have committed to fight DDoS attacks collaboratively. One such coalition is the Dutch Anti-DDoS Coalition, which connects eighteen organisations from different sectors in the Netherlands, such as telecom, internet infrastructure, banking, and government. Coalition members are potential targets of DDoS attacks, or organisations in the business of DDoS mitigation. They share knowledge and expertise, for instance through the Clearing House and through joint large-scale DDoS drills. Joining forces in a coalition enables them to organise collaborative and proactive anti-DDoS activities.
The DDoS Clearing House consists of three core components: the Dissector, DDoS-DB, and Converters.
- The Dissector analyses collected DDoS traffic captures and generates a fingerprint that summarize the attack’s key characteristics, such as protocol types, packet lengths, and source IP addresses. Coalition members run the Dissector in their networks and process their own network traffic internally.
- DDoS-DB is a logically centralised interactive database that stores and shares DDoS fingerprints from Dissectors operated by the members of an anti-DDoS coalition. DDoS-DB is a shared facility among coalition partners.
- A Converter receives fingerprints from DDoS-DB and enables members to ward off the attack described in a fingerprint targeted at them. A Converter transforms DDoS fingerprints into mitigation rules. Since fingerprints only contain the metadata of the attacks and not actual attack traffic, coalition partners don’t run the risk of unintentionally sharing privacy or commercially sensitive network traffic information with others.
Why do we need a DDoS clearing house testbed?
We need a testbed to learn how the DDoS Clearing House operates in a realistic setting, but without the members of an Anti-DDoS Coalition having to add a Dissector to their networks or to exchange fingerprints with actual IP addresses in them. This is important because we have experienced that such processes often take a significant amount of time, typically in the order of months. For example, organisations need to modify their production networks to add the Dissector, and they need to sign a data sharing agreement because the IP addresses in DDoS fingerprints are personally identifiable information (PII). While such processes are indispensable for a production version of the Clearing House (TRL8-9), they can significantly slow down the development and evaluation of the system.
We aimed to achieve TRL6 with our testbed (“Technology demonstrated in relevant environment”). It precedes an actual pilot with the Dutch Anti-DDoS Coalition, which will be at TRL7 (“System prototype demonstration in operational environment”). In the pilot, the members of the coalition will install Dissectors in their networks and sign data sharing agreements covering the exchange of fingerprints via the DDoS-DB. Our goal is to use the pilot as input for a production system (TRL8-9).
Technical requirements
We designed the Clearing House testbed based on four key requirements:
- The testbed must allow us to test the DDoS Clearing House without the typically time-consuming deployment and legal processes that are needed for a production-level system. At the same time, the testbed must meet the basic legal requirements described in the next section.
- It must closely resemble the technical operational setting in which the Clearing House will be deployed after development, by which we mean that it operates in an environment distributed over the internet, and not in an isolated virtualised network.
- It must allow us to test and demonstrate each component of the Clearing House, and importantly, the system as a whole: from DDoS traffic captures at an anti-DDoS coalition member to the use of mitigation rules by other members; the potential victims.
- The testbed must be able to emulate DDoS attacks in which distributed remote sources send traces of DDoS traffic to a target participant on the testbed. This doesn’t mean that it should send a real DDoS attack; a small sample of test traffic is sufficient to adequately test the Clearing House. The testbed should therefore prevent members sending more traffic than required by the Dissector.
Legal requirements
Before we started developing the Clearing House testbed, we first consulted our legal department to discuss the legal requirements. This is because the testbed involves generating DDoS test traffic to trigger the operation of the Clearing House, which may create liability risks. We were also interested in our options regarding testbed agreements between coalition members, and how to minimise them for testing purposes.
We learnt from our legal experts that we could operate the Clearing House testbed under three types of agreement, ranging from strict to informal:
- A fully-fledged agreement among the testbed partners with all details set in stone, which would be similar to an agreement for a pilot or production system.
- A waiver/disclaimer to relieve the traffic source or operating party of any liability, for instance as a result of receiving the DDoS test traffic on their systems.
- An agreement by e-mail, covering only the basic operational details of the testbed.
The choice depends on the members participating in the testbed and the level of trust between them. In our case, SIDN Labs and SURF collaborate on the testbed in the form of a virtual anti-DDoS coalition. We opted for the third and most informal agreement, because SIDN Labs and SURF have often collaborated in the past and have considerable mutual trust. Furthermore, both parties have separate networks and resources reserved for research, outside the operational infrastructures of the respective parent organisations, reducing the potential for an accident to have seriously detrimental consequences. Lastly, both parties are Dutch, meaning that they operate in the same legal jurisdiction and are both familiar with the laws that would apply if a legal issue unexpectedly arose.
Another legal requirement is that members of a virtual coalition are only able to send simulated DDoS traffic to themselves, thus avoiding liability to others. The testbed is not meant to load-test other partners, but to receive test traffic for evaluation of the Clearing House.
Unlike the testbed, pilots in established anti-DDoS coalitions and production-level versions of the DDoS Clearing House will require formal data sharing agreements.
Design of the Clearing House testbed
Figure 1 provides an overview of the design of our testbed, which we will explain in more detail in the following sections. The testbed consists of three elements: a virtual anti-DDoS coalition (“vCoalition” for short), a DDoS attack traffic generator, and the DDoS Clearing House deployed throughout the vCoalition. In our design, we distributed these components over the internet, as opposed to virtualized in a single network, because this is more representative of the situation in which the DDoS Clearing House will operate.
Virtual anti-DDoS coalition
To test the Clearing House in a realistic environment, we need to deploy its components in a virtual anti-DDoS coalition (vCoalition, for short). Our vCoalition consists of SIDN Labs and SURF, two CONCORDIA partners tasked with the development of the DDoS Clearing House. The reason we can execute pilots on this platform without a data sharing agreement is that no real DDoS data is used. To avoid sharing PII, we generate traces of DDoS attack traffic using our traffic generator. We are free to share these traffic captures with whomever, since they do not contain any personal IP addresses.
Traffic generator
The traffic generator allows a member of a vCoalition to generate traffic samples of various kinds of DDoS attack and send it to themselves. The traffic generator does not launch real DDoS attacks and only allows for negligible traffic volumes (5 Mbps tops). It currently supports TCP-no-flag attacks, TCP SYN/ACK/FIN-floods, UDP floods, fragmentation attacks with various protocols, and ICMP floods.
The traffic generator is made up of five virtual machines distributed across the world in a cloud platform to resemble a botnet that generates a DDoS attack. These five machines simultaneously send traffic to one of the coalition partners only at that partner’s request. The source IP addresses in the traffic belong to the cloud provider and not individual people, which means we can use the traffic captures for the testbed because they do not contain any personal IP addresses (no PII).
We have created an online dashboard through which vCoalition members can send customisable traffic to themselves (see Figure 2). Many details of the internet traffic that makes up a DDoS attack can be tweaked on this dashboard, including the communication protocol, traffic speed, destination port(s), packet fragmentation, TCP flags, ICMP mode, packet size, etc.
Because we only need a small sample of DDoS data to test the system, we have limited the capacity of the traffic generator so that it cannot send more than 1,000 packets per second from each machine, with no more than 1,000 bytes per packet. With the five virtual machines we use to generate traffic that amounts to 5MB/s at most, which is insufficient to actually DDoS a typical contemporary network.
When a vCoalition member initiates an attack, our dashboard shows a confirmation screen with the exact details of the traffic that the traffic generator will send. As an additional safety measure, the transmission can be stopped on the dashboard at any point. Also, the dashboard is only accessible to the members of the vCoalition (SURF and SIDN Labs in our current setup). Access is controlled using an IP allow-list and password login.
The server that hosts the dashboard instructs the five globally distributed virtual machines through commands sent over SSH. We use a packet assembly tool called Hping3 to construct and send internet packets to the target vCoalition member.
The Clearing House testbed in action
Our testbed allows us to test the technical components of the Clearing House in a distributed manner. The generated “DDoS traffic” originates from all over the world, partners in the vCoalition are physically separated, and the DDoS Clearing House is deployed on the systems of vCoalition members.
Figure 3 shows the operation of our testbed in 8 steps using an animation.
Firstly, Member 1 (SIDN Labs in our current setup) instructs the traffic generator to send them customised DDoS attack traffic. Member 1 captures this traffic on the target machine using tcpdump and passes the packet capture (PCAP) files to its Dissector. The Dissector examines the PCAPs, summarises them into a fingerprint, and automatically shares the fingerprint with the central instance of DDoS-DB (ddosdb.eu). Another option is that Member 1 uploads the fingerprint to a local instance of DDoS-DB first, which subsequently updates the remote, central instance of DDoS-DB.
Next, Member 2 downloads the fingerprint from the central DDoS-DB and uses its Converter to generate a mitigation strategy in the form of iptables rules. Our current implementation of the Converter simply blocks IP addresses on the targeted ports and protocols.
After Member 2 has generated the mitigation rules, they request the same attack previously sent to Member 1 from the traffic generator. We can then determine if the generated mitigation rules can indeed proactivity handle DDoS attacks using the fingerprints shared by other partners in the vCoalition.
Results
We used the testbed to test the DDoS Clearing House with various types of synthetic DDoS traffic. All attacks were successfully sent and processed by the target machines. Using Elasticsearch, Kibana and network analysis package Packetbeat, we can visualise the incoming traffic. Figure 4 shows a visualisation of a 15-second UDP flood attack.
Following the testbed procedure as illustrated in Figure 3, we found that we could indeed block a DDoS attack on Member 2 by applying the mitigation rules created by the Converter.
Using the testbed, we demonstrated the entire cycle of the DDoS Clearing House, from (simulated) DDoS attack on one partner in the coalition to the successful mitigation of that attack on another partner. Importantly, we did so without the need to share real DDoS fingerprints, which, in production, would invariantly include personal IP addresses and would therefore require a data sharing agreement with the owners and all partners in the coalition.
What have we learnt?
The first thing we learnt is what it takes to thoroughly test the DDoS clearing house without going through the time-consuming deployment and legal processes required for a pilot or production-level system, while still fulfilling basic legal requirements such as members of a vCoalition being able to only initiate “DDoS attacks” onto themselves.
Our second lesson learnt is that we now know how to construct a simulated DDoS Clearing House environment that is representative of a pilot run in an actual anti-DDoS coalition. We did this by distributing the Clearing House components globally over the internet, using representative data, and with a simulated anti-DDoS coalition of two companies.
Our third lesson learnt is how to organise a simulation of the DDoS Clearing House from a legal perspective. We reviewed the various options for legal agreements, which are also beneficial for parties wanting to set up their own DDoS Clearing House and coalitions. We will also include these considerations in the cookbook which we will publish in the context of CONCORDIA.
Importantly, we have also gained knowledge on how the DDoS Clearing House will operate as a whole in a pilot or production environment and can use those lessons to improve the Clearing House components. Our testbed has already enabled us to make various improvements, such as better fingerprint synchronisation between multiple instances of DDoS-DB.
Finally, our testbed meets the requirements we laid out earlier in this blog: it closely represents the operational environment due to its distributed nature, it works with lightweight legal and deployment requirements, and it allows us to test the entire Clearing House, from start to finish.
Plans for the future
Our next steps are to carry out a pilot with the Dissector deployed on the systems of members of the Dutch Anti-DDoS Coalition as well as a consortium in Italy, using the lessons we learnt from the testbed. We also expect that our testbed will further encourage the partners in those coalitions to deploy the Dissector on their infrastructures.
We’ll improve the testbed further by enabling the traffic generator to produce traces of more intricate types of DDoS attack, such as amplification attacks, reflection attacks and combinations of the current possibilities, for example.
We also want to improve the Converter component, since a simple blocklist is not sufficient for all types of DDoS attack, such as DNS reflection attacks or attacks with spoofed IP addresses.
Finally, we plan to explore how the testbed could serve as a cyber-range in CONCORDIA, for instance as part of a DDoS drill.
It’s open source
Both the Clearing House software and the testbed are open source. You can view the source code on our GitHub organisation page.
Acknowledgements
SIDN, the University of Twente, SURF, and FORTH were partly funded by the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement No 830927. Project website: https://www.concordia-h2020.eu/
SIDN, the University of Twente, and SURF are members of the Dutch National Anti-DDoS Coalition, a self-funded public-private initiative to collaboratively protect members and the wider internet community from DDoS attacks. Website: https://www.nomoreddos.org/en/