The Hedge 10: Pavel Odintsov and Fastnetmon

Fastnetmon began life as an open source DDoS detection tool, but has grown in scope over time. By connecting Fastnetmon to open source BGP implementations, operators can take action when a denial of service event is detected, triggering black holes and changing route preferences. Pavel Odintsov joins us to talk about this interesting and useful open source project.

download episode

you can subscribe to the Hedge on iTunes and other podcast listing sites, or use the RSS feed directly from

Research: Practical Challenge-Response for DNS

Because the speed of DNS is so important to the performance of any connection on the ‘net, a lot of thought goes into making DNS servers fast, including optimized software that can respond to queries in milliseconds, and connecting DNS servers to the ‘net through high bandwidth links. To set the stage for massive DDoS attacks based in the DNS system, add a third point: DNS responses tend to be much larger than DNS queries. In fact, a carefully DNS response can be many times larger than the query.

To use a DNS server as an amplifier in a DDoS attack, then, the attacker sends a query to some number of publicly accessible DNS servers. The source of this query is the address of the system to be attacked. If the DNS query is carefully crafted, the attacker can send small packets that cause a number of DNS servers to send large responses to a single IP address, causing large amounts of traffic to the system under attack.

Rami Al-Dalky, Michael Rabinovich, and Mark Allman. 2018. Practical Challenge-Response for DNS. In Proceedings of the Applied Networking Research Workshop (ANRW ’18). ACM, New York, NY, USA, 74-74. DOI:

Carrying DNS over TCP is one way to try to resolve this problem, because TCP requires a three-way handshake. When the attacker sends a request with a spoofed source address, the server attempts to build a TCP session with the system who owns the spoofed address, which will fail. A key point: TCP three-way handshake packets are much smaller than most DNS responses, which means the attacker’s packet stream is not being amplified (in size) by the DNS server.

DNS over TCP is problematic, however. For instance, many DNS resolvers cannot reach an authoritative DNS server using TCP because of stateful packet filters, network address translators, and other processes that either modify or block TCP sessions in the network. What about DNSSEC? This does not prevent the misuse of a DNS server; it only validates the records contained in the DNS database. DNSSEC just means the attacker can send even larger really secure DNS records towards an unsuspecting system.

Another option is to create a challenge-response system much like the TCP handshake, but embed it in DNS. The most obvious place to embed such a challenge response system is in CNAME records. Assume a recursive DNS server requests a particular record; an authoritative server can respond with a CNAME record effectively telling the recursive server to ask someone else. When the recursive server sends the second query, presumably to a different server, it includes the response information it has in order to give the second server the context of its request.

To build a challenge-request system, the authoritative server sends back a CNAME telling the recursive server to contact the very same authoritative server. In order to ensure the three-way handshake is effective, the source IP address of the querying recursive DNS server is encoded into the CNAME response. When the authoritative server receives the second query, it can check the source address encoded in the second resolution request against the source of the packet containing the new query. If they do not match, the authoritative server can drop the second request; the three-way handshake failed.

If the source of the original request is spoofed, this causes the victim to receive a CNAME response telling it to ask again for the answer—which the victim will never respond to, because it did not send the original request. Since CNAME responses are small, this tactic removes the amplification the attacker is hoping for.

There is one problem with this solution, however: DNS resolvers are often pooled behind a single anycast address. Consider a resolving DNS server pool with two servers labeled A and B. Server A receives a DNS request from a host, and finding it has no cache entry for the destination, recursively sends a request to an authoritative server. The authoritative server, in turn, sends a challenge to the IP address of server A. This address, however, is an anycast address assigned to the entire pool of recursive servers. For whatever reason, the challenge—a CNAME response asking the recursive server to ask at a different location—is directed to B.

If the DNS software is set up correctly, B will respond to the request. However, this response will be sourced from B’s IP address, rather than A’s. Remember the source of the original query is encoded in the CNAME response from the responding server. Since the address encoded in the follow-on query will not match B’s address, the authoritative server will drop the request.

To solve this problem, the authors of this paper suggest a chained response. Rather than dropping the request with an improperly encoded source address, encode the new source address in the packet and send another challenge in the form of a CNAME response. Assuming there are only two servers in the pool, the next query with the encoded list of IP addresses from the CNAME response will necessarily match one of the two available source addresses, and the authoritative server can respond with the correct information.

What if the pool of recursive servers is very large—on the order of hundreds or thousands of servers? While one or two “round trips” in the form of a three-way handshake might not have too much of a performance impact, thousands could be a problem. To resolve this issue, the authors suggest taking advantage of the observation that once the packets being transmitted between the requester and the server are as large as the request itself, any amplification gain an attacker might take advantage of has been erased. Once the CNAME packet grows to the same size as a DNS request by adding source addresses observed in the three-way handshake process, the server should just answer the query. This (generally) reduces the number of round trips down to three or four before the DNS is not going to generate any more data than the attacker could send to the victim directly, and dramatically improves the performance of the scheme.

I was left with one question after reading this paper: there are carefully crafted DNS queries that can cause very large, multipacket responses. These are not mentioned at all in the paper; this seems like an area that would need to be considered and researched more deeply. Overall, however, this seems like it would be an effective system to reduce or eliminate the use of authoritative servers in DDoS reflection attacks.

Research: Tail Attacks on Web Applications

When you think of a Distributed Denial of Service (DDoS) attack, you probably think about an attack which overflows the bandwidth available on a single link; or overflowing the number of half open TCP sessions a device can have open at once, preventing the device from accepting more sessions. In all cases, a DoS or DDoS attack will involve a lot of traffic being pushed at a single device, or across a single link.


  • Denial of service attacks do not always require high volumes of traffic
  • An intelligent attacker can exploit the long tail of service queues deep in a web application to bring the service down
  • These kinds of attacks would be very difficult to detect


But if you look at an entire system, there are a lot of places where resources are scarce, and hence are places where resources could be consumed in a way that prevents services from operating correctly. Such attacks would not need to be distributed, because they could take much less traffic than is traditionally required to deny a service. These kinds of attacks are called tail attacks, because they attack the long tail of resource pools, where these pools are much thinner, and hence much easier to attack.

There are two probable reasons these kinds of attacks are not often seen in the wild. First, they require an in-depth knowledge of the system under attack. Most of these long tail attacks will take advantage of the interaction surface between two subsystems within the larger system. Each of these interaction surfaces can also be attack surfaces if an attacker can figure out how to access and take advantage of them. Second, these kinds of attacks are difficult to detect, because they do not require large amounts of traffic, or other unusual traffic flows, to launch.

The paper under review today, Tail Attacks on Web Applications, discusses a model for understanding and creating tail attacks in a multi-tier web application—the kind commonly used for any large-scale frontend service, such as ecommerce and social media.

Huasong Shan, Qingyang Wang, and Calton Pu. 2017. Tail Attacks on Web Applications. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS ’17). ACM, New York, NY, USA, 1725-1739. DOI:

The figure below illustrates a basic service of this kind for those who are not familiar with it.

The typical application at scale will have at least three stages. The first stage will terminate the user’s session and render content; this is normally some form of modified web server. The second stage will gather information from various backend services (generally microservices), and pass the information required to build the page or portal to the rendering engine. The microservices, in turn, build individual parts of the page, and rely on various storage and other services to supply the information needed.

If you can find some way to clog up the queue at one of the storage nodes, you can cause every other service along the information path to wait on the prior service to fulfill its part of the job in hand. This can cause a cascading effect through the system, where a single node struggling because of full queues can cause an entire set of dependent nodes to become effectively unavailable, cascading to a larger set of nodes in the next layer up. For instance, in the network illustrated, if an attacker can somehow cause the queues at storage service 1 to fill up, even for a moment, this can cascade into a backlog of work at services 1 and 2, cascading into a backlog at the front-end service, ultimately slowing—or even shutting—the entire service down. The queues at storage service 1 may be the same size as every other queue in the system (although they are likely smaller, as they face internal, rather than external, services), but storage system 1 may be servicing many hundreds, perhaps thousands, of copies of services 1 and 2.

The queues at storage service 1—and all the other storage services in the system—represent a hidden bottleneck in the overall system. If an attacker can, for a few moments at a time, cause these internal, intra-application queue to fill up, the overall service can be made to slow down to the point of being almost unusable.

How plausible is this kind of attack? The researchers modeled a three-stage system (most production systems have more than three stages) and examined the total queue path through the system. By examining the queue depths at each stage, they devised a way to fill the queues at the first stage in the system by sending millibursts of valid sessions requests to the rend engine, or the use facing piece of the application. Even if these millibursts are spread out across the edge of the application, so long as they are all the same kind of requests, and timed correctly, they can bring the entire system down. In the paper, the researchers go further and show that once you understand the architecture of one such system, it is possible to try different millibursts on a running system, causing the same DoS effect.

This kind of attack, because it is built out of legitimate traffic, and it can be spread across the entire public facing edge of an application, would be nearly impossible to detect or counter at the network edge. One possible counter to this kind of attack would be increasing capacity in the deeper stages of the application. This countermeasure could be expensive, as the data must be stored on a larger number of servers. Further, data synchronized across multiple systems will subject to CAP limitations, which will ultimately limit the speed at which the application can run anyway. Operators could also consider fine grained monitoring, which increases the amount of telemetry that must be recovered from the network and processed—another form of monetary tradeoff.


On the ‘web: The Value of MANRS

Route leaks and Distributed Denial of Service (DDoS) attacks have been in the news a good deal over the last several years; but the average non-transit network operator might generally feel pretty helpless in the face of the onslaught. Perhaps you can buy a DDoS mitigation service or appliance, and deploy the ubiquitous firewall at the edge of your network, but there is not much else to be done, right? Or maybe wait on the Internet at large to “do something” about these problems by deploying some sort of BGP security. But will adopting a “secure edge,” and waiting for someone else to solve the problem, really help? @ECI