Paul Vixie joins us on the History of Networking to talk about the spread of the DNS system—like a virus through the body network. All those radios in the background at a bit of history; Paul is an Amateur Radio Operator of many years, though, like me, he is not as active as he used to be in this realm.
In this recording, Jordan, Donald, and I talk to Paul Mockapetris, who took on the problem of naming in the original Internet. Back when routers were called fuzzballs, and the only way to get to a destination was by referencing a host file…
One interesting trend of the last year or two is the rising use of data analytics and ANI (Artificial Narrow Intelligence) in solving network engineering problems. Several ideas (and/or solutions) were presented this year at the IETF meeting in Seoul; this post takes a look at one of these. To lay the groundwork, botnets are often controlled through a set of domain names registered just for this purpose. In the same way, domain names are often registered just to provide a base for sending bulk mail (SPAM), phishing attacks, etc. It might be nice for registrars to make some attempt to remove such domains abused for malicious activities, but it’s difficult to know what “normal” activity might look like, or for the registrar to even track the usage of a particular domain to detect malicious activity. One of the papers presented in the Software Defined Network Research Group (SDNRG) addresses this problem directly.
The first problem is actually collecting enough information to analyze in a useful way. DNS servers, even top level domain (TLD) servers collect a huge amount of data—much more than most engineers might suspect. In fact, the DNS system is one of those vast sources of information about people and organizations not many engineers are aware of; there is much you can learn by looking at patterns of DNS queries, even DNS servers that don’t directly serve individual client machines. To give a sense of the amount of information DNS servers throw off, according to the presentation, the .nl DNS servers produce 50Tb+ of information on a regular enough basis that storage and analysis of the data can become a problem. To solve the problem, SIDN labs built ENTRADA.
Essentially, ENTRADA is a custom built streaming data store that accepts PCAP files pulled from SIDN’s TLD servers for the .nl domain, and then pushes the data through a series of map/reduce jobs on a Hadoop cluster to discover domains being used for malicious purposes. They built a custom streaming data store and back end processors—
A map/reduce job is run across the PCAP data on a regular basis, just looking for patterns in the information. What turns up is actually pretty interesting. For instance, this is the usage chart (the query rate/etc.) for two different domain names that show up in the PCACP files, as shown in their slides—
The usage patterns of these two domains look completely different. As it turns out in each of the cases SIDN labs has looked at, the domain on the right is characteristic of a domain being used to source phishing attacks. The domain on the left, however, is what “normal usage” actually looks like. These usage patterns, then, are often able to allow the operator to distinguish between valid and malicious use of a domain name. Domain names that appear to be used for malicious activity can be investigated further by a human, referred to local law enforcement, or simply not renewed when their contract expires. Other things that can be detected through these patters are SPAM injection events and other problems.
ENTRADA is an interesting way to find and squash domain names being used for malicious activity. If you’re interested in seeing how this works, they have more information, and even a demo, on their web site—http://entrada.sidnlabs.nl You can not only learn more about the project there, you can even find information on how to contribute to the code.
Could you use DNS names to translate human-readable rules into packet filters? The traditional answer was “no, because I don’t trust DNS”.
This has been a pet peeve of mine for some years—particularly after my time at Verisign Labs, looking at the DNS system, and its interaction with the control plane, in some detail. I’m just going to say this simply and plainly; maybe someone, somewhere, will pay attention—
The Domain Name System is a part of the IP networking stack.
Network engineers and application developers seem to treat DNS as some sort of red-headed-stepchild; it’s best if we just hide it in some little corner someplace, and hope someone figures out how to make it work, but we’re not going to act like it should or will work. We’re just going to ignore it, and somehow hope it goes away so we don’t have to deal with it.
Let’s look at some of the wonderful ideas this we’ll just ignore DNS has brought us over the years, like, “let’s embed the IP address in the packet someplace so we know who we’re talking to,” and “we must build all filters based on IP addresses” (the instance Ivan is talking about). We’ve gone for years treating one thing as the other, so that now we actually want to split the IP address space up so we can have “locators” and “identifiers.”
It’s all nonsense—IP addresses are topological locations on the network, and DNS names are host identifiers. If you want to know which host you’re talking to, you need to know what it’s DNS name is, not where it is attached to the network. So why don’t we use DNS names for hosts? Because “DNS is too slow,” and “it would be too much work to make DNS interactive, so that individual hosts can tell the DNS system where they are.”
Perhaps this was all true years ago—but it was also true that routing was slow years ago. Why was routing slow? Because processors were slow, bandwidth was slow, and memory was precious. Routing is now fast because processors are fast, bandwidth is high, and memory is (at least) not extremely expensive. Today we stream HD movies in near real time, and yet we still whine about a couple of DNS packets. Really?
What we’ve done is carted out the same old tired excuses for not using DNS for years on end, rather than fixing it. As a result, we’ve moved policy and identification into routing, making a complete mess to solve problems that don’t really make sense to solve in routing.
The reality is: if you own the DNS system in your network, DNS is only as limited and slow as you make it.
Even on the global ‘net, it’s not as slow as you think. We’re all so conditioned to our service providers telling us “it could take up to 24 hours for DNS information to propagate throughout the Internet,” that we just think even the best run DNS servers always take 24 hours to “get the information out.” There are even services out there promising to solve this problem, making your DNS changes fast. I can’t find any real information on this, so I’m going to fall back to anecdotal evidence—I shift a few sites around from service provider to service provider from time to time, and I’ve never seen it take more than 2-3 minutes to switch from IP address to another.
The bottom line is this—don’t be afraid to use DNS for what it’s designed for in your network. There are specific applications where it might not make sense to use a DNS query—when the destination address is an anycast for a specific service, for instance, or when milliseconds actually count. But by and large, the more you stick with using DNS to indicate devices, and addresses to indicate locations, the easier your network is going to be to manage over the long term.
Location and identity are definitely two different things, but we don’t need to treat IP addresses as if they were identifiers to solve the locater/identifier problem set.
We need to learn to treat DNS like it’s a part of the IP stack, rather than something that “only the server folks care about,” or “a convenience for users we don’t really take seriously for operations.”
DDoS attacks, particularly for ransom—essentially, “give me some bitcoin, or we’ll attack your server(s) and bring you down,” seem to be on the rise. While ransom attacks rarely actually materialize, the threat of DDoS overall is very large, and very large scale. Financial institutions, content providers, and others regularly consume tens of gigabits of attack traffic in the normal course of operation. What can be done about stopping, or at least slowing down, these attacks?
To answer, this question, we need to start with some better idea of some of the common mechanisms used to build a DDoS attack. It’s often not effective to simply take over a bunch of computers and send traffic from them at full speed; the users, and the user’s providers, will often notice machine sending large amounts of traffic in this way. Instead, what the attacker needs is some sort of public server that can (and will) act as an amplifier. Sending this intermediate server should cause the server to send an order of a magnitude more traffic towards the attack target. Ideally, the server, or set of servers, will have almost unlimited bandwidth, and bandwidth utilization characteristics that will make the attack appear as close to “normal operation” as possible.
It is at this point that DNS servers often enter the picture. There are thousands (if not tens of thousands) of DNS servers out there, they all have public facing DNS services ripe for exploitation, and they tend to be connected to large/fat pipes. What the attacker can do is send a steady stream of DNS queries that appear to be originating at the target towards the DNS server. The figure below illustrates the concept.
The attacker will carefully examine the DNS table, of course, choosing a large record—the largest TXT record possible is ideal—and send a request for this item, or a series of similar large items, asking the DNS server to send the response to the target. The DNS server, having no idea where the query actually originates, will normally reply with the information requested.
But one DNS server isn’t going to be enough. As shown in the illustration, the attacker can actually send queries to hundreds or thousands of DNS servers; an investment of ten to fifteen packets per second in queries can generate tens of thousands replies each second in return. To ramp the attack up, the attacker can install software (probably through a botnet) that can use hundreds or thousands of hosts to generate packets towards thousands of DNS servers—this kind of amplification can easily overwhelm even the largest edge circuits available.
How can this sort of attack be countered? The key point is to remove amplification wherever possible—in this case, DNS servers seem like a likely point at which the amplification attack can be cut down to size. But how? RFC7873, DNS Cookies, published in May of 2016, provides a weak authentication mechanism to reduce the effectiveness of DNS as an amplification platform.
Essentially, this RFC specifies that each DNS query should include a cookie, or what might be called a nonce. This cookie is calculated using a consistent algorithm (a hash) computed based on several items, such as the IP address of the client (sender), the IP address of the server, etc. The idea is this: the first time a host queries a DNS server, it will send a client cookie. The server can respond with an error, simply ignore the request, or respond with a DNS packet containing the server’s cookie. The rate at which the server hands the cookies out can be rather limited; assuming clients can cache the server’s cookie as well as previous DNS records retrieved from the server, the additional rate limiting should have very little impact on the host’s operation. The next time the client sends a request, it will include this server cookie. The server, on seeing a valid cookie, will reply to the request as normal.
The figure below illustrates.
This process accomplishes two things:
- Rate limiting responses to “cookie requests” effectively caps the speed at which any given client can send packets with a forged source address to the server and expect a reply. It doesn’t matter how fast the attacker sends packets, it will only receive a cookie making it possible to send a valid request on an infrequent basis.
- Given the attacker will not be able to successfully forge a cookie for the target, the only thing the attacker can force the DNS server to send to the target is a (purposefully) short error message. These error messages will have nowhere near the impact of the carefully chosen large TXT (and other) records.
This is, as the draft says, a “weak protection,” but it is enough to remove DNS servers out of the realm of “easy targets” for amplification duty in DDoS attacks.