When rolling out a new protocol such as IPv6, it is useful to consider the changes to security posture, particularly the network’s attack surface. While protocol security discussions are widely available, there is often not “one place” where you can go to get information about potential attacks, references to research about those attacks, potential counters, and operational challenges. In the case of IPv6, however, there is “one place” you can find all this information: draft-ietf-opsec-v6. This document is designed to provide information to operators about IPv6 security based on solid operational experience—and it is a must read if you have either deployed IPv6 or are thinking about deploying IPv6.
The draft is broken up into four broad sections; the first is the longest, addressing generic security considerations. The first consideration is whether operators should use Provider Independent (PI) or Provider Assigned (PA) address space. One of the dangers with a large address space is the sheer size of the potential routing table in the Default Free Zone (DFZ). If every network operator opted for an IPv6 /32, the potential size of the DFZ routing table is 2.4 billion routing entries. If you thought converging on about 800,000 routes is bad, just wait ‘til there are 2.4 billion routes. Of course, the actual PI space is being handed out on /48 boundaries, which makes the potential table size exponentially larger. PI space, then, is “bad for the Internet” in some very important ways.
This document provides the other side of the argument—security is an issue with PA space. While IPv6 was supposed to make renumbering as “easy as flipping a switch,” it does not, in fact, come anywhere near this. Some reports indicate IPv6 re-addressing is more difficult than IPv4. Long, difficult renumbering processes indicate many opportunities for failures in security, and hence a large attack surface. Preferring PI space over PA space becomes a matter of reducing the operational attack surface.
Another interesting question when managing an IPv6 network is whether static addressing should be used for some services, or if all addresses should be dynamically learned. There is a perception out there that because the IPv6 address space is so large, it cannot be “scanned” to find hosts to attack. As pointed out in this draft, there is research showing this is simply not true. Further, static addresses may expose specific servers or services to easy recognition by an attacker. The point the authors make here is that either way, endpoint security needs to rely on actual security mechanisms, rather than on hiding addresses in some way.
Other very useful topics considered here are Unique Local Addresses (ULAs), numbering and managing point-to-point links, privacy extensions for SLAAC, using a /64 per host, extension headers, securing DHCP, ND/RA filtering, and control plane security.
If you are deploying, or thinking about deploying, IPv6 in your network, this is a “must read” document.
When deploying IPv6, one of the fundamental questions the network engineer needs to ask is: DHCPv6, or SLAAC? As the argument between these two has reached almost political dimensions, perhaps a quick look at the positive and negative attributes of each solution are. Originally, the idea was that IPv6 addresses would be created using stateless configuration (SLAAC). The network parts of the address would be obtained by listening for a Router Advertisement (RA), and the host part would be built using a local (presumably unique) physical (MAC) address. In this way, a host can be connected to the network, and come up and run, without any manual configuration. Of course, there is still the problem of DNS—how should a host discover which server it should contact to resolve domain names? To resolve this part, the DHCPv6 protocol would be used. So in IPv6 configuration, as initially conceived, the information obtained from RA would be combined with DNS information from DHCPv6 to fully configure an IPv6 host when it is attached to the network.
There are several problems with this scheme, as you might expect. The most obvious is that most network operators do not want to deploy two protocols to solve a single problem—configuring IPv6 hosts. What might not be so obvious, however, is that many network operators care a great deal about whether hosts are configured statelessly or through a protocol like DHCPv6.
Why would an operator want stateful configuration? Primarily because they want to control which devices can receive an IPv6 address, and hence communicate with other devices on the network. When using DHCPv6, just like DHCP with IPv4, the operator can set parameters around what kinds of devices, or perhaps even which specific devices, will be able to receive an IPv6 address. Further, the DHCPv6 server can be tied to the DNS server, so each host which connects to the network can also be given a DNS entry. Proper DNS entries are often a requirement for many applications. There are Dynamic DNS (DDNS) implementations that can solve this problem, but they are not often considered secure enough for a controlled network environment.
Why would an operator want stateless autoconfiguration? First, because they want any random user who can successfully connect to the network to be able to get an IPv6 address without any other configuration, and without the provider needing run any sort of special protocol or configuration to allow this. In fact, DHCPv6, in some environments, at least, can be seen as an attack surface, or rather a hole through which attacks can potentially be driven. Second, stateful configuration also has a failover problem; if the DHCPv6 server fails, then hosts can no longer obtain an IPv6 address, and the network no longer works. This could be, to say the least, problematic for service providers. Finally, SLAAC has a set of privacy extensions outlined in RFC4941 that (theoretically) prevent a host from being tracked based on its IPv6 address over time. This is a very attractive property for edge facing service providers.
The original set of drafts, however, only provided for DNS information to be carried through DHCPv6, and had no failover mechanism for DHCPv6. These two things, together, made it impossible to use just one of these two options. More recent work, however, has remedied both parts of this problem, making either option able to stand on its own. RFC6106, which is a bit older (2010), provides for DNS advertisement in the RA protocol. This allows an operator who would like to run everything completely stateless to do so, including hosts learning which DNS resolver to use. On the other side, RFC8156, which was just ratified in July of 2017, allows a pair of DHCPv6 servers to act as a failover pair. While this is more complex than simple DHCPv6, it does solve the problem of a host failing to operate correctly simply because the DHCPv6 server has failed.
Which of the two is now the best choice? If you do not have any requirement to restrict the hosts that can attach to the network using IPv6, then SLAAC, combined with DNS advertisement in the RA, and possibly with DDNS (if needed), would be the right choice. However, if the environment must be more secure, then DHCPv6 is likely to be the better solution.
A word of warning, though—using DHCPv6 to ensure each host received an IPv6 address that can be used anyplace in the network, and then stretching layer 2 to allow any host to roam “anywhere,” is really just not a good idea. I have worked on networks where this kind of thing has been taken to a global scale. It might seem cute at first, but this kind of solution will ultimately become a monster when it grows up.
Does this mean we ban all filtering of traffic on the public Internet, imposing the end-to-end rule in earnest, leaving all security to the end hosts? This does seem to be the flavor of the original IPv6 discussions around stateful packet filters. This does not, however, seem like the most realistic option available; the stronger defense is not a single perfect wall, but rather a series of less than perfect walls. Defense in depth will beat a single firewall every time. Another alternative is to accept another bit of reality we often forget in the network engineering world: abstractions leak. The end-to-end principle describes a perfectly abstracted system capable of carrying traffic from one host to another, and a perfectly abstracted set of hosts between which traffic is being carried.
The full post can be read over at the ECI blog.
In Don’t Forget to Lock the Back Door! A Characterization of IPv6 Network Security Policy, the authors ran an experiment that tested for open ports in IPv4 and IPv6 across a wide swath of the network. What they discovered was interesting—
IPv6 is more open than IPv4. A given IPv6 port is nearly always more open than the same port is in IPv4. In particular, routers are twice as reachable over IPv6 for SSH, Telnet, SNMP, and BGP. While openness on IPv6 is not as severe for servers, we still find thousands of hosts open that are only open over IPv6.
This result really, on reflection, should not be all that surprising. There are probably thousands of networks in the world with “unintentional” deployments of IPv6. The vendor has shipped new products with IPv6 enabled by default, because one large customer has demanded it. Customers who have not even thought about deploying IPv6, however, end up with an unprotected attack surface.
The obvious solution to this problem is—deploy IPv6 intentionally, including security, and these problems will likely go away.
But the obvious solution, as obvious as it might be, is only one step in the right direction. Instead of just attacking the obvious problem, we should think through the process that caused this situation in the first place, and plug the hole in our thinking. The hole in our thinking is, of course, this:
“More features” is always better, so give me more features.
One of the lessons of the hyperscaler, lessons the rest of the market is just beginning to catch sight of off in the distance, is this more features mantra has led, and is still leading, into dangerous territory. Each feature, no matter how small it might seem, opens some new set of vulnerabilities in the code. Whether the vulnerability is a direct attack vector (such as the example in the paper), or just another interaction surface buried someplace in the vendor’s code, it is still a vulnerability.
In other words, simplicity is not just about the networks we design. Simplicity is not just about the user interface, either. It is integral to every product we buy, from the simplest switch to the most complex “silver bullet” appliance. Hiding complexity inside an appliance does not really make it go away; it just hides it.
You should really go read the paper itself, of course—and you should really deploy IPv6 intentionally in your network, as in yesterday. But you should also not fail to see the larger lesson that can be drawn from such studies. Sometimes it is better to have the complexity on the surface, where you can see and manage it, than it is to bury the complexity in an appliance, user interface, or…
Simplicity needs to be greater than skin deep.
I ran into an interesting paper on the wide variety of options for assigning addresses, and providing DNS information, in IPv6, over at ERNW. As always, with this sort of thing, it started me thinking about the power of unintended consequences, particularly in the world of standardization. The authors of this paper noticed there are a lot of different options available in the realm of assigning addresses, and providing DNS information, through IPv6.
Alongside these various options, there are a number of different flags that are supposed to tell the host which of these options should, and which shouldn’t, be used, prioritized, etc. The problem is, of course, that many of these flags, and many of the options, are, well, optional, which means they may or may not be implemented across different versions of code and vendor products. Hence, combining various flags with various bits of information can have a seemingly random impact on the IPv6 addresses and DNS information different hosts actually use. Perhaps the most illustrative chart is this one—
Each operating system tested seems to act somewhat differently when presented with all possible flags, and all possible sources of information. As the paper notes, this can cause major security holes. For instance, if an attacker simply brings up a DHCPv6 server on your network, and you’re not already using DHCPv6, the attacker can position itself to be a “man in the middle” for most DNS queries.
What lessons can we, as engineers and network operators, take away from this sort of thing?
First, standards bodies aren’t perfect. Standards bodies are, after all, made up of people (and a lot less people than you might imagine), and people are not perfect. Not only are people not perfect, they are often under pressures of various sorts which can lead to “less than optimal” decisions in many situations, particularly in the case of systems designed by a lot of different people, over a long stretch of time, with different pieces and parts designed to solve particular problems (corner or edge cases), and subjected to the many pressures of actually holding a day job.
Second, this means you need to be an engineer, even if you are relying on standards. In other words, don’t fall back to “but the RFC says…” as an excuse. Do the work of researching why “the RFC says,” find out what implementations do, and consider what alternatives might be. Ultimately, if you call yourself an engineer, be one.
Third, always know what is going on, on your network, and always try to account for negative possibilities, rather than just positive ones. I wonder how many times I have said, “but I didn’t deploy x, so I don’t need to think about how x interacts with my environment.” We never stop to ask if not deploying x leaves me open to security holes or failure modes I have not even considered.
Unintended consequences are, after all, unintended, and hence “Out of sight, out of mind.” But out of sight, and even out of mind, definitely does not mean out of danger.