TECH
Technologies that Didn’t: Asynchronous Transfer Mode
One of the common myths of the networking world is there were no “real” networks before the early days of packet-based networks. As myths go, this is not even a very good myth; the world had very large-scale voice and data networks long before distributed routing, before packet-based switching, and before any of the packet protocols such as IP. I participated in replacing a large scale voice and data network, including hundreds of inverse multiplexers that tied a personnel system together in the middle of the 1980’s. I also installed hundreds of terminal emulation cards in Zenith Z100 and Z150 systems in the same time frame to allow these computers to connect to mainframes and newer minicomputers on the campus.
All of these systems were run through circuit-switched networks, which simply means the two end points would set up a circuit over which data would travel before the data actually traveled. Packet switched networks were seen as more efficient at the time because the complexity of setting these circuits up, along with the massive waste of bandwidth because the circuits were always over provisioned and underused.
The problem, at that time, with packet-based networks was the sheer overhead of switching packets. While frames of data could be switched in hardware, packets could not. Each packet could be a different length, and each packet carried an actual destination address, rather than some sort of circuit identifier—a tag. Packet switching, however, was quickly becoming the “go to” technology solution for a lot of problems because of its efficient use of network resources, and simplicity of operation.
Asynchronous Transfer Mode, or ATM, was widely seen as a compromise technology that would provide the best circuit and packet switching in a single technology. Data would be input into the network in the form of either packets or circuits. The data would then be broken up into fixed sized cells, which would then be switched based on a fixed label-based header. This would allow hardware to switch the cells in a way that is like circuit switching, while retaining many of the advantages of a circuit switched network. In fact, ATM allowed for both circuit- and packet-switched paths to be both be used in the same network.
With all this goodness under one technical roof, why didn’t ATM take off? The charts from the usual prognosticators showed markets that were forever “up and to the right.”
The main culprit in the demise of ATM turned out to be the size of the cell. In order to support a good combination of voice and data traffic, the cell size was set to 53 octets. A 48-octet packet, then, should take up a single cell with a little left over. Larger packets, in theory, should be able to be broken into multiple cells and carried over the network with some level of efficiency, as well. The promise of the future was ATM to the desktop, which would solve the cell size overhead problem, since applications would generate streams pre-divided into the correctly sized packets to use the fixed cell size efficiently.
The reality, however, was far different. The small cell size, combined with the large overhead of carrying both a network layer header, the ATM header, and the lower layer data link header, caused ATM to be massively inefficient. Some providers at the time had research showing that while they were filling upwards of 80% of any given link’s bandwidth, the goodput, the amount of data being transmitted over the link, was less than 40% of the available bandwidth. There were problems with out of order cells and reassembly to add on top of this, causing entire packets worth of data, spread across multiple cells, to be discarded. The end was clearly near when articles appeared in popular telecommunications journals comparing ATM to shredding the physical mail, attaching small headers to each resulting chad, and reassembling the original letters at the recipient’s end of the postal system. The same benefits were touted—being able to pack mail trucks more tightly, being able to carry a wider array of goods over a single service, etc.
In the end, ATM to the desktop never materialized, and the inefficiencies of ATM on long-haul links doomed the technology to extinction.
Lessons learned? First, do not count on supporting “a little inefficiency” while the ecosystem catches up to a big new idea. Either the system has immediate, measurable benefits, or it does not. If it does not, it is doomed from the first day of deployment. Second, do not try to solve all the problems in the world at once. Build simple, use it, and then build it better over time. While we all hate being beta testers, sometimes real-world beta testing is the only way to know what the world really wants or needs. Third, up-and-to-the-right charts are easy to justify and draw. They are beautiful and impressive on glossy magazine pages, and in flashy presentations. But they should always be considered carefully. The underlying technology, and how it matches real-world needs, are more important than any amount of forecasting and hype.
The Hedge 54: Bob Friday and AI in Networks
AI in networks is a hotly contested subject—so we asked Bob Friday, CTO of Mist Systems, to explain the value and future of AI in networks. Bob joins Tom Ammon and Russ White for this episode.
The Hedge 53: Deprecating Interdomain ASM
Interdomain Any-source Multicast has proven to be an unscalable solution, and is actually blocking the deployment of other solutions. To move interdomain multicast forward, Lenny Giuliano, Tim Chown, and Toerless Eckert wrote RFC 8815, BCP 229, recommending providers “deprecate the use of Any-Source Multicast (ASM) for interdomain multicast, leaving Source-Specific Multicast (SSM) as the recommended interdomain mode of multicast.”
Technologies that Didn’t: CLNS
Note: RFC1925, rule 11, reminds us that: “Every old idea will be proposed again with a different name and a different presentation, regardless of whether it works.” Understanding the past not only helps us to understand the future, it also helps us to take a more balanced and realistic view of the technologies being created and promoted for current and future use.
The Open Systems Interconnect (OSI) model is the most often taught model of data transmission—although it is not all that useful in terms of describing how modern networks work. What many engineers who have come into network engineering more recently do not know is there was an entire protocol suite that went with the OSI model. Each of the layers within the OSI model, in fact, had multiple protocols specified to fill the functions of that layer. For instance, X.25, while older than the OSI model, was adopted into the OSI suite to provide point-to-point connectivity over some specific kinds of physical circuits. Moving up the stack a little, there were several protocols that provided much the same service as the widely used Internet Protocol (IP).
The Connection Oriented Network Service, or CONS, ran on top of the Connection Oriented Network Protocol, or CONP (which is X.25). Using CONP and CONS, a pair of hosts can set up a virtual circuit. Perhaps the closest analogy available in the world of networks today would be an MPLS Label Switched Path (LSP). Another protocol, the Connectionless Network Service, or CLNS, ran on top of the Connectionless Network Protocol, or CLNP. A series of Transport Protocols ran on top of CLNS (these might also be described as modes of CLNS in some sense). Together, CLNS and these transport protocols provided a set of services similar to IP, the Transmission Control Protocol (TCP), and the User Datagram Protocol (UDP)—or perhaps even closer to something like QUIC.
The routing protocol that held the network together by discovering the network topology, carrying reachability, and calculating loop free paths was the venerable Intermediate System to Intermediate System (IS-IS) protocol, which is still in wide use today. The OSI protocol suite had some interesting characteristics.
For instance, each host had a single address, rather than each interface. The host address was calculated automatically based on a local Media Access Control (MAC) address, combined with other bits of information that were either learned, assumed, or locally configured. A single host could have multiple addresses, using each one to communicate to the different networks it might be connected to, or even being used to contact hosts in different domains. The Intermediate System (IS) played a vital role in the process of address calculation and routing; essentially routing ran all the way down to the host level, which allowed smart calculation of next hops. While a default router could be configured in some implementations, it was not required, as the hosts participated in routing.
A tidbit of interesting history—one of the earliest uses of address families in protocols, and one of the main drivers of the Type-Length-Vector encoding of the IS-IS routing protocol, were driven by the desire to run CLNS and TCP/IP side-by-side on a single network. ISO-IGRP was one of the first multi-protocol routing protocols to use a concept similar to address families. Some of the initial work in BGP address families was also done to support CLNS routing, as the OSI stack did not specify an interdomain routing protocol.
Why is this interesting protocol not in wide use today? There are many reasons—probably every engineer who ever worked on both can give you a different set of reasons, in fact. From my perspective, however, there are a few basic reasons why TCP/IP “won” over the CLNS suite of protocols.
First, the addressing used with CLNS was too complex. There were spots in the address for the assigning authority, the company, subdivisions within the company, flooding domains, and other elements. While this was all very neat, it assumed either one of two things: that network topologies would be built roughly parallel to the organizational chart (starting from governmental entities), or that the topology of the network and addressing did not need to be related to one another. This flies in the face of Yaakov’s rule: either the topology must follow the addressing, or the addressing must follow the topology, if you expect the network to scale.
In the later years of CLNS, this entire mess was replaced by people just using the “private” organizational information to build internal networks, and assuming these networks would never be interconnected (much like using private IPv4 address space). Sometimes this worked, of course. Sometimes it did not.
This addressing scheme left users with the impression, true or false, that CLNS implementations had to be carefully planned. Numbers must be procured, the organizational structure must be consulted, etc. IP, on the other hand, was something you could throw out there and play with. You could get it all working, and plan later, or change you plans. Well, in theory, at least—things never seem to work out in real life the way they are supposed to.
Second, the protocol stack itself was too complex. Rather than solving a series of small problems, and fitting the solutions in a sort of ad-hoc layered set of protocols, the designers of CLNS carefully thought through every possible problem, and considered every possible solution. At least they thought they did, anyway. All this thinking, however, left the impression, again, that to deploy this protocol stack you had to carefully think about what you were about. Further, it was difficult to change things in the protocol stack when problems were found, or new use cases—that had not been thought of—were discovered.
It is better, most of the time, to build small, compact units that fit together, than it is to build a detailed grand architecture. The network engineering world tends to oscillate between the two extremes of no planning or all planned; rarely do we get the temperature of the soup (or porridge) just right.
While CLNS is not around today, it is hard to call the protocol stack a failure. CLNS was widely deployed in large scale electrical networks; some of these networks may well be in use today. Further, its effects are everywhere. For instance, IS-IS is still a widely deployed protocol. In fact, because of its heritage of multiprotocol design from the beginning, it is arguably one of the easiest link state protocols to work with. Another example is the multiprotocol work that has carried over into other protocols, such as BGP. The ideas of a protocol running between the host and the router and autoconfiguration of addresses, have come back in very similar forms in IPv6, as well.
CLNS is another one of those designs that shape the thinking of network engineers today, even if network engineers don’t even know it existed.
Reducing RPKI Single Point of Takedown Risk
The RPKI, for those who do not know, ties the origin AS to a prefix using a certificate (the Route Origin Authorization, or ROA) signed by a third party. The third party, in this case, is validating that the AS in the ROA is authorized to advertise the destination prefix in the ROA—if ROA’s were self-signed, the security would be no better than simply advertising the prefix in BGP. Who should be able to sign these ROAs? The assigning authority makes the most sense—the Regional Internet Registries (RIRs), since they (should) know which company owns which set of AS numbers and prefixes.
The general idea makes sense—you should not accept routes from “just anyone,” as they might be advertising the route for any number of reasons. An operator could advertise routes to source spam or phishing emails, or some government agency might advertise a route to redirect traffic, or block access to some web site. But … if you haven’t found the tradeoffs, you haven’t looked hard enough. Security, in particular, is replete with tradeoffs.
Every time you deploy some new security mechanism, you create some new attack surface—sometimes more than one. Deploy a stateful packet filter to protect a server, and the device itself becomes a target of attack, including buffer overflows, phishing attacks to gain access to the device as a launch-point into the private network, and the holes you have to punch in the filters to allow services to work. What about the RPKI?
When the RKI was first proposed, one of my various concerns was the creation of new attack services. One specific attack surface is the control a single organization—the issuing RIR—has over the very existence of the operator. Suppose you start a new content provider. To get the new service up and running, you sign a contract with an RIR for some address space, sign a contract with some upstream provider (or providers), set up your servers and service, and start advertising routes. For whatever reason, your service goes viral, netting millions of users in a short span of time.
Now assume the RIR receives a complaint against your service for whatever reason—the reason for the complaint is not important. This places the RIR in the position of a prosecutor, defense attorney, and judge—the RIR must somehow figure out whether or not the charges are true, figure out whether or not taking action on the charges is warranted, and then take the action they’ve settled on.
In the case of a government agency (or a large criminal organization) making the complaint, there is probably going to be little the RIR can do other than simply revoke your certificate, pulling your service off-line.
Overnight your business is gone. You can drag the case through the court system, of course, but this can take years. In the meantime, you are losing users, other services are imitating what you built, and you have no money to pay the legal fees.
A true story—without the names. I once knew a man who worked for a satellite provider, let’s call them SATA. Now, SATA’s leadership decided they had no expertise in accounts receivables, and they were spending too much time on trying to collect overdue bills, so they outsourced the process. SATB, a competing service, decided to buy the firm SATA outsourced their accounts receivables to. You can imagine what happens next… The accounting firm worked as hard as it could to reduce the revenue SATA was receiving.
Of course, SATA sued the accounting firm, but before the case could make it to court, SATA ran out of money, laid off all their people, and shut their service down. SATA essentially went out of business. They won some money later, in court, but … whatever money they won was just given to the investors of various kinds to make up for losses. The business itself was gone, permanently.
Herein lies the danger of giving a single entity like an RIR, even if they are friendly, honest, etc., control over a critical resource.
A recent paper presented at the ANRW at APNIC caught my attention as a potential way to solve this problem. The idea is simple—just allow (or even require) multiple signatures on a ROA. To be more accurate, each authorizing party issues a “partial certificate;” if “enough” pieces of the certificate are found and valid, the route will be validated.
The question is—how many signatures (or parts of the signature, or partial attestations) should be enough? The authors of the paper suggest there should be a “Threshold Signature Module” that makes this decision. The attestations of the various signers are combined in the threshold module to produce a single signature that is then used to validate the route. This way the validation process on the router remains the same, which means the only real change in the overall RPKI system is the addition of the threshold module.
If one RIR—even the one that allocated the addresses you are using—revokes their attestation on your ROA, the remaining attestations should be enough to convince anyone receiving your route that it is still valid. Since there are five regions, you have at least five different choices to countersign your ROA. Each RIR is under the control of a different national government; hence organizations like governments (or criminals!) would need to work across multiple RIRs and through other government organizations to have a ROA completely revoked.
An alternate solutions here, one that follows the PGP model, might be to simply have the threshold signature model consider the number and source of ROAs using the existing model. Local policy could determine how to weight attestations from different RIRs, etc.
This multiple or “shared” attestation (or signature) idea seems like a neat way to work around one of (possibly the major) attack surfaces introduced by the RPKI system. If you are interested in Internet core routing security, you should take a read through the post linked above, and then watch the video.
The Hedge 51: Tim Fiola and pyNTM
Have you ever looked at your wide area network and wondered … what would the traffic flows look like if this link or that router failed? Traffic modeling of this kind is widely available in commercial tools, which means it’s been hard to play with these kinds of tools, learn how they work, and understand how they can be effective. There is, however, an open source alternative—pyNTM.
Technologies that Didn’t: The Fiber Distributed Data Interface
Token Ring, in its original form, was—on paper—a very capable physical transport. For instance, because of the token passing capabilities, it could make use of more than 90% of the available bandwidth. In contrast, Ethernet systems, particularly early Ethernet systems using a true “single wire” broadcast domain, cannot achieve nearly that kind of utilization—hence “every device on its own switch port.” The Fiber Distributed Data Interface, or FDDI, is like Token Ring in many ways. For instance, FDDI uses a token to determine when a station connected to the ring can transmit, enabling efficient use of bandwidth.
And yet, Ethernet is the common carrier of almost all wired networks today, and even wireless standards mimic Ethernet’s characteristics. What happened to FDDI?
FDDI is a 100Mbit/second optical standard based on token bus, which used an improved timed token version of the token passing scheme developed for token ring networks. This physical layer protocol standard had a number of fairly unique features. It was a 100Mbit/second standard at a time when Ethernet offered a top speed of 10Mbits/second. Since FDDI could operate over a single mode fiber, it could support distances of 200 kilometers, or around 120 miles. Because it was based on a token passing system, FDDI could also support larger frame sizes; while Ethernet was supporting 1500 octet frames, FDDI could support 4352 octet frames.
FDDI provided three different ways to connect a station to the ring. A Single Attached Station (SAS) used one ring as the primary and the second as a backup. A Dual Attached Station (DAS) connects to both rings, which allows a DAS to send and receive at 200mbits/second. A concentrator is dual attached, as well. FDDI can react to a failure of a node by optically bypassing the failed node, and can react to a failed link in the ring by looping all traffic onto the second ring. Error detection is handled by the loss of a token so even the failure of an optically bypassed node, either for failure or just for faster switching, can be easily detected.
If this is your first encounter with FDDI, you might be wondering: how did Ethernet win against this technology? It seems, on first examination, like the legendary war between VHS and Betamax.
The primary reason given for FDDI’s loss is the cost of the hardware. Once Ethernet reached 100mbit/second speeds, the lower cost of Ethernet simply made the choice between the two obvious. This is a simple answer, but as with almost all technology stories, it is too simple.
Consider all the terrific features FDDI has natively. It takes much of the load of error detection and correction off upper layer protocols—but upper layer protocols still need error correction in the end-to-end path. FDDI has the ability to wrap onto the second ring for a SAS, but this means maintaining two optical systems, including the fiber, transceiver, CMOS, and ASIC ports. Half of these resources will only be rarely used. To make matter worse, the fibers were often buried together, and terminate in the same physical hardware, so they share fate in many regards. The chipsets required to provide optical bypass, the ability to wrap, and all the detection and other mechanisms included are very expensive to produce, and the process of building these higher end features into software and hardware are prone to defects and problems. So, yes—FDDI was more expensive than Ethernet. This is primarily because it tried to bury a lot of functionality in a single layer, using advanced optics and silicon.
The story of FDDI, then, is that it was introduced at just the point where Ethernet was gaining momentum, largely in an attempt to create a better token passing physical layer that would use links more efficiently. Laying fiber, even in a campus, is expensive; it just seemed better to make the best use of the available fiber by transferring costs to the hardware rather than laying more. As the market shifted, and laying fiber became less expensive, the return on investment in all that fancy silicon and software fell until Ethernet had a similar ROI.
A bottom line lesson network engineers can take from this? Solving hard problems requires highly complex systems. Pushing the complexity into a single layer might seem simpler at first, but the costs in that single layer will multiply until the ROI makes the solution unattractive. The fruits of complexity sell, but the costs can be overwhelming.
This is a repost from the ECI blog, which is apparently being taken down.