Shows in left side column — all but worth reading should be in this category

On the ‘net: The Tradeoffs of Information Hiding

I recently joined Ethan Banks for a Packet Pushers episode around the trade offs of hiding information in the control plane. This was a terrific show; you can listen to it by clicking on the link below.

Today on the Priority Queue, we’re gonna hide some information. Oh, like route summarization? Sure, like route summarization. That’s an example of information hiding. But there’s much more to the story than that. Our guest is Russ White. Russ is a serial networking book author, network architect, RFC writer, patent holder, technical instructor, and much of the motive force behind the early iterations of the CCDE program.

Weekend Reads 071318: Nice to Haves

I had about four hours of highway driving yesterday. Even though I probably could’ve navigated it on my own, I opted to use Apple Maps, which is integrated with my car’s Apple CarPlay “infotainment center.” It was nice. It told me how many miles I had remaining and my expected time of arrival. But it wasn’t a life changer. @The Old Reader

More than ever before Internet users are now interacting with people living/working in other economies. And as a result of these interactions, there are an increasing number of ‘legal contracts’ (intentional or not). Internet policy researchers and academics debate about the changing landscape and the boundaries of the international and domestic laws, without conclusive agreements. —Yeseul Kim @APNIC

The plague that is Spectre continues to evolve and adapt, showing up in two new variants this week dubbed Spectre 1.1 and Spectre 1.2 that follow the original Spectre’s playbook while expanding on the ways they can do damage. —Curtis Franklin Jr. @Dark Reading

These vast routing events that are propagated globally already provide a hint that some ISPs do not set filters at all, or there are vastly malformed AS-SETs. We decided to measure the number of filters that were already bypassed by routing anomalies. To do so, we checked the way route leaks were propagated: if a route leak is received from a customer link and it does not belong to the customer cone then IRR filters were malformed. —Alexander Azimov @APNIC

Recently, a CEO of a roaring unicorn in Silicon Valley drew my attention to the following: “If you compare Amazon’s stock price over the recent years against the cost of housing and the rise of homelessness in Seattle, the progression is identical…” —Frederic Filloux @MondayNote

Why do many problems in life seem to stubbornly stick around, no matter how hard people work to fix them? It turns out that a quirk in the way human brains process information means that when something becomes rare, we sometimes see it in more places than ever. —David Levari @The Conversation

Two web-based attacks against IoT devices made the rounds this week. Researchers Craig Young and Brannon Dorsey showed that a well known attack technique called “DNS rebinding” can be used to control your smart thermostat, detect your home address or extract unique identifiers from your IoT devices. —Gunes Acar

Reaction: Some Sayings that Sum Up Networking

Over at the CIMI blog, Tom Nolle has a mixed bag of sayings and thoughts about the computer networking world, in particular how it relates to the media. Some of these were interesting enough that they seemed worth highlighting and writing a bit more on.

“News” means “novelty”, not “truth”. In much of the computer networking world, news is what sells products, rather than business need. In turn, Novelty is what drives the news. The “straight line” connection, then is from novelty to news to product, and product manufacturers know this. This is not just a vendor driven problem, however; this is also driven by recruitment, and padding resumes, and many other facets of the networking nerd culture.

On the other hand, novelty is never a good starting place for network design. Rather, network design needs to start with problems that need to be solved, proceeds by considering how those problems can be solved with technologies, then builds requirements based on the problems and technologies, and finally considers which products can be used to implement all of this at the lowest long term cost. This is not to say novelty is not useful, or is not justified, but rather that novelty is not the point.

How can you overcome the drive to novelty through the news cycle? Go back to basics. Every “novel” thing you are looking at in the latest news story is something that has been invented and implemented before in a different package, and with a different name. Apply rule 11 liberally to all marketing claims, look for the problem to be solved, push back on the requirements, think systemically, manage your own expectations, and go back to basics.

To a user, “the network” is whatever isn’t on their desk or in their device. This is a point folks who work on the network for a living often forget. Talking to a non-networking person about networking technology is often like talking to someone who commutes on the train about how the train works; it might be interesting, but they often just do not care. There are several implications here: the first is that if your business relies on the network (and most do, whether or not they realize it), as the network engineer, you need to go beyond just making the train work, to helping others understand that why and how the network (the train) runs is important to reaching the overall business goals. There is an entire movement within the networking world that would say: “networks are a commodity, just like the train is, just move the packets and shut up.” I do not tend to agree with this; for a city, a train is not a commodity, it is a vital resource that grows business and interacts with people’s lives. The network is like the train to a city; it might be a commodity for the person riding it, but it is not for the overall business.

There’s no substitute for knowing what you’re doing. But what does it mean to “know what you are doing?” In a large complex system, you can know what is on “your layer,” or “your piece of the system,” plus one or two levels above and below. The rest is rumor and pop psychology.

In a world where there is just too much information, how can you “know what you are doing?” First, you can use rule 11 to your advantage, and realize that everything that is, has been before. If you know the underlying technology, then the implementation is much easier to learn (if you need to learn it at all!). If you know the pattern, then you can see the details much more easily. Second, you can insist on radical simplicity, which will make the process of knowing the entire system much easier. Third, you can intentionally think systematically, and functionally, rather than orienting yourself to products.

Recent BGP Peering Enhancements

BGP is one of the foundational protocols that make the Internet “go;” as such, it is a complex intertwined system of different kinds of functionality bundled into a single set of TLVs, attributes, and other functionality. Because it is so widely used, however, BGP tends to gain new capabilities on a regular basis, making the Interdomain Routing (IDR) working group in the Internet Engineering Task Force (IETF) one of the consistently busiest, and hence one of the hardest to keep up with. In this post, I’m going to spend a little time talking about one area in which a lot of work has been taking place, the building and maintenance of peering relationships between BGP speakers.

The first draft to consider is Mitigating the Negative Impact of Maintenance through BGP Session Culling, which is a draft in an operations working group, rather than the IDR working group, and does not make any changes to the operation of BGP. Rather, this draft considers how BGP sessions should be torn down so traffic is properly drained, and the peering shutdown has the minimal effect possible. The normal way of shutting down a link for maintenance would be to for administrators to shut down BGP on the link, wait for traffic to subside, and then take the link down for maintenance. However, many operators simply do not have the time or capability to undertake scheduled shutdowns of BGP speakers. To resolve this problem, graceful shutdown capability was added to BGP in RFC8326. Not all implementations support graceful shutdown, however, so this draft suggests an alternate way to shut down BGP sessions, allowing traffic to drain, before a link is shut down: use link local filtering to block BGP traffic on the link, which will cause any existing BGP sessions to fail. Once these sessions have failed, traffic will drain off the link, allowing it to be safely shut down for maintenance. The draft discusses various timing issues in using this technique to reduce the impact of link removal due to maintenance (or other reasons).

Graceful shutdown, itself, is also in line to receive some new capabilities through Extended BGP Administrative Shutdown Communication. This draft is rather short, as it simply allows an operator to send a short freeform message (presumably in text format) along with the standard BGP graceful shutdown notification. This message can be printed on the console, or saved to syslog, to provide an operator with more information about why a particular BGP has been shut down, whether it coming back up again, how long the shutdown is expected to last, etc.

Graceful Restart (GR) is a long available feature in many BGP implementations that aims to prevent the disruption of traffic flow; the original purpose was to handle a route processor restart in a router where the line cards could continue forwarding traffic based on local forwarding tables (the FIB), including cases where one route processor fails, causing the router switches to a backup route processor in the same chassis. Over time, GR began to be applied to NOTIFICATION messages in BGP. For instance, if a BGP speaker receives a malformed message, it is required (by the BGP RFCs) to send a NOTIFICATION, which will cause the BGP session to be torn down and restarted. GR has been adapted to these situations, so traffic flow is either not impacted, or minimally impacted through the NOTIFICATION/session restart process. This same processing takes place for a hold timer timeout in BGP.

The problem is that only one of the two speakers in a restarting pair will normally retain its local forwarding information. The sending speaker will normally flush its local routing tables, and with them its local forwarding tables, on sending a BGP NOTIFICATION. Notification Message support for BGP Graceful Restart changes this processing, allowing both speakers to enter the “receiving speaker” mode, so both speakers would retain their local forwarding information. A signal is provided to allow the sending speaker to indicate the sessions should be hard reset, rather than gracefully reset, if needed.

Finally, BGP allows speakers to send a route with a next hop other than themselves; this is called a third party next hop, and is illustrated in the figure below.

In this network, router C’s best path to 2001:db8:3e8:100::/64 might be through A, but the operator may prefer this traffic pass through B. While it is possible to change the preferences so C chooses the path through B, there are some situations where it is better for A to advertise C as the next hop towards the destination (for instance, a route server would not normally advertise itself as the nexthop towards a destination). The problem with this situation is that B might not have the same capabilities as a BGP speaker as A. If B, for instance, cannot forward for IPv6, the situation shown in the illustration would clearly not work.

To resolve this, BGP Next-Hop dependent capabilities allows a speaker to advertise the capabilities of these alternate next hops to peered BGP speakers.

Complexity Sells

According to Roman philosophers, simplicity is the hallmark of truth. And yet, networks have become ever more complex over time. Why is this? Because complexity sells. In this short take, I talk about why complexity sells, and some of the mental habits you can use to overcome our natural tendency to prefer the complex.

Weekend Reads 070618

Our security analysis of the mobile communication standard LTE ( Long-Term Evolution, also know as 4G) on the data link layer (so called layer two) has uncovered three novel attack vectors that enable different attacks against the protocol. —David Rupprecht, Katharina Kohls, Thorsten Holz, and Christina Pöpper

To be fair, the tech sector has been the United States’ economic pride and joy in recent decades, a seemingly endless wellspring of innovation. The speed and power of Google’s search engine is breathtaking, putting extraordinary knowledge at our fingertips. Internet telephony allows friends, relatives, and co-workers to interact face to face from halfway around the world, at very modest cost. Yet, despite all this innovation, the pace of productivity growth in the broader economy remains lackluster. —Kenneth Rogoth @MarketWatch

The world of scholarly communication is broken. Giant, corporate publishers with racketeering business practices and profit margins that exceed Apple’s treat life-saving research as a private commodity to be sold at exorbitant profits. Only around 25 per cent of the global corpus of research knowledge is ‘open access’, or accessible to the public for free and without subscription, which is a real impediment to resolving major problems, such as the United Nations’ Sustainable Development Goals. —John Tennant @Intellectual Takeout

It’s become increasingly impossible to talk about spectrum policy without getting into the fight over whether 5G is a miracle technology that will end poverty, war and disease or an evil marketing scam by wireless carriers to extort concessions in exchange for magic beans. @Wetmachine

Research: P Fat Trees

Link speeds in data center fabrics continue to climb, with 10g, 25g, 40g, and 100g widely available, and 400g promised in just a few short years. What isn’t so obvious is how these higher speeds are being reached. A 100g link, for instance, is really four 25g links bundled as a single link at the physical layer. If the optics are increasing in speed, and the processors are increasing in their ability to switch traffic, why are these higher speed links being built in this way? According to the paper under investigation today, the reason is the speed of the chips that serialize traffic from and deserialize traffic off the optical medium. The development of the Complementary metal–oxide–semiconductor, of CMOS, chips required to build ever faster optical interfaces seems to have stalled out at around 25g, which means faster speeds must be achieved by bundling multiple lower speed links.

Mellette, William M., Alex C. Snoeren, and George Porter. “P-FatTree: A Multi-Channel Datacenter Network Topology.” In Proceedings of the 15th ACM Workshop on Hot Topics in Networks, 78–84. HotNets ’16. New York, NY, USA: ACM, 2016. https://doi.org/10.1145/3005745.3005746.

The authors then point out that many data operators have moved towards some form of chassis device in order to reduce the costs of cabling and optics. Chassis devices most often use some form of spine and leaf internally to switch traffic between the input and output ports across a short run copper fabric, resulting in a switching path within the chassis router that looks something like the following figure.

The spine and leaf in connecting the switching ASICs are one of the main reasons data center operators move away from chassis devices; the number of hops through the network becomes unstable with the addition of these internal spine and leaf fabrics, the backpressure and quality of service is essentially unmanageable across this fabric on most devices, and there is little in the way of traffic analysis that can be done on this internal fabric. The authors do not address these problems, however.

Rather, they address the added set of switching ASICs in the spine layer of the internal spine and leaf network. As it turns out, the switching ASICs themselves are a major consumer of power, and heat generator, in switches. They argue that removing this internal spine layer would greatly reduce the amount of power required in a fabric, as well as the amount of heat generated.
To do this, they propose unbundling the links attached to each SerDes CMOS chip, exposing them as individual links to the control plane. This would allow the switching path to be shortened to something like the figure below.

Exposing the unbundled links to the external control plane allows each stage of the internal fabric to be treated as another hop in the network, and hence for “normal” ECMP to choose the path through the chassis fabric.

The authors suggest the four unbundled links attached to a single switching ASIC can be treated as a member of a different “switching plane,” which, in effect, creates four virtual topologies across the fabric, each of which is one quarter the speed of the total fabric bandwidth. Each virtual topology could run its own control plane, producing four somewhat redundant networks, and the ability to steer traffic onto any given plane at the edge of the network for traffic engineering, policy separation, or any other purpose. The result is a fabric that is more flexible in use, while retaining a fixed hop count through the fabric, and reducing the ASIC count in the fabric by around one third.

This is an interesting concept, but it would require an entire fabric to be built this way from the ground up; there is little chance of a brown field deployment of this kind of design. One tradeoff in this kind of design would be the additional control plane state, including assigning four addresses to each host (although this might be mitigated by the clever use of anycast), and the maintenance of four control planes, etc. Another design tradeoff would be the shared risk link groups involved in splitting a single optical fiber and ASIC into four circuits—these aren’t exactly “virtual circuits,” but they share many of the same characteristics.

Weekend Reads 062918

The Internet and related digital systems that the United States did so much to create have effectuated and symbolized US military, economic, and cultural power for decades. The question raised by this essay is whether these systems, like the Roman Empire’s roads, will come to be seen as a platform that accelerated US decline. @The Hoover Institute

Article 13 reverses one of the key legal doctrines that allowed the Internet to thrive: the idea that computer networks are not “publishers” and are therefore not liable for the actions or statements of their users. This means that you can sue an individual user for libel or copyright infringement, but not the e-mail service or bulletin board or social media platform on which he did it. This immunity made it possible for computer networks to open a floodgate of content produced by independent individuals, without requiring service providers to serve as editors or moderators. —Robert Tracinski @The Federalist

The U.S. Supreme Court today ruled that the government needs to obtain a court-ordered warrant to gather location data on mobile device users. The decision is a major development for privacy rights, but experts say it may have limited bearing on the selling of real-time customer location data by the wireless carriers to third-party companies. —Krebs on Security

The need for an access model for non-public Whois data has been apparent since GDPR became a major issue before the community well over a year ago. Now is the time to address it seriously, and not with half measures. We urgently need a temporary model for access to non-public Whois data for legitimate uses, while the community undertakes longer-term policy development efforts. —Fabricio Vayra @CircleID

More and more companies, government agencies, educational institutions, and philanthropic organizations are today in the grip of a new phenomenon. I’ve termed it “metric fixation.” The key components of metric fixation are the belief that it is possible–and desirable–to replace professional judgment (acquired through personal experience and talent) with numerical indicators of comparative performance based upon standardized data (metrics); and that the best way to motivate people within these organizations is by attaching rewards and penalties to their measured performance. —Jerry Muller @Fast Company

I wish 5G, with its 490 Mbit/sec. speeds and download latency times of 17 milliseconds, was just around the corner. It’s not. I know, I know. AT&T Mobility, Verizon Wireless, and the pairing of T-Mobile and Sprint are all promising 5G real soon now. They’re … fibbing. —Steven J. Vaughan-Nichols @IT World

Digital collaboration technologies are accelerating productivity in the post-phone-call workplace, but tools like Yammer, Workplace by Facebook, and Slack have their dark side. While these channels can help speed group decision-making, they also serve as an enterprise blind spot for insider threats to do their worst – not to mention being open conduits for spreading negativity and toxic behaviors among the ranks. —Ericka Chickowski @Dark Reading

We have reached a point in the evolution of cyber security where handsoff, behind-the-scenes cyber defense should be the norm. Clearly, the best solution would be to deploy less-vulnerable systems. This is a topic that has received great attention for approximately five decades, but developers continue to resist using tools and techniques that have been shown to be effective, such as code minimization, employing formal development methods, and using type-safe languages. —Josiah Dykstra, Eugene H. Spafford @ACM

On the ‘web: Considerations in Network Complexity

One of my articles was published in the most recent Internet Protocol Journal:

Computer networks are complex—and getting more complex by the day. At one time, knowing the Internet Protocol (IP) was enough; today there are underlays, overlays, virtualized services, service chains, and a host of other technologies engineers need to plan around and for. With complexity on the rise, maybe it’s time to ask some fundamental questions, such as—what does complexity mean? Can complexity be solved? How can engineers manage complexity? @The Internet Protocol Journal

If you don’t subscribe to or read IPJ, you should—it’s a great source of information written by industry leaders and thinkers.

Weekend Reads 062218: Bitcoin and Security Flaws

But, as with all new technology, security risks can be found beneath the hype. Indeed, threat actors are finding new targets amid the rise of blockchain as they serve up social-engineering attacks, malware, and exploits to businesses and consumers, according to a recently published report by McAfee’s Advanced Threat Research Team. —Kelly Sheridan @Dark Reading

After last Thursday’s regulatory news that bitcoin and ether were not considered securities, which was widely considered a win for crypto enthusiasts, digital currencies had failed to hold on to gains, a sign the bear market is firmly intact. However, investors were hoping Monday’s move above the post-SEC-statement high will be the start of a more fruitful period for digital currency owners. —Aaron Hankin @MarketWatch

Dubbed Lazy FP State Restore, the vulnerability (CVE-2018-3665) within Intel Core and Xeon processors has just been confirmed by Intel, and vendors are now rushing to roll out security updates in order to fix the flaw and keep their customers protected. The company has not yet released technical details about the vulnerability, but since the vulnerability resides in the CPU, the flaw affects all devices running Intel Core-based microprocessors regardless of the installed operating systems, except some modern versions of Windows and Linux distributions. —Mohit Kumar @The Hacker News

GnuPG, Enigmail, GPGTools and potentially other applications using GnuPG can be attacked with in-band signaling similar to phreaking phone lines in the 1970s (“Cap’n Crunch”). We demonstrate this by creating messages that appear to be signed by arbitrary keys. —Marcus Brinkmann @neopg

Earlier this month the New York Times reported that Facebook had provided highly privileged access to the social network’s platform to more than 60 device makers to allow them to build their own “Facebook experiences” in the era prior to smartphone apps became popular and that this access continued at least in part through earlier this year. Facebook pushed back on the report, arguing that the device makers were acting as extensions of itself, rather than as third parties. Making matters worse, one of those partners has been flagged by the US intelligence community as a national security threat. What can we learn from this latest revelation about Facebook’s approach to user privacy and security? —Kalev Leetaru @Forbes

The number of Resource Certificates and Route Origin Authorizations (ROAs) is steadily growing. However, it remains unclear how widely BGP speakers on the Internet are actually using Route Origin Validation (ROV) to drop or de-preference invalid announcements. —Andreas Reuter @APNIC

Apple Inc. was fined in Australia for refusing to offer free fixes for iPhones and iPads that were previously serviced by non-Apple stores, the latest episode in a global dispute between companies and consumers about the right to repair. —Mike Cherney @MarketWatch

Netflix Open Connect is our purpose-built Content Delivery Network (CDN) responsible for serving 100% of our video traffic. Close to 95% of our traffic globally is delivered via direct connections between Open Connect and the residential ISPs our members use to access the Internet. Most of these connections are localized to the regional point of interconnection geographically closest to the member watching. —Nihit Tandon @APNIC