Archive for 2020
The Hedge 27: New directions in network and computing systems
On this episode of the Hedge, Micah Beck joins us to discuss a paper he wrote recently considering a new model of compute, storage, and networking. Micah Beck is Associate Professor in computer science at the University of Tennessee, Knoxville, where he researches and publishes in the area of networking technologies, including the hourglass model and the end-to-end principle.
Enterprise and Service Provider—Once more into the Windmill
There is no enterprise, there is no service provider—there are problems, and there are solutions. I’m certain everyone reading this blog, or listening to my podcasts, or listening to a presentation I’ve given, or following along in some live training or book I’ve created, has heard me say this. I’m also certain almost everyone has heard the objections to my argument—that hyperscaler’s problems are not your problems, the technologies and solutions providers user are fundamentally different than what enterprises require.
Let me try to recap some of the arguments I’ve heard used against my assertion.
The theory that enterprise and service provider networks require completely different technologies and implementations is often grounded in scale. Service provider networks are so large that they simply must use different solutions—solutions that you cannot apply to any network running at a smaller scale.
The problem with this line of thinking is it throws the baby out with the bathwater. Google is using automation to run their network? Well, then… you shouldn’t use automation because Google’s problems are not your problems. Microsoft is deploying 100g Ethernet over fiber? Then clearly enterprise networks should be using Token Ring or ARCnet because… Microsoft’s problems are not your problems.
The usual answer is—“I’m not saying we shouldn’t take good ideas when we see them, but we shouldn’t design networks the way someone else does just because.” I don’t see how this clarifies the solution, though—when is it a good idea or a bad one? What is our criterion to decide what to adopt and what not to adopt? Simply saying “X’s problems aren’t your problems” doesn’t really give me any actionable information—or at least I’m not getting it if it’s buried in there someplace.
Instead—maybe—just maybe—we are looking at this all wrong. Maybe there is some other way classify networks that will help us see the problem set better.
I don’t think networks are undifferentiated—I think the enterprise/service provider/hyerpscaler divide is not helpful to understand how different networks are … different, and how to correctly identify an environment and build to it. Reading a classic paper in software design this week—Programs, Life Cycles, and Laws of Software Evolution—brought all this to mind. In writing this paper, Meir Lehman was facing many of the same classification problems, just in software development rather than in building networks.
Rather than saying “enterprise software is different than service provider software”—an assertion absolutely no-one makes—or even “commercial software is different than private software, and developers working in these two areas cannot use the same tools and techniques,” Lehman posits there are three kinds of software systems. He calls these S-Programs, in which the problem and solution can be fully specified; P-Programs, in which the problem can be fully specified, but the program can only be partially specified because of complexity and scale; and E-Programs, where the program itself become part of the world it models. Lehman thinks most software will move towards S-Program status as time moves on—something that hasn’t happened (the reasons are out of scope for this already-too-long-blog-post).
But the classification is useful. For S-Programs, the inputs and outputs can be fully specified, full-on testing can take place before the software is deployed, and lifecycle management is largely about making the software more fully conform to its original conditions. Maybe there are S-Networks, too? Single-purpose networks which are aimed at fulfilling on well-defined thing, and only that thing. Lehman talks about learning how to breaking larger problems into smaller one so the S-Problems can be dealt with separately—is this anything different than separating out the basic problem of providing IP connectivity in a DC fabric underlay, or even providing basic IP connectivity in a transit or campus network, treating it as a separate module with fairly well design goals and measurements?
Lehman talks about P-Programs, where the problem is largely definable, but the solutions end up being more heuristic. Isn’t this similar to a traffic engineering overlay, where we largely know what the goals are, but we don’t necessarily know what specific solution is going to needed at any moment, and the complete set of solutions is just too large to initially calculate? What about E-Programs, where the software becomes a part of the world it models? Isn’t this like the intent-based stuff we’ve been talking about networking for going one 30 years now?
Looking at it another way, isn’t it possible that some networks are largely just S-Networks? And others are largely E-Networks? And that these classifications have nothing to do with whether the network is being built by what we call an “enterprise” or a “service provider?” Isn’t is possible that S-Networks should probably all use the same basic sort of structure and largely be classified as a “commodity,” while E-Networks will all be snowflakes, and largely classified as having high business importance?
Just like I don’t think the OSI model is particularly helpful in teaching and understanding networks any longer, I don’t find the enterprise/service/hyperscaler model very useful in building and operating networks. The service enterprise/service provider divide tends to artificially limit idea transfer when it wants to be transferred, and artificially “hype up” some networks while degrading others—largely based on perceptions of scale.
Scale != complexity. It’s not about service providers and enterprises. It doesn’t matter if Google’s problems are not your problems; borrowing from the hyperscale is not a “bad thing.” It’s just a “thing.” Think clearly about the problem set, understand the problem set, and borrow liberally. There is no such thing as a “service provider technology,” nor is there any such thing as an “enterprise technology.” There are problems, and there are solutions. To be an engineer is to connect the two.
The Hedge 26: Jason Gooley and CHINOG
CHINOG is a regional network operators group that meets in Chicago once a year. For this episode of the Hedge, Jason Gooley joins us to talk about the origins of CHINOG, the challenges involved in running a small conference, some tips for those who would like to start a conference of this kind, and thoughts on the importance of community in the network engineering world.
The Art and Necessity of Refocusing
Over at his blog The Forwarding Plane, Nick Buraglio posted about embracing change and how technology is mostly unimportant. In the technology-driven world networking folks live in, how can technology be mostly inconsequential? One answer is people drive technology, rather than the other way around—but this misses the real-world consequences of technological adoption on culture. To paraphrase Andy Crouch, technology makes some things possible that were once impossible, some things easy that were once hard, some things hard that were once easy, and some things impossible that were once possible.
There is another answer to this question, though—the real versus the perceived rate of change. When I was a kid, I would ride around with my uncle in his Jeep, a 1968 CJ5 with a soft top and soft doors. He would take the doors off when he took the top down, and—these older Jeeps being much smaller than current models—you could look just to your right and see the road passing by just there under your feet. What always amazed me was I could make myself think I was moving at different speeds just by changing my focus. If I looked across a field at a telephone pole in the distance, it didn’t seem like I was moving all that quickly. If I stared down at the white line on the side of the road, it looked like I was moving very fast indeed. By shifting my focus from here to there, I could adjust my perceived speed.
Here is where the focus on details becomes critical in networking. We do tend to focus on the details. To make matters worse, the average network operator tends to be something of a generalist. Being a generalist focused on details can be a frightening experience.
If you live entirely in the world of Ethernet, then you see past and future changes in the context of the history of Ethernet. This is something like looking at an object a few hundred feet off the road, perhaps. Things are moving quickly, but they aren’t insanely fast, blurry, up close and personal. If you live in wholly in the world of routing protocols, you are going to have a different picture, but the apparent speed is going to be similar, or perhaps even slower.
If you’re a generalist who focuses on detail, though, you’re going to be staring at the white line—at all the features, physical form factors, and products created by a combination of the changes made in routing and Ethernet. If there are two changes in Ethernet, and two in routing, product marketing will create at least four, and probably eight, new features out of the combination of these two, across twenty or thirty product lines. Each of these features will likely be called something different and sold to solve completely different problems.
Staring at the white line is fun at first, then mesmerizing, then it is frightening… then finally it is just plain dull. But let’s talk about the terrifying bit because it’s the scary stage that makes us all reject change out of fear for the future. And, trust me, a kid sitting in a car with no doors staring down at the white line while his uncle drives 60 miles-per-hour is going to be frightened from time to time.
The point Nick is making is we should back off the details and embrace the change. This is great advice—but how? It can feel like you’re going to run off the road if you don’t keep staring at the white line. The answer lies in putting your eyes someplace else—on the posts way out in the field. Ethernet still solves the same problems Bob Metcalf designed it to solve, and it always solves those problems using a small’ish set of solutions. Routing still solves the same problems it did when Dijkstra was mulling around toy problems to show off the processing power of a new computer some 60-odd years ago, and it still solves these problems using a small set of tools.
If you stop looking at the white line and start looking at the poles out in the distance, you’ll not only save your sanity, you’ll also permit yourself to start looking at the sociological and business impacts of new technology, including what matters and what doesn’t.
Two hundred years ago, if you wanted to get from Memphis to say… Lake Providence, Mississippi, you could take a boat directly between the two. Today you would take a car, and the only paths between the two are pretty round-a-about and “small country road” sorts of affairs. On the other hand, getting from Memphis Tennessee to Atlanta, Georgia, is now easy, while a couple of hundred years ago would be a big deal indeed. The sociological changes wrought by moving from rivers to roads are almost impossible to fathom. But you wouldn’t know that if you just stared at the white lines.
The Hedge 25: Building the Next Generation of Network Engineer
If there is one thing I notice when I look around at the IETF—and many other places where I meet a lot of network operations and engineering folk—it’s that we all seem to be getting a bit older. This should lead us to an obvious question—what are we doing about bringing up a new generation of network engineers? David Huberman joins Tom Ammon and I to discuss this interesting question. David i s involved in a number of community-based efforts to train next generation network engineers, some of which he discusses in his excellent article at the APNIC blog.
Whither Cyber-Insurance?
Note: I’m off in the weeds a little this week thinking about cyber-insurance because of a paper that landed in one of my various feeds—while this isn’t something we often think about as network operators, it does impact the overall security of the systems we build.
When you go to the doctor for a yearly checkup, do you think about health or insurance? You probably think about health, but the practice of going to the doctor for regular checkups began because of large life insurance companies in the United States. These companies began using statistical methods to make risk, or to build actuarial tables they could use to set the premiums properly. Originally, life insurance companies relied on the “hunches” of their salesmen, combined with some checking by people in the “back office,” to determine the correct premium. Over time, they developed networks of informers in local communities, such as doctors, lawyers, and even local politicians, who could describe the life of anyone in their area, providing the information the company needed to set premiums correctly.
Over time, however, statistical methods came into play, particularly relying on an initial visit with a doctor. The information these insurance companies gathered, however, gave them insight into what habits increased or decreased longevity—they decided they should use this information to help shape people’s lives so they would live longer, rather than just using it to discover the correct premiums. To gather more information, and to help people live better lives, life insurance companies started encouraging yearly doctor visits, even setting up non-profit organizations to support the doctors who gave these examinations. Thus was born the yearly doctor’s visit, the credit rating agencies, and a host of other things we take for granted in modern life.
You can read about the early history of life insurance and its impact on society in How Our Days Became Numbered.
What does any of this have to do with networks? Only this—we are in much the same position in the cyber-insurance market right now as the life insurance market in the late 1800s through the mid-1900s—insurance agents interview a company and make a “hunch bet” on how much to charge the company for cyber-insurance. Will cyber-insurance ever mature to the same point as life insurance? According to a recent research paper, the answer is “probably not.” Why not?
First, legal restrictions will not allow a solution such as the one imposed by payment processors. Second, there does not seem to be a lot of leverage in cyber-insurance premiums. The cost of increasing security is generally much higher than any possible premium discount, making it cheaper for companies just to pay the additional premium than to improve their security posture. Third, there is no real evidence tying the use of specific products to reductions in security breaches. Instead, network and data security tend to be tied to practices rather than products, making it harder for an insurer to precisely specify what a company can and should to improve their posture.
Finally, the largest problem is measurement. What does it look like for a company to “go to the doctor” regularly? Does this mean regular penetration tests? Standardizing penetration tests is difficult, and it can be far too easy to counter pentests without improving the overall security posture. Like medical care in the “early days,” there is no way to know you have gathered enough information on the population to know if you correctly understand the kinds of things that improve “health”—but there is no way to compel reporting (much less accurate reporting), nor is there any way to compel insurance companies to share the information they have about cyber incidents.
Will cyber-insurance exist as a “separate thing” in the future? The authors largely answer in the negative. The pressures of “race to the bottom,” providing maximal coverage with minimal costs (which they attribute to the structure of the cyber-insurance market), combined with lack of regulatory clarity and inaccurate measurements, will probably end up causing cyber-insurance to “fold into” other kinds of insurance.
Whether this is a positive or negative result is a matter of conjecture—the legacy of yearly doctor’s visits and public health campaigns is not universally “good,” after all.
The Hedge 24: Single Source of Truth
Tim Schreyack recently presented at NANOG on the topic of building a single source of truth for network automation. Tim joins Tom and Russ in a wide-ranging discussion about single sources of truth, changing the way we see the network, and the changing skills of network engineers.