CONTENT TYPE
The Hedge 26: Jason Gooley and CHINOG
CHINOG is a regional network operators group that meets in Chicago once a year. For this episode of the Hedge, Jason Gooley joins us to talk about the origins of CHINOG, the challenges involved in running a small conference, some tips for those who would like to start a conference of this kind, and thoughts on the importance of community in the network engineering world.
The Art and Necessity of Refocusing
Over at his blog The Forwarding Plane, Nick Buraglio posted about embracing change and how technology is mostly unimportant. In the technology-driven world networking folks live in, how can technology be mostly inconsequential? One answer is people drive technology, rather than the other way around—but this misses the real-world consequences of technological adoption on culture. To paraphrase Andy Crouch, technology makes some things possible that were once impossible, some things easy that were once hard, some things hard that were once easy, and some things impossible that were once possible.
There is another answer to this question, though—the real versus the perceived rate of change. When I was a kid, I would ride around with my uncle in his Jeep, a 1968 CJ5 with a soft top and soft doors. He would take the doors off when he took the top down, and—these older Jeeps being much smaller than current models—you could look just to your right and see the road passing by just there under your feet. What always amazed me was I could make myself think I was moving at different speeds just by changing my focus. If I looked across a field at a telephone pole in the distance, it didn’t seem like I was moving all that quickly. If I stared down at the white line on the side of the road, it looked like I was moving very fast indeed. By shifting my focus from here to there, I could adjust my perceived speed.
Here is where the focus on details becomes critical in networking. We do tend to focus on the details. To make matters worse, the average network operator tends to be something of a generalist. Being a generalist focused on details can be a frightening experience.
If you live entirely in the world of Ethernet, then you see past and future changes in the context of the history of Ethernet. This is something like looking at an object a few hundred feet off the road, perhaps. Things are moving quickly, but they aren’t insanely fast, blurry, up close and personal. If you live in wholly in the world of routing protocols, you are going to have a different picture, but the apparent speed is going to be similar, or perhaps even slower.
If you’re a generalist who focuses on detail, though, you’re going to be staring at the white line—at all the features, physical form factors, and products created by a combination of the changes made in routing and Ethernet. If there are two changes in Ethernet, and two in routing, product marketing will create at least four, and probably eight, new features out of the combination of these two, across twenty or thirty product lines. Each of these features will likely be called something different and sold to solve completely different problems.
Staring at the white line is fun at first, then mesmerizing, then it is frightening… then finally it is just plain dull. But let’s talk about the terrifying bit because it’s the scary stage that makes us all reject change out of fear for the future. And, trust me, a kid sitting in a car with no doors staring down at the white line while his uncle drives 60 miles-per-hour is going to be frightened from time to time.
The point Nick is making is we should back off the details and embrace the change. This is great advice—but how? It can feel like you’re going to run off the road if you don’t keep staring at the white line. The answer lies in putting your eyes someplace else—on the posts way out in the field. Ethernet still solves the same problems Bob Metcalf designed it to solve, and it always solves those problems using a small’ish set of solutions. Routing still solves the same problems it did when Dijkstra was mulling around toy problems to show off the processing power of a new computer some 60-odd years ago, and it still solves these problems using a small set of tools.
If you stop looking at the white line and start looking at the poles out in the distance, you’ll not only save your sanity, you’ll also permit yourself to start looking at the sociological and business impacts of new technology, including what matters and what doesn’t.
Two hundred years ago, if you wanted to get from Memphis to say… Lake Providence, Mississippi, you could take a boat directly between the two. Today you would take a car, and the only paths between the two are pretty round-a-about and “small country road” sorts of affairs. On the other hand, getting from Memphis Tennessee to Atlanta, Georgia, is now easy, while a couple of hundred years ago would be a big deal indeed. The sociological changes wrought by moving from rivers to roads are almost impossible to fathom. But you wouldn’t know that if you just stared at the white lines.
The Hedge 25: Building the Next Generation of Network Engineer
If there is one thing I notice when I look around at the IETF—and many other places where I meet a lot of network operations and engineering folk—it’s that we all seem to be getting a bit older. This should lead us to an obvious question—what are we doing about bringing up a new generation of network engineers? David Huberman joins Tom Ammon and I to discuss this interesting question. David i s involved in a number of community-based efforts to train next generation network engineers, some of which he discusses in his excellent article at the APNIC blog.
Whither Cyber-Insurance?
Note: I’m off in the weeds a little this week thinking about cyber-insurance because of a paper that landed in one of my various feeds—while this isn’t something we often think about as network operators, it does impact the overall security of the systems we build.
When you go to the doctor for a yearly checkup, do you think about health or insurance? You probably think about health, but the practice of going to the doctor for regular checkups began because of large life insurance companies in the United States. These companies began using statistical methods to make risk, or to build actuarial tables they could use to set the premiums properly. Originally, life insurance companies relied on the “hunches” of their salesmen, combined with some checking by people in the “back office,” to determine the correct premium. Over time, they developed networks of informers in local communities, such as doctors, lawyers, and even local politicians, who could describe the life of anyone in their area, providing the information the company needed to set premiums correctly.
Over time, however, statistical methods came into play, particularly relying on an initial visit with a doctor. The information these insurance companies gathered, however, gave them insight into what habits increased or decreased longevity—they decided they should use this information to help shape people’s lives so they would live longer, rather than just using it to discover the correct premiums. To gather more information, and to help people live better lives, life insurance companies started encouraging yearly doctor visits, even setting up non-profit organizations to support the doctors who gave these examinations. Thus was born the yearly doctor’s visit, the credit rating agencies, and a host of other things we take for granted in modern life.
You can read about the early history of life insurance and its impact on society in How Our Days Became Numbered.
What does any of this have to do with networks? Only this—we are in much the same position in the cyber-insurance market right now as the life insurance market in the late 1800s through the mid-1900s—insurance agents interview a company and make a “hunch bet” on how much to charge the company for cyber-insurance. Will cyber-insurance ever mature to the same point as life insurance? According to a recent research paper, the answer is “probably not.” Why not?
First, legal restrictions will not allow a solution such as the one imposed by payment processors. Second, there does not seem to be a lot of leverage in cyber-insurance premiums. The cost of increasing security is generally much higher than any possible premium discount, making it cheaper for companies just to pay the additional premium than to improve their security posture. Third, there is no real evidence tying the use of specific products to reductions in security breaches. Instead, network and data security tend to be tied to practices rather than products, making it harder for an insurer to precisely specify what a company can and should to improve their posture.
Finally, the largest problem is measurement. What does it look like for a company to “go to the doctor” regularly? Does this mean regular penetration tests? Standardizing penetration tests is difficult, and it can be far too easy to counter pentests without improving the overall security posture. Like medical care in the “early days,” there is no way to know you have gathered enough information on the population to know if you correctly understand the kinds of things that improve “health”—but there is no way to compel reporting (much less accurate reporting), nor is there any way to compel insurance companies to share the information they have about cyber incidents.
Will cyber-insurance exist as a “separate thing” in the future? The authors largely answer in the negative. The pressures of “race to the bottom,” providing maximal coverage with minimal costs (which they attribute to the structure of the cyber-insurance market), combined with lack of regulatory clarity and inaccurate measurements, will probably end up causing cyber-insurance to “fold into” other kinds of insurance.
Whether this is a positive or negative result is a matter of conjecture—the legacy of yearly doctor’s visits and public health campaigns is not universally “good,” after all.
The Hedge 24: Single Source of Truth
Tim Schreyack recently presented at NANOG on the topic of building a single source of truth for network automation. Tim joins Tom and Russ in a wide-ranging discussion about single sources of truth, changing the way we see the network, and the changing skills of network engineers.
Ironies of Automation
Ironies of Automation
In 1983 I was just joining the US Air Force, and still deeply involved in electronics (rather than computers). I had written a few programs in BASIC and assembler on a COCOII with a tape drive, and at least some of the electronics I worked on were used vacuum tube triodes, plate oscillators, and operational amplifiers. This was a magical time, though—a time when “things” were being automated. In fact, one of the reasons I left electronics was because the automation wave left my job “flat.” Instead of looking into the VOR shelter to trace through a signal path using a VOM (remember the safety L!) and oscilloscope, I could sit at a terminal, select a few menu items, grab the right part off the depot shelf, replace, and go home.
Maybe the newer way of doing things was better. On the other hand, maybe not.
What brings all this to mind is a paper from 1983 titled The Ironies of Automation. It might often seem, because of our arrogant belief that we can remake the world through disruption (was the barbarian disruption of Rome in 455 the good sort of disruption, or the bad sort?), we often think we can learn nothing from the past. Reality check: the past is prelude.
What can the past teach us about automation? This is as good a place to start as any other:
This is the first of the ironies of automation Lisanne Bainbridge discusses—and this is the irony I’d like to explore. The irony she is articulating is this: the less you work on a system, the less likely you are to be able to control that system efficiently. Once a system is automated, however, you will not work on the system on a regular basis, but you will be required to take control of the system when the automated controller fails in some way. Ironically, in situations where the automated controller fails, the amount of control required to make things right again will be greater than in normal operation.
In the case of machine operation, it turns out that the human operator is required to control the machine in just the situations where the least amount of experience is available. This is analogous to the automated warehouse in which automated systems are used to stack and sort material. When the automated systems break down, there is absolutely no way for the humans involved to figure out why things are stacked the way they are, nor how to sort things out to get things running again.
This seems intuitive. When I’m running the mill through manual control, after I’ve been running it for a while (I’m out of practice right now), I can “sense” when I’m feeding too fast, meaning I need to slow down to prevent chatter from ruining the piece, or worse—a crash resulting in broken bits of bit flying all over the place.
How does this apply to network operations? On the one hand, it seems like once we automate all the things we will lose the skills of using the CLI to do needed things very quickly. I always say “I can look that command up,” but if I were back in TAC, troubleshooting a common set of problems every day, I wouldn’t want to spend time looking things up—I’d want to have the right commands memorized to solve the problem quickly so I can move to the next case.
This seems to argue against automation entirely, doesn’t it? Perhaps. Or perhaps it just means we need to look at the knowledge we need (and want) in a little different way (along with the monitoring systems we use to obtain that knowledge).
Humans think quick and slow. We either react based on “muscle memory,” or we must think through a situation, dig up the information we need, and weigh out the right path forward. When you are pulling a piece of stainless through a bit and the head starts to chatter, you don’t want to spend time assessing the situation and deciding what to do—you want to react.
But if you are working on an automated machine, and the bit starts to chatter, you might want to react differently. You might want to stop the process entirely and think through how to adjust the automated sequence to prevent the bit from chattering the next time through. In manual control, each work piece is important because each one is individually built. In the automated sequence, the work piece itself is subsumed within the process.
It isn’t that you know “less” in the automated process, it’s that you know different things. In the manual process, you can feel the steel under the blade, the tension and torque, and rely on your muscle memory to react when its needed. In the automated process, you need to know more about the actual qualities of the bit and metal under the bit, the mount, and the mill itself. You have to have more of an immediate sense of how things work if you are doing it manually, but you have to have more of a sense of the theory behind why things work the way if it is automated.
A couple of thoughts in this area, then. First, when we are automating things, we need to be very careful to assume there is no “fast thinking” when things ultimately do fail (it’s not if, it’s when). We need to think through what information we are collecting, and how that information is being presented (if you read the original paper, the author spends a great deal of time discussing how to present information to the operator to overcome the ironies she illuminates) so we take maximum advantage of the “slow path” in the human brain, and stop relying on the “fast path” so much. Second, as we move towards an automated world, we need to start learning, and teaching, more about why and less about how, so we can prepare the “slow path” to be more effective—because the slow path is the part of our thinking that’s going to get more of a workout.
The Hedge 23: The MOPS Working Group
The IETF works on many things beyond IP and routing—the Media Operations (MOPS) working group is gathering input on media-related operational issues and practices, including “proposed technologies related to the deployment, engineering, and operation of media streaming and manipulation protocols and procedures in the global Internet (inter-domain) and within-domain networking.” Leslie Daigle and Eric Vyncke, the co-chairs of the MOPS working group, join Alvaro Retana and Russ White to discuss the work they are doing.