CULTURE
Hedge 284: Netops and Corporate Culture
We all know netops, NRE, and devops can increase productivity, increase Mean Time Between Mistakes (MTBM), and decrease MTTR–but how do we deploy and use these tools? We often think of the technical hurdles you face in their deployment, but most of the blockers are actually cultural. Chris Grundemann, Eyvonne, Russ, and Tom discuss the cultural issues with deploying netops on this episode of the Hedge.
download
Fast Following Fails
Fast following fails.
Whenever I hear a leader in a technology business say, “We’re going to fast follow because it’s the most profitable place to be,” I know I’m looking at a failed organization. I didn’t come to this conclusion by thinking about it. I came to this conclusion by observing it repeatedly.
After observing it, however, I wanted to understand why this particular strategy fails so consistently and spectacularly. Why? To understand my theory, we need to start in a somewhat different place than business—we need to start with the nature of goals and humans.
You can place goals into two buckets: first things and second things.
First things are foundational. If you are a technology company, the first thing is building a stable, resilient, and flexible platform (or foundation). The products you sell will only be as stable as your platform. The innovation you achieve will only be as consistent as your platform is.
Second things are goals you can only achieve once you’ve built the first things.
Here’s the hard truth no one wants to hear: Generating revenue is a second thing.
Humans become what they do.
We all want to believe we can become what we desire—but we actually become what we do. In Aristotelian philosophy, this is called the virtue ethic. You become physically virtuous by exercising your body. You become intellectually virtuous by thinking about hard things.
Companies are the same way. A company can only become innovative by innovating. Innovating becomes a habit—or it doesn’t.
What does this have to do with fast following?
The theory of the “fast follower” is: “I’m going to let other people spend money on research and development, I’m going to let them carry the burden of innovating and making all the mistakes, then I’m going to jump in and scoop up their innovation.”
This seems sound at first glance. It’s a compelling story.
It doesn’t work, however, because you are chasing another organization’s success without building their platform. You’ve placed a second thing—revenue generation—in first place, and first things—building a platform and innovating—in second place.
When you put building a platform and innovating on top of that platform in second place—when you “fast follow”—you lose the habit of building a solid platform and the habit of innovating.
Building a platform on which you can actually ship innovative products—no matter who invented them—and cultivating a mindset that seeks out good innovation creates a culture of innovation. When you build the mental habit of waiting until someone else’s innovation succeeds and then building “just enough platform to make it work here, too,” you are building an unstable platform and killing innovation.
“But what about all those fast-following success stories?”
One reason “fast following” success stories abound is that you can make a lot of money for a little while with the fast-following strategy. Another is that when an organization first moves to fast following, they have the leftover platform and innovation culture to carry them for a little while.
But time will out all fast following organizations. When the market shifts, fast followers will have neither the platform to shift with it nor the innovation to change with the market.
By putting second things first, the fast follower loses the first things that make the second thing possible.
“But I’ll make a lot of money until it fails, right? I don’t care about the future, just making a lot of money quickly!”
Sure, if that’s the life you want to lead, go for it. If you want to live a life devoid of community, and you want to lie on your deathbed and say, “I don’t care what damage I caused,” if sheer wealth is all that matters, feel free to fast follow.
If you want to build something, however, go build it.
Fast following gives up building platforms and innovating for immediate success, and winds up failing to innovate or succeed.
Making Networking Cool Again? (2)
Network engineering is not “going away.” Network engineering is not less important than it was yesterday, last year, or even a decade ago.
But there still seems to be a gap somewhere. There are fewer folks interested than we need. We need more folks who want to work as full-time network engineers, and more folks with network engineering skills diffused within the larger IT community. More of both is the right answer if we’re going to continue building large-scale systems that work. The real lack of enthusiasm for learning network engineering is hurting all of IT, not just network engineering.
How do we bridge this gap? We’re engineers. We solve problems. This seems to be a problem we might be able to solve (unlike human nature). Let’s try to solve it.
As you might have guessed, I have some ideas. These are not the only ideas in the world—feel free to think up more!
If you walk into a robotics class, even an introductory robotics class, you see people … building robots. If you walk into a coding class, even an introductory one, you see people … writing software. If you walk into a network network engineering class you see … someone lecturing about the OSI model, packet formats, or how to configure BGP.
What problems are people learning to solve robotic engineering? How to build a robot and get it to do something to solve a real-world problem. What problems are people learning to code solving? How to tackle some real-world problem.
Sure, the problems being solved at an introductory level might be trivial, like: “Read this file and spit out a sum of the numbers in the fourth column.” But they are still starting, right from the beginning, by taking requirements and converting them into solutions.
What problems are network engineers learning how to solve? How to choose hardware, string it all together, and configure BGP.
Do you see the difference?
All engineers solve problems—it’s the nature of engineering. But are we creating a mindset in prospective network engineers, or even adjacent fields, that we solve real-world problems? Or are we giving them the impression that we solve whiteboard problems by talking about bits, bytes, configurations, and cable types?
Have you ever seen the glazing over of eyes while explaining how you put four transport protocols on top of one another (look at all the pretty tunnels)? How about when you create a chart showing how TCP and QUIC can be “kind-of sort-of” forced into the OSI model? Or when you spin out your BGP packet format charts, showing how we’ve (mis)used address families to carry everything anyone can imagine?
I’ve been teaching this stuff for years (okay, decades). Over time, I’ve moved away from teaching configurations and packet formats. I’ve gone from Advanced IP Network Design to Computer Networking Problems and Solutions. These are very different ways of looking at network engineering.
Focusing on real-world problems would help connect business and other IT folks to the network, connect theory to practice, and people to network engineering. Going home at the end of the day saying, “I solved a problem,” can be satisfying. Going home at the end of the day saying, “I configured BGP?”
Another thing adopting the mindset of solving real-world problems might do is help us lose unnecessary complexity. I know complexity is necessary to build resilient systems; we cannot build what we build without creating and encountering complexity.
But we often run ourselves into the ditch on both sides of the road.
We unintentionally build too complex because we try to make it too simple. Quick, which is simpler: building a data center fabric with one routing protocol or two? A single chassis system or several smaller fixed format devices? A proprietary system or something built on open standards?
How many balloons fit in a bag? (thanks, Don)
Failing to start with the tradeoffs, and thinking through what problem we’re actually trying to solve, leads to unnecessary complexity. Such designs might not immediately fail, but they will fail, and “it’s so complex” just isn’t an excuse.
Don’t even try to tell me there aren’t any tradeoffs. If you think there aren’t any tradeoffs, that just means you haven’t looked hard enough. Go find them, think about them, and document them.
We also build complex things because we think it offers job security, or it’s neat, or we like to feel like the kid who says to the world, “look what I built!”
I know it’s exciting to hear stories about that time someone rescued a network from a major failure—after all, that’s solving a real-world problem. Building a network that just works might be “boring,” but it solves many more real-world problems than raising a network from the dead.
We love our fashionable capes, but … capes can get caught in a nearby jet engine. Lose the cape. In the long run, it’ll make network engineering more attractive as a career field and field of knowledge.
The Bottom Line
No, the sky is not falling. We still need networks, and we still need network engineers.
Yes, there is a problem. Too many companies are going “to the cloud” because they cannot find people qualified to build and maintain their very complex networks. There’s too much centralization and too little oppeness.
So maybe let’s stop saying “we don’t need network engineers.” And maybe let’s really think about how we’re building things. And maybe let’s focus on solving real-world problems, starting from day one in network engineering classrooms.
Network engineering is still cool—let’s go out there believing—and selling—that idea to the world.
Hedge 207: Being a Network Engineer with Phil Grevasi

What does it mean to be a network engineer in today’s world of information technology? Phil Gervasi joins Tom and Russ to discuss the ins and outs of network engineering, and what it’s really like to be in this small corner of information technology today.
download
Hedge 206: Taking Care of Yourself with Ethan Banks
As we reach the end of what has been a hard two-year stretch for what seems like the entire world, Ethan Banks joins Tom, Eyvonne, and Russ to talk about the importance of taking care of yourself. In the midst of radical changes, you can apply self-discipline to make your little part of the world a better place by keeping yourself sane, fit, and well-rested.
Mean Time to Innocence is not Enough
A long time ago, I supported a wind speed detection system consisting of an impeller, a small electric generator, a 12 gauge cable running a few miles, and a voltmeter. The entire thing was calibrated through a resistive bridge–attach an electric motor to the generator, run it at a series of fixed speed, and adjust the resistive bridge until the voltmeter, marked in knots of wind speed, read correctly.
The primary problem in this system was the several miles of 12 gauge cable. It was often damaged, requiring us to dig the cable up (shovel ready jobs!), strip the cable back, splice the correct pairs together, seal it all in a plastic container filled with goo, and bury it all again. There was one instance, however, when we could not get the wind speed system adjusted correctly, no matter how we tried to tune the resistive bridge. We pulled things apart and determined there must be a problem in one of the (many) splices in the several miles of cable.
At first, we ran a Time Domain Reflectometer (TDR) across the cable to see if we could find the problem. The TDR turned up a couple of hot spots, so we dug those points up … and found there were no splices there. Hmmm … So we called in a specialized cable team. They ran the same TDR tests, dug up the same places, and then did some further testing and found … the cable was innocent.
This set up an argument, running all the way to the base commander level, between our team and the cable team. Who’s fault was this mess? Our inability to measure the wind speed at one end of the runway was impacting flight operations, so this had to be fixed. But rather than fixing the problem, we were spending our time arguing about who’s fault the problem was, and who should fix it.
When I read this line in a recent CAIDA research paper–
“Measurement is political, and often adversarial.”
It rang very true. In Internet terms, speed, congestion, and even usage are often political and adversarial. Just like the wind speed system, two teams were measuring the same thing to prove the problem wasn’t their’s–rather than to figure out what the problem is and how to fix it.
In other words, our goal is too often Mean Time to Innocence (MTTI), rather than Mean Time to Repair (MTTR).
MTTI is not enough. We need to work with our application counterparts to find and fix problems, rather than against them. Measurement should not be adversarial, it should be cooperative.
We need to learn to fix the problem, not the blame.
This is a cultural issue, but it also impacts the way we do telemetry. For instance, in the case of the wind speed indicator, the problem was ultimately a connection that “worked,” but with high capacive reactance such that some kinds of signals were attenuated while others were not. None of us were testing the cable using the right kind of signal, so we all just sat around arguing about who’s problem it was rather than solving the problem.
When a user brings a problem to you, resist the urge to go prove yourself–or your system–innocent. Even if your system isn’t the problem, your system can provide information that can help solve the problem. Treat problems as opportunities to help rather than as opportunies to swish your superhero cape and prove your expertise.
Hedge 153: Security Perceptions and Multicloud Roundtable
Tom, Eyvonne, and Russ hang out at the hedge on this episode. The topics of discussion include our perception of security—does the way IT professionals treat security and privacy helpful for those who aren’t involved in the IT world? Do we discourage users from taking security seriously by making it so complex and hard to use? Our second topic is whether multicloud is being oversold for the average network operator.
