Archive for 2015
Fear itself: Thinking through change and turmoil
Fair warning: this is going to be a controversial post, and it might be considered a bit “off topic.”
Maybe it’s just the time of year for fear. Or maybe it’s several conversations I’ve been involved in recently. Or maybe it’s the result of following over 150 blogs on a daily basis covering everything from religion to politics to technology to philosophy. Whatever it is, there’s one thing I’ve noticed recently.
We’re really afraid.
I don’t mean “concerned about what the future might hold,” but rather — it seems, at least sometimes — sinking into a state of fear bordering on the irrational. Sometimes it feels like the entire world is one long troubleshooting session in the worst designed network I’ve ever encountered. Let me turn to a few completely different areas to illustrate my point. Some of these are going to make people mad, so hold on to your hats — and hear me out before you jump all over me or shut down.
We’re afraid of what the future might hold for us as engineers and as people. Maybe this entire software defined thing is going to destroy my entire career. Maybe I’ll end up like a buggy whip maker a few years after the first car was built. Maybe the entire world is going to sink under the oceans as they rise due to man made global warming. Maybe we’re all going to be replaced by robots, leaving none of us anything to do for a living at all. Maybe we’re all going to eat GMO foods and die. Maybe I don’t have the right certifications, or maybe I have too many certifications. Maybe cell phones are going to give us all cancer.
Or maybe, just maybe, we’ve come too close to perfecting fear as the ideal motivator for selling just about everything from things to training to politics. Maybe the noise level has gotten so high that we won’t listen until it’s a existential crisis right now. Maybe we’re rushing from crisis to crisis like a boat out in a huge storm trying to stay above water and forgetting to ask where it is we’re going — which port we actually should call home.
Maybe it’s time to reassess, to find some strategy that will help us cope with all this information and all this fear. Some thoughts to that end.
First, ask what claim is actually being made. This might be painful, but learn logical syllogisms, and make it a habit to turn enthymemes into a proper syllogistic form so you can actually evaluate the claim. We’re too fast to accept straw men, too quick to dismiss with a casual wave of the hand, an appropriate bit of snark, and a quick dose of name calling. We’re too slow to listen and spend time really trying to understand. We’ve sown a world of 140 character snippets, and we’re reaping a whirlwind of thoughtlessness.
Second, ask what supports the claim. I don’t mean who supports the claim or why they support it. Stop asking about feelings and motives. Start asking about facts.
Third, ask why you might have any reason to doubt the claim. Intentionally fight against your confirmation bias and seek out the most credible sources you can find that disagree with the claim. Read them carefully, intentionally, and as honestly as you can.
Okay, you’ve done all of this, and you believe the claim is correct. Now is the time to jump to action, right? Wrong. In fact, the hard work has just begun.
First, ask what it is you can actually do about it. Second, find the tradeoffs, including who pays and how.
The climate of fear we live in particularly shuts down our ability to think about tradeoffs. When we’re afraid, we move to “there is no tradeoff,” “we need to do something about this,” and “anyone who disagrees is a moral monster” far too quickly. Engineers should know from long experience with real world systems there are always tradeoffs. If you’ve not found them, then you’re not looking — and if you’re not looking, then you’re not really engaged in thinking.
Let me try to take a personal example here. “What happens if my job ends tomorrow, because the technology I know goes away?” Well, you could run around like a turkey the day before Thanksgiving. I don’t how useful that’s going to be, but it’s certainly entertaining, and, in some ways actually satisfying.
Or you could process the question, ask if it’s true (it probably is on some level all the time), think about what you can do about it, and then focus on finding the tradeoffs so you can make a rational set of decisions about what actions to take in response. Maybe you should make it a practice to learn new skills on a regular basis? “But what if I bet wrong, and learn the wrong skills?” How is that better than not betting at all? Learning is, itself, a skill that takes regular practice.
We need to use the same process across the board. Before we casually cast aside anyone’s rights (or responsibilities) in the name of creating a “safer world,” before we radically alter our entire way of life to solve the fifteenth world crisis that has a celebrity “do something now” video attached, before we all collapse in despair at the collapse of our world and our careers, we need to make certain we ask the questions — what does this really mean, what are the facts supporting it, why should I doubt it, what can I really do about it, and what are the tradeoffs?
I don’t want to get into a long, drawn out, political discussion. That’s not what this blog is about. I’m not trying to make a political point, but rather a thinking point. Fear makes us treat one another like objects when we really need to listen to one another as people. We really need to learn to get past the fear our world seems to be drowning in. There are things we should rationally be afraid of. But there is also a sense in which fear removes our capacity to react rationally, and hence makes our nightmares into reality.
Why aren’t you teaching?
There is an old saw about teaching and teachers: “Those who can, do. Those who can’t, teach.” This seems to be a widely believed thought in the engineering world (though perhaps less in the network engineering world than many other parts of engineering) — but is it true? In fact, to go farther, does this type of thinking actually discourage individual engineers teaching, or training, in a more formal way in the networking world? Let me give you my experience.
What I’ve discovered across the years is something slightly different: if you can’t explain it to someone else in a way they can understand it, then you don’t really know it. There are few ways to put this into practice in the real world better than intentionally taking on the task of teaching others what you know. In fact, I’ve probably learned much more in the process of preparing to teach than I ever have in “just doing.” There is something about spending the time in thinking through how to explain something in a number of different ways that encourages understanding. To put it in other terms, teaching makes you really think about how something works.
Don’t get me wrong here — engineers shouldn’t lose their focus on doing. But we need to learn to blend doing with understanding in a way that we’ve not done well with up until now. We’ve often been so focused on the what that we forget about the why.
Given that one excellent way to develop the thinking skills, to exercise our why skills as well as our what skills, is to tech, why aren’t you teaching?
Is it that you don’t think you have the skills to teach? Is it that you don’t think you have the opportunity? Is it that you don’t think you have the knowledge?
All of these are excuses, rather than real reasons. You can always take the time to put together a basic course in networking for the people in your company. In fact, maybe the reason they don’t really understand your job is because you never explain the technology you work on. You can always take the time to teach your peers, or even the junior engineers on your team, or another team. There are local high schools that could use your time in the classroom teaching networking technology. Where else are new network engineers coming from, after all?
I’m also not saying you shouldn’t rely on professional education — after all, I still want you to buy my books. 🙂 But there’s something about building and giving a class that teaches things you just can’t learn many other places.
So — let me ask again — why aren’t you teaching?
Innovation and the Internet
Industries mature, of course. That they do so shouldn’t be surprising to anyone who’s watched the world for very long. The question is — do they mature in a way that places a few players at the “top,” leaving the rest to innovate along the edges? Or do they leave broad swaths of open space in which many players can compete and innovate? Through most of human history, the answer has been the first: industries, in the modern age, tend to ossify into a form where a few small players control most of the market, leaving the smaller players to innovate along the edges. When the major impetus in building a new company is to “get bought,” and the most common way for larger companies to innovate is by buying smaller companies (or doing “spin ins”), then you’ve reached a general point of stability that isn’t likely to change much.
Is the networking industry entering this “innovation free zone?” Or will the networking industry always be a market with more churn, and more innovation? There are signs in both directions.
For instance, there’s the idea that once technology reaches a certain level of capability, there’s just no reason for any further forward motion. Fifty years ago, if you would have asked people what airplanes could do, and what they would look like, you have have gotten some wild feedback. Today, ask the same question, and you’ll likely get the same wild ideas. Things haven’t changed much in air travel (other than reductions in the amount of space in the cattle cars, it seems) because we’ve reached the point where new advances don’t bring much in the way of new benefits.
Another instance: there is a growing group of “old” companies with a lot of money, and they’re turning that money into political power. The one sure way to ensure stagnation is to get the government involved. A case in point here is LTE-U, which bids fair to turn the last mile upside down. It seems a number of large companies are using their lobbying mojo to make certain older carriers aren’t allowed to use unlicensed space. A lot of top flight engineers don’t seem to agree on the overall impact of allowing AT&T, for instance, to expand their wireless network on WiFi frequencies; much of the argument at the moment seems to come down to the political, rather than the engineering aspects of the problem. When lobbying takes over engineering, it’s a sure sign the industry is moving into an ossified state. Robotics are the new and exciting thing now; the Internet seems like a “given.”
On the other hand, routing is more interesting right now than it has been in a long time. Software Defined and cloud are taking over the world, it seems (though a few of us do try to inject a bit of sanity into the news stream every now and then). Over the top services, like SD-WAN, seem to be creating new value in spaces long thought completely ossified. In a somewhat virtual world (hardware still counts, but the intelligence tends to move into the overlay), there isn’t any apparent point at which you can say, “we’re done with this, let’s move to the next thing.”
It seems, to me, that we’re on a bit of a cusp, a turning point. Which way the industry goes depends, in some part, on the way the larger players go. Will they continue to turn to the government, using political muscle to solidify revenue streams? Or will they turn back to real innovation?
Let’s not lose sight of the role each of us, as individual network engineers, play in the path from this point forward — the choice between the safe vendor bet, and innovating even on a small scale, played out over the thousands of networks in the world, can make a huge difference. We tend to divide the world into small networks with boring problems and large networks with interesting problems. This is a false dichotomy — interesting problems are interesting problems, no matter what the network size. Interested people make for interesting solutions, and in turn, interesting innovation.
We need to realize that no matter how small it seems, we’re at a point where the small decisions, en mass, will make a big difference. What decisions will you make today?
Assuming the worst is not the best assumption
It was too bad to be true, but I should have known that assuming the worst was not the best assumption. I was driving the “other” car, the Saab, on the way back from the METNAV shop around eight in the morning. Since the shop was located in the middle of the three runways, this meant I had to drive across the 18 taxiway, along the white lines painted between the C-141’s, C-130’s, KC-10’s, F-4’s, and sometimes other odds and ends, and then past the Tower, off the flightline, and onto the “surface streets.” As I was coming off a call at around three in the morning, I wasn’t in uniform. For some reason, I hadn’t driven my normal car — a white Jeep — so the folks in the Tower certainly wouldn’t recognize me.
So when the SP flipped his lights on and pulled in behind me, I was worried. Just as the lights came on, I remembered something really important: I had forgotten to put my sticker on the car. You see, to drive on the flightline, you had to have a sticker on your car. There were various colors for the different areas you could gain access to; mine was red, which meant I had access to everything on the flightline other than the red zone and hot spot. But here I was at eight in the morning, after spending five hours putting the glideslope back on the air for the morning’s landing runs, in a plain pair of jeans, a ratty T-Shirt, without a shower, electronics junk and tools strewn in the back seat of the Saab, and no sticker.
As an aside, I’d encountered the SP’s before on the flightline. Several times, in fact. I was once pushed to the ground face first because I’d accidentally crossed the red line. One night a friend and I walked out of the shelter at the localizer to find ourselves staring down the barrels of at least a dozen M16’s. It seems there was a shift change while we were inside working on something, and the outgoing duty officer had forgotten to brief in the oncoming duty officer. Not a happy memory.
Needless to say, then, I was assuming the worst.
I stopped (there is no place to “pull over” on a flightline”), rolled down the window, and waited. The officer walked up to the car, took a look at the back seat, took a look at me, and said, “I just wanted you to know your lights are on. Don’t forget when you park to turn them off. I wouldn’t want you to have to call a tow truck because of a failed battery.” With that, he turned, went back to his car, and drove off.
I’m glad he didn’t give me time to go through all my excuses. On reflection, it would have only made it worse. Of course I had my military ID handy, but just having an ID doesn’t help you if you’re on the flightline without authorization. In fact, it might just make things worse.
Thinking back through my life, I can recall a lot of times that I’ve made things a lot worse by assuming the worst — by making the worst assumption my first, and best, assumption. By assuming the worst about a situation (and about people), I’ve probably made a lot of things a lot worse than they ever needed to be.
Don’t do this.
What I learned that morning, even though my head was foggy, even though I was tired, and even though I had a few hours of paperwork staring me in the face, is this: don’t assume you’re being stopped for doing something wrong. You should allow each person who enters your life at least a neutral frame of reference, if not a positive one. In a court of law, you’re guilty until proven innocent. In real life, if you treat everyone as if they’re guilty, you’re going to make them all act like their guilty.
Sometimes someone just wants to tell you that you left your lights on.
Rule 11 is your friend
It’s common enough in the networking industry — particularly right now — to bemoan the rate of change. In fact, when I worked in the Cisco Technical Assistance Center (TAC), we had a phrase that described how we felt about the amount of information and the rate of change: sipping through the firehose. This phrase has become ubiquitous in the networking world to describe the feeling we all feel of being left out, left behind, and just plain not able to keep up.
It’s not much better today, either. SDNs threaten to overturn the way we build control planes, white boxes threaten to upend the way we view vendor relationships, virtualization threatens to radically alter the way we think about the relationship between services and the network, and cloud computing promises just to make the entire swatch of network engineers redundant. It’s enough to make a reasonable engineer ask some rather hard questions, like whether it’s better to flip burgers or move into management (because the world always needs more managers). Some of this is healthy change, of course — we need to spend more time thinking about why we’re doing what we’re doing, and the competition of the cloud is probably a good thing. But there’s another aspect here I don’t think we’ve thought about enough.
Sure there’s a firehose here. But there are fields all over the world where there’s a veritable firehose of new information, new thinking, and new products being designed, developed, and introduced. The actual work of building buildings has radically changed over the last 50–100 years. There have been some folks thrown out of the business in the process, but what we tend to see is more buildings being put up faster, not a bunch of mid life hamburger flippers who used to design buildings. All around us we see tons of new technology being pressed into service, and yet we don’t seem to always have the massive fear of dislocation combined with the constant angst that always seems to be in the air in network engineering (and the information technology industry at large).
I know it’s easy to fly the black flag and say, “well, if you can’t keep up, get out.” I don’t know if this is precisely fair to the old, grizzled folks who have families and lives outside work. I don’t even know if this is fair to the newbies coming in—a career field that eats people by the time they are 50, and says, “just save up while you make enough to do so, and forget having a family,” just doesn’t seem all that healthy to me. Instead, we need to find ways to mitigate the firehose. Somehow, we need to learn to cut it down so we can actually learn, and understand, and still live our lives.
But before I talk about Rule 11, let me be honest for a second — this industry isn’t going to change unless we change it. There’s no real reason for it to change. After all, 20 year olds cost less than 50 year olds to keep on staff, the firehose makes a lot of money for vendors, and it’s a large ego boost in asking questions like, “did you see the latest vendor x box,” or in “beating” someone in an interview.
For those of us who do want to change the networking world, or even just to keep up without sipping from the firehose, what can we use as a handle? This is where Rule 11 comes in. To refresh your memory—
Most people sniggle when they read this, because it really is funny. But if rule 11 is true, 90% of the water coming out of the firehose is, in fact, recycled.
Do you see it yet? If you can successfully build a mental model of each technology, and then learn to expand that mental model to each new technology you encounter, you will be able to mitigate the firehose.
If we’re going to survive as an industry, we need to get past the firehose. We need to stop thinking about the sheet metal and the cable colors, and start thinking about processes, ideas, and models. We need to stop flying by the seat of our pants, and start trying to make this stuff into real engineering, rather than black magic. Yes, I moved from working on airfield electronics to network engineering because I craved the magical side of this world, but magic just isn’t a sustainable business model, nor a sustainable way of life.
Information wants to be protected: Security as a mindset
I was teaching a class last week and mentioned something about privacy to the students. One of them shot back, “you’re paranoid.” And again, at a meeting with some folks about missionaries, and how best to protect them when trouble comes to their door, I was again declared paranoid. In fact, I’ve been told I’m paranoid after presentations by complete strangers who were sitting in the audience.
Okay, so I’m paranoid. I admit it.
But what is there to be paranoid about? We’ve supposedly gotten to the point where no-one cares about privacy, where encryption is pointless because everyone can see everything anyway, and all the rest. Everyone except me, that is—I’ve not “gotten over it,” nor do I think I ever will. In fact, I don’t think any engineer should “get over it,” in terms of privacy and security. Even if you think it’s not a big deal in your own life, engineers should learn to treat other people’s information with the utmost care.
In moving from the person to the digital representation of the person, we often forget it’s someone’s life we’re actually playing with. I think it’s time for engineers to take security—and privacy—personally. It’s time to actually do what we say we do, and make security a part of the design from day one, rather than something tacked on to the end.
And I don’t care if you think I’m paranoid.
Maybe it’s time to replace the old saying information wants to be free. Perhaps we should replace it with something a little more realistic, like:
Information wants to be protected.
It’s true that there are many different kinds of information. For instance, there’s the information contained in a song, or the information contained in a book, or a blog, or information about someone’s browsing history. Each piece of information has a specific intent, or purpose, a goal for which it was created. Engineers should make their default design such that information is only used for its intended purpose by the creator (or owner) of that information. We should design this into our networks, into our applications, and into our thought patterns. It’s all too easy to think, “we’ll get to security once things are done, and there’s real data being pushed into the system.” And then it’s too easy to think, “no-one has complained, and the world didn’t fall apart, so I’ll do it later.”
But what does it mean to design security into the system from day one? This is often, actually, the hard part. There are tradeoffs, particularly costs, involved with security. These costs might be in terms of complexity, which makes our jobs harder, or in terms of actual costs to bring the system up in the first place.
But if we don’t start pushing back, who will? The users? Most of them don’t even begin to understand the threat. The business folks who pay for the networks and applications we build? Not until they’re convinced there’s an ROI they can get their minds around. Who’s going to need to build that ROI? We are.
A good place to start might be here.
And we’re not going to until we all start nurturing the little security geek inside every engineer, until we start taking security (and privacy) a little more seriously. Until we stop thinking about this stuff as just bits on the wire, and start thinking about it as people’s lives. Until we reset our default to “just a little paranoid,” perhaps.
P.S. I’m not so certain we should get over it. Somehow I think we’re losing something of ourselves in this process of opening our lives to anyone and everyone, and I fear that by the time we figure out what it is we’re losing, it’ll be too late to reverse the process. Somehow I think that treating other people as a product (if the service is free, you are the product) is just wrong in ways we’ve not yet been able to define.
Micromanaging networks considered harmful: on (k)nerd knobs
Nerd Knobs (or as we used to call them in TAC, knerd knobs) are the bane of the support engineer’s life. Well, that and crashes. And customer who call in with a decoded stack trace. Or don’t know where to put the floppy disc that came with the router into the router. But, anyway…
What is it with nerd knobs? Ivan has a great piece up this week on the topic. I think this is the closest he gets to what I think of as the real root cause for nerd knobs —
Greg has a response to Ivan up; again, I think he gets close to the problem with these thoughts —
A somewhat orthogonal article caught my eye, though, that I think explains what is actually going on here with those pesky nerd knobs. The article is really about SQL and the concept of micromanaging software. To give you a flavor (in case you’re too lazy/busy to head over there and read the whole thing) —
I think this gets to the heart of the nerd knob problem. What’s happening with nerd knobs is it’s easier to tell the system how we want something done than it is to tell the system what we want to do. Think about this way: you install a routing protocol, and you tell it what you want in broad, general terms. Something like, “I want the shortest path between each pair of points in the network.” Then you run into a situation where you need that modified, so you mess around with the metrics some, and get on with your life. Then you run into a situation where you need this flow to go here, and that flow to go there, so you install some policy based routing along the way.
Per link metrics are just the first level of nerd knobs. Policy based routing is just the second. The more precise we want to get, the deeper the nerd knobs go. Want to load share over links that aren’t truly equal cost? Oh, just nerd knob it. Want to send AS’ in the AS path you shouldn’t? Just nerd knob it.
The reality is every nerd knob in routing represents a policy driven by a business requirement expressed as a tweak to the underlying fundamental routing algorithm. As Ivan rightly points out, going to SDNs isn’t going to solve this problem. If anything, it’s going to make it worse. Now, rather than seeing the nerd knob for what it is, a pain in the butt that needs to be explained and dealt with at 2AM when you’re half asleep and the TAC engineer is halfway around the world, it’s going to be “just another line of code.”
This might sound brilliant to someone who hasn’t managed, or dealt with, multi-million line projects and the vagaries of codebase management. Ask someone who has, though, before you get into this. It’s just a different set of problems, not a better set of problems.
The root cause here, though, isn’t nerd knobs. And it’s not business requirements. And it’s not really laziness (most of the time). It’s not even machismo most of the time (though I will admit the natural arrogance of the geek is probably worth studying by some anthropologist somewhere). There are two root causes, really.
First, we, the networking industry, haven’t really thought through what a control plane actually does. Oh, we have the seven layer model with the control plane thrown off to the side, or the claim that there shouldn’t even be a control plane. But this is part of why I think the seven layer model needs to die — because it’s a host focused view of the networking world. End-to-end and dumb as rocks routers are nice to contemplate, but I think we need to admit that even the dumb rocks are a bit more complex than we first thought.
Second, I don’t think we’ve really incorporated complexity into our souls. As someone once told me, “the CAP theorem is just an observer problem!” Or rather, we somehow believe that by making virtual things we can skip all that ugly physical reality stuff. Faster, cheaper, and better are all three available “on tap,” if we can just figure out how to see the problem right. This is nonsense on stilts.
We need to get in here and do some serious thinking about complexity, and how to manage it in network design. We need to do things like think about interaction surfaces, and how to prevent them from becoming so deep and broad as to be unmanageable. As the article on SQL says, from above —
In a world of regulation and increasing interdependencies between organizations, expressing intent independently of implementation means that you can avoid a class of unintended consequences of systems building.
Where have I heard this before? Oh, maybe it’s in that new book on network complexity someplace.
Seriously — I know this is a long rant, so I’ll quit now, but — seriously (!) we need to grow up and start treating the control plane as an engineering problem. Then, and only then, will we get rid of nerd knobs, no matter whether they’re some hidden CLI command, or some “if/then/else” or “goto” statement hidden someplace in the controller code.
P.S. BTW, Greg, I disagree with you about routing protocols. They’ll “go away” for a short while, until we start trying to deal with networks that don’t run on standards based routing protocols. And then we’ll beg for them to come back. We’ll form something like the IETF, and solve all the same problems all over again, convinced that we can do better than that last group of engineers did. Been there. Done that. Got the t-shirt (someplace).