Intellectual virtue and the engineer

Plane_crash_into_Hudson_River_(crop)On the 19th of January in 2009, Captain “Sully” Sullenberger glided an Airbus A320 into the Hudson River just after takeoff from LaGuardia airport in New York City. Both engines failed due to multiple bird strikes, so the ditching was undertaken with no power, in a highly populated area. Captain Sullenberger could have attempted to land on one of several large highways, but all of these tend to have heavy traffic patterns; he could not make it to any airport with the power he had remaining, so he ditched the plane in the river. Out of the 155 passengers on board, only one needed overnight hospitalization.

There are a number of interesting things about this story, but there is one crucial point that applies directly to life at large, and engineering in detail. Here’s a simple question that exposes the issue at hand—

Do you think the Captain had time to read the manual while the plane was gliding along in the air after losing both engines? Or do you think he just knew what to do?

Way back in the mists of time, a man named Aristotle struggled over the concept of ethics. Not only was he trying to figure out where ethics come from (normative ethics), he was also trying to figure out how to transfer those normative ethics to individual people (aretaic ethics). Aristotle was, above all, a practical man; he wanted to know how any given person could live the “good life,” which he defined as “in accordance with the normative ethics that produce the greatest happiness for an individual.”

What does Aristotle have to do with Captain Sullenberger gliding an airplane into a clean ditch in 2009? Let’s go back to the question just above — do you think he had time to read the manual before ditching that airplane? I’m pretty certain we all know the answer to this question: he didn’t need to read the manual, because he knew what to do. Which leads to the next question: how did he know what to do? As it turns out Captain Sullenberger was a glide instructor, so he not only knew what to do in theory, he’d actually practiced it before. And this, I think, is a crucial point.

What Aristotle posited was that action and belief are not independent “things,” as we often believe, but rather interconnected. As we do things we change our beliefs. As we believe things, we change our actions. The two run in tandem like a Möbius strip, each one supporting the other. This theory of aretaic ethics is called the virtue ethic.

What does any of this have to do with engineering?

Have you ever met an engineer who can quickly assess a network design, seemingly at a glance, tell you where the problems will be, and propose a set of solutions? Have you ever met an engineer who can sit calmly through a huge network outage, looking at the various outputs and figuring out what is wrong, apparently oblivious to the storms surrounding them?

What you’ve witnessed is the virtue ethic in operation. Knowledge builds on experience, and experience builds knowledge. It takes time to build the two, but neither one can exist without the other. To put it in another context, in the shooting world there is a saying: when you’re under pressure, you don’t shoot the way you think, you shoot the way you’ve practiced.

We can apply this concept to our lives in many ways as engineers. To be the engineer who can take in a network “all at once,” and “grock” it, you have to both gain knowledge and practice. The knowledge I’m talking about, by the way, isn’t about command lines; it’s about understanding how networks work at a systemic, integral level. It’s about having a set of mental models you can put around any situation to make sense of it quickly and efficiently — for instance, the OODA loop. It’s about understanding complexity, and all the rest. The virtue ethic says we need to experience and know; that these two things need to reinforce one another.

But how can we learn? Learning isn’t just about reading a few things here and there, or watching the occasional webinar. Learning is, itself, a learned skill — subject to the virtue ethic. You can learn how to learn (a subject for another entire set of posts, and at least part of the point of every other post here).

In fact, the concept of virtue is closely tied to something else that’s always close to the surface of my thinking: culture. One possible definition of a group’s culture is the intertwined surfaces of knowledge and action built up through repetition over time — the virtue ethic.

What defines culture for a group defines culture for you, as well. What are the habits you’re building today? Are you learning to learn? Are you intentionally building a set of practiced skills that will carry you through when there’s no time to read the manual?

What’s your culture? After all, culture eats technology for breakfast.

And you didn’t think Aristotle could teach you anything.

Out with the old: Make removing old technology part of your culture

Friday afternoon, late, and the new system is finally up. Users are logged in, getting their work done, and you’ve just received an email from the CTO (your boss’ boss’ boss’ boss, probably), saying what a good job the team did in getting things up and running so quickly. For once, in fact, the system went in perfectly. There was no close to team breakups over which technology or vendor to use; there were very few unexpected items that crept into the budget, the delays were minimal, and you even learned a couple of new skills to top it all off.

Wonderful, right? The perfect unicorn project.

But before you break open that bottle of bubbly (or whatever cold beverage is your choice), or maybe pop up a bowl of popcorn and sit down to a long deserved break binge watching the shows you missed pulling this thing together, you need to ask one more question:

Did you strip and sand first? Or did you just paint right on top?

Or don’t you remember the time you tried to paint that old trailer that had been sitting in your back yard for ages? Sure, it was covered in rust, dirt, and who knows what else, but the paint said right on the can, “covers anything,” right? Sure, it does, for about fifteen minutes — until you hit the first bump in the road, and all that rust comes loose, flying off in a shower of chips, and making everyone behind you on the highway mad. You know what’s attached to every one of those chips of rust? A little bit of your beautiful new paint, that’s what.

Our networks, today, are testament to the difficulty of translating physical principles into virtual ones. I remember asking a network architect at a major bank once, “what architecture do you use in your data centers?” His reply? “We have every architecture cisco has ever created and sold in some data center someplace.” Or the time I worked on a failed network because they had accrued hundreds of parallel lower speed links, combined with a control plane of mixed routing protocols all munged together with redistribution.

Ugly? Yes.

And of course, you’d never do something like that, right? It takes so little time to layer on a new system, a new protocol, a new virtual topology, a new… And the system keeps running. We can add a new area of the network with a different design over there, partitioning it off with a few aggregates, and forget it even exists.

Here’s the point: in the network engineering world, we need to build a culture of scraping and sanding. Rather than just looking at how we can layer the new onto the old, we need to think about how we can actually remove technologies as we’re adding them. Here’s your challenge:

In every project, don’t just ask how the new stuff is going to work, or how it fits into the network, or what needs to change to support it. Ask what this new technology can replace. Ask what you can get rid of. A conscious part of every project undertaken in the network should include stripping and sanding — thinking through what can be gotten rid of as a part of the project.

We need to make cleaning up — removing old technology — a part of our culture, too.

Information wants to be protected: Security as a mindset

George-Orwell-house-big-brotherI was teaching a class last week and mentioned something about privacy to the students. One of them shot back, “you’re paranoid.” And again, at a meeting with some folks about missionaries, and how best to protect them when trouble comes to their door, I was again declared paranoid. In fact, I’ve been told I’m paranoid after presentations by complete strangers who were sitting in the audience.

Okay, so I’m paranoid. I admit it.

But what is there to be paranoid about? We’ve supposedly gotten to the point where no-one cares about privacy, where encryption is pointless because everyone can see everything anyway, and all the rest. Everyone except me, that is—I’ve not “gotten over it,” nor do I think I ever will. In fact, I don’t think any engineer should “get over it,” in terms of privacy and security. Even if you think it’s not a big deal in your own life, engineers should learn to treat other people’s information with the utmost care.

In moving from the person to the digital representation of the person, we often forget it’s someone’s life we’re actually playing with. I think it’s time for engineers to take security—and privacy—personally. It’s time to actually do what we say we do, and make security a part of the design from day one, rather than something tacked on to the end.

And I don’t care if you think I’m paranoid.

Maybe it’s time to replace the old saying information wants to be free. Perhaps we should replace it with something a little more realistic, like:

Information wants to be protected.

It’s true that there are many different kinds of information. For instance, there’s the information contained in a song, or the information contained in a book, or a blog, or information about someone’s browsing history. Each piece of information has a specific intent, or purpose, a goal for which it was created. Engineers should make their default design such that information is only used for its intended purpose by the creator (or owner) of that information. We should design this into our networks, into our applications, and into our thought patterns. It’s all too easy to think, “we’ll get to security once things are done, and there’s real data being pushed into the system.” And then it’s too easy to think, “no-one has complained, and the world didn’t fall apart, so I’ll do it later.”

But what does it mean to design security into the system from day one? This is often, actually, the hard part. There are tradeoffs, particularly costs, involved with security. These costs might be in terms of complexity, which makes our jobs harder, or in terms of actual costs to bring the system up in the first place.

But if we don’t start pushing back, who will? The users? Most of them don’t even begin to understand the threat. The business folks who pay for the networks and applications we build? Not until they’re convinced there’s an ROI they can get their minds around. Who’s going to need to build that ROI? We are.

A good place to start might be here.

And we’re not going to until we all start nurturing the little security geek inside every engineer, until we start taking security (and privacy) a little more seriously. Until we stop thinking about this stuff as just bits on the wire, and start thinking about it as people’s lives. Until we reset our default to “just a little paranoid,” perhaps.

P.S. I’m not so certain we should get over it. Somehow I think we’re losing something of ourselves in this process of opening our lives to anyone and everyone, and I fear that by the time we figure out what it is we’re losing, it’ll be too late to reverse the process. Somehow I think that treating other people as a product (if the service is free, you are the product) is just wrong in ways we’ve not yet been able to define.

Micromanaging networks considered harmful: on (k)nerd knobs

Nerd Knobs (or as we used to call them in TAC, knerd knobs) are the bane of the support engineer’s life. Well, that and crashes. And customer who call in with a decoded stack trace. Or don’t know where to put the floppy disc that came with the router into the router. But, anyway…

What is it with nerd knobs? Ivan has a great piece up this week on the topic. I think this is the closest he gets to what I think of as the real root cause for nerd knobs —

Instead of using cookie-cutter designs, we prefer to carefully craft unique snowflakes that magically integrate the legacy stuff that should have been dead years ago with the next-generation technologies… and every unique snowflake needs at least a nerd knob or two to make it work.

Greg has a response to Ivan up; again, I think he gets close to the problem with these thoughts —

Most IT managers have lost the ability to recognise technical debt and its impacts … Nerd Knobs are symptoms of much deeper problems/technical debt in the networking market and treat the cause not the symptom.

A somewhat orthogonal article caught my eye, though, that I think explains what is actually going on here with those pesky nerd knobs. The article is really about SQL and the concept of micromanaging software. To give you a flavor (in case you’re too lazy/busy to head over there and read the whole thing) —

So, here’s an analogy that highlights the key difference between what “imperative” languages like Java or Python and “declarative” languages like SQL do to your computation. In Python, say, you specify step-by-step what the computer should do: open the file; read the first line; if the line doesn’t match some requirement, skip it; update the counter; read the next line; update the counter again; if the counter exceeds some value, stop; if the end of file is reached, close the file; return the counter. Code often accumulates like this and builds up into complex business rules that are usually poorly understood. via infoworld

I think this gets to the heart of the nerd knob problem. What’s happening with nerd knobs is it’s easier to tell the system how we want something done than it is to tell the system what we want to do. Think about this way: you install a routing protocol, and you tell it what you want in broad, general terms. Something like, “I want the shortest path between each pair of points in the network.” Then you run into a situation where you need that modified, so you mess around with the metrics some, and get on with your life. Then you run into a situation where you need this flow to go here, and that flow to go there, so you install some policy based routing along the way.

Per link metrics are just the first level of nerd knobs. Policy based routing is just the second. The more precise we want to get, the deeper the nerd knobs go. Want to load share over links that aren’t truly equal cost? Oh, just nerd knob it. Want to send AS’ in the AS path you shouldn’t? Just nerd knob it.

The reality is every nerd knob in routing represents a policy driven by a business requirement expressed as a tweak to the underlying fundamental routing algorithm. As Ivan rightly points out, going to SDNs isn’t going to solve this problem. If anything, it’s going to make it worse. Now, rather than seeing the nerd knob for what it is, a pain in the butt that needs to be explained and dealt with at 2AM when you’re half asleep and the TAC engineer is halfway around the world, it’s going to be “just another line of code.”

This might sound brilliant to someone who hasn’t managed, or dealt with, multi-million line projects and the vagaries of codebase management. Ask someone who has, though, before you get into this. It’s just a different set of problems, not a better set of problems.

The root cause here, though, isn’t nerd knobs. And it’s not business requirements. And it’s not really laziness (most of the time). It’s not even machismo most of the time (though I will admit the natural arrogance of the geek is probably worth studying by some anthropologist somewhere). There are two root causes, really.

First, we, the networking industry, haven’t really thought through what a control plane actually does. Oh, we have the seven layer model with the control plane thrown off to the side, or the claim that there shouldn’t even be a control plane. But this is part of why I think the seven layer model needs to die — because it’s a host focused view of the networking world. End-to-end and dumb as rocks routers are nice to contemplate, but I think we need to admit that even the dumb rocks are a bit more complex than we first thought.

Second, I don’t think we’ve really incorporated complexity into our souls. As someone once told me, “the CAP theorem is just an observer problem!” Or rather, we somehow believe that by making virtual things we can skip all that ugly physical reality stuff. Faster, cheaper, and better are all three available “on tap,” if we can just figure out how to see the problem right. This is nonsense on stilts.

We need to get in here and do some serious thinking about complexity, and how to manage it in network design. We need to do things like think about interaction surfaces, and how to prevent them from becoming so deep and broad as to be unmanageable. As the article on SQL says, from above —

In a world of regulation and increasing interdependencies between organizations, expressing intent independently of implementation means that you can avoid a class of unintended consequences of systems building.

Where have I heard this before? Oh, maybe it’s in that new book on network complexity someplace.

Seriously — I know this is a long rant, so I’ll quit now, but — seriously (!) we need to grow up and start treating the control plane as an engineering problem. Then, and only then, will we get rid of nerd knobs, no matter whether they’re some hidden CLI command, or some “if/then/else” or “goto” statement hidden someplace in the controller code.

P.S. BTW, Greg, I disagree with you about routing protocols. They’ll “go away” for a short while, until we start trying to deal with networks that don’t run on standards based routing protocols. And then we’ll beg for them to come back. We’ll form something like the IETF, and solve all the same problems all over again, convinced that we can do better than that last group of engineers did. Been there. Done that. Got the t-shirt (someplace).

The Silo of Focus

FocusHow often, in our careers, are we told to focus on one thing at a time? I would guess I see some message about this, such as the image to the left in this post, at least once a week, if not once a day.

In general, I agree with the sentiment. If you really want to get something done, do it, rather than doing a lot of things at once. The reason for this, I think, is because multitasked work tends to result in half-work, which is something to be avoided at all costs.

Avoid half-work more than anything. Do not imitate those people who sit long at their desks but let their minds wander. It is better to shorten the time and use it intensely, to increase its value, which is all that counts. Do something, or do nothing at all. Do ardently whatever you decide to do; do it with your might; and let the whole of your activity be a series of vigorous fresh starts. Half-work, which is half-rest, is good neither for rest nor for work. via Sertillanges, The Intellectual Life

But there is another side to focus we need to be wary of as engineers — particularly, in fact, as engineers. It’s easy to become so focused on a single problem that we lose sight of what we were trying to do in the first place. To put it more precise terms, we can focus so hard on something that we end up building a silo. And silos make you stupid, as illustrated in the difference between the Sony Walkman and the Apple iPod (via Financial Review)

There are a number of things a company can do to break up its silo’d structure; this has been an area of study for a number of years, and from a number of different angles. The article in FR, for instance, mentions using anthropology as an entry point in prevent companies with silos. But what about on a more personal level? Of all the silos you can be trapped in, the most dangerous is a silo of one. How can we, as individual engineers, avoid this silo? Some suggestions.

First, stop talking to engineers all the time. I know it’s pretty pleasant to talk to engineers, as they tend to talk the same language as you do. But if all your friends know what interprovider option C means, you need some new friends. Seriously. There are two great places to start in the endeavor: in your actual local community, and in your company. Seeking out people in your community might mean you need to develop some interests other than engineering—but that’s the point, isn’t it? It doesn’t have to be sports, by the way. In my case it’s philosophy, shooting, and a few other things I have an interest in.

Within your company, you need to be intentional about finding reasons to connect with other people. In all honesty, I find LinkedIn a better way to find interesting people inside the companies I work for than the internal directory, or just asking who people are in meetings. Other social media might work better for you—but whatever the means, break out of your department and get to know some people.

Second, read more stuff. I know I’m like a broken record on this score, but, seriously, folks… If the last thing you read that wasn’t either fiction or engineering related was in college or high school, you need to find a library or something. A little economics or philosophy or ethics or religion or… whatever—it’s not going to kill you. Honestly. Check out my 60 books page, and go past the engineering tab. I record most of my book reading on LibraryThing (in fact, looking at my reading you’re going to be startled to find I read very few technical books).

Third, find someone you disagree with and listen to them. I know this is hard—there is so much stuff out there you do agree with that you probably feel like you don’t have time to waste on people you think are complete dolts. But, you see, if you think that way you’ve already succumbed to the silo. Find someone you think is really the opposite of you — politically, religiously, technically, or whatever else — and really try and figure out what they’re saying. Don’t just get halfway through and say, “oh, they’re just an idiot.” If you don’t understand the argument they’re making, or they make logical mistakes, don’t shut down. Work through it, put the best spin you can on what they’re saying, put some time and effort into it.

One of the worst attributes of our social media driven world is the five minute hate cycle we’ve gotten in to—a perfect illustration of focus taken to the level of a silo. We have a horrible habit of calling someone hateful, ignorant, stupid, or evil just because they don’t agree with us. It’s a habit we need to break.

Fourth, be intentional about learning stuff that’s outside your area of expertise. We’ve talked about this before here, so I’m not going to go into detail. Instead, I’ll just point you at that post. Go read it (or read it again, as the case might be).

Don’t get trapped in silos. Definitely don’t get trapped in a silo of one. Silos make you stupid, and the silo of one is the stupidest silo of all.

Worth Learning: The Power Grid

Stop mulling over the latest (now dead) command line, and learn something useful. If you work in networking, you work with electricity. But how many people really know how the power grid works? Even though I have relatives and friends who’ve worked in the power industry all their lives, I’m still learning new things about the grid, and the way it works.

Four items of interest in this area for today.

A really short and simple video

A longer, boring video with lots of presentations and details

An interesting paper on coal to data

An article giving the other side of the renewable hype

Skipping the Hype Cycle

Sipping from the firehose is a big problem in the tech industries. Every time I turn around there’s yet another new technology to make everything better. If you can’t quote rule 11, you need to learn it by heart. Now. I’ll wait right here while you do the memorization thing.

So — why the hype cycle? Essentially, it comes down to this: we’re emotionally driven creatures. Advertisers have known this for years; to wit —

On average, consumers are exposed to more than 5000 advertisements per day. Among those 5000-plus advertisements, only about 12 will make an impression on the average consumer. How can your business stand out among the clutter? Whether you are a Fortune 500 company or a recently funded startup, the best way for your business to stand out is by building emotional connections with your audience. Your business needs to acknowledge that selling a product is no longer enough. Now it’s all about the experience you provide with it. This experience is dependent on your ability to trigger the right emotions, from the right audience, at the right time. via entrepreneur

Mark the last words in that quote: it’s about selling an experience, rather than a product. Then ask yourself this simple question: what — really — is the difference between a unified, hyperconverged, high performance cloud platform and a mainframe? Or, to back a few years, what is really the difference between a layer 3 switch and a router? The answer really lies in one word.


Not how much you have, but the experience of the buyer. What does it feel like to buy this product? Does it make you feel smart (this is what everyone else is doing)? Does it make you feel comfortable (nobody ever gets fired for buying IBM/Cisco/etc.)? Does it make you feel like you’re on the cutting edge? Does it make you, well, just, feel?

We tend to think of the engineering market as somehow detached from this emotional cycle. After all, we’re not talking about the latest in fashion (particularly in a world where anti-fashion is the new fashion, if you can grock that one). We’re not talking about cars. We’re talking about computers, and routers, and switches, and services.

I can just hear you saying, “I’m an engineer! I don’t let my emotions get in the way!” Yeah, right. If there’s anything I’ve learned from my study of philosophy over the last several years, it’s that those who most believe they are least susceptible to emotional manipulation are actually the most likely to fall into focusing on emotions and what advertisers call “the experience.”

Philosophy is a long story of balanced systems being decried for their irrational roots, followed by someone trying to provide rational roots, followed by someone realizing the rational roots are bunk, followed by the collapse of the rational roots into pure unadulterated mysticism with no rational component. If it weren’t so, we might have actually learned something through philosophy by now.

Our rational approach to the problem at hand is often what makes us vulnerable to the emotional manipulation of the hype cycle.

Okay — so how do we get off the hype cycle? To go back to the beginning, rule 11 plays a major role. Let’s look at some basic rules.

First, learn to recognize when technology is being repeated. This means learning the theory (hard as that might be), and learning to see things as a set of problems and solutions that span across many generations of time. Getting there is going to take reshaping some models, and that’s going to hurt. Just keep reading my blog, I have a whole series of posts in these areas. (see what I did there?).

Second, stop looking for the technology that’s going to solve your business problems. There isn’t one.

While new technology might help you get the job done quicker, no amount of shining magic and unicorn dust peddled by vendor marketers will solve your problems until you realize it’s time to change the processes and responsibilities. Ivan P

Third, stop hiring and following people based on the hype cycle. Engineering sense is a real thing, and it’s an important thing, even if the HR system can’t see it. Learn it, develop it, hire it, teach it.

Finally, learn to balance the emotional in the buy with the rational. Look at the underlying problems, look at the available models, and match them together. In other words, be an engineer, not a consumer. There’s a difference for a reason.