Making Networking Cool Again? (2)

Network engineering is not “going away.” Network engineering is not less important than it was yesterday, last year, or even a decade ago.

But there still seems to be a gap somewhere. There are fewer folks interested than we need. We need more folks who want to work as full-time network engineers, and more folks with network engineering skills diffused within the larger IT community. More of both is the right answer if we’re going to continue building large-scale systems that work. The real lack of enthusiasm for learning network engineering is hurting all of IT, not just network engineering.

How do we bridge this gap? We’re engineers. We solve problems. This seems to be a problem we might be able to solve (unlike human nature). Let’s try to solve it.

As you might have guessed, I have some ideas. These are not the only ideas in the world—feel free to think up more!

If you walk into a robotics class, even an introductory robotics class, you see people … building robots. If you walk into a coding class, even an introductory one, you see people … writing software. If you walk into a network network engineering class you see … someone lecturing about the OSI model, packet formats, or how to configure BGP.

What problems are people learning to solve robotic engineering? How to build a robot and get it to do something to solve a real-world problem. What problems are people learning to code solving? How to tackle some real-world problem.

Sure, the problems being solved at an introductory level might be trivial, like: “Read this file and spit out a sum of the numbers in the fourth column.” But they are still starting, right from the beginning, by taking requirements and converting them into solutions.

What problems are network engineers learning how to solve? How to choose hardware, string it all together, and configure BGP.

Do you see the difference?

All engineers solve problems—it’s the nature of engineering. But are we creating a mindset in prospective network engineers, or even adjacent fields, that we solve real-world problems? Or are we giving them the impression that we solve whiteboard problems by talking about bits, bytes, configurations, and cable types?

Have you ever seen the glazing over of eyes while explaining how you put four transport protocols on top of one another (look at all the pretty tunnels)? How about when you create a chart showing how TCP and QUIC can be “kind-of sort-of” forced into the OSI model? Or when you spin out your BGP packet format charts, showing how we’ve (mis)used address families to carry everything anyone can imagine?

I’ve been teaching this stuff for years (okay, decades). Over time, I’ve moved away from teaching configurations and packet formats. I’ve gone from Advanced IP Network Design to Computer Networking Problems and Solutions. These are very different ways of looking at network engineering.

Focusing on real-world problems would help connect business and other IT folks to the network, connect theory to practice, and people to network engineering. Going home at the end of the day saying, “I solved a problem,” can be satisfying. Going home at the end of the day saying, “I configured BGP?”

Another thing adopting the mindset of solving real-world problems might do is help us lose unnecessary complexity. I know complexity is necessary to build resilient systems; we cannot build what we build without creating and encountering complexity.

But we often run ourselves into the ditch on both sides of the road.

We unintentionally build too complex because we try to make it too simple. Quick, which is simpler: building a data center fabric with one routing protocol or two? A single chassis system or several smaller fixed format devices? A proprietary system or something built on open standards?

How many balloons fit in a bag? (thanks, Don)

Failing to start with the tradeoffs, and thinking through what problem we’re actually trying to solve, leads to unnecessary complexity. Such designs might not immediately fail, but they will fail, and “it’s so complex” just isn’t an excuse.

Don’t even try to tell me there aren’t any tradeoffs. If you think there aren’t any tradeoffs, that just means you haven’t looked hard enough. Go find them, think about them, and document them.

We also build complex things because we think it offers job security, or it’s neat, or we like to feel like the kid who says to the world, “look what I built!”

I know it’s exciting to hear stories about that time someone rescued a network from a major failure—after all, that’s solving a real-world problem. Building a network that just works might be “boring,” but it solves many more real-world problems than raising a network from the dead.

We love our fashionable capes, but … capes can get caught in a nearby jet engine. Lose the cape. In the long run, it’ll make network engineering more attractive as a career field and field of knowledge.

The Bottom Line

No, the sky is not falling. We still need networks, and we still need network engineers.

Yes, there is a problem. Too many companies are going “to the cloud” because they cannot find people qualified to build and maintain their very complex networks. There’s too much centralization and too little oppeness.

So maybe let’s stop saying “we don’t need network engineers.” And maybe let’s really think about how we’re building things. And maybe let’s focus on solving real-world problems, starting from day one in network engineering classrooms.

Network engineering is still cool—let’s go out there believing—and selling—that idea to the world.

Hedge 206: Taking Care of Yourself with Ethan Banks

As we reach the end of what has been a hard two-year stretch for what seems like the entire world, Ethan Banks joins Tom, Eyvonne, and Russ to talk about the importance of taking care of yourself. In the midst of radical changes, you can apply self-discipline to make your little part of the world a better place by keeping yourself sane, fit, and well-rested.




Mean Time to Innocence is not Enough

A long time ago, I supported a wind speed detection system consisting of an impeller, a small electric generator, a 12 gauge cable running a few miles, and a voltmeter. The entire thing was calibrated through a resistive bridge–attach an electric motor to the generator, run it at a series of fixed speed, and adjust the resistive bridge until the voltmeter, marked in knots of wind speed, read correctly.

The primary problem in this system was the several miles of 12 gauge cable. It was often damaged, requiring us to dig the cable up (shovel ready jobs!), strip the cable back, splice the correct pairs together, seal it all in a plastic container filled with goo, and bury it all again. There was one instance, however, when we could not get the wind speed system adjusted correctly, no matter how we tried to tune the resistive bridge. We pulled things apart and determined there must be a problem in one of the (many) splices in the several miles of cable.

At first, we ran a Time Domain Reflectometer (TDR) across the cable to see if we could find the problem. The TDR turned up a couple of hot spots, so we dug those points up … and found there were no splices there. Hmmm … So we called in a specialized cable team. They ran the same TDR tests, dug up the same places, and then did some further testing and found … the cable was innocent.

This set up an argument, running all the way to the base commander level, between our team and the cable team. Who’s fault was this mess? Our inability to measure the wind speed at one end of the runway was impacting flight operations, so this had to be fixed. But rather than fixing the problem, we were spending our time arguing about who’s fault the problem was, and who should fix it.

When I read this line in a recent CAIDA research paper–

“Measurement is political, and often adversarial.”

It rang very true. In Internet terms, speed, congestion, and even usage are often political and adversarial. Just like the wind speed system, two teams were measuring the same thing to prove the problem wasn’t their’s–rather than to figure out what the problem is and how to fix it.

In other words, our goal is too often Mean Time to Innocence (MTTI), rather than Mean Time to Repair (MTTR).

MTTI is not enough. We need to work with our application counterparts to find and fix problems, rather than against them. Measurement should not be adversarial, it should be cooperative.

We need to learn to fix the problem, not the blame.

This is a cultural issue, but it also impacts the way we do telemetry. For instance, in the case of the wind speed indicator, the problem was ultimately a connection that “worked,” but with high capacive reactance such that some kinds of signals were attenuated while others were not. None of us were testing the cable using the right kind of signal, so we all just sat around arguing about who’s problem it was rather than solving the problem.

When a user brings a problem to you, resist the urge to go prove yourself–or your system–innocent. Even if your system isn’t the problem, your system can provide information that can help solve the problem. Treat problems as opportunities to help rather than as opportunies to swish your superhero cape and prove your expertise.

Hedge 153: Security Perceptions and Multicloud Roundtable

Tom, Eyvonne, and Russ hang out at the hedge on this episode. The topics of discussion include our perception of security—does the way IT professionals treat security and privacy helpful for those who aren’t involved in the IT world? Do we discourage users from taking security seriously by making it so complex and hard to use? Our second topic is whether multicloud is being oversold for the average network operator.


On Building a Personal Brand

How do you balance loyalty to yourself and loyalty to the company you work for?

This might seem like an odd question, but it’s an important component of work/life balance many of us just don’t think about any longer because, as Pete Davis says in Dedicated, we live in a world of infinite browsing. We’re afraid of sticking to one thing because it might reduce our future options. If we dedicate ourselves to something bigger than ourselves, then we might lose control of our direction. In particular, we should not dedicate ourselves to any single company, especially for too long. As a recent (excellent!) blog post over at the ACM says:

Loyalty is generally a good trait, but extreme loyalty to the organization or mission may cause you to stay in the same job for too long.

The idea that we should control our own destiny, never getting lost in anything larger than ourselves, is ubitiquos like water is to a fish. We don’t question it. We don’t argue. It is just true. We assume there are three people who are going to look after “me:” me, myself, and I.

I get it. Honestly, I do. I’ve been there more times than I want to think about. I was the scapegoat in an argument between people far above my pay grade early in my career, causing much angst and pain. I’ve been laid off,—I cared about a company that simply didn’t care about me. Most recently, the family I’d dedicated more than twenty years of my life to ended through a divorce.

I can see why you might ask yourself hard questions about dedicating yourself to anything or anyone.

The problem, as Pete Davis points out, is that the human person was not designed for the kind of digital nomad life represented by the phrase “live for yourself.” We can try to substitute an online community. We can try to replace community with a string of novel experiences. But the truth is it will eventually catch up with you. When you’re young it’s hard to see how it will ever catch up with you, but it will.

Returning to the top—the author of the ACM article advises balancing between dedicating yourself to a company and dedicating yourself to your career. This is wise advice, but it leaves me wondering “how?” Let me lay out some thoughts here. They may not be all of the answer, but they will, I hope, point in the right direction.

First, resist seeing these two choices as orthogonal. They might be at odds in some companies—there are publishers who want your content to build their brand, and they specifically work at preventing you from building your brand. There are companies that explicitly want to own “your whole professional life.” They don’t want you blogging, going to conferences to speak, etc. Avoid these companies.

Instead, find companies that understand your personal brand is an asset to the company. Having a lot of people with strong personal brands in a company makes the company stronger, not weaker. People with strong brands will form communities around themselves. This community is a pool of people from which to recruit top-flight talent. This community allows them to collect new ideas that can be directly applied to problems in the organization. People with strong personal brands will have greater influence when they walk into a room to meet with a customer, a supplier, or just about anyone else. A company full of people with strong personal brands is stronger than one where everyone is faceless, consumed by/hiding behind the company logo.

Second, learn to manage your time effectively. I understand it’s possible to spend so much time building your brand that you don’t get your job done. As an individual, you need to be sensitive to this and learn how to manage your time effectively.

Third, seek out the win/win. Don’t think of every situation through the lens of “it’s either my brand or my employer’s.” There may be times when you cannot do something because, while it would help your brand immensely, it would harm your company’s. There may be times where you need to have a delicate discussion with your manager because you’ve been asked to do something that would be great for the company but would harm your brand. There is almost always a win/win, you just have to find it.

Fourth, seek out a community that’s not attached to work and dedicate yourself to it. Find something larger than yourself. A community that’s not tied to work will be your lifeline when things go wrong.

Finally, expect to get hurt. I know I have (an old saying in my community—never trust a man who doesn’t walk with a limp). You can be the nicest, humblest person in the world. Someone is still going to take advantage of you. In fact, the nicer and humbler you are—the more you care, the more likely it is people are going to take advantage of you. I am amazed at how much people seem to enjoy hurting one another when they believe there won’t be any consequences.

But … if you expect your life to be perfect, you were born in the wrong world. Build up the mental reserves to deal with this. Build a community that will help carry you through. There is nothing better than sitting down and sharing your hurt over a cup of coffee with a good friend (except I don’t drink coffee).

I get it—the world has moved into a YOLO/FOMO phase. If you don’t “grab it,” and right now! you risk missing something really important. We pile up alternative possibilities in our minds, wondering what might have happened if we’d chosen otherwise. We have deep angst over our personal brand, overthinking the concept to the point of diminishing returns.

The solution, though, is not to draw into yourself, to become self-centered. The solution is to find the balance, seek the win/win, dedicate yourself to something bigger than yourself, and find the right way to build your personal brand.

Quality is (too often) the missing ingredient

Software Eats the World?

I’m told software is going to eat the world very soon now. Everything already is, or will be, software based. To some folks, this sounds completely wonderful, but—leaving aside the privacy issues—I still see an elephant in the room with this vision of the future.


Let me give you some recent examples.

First, ceiling fans. Modern ceiling fans, in case you didn’t know, don’t rely on the wall switch and pull chains. Instead, they rely on remote controls. This is brilliant—you can dim the light, change the speed of the fan, etc., from a remote control. No unsightly chains hanging from the ceiling.

Well, it’s brilliant so long as it works. I’ve replaced three of the four ceiling fans in my house. Two of the remote controls have somehow attached themselves to two of the three fans. It’s impossible to control one of the fans without also controlling the other. They sometimes get into this entertaining mode where turning one fan off turns the other one on.

For the third one—the one hanging from a 13-foot ceiling—the remote control sometimes operates one of the other fans, and sometimes the fan its supposed to operate. Most of the time it doesn’t seem to do much of anything.

The fan manufacturer—a large, well-known company—mentions this situation in their instructions and points to a FAQ that doesn’t exist. Searching around online I found instructions for solving this problem that involve unwiring the fans and repeating a set of steps 12 times for each fan to correct the situation. These instructions, needless to say, don’t work.

There is no way to reset the remote, nor the connection between the remote and the fan. There is no way to manually select some dip switch so the remote has a specific fan it talks to. Just some mystical software that’s supposed to work (but doesn’t) and no real instructions on how to resolve the problem. The result will be a multi-hour wait on a customer support line, spending hours of my time to sort the problem out, and the joy of climbing (tall) ladders to unwire and wire ceiling fans in four different rooms.

Thinking through possible problems and building software interfaces that take those situations into account … might be a bit more important than we think they are if software is really going to eat the world.

Second, the retailer’s web site—a large retailer with thousands of physical stores across the United States. Twice I’ve ordered from this site, asking to have the item held in the local store so I can pick it up. The site won’t let you order the item for store pickup unless they have it in stock.

The first time they called me to say they couldn’t find the item I ordered, but they found a “newer model” that was a lot less expensive. It was a lot less expensive because it wasn’t the same item. They never did find the item I originally ordered.

The second time they called me to say they couldn’t find the item I ordered. I asked if they could just ship the item to my house when it’s back in stock. “I’m sorry, our system doesn’t allow us to do that …” Several hours later, they called back to tell me they found it, but they cannot reinstate my order—I must place a new order.

Again, software quality strikes … what should be a simple process just isn’t. There will always be mismatches between the state in software and the state in the real world—but design the system so it’s possible to adapt when this happens, rather than shutting down the process and starting over.

Third, I own a car that has all the “bells and whistles,” including an adaptive cruise control system. There are certain situations, however, where this adaptive control does the wrong thing, producing potentially dangerous results. There is no way to set the car to use the non-adaptive cruise control permanently (I called and waited on the phone for several hours to discover this). You can set the non-adaptive cruise control on a per-use basis by going through set of menus to change the settings … while driving.

Software quality anyone?

Software eats the world might be someone’s ultimate dream—but I suspect that software quality will always be the fly in the ointment. People are not perfect (even in crowds); software is created by people; hence software will always suffer from quality problems.

Maybe a little humility about our ability to make things as complex as we might like because “we can always have software do that bit” would be a good thing—even in the networking world.