Reaction: Overly Attached

In a recent edition of ACM Queue, Kate Matsudaira has an article discussing the problem of being overly attached to a project or solution.

The longer you work on one system or application, the deeper the attachment. For years you have been investing in it—adding new features, updating functionality, fixing bugs and corner cases, polishing, and refactoring. If the product serves a need, you likely reap satisfaction for a job well done (and maybe you even received some raises or promotions as a result of your great work).

Attachment is a two-edged sword—without some form of attachment, it seems there is no way to have pride in your work. On the other hand, attachment leads to poorly designed solutions. For instance, we all know the hyper-certified person who knows every in and out of a particular vendor’s solution, and hence solves every problem in terms of that vendor’s products. Or the person who knows a particular network automation system and, as a result, solves every problem through automation.

The most pernicious forms of attachment in the network engineering world are to a single technology or vendor. One of the cycles I have seen play out many times across the last 30 years is: a new idea is invented; this new idea is applied to every possible problem anyone has ever faced in designing or operating a network; the  resulting solution becomes overburdened and complicated; people complain about the complexity of the solution and rush to… the next new idea. I could name tens or hundreds of technologies that have been through this cycle over time.

Another related cycle: a team adopts a new technology in order to solve a problem.

Kate points out some very helpful ways to solve over-attachment at an organizational level. For instance, aligning on goals and purpose, and asking everyone to be open to ideas and alternatives. But these organizational level solutions are more difficult to apply at an individual level. How can this be applied to the individual—to your life?

Perhaps the most important piece of advice Kate gives here is ask for stories, not solutions. In telling stories you are not eliminating attachment but refocusing it. Rather than becoming attached to a solution or technology, you are becoming attached to a goal or a narrative. This accepts that you will always be attached to something—in fact, that it is ultimately healthy to be attached to something outside yourself in a fundamental way. The life that is attached to nothing is ugly and self-centered, ultimately failing to accomplish anything.

Even here, however, there are tradeoffs. You can attach yourself to the story of a company, dedicating yourself to that single brand. To expand this a little, then, you should focus on stories about solving problems for people rather than stories about a product or technology. This might mean telling people they are wrong, by the way—sometimes the best thing is not what someone thinks they want.

Stories are ultimately about people. This is something not many people in engineering fields like to hear, because most of us are in these kinds of fields because we are either introverted, or because we struggle to relate to people in some other way.

To expand this a bit more, you should be willing to tell multiple stories, rather than just one. These stories might overlap or intersect, of course—I have been invested in a story about reducing complexity, disaggregation, and understanding why rather than how for the last ten or fifteen years. These three stories are, in many ways, the same story, just told from different perspectives. You need to allow the story to be shaped, and the path to tell that story to change, over time.

Realize your work is neither as bad as you think it is, nor as good as you think it is. Do not take criticism personally. This is a lesson I had to learn the hard way, from receiving many manuscripts back covered in red marks, either physical or virtual. Failure is not an option; it is a requirement. The more you fail, the more you will actively seek out the tradeoffs, and approach problems and people with humility.

Finally, you need to internalize modularity. Do not try to solve all the problems with a single solution, no matter how neat or new. Part of this is going to go back to understanding why things work the way they do and the limits of people (including yourself!). Solve problems incrementally and set limits on what you will try to do with any single technology.

Ultimately, refusing to become overly attached is a matter of attitude. It is something that is learned through hard work, a skill developed across time.

meeting madness, ONUG, and software defined

Jordan, Eyvonne, and I sit down for a conversation that begins with meetings, and ends with talking about software defined everything (including meetings??).

Outro Music:
Danger Storm Kevin MacLeod (
Licensed under Creative Commons: By Attribution 3.0 License

Throwing the baby out with the bathwater (No, you’re not Google, but why does this matter?)

It was quite difficult to prepare a tub full of bath water at many points in recent history (and it probably still is in some many parts of the world). First, there was the water itself—if you do not have plumbing, then the water must be manually transported, one bucket at a time, from a stream, well, or pump, to the tub. The result, of course, would be someone who was sweaty enough to need the forthcoming bath. Then there is the warming of the water. Shy of building a fire under the tub itself, how can you heat enough water quickly enough to make the eventual bathing experience? According to legend, this resulted in the entire household using the same tub of water to bathe. The last to bathe was always the smallest, the baby. By then, the water would be murky with dirt, which means the child could not be seen in the tub. When the tub was thrown out, then, no-one could tell if the baby was still in there.

But it doesn’t take a dirty tub of water to throw the baby out with the bath. All it really takes is an unwillingness to learn from the lessons of others because, somehow, you have convinced yourself that your circumstances are so different there is nothing to learn. Take, for instance, the constant refrain, “you are not Google.”

I should hope not.

But this phrase, or something similar, is often used to say something like this: you don’t have the problems of any of the hyperscalers, so you should not look to their solutions to find solutions for your problems. An entertaining read on this from a recent blog:

Software engineers go crazy for the most ridiculous things. We like to think that we’re hyper-rational, but when we have to choose a technology, we end up in a kind of frenzy — bouncing from one person’s Hacker News comment to another’s blog post until, in a stupor, we float helplessly toward the brightest light and lay prone in front of it, oblivious to what we were looking for in the first place. —Oz Nova

There is a lot of truth here—you should never choose a system or solution because it solves someone else’s problem. Do not deploy Kafka if you you need the scale Kafka represents. Maybe you don’t need four links between every pair of routers “just to be certain you have enough redundancy.”

On the other hand, there is a real danger here of throwing the baby out with the bathwater—the water is murky with product and project claims, so just abandon the entire mess. To see where the problem is here, let’s look at another large scale system we don’t think about very much any longer: the NASA space program from the mid-1960’s. One of the great things the folks at NASA have always liked to talk about is all the things that came out of the space program. Remember Tang? Or maybe not. It really wasn’t developed for the space program, and it’s mostly sugar and water, but it was used in some of the first space missions, and hence became associated with hanging out in space.

There are a number of other inventions, however, that really did come directly out of research into solving problems folks hanging out in space would have, such as the space pen, freeze-dried ice cream, exercise machines, scratch-resistant eyeglass lenses, cameras on phones, battery powered tools, infrared thermometers, and many others.

Since you are not going to space any time soon, you refuse to use any of these technologies, right?

Do not be silly. Of course you still use these technologies. Because you are smart enough not to throw the baby out with the bathwater, right?

You should apply the same level of care to the solutions Google, Amazon, LinkedIn, Microsoft, and the other hyperscalers. Not everything is going to fit in your environment, of course. On the other hand, some things might fit. And regardless of whether any particular technology fits or not, you can still learn something about how systems work by considering how they are building things to scale to their needs. You can adopt operational processes that make sense based on what they have learned. You can pick out technologies and ways of thinking that make sense.

No, you’re (probably not) Google. On the other hand, we are all building complex networks. The more we can learn from those around us, the better what we build will be. Don’t throw the baby out with the bathwater.

About that Easy Button …

We love layers and abstraction. After all, building in layers and it’s corollary, abstraction, are the foundation of large-scale system design. The only way to build large-scale systems is to divide and conquer, which means building many different component parts with clear and defined interaction surfaces (most often expressed as APIs) and combining these many different parts into a complete system. But abstraction, layering, and modularization have negative aspects as well as positive ones. For instance, according to the State/Optimization/Surface triad, any time we remove state in order to control complexity, we either add an interaction surface (which adds complexity) or we reduce optimization.

Another impact of abstraction, though, is the side effect of Conway’s Law: “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.” The structure of the organization that designs a system is ultimately baked into the modularization, abstraction, and API schemes of the system itself.

To take a networking instance, many networks use one kind of module for data centers and another for campuses. The style of network built in each place, where the lines are between these different topological locations in the network, the kind of policies expressed at the borders of these networks, and many other factors are baked into the network design based on the organizational design. This is all well and good within an organization, but what happens when you reach outside the organization and purchase hardware and software from another company to build your network?

When you buy systems designed by other organizations, you are importing their organizational structure into your organization. A corollary: the more vertically integrated a system is, or more aggregated, the more of the external organization’s structure you are importing into your organization. It seems like purchasing a strongly integrated system is “pressing the easy button,” but the result is often actually a mess of tech debt from trying to make the organizing principles of the purchased system fit with the internal logic of your organization.

This is why, for instance, some software developers advocate not using open source libraries and frameworks in building internal projects. The idea sounds radical, contrary to the direction of the entire tech culture. The easy button calls; the network is just an undifferentiated bit moving machine, just give me a black box and two lines of configuration, and I’m happy. I don’t want to know how it works.

Until it doesn’t. Or until it doesn’t do what you want for the thousandth time, and you add the thousandth line of configuration to tweak something, or you run into the thousandth assumption in the way the black box works that you just need to change to make your business run better. When you look at the system architecture as a reflection of organizational structure, these little mismatches begin to make sense.

Lessons for the network? It isn’t really possible to build all the bits and pieces you need to build a network. You are not going to be able to write your own network operating system. You are not going to be able to (from scratch) build your own control plane.

But you can be more careful about the organizational structures you are importing. You can understand the internal components and how they connect. You can understand the problems being solved, the general solution being implemented, and how these solutions ultimately fit together to make a whole.
You can insist on having access to the same APIs the system developers use. You can minimize the number of system elements you rely on, such as protocols and nerd knobs. You can, ultimately, disaggregate, treating your software and hardware as two different “things,” each of which has its own lifecycle, purposes, and value.

This is as much about mindset as anything else; the easy button is an abstraction. Abstractions are abstracting something. Don’t just watch the vendor presentation and gawk; dig in and understand.

Choose Simple Solutions

In my experience, simplicity is not valued enough in software development. Instead, there is a lot of emphasis placed on flexibility. —Felix

Replace “software” with “network,” and think about it. How often do network engineers select the chassis-based system that promises to “never need to be replaced?” How often do we build networks like they will be “in use” 20+ years from now? Now it does happen from time to time; I have heard of devices with many years of uptime, for instance. I have worked on AT&T Brouters in production—essentially a Cisco AGS+ rebranded and resold by AT&T—that were some ten or fifteen years old even back when I worked on them. These things certainly happen, and sometimes they even happen for good reasons.

But knowing such things happen and planning for such things to happen are two different mindsets. At least some of the complexity in networks comes from just this sort of “must make it permanent: thinking:

Many developers like to write code which handles any problem which might appear at any point in the future. In that regard, they are fortune tellers, trying to find a solution for eventual problems. This can work out very well if their predictions are right. Most of the time, however, this flexibility only causes unneeded complexity in the code which gets in the way and does not actually solve any problems. This is not surprising as telling the future is a messy business.

Let’s refactor: many network engineers like to build networks that can handle any problem or application that might appear at any point in the future. I know I’m close to the truth, because I’ve been working on networks since the mid- to late-1980’s.

So now you are reading this and thinking: “but it is important to plan for the future.” You are not wrong—but there is a balance that often is not well thought out. You should not build for the immediate problem ignoring the future; down this path leads technical debt. You should not plan for the distant future, because this injects complexity that does not need to be there.

How do you find the balance? The place to begin is knowing how things work, rather than just how to make them work. If you know how and why things work, then you can see what things might last for a long time, and what might change quickly.

When you are designing a protocol, does it make sense to use TLVs rather than fixed length fields? Protocols last for 20+ years and are used across many different network devices. Protocols are often extended to solve new problems, rather than being replaced wholesale. Hence, it makes sense to use TLVs.

When you are designing a data center or campus network, does it make sense to purchase chassis boxes that are twice as large as you foresee needing over the next three years to future proof the design? Hardware changes are likely to make a device more than three years old easier to replace than upgrade—if you can even get the parts you need in three years. Hence, it makes more sense to plan for the immediate future and leave the crystal ball gazing to someone else.

If you haven’t found the tradeoffs, then you haven’t looked hard enough.

But to look hard enough, you need to go beyond the hype and “future proofing,” beyond how to make things work. You need to ask how and why things work the way they do so you know where to accept complexity to make the design more flexible, and where to limit complexity by planning for today.

Social Media, Limits, and Productivity

If there is one question I get most often, it is “how do you get so much done?” One answer to this question is: I limit my use of social media. There is, another angle to social media use which is a bit more… philosophical.

Some of you might know that I am currently working on a PhD in Philosophy—which might seem like an odd thing to do for someone who has been in the engineering world for, well, pretty much my entire life. My particular area of study, however, is what might be called media ecology and humanness. How do these two interact? What impact does, for instance, social media have on things like human freedom and dignity?

Social media (and mediated reality in general) has a bad habit of making people into objects—objectification is just part of the mediation process. If you go “all in” to the mediated world, then you become wholly mediated. This is ultimately dehumanizing, and a very bad thing.

Returning to the first question I raised above: what impact does social media have on my use of time? Does it make me more or less productive?

If we think social media does have a negative impact on our use of time, and even on humanness, what do we do about it? Should I just drop out? This does not seem like a very useful answer; there are too many benefits from social media, and it has become too embedded in our lives to just drop out. What about building “trusted gatekeepers” to manage what people can, and cannot, say? This, it seems to me, leads to a place where we don’t really want to go.

In answer to all of this, I would suggest putting personal rules in place about social media. Let me tell you what I’ve adopted (although I’m pretty flexible about them).

First, I have a daily time limit. I have a maximum amount of time I will spend across all social media sites each day. I’m not going to divulge my limit, because I think it is probably best left to each person to decide what their limits should be, but I also think having a limit is important. Things like social media can be addicting even if not designed to be (and it is, generally designed to be addictive). One of the best ways to fight addiction is to set clear limits of some kind and stick to them. I’ve chosen time as my limit for the moment, though others may make themselves obvious later.

Second, I have specific topical limits. If you want to talk about religion, politics, or the meaning of life, I’m perfectly happy to do so—but only in person (and not recorded). I do not have any interest in slogging out the Christian versus Islamic versus naturalistic concepts of personhood in 180 character microblogs. Nor even in the messaging section of a social media site. If you want to talk about these things, find me in person. Likewise, I don’t talk about what I had for dinner, where my last family vacation was, etc.

If you have different limits, or think this is an interesting topic, I do have the comments section open on this post (eventually I want to move to a platform that has no comment section, but this is turning out to be harder than I thought it would be).

Or… you can find me in person at Interop, CHINOG, the IETF, NXTWORK, and a number of other places… 🙂

Bridging the Gap

Mike Bushong and Denise Donohue join Eyvonne, Jordan, and I to discuss the gap between network engineering and “the business,” and give us some thoughts on bridging it.

Outro Music:
Danger Storm Kevin MacLeod (
Licensed under Creative Commons: By Attribution 3.0 License