Reaction: Devops and Dumpster Fires

Networking is often a “best effort” type of configuration. We monkey around with something until it works, then roll it into production and hope it holds. As we keep building more patches on to of patches or try to implement new features that require something to be disabled or bypassed, that creates a house of cards that is only as strong as the first stiff wind. It’s far too easy to cause a network to fall over because of a change in a routing table or a series of bad decisions that aren’t enough to cause chaos unless done together. —Networking Nerd

Precisely.

But what are we to do about it. Tom’s Take is that we need to push back on applications. This, also, I completely agree with. But this only brings us to another problem—how do we make the case that applications need to be rewritten to work on a simpler network? The simple answer is—let’s teach coders how networks really work, so they can figure out how to better code to the environment in which their applications live. Let me be helpful here—I’ve been working on networks since somewhere around 1986, and on computers and electronics since before then. When I first started in network engineering, I could still wander up in the hills and see Noah’s Ark…

And in all that time, “we,” as in network engineers, have been trying to teach coders how to make their applications run on the network.

Maybe—just maybe—this quest isn’t actually going anyplace. Maybe we convince a few coders here and there. And then they’re replaced by a new generation of coders (just like old network engineers are replaced with new ones every now and again) who never learned those lessons, and want to do really cool stuff, and see the network engineering team as a bunch of old fuddy-duddies who don’t know how to get things done.

The root of this problem isn’t coders. It’s the people who run the businesses coders work for. In most places, IT is just an inconvenience—something I must use to get my “real job” done, rather than an invaluable tool that enables me. Selling X is the focus, IT just gets in the way by making me jump through hoops to get to the information I need to sell X, and by making me put stuff in that I don’t care about once I’ve sold X. In this world, the network is one layer back from the actual system I need to work so I can get my “real work” done, so it’s an enigma wrapped in a painful GUI I hate to use but I must.

This is how networks are really seen by folks who use our systems. Network engineering is a bunch of whiners piled on top of a bunch of folks who make things that make my job harder.

If teaching coders isn’t going to solve the problem, then what do we do?

Russ’ Take

This is what I think we need to do: we need to go to where the money is. Applications aren’t bought by coders, just like networks aren’t. When your manager comes to you and says, “we need a new network,” do they also tell you which gear to buy, and how to configure it? That’s not generally my experience (just remember to keep your managers hermetically sealed off from sales engineers and in flight “CIO” magazines, and your life will be easier). The same applies to coders—when someone says, “I want an application that does this,” they don’t specify how it should work, just that it should. Which means the coder is going to take the shortest of short cuts to make it work, knowing that when it comes down to deployment day, the exec is going to push on the network people to “make it work.”

Why on the network folks? First, we always geek out and say “yes.” Second, we don’t know how to effectively say “no.” We don’t have any sort of language, or process, etc., to say “no” with.

This is what I think we need to change. We need to learn how to talk in terms of tradeoffs and complexity. We need to figure out how to say things like, “sure, I can deploy that, and it won’t cost a penny today—but it will cost you in downtime and operational costs in the future.” As a field, we don’t even have the language for this sort of discussion, much less any way to measure it and make it real. I’ve worked in the area of complexity for several years now because I believe this is where that language is going to come from. I don’t think we’re there yet, but I do think understanding complexity is the right tail to grab when trying to get to this dog. The applications folks already know all about this stuff in their own world; they talk a lot about modularity, APIs, and the like. This isn’t to say they always get it right, but at least they’re talking about it. We’re not even talking about it—instead, we’re building patch on patch, feature on feature, with little thought about the technical debt we’re creating.

Only when we can talk complexity against complexity, technical debt against technical debt, head to head with companies that develop applications, will we begin to truly participate in these conversations.

The network vendors aren’t going to help us here, because they’re keen to sell boxes with the latest features—this is something we must do. There is no calvary off on the horizon.

And then again…

The reality is we shouldn’t need DevOps for configuration at all. This is a bit of a revolution in my thinking in the last two or three years, but what I’m trying to do is to simply make DevOps, as it’s currently constituted, obsolete. DevOps should be about understanding how the network is working and making the network work better, rather than about making the network work in the first place. We need to get to the point where configuration just isn’t something that’s “done” any longer, beyond a few basic points that get things up and running. I know we won’t ever get there, but this is an attitude, not an absolute destination, that keeps me thinking about how to make things simpler.