Reaction: Devops and Dumpster Fires

Networking is often a “best effort” type of configuration. We monkey around with something until it works, then roll it into production and hope it holds. As we keep building more patches on to of patches or try to implement new features that require something to be disabled or bypassed, that creates a house of cards that is only as strong as the first stiff wind. It’s far too easy to cause a network to fall over because of a change in a routing table or a series of bad decisions that aren’t enough to cause chaos unless done together. —Networking Nerd

Precisely.

But what are we to do about it. Tom’s Take is that we need to push back on applications. This, also, I completely agree with. But this only brings us to another problem—how do we make the case that applications need to be rewritten to work on a simpler network? The simple answer is—let’s teach coders how networks really work, so they can figure out how to better code to the environment in which their applications live. Let me be helpful here—I’ve been working on networks since somewhere around 1986, and on computers and electronics since before then. When I first started in network engineering, I could still wander up in the hills and see Noah’s Ark…

And in all that time, “we,” as in network engineers, have been trying to teach coders how to make their applications run on the network.

Maybe—just maybe—this quest isn’t actually going anyplace. Maybe we convince a few coders here and there. And then they’re replaced by a new generation of coders (just like old network engineers are replaced with new ones every now and again) who never learned those lessons, and want to do really cool stuff, and see the network engineering team as a bunch of old fuddy-duddies who don’t know how to get things done.

The root of this problem isn’t coders. It’s the people who run the businesses coders work for. In most places, IT is just an inconvenience—something I must use to get my “real job” done, rather than an invaluable tool that enables me. Selling X is the focus, IT just gets in the way by making me jump through hoops to get to the information I need to sell X, and by making me put stuff in that I don’t care about once I’ve sold X. In this world, the network is one layer back from the actual system I need to work so I can get my “real work” done, so it’s an enigma wrapped in a painful GUI I hate to use but I must.

This is how networks are really seen by folks who use our systems. Network engineering is a bunch of whiners piled on top of a bunch of folks who make things that make my job harder.

If teaching coders isn’t going to solve the problem, then what do we do?

Russ’ Take

This is what I think we need to do: we need to go to where the money is. Applications aren’t bought by coders, just like networks aren’t. When your manager comes to you and says, “we need a new network,” do they also tell you which gear to buy, and how to configure it? That’s not generally my experience (just remember to keep your managers hermetically sealed off from sales engineers and in flight “CIO” magazines, and your life will be easier). The same applies to coders—when someone says, “I want an application that does this,” they don’t specify how it should work, just that it should. Which means the coder is going to take the shortest of short cuts to make it work, knowing that when it comes down to deployment day, the exec is going to push on the network people to “make it work.”

Why on the network folks? First, we always geek out and say “yes.” Second, we don’t know how to effectively say “no.” We don’t have any sort of language, or process, etc., to say “no” with.

This is what I think we need to change. We need to learn how to talk in terms of tradeoffs and complexity. We need to figure out how to say things like, “sure, I can deploy that, and it won’t cost a penny today—but it will cost you in downtime and operational costs in the future.” As a field, we don’t even have the language for this sort of discussion, much less any way to measure it and make it real. I’ve worked in the area of complexity for several years now because I believe this is where that language is going to come from. I don’t think we’re there yet, but I do think understanding complexity is the right tail to grab when trying to get to this dog. The applications folks already know all about this stuff in their own world; they talk a lot about modularity, APIs, and the like. This isn’t to say they always get it right, but at least they’re talking about it. We’re not even talking about it—instead, we’re building patch on patch, feature on feature, with little thought about the technical debt we’re creating.

Only when we can talk complexity against complexity, technical debt against technical debt, head to head with companies that develop applications, will we begin to truly participate in these conversations.

The network vendors aren’t going to help us here, because they’re keen to sell boxes with the latest features—this is something we must do. There is no calvary off on the horizon.

And then again…

The reality is we shouldn’t need DevOps for configuration at all. This is a bit of a revolution in my thinking in the last two or three years, but what I’m trying to do is to simply make DevOps, as it’s currently constituted, obsolete. DevOps should be about understanding how the network is working and making the network work better, rather than about making the network work in the first place. We need to get to the point where configuration just isn’t something that’s “done” any longer, beyond a few basic points that get things up and running. I know we won’t ever get there, but this is an attitude, not an absolute destination, that keeps me thinking about how to make things simpler.

6 Comments

  1. rjhintz on 19 September 2016 at 2:29 pm

    > The reality is we shouldn’t need DevOps for configuration at all. This is a bit of a revolution in my thinking in the last two or three years, but what I’m trying to do is to simply make DevOps, as it’s currently constituted, obsolete. DevOps should be about understanding how the network is working and making the network work better, rather than about making the network work in the first place.

    I really don’t understand this. DevOps is cooperation among groups from Dev, Ops, Security, Storage, Networking, Compliance (and any other group that gates the application to production status) to create and deploy applications that meet business goals. I’m using the term DevOps as generally understood (Gene Kim, John Willis, Damon Edwards), so maybe there’s a variant meaning I’m not thinking of.

    So, yes, I’d say under all circumstances we need DevOps practices for network configuration as part of the application moving from development to deployment. What needs to be made obsolete and why?



    • Russ on 20 September 2016 at 1:35 pm

      Thanks for stopping by — I’m going to answer this in a separate post. ? Russ



  2. Evil CCIE on 20 September 2016 at 3:20 am

    While I have very basic understanding of Dev OPS world, would like you to throw some more light on:

    – How understanding more about networks should help Dev OPS guys. Can you share few examples around it.

    – I agree that network runs applications and it’s important for network engineers to understand at least some basic things to begin with

    : What are end to end requirements in terms of latency, jitter etc
    : How application recover in event of failure or failures
    : How application redundancy is planned etc

    But talking about application aware networking, doesn’t it add it’s own sets of complexities into the network and suddenly overhead grows ?

    Don’t get me wrong but in couple of years I spent working in industry, not many times I find application aware devices like IPS/IDS are well deployed simply because:

    : Security Engineers/network engineers don’t understand applications and it’s behavior very well
    : You need to have good assessment and review processes in place
    : Old age NMS were not very well designed or build to give you granular information around application and it’s statistics

    I haven’t dealt much with application developers per say, but most of the application admins usually don’t have much details on application internals in my personal exp.



    • Russ on 20 September 2016 at 1:35 pm

      Thanks for stopping by — I’m going to answer this in a separate post. 🙂 Russ



  3. Mohammad Moghaddas on 20 September 2016 at 3:28 am

    ​I found it interesting that you wrote about this topic. I’m in full agreement with the whole post, however just wanna add some points about the first part.

    I remember during last IETF, we talked about the need for Network Engineers to learn coding and how it can be helpful. I brought up the point that Developers need more to learn networking as most of the time it’s their lack of familiarity of how networks work (though personally I still don’t know it myself :D) and actually any Network Engineer I know of, is at least aware of some coding and algorithms fundamentals, however this also needs much more improvement.

    Perhaps we should change something in the job market.
    Taking a look into Networking job postings, it has became common asking for programming skills. Why not wanting Developers to have some Networking skills? Maybe we, as network guys, should discuss such options with CIOs and HR people…

    In a world where both parties understand each other, there certainly is positive effects on the business, besides an easier life for both networkers and developers, and more efficiency in their daily work. I wish to live in that world 😉
    Thanks.​​



    • Russ on 20 September 2016 at 1:22 pm

      Thanks for stopping by and commenting — I would agree that we need to get coders to understand networks much better — but I don’t think they’re driven to do so because of the way the business is built right now. What we need to do is somehow change the business, rather than just expecting coders to go out and learn this stuff.