Reaction: The Pace of Innovation

Dave Ward has an excellent article over at the Cisco blog on the three year journey since he started down the path of trying to work the standards landscape (called SDOs) to improve the many ways in which these organizations are broken. Specifically, he has been trying to connect the open source and open standards communities better—a path I heartily endorse, as I have been intentionally trying to work in both communities in parallel over the last several years, and find places where I can bring them together.

While the entire blog is worth reading, there are two lines I think need some further thought. The first of this is a bit of a scold, so be prepared to have your knuckles rapped.

My real bottom line here is that innovators can’t go faster than their customers and customers can’t go faster than their own understanding of the technology and integration, deployment and operational considerations.

Precisely. Maybe this is just an old man talking, but I sometimes want to scold the networking industry on this very point. We fuss about innovation, but innovation requires customers who understand the technology—and the networking world has largely become a broad set of meta-engineers, interspersed with the occasional engineer. It could be there are just as many engineers as there have always been, but there are so many more administrators calling themselves engineers that it doesn’t not seem like it—but this is not my sense of where we have come since I first started working on the McGuire AFB backbone in in the late 1980’s.

My time in the Cisco TAC compared to today is telling; jbash used to open cases with the stack trace decodes in his initial email. Rodney Dunn and I used to speak to standing room only crowds on router architecture. Today, on the other hand, I go to sessions where speakers are talking about router architecture, and they are flat out wrong on how these things work. When I ask why, the speaker says “this is all this crowd is ever going to understand.” This sort of cycle is self-reinforcing. Another sign: I’m told my audience is so small because most networking folks “just don’t see the point.” They can’t see how to apply the theory I teach to the “real world,” which is driven by vendor product and configuration.

If you want innovation, be an educated consumer.

A corollary: if you want to effectively use innovation, you need to be an educated consumer. To innovate, there needs to be a tight interaction between the consumer and the creator. This is why car manufacturers sponsor racing teams, for instance—they need educated consumers to interact with in order to invent.

The second line that caught my eye was this one—

And, we need to reduce the fracturing of the industry because, in this interim period, a technology landscape has evolved that is littered with “Stacks”, “Controllers”, and “Virtual Fubars”.

This is a real problem. I think it is a product of the first problem Dave mentions—the uneducated consumer. The tendency, in the networking world, is to chase after new solutions, and to layer solutions one on top of the other. The result is often massively intertwined complexity, as we create new layers to hide the complexity of the layers below the current one. We often don’t even understand how the underlying layer works, we just assume it will. Further, we assume there will be no interaction between the layer we are smearing on top and the underlying layers we built before.

The result is a system hardened to the point of ossification. These brittle systems fail when the wind blows the wrong way, causing much heartache and distrust among our users. What is our solution?

Smear another layer on top.

Further, this kind of thinking leads to the market fragmentation Dave is talking about. As each layer becomes more complex, only “specialists” know how to work on it, creating silos.

If you want to see the market innovate, and you want to break down the silos being built right now, and you want to reduce the complexity in your network, there is a solution at hand. It is not an easy solution; it requires real work.

Get educated. Learn how this stuff works.


  1. Randall Greer on 15 February 2018 at 11:05 am

    I wonder how many other problems can be solved by the proverbial “people” getting more educated.

    If we were to assume that folks will not do so, what are we left with as a solution? I think we’ve come to the point where vendors have to keep pushing new boxes or software to stay in business (as always), and since folks won’t get more educated, someone (usually the vendor or some other developer) has to add a layer of abstraction to simplify operations, regardless of how it affects the things further down the stack. This is to solve the issue of growing number of devices to manage with the same size teams that have always been had.

    I personally am not hopeful for some big change to happen in networking, as it is generally cost prohibitive and too disruptive to simply start over by ripping and replacing. And then someone went and invented overlay networks.

    Russ, I think you’ve done a very good job at stating the problem we’re facing in the industry today. I agree that the solution will not be easy at all. I also think that most organizations will not make the change required due to their own various reasons – businesses are complex creatures and each requires their own custom solutions, and it takes a those “specialists” to figure out those solutions in a way that conforms to all the various goals we have for a design (uptime, ease of management, simplicity, scalability, etc). Not for a long time at least, maybe when the talent required becomes ubiquitous enough that it is no longer prohibitive in any way to not put the effort forth.

    • Russ on 16 February 2018 at 8:36 am

      If you take the low view, the only real choice is for vendors to “make up the rest” by providing “push this button” solutions… I do think there is a place for such systems, and this is part of what the move to hyperconvergence is about. On the other hand, I think there will always be some part of the market that sees value in educated engineers who are building solutions for a particular business.

      If you want to get into this part of the market, education is going to be a key point.