CULTURE
Innovation and the Internet
Industries mature, of course. That they do so shouldn’t be surprising to anyone who’s watched the world for very long. The question is — do they mature in a way that places a few players at the “top,” leaving the rest to innovate along the edges? Or do they leave broad swaths of open space in which many players can compete and innovate? Through most of human history, the answer has been the first: industries, in the modern age, tend to ossify into a form where a few small players control most of the market, leaving the smaller players to innovate along the edges. When the major impetus in building a new company is to “get bought,” and the most common way for larger companies to innovate is by buying smaller companies (or doing “spin ins”), then you’ve reached a general point of stability that isn’t likely to change much.
Is the networking industry entering this “innovation free zone?” Or will the networking industry always be a market with more churn, and more innovation? There are signs in both directions.
For instance, there’s the idea that once technology reaches a certain level of capability, there’s just no reason for any further forward motion. Fifty years ago, if you would have asked people what airplanes could do, and what they would look like, you have have gotten some wild feedback. Today, ask the same question, and you’ll likely get the same wild ideas. Things haven’t changed much in air travel (other than reductions in the amount of space in the cattle cars, it seems) because we’ve reached the point where new advances don’t bring much in the way of new benefits.
Another instance: there is a growing group of “old” companies with a lot of money, and they’re turning that money into political power. The one sure way to ensure stagnation is to get the government involved. A case in point here is LTE-U, which bids fair to turn the last mile upside down. It seems a number of large companies are using their lobbying mojo to make certain older carriers aren’t allowed to use unlicensed space. A lot of top flight engineers don’t seem to agree on the overall impact of allowing AT&T, for instance, to expand their wireless network on WiFi frequencies; much of the argument at the moment seems to come down to the political, rather than the engineering aspects of the problem. When lobbying takes over engineering, it’s a sure sign the industry is moving into an ossified state. Robotics are the new and exciting thing now; the Internet seems like a “given.”
On the other hand, routing is more interesting right now than it has been in a long time. Software Defined and cloud are taking over the world, it seems (though a few of us do try to inject a bit of sanity into the news stream every now and then). Over the top services, like SD-WAN, seem to be creating new value in spaces long thought completely ossified. In a somewhat virtual world (hardware still counts, but the intelligence tends to move into the overlay), there isn’t any apparent point at which you can say, “we’re done with this, let’s move to the next thing.”
It seems, to me, that we’re on a bit of a cusp, a turning point. Which way the industry goes depends, in some part, on the way the larger players go. Will they continue to turn to the government, using political muscle to solidify revenue streams? Or will they turn back to real innovation?
Let’s not lose sight of the role each of us, as individual network engineers, play in the path from this point forward — the choice between the safe vendor bet, and innovating even on a small scale, played out over the thousands of networks in the world, can make a huge difference. We tend to divide the world into small networks with boring problems and large networks with interesting problems. This is a false dichotomy — interesting problems are interesting problems, no matter what the network size. Interested people make for interesting solutions, and in turn, interesting innovation.
We need to realize that no matter how small it seems, we’re at a point where the small decisions, en mass, will make a big difference. What decisions will you make today?
Information wants to be protected: Security as a mindset
I was teaching a class last week and mentioned something about privacy to the students. One of them shot back, “you’re paranoid.” And again, at a meeting with some folks about missionaries, and how best to protect them when trouble comes to their door, I was again declared paranoid. In fact, I’ve been told I’m paranoid after presentations by complete strangers who were sitting in the audience.
Okay, so I’m paranoid. I admit it.
But what is there to be paranoid about? We’ve supposedly gotten to the point where no-one cares about privacy, where encryption is pointless because everyone can see everything anyway, and all the rest. Everyone except me, that is—I’ve not “gotten over it,” nor do I think I ever will. In fact, I don’t think any engineer should “get over it,” in terms of privacy and security. Even if you think it’s not a big deal in your own life, engineers should learn to treat other people’s information with the utmost care.
In moving from the person to the digital representation of the person, we often forget it’s someone’s life we’re actually playing with. I think it’s time for engineers to take security—and privacy—personally. It’s time to actually do what we say we do, and make security a part of the design from day one, rather than something tacked on to the end.
And I don’t care if you think I’m paranoid.
Maybe it’s time to replace the old saying information wants to be free. Perhaps we should replace it with something a little more realistic, like:
Information wants to be protected.
It’s true that there are many different kinds of information. For instance, there’s the information contained in a song, or the information contained in a book, or a blog, or information about someone’s browsing history. Each piece of information has a specific intent, or purpose, a goal for which it was created. Engineers should make their default design such that information is only used for its intended purpose by the creator (or owner) of that information. We should design this into our networks, into our applications, and into our thought patterns. It’s all too easy to think, “we’ll get to security once things are done, and there’s real data being pushed into the system.” And then it’s too easy to think, “no-one has complained, and the world didn’t fall apart, so I’ll do it later.”
But what does it mean to design security into the system from day one? This is often, actually, the hard part. There are tradeoffs, particularly costs, involved with security. These costs might be in terms of complexity, which makes our jobs harder, or in terms of actual costs to bring the system up in the first place.
But if we don’t start pushing back, who will? The users? Most of them don’t even begin to understand the threat. The business folks who pay for the networks and applications we build? Not until they’re convinced there’s an ROI they can get their minds around. Who’s going to need to build that ROI? We are.
A good place to start might be here.
And we’re not going to until we all start nurturing the little security geek inside every engineer, until we start taking security (and privacy) a little more seriously. Until we stop thinking about this stuff as just bits on the wire, and start thinking about it as people’s lives. Until we reset our default to “just a little paranoid,” perhaps.
P.S. I’m not so certain we should get over it. Somehow I think we’re losing something of ourselves in this process of opening our lives to anyone and everyone, and I fear that by the time we figure out what it is we’re losing, it’ll be too late to reverse the process. Somehow I think that treating other people as a product (if the service is free, you are the product) is just wrong in ways we’ve not yet been able to define.
Micromanaging networks considered harmful: on (k)nerd knobs
Nerd Knobs (or as we used to call them in TAC, knerd knobs) are the bane of the support engineer’s life. Well, that and crashes. And customer who call in with a decoded stack trace. Or don’t know where to put the floppy disc that came with the router into the router. But, anyway…
What is it with nerd knobs? Ivan has a great piece up this week on the topic. I think this is the closest he gets to what I think of as the real root cause for nerd knobs —
Greg has a response to Ivan up; again, I think he gets close to the problem with these thoughts —
A somewhat orthogonal article caught my eye, though, that I think explains what is actually going on here with those pesky nerd knobs. The article is really about SQL and the concept of micromanaging software. To give you a flavor (in case you’re too lazy/busy to head over there and read the whole thing) —
I think this gets to the heart of the nerd knob problem. What’s happening with nerd knobs is it’s easier to tell the system how we want something done than it is to tell the system what we want to do. Think about this way: you install a routing protocol, and you tell it what you want in broad, general terms. Something like, “I want the shortest path between each pair of points in the network.” Then you run into a situation where you need that modified, so you mess around with the metrics some, and get on with your life. Then you run into a situation where you need this flow to go here, and that flow to go there, so you install some policy based routing along the way.
Per link metrics are just the first level of nerd knobs. Policy based routing is just the second. The more precise we want to get, the deeper the nerd knobs go. Want to load share over links that aren’t truly equal cost? Oh, just nerd knob it. Want to send AS’ in the AS path you shouldn’t? Just nerd knob it.
The reality is every nerd knob in routing represents a policy driven by a business requirement expressed as a tweak to the underlying fundamental routing algorithm. As Ivan rightly points out, going to SDNs isn’t going to solve this problem. If anything, it’s going to make it worse. Now, rather than seeing the nerd knob for what it is, a pain in the butt that needs to be explained and dealt with at 2AM when you’re half asleep and the TAC engineer is halfway around the world, it’s going to be “just another line of code.”
This might sound brilliant to someone who hasn’t managed, or dealt with, multi-million line projects and the vagaries of codebase management. Ask someone who has, though, before you get into this. It’s just a different set of problems, not a better set of problems.
The root cause here, though, isn’t nerd knobs. And it’s not business requirements. And it’s not really laziness (most of the time). It’s not even machismo most of the time (though I will admit the natural arrogance of the geek is probably worth studying by some anthropologist somewhere). There are two root causes, really.
First, we, the networking industry, haven’t really thought through what a control plane actually does. Oh, we have the seven layer model with the control plane thrown off to the side, or the claim that there shouldn’t even be a control plane. But this is part of why I think the seven layer model needs to die — because it’s a host focused view of the networking world. End-to-end and dumb as rocks routers are nice to contemplate, but I think we need to admit that even the dumb rocks are a bit more complex than we first thought.
Second, I don’t think we’ve really incorporated complexity into our souls. As someone once told me, “the CAP theorem is just an observer problem!” Or rather, we somehow believe that by making virtual things we can skip all that ugly physical reality stuff. Faster, cheaper, and better are all three available “on tap,” if we can just figure out how to see the problem right. This is nonsense on stilts.
We need to get in here and do some serious thinking about complexity, and how to manage it in network design. We need to do things like think about interaction surfaces, and how to prevent them from becoming so deep and broad as to be unmanageable. As the article on SQL says, from above —
In a world of regulation and increasing interdependencies between organizations, expressing intent independently of implementation means that you can avoid a class of unintended consequences of systems building.
Where have I heard this before? Oh, maybe it’s in that new book on network complexity someplace.
Seriously — I know this is a long rant, so I’ll quit now, but — seriously (!) we need to grow up and start treating the control plane as an engineering problem. Then, and only then, will we get rid of nerd knobs, no matter whether they’re some hidden CLI command, or some “if/then/else” or “goto” statement hidden someplace in the controller code.
P.S. BTW, Greg, I disagree with you about routing protocols. They’ll “go away” for a short while, until we start trying to deal with networks that don’t run on standards based routing protocols. And then we’ll beg for them to come back. We’ll form something like the IETF, and solve all the same problems all over again, convinced that we can do better than that last group of engineers did. Been there. Done that. Got the t-shirt (someplace).