Warning: in this post, I am going to cross a little into philosophy, governance, and other odd subjects. Here there be dragons. Let me begin by setting the stage:
Decentralized systems will continue to lose to centralized systems until there’s a driver requiring decentralization to deliver a clearly superior consumer experience. Unfortunately, that may not happen for quite some time. —Todd Hoff @High Scalability
And the very helpful diagram which accompanies the quote—
The point Todd Hoff, the author makes, is that five years ago he believed the decentralized model would win, in terms of the way the Internet is structured. However, today he doesn’t believe this; centralization is winning. Two points worth considering before jumping into a more general discussion.
First, the decentralized model is almost always the most efficient in almost every respect. It is the model with the lowest signal-to-noise ratio, and the model with the highest gain. The simplest way to explain this is to note the primary costs in a network is the cost of connectivity, and the primary gain is the amount of support connections provide. The distributed model offers the best balance of these two.
Second, what we are generally talking about here is data and the people connections, rather than the physical infrastructure. While physical hardware is important, it will always lag behind the data and people connections by some amount of time.
These things said, what is the point?
If decentralized is the most efficient model, then why is centralization winning? There are two drivers that can cause centralization to win. The first is scarcity of resources. For instance, you want a centralized authority (within some geographic area) to build roads. Who wants two roads to their house? And would that necessitate two garages, one for each road system? Physical space is limited in a way that, in relation to road systems, centralization (within geographic areas!) makes sense.
The second is regulatory capture. Treating a resource that is not physically constrained as a scarce resource forces centralization, generally to the benefit of the resulting private/public partnership. The overall system is less efficient, but through regulatory capture, or rent seeking, the power accrues to a single entity, which then uses its power to enforce the centralized model.
In the natural order of things, there is disruption of some sort. During the disruption phase, things are changing, and the cost of maintaining a connection in the network is low, so the network tends towards a distributed model. Over time, a distributed model will emerge naturally. Finally, as one node, or small set of nodes, gain(s) power, the network will tend towards centralization. If regulatory bodies can be found and captured, then the centralized model will be enforced. The more capture, the more strongly the centralized model will become entrenched.
Over time, if innovation is allowed (there is often an attempt to suppress innovation, but this is a two-edged sword), some new “thing,” whether a social movement or a technology—generally a blend of both—will build a new distributed network of some sort, and thus disrupt the centralized network.
What does all of this have to do with network engineering? We are currently moving towards strong regulatory capture with highly centralized networks. The owners of those centralized resources are battling it out, trying to strengthen their regulatory capture in every way possible—the war over net neutrality is but one component of this ongoing battle (which is why it often seems there are no “good folks” and “bad folks” in this argument). At some point, that battle will be decisively won, and one kind of information broker is going to win over the other.
In the process, the “old guard,” the “vendors,” are being required to change their focus, and trying to survive. Disaggregation, and “software defined,” are two elements of this shift in power.
The question is: will we reach full centralization? Or will some new idea—a new/old technology that organizes information in a different way—disrupt the coalescing centralized powers?
The answer to this question impacts what skills you should be learning now, and how you approach the rest of your career (or life!). A lot of your career rests not just on understanding the command lines and hardware, but reaching beyond these into understanding the technologies, and even beyond the technologies to understand the way organizations work. What you should be learning now, and what you are paying attention to, should not reflect the last war, but the next one. Should you study for a certification? Which one? Should you focus on a vendor, or a software package, or a specific technology? For how long, and how deeply?
And yes, I fully understand you cannot sell your longer term technical ability to prospective employers; the entire hiring process today is tailored to hedgehogs rather than foxes. This, however, is a topic for some other time.