Posts by Russ
Mean Time to Innocence is not Enough
A long time ago, I supported a wind speed detection system consisting of an impeller, a small electric generator, a 12 gauge cable running a few miles, and a voltmeter. The entire thing was calibrated through a resistive bridge–attach an electric motor to the generator, run it at a series of fixed speed, and adjust the resistive bridge until the voltmeter, marked in knots of wind speed, read correctly.
The primary problem in this system was the several miles of 12 gauge cable. It was often damaged, requiring us to dig the cable up (shovel ready jobs!), strip the cable back, splice the correct pairs together, seal it all in a plastic container filled with goo, and bury it all again. There was one instance, however, when we could not get the wind speed system adjusted correctly, no matter how we tried to tune the resistive bridge. We pulled things apart and determined there must be a problem in one of the (many) splices in the several miles of cable.
At first, we ran a Time Domain Reflectometer (TDR) across the cable to see if we could find the problem. The TDR turned up a couple of hot spots, so we dug those points up … and found there were no splices there. Hmmm … So we called in a specialized cable team. They ran the same TDR tests, dug up the same places, and then did some further testing and found … the cable was innocent.
This set up an argument, running all the way to the base commander level, between our team and the cable team. Who’s fault was this mess? Our inability to measure the wind speed at one end of the runway was impacting flight operations, so this had to be fixed. But rather than fixing the problem, we were spending our time arguing about who’s fault the problem was, and who should fix it.
When I read this line in a recent CAIDA research paper–
“Measurement is political, and often adversarial.”
It rang very true. In Internet terms, speed, congestion, and even usage are often political and adversarial. Just like the wind speed system, two teams were measuring the same thing to prove the problem wasn’t their’s–rather than to figure out what the problem is and how to fix it.
In other words, our goal is too often Mean Time to Innocence (MTTI), rather than Mean Time to Repair (MTTR).
MTTI is not enough. We need to work with our application counterparts to find and fix problems, rather than against them. Measurement should not be adversarial, it should be cooperative.
We need to learn to fix the problem, not the blame.
This is a cultural issue, but it also impacts the way we do telemetry. For instance, in the case of the wind speed indicator, the problem was ultimately a connection that “worked,” but with high capacive reactance such that some kinds of signals were attenuated while others were not. None of us were testing the cable using the right kind of signal, so we all just sat around arguing about who’s problem it was rather than solving the problem.
When a user brings a problem to you, resist the urge to go prove yourself–or your system–innocent. Even if your system isn’t the problem, your system can provide information that can help solve the problem. Treat problems as opportunities to help rather than as opportunies to swish your superhero cape and prove your expertise.
Hedge 154: Path Aware Networking Research Group
Applications generally assume the network provides near-real-time packet transmission without regard for what the application is trying to do, what kind of traffic is being transmitted, etc. Back in the real world, its often important for the network to coordinate with applications to more efficiently carry traffic offered. The Path Aware Research Group (PANRG) in the Internet Research Task Force (IRTF) is looking at the problems involved in understanding and signaling the path characteristics to applications.
In this episode of the Hedge, Brian Trammel joins Tom Ammon and Russ White to discuss the current work on path aware networking.
Hedge November Update
November update on upcoming shows and training. My upcoming training on Safari Books Online is here.
Hedge 153: Security Perceptions and Multicloud Roundtable
Tom, Eyvonne, and Russ hang out at the hedge on this episode. The topics of discussion include our perception of security—does the way IT professionals treat security and privacy helpful for those who aren’t involved in the IT world? Do we discourage users from taking security seriously by making it so complex and hard to use? Our second topic is whether multicloud is being oversold for the average network operator.
Hedge 152: Joel King on the network and DevOps
DevOps, SecDevOps, GitDevOps—stick DevOps on the end of anything, and it will sound cool, generation FOMO in thousands (maybe millions). What does DevOps really mean to network engineers, though? In this episode of The Hedge, we discuss examples of how the Three Ways, (described in Part One of The DevOps Handbook) of Flow, Feedback, and Continual Learning with Joel King, a leading light in this field.
On the ‘net: Privacy and Networking
The final three posts in my series on privacy for infrastructure engineers is up over at Packet Pushers. While privacy might not seem like a big deal to infrastructure folks, it really is an issue we should all be considering and addressing—if for no other reason than privacy and security are closely related topics. The primary “thing” you’re trying to secure when you think about networking is data—or rather, various forms of privacy.
Hedge 151: Cecilia Testart and the Value of the RPKI
If you advertise routes through a provider to the global Internet, you might be wondering if you should go through the trouble of registering in the RPKI and advertising ROAs. What is the tradeoff for the work involved in what seems like a complex process? Cecelia Testart joins Jeremy White and Russ White to discuss recent work in measuring the value of the RPKI.