Archive for 2022
Hedge 145: Roundtable on Professional Liability
The software world is known for overdue projects, costs overrun, lots of defects, and lots of failure all the way around. Many other engineering fields have stricter requirements to take on projects and liability insurance driving correct practice and care. The networking world, and the larger IT world, however, has neither of these things. Does this make IT folks less likely to “do the right thing,” or is the self-regulation we have today enough? Join Tom Ammon, Eyvonne Sharp, and Russ White as they discuss the possibilities of professional liability in information technology.
Hedge 144: IPv6 Lessons Learned
We don’t often do a post-mortem on the development and deployment of new protocols … but here at the Hedge we’re going to brave these deep waters to discuss some of the lessons we can learn from the development and deployment of IPv6, especially as they apply to design and deployment cycles in the “average network” (if there is such at thing). Join us as James Harr, Tom Ammon, and Russ White consider the lessons we can learn from IPv6’s checkered history.
Hedge 143: Being Prepared to be Laid Off with Giovanni Messina
Forty years ago there was an implied loyalty between companies and employees—but that world is long gone. As much as companies would like their employees to be loyal, layoff culture has crept into every corner of the modern world, especially as we move into an economic downturn. Giovanni Messina joins Russ White and Tom Ammon to talk about being prepared to be laid off, including such topics as being financially prepared, building skills for the long term, and finding community.
Route Servers and Loops
From the question pile: Route servers (as opposed to route reflectors) don’t change anything about a BGP route when re-advertising it to a peer, whether iBGP or eBGP. Why don’t route servers cause routing loops (or other problems) in a BGP network?
Route servers are often used by Internet Exchange Points (IXPs) to distribute routes between connected BGP speakers. BGP route servers
- Don’t change anything about a received BGP route when advertising the route to its peers (other BGP speakers)
- Don’t install routes received through BGP into the local routing table
Shouldn’t using route servers in a network—pontentially, at least—cause routing loops or other BGP routing issues? Maybe a practical example will help.
Assume b, e, and s are all route servers in their respective networks. Starting at the far left, a receives some route, 101::/64, and sends it on to b,, which then sends the unmodified route to c. When c receives traffic destined to 101::/64, what will happen? Regardless of whether these routers are running iBGP or eBGP, b will not change the next hop, so when c receives the route, a is still the next hop. If there’s no underlying routing protocol, c won’t know how to reach A, so it will ignore the route and drop the traffic. Even if there is an underlying routing protocol, c’s route to 101::/64’s route passes through b, and b isn’t installing any routing information learned from BGP into its local routing table (because it’s a route server). b is going to drop traffic destined to 101::/64.
We can solve this simple problem by adding a new link between the two clients of the route server, as shown in the center diagram. Here, d sends 101::/64 to e, which then sends the unchanged route to g. Since g has a direct connection to d, we can assume g will send traffic destined to 101::/64 directly to d, where it will be forwarded to the destination. Why wouldn’t d and g peer directly instead of counting on e to carry routes between them? In most cases this kind of indirect peering is done to increase network scale. If there are thousand routes like d and g, it will be simpler for them all to peer to e than to build a full mesh of connections.
Why not use a route reflector rather than a route server in this situation? Route reflectors can only be used to carry routes between iBGP peers. If d, e, and g are all in different autonomous systems, route reflectors cannot be used to solve this problem.
But this brings us back to the original question—route reflectors use the cluster list to prevent loops within an AS (the cluster list is similar in form and function to the AS path carried between autonomous systems, but it uses router ID’s rather than AS numbers to describe the path)?
If you have multiple route servers connected to one another you can, in fact, form routing loops.
In this network, a is sending 101::/64 to b, which is then sending the route, unmodified, to e. Because of some local policy, e is choosing the path through a, which means e forwards traffic destined to 101::/64 to c. At the same time, e is advertising 101::/64 to b, which is then sending the route (unmodified) to a, and a is choosing the path through c. In this case, a permanent (persistent) routing loop is formed through the control plane, primarily because no single BGP speaker has a complete view of the topology. The two route servers, by hiding the real path to 101::/64, makes is possible to form a routing loop.
The deploy route servers without forming these kinds of loops—
- BGP speakers learning routes from route servers should be directly connected—there should not be destinations reachable via some “hidden” intermediate hop
- Route servers should send all the routes they learn from clients; they should not use bestpath to choose which routes to send to clients
These restrictions prevent routing loops from forming when deploying route servers—but they also restrict the use of route servers to situations like carrying routes between BGP speakers connected to a single fabric.
Cisco filed a patent some time back describing a method to prevent routing loops when using BGP route servers; it makes interesting reading for folks who want to dive a little deeper.
BGP Peering (2)
I recorded the beginnings of a BGP training series over at Packet Pushers a short while back; they’ve released these onto youtube (so you can find the entire series there). I’m highlighting one of these every couple of weeks ’til I’ve gone through the entire set of recordings. In this recording, I’m talking through some more interesting aspects of BGP peering, including challenges with IPv6 link local nexthops, promiscuous peering, and capabilities.
https://packetpushers.net/learning-bgp-module-2-lesson-2-peering-part-2-video/
Hedge 142: George Michaelson and the Pace of IPv6 Deployment
IPv6 is still being deployed, years after the first world IPv6 day, even more years after its first acceptance as an Internet standard by the IETF. What is taking so long? George Michaelson (APNIC) joins Tom Ammon and Russ White on this episode of the Hedge to discuss the current pace of IPv6 deployment, where there are wins, and why things might be moving more slowly in other areas.
RFC9199: Lessons in Large-scale Service Deployment
While RFC9199 (are we really in the 9000’s?) is targeted at large-scale DNS deployments–specifically root zone operators–so it might seem the average operator won’t find a lot of value here.
This is, however, far from the truth. Every lesson we’ve learned in deploying large-scale DNS root servers applies to any other large-scale user-facing service. Internally deployed DNS recursive servers are an obvious instance, but the lessons here might well apply to a scheduling, banking, or any other multi-user application accessed from a lot of places by a lot of different users. There are some unique points in DNS, such as the relatively slower pace of database synchronization across nodes, but the network-side lessons can still be useful for a lot of applications.
What are those lessons?
First, using anycast dramatically improves performance for these kinds of services. For those who aren’t familiar with the concept, anycase turns an IP address into a service identifier. Any host with a copy (or instance) or a given service advertises the same address, causing the routing table to choose the (topologically) closest instance of the service. If you’re using anycast, traffic destined to your service will automatically be forwarded to the closest server running the application, providing a kind of load sharing among multiple instances through routing. If there are instances in New York, California, France, and Taipei, traffic from users in North Carolina will be routed to New York and traffic from users in Singapore will be routed to Taipei.
You can think of an anycast address something like a cell tower; users within a certain desintance will be “captured” by a particular instance. The more copies of the service you deploy, the smaller the geographic region the service will support. Hence you can control the number of users using a particular copy of the service by controlling the number and location of service copies.
To understand where and how to deploy service instances, create anycast catchment maps. Again, just like a wifi signal coverage map, or a cellphone tower coverage map, it’s important to understand which users will be directed to which instances. Using a catchment map will help you decide where new instances need to be deployed, which instances need the fastest links and hardware, etc. The RIPE ATLAS pobes and looking glass servers are good ways to start building such a map. If the application supports a large number of users, you might be able to convince the application developer to include some sort of geographic information in requests to help build these maps.
Third, when deploying service instance, pay as much attention to routing and connectivity as you do the number of instances deployed. As the authors note, sometimes eight instances will provide the same level of service as several thousand instances. The connectivity available into each instance of the service–bandwidth, delay, availability, etc.–still has a huge impact on service speed.
Fourth, reduce the speed at which the database needs to be synchronized where possible. Not every piece of information needs to be synchronized at the same rate. The less data being synchronized, the more consistent the view from multiple users is going to be.
RFC9199 is well worth reading, even for the average network engineer.