When prepend fails, what next? (3)

We began this short series with a simple problem—what do you do if your inbound traffic across two Internet facing links is imbalanced? In other words, how do you do BGP load balancing? The first post looked at problems with AS Path prepend, while the second looked at de-aggregating and using communities to modify the local preference within the upstream provider’s network.

There is one specific solution I want to discuss a bit more before I end this little series: de-aggregation. Advertising longer prefixes is the “big hammer” of routing; you should always be careful when advertising more specifics. The Default Free Zone (DFZ) is much like the “commons” of an old village. No-one actually “owns” the routing table in the global Internet, but everyone benefits from it. De-aggregating don’t really cost you anything, but it does cost everyone else something. It’s easy enough to inject another route into the routing table, but remember the longer prefix you inject shows up everywhere in the world. You’re fixing your problem by taking up some small amount of memory in every router that’s connected to the DFZ in the world. If everyone de-aggregates, everyone has to buy larger routers and more memory. Including you.

There is a fine line between using a commonly held resource and abusing a commonly held resource. If everyone abuses the commons because it “does not cost them anything,” what results is the tragedy of the commons. Once a set of commons are ruined, it’s very difficult to recover the original intent and trust relationships that caused the commons in the first place. So before you de-aggregate, you should think about whether or not it is really necessary.

Is this really necessary? Does it really matter if your two inbound links are not balanced? There may be financial reasons why it does matter, such as the costs of the two links, or the cost of bursting over a set level on one of the two links. These are certainly considerations, but it might make more sense to modify the sizing of the available links rather than putting a technical solution in place that will need to be managed and maintained.

Remember everything you configure will eventually break, and everything that breaks results in a call at 2AM. Think through the options you have available before putting a optimization in place.

Are there ways I can limit the damage to the commons?

Returning to our original network, is it possible to de-aggregate in a way that pulls traffic from AS65001 into AS65004, but doesn’t impact the table size of anyone these two providers are connected to? Most providers to, in fact, allow you to not only send a community to set the local preference within their AS, but also to block the advertisement of any particular route to their peers. You might need to play around with these communities a bit to understand the relationship between the community and inbound traffic flow; for instance, what impact will blocking the advertisement of a more specific to the transit peers of one upstream versus blocking the route to some set of customers connected to the provider? As there is no way for you to directly know how and where the provider is connected. You can work directly with the provider to sort out what to advertise where while reducing your global impact, or you might just need to play around with different combinations to see what works best.

Is my peering the right peering? Another option is to think through who you are peering with. Assume, for a moment, that you are peering with one more regional provider, and one more global provider. In this case, your customer base is going to play a large role in which provider sends you more traffic.

For instance, if you are a regional bank, or health care provider, most of your customers are going to be connected to a regional provider (rather than a tier 1), and hence you are likely to receive most of your traffic on the regional provider’s link. If, however, your business is more global, the regional provider is not going to send you a lot of traffic—mostly just people who happen to be accessing your network from within your region. In this case, the imbalance between the two inbound links should be expected.

An observation: if this is so, maybe it is better to peer with two providers that will bring you closer to your customers. If your customers are global, maybe it’s better to peer with two providers at the national or global level, rather than one global and one regional—and the other way around. Perhaps it is better to balance your inbound traffic by carefully considering who your customers are, and how to best reach them, than it is to try and play engineering tricks to draw equal amounts of traffic over the networks of two completely different kinds of providers.

The bottom line is this: the engineering solution is the last solution you should reach for. I know—we are all engineers here, and there’s nothing quite like getting under a heavy load and solving it with a nice, long set of configuration commands that make you feel like you spent your money well in buying that big hunk of iron racked up in the DMARC.

But real engineering begins when you ask the background questions, and really understand the problem.