In high-traffic global content delivery networks, millisecond latency increases directly correlate with user drop-offs and degraded quality of service. When auditing our multi-site WAN infrastructure, we discovered routing-resilience gaps and path selection inefficiencies that added significant latency to sensitive application traffic.
This article outlines how we tuned BGP attributes and implemented traffic engineering to reduce latency by approximately 25%.
The Problem: Hot-Potato Routing and Suboptimal Paths
By default, BGP is a path-vector protocol designed to find the shortest path based on Autonomous System (AS) hops, not path latency or link quality. In our multi-homed data center edges, traffic frequently exited through suboptimal transit providers because BGP preferred a shorter AS-path length, despite that path suffering from congestion and a higher physical fiber distance.
Specifically, we noticed:
- Asymmetric Routing: Return traffic was choosing different, higher-latency paths.
- Hot-Potato Routing: Traffic exiting our network as quickly as possible to the nearest ISP peering node, rather than transiting our optimized private backbone to exit closer to the destination.
The Solution: Multi-Layer Traffic Engineering
To fix this, we implemented a structured BGP path tuning framework focusing on both inbound and outbound traffic flows.
1. Outbound Traffic Tuning (Local Preference)
We designated transit providers and peerings based on continuous latency probes. For outgoing traffic, we tuned the Local Preference attribute on our edge routers.
By default, Local Preference is 100. We created route-maps to assign higher Local Preference values (e.g., 150 or 200) to low-latency transit connections for specific IP prefixes:
# Nokia SR OS routing policy sample
configure router policy-options
begin
prefix-list "sensitive-apps"
prefix 198.51.100.0/24 exact
exit
policy-statement "outbound-te"
entry 10
from
prefix-list "sensitive-apps"
exit
to
neighbor 192.0.2.1
exit
action accept
local-preference 150
exit
exit
exit
commit
2. Inbound Traffic Tuning (AS-Path Prepending & MED)
Controlling incoming traffic is notoriously more difficult since external networks make their own routing decisions. However, we influenced inbound path selection using two primary tools:
- AS-Path Prepending: For transit links that had high latency, we artificially lengthened our AS path by prepending our own AS number multiple times (e.g., prepending 3 times). This made our high-latency connections look less attractive to remote ASNs.
- Multi-Exit Discriminator (MED): Where we had multiple interconnections with the same ISP, we advertised MED values. A lower MED (e.g., 50 vs. 100) tells the neighboring AS which peering point we prefer they use to reach our network.
Results and Verification
After deploying the policy changes in a phased maintenance window, we verified the results using Obkio latency probes and telemetry streaming to our Zabbix dashboards.
- Latency Reduction: Latency across our main WAN paths dropped from an average of 48ms down to 36ms (~25% decrease).
- Path Stability: Symmetric routing was restored for 92% of targeted IP ranges, reducing packet reordering and jitter for real-time application streams.
Key Takeaways
When designing WAN routing, never assume BGP's default path selection is optimal. Actively monitor path latency and use Local Preference, AS-Path Prepending, and MED to mold the traffic to fit your network's physical realities.