Core Routing Security: Hardening the Routing Protocols
In this blog
Routing protocols such as ISIS, OSPF and BGP are critical for core networks. Without them, routers don't know how to communicate with each other, which means they can't deliver transport services or perform traffic engineering across the network. In the first segment of this Core Routing Security series, we discussed the importance of protecting the core network's control plane from intentional or accidental disruption—a critical step that ensures that the routing protocols have enough system resources to do their jobs. Now, in this third installment, we turn our attention to hardening the protocols themselves. We'll explore hardening strategies for interior routing protocols—ISIS and OSPF—as well as for BGP. At each step, we'll give you concrete action items you can take to improve your core network's security today.
1. Neighbor authentication: The first line of defense
Routing protocols assume a level of trust between peers. If one router says, "Here's a new prefix," its peer believes it. Without authentication, this trust can be easily abused—any device can attempt to form an adjacency, potentially injecting malicious routes, causing instability, or exhausting control plane resources. Protocol authentication provides the first and most essential barrier.
IGP Authentication
Neighbor authentication provides a basic but critical safeguard for IGPs such as ISIS and OSPF, protecting them against unauthorized adjacencies and control-plane tampering. Both protocols support cleartext, HMAC-MD5, and HMAC-SHA authentication options. These mechanisms ensure that only authenticated routers are permitted to exchange routing information.
BGP Authentication
BGP uses TCP MD5 authentication (such as RFC 2385 and RFC 5925) to protect the initial TCP handshake and session maintenance. The use of keychains is strongly recommended, as they support time-based key rotation and minimize operational disruptions during key changes.
Authentication Key Management
One of the most common missteps is over-reliance on a single static key shared across an entire domain, which increases the blast radius in the event of compromise. Instead, operators should deploy unique keys per peer or region and rotate them regularly. It's also a good idea to configure alerting for any failed authentication attempts, which could signal either benign misconfiguration or more malicious activity like reconnaissance.
2. IGP Hardening: ISIS and OSPF
Once neighbors are authenticated, the next hardening step is to protect the IGP domain itself. As link-state protocols, ISIS and OSPF are vulnerable to flooding attacks and route table instability if misbehaving devices are allowed to inject routing information.
Adjacency Control
To prevent unauthorized adjacencies, operators should explicitly configure passive interfaces on routers that should never form routing sessions. Interfaces that are active should include strict authentication and, where possible, neighbor-specific expectations. Once adjacency is formed, the emphasis shifts to flooding control.
Flooding Protection
Both protocols offer link-state throttling mechanisms, which control how frequently updates can be sent or flooded. Throttling helps guard against excessive churn and route flapping. In addition, many platforms support maximum link-state thresholds, which act as a circuit breaker if a peer exceeds a reasonable number of advertisements. SPF backoff timers can also help, delaying recalculations during periods of instability to reduce control plane load.
Operational Flags
Operators should also leverage protocol-specific operational flags. For example, ISIS includes the Overload Bit, which suppresses a router's prefixes for transit use—ideal during maintenance or bootstrap events. OSPF utilizes a Down Bit, which helps prevent routing loops in multi-protocol environments where redistribution often poses a threat to routing stability. These features can help segment IGP domains safely and predictably.
Route Redistribution Filtering
Finally, it's critical to apply route filtering wherever redistribution is necessary. Redistribution is one of the most common sources of control plane headache, and unfiltered routes can lead to black holes or loops across the network.
3. BGP Session Protection
Due to its use across administrative boundaries, BGP presents a broader attack surface than IGPs. Securing BGP requires not just authenticating the session but also restricting the type and volume of routing information exchanged. Many different mechanisms can help operators achieve more secure and stable BGP routing.
TTL Security (GTSM)
RFC 5082, known as the Generalized TTL Security Mechanism (GTSM), is a simple yet effective enhancement to BGP. It enforces a minimum TTL of 255 on incoming BGP packets, ensuring that only directly connected peers can establish or maintain a session—eliminating many off-path attack vectors. On most carrier-grade core networking routers, this feature is enabled on external BGP (eBGP) by default, while disabled on internal BGP (iBGP) by default. This standard default behavior reflects the fact that BGP peerings with external routers should always be highly scrutinized and expected to be directly connected.
Prefix and AS Path Filtering
BGP is a particularly good candidate for route filtering, where filters can be configured to match on route prefixes, AS paths, communities, and more. This is especially important in external BGP (eBGP), where prefix-lists should always tightly define which prefixes an external peer is authorized to advertise. AS-path filters should also be used to prevent leaks or route reflection loops, and max-prefix limits provide an important safeguard against runaway advertisements.
For iBGP sessions, communities and route targets can be used to enforce policy, while route-reflectors should be configured with filters to prevent invalid routes from spreading through the core.
BGP route dampening helps mitigate the effects of prefix flapping, though it must be deployed carefully to avoid suppressing legitimate routing updates.
In all cases, alerting and logging are critical—unexpected routing behavior often points to operational or security issues that require further investigation.
Resource Public Key Infrastructure (RPKI)
Finally, while session authentication can secure BGP connections, it does not verify the legitimacy of advertised prefixes. For this, operators should consider integrating RPKI-based validation, which checks whether the originating AS is authorized to advertise a given prefix.
Conclusion: Securing core routers requires layered defense
Even well-designed networks are vulnerable to basic oversights. It's surprisingly common to find IGP adjacencies running without authentication, eBGP sessions established without any filters, or MD5 keys that haven't been changed in years. Performing a simple audit against these known pitfalls can yield rapid security gains with minimal operational impact.
Just as Control Plane Protection protects the router's CPU and MACsec protects the traffic on the wire, protocol hardening protects the routing decisions being made—decisions which are the lifeblood of core networks. Securing them is not optional; it's foundational.