Just on the heels of the YouTube Route Hijack a little over two weeks ago, Africa Online Kenya (AFOL-KE – AS 36915) seems to have become the latest casualty of Internet routing insecurity. Earlier this morning Abovenet (AS 6461) began announcing reachability for one of AFOL-KE’s prefixes, 188.8.131.52/24.
According to RIPE’s BGP Play tool kit, the seemingly erroneous route announcement first appeared around 3:05A UTC this morning:
# 1/361 2008-03-15 03:05:27 Path Change from 29636 6461 2914 8513 25228 36915
rrc01 184.108.40.206 to 29636 2914 6461
# 2/361 2008-03-15 03:05:27 Route Announcement 20485 2914 6461
Some ~17 minutes later AS 6461 withdraw the route announcement, yet they began announcing it again another ~12 minutes later:
# 41/361 2008-03-15 03:22:56 Route Withdrawal ( 4777 2497 2914 6461 )
# 42/361 2008-03-15 03:35:26 Path Change from 29636 6461 2914 8513 25228 36915
rrc01 220.127.116.11 to 29636 2914 6461
Around 5:53 UTC a huge amount of instability associated with this prefix ensued:
# 66/361 2008-03-15 05:53:40 Route Announcement 25462 6461
And then again around 7:43 UTC, for around 40 minutes, a considerable amount of instability and path oscillation and instability associated with this prefix between path via AS 36915 (AFOL-KE) and the path via AS 6461 (Abovenet):
# 183/361 2008-03-15 07:43:48 Path Change from 8468 6453 6461
rrc01 18.104.22.168 to 8468 3491 25228 25228 25228 25228 25228 36915
Interestingly, although the AS 6461 route announcement still exists in the routing system, it appears as though for the past two hours or so that many networks are manually overriding default shortest AS path route selection algorithms, either by manipulating attributes such as BGP’s local preference, or filtering the announcement from AS 6461 altogether, in order to prefer the route announcement from AFOL-KE.
Unfortunately for AFOL-KE, a couple of things that are inherent to the Internet routing system are working against them in resolving this problem.
- First and most obvious, the simple fact that in it’s current state inter-provider filtering doesn’t exist in any interesting or useful way on the Internet. This allows pretty much any AS to assert reachability for any set of Internet addresses on the Internet.
- Subsequent to this is the property IP routing employs that “the most specific route is always the preferred route”. A general rule of routing on the Internet today is that most ISPs aren’t going to accept any route announcement that’s more-specific than a /24 (i.e., a /25 – /32). Unlike with the YouTube incident where they were announcing a /22 of which the hijacked route was a more-specific (a /24 subset of that /22 – to which YouTube responded with their own /24 announcement), in this case we immediately arrived at the later state of brokenness, as 22.214.171.124/24 was the original announcement by AFOL-KE, and a route of the same /24 length was announced by AS 6461.
- So the next thing that comes into play, after prefixes of the same length are announced, is where an AS ‘sits’ in the Internet routing topology mix. In general, the closer an AS is to the core, the more likely they’ll be to have their routes preferred than an AS that’s less well-connected – namely because of the shortest AS path associated with the prefix. In this case, as you might well expect, AS 6461 is generally much better interconnected with other networks than AS 36915, from Africa.
Either way, this all leads to fragmented route selection that’s topologically-dependent, and essentially non-deterministic for the greater part of the Internet. For the most part, in this case, AFOL-KE loses. As illustrated in the BGP route preference snapshot in the diagram, most of the pleasant-colored lines terminating in AS 6461 are representative of something not so pleasant for AFOL-KE.
When looking at the current announcement received from AS 6461 at route-views.routeviews.org, there’s nothing particularly telling of what may have motivated it:
126.96.36.199 from 188.8.131.52 (184.108.40.206)
Origin IGP, metric 0, localpref 100, valid, external, best
The Origin code of IGP is very common. Often times when routes are implicitly “redistributed” into BGP their Origin code is set to Incomplete, or just ?, depending how its accessed or displayed. There’s nothing particularly telling here, although there’s also no hard and fast rules about how policies should be defined to handle Origin code values.
The one potentially telling bit is the BGP Community attribute value, which is what’s often employed for coloring route sets for an announcement as “blackhole”, “internal only”, “transit route” and other policies. The value currently contained here in the announcement from AS 6461 is 6461:5999, which according to this, is used to represent “internal prefixes”. I thought this interesting at first, until I realized that all of the AS 6461 originated prefixes (130 of them currently) contain this community value.
Just for grins, I also checked to see if there were any similar prefixes originated by AS 6461 that were in adjacent address space and may have triggered a typo or the like during router or policy configuration. The closest prefix to the hijacked block (194.9.82/24) they’re announcing is 194.9.40/24, which appears to be a legitimate AS 6461 prefix:
*> 220.127.116.11 18.104.22.168 0 0 6461 i
*> 22.214.171.124 126.96.36.199 0 0 6461 i
danny@pork% whois -h whois.cymru.com 188.8.131.52
AS | IP | AS Name
6461 | 184.108.40.206 | MFNX MFN – Metromedia Fiber Network
Which does seem to invalidate any such suspicions.
So what’s the latest? Well, this traceroute tells its version of the story from where I currently sit:
danny@pork% tr 220.127.116.11
traceroute to 18.104.22.168 (22.214.171.124), 64 hops max, 40 byte packets
8 so-1-1-0.mpr3.iah1.us.above.net (126.96.36.199) 161.771 ms 140.263 ms 133.813 ms
9 so-2-0-0.mpr1.dca2.us.above.net (188.8.131.52) 127.374 ms 153.478 ms 198.339 ms
10 so-1-1-0.mpr1.lhr2.uk.above.net (184.108.40.206) 256.579 ms 233.202 ms 217.883 ms
11 so-1-0-0.mpr2.ams5.nl.above.net (220.127.116.11) 247.089 ms 251.507 ms 237.567 ms
12 ten-gige-1-1.mpr2.ams2.nl.above.net (18.104.22.168) 225.673 ms 248.454 ms 264.090 ms
13 ten-gige-2-2.mpr1.ams2.nl.above.net (22.214.171.124) 286.877 ms 282.975 ms 200.335 ms
14 ge-1-2-0.mpr1.ams1.nl.above.net (126.96.36.199) 197.288 ms 200.963 ms 262.926 ms
15 ten-gige-1-1.mpr1.ams2.nl.above.net (188.8.131.52) 185.955 ms 175.015 ms 197.161 ms
16 ten-gige-2-2.mpr2.ams2.nl.above.net (184.108.40.206) 406.662 ms 271.759 ms 235.472 ms
17 ten-gige-2-2.mpr1.ams2.nl.above.net (220.127.116.11) 192.466 ms 222.556 ms 238.616 ms
18 ge-1-2-0.mpr1.ams1.nl.above.net (18.104.22.168) 223.433 ms 361.507 ms 255.553 ms
19 ten-gige-1-1.mpr1.ams2.nl.above.net (22.214.171.124) 248.874 ms 236.173 ms 221.084 ms
20 ten-gige-2-2.mpr2.ams2.nl.above.net (126.96.36.199) 214.834 ms 209.944 ms 228.589 ms
21 ten-gige-2-2.mpr1.ams2.nl.above.net (188.8.131.52) 243.864 ms 268.004 ms 255.307 ms
22 ge-1-2-0.mpr1.ams1.nl.above.net (184.108.40.206) 288.960 ms 230.006 ms 206.478 ms
23 ten-gige-1-1.mpr1.ams2.nl.above.net (220.127.116.11) 175.457 ms 175.486 ms 205.783 ms
24 ten-gige-2-2.mpr2.ams2.nl.above.net (18.104.22.168) 205.536 ms 202.752 ms 215.748 ms
25 ten-gige-2-2.mpr1.ams2.nl.above.net (22.214.171.124) 248.825 ms 227.442 ms 248.768 ms
26 ge-1-2-0.mpr1.ams1.nl.above.net (126.96.36.199) 261.013 ms 277.980 ms 257.569 ms
27 ten-gige-1-1.mpr1.ams2.nl.above.net (188.8.131.52) 256.352 ms 215.744 ms 194.133 ms
28 ten-gige-2-2.mpr2.ams2.nl.above.net (184.108.40.206) 237.354 ms^C
I.e., a forwarding loop exists inside AS 6461 for the route, and route announcements are still seen as being originated by AS 6461 for the prefix. RIPE’s BGP Play, which I’m becoming more and more fond of, shows that a large numbers of networks are still preferring the path via AS 6461. Some traffic seems to have shifted manually back to the original AFOL-KE path, but not most.
If you’re AFOL-KE, stay on the phone with upstream, and their upstream, and the folks at Abovenet/MFN until the route is withdrawn. You can verify the current state of announcements via one of the many route servers (e.g., routeviews) by doing something like this:
username == rviews
sh ip bgp 220.127.116.11/24
After a few messages on NANOG about this, the good folks at Abovenet have sent me email with the case number and assured me they’re working on resolving the problem. Apparently, the problem they’re having that’s triggering this announcement appears to not be such a trivial thing to resolve, given the the slew of instability with route announcements for this prefix over the past hours, the existence of forwarding loops in their network for this prefix currently, and the simple fact that the problem still exists.
With no insight whatsoever to the problem, they (AS 6461) could always consider a quick routing hack like this in an attempt to restore connectivity in the interim:
- Statically route two more-specific prefixes (e.g., 18.104.22.168/25 and 22.214.171.124/25) to a legit upstream transit provider of AS 36915 that is currently utilizing that path, e.g., AS 3491 (PCCW/BTN-ASN, ironically enough) inside the AS 6461 AS.
- Inject them into BGP internally as /25s (with a NO_EXPORT community)
- Fix the problem
Then again, I have no insight into the actual cause, so that might be completely useless information, and likely is just that
Interestingly, when this first occurred Felix Bako, the Network Engineer at Africa Online, Kenya, had sent an email to the Cisco NSP list mentioning that they had inadvertently let some prefix allocation renewals with AfriNIC lapse, that he’d just renewed them, and thought that folks at some ISPs were dropping his route announcements because they’d been reclassified into BOGON space. Normally, routes are only classified as bogons if they either haven’t been allocated from IANA to a Regional Internet Registry (RIR), or they’re chunks addresses that RIRs have yet to allocate to LIRs/NIRs or directly to ISPs or other network operators.
Fortunately, there is considerable work going on in the IETF, and with the RIRs, mostly centering around SIDR work at the moment. SIDR is intended to provide an infrastructure for IP address and AS number allocation authentication via a numbers Resource PKI (RPKI), enabled by RFC 3779’s X.509 Extensions for IP Addresses and AS Identifiers. From there, it can be employed to populate information such as route objects in Internet Routing Registries (IRRs, e.g., RABD), or perhaps down the road employed by an overlay routing infrastructure, or directly by Internet routing protocols themselves via mechanisms akin to SBGP or SoBGP. If you’re a network operator, you should be paying attention to this work.
We’ve obviously got a long way to go, and good reason to get there.
As for the AFOL-KE folks, I wish you well. The Abovenet folks to seem to be well aware of the issue, and are typically quite diligent in resolving operational issues of this sort…
Finally, if there’s some reasonable explanation as to why AS 6461 is announcing this route, and its made public, I’ll happily share that information when it’s available.