Africa Online Kenya Latest Internet Routing Insecurity Casualty

By: Danny McPherson -

Just on the heels of the YouTube Route Hijack a little over two weeks ago, Africa Online Kenya (AFOL-KE – AS 36915) seems to have become the latest casualty of Internet routing insecurity. Earlier this morning Abovenet (AS 6461) began announcing reachability for one of AFOL-KE’s prefixes, 194.9.82.0/24.

According to RIPE’s BGP Play tool kit, the seemingly erroneous route announcement first appeared around 3:05A UTC this morning:

# 1/361 2008-03-15 03:05:27 Path Change from 29636 6461 2914 8513 25228 36915
rrc01 195.66.224.132 to 29636 2914 6461
# 2/361 2008-03-15 03:05:27 Route Announcement 20485 2914 6461
rrc01 195.66.224.212

Some ~17 minutes later AS 6461 withdraw the route announcement, yet they began announcing it again another ~12 minutes later:

# 41/361 2008-03-15 03:22:56 Route Withdrawal ( 4777 2497 2914 6461 )
rrc06 202.249.2.20
….

# 42/361 2008-03-15 03:35:26 Path Change from 29636 6461 2914 8513 25228 36915
rrc01 195.66.224.132 to 29636 2914 6461

Around 5:53 UTC a huge amount of instability associated with this prefix ensued:

# 66/361 2008-03-15 05:53:40 Route Announcement 25462 6461
rrc07 194.68.123.157

And then again around 7:43 UTC, for around 40 minutes, a considerable amount of instability and path oscillation and instability associated with this prefix between path via AS 36915 (AFOL-KE) and the path via AS 6461 (Abovenet):

# 183/361 2008-03-15 07:43:48 Path Change from 8468 6453 6461
rrc01 195.66.224.151 to 8468 3491 25228 25228 25228 25228 25228 36915

Interestingly, although the AS 6461 route announcement still exists in the routing system, it appears as though for the past two hours or so that many networks are manually overriding default shortest AS path route selection algorithms, either by manipulating attributes such as BGP’s local preference, or filtering the announcement from AS 6461 altogether, in order to prefer the route announcement from AFOL-KE.

Unfortunately for AFOL-KE, a couple of things that are inherent to the Internet routing system are working against them in resolving this problem.

  • First and most obvious, the simple fact that in it’s current state inter-provider filtering doesn’t exist in any interesting or useful way on the Internet. This allows pretty much any AS to assert reachability for any set of Internet addresses on the Internet.
  • Subsequent to this is the property IP routing employs that “the most specific route is always the preferred route”. A general rule of routing on the Internet today is that most ISPs aren’t going to accept any route announcement that’s more-specific than a /24 (i.e., a /25 – /32). Unlike with the YouTube incident where they were announcing a /22 of which the hijacked route was a more-specific (a /24 subset of that /22 – to which YouTube responded with their own /24 announcement), in this case we immediately arrived at the later state of brokenness, as 194.9.82.0/24 was the original announcement by AFOL-KE, and a route of the same /24 length was announced by AS 6461.
  • So the next thing that comes into play, after prefixes of the same length are announced, is where an AS ‘sits’ in the Internet routing topology mix. In general, the closer an AS is to the core, the more likely they’ll be to have their routes preferred than an AS that’s less well-connected – namely because of the shortest AS path associated with the prefix. In this case, as you might well expect, AS 6461 is generally much better interconnected with other networks than AS 36915, from Africa.

Either way, this all leads to fragmented route selection that’s topologically-dependent, and essentially non-deterministic for the greater part of the Internet. For the most part, in this case, AFOL-KE loses. As illustrated in the BGP route preference snapshot in the diagram, most of the pleasant-colored lines terminating in AS 6461 are representative of something not so pleasant for AFOL-KE.

AFOL-KE Woes

When looking at the current announcement received from AS 6461 at route-views.routeviews.org, there’s nothing particularly telling of what may have motivated it:


6461
64.125.0.137 from 64.125.0.137 (64.125.0.137)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: 6461:5999

The Origin code of IGP is very common. Often times when routes are implicitly “redistributed” into BGP their Origin code is set to Incomplete, or just ?, depending how its accessed or displayed. There’s nothing particularly telling here, although there’s also no hard and fast rules about how policies should be defined to handle Origin code values.

The one potentially telling bit is the BGP Community attribute value, which is what’s often employed for coloring route sets for an announcement as “blackhole”, “internal only”, “transit route” and other policies. The value currently contained here in the announcement from AS 6461 is 6461:5999, which according to this, is used to represent “internal prefixes”. I thought this interesting at first, until I realized that all of the AS 6461 originated prefixes (130 of them currently) contain this community value.

Just for grins, I also checked to see if there were any similar prefixes originated by AS 6461 that were in adjacent address space and may have triggered a typo or the like during router or policy configuration. The closest prefix to the hijacked block (194.9.82/24) they’re announcing is 194.9.40/24, which appears to be a legitimate AS 6461 prefix:

*> 194.9.40.0 64.125.0.137 0 0 6461 i
*> 194.9.82.0 64.125.0.137 0 0 6461 i

danny@pork% whois -h whois.cymru.com 194.9.40.0
AS | IP | AS Name
6461 | 194.9.40.0 | MFNX MFN – Metromedia Fiber Network

Which does seem to invalidate any such suspicions.

So what’s the latest? Well, this traceroute tells its version of the story from where I currently sit:

danny@pork% tr 194.9.82.1
traceroute to 194.9.82.1 (194.9.82.1), 64 hops max, 40 byte packets

8 so-1-1-0.mpr3.iah1.us.above.net (64.125.26.130) 161.771 ms 140.263 ms 133.813 ms
9 so-2-0-0.mpr1.dca2.us.above.net (64.125.29.38) 127.374 ms 153.478 ms 198.339 ms
10 so-1-1-0.mpr1.lhr2.uk.above.net (64.125.31.185) 256.579 ms 233.202 ms 217.883 ms
11 so-1-0-0.mpr2.ams5.nl.above.net (64.125.27.178) 247.089 ms 251.507 ms 237.567 ms
12 ten-gige-1-1.mpr2.ams2.nl.above.net (64.125.26.77) 225.673 ms 248.454 ms 264.090 ms
13 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 286.877 ms 282.975 ms 200.335 ms
14 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 197.288 ms 200.963 ms 262.926 ms
15 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 185.955 ms 175.015 ms 197.161 ms
16 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 406.662 ms 271.759 ms 235.472 ms
17 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 192.466 ms 222.556 ms 238.616 ms
18 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 223.433 ms 361.507 ms 255.553 ms
19 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 248.874 ms 236.173 ms 221.084 ms
20 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 214.834 ms 209.944 ms 228.589 ms
21 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 243.864 ms 268.004 ms 255.307 ms
22 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 288.960 ms 230.006 ms 206.478 ms
23 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 175.457 ms 175.486 ms 205.783 ms
24 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 205.536 ms 202.752 ms 215.748 ms
25 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 248.825 ms 227.442 ms 248.768 ms
26 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 261.013 ms 277.980 ms 257.569 ms
27 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 256.352 ms 215.744 ms 194.133 ms
28 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 237.354 ms^C

I.e., a forwarding loop exists inside AS 6461 for the route, and route announcements are still seen as being originated by AS 6461 for the prefix. RIPE’s BGP Play, which I’m becoming more and more fond of, shows that a large numbers of networks are still preferring the path via AS 6461. Some traffic seems to have shifted manually back to the original AFOL-KE path, but not most.

If you’re AFOL-KE, stay on the phone with upstream, and their upstream, and the folks at Abovenet/MFN until the route is withdrawn. You can verify the current state of announcements via one of the many route servers (e.g., routeviews) by doing something like this:

telnet route-views.routeviews.org
username == rviews
sh ip bgp 194.9.82.0/24

Or, of course, for a historical view, have a look at RIPE’s BGP Play for graphical representation, or raw data in routeviews archives.

After a few messages on NANOG about this, the good folks at Abovenet have sent me email with the case number and assured me they’re working on resolving the problem. Apparently, the problem they’re having that’s triggering this announcement appears to not be such a trivial thing to resolve, given the the slew of instability with route announcements for this prefix over the past hours, the existence of forwarding loops in their network for this prefix currently, and the simple fact that the problem still exists.

With no insight whatsoever to the problem, they (AS 6461) could always consider a quick routing hack like this in an attempt to restore connectivity in the interim:

  • Statically route two more-specific prefixes (e.g., 194.9.82.0/25 and 194.9.82.128/25) to a legit upstream transit provider of AS 36915 that is currently utilizing that path, e.g., AS 3491 (PCCW/BTN-ASN, ironically enough) inside the AS 6461 AS.
  • Inject them into BGP internally as /25s (with a NO_EXPORT community)
  • Fix the problem :-)

Then again, I have no insight into the actual cause, so that might be completely useless information, and likely is just that :-)

Interestingly, when this first occurred Felix Bako, the Network Engineer at Africa Online, Kenya, had sent an email to the Cisco NSP list mentioning that they had inadvertently let some prefix allocation renewals with AfriNIC lapse, that he’d just renewed them, and thought that folks at some ISPs were dropping his route announcements because they’d been reclassified into BOGON space. Normally, routes are only classified as bogons if they either haven’t been allocated from IANA to a Regional Internet Registry (RIR), or they’re chunks addresses that RIRs have yet to allocate to LIRs/NIRs or directly to ISPs or other network operators.

Fortunately, there is considerable work going on in the IETF, and with the RIRs, mostly centering around SIDR work at the moment. SIDR is intended to provide an infrastructure for IP address and AS number allocation authentication via a numbers Resource PKI (RPKI), enabled by RFC 3779′s X.509 Extensions for IP Addresses and AS Identifiers. From there, it can be employed to populate information such as route objects in Internet Routing Registries (IRRs, e.g., RABD), or perhaps down the road employed by an overlay routing infrastructure, or directly by Internet routing protocols themselves via mechanisms akin to SBGP or SoBGP. If you’re a network operator, you should be paying attention to this work.

We’ve obviously got a long way to go, and good reason to get there.

As for the AFOL-KE folks, I wish you well. The Abovenet folks to seem to be well aware of the issue, and are typically quite diligent in resolving operational issues of this sort…

Finally, if there’s some reasonable explanation as to why AS 6461 is announcing this route, and its made public, I’ll happily share that information when it’s available.

Comments

  1. Wade Blackwell 03/15/2008, 5:06 pm

    Thank you for this fantastically informative article. I don’t know where else I could get this level of detail (in one place). Thanks againm

    Wade B

  2. Thanks for this Danny – lots more interesting info here than I could put together with my view. Appreciate you taking the time!

  3. My AS path to the prefix is still “6461 I”, here are a couple of traceroutes with MPLS labels still. The first one taken an hour ago shows the loop between 550 and 315, the second one (recent) shows the loop between 297 and 315

    3 so-6-0-0.mpr2.bos2.us.above.net (64.125.27.69) 36.862 ms 39.099 ms 36.874 ms
    MPLS Label=105872 CoS=0 TTL=1 S=1
    4 so-5-0-0.cr2.lga2.us.above.net (64.125.29.90) 36.989 ms 36.902 ms 36.784 ms
    MPLS Label=240289 CoS=0 TTL=1 S=1
    5 so-2-0-0.mpr2.lga5.us.above.net (64.125.28.253) 36.767 ms 36.609 ms 36.735 ms
    MPLS Label=635072 CoS=0 TTL=1 S=1
    6 so-0-0-0.mpr1.lga5.us.above.net (64.125.27.237) 36.662 ms 36.762 ms 36.615 ms
    MPLS Label=618240 CoS=0 TTL=1 S=1
    7 so-1-0-0.mpr1.ams1.nl.above.net (64.125.27.186) 116.703 ms 118.365 ms 143.942 ms
    MPLS Label=313488 CoS=0 TTL=1 S=1
    8 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 120.308 ms 124.344 ms 120.360 ms
    MPLS Label=54 CoS=0 TTL=1 S=1
    9 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 120.379 ms 122.766 ms 120.214 ms
    10 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 116.995 ms 117.046 ms 117.003 ms
    MPLS Label=550 CoS=0 TTL=1 S=1
    11 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 126.895 ms 116.918 ms 117.953 ms
    12 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 136.514 ms 120.823 ms 120.851 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    13 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 124.697 ms 120.891 ms 120.813 ms
    14 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 126.329 ms 131.416 ms 137.330 ms
    MPLS Label=550 CoS=0 TTL=1 S=1
    15 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 117.672 ms 117.448 ms 117.460 ms
    16 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 121.533 ms 126.422 ms 121.362 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    17 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 121.455 ms 121.316 ms 121.271 ms
    18 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 118.355 ms 119.046 ms 118.536 ms
    MPLS Label=550 CoS=0 TTL=1 S=1
    19 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 120.113 ms 123.336 ms 118.038 ms
    20 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 126.628 ms 134.596 ms 121.941 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    21 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 125.440 ms 121.863 ms 121.873 ms
    22 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 131.022 ms 118.523 ms 118.581 ms
    MPLS Label=550 CoS=0 TTL=1 S=1
    23 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 126.494 ms 118.624 ms 118.530 ms
    24 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 122.383 ms 122.400 ms 122.423 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    25 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 122.468 ms 122.458 ms 122.401 ms
    26 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 119.312 ms 119.059 ms 119.099 ms
    MPLS Label=550 CoS=0 TTL=1 S=1
    27 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 119.097 ms 118.943 ms 119.091 ms
    28 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 122.822 ms 122.870 ms 122.863 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    29 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 123.830 ms 122.885 ms 123.383 ms
    30 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 119.807 ms 119.641 ms 119.669 ms
    MPLS Label=550 CoS=0 TTL=1 S=1

    #2

    3 so-6-0-0.mpr2.bos2.us.above.net (64.125.27.69) 36.765 ms 36.919 ms 36.892 ms
    MPLS Label=105872 CoS=0 TTL=1 S=1
    4 so-5-0-0.cr2.lga2.us.above.net (64.125.29.90) 36.826 ms 36.760 ms 36.924 ms
    MPLS Label=240289 CoS=0 TTL=1 S=1
    5 so-2-0-0.mpr2.lga5.us.above.net (64.125.28.253) 36.686 ms 36.614 ms 47.157 ms
    MPLS Label=635072 CoS=0 TTL=1 S=1
    6 so-0-0-0.mpr1.lga5.us.above.net (64.125.27.237) 36.915 ms 45.372 ms 36.628 ms
    MPLS Label=618240 CoS=0 TTL=1 S=1
    7 so-1-0-0.mpr1.ams1.nl.above.net (64.125.27.186) 116.556 ms 116.619 ms 116.579 ms
    MPLS Label=313488 CoS=0 TTL=1 S=1
    8 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 120.301 ms 120.373 ms 120.346 ms
    MPLS Label=54 CoS=0 TTL=1 S=1
    9 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 120.573 ms 120.702 ms 121.144 ms
    10 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 117.257 ms 117.646 ms 117.001 ms
    MPLS Label=297 CoS=0 TTL=1 S=1
    11 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 116.915 ms 130.513 ms 117.007 ms
    12 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 120.768 ms 129.930 ms 121.263 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    13 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 122.641 ms 120.947 ms 120.964 ms
    14 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 117.589 ms 117.605 ms 117.923 ms
    MPLS Label=297 CoS=0 TTL=1 S=1
    15 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 117.515 ms 117.492 ms 117.480 ms
    16 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 121.794 ms 121.442 ms 121.375 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    17 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 121.829 ms 131.167 ms 121.693 ms
    18 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 122.098 ms 118.655 ms 122.008 ms
    MPLS Label=297 CoS=0 TTL=1 S=1
    19 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 118.068 ms 128.275 ms 118.027 ms
    20 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 122.560 ms 121.856 ms 121.921 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    21 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 122.004 ms 122.282 ms 122.114 ms
    22 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 129.703 ms 119.529 ms 125.491 ms
    MPLS Label=297 CoS=0 TTL=1 S=1
    23 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 134.554 ms 123.187 ms 123.037 ms
    24 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 122.407 ms 123.023 ms 122.389 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    25 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 122.622 ms 123.351 ms 122.561 ms
    26 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 119.929 ms 119.062 ms 120.116 ms
    MPLS Label=297 CoS=0 TTL=1 S=1
    27 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 119.116 ms 119.813 ms 119.526 ms
    28 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 122.977 ms 122.895 ms 123.338 ms
    MPLS Label=315 CoS=0 TTL=1 S=1
    29 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 123.498 ms 124.213 ms 123.307 ms
    30 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 120.417 ms 120.084 ms 120.698 ms
    MPLS Label=297 CoS=0 TTL=1 S=1