Cyber Black Monday Traffic – Non Issue?

By: Jose -

Yesterday was “Cyber Monday”, so named for the kickoff of the online sales binge that so many Americans go through. The thinking is that Friday is for brick and mortar sales, and Monday is when folks get to work and use their high speed links to browse, shop, and spend. A few folks measure this specifically. Check the dynamic report Net Usage Index for Retail from Akamai is a very neat visualization of backbone traffic to e-commerce sites.

Looking at HTTP flows from a US provider to the outside world, we can see that there isn’t a big spike Monday relative to the previous Wednesday. We can clearly see the US holiday Thanksgiving (Thursday) and the associated day off work Friday in the traffic depressions. By the time Monday peaks we’re back at pre-holiday levels.


Year on year, however, is another story, but I don’t think it’s all attributable to the growth of Cyber Monday, it looks just like traffic growth. Bloggers and news reports, such asCyber Monday Internet Traffic Spikes 30 Percent from MarketWatch, are citing data from Integra Telecom about a 30% Y-on-Y growth. Cyber Monday traffic better than last year by ZDNet blogger Sam Diaz also discusses these numbers. If we look at backbone HTTP traffic over a 52 week period we can see growth even before Cyber Monday. This analysis is just raw web traffic, not analyzing specific sites or types of sites.


The net is indeed growing, but to come to conclusions it’s important to keep context in mind. If you just measure one day a year, you’ll infer growth due to one phenomenon. I trust Akamai’s data as it’s targeted at e-commerce sites. So there is growth, but not all of it is specific to Cyber Monday.

Google Report on IPv6 Capable Clients

By: Danny McPherson -

Given that I’ve seen a number of references to a report Google provided at a RIPE meeting in Dubai a couple weeks ago titled “Global IPv6 Statisics: Measuring the current state of IPv6 for ordinary users“, and several folks comparing it to a report titled Tracking the IPv6 Migration we at Arbor released several months back, I feel obligated to share an observation or two here about this data.

First and foremost, there’s a big difference between measuring end systems or clients capable of usable IPv6 connectivity (what Google measured) and how much actual traffic on [a defined subset of] the Internet is IPv6 (what we measured).  That is, just because some percentage of end systems on the Internet support IPv6, that doesn’t necessarily mean the percentage of IPv6 traffic on the Internet parallels it – i.e., the data points provided are indeed complementary, not at odds.

Basically, as our friend  Iljitsch van Beijnum points out at ars technica from the Google presentation, Google observed that .238% of all users worldwide have working IPv6 connectivity.  Another .09% of users have IPv6 connectivity, but it’s apparently broken in some manner.  What we observed is that .0026% of traffic measured was IPv6 traffic (with noted methodology caveats such as some limitations in supported flow types, Teredo data channels not being measured, etc, and being captive to what the production ISP networks we were measuring were capable of reporting).  There are several other IPv6-related measurements out there, to include Geoff Huston’s IPv6 routing system report, DNS IPv6 enabled queries, and IPv6 address space delegation from RIRs (pdf).

I want to state explicitly that the point of the Arbor IPv6 traffic report certainly wasn’t to discredit IPv6 in any way, quite the contrary, as the goal was to foster discussion on the topic of the Internet community’s prepardness for the exhausiton of IPv4 addresses.  What’s key here is that folks are measuring these datapoints, and solong as we continue to measure and ensure the methodology is well understood, growth rates [of what’s being measured] can be observed, and progress can be gauged.

Finally, I suspect that had Google simply put up a traffic graph of Google inter-domain IPv4 v. IPv6 percentages the numbers would be more akin to what we observed – although I might be otherwise pleasantly surprised.

The End is Near, but is IPv6?

By: Craig Labovitz -

As of this blog posting, exactly 900 days remain until the end of the Internet, or at least the exhaustion of IPv4 registry allocations. And you don’t have to take my word for it, even the normally staid London Times and Fox News proclaimed, “Internet meltdown… The world is heading for a digital doomsday”.

Heady stuff.

Of course, IPv6 (or the new IPv4) was supposed to take care of all of this — billions of billions of new IP addresses, hardened security built in from the start, and an elegant new architecture to replace all of IPv4’s hacks.

So what happened to IPv6?

Well, it has been a strange, long year…

The year began with fears over the “end of the Internet” (due to lack of IPv6 adoption) and ends this month with renewed IPv6 enthusiasm centered around the Olympics and a successful US government IPv6 mandate. In between these two extremes of IPv6 despair and enthusiasm, IPv6 generated a surge of news coverage (see graph below). At its peak this past June, print media around the world published nearly 3,000 articles a month on IPv6 (almost twice as much as the comparatively uninteresting IPv4).

Much of the recent coverage has focused on the summer Olympics this week. Chinese organizers have planned the summer Olympics game as a showcase for IPv6 technology. From a recent article, “… IPv6 will herald the arrival of China as a major center for technological and scientific advancement in a way that will overshadow its own unbeatable record as a world leader…”. Through China’s Next Generation Internet (CNGI) initiative, China has reportedly spent several billion dollars making sure they got a national IPv6 backbone right.

In the US, the recent government deadline for IPv6 compliance also generated a flurry of IPv6 activity: All major vendors publicly declared their IPv6 readiness. Popular press and industry magazines filed thousands of stories on IPv6. US Federal Departments officially declared success and the Internet IPv6-ready this past June 30th.

So has imminent collapse of the Internet has been avoided?

Is the Internet moving full steam ahead towards IPv6?


The truth is that as an industry we don’t have a good measure on the relative adoption success of IPv6 with respect to Internet traffic.

No idea really.

We do have some anecdotal measurements on IPv6 registry allocation and BGP announcements. But, very little data on actual IPv6 usage.

As our small effort to fill this gap, we spent much of the last year looking for IPv6 traffic in the Internet. In cooperation with the University of Michigan and close to 100 Internet providers, we leveraged commercial traffic probes across the world to measure inter-domain IPv6 traffic in the Internet. We believe this is the largest study of IPv6 and Internet traffic in general to date (by several orders of magnitude).

Our dataset covered 87 ISPs including one quarter of the tier1 ISPs and a sizable percentage of the regional / PTT providers in North America and EMEA. In all, we monitored 2,389 peering and backbone routers, 278,268 customer and peering interfaces, and an aggregate 15 exabytes of Internet inter-domain traffic at an average daily rate of 4 terabits per second (we spoke about some of this measurement infrastructure at a recent NANOG). We believe this gave us a pretty good view of overall IPv6 traffic trends in the Internet.

You can view the full technical report at

What did we find?

Not much. In fact, less than not much — very, very little.

The below shows the percentage of IPv6, both native and tunneled, as a percentage of all Internet traffic. At its peak, IPv6 represented less than one hundredth of 1% of Internet traffic. This is somewhat equivalent to the allowed parts of contaminants in drinking water (my household water comes from the Detroit river).

IPv6 Traffic Graphed as Percentage of IPv4

Now the above graph may not be completely fair since many of the ISPs do not have infrastructure to monitor native IPv6 (more about this later). But our numbers seem to agree with data from a variety of other sources on IPv6 adoption rates.

Some related IPv6 statistics:

Percentage of ASN with IPv6 BGP announcements 3%
Percentage of Internet2 sites with passing IPv6 grade 1%
Percentage of Alexa Top 500 websites using IPv6 0.4%
IPv6 DNS queries as percentage of IPv4 DNS load 0.2%
IPv6 as a percentage of all Internet traffic 0.002%

We are not the first to raise concern over the small amount of IPv6 traffic (see Geoff’s slides last month) — just the first to have Internet wide IPv6 traffic usage measurements.

And the lack of IPv6 traffic is not for lack of trying. Many organizations and individuals offer a range of lures to encourage IPv6 adoption. For example, the next generation research and education backbone in the US, Internet2, offers free transit for IPv6 traffic. And unlike IPv4, many large ISPs have very liberal IPv6 peering policies.

The single greatest lure? For ISPs or large multi-homed enterprises struggling to justify just one more tiny, little IPv4 /16 allocation, the minimum IPv6 allocation is /32 or a staggering 2^64 larger than the entire IPv4 Internet today.

On the less pragmatic side, other IPv6 proponents offer free high quality IPv6 porn. Others yet provide ASCII animation of Star Wars movies (IPv4 users get only black & white — make sure you watch the “Return of the Jedi” version). And, of course, the dancing turtle. Several web sites provide more listings of exotic IPv6 only Internet content.

But none of these efforts have been enough to generate any significant IPv6 traffic.

So, why so little IPv6 traffic?

Well, the biggest issue is money. Specifically, the department of commerce estimates it will cost $25 billion for ISPs to upgrade to native IPv6.

And this massive expense comes without the lure of additional revenue since IPv6 offers diminishingly few incentives nor new competitive features to attract or upsell customers. In many ways, IPv6 in the United States is much like the high definition television federal mandate (but without the mandate or the really crisp looking football games).

The harsh logic of the Metcalfe Effect also applies. With so few web sites and end users currently supporting IPv6, the incremental value to any single new IPv6 end site is limited. For many end users, v6 is an IP version all dressed up but with nowhere to go.

The third major issue is technical. While most vendors passed the OMB IPv6 requirements, it kind of depends on what you mean by “IPv6″ and “requirement”.

For example, some backbone routers “support” IPv6 forwarding, but not in hardware (at least not at line rate). Or IPv6 “support” does not include ACLs nor other critical security features. An ICANN survey of security vendors found that less than one in three commercial firewalls support IPv6.

Maybe you want an IPv6 MPLS VPN backbone? Sorry, not supported.

And even if your router supports IPv6, you might not be able to test or monitor it. Few vendors offer complete IPv6 SNMP / MIB support and even fewer support IPv6 Flow export (in fairness, V9 flow support is included on many Cisco cards today and Juniper has announced IPFIX support sometime in the next year). We blogged about many of these deployment issue earlier this year and Randy gave a presentation on the topic at a recent NANOG. The CAIDA and ARIN IPv6 Survey also has a nice summary on market / business forces limiting ISP IPv6 adoption.

Perhaps the biggest problem is that IPv4 works. And works well.

While IPv4 addresses are still relatively plentiful and cheap, ISPs and end customers have few incentives to migrate to IPv6. Some recent research even suggests IPv4 addresses may be more plentiful than previously believed. This tech report found that less than 4% of allocated addresses are actually occupied by visible end hosts. The authors concluded that most Internet space is likely, in fact, unused (though allocated).

All of this lack of IPv6 adoption has lead to quite a bit of hand wringing in the ISP technical community. While not declaring IPv6 a failure, discussions do wander into questions about “premature extinction of IPv6″ or whether “IPv6 is an exercise in futility”.

Imminent Collapse

Predicting the imminent collapses of the Internet has a long and storied history over the last twenty years. But despite all of these predictions, the Internet has survived. Sure we crashed a few routers, announced some bogus routes, and dropped a few bits here and there. But the Internet survived — even grew a bit and gained some new users. I saw Bob Metcalfe eat a tie.

And the Internet will undoubtedly change and evolve past the impending IPv4 exhaustion.

But how?

Well, the questions is more about market forces than technology. IPv4 address allocations already have a minimal cost ($18,000 for ARIN large allocation). And growing registry management justification requirements and shrinking allocation size have steadily increased the overall cost of address space to ISPs over the last ten years. During the heady Internet technology bubble days, several companies made acquisitions in significant part based on the valuation of large legacy IP allocations.

Many think the price of IPv4 and scarcity will lead to open or if not sanctioned, black markets for address space. And debates continue whether an open market for IPv4 would be good or bad thing for Internet policy. Personally, I think an IPv4 market is inevitable.

The Future of IPv6

It is now clear the original optimistic IPv6 deployment plans have failed.

While the end of the Internet is not near, neither is IPv6. At the current rate of adoption, we are a decade or more away from pervasive adoption of dual stack support for IPv6. As Alain correctly notes in a recent IETF draft, “The IANA free pool of IPv4 addresses will be depleted soon, way before any significant IPv6 deployment will have occurred”.

So IPv6 adoption will take far longer and will look far different than most of us expected back in 1994 when the IAB announced the selection of IPv6. Clearly things need to change, including IETF and vendor exploration of other technologies to facilitate IPv6 adoption such as better NAT interoperability or lighter weight dual stack.

Number of IPv6 News Articles per Month

Still, despite some of the rather anemic IPv6 traffic statistics above, IPv6 is growing. The graph above shows the number of print media articles per month mentioning IPv6 and IPv4 in the first 30 words (source MetaNews). Note that IPv6 is running almost two to one against IPv4. If judged purely by public interest, IPv6 is a winning (by comparison, DNSSEC averages only 50 articles per month and barely peaked at 150 during the DNS crisis. BGPSEC fared even worse).

The below graph shows the aggregate average daily IPv6 (tunneled and native) traffic across 87 ISPs over the last year. Since July 2007, IPv6 traffic has grown by nearly a factor of 5 to an average of 100 Mbps per day. BGP tables show an even larger proportional growth. Though not a landslide of adoption, it is still something.

While it is easy to poke fun at predictions of the “Imminent Collapse of the Internet”, the eventual exhaustion of IPv4 allocations is real. We need to do something. And IPv6 is our best bet.

Aggregate IPv6 Traffic Internet Wide

So, I’ll end with my top four predictions on IPv6 growth:

  1. Islands are beautiful too.
    IPv6 may succeed in the same way multicast failed. And by multicast failing, I really mean multicast succeeded. Though multicast never evolved a business model to justify its originally envisioned Internet wide inter-domain deployment, multicast has been astonishingly successful within the enterprise and ISP service infrastructure. Almost all Fortune 500 enterprises use multicast in some form to broadcast company video or update applications. Similarly, multicast is at the core of most ISP IPTV infrastructure.Like multicast, we are seeing rapid adoption of IPv6 within consumer, IPTV and 3G / 4G mobile providers for management of their internal infrastructure. You can pick your favorite market driver: 3G / 4G Mobile IP, Digital Radio, RFID, Control Networks, etc. But the numbers for globally unique end devices is staggering no matter which trend you choose.

    For example, Comcast has migrated all of their internal control network to IPv6 with plans to manage 100 million cable modems.

    Various estimates place number of network devices that will need to be managed at 12 billion by 2012. Note that these devices may not need global visibility, but providers will need to at least internally provision and manage (and RFC1918 space is not a reasonable option).

  2. Market trumps technology. And politics trumps markets.
    The future of the Internet is not fixed line devices. Nor is it users in the United States.The future of the Internet is likely both mobile devices and emerging Internet countries like China which reportedly surpassed the number of US web users at 253 million last month.While politics and government mandates do not always drive technology (see GOSIP or metric system in the United States), sometimes they do (see metric system in United Kingdom).

    Throughout the world, government mandates are spurring IPv6 adoption. China’s CNGI initiative and billions in spending uses IPv6 as the centerpiece. Similarly, Japan, Korea all have major IPv6 initiatives. The EU called for mass migration to IPv6 by 2010.

    The important bit to realize about governmental focus on IPv6 is that it is not about technology nor is it even really about IPv6. Many governments view IPv6 through the prism of politics. These countries, rightly or wrongly, view ICANN and the large US centric legacy IPv4 allocations as instruments of US control. For China, Japan and many EU nations, IPv6 is really about no less than who will control the future of the Internet.

  3. IPv6 has already succeed.
    You can now get native IPv6 from many providers, including Verizon, Tata (formerly VSNL/Teleglobe), Global Crossing and others. Over half of surveyed providers say they have plans to roll out commercial IPv6 offerings in the next year. As more vendors integrate IPv6 into their products lines, the ISP IPv6 tax has correspondingly dropped. For many, IPv6 comes with latest refresh of hardware which ISPs generally amortize over 5-8 year periods. While it will be many years before the millions of embedded end devices support IPv6, your backbone routers likely already do.Most encouraging of all, there is finally the beginning of real IPv6 content. Or at least you can now use IPv6 to search content (as long as the indexed content is IPv4). At the Philadelphia IETF this year, Google announced support for production quality IPv6 search at

  5. And the final reason IPv6 will succeed?
    No one has the stomach to spend another twenty years replacing IPv6 with something else.

Personally though, I just configured our local router for IPv6 so I can watch Michael Phelps (former University of Michigan athlete) win eight golds this week at http://2001:252:0:1::2008:6.

Full disclosure — I worked on the failed TUBA counter-proposal to IPv6 and still harbor a grudge.

2% of Internet Traffic Raw Sewage

By: Danny McPherson -

For the last 18 months or so here at Arbor we’ve been recruiting ISPs that currently use Peakflow SP systems to participate in our statistics sharing program. The goal of the program is to try and better understand Internet traffic and attack characteristics over time, to include protocol and packet size distributions, attack vectors, frequency and scale, source and target distributions, etc. Some of this information can be accessed in real-time via our ATLAS portal, with more to come.

The statistics sharing program is based on flow data (e.g., NetFlow, JFlow, IPFIX) collection systems, which deal primarily with Network and Transport Layer (Layer 3 & 4) traffic information, and data currently being collected here is only from interfaces participants have classified as inter-domain (i.e., not internal or customer).

We’ve currently got 68 discrete ISPs participating, covering over 100k interfaces on nearly 1300 routers, and peak inter-domain traffic rates are currently nearing 1.5 Tbps, which is a statistically significant number.

ASP Dashboard

We currently see somewhere around 1300 DDoS attacks a day on average, we’ve seen nearly 1 million since we began the program, and we’re getting to a point where after 1.5 years of collection, some trends are beginning to emerge. For example, attack frequency seems to drop significantly on Christmas Day, New Year’s Eve, and New Years Day (perhaps while the miscreants are either hung over or expending their spoils :-). The most common targets we see are IRC servers, although those attacks are usually lower-scale and not as well distributed as some of the larger attacks. The most common attack vectors are TCP SYN floods, with ICMP floods being a close second. It’s also particularly interesting to compare and contrast protocol distributions (e.g., peer-2-peer, http, etc..) and rates for inter-domain traffic versus broadband dense segments or other demographics. We’re intending to publish a report based on this data in the near future, so I won’t spoil it with any more details here.

24 hour Attack Rate Snapshot

However, one finding I did want to point out that was somewhat surprising is that DDoS (i.e., brute-force flood-based attacks) have over the past 18 months consistently accounted for ~1-3% of all all inter-domain Internet traffic. Again, this is raw attack traffic, simply meant to exhaust connection state or fill links, nowhere in this mix is spam, phishing, scans, or other malicious or similarly annoying traffic. We have seen peaks well above 5% of aggregate reported traffic, although not consistently.

As you might suspect, that’s no small amount of wasted resources consumed by DDoS attack traffic. TCP/25 (SMTP – email) seems to hover around 10-15 Gbps, so 1-1.5%. If you were to assume that only 66% of that is spam (which is likely a very low estimate, and one that varies rather widely), you get ~1%, so we’re at nearly 4% of all inter-domain traffic as junk, with over half being raw sewage.. Anyways, we’ve got some work to do to sure up these numbers and provide something folks can reference, you should be seeing something more definitive along these lines in the coming months.

ddos de da: Internet attacks still considerable

By: Danny McPherson -

Here at Arbor we’re working with many of our service provider partners on trying to qualify and quantify denial of service attacks and other network threats. Here are a few data points relative to DDoS attacks we’ve observed over the past 255 days of data collection:

  • 255 days of data collection
  • 39 ISPs participation average
  • ~1 Tbps of monitored inter-domain traffic
  • ~143k rate-based attacks (~278k total attacks)
  • 58% of attacks were TCP-based (80, 25, 6667, 22 leader ports)
  • 36% were ICMP
  • 5% were UDP (fragments well over majority)
  • 15% of attacks were TCP SYN (>94% had constant source sets and were likely NOT spoofed)

As for scale and frequency of attacks, of the 255 days of collection the following number of days had at least one attack exceeding the indicated threshold:

  • 6 Mpps – 1
  • 5 Mpps – 12
  • 4 Mpps – 33
  • 3 Mpps – 53
  • 2 Mpps – 91
  • 1 Mpps – 151

Total attacks over 255 days exceeding a given threshold:

  • 6 Mpps – 1
  • 5 Mpps – 17
  • 4 Mpps – 82
  • 3 Mpps – 135
  • 2 Mpps – 352
  • 1 Mpps – 823

Note that the above million packet per second (Mpps) attacks are from the perspective of a single participating ISP, an ISP which could be ingress, transit or edge network for the attack target. As such, it’s extremely likey that upon performing cross-ISP correlation (which is done but not fully analyzed) of the attack target data sets a much larger number of attacks will exceed the one million packets per second mark, and manual inspection already reveals that the aggregate of some of these attacks is far greater than even 10 Mpps!

To put this in perspective, the most crippling of the Estonian attacks had peak rates averaged over a 24 hour period of about 4 Mpps. 4 Mpps is a very large attack, and while less than 1% of the attacks we see exceed the Mpps mark, these attacks are nothing to ignore, pretty much regardless of who you are or what’s motivating an attacker.

We hope to release some formal analysis on the attack and traffic statistics we’ve been collecting, look for something here sometime soon. Volume III of the Infrastructure Security Survey is currently being compiled as well. With any luck, between these data sets we’ll be able to provide qualitative information on denial of service and Internet attack trends.

On DDoS Attack Activity

By: Danny McPherson -

We’ve been doing analysis on the DDoS attack and network traffic distribution data some of our Peakflow SP customers are providing and I figured I’d share a bit of a teaser. The data is shared with Arbor via an optional module within Peakflow SP, so if you’re wondering how it’s gathered have a look here.

We’ve got 26 SP deployments participating at the moment (and still growing) and have been archiving attack and traffic data daily for about four months now. Some stats on the data gathered thus far:

  • the data is representative of only inter-provider traffic and attack activity (customer and internal attack activity explicitly excluded)
  • about .5 Tbps in aggregate
  • about 500 routers, 30,000 unique interfaces
  • ~126 day collection period
  • ramp from 12 to 26 participants during that period
  • 120,231 attacks reported (954 attacks/day average)

A daily high of 1991 attacks was observed on 11/8/2006. There are also some discernible drops in aggregate attack activity around Christmas and New Years, perhaps the miscreants were distracted with the holidays?

Plot of Aggregate Attacks Per Day

Lots of interesting information can be gleaned from the data. For example, TCP attacks lead the pack at the moment, followed by ICMP and UDP-based attacks. Of the TCP-based attacks, SYN floods are the most prominent attacks, followed closely by Null and “Christmas tree” attacks.

Attack and routing data is shared via XML, a typical attack fragment looks like this:

Attack 122002

This attack was one of the larger attacks observed over the current period, with a maximum packet rate of ~6.2m packets per second. It appears to have been source IP and port spoofed (hence the and 0-65535 fields versus something more specific). The SP reporting the attack above observed it ingressing the network via 19 different routers, 52 different interfaces, pretty well distributed. It was a Null TCP attack (no flags) targeting a, umm.., “popular” IRC server (TCP/6667), whose IP has been anonymized here. Not surprisingly, given the scale and distribution of this attack, several of the other participating SPs reported some of the attack flows via their networks as well.

Many other attack and traffic attributes are available, from packet sizes, ports and protocols to detailed Network and Transport Layer attack vectors for each reported attack, to include sources and targets, etc.. Both the specific attacks and data on aggregate, correlated over time will provide some interesting perspective.

If you’ve got thoughts, questions, or comments, ping Craig Labovitz or I. Otherwise, stay tuned, as you’ll be seeing a great deal more analysis of this and related data in the near future…

Battling the Stupid-Bit

By: Rob Malan -

The Evil Bit! I’ve been thinking about RFC-3514 often over the last few quarters; that and what should be its cousin: the Stupid-Bit. I know you’re shaking your head now – poor CTO…too much time in the sales/marketing dunk tank. I’m serious, though. Not that you should be able to look at a bit in a packet and know its intent (e.g. malicious, web-surfing, financial transaction…), but rather that by understanding the context of the packet, you should be able to decide its fate. As a thought exercise, put yourself at any point in the network, and, given a packet, ask yourself, “What do I need to know about this packet in order to forward it along the way?” The set of things that our current security and networking infrastructure ask are pretty rudimentary: do we know how to get where it wants to go, i.e. have a route, arp mapping, etc.? Is it allowed to go to that host and service – match a firewall ruleset? Does it carry obvious maliciousness – match an IPS vulnerability or exploit signature? The industry is pushing this a bit further: is it allowed to talk on this port and is it coming from an authorized machine (NAC)? However, it’s my guess that you could ask yourselves a bunch of other pretty relevant questions — some of these divining an implicit evil-bit or more likely a stupid-bit set to one.

Arbor’s spent the last five years building both provider and enterprise solutions that measure network-wide normal behavior. Give us a point in your network and a packet — we can tell you if you’ve seen it before; how often; to which other hosts; between which users; what other applications were in the mix at the time; etc. We have done this in the past with flow-based inputs, and increasingly with application-specific data generators — such as our Authflow Identity Tracking agents. If you have talked to our sales-force, pm-force, or me in the last nine months, you’ve probably heard about what we’re doing in the area of deep packet data inputs. The idea is to build out our context not just network-wide, but also up and down the stack. I won’t spoil the marketing launches, so enough said about that; but where we’re going is increasingly towards putting this network-wide context into action. If you can answer a lot more questions about a packet, you can make a much better forwarding decision. Networks need to get smarter or at least not work so hard at forwarding traffic with the stupid-bit set to one.

What’s the Future of Application Analysis in Provider Networks?

By: Carlos Morales -

Application analysis products are in vogue. The bandwidth that is now available to broadband users has enabled providers to offer converged solutions consisting of voice, data and in some cases video to these same end users. These services provide a much-needed set of revenue streams to providers but they also present them with a lot more challenges. The most notable effect is that this pretty much eliminates the idea that data transport is done on a “best effort” basis. Just try telling a consumer that his 911 calls cannot go through because his neighbor is downloading the latest Shakira tune from a PC in Albuquerque. The result: providers have to be much more cognizant of what is traveling over their network and have to take steps to make sure that these revenue producing services take precedence. That brings us back to application analysis. How can you assign priorities when you don’t know what is going on in the network?

Several techniques can be used to figure this out. Flow based solutions such as Cisco Netflow, Juniper cFlow and Foundry sFlow offer a scalable, low cost approach to gain insight into application use throughout the network based on layer 3 and 4 packet information. With it, you can get a lot of useful data on what the makeup is of congestion in key parts of the network and who the hosts are that are generating a lot of the traffic. This approach provides you with the information necessary to make capacity planning decisions and to detect many forms of network abuse. However, it does not completely address all of the applications that providers really want to analyze and does not answer all the questions that will be asked about these applications. Applications such as peer-to-peer can masquerade using ports commonly used on the Internet like http port 80. Other applications such as VOIP RTP use a range of UDP ports that are negotiated out of band and are difficult to key in on using just the port numbers. These applications require delving into the payload information in layers 5-7 of the packets so most flow based data cannot be used to definitively pinpoint this traffic. Extensible versions of flow have come onto the scene recently in the form of Netflow v9 and IPFIX, which have added the capability of including payload data in the export; sFlow has had this functionality for some time. This bears further attention in the future but there are a few barriers to deployment of this today. The support for these new standards is not universal among vendors or even within vendors, the level of support can vary from platform to platform, and there is some legitimate concern about what the impact will be on performance on routers and in network bandwidth when payload is included in the flow exports.

Span or tap based solutions offer considerably more insight into this type of traffic but also bring along some significant deployment challenges. They have visibility into the layer 5-7 information so that they are more capable of fingerprinting applications based on content. With a span port, you can look into the packet payload and potentially re-assemble application data. This gives the ability to accurately say how much of your port 80 traffic is made up of Kazaa vs. BitTorrent vs. http and it could give you the ability to intercept and record a VOIP call from the wire. The downsides of this method are cost scale and management. To get full network coverage across the edge of the provider network, say just upstream of your dslams, or on an MSO just above the headends, there would be a need to deploy 10s or hundreds of these devices. To avoid the problems presented by asymmetry, you really have to deploy much closer to the edge to be effective. The devices themselves are not cheap either. To achieve layer 5-7 inspection on Gbps + links that are feeding end user segments, the devices need to be equipped with reasonably sized switching backplanes and higher end network processors. Another challenge of this method is the actual management of the devices. When you deploy a number of these across your network, it becomes more critical to have some type of central platform that can be used for correlation, reporting and configuration management.

In-line solutions offer yet another set of options to the user but introduce more challenges. With an in-line solution, mitigation in the form of traffic blocking, rate limiting, traffic shaping and stateful inspection becomes possible. Not only are you able to analyze applications and detect issues on the applications, you are able to do something about it. That is very compelling. However, in-line solutions are the bane of a net ops guy’s existence. It is something new to put in my network that could potentially go down and lead to me being paged at 3 in the morning. Things like full redundancy, 5 9’s reliability, fail open relays, hitless upgrades and other measures become immediate requirements. This will delay the time to market for solutions and will definitely inflate the cost associated with these systems.

The next consideration is feature set. Is it enough just to know how much Napster traffic is parading on port 80 in my network or how many times Jim has called Jane using Vonage? That is certainly interesting data and worth having but as with all data, it tends to open up the door to more questions and requirements. This is particularly true given the investment that is necessary to profile some of these applications across a provider edge. Looking at recent security markets, it is clear that the requirements on products adapt over time from a “tell me what’s going on” to a “do something about it” scenario. You need not look further than the IDS market for an example of this. Intrusion detection emerged commercially en masse in the mid 90’s and most major enterprises deployed them along their perimeters. Over time, these customers realized that just knowing that a worm or a hacker is intruding on the network was not enough; they needed to be able to block them in real time. This spurred on the birth of the IPS, which has dominated the perimeter security space for the last few years. Application reporting is already taking a similar trajectory. Some peer-to-peer reporting solutions available today already include the ability to do rate limiting in real time. VOIP solutions are being developed to detect inappropriate use of VOIP, man in the middle attacks, and VOIP based DDoS. This is just the tip of the iceberg in terms of feature sets that are going to be required. Application analysis solutions need to have a roadmap towards providing more of this functionality while providing for scale and central management requirements.

So all of these options have virtues and challenges, which is the right one? The answer can be found by taking a page out your standard security approach: layer your defenses and adapt them over time. You do not have to choose just one option; you just have to choose your options wisely such that they solve your immediate needs, will ultimately integrate together and will grow with your needs. One could start with a flow-based system that for short money could provide a very decent level of coverage and provide for congestion analysis, capacity planning, acceptable use and quality of service analysis. You can then choose the key locations on your network where to deploy deeper application analysis targeted at the applications that are strategic or potentially more harmful to you. For instance, peer-to-peer traffic on the network is not that big of an issue for a provider unless there is congestion in a portion of the network where applications that are more useful are vying for the same network bandwidth. Initial peer-to-peer analysis appliances using span ports can be deployed in these parts of the network first. VOIP analysis can be performed initially at key provider inter exchange locations and regions of the network where these services are more predominant. As the solutions mature and services continue to grow, the footprint can be gradually expanded until full network coverage is achieved.

The key to making this all work is integration. Whether buying a complete solution from a single vendor or buying from multiple vendors and tying them together using a network management or security event management product, it is crucial to maintain an ability to manage and control the elements in a hierarchical manner. The more fragmented the management and reporting for the system, the bigger the burden that will be put on the operations group that will need to manage it. You can deploy a number of best of breed solutions for specific purposes but if there is no way to ultimately integrate them together or obtain useful correlation, the operations burden could outweigh the benefits of the solution.