Tag Archives : Internet service provider March 2010

How Big is Google?

By: Craig Labovitz -

Google’s recent FTTH announcement generated a wave of media coverage and industry discussion. Responses ranged from exuberant local communities racing to sign up to anti-competitive howls from incumbent carriers.

Industry pundits wondered what is Google up to? What will the search giant do with 1Gbps to the home? And more ominously, is Google getting too big?

While this blog post won’t explore the politics / strategy behind Google’s FTTH initiative (except to suggest Google should choose Ann Arbor), we will share some data on Google’s relative size and growth from a global Internet perspective.

Google is big.

And by “big”, I mean really big. If Google were an ISP, it would be the fastest growing and third largest global carrier. Only two other providers (both of whom carry significant volumes of Google transit) contribute more inter-domain traffic. But unlike most global carriers (i.e. the “tier1s”), Google’s backbone does not deliver traffic on behalf of millions of subscribers nor thousands of regional networks and large enterprises. Google’s infrastructure supports, well, only Google.

Based on anonymous data from 110 ISPs around the world, we estimate Google contributes somewhere between 6-10% of all Internet traffic globally as of the of summer of 2009.

The below graph shows the weighted average percentage of all Internet traffic contributed by Google ASNs between June 2007 and July 2009. Most of Google’s rapid growth comes after the acquisition of YouTube in 2007.

Google's Contribution to Global Internet Traffic

Before getting much further, a few words about what we’re measuring. Traffic volumes provide only the most indirect measure of a network’s size or popularity (for example, it takes tens of thousands of Tweets to match the bandwidth of a single HD video). Our anonymous data also does not include internal provider services (e.g. IPTV or VPN) nor data served from co-located caches within provider data centers. Rather, we’re measuring inter-domain traffic, i.e. the traffic between providers (the “inter” in “Internet”).

With all of the above said, inter-domain traffic volumes provide a key metric for understanding Internet topology and the evolution of Internet traffic patterns.

But even traffic volumes tell only part of the story.

The competition between Google, Microsoft, Yahoo and other large content players has long since moved beyond just who has the better videos or search. The competition for Internet dominance is now as much about infrastructure — raw data center computing power and about how efficiently (i.e. quickly and cheaply) you can deliver content to the consumer.

And here again, Google is at the head of the pack.

In 2007, Google used transit providers for the majority of their Internet traffic (including Level(3)). But over the last three years, Google both built out their global data center and content distribution capability as well as aggressively pursued direct interconnection with most consumer networks.

The graph below shows an estimate of the average percentage of Google traffic per month using direct interconnection (i.e. not using a transit provider). As before, this estimate is based on anonymous statistics from 110 providers. In 2007, Google required transit for the majority of their traffic. Today, most Google traffic (more than 60%) flows directly between Google and consumer networks.


But even building out millions of square feet of global data center space, turning up hundreds of peering sessions and co-locating at more than 60 public exchanges is not the end of the story.

Over the last year, Google deployed large numbers of Google Global Cache (GGC) servers within consumer networks around the world. Anecdotal discussions with providers, suggests more than half of all large consumer networks in North America and Europe now have a rack or more of GGC servers.

So, after billions of dollars of data center construction, acquisitions, and creation of a global backbone to deliver content to consumer networks, what’s next for Google?

Well, I’m hoping for delivery of content directly to the consumer via a nice, fat 1 Gbps FTTH pipe.

Google, please choose Ann Arbor.

Who Put the IPv6 in my Internet?

By: Craig Labovitz -

About this time last year, we released a study on the state of IPv6 deployment in the Internet. Our August 2008 paper found diminishingly small traces of IPv6 — less than one hundredth of 1% of Internet traffic.

This year?

In a dramatic reversal of long-term IPv6 stagnation, global IPv6 traffic globally grew more than 1,400% in the last 12 months. Even more remarkable, this growth is due primarily to one application and one ISP.

We’ll explain in a moment, but first some background: Both our 2008 work and this IPv6 study used traffic statistics from 110 ISPs participating in the Internet Observatory. Though the Observatory is capable of collecting native (i.e. not tunneled) IPv6 traffic statistics, only six ISPs out of the 110 in the study currently have routers and collection infrastructure with native IPv6 enabled. As a result, our data generally includes only IPv6 traffic through Teredo and 6to4 tunnels. Further, since only a handful of Observatory participants use monitoring infrastructure with payload visibility, our study sees only the UDP Teredo control traffic (i.e. not the data portion).

The above technical limitations and our somewhat dismal 2008 assessment of IPv6 deployment engendered a bit of criticism. The main critique (such as this posting) seems to be that we significantly under counted IPv6. In particular, many pointed to the Amsterdam Internet Exchange (AMS-IX) switch statistics which show a Gigabit or more of IPv6 traffic (far more than we found in our study). Others pointed to the high rate of IPv6 address allocations as evidence of broader IPv6 deployment.

From the perspective of a year later, we stand by our 2008 IPv6 findings. A July 2009 news server outage confirmed suspicions that AMS-IX IPv6 traffic mainly consisted of file sharing through the free AMS-IX based IPv6 news servers. And a PAM paper earlier this year found both minuscule levels of IPv6 traffic in a tier1 network and confirmed that registry allocations provide a poor indicator of IPv6 usage. As a side note, the PAM paper also found that the small amount of tier1 IPv6 traffic consisted mainly of DNS and ICMP (i.e. test traffic and not real IPv6 usage).

So in August of 2008 real IPv6 Internet traffic was mostly non-existent.

And then things changed…
The above graph shows IPv6 traffic (Teredo and 6to4) as a normalized weighted average percentage of all Internet traffic between July 2007 and July 2009. In July of 2007, IPv6 represented less than 0.002% of Internet traffic. Beginning in August of 2008, tunneled IPv6 traffic begin to grow dramatically followed by an abrupt and even larger jump in April of 2009 (the E. Karpilovsky et al. PAM paper also observed this first 2008 jump in traffic but did not speculate as to the causality).

What happened?

This stark August 19, 2008 warning to the NANOG mailing list by Nathan Ward provides a strong clue:

Sit up and pay attention, even if you don't now run IPv6, or even if you don't ever intend to run IPv6.
Your off-net bandwidth is going to increase, unless you put some relays in.
As a friend of mine just said to me: "Welcome to your v6-enabled transit network, whether you like it or not ;-)".
uTorrent 1.8 is out, as of Aug 9.

Nathan was mostly right. While uTorrent never generated the expected flood of new traffic (at least by IPv4 standards), the introduction of IPv6 P2P succeeded where most previous IPv6 inducement efforts had failed (i.e. liberal peering, high quality IPv6 porn, IPv6 ASCII animation of Star Wars, etc.). In the space of ten months uTorrent helped drive IPv6 traffic from .002% to .03% of all Internet traffic (a dramatic 15x jump).

But the more interesting (and from an infrastructure perspective, far more important) IPv6 traffic increase came on April 21, 2009 with Hurricane Electric’s turn up of a global anycast’ed Teredo relay service. Hurricane Electric enabled 14 Teredo relays in Seattle, Fremont, Los Angeles, Chicago, Dallas, Toronto, New York, Ashburn, Miami, London, Paris, Amsterdam, Frankfurt and Hong Kong.

More details of Hurricane Electric’s infrastructure is available in this May 2009 LACNIC presentation.

Historically, IPv6 connectivity across the Internet has been, well, abysmal. Inefficient routing, multiple IPv6 tunnel encapsulations and overall lack of coordination between Teredo and 6to4 relay providers added latency, loss and played havoc with jitter (i.e. mangling VoIP). Frequently, a traceroute between two providers at the same exchange could traverse multiple countries or continents en route. For added background, see this 2009 Google IPv6 Conference presentation, this 2008 RIPE study and related 2007 study.

By all accounts, Hurricane Electric’s Teredo service significantly improved the IPv6 goodput for the average Internet end user over night. In particular, Microsoft Windows users got a big boost. Though Windows has shipped with a Teredo client (on by default) since XP, Microsoft never provided a public relay service. teredo.ipv6.microsoft.com now uses Hurricane’s 6to4 relays. And the dramatic improvement in Teredo and 6to4 relays seems to have lead to a corresponding jump in IPv6 traffic.

This is good news.

Finally, in the below graph, you can see the impact of both uTorrent and Hurricane’s relay deployment by region. We again show IPv6 tunneled traffic as a weighted normalized percentage of all Internet traffic. The most important take away is that the IPv6 growth after August 2008 is a global phenomena (with Asia at the forefront follow by Europe).
We look forward to revisiting IPv6 traffic in another year as relays improve, meaningful IPv6 content becomes available and more providers offer native IPv6 service.

Editor’s Note: This blog is the fourth in a series of weekly posts leading up to the publication of the joint University of Michigan, Merit Network and Arbor Networks “2009 Internet Observatory Report”. The full technical report goes into detail on the evolving Internet topology, commercial ecosystem and traffic patterns — available this October. Next week: “How Big is Google?”


Reblog this post [with Zemanta]

The Internet After Dark (Part 2)

By: Craig Labovitz -

This blog completes our informal three week study of Internet daily traffic patterns. Using data from the Internet Observatory, we analyzed weekday application traffic across 110 geographically diverse ISPs, including some of the largest carriers in North American and Europe. We believe this report (and upcoming paper) represent the largest study of Internet traffic temporal characteristics to date.

In the first half of this post, we showed unlike European Internet traffic which peaks in the early evening and then drops off until the next day’s business hours, US Internet traffic reaches its peak at 11pm EDT and then stays relatively high until 3am in the morning.

The question is what are Internet users doing after dark?

The answer: long after Exchange and Oracle business traffic slows to a crawl, Internet users turn to the web to surf, watch videos, send IM’s and happily try to kill each other.

We illustrate these trends with graphs of four application categories below.


The top two graphs show the daily average traffic fluctuations of TCP / UDP ports related two popular online game multi-player platforms: World of Warcraft and Steam (which includes many popular first person shooter games like Half Life). The bottom two graphs show common video and instant messaging protocols. As in earlier analysis, we take the average of North American consumer / regional providers traffic over 10 weekdays in July. To make the graph more readable, we show traffic as a percentage of peak traffic levels. All times are EDT.

Some observations:

  • Gamers Come Out at Night: Unlike most Internet applications which peak midday or late afternoon, online game traffic grows by more than 60% after 2pm. Gaming prime time appears to be between 8pm and 11pm EDT weekday nights (corresponding to the traditional and now declining television prime time hours). By comparison, web traffic levels remain relatively constant through the late afternoon and peaks much earlier at 5pm.
  • A Guild that Plays Together Stays Together: Unlike other online game traffic, World of Warcraft’s Battlenet shows a distinct 30% jump exactly at 8pm EDT every evening. In-house WoW level 80 colleagues suggest 8pm is a common time for guilds to set out on quests. Also unlike other game traffic, WoW declines rapidly after 11pm every night. Again, we suspect WoW traffic patterns are related to the more large group, social nature of World of Warcraft.
  • Midnight Video: Of all Internet applications, streaming video protocols reach their traffic peak the latest around midnight EDT every evening. We do not have very good visibility into what Internet users are watching this late, but correlation with large content site traffic patterns (below) provides some clues.
  • Always in Touch: Beginning at 9am EDT at lasting though midnight, Internet users IM constantly. The IM graph above shows traffic reaches 80% of peak by 10am and stays above 80% until midnight (with a 5pm EDT peak — perhaps related to millions of users making dinner plans). Interestingly, email exhibits a very different pattern and plummets by more than 30% immediately after 5pm EDT.

As mentioned earlier, we do not have detailed visibility into what Internet users are watching at midnight but ASN level traffic analysis provides some hints. Predictably, traffic grows dramatically to consumer sites like Google’s YouTube and large CDN / video providers. Also not surprisingly, we see a large jump in traffic to colo / hosting companies with adult content such as a 40% jump to ISPrime (AS23393) between 10pm and 1am EDT. We will explore one of the fastest growing and largest nighttime sites, Carpathia Hosting (AS29748), in an upcoming blog.

Editor’s Note: This blog is the third in a series of weekly posts leading up to the publication of the joint University of Michigan, Merit Network and Arbor Networks “2009 Internet Observatory Report”. The full technical reports goes into detail on the evolving Internet topology, commercial ecosystem and traffic patterns — available this October. Next week: “Who Put the IPv6 in My Internet?”

Reblog this post [with Zemanta]

The Great GoogleLapse

By: Craig Labovitz -

Web sites go down. Circuits fail. Network engineers goof router configs. And few of these outages ever make the nightly news…

But if you happen to be Google and your content constitutes up to 5% of all Internet traffic, people notice.  Network engineers around the world frantically email traceroutes to mailing lists. IRC channels fill with speculation (“definitely was a DDoS attack”, “no, a worm”, “it was ISP xxx’s  fault!”). And end users Twitter (a lot).

So what does it look like when 5% of the Internet disappears on an otherwise uneventful Thursday morning? The below graph shows average traffic across 10 tier1/2 ISPs in North America from Google’s network (ASN 15169). Outage began roughly at 10:15am and lasted through 12:15pm EDT.

Looking at the data, most large transit providers appear to have been impacted (e.g., Level3, AT&T, etc.). Other providers (e.g. large consumer DSL / Cable) showed no drop in traffic from/to Google.

Looking at BGP (below snapshot is from Arbor’s Routeviews Servers) we see a lot of churn in Google’s BGP routes around the outage timeframe — one prefix I choose at random flaps across half a dozen providers before getting withdrawn.

In a recent official company blog post, Google blamed some combination of airplanes and BGP for the outage.

Reblog this post [with Zemanta]

Highlights of Arbor Networks’ Fourth Annual Worldwide Infrastructure Security Report

By: Arbor Networks -

Highlights of Arbor Networks’ Fourth Annual Worldwide Infrastructure Security Report


  • Tom Bienkowski, Director of Product Marketing, Arbor Networks
  • Danny McPherson, Vice President and Chief Security Officer, Arbor Networks


For the past four years Arbor Networks has conducted a survey of many of the world’s network operators. This survey covers topics such as: Most significant network based threats; common attack vectors and targets; methods of attack detection and mitigation; and other related questions regarding size of staff, use of law enforcement, managed security services, etc. The results of the survey are compiled into the Worldwide Infrastructure Security Report.

This 45 minute video will focus on the more interesting highlights of this year’s report which was a culmination of responses from approximately 60 different network operators from around the world and their experiences in 2008.

Run time: 49:41 (Registration is required to view this Webcast – Click here to be taken to the registration page)

Reblog this post [with Zemanta]

The Great Obama Traffic Flood

By: Craig Labovitz -

Streaming video traffic coverage of Obama’s inauguration flooded North American backbones today. Traffic increases varied wildly across US providers with some seeing an overall 5% increase in backbone traffic and others jumping more than 40%.

This multi terabit per second flood represents one of the single largest one day spikes in Internet traffic since ATLAS Internet Observatory monitoring began five years ago. Apparently, US presidents are more popular than pro golfers — the inauguration traffic handily beat the last Internet traffic record set during the US Open.

While most of the US infrastructure appears to have withstood the flood, at least two ISPs showed clear failures and traffic drops during the traffic peaks coincident with Obama’s swearing in (traffic levels started to drop quickly beginning with the subsequent poetry readings).

Chiefly the traffic surge centered on Flash (TCP port 1935) and UDP port 8247 (which includes CNN streaming). In the US, most of these increases focused on consumer (DSL / MSO) providers and transit ISPs (especially those interconnecting large CDNs). Flash traffic spiked by more than 60% in most providers and by 400% in a few of the larger cable operators.

The below graph shows both of these ports across 10 of the largest US ISPs participating in Arbor’s ATLAS Internet Observatory traffic sharing initiative (see NANOG presentation for more details).

Great Obama Traffic Flood

While US backbones saw a large inauguration traffic spike, Europeans and Asian viewers appeared less interested in US politics with an under 1% increase in backbone traffic (in fairness, timezone differences also likely had a significant impact). Our Canadian neighbors proved more interested with a 2-5% growth in backbone traffic today.

Though multiple content providers hosted the traffic streams today, Limelight (AS 22822) was one of the clear winners — ATLAS data across the ten US consumers ISPs show a massive increase in AS22822 traffic (median of 160%). Akamai showed a more modest increase of 17%.

The Obama inauguration marks a historic day in US politics and a remarkable day for the popularity of Internet streaming video. We look forward to watching more great things to come.

(Co-authored with Scott Iekel-Johnson)

Reblog this post [with Zemanta]

New attack patterns emerge in 2009

By: Arbor Networks -

Botnets were just the beginning. The bad guys will continue to use these to try and steal your data, but more sophisticated attacks over the application layer and targeted network attacks are on the way.  In this Network World Podcast, Danny McPherson from Arbor Networks discusses the new ways that hackers will be trying to get into (and steal information) from your network in 2009.

Reblog this post [with Zemanta]

2008 Worldwide Infrastructure Security Report

By: Craig Labovitz -

Growing financial pressures, unforeseen threats, and a volatile and rapidly changing business landscape — apt descriptions for both the world economy and this years Worldwide Infrastructure Security Survey.

Arbor Networks once again has completed a survey of the largest ISPs and content providers around the world. Some 70 lead security engineers responded to 90 questions covering a spectrum of Internet backbone security threats and engineering challenges. This fourth annual survey covered the 12-month period from August 2007 through July 2008.

A copy of the full report is available at http://www.arbornetworks.com/report

The most significant findings:

  • ISPs Fight New Battles
    In the last four surveys, ISPs reportedly spent most of their available security resources combating distributed denial of service (DDoS) attacks. For the first time, this year ISPs describe a far more diversified range of threats, including concerns over domain name system (DNS) spoofing, border gateway protocol (BGP) hijacking and spam. Almost half of the surveyed ISPs now consider their DNS services vulnerable. Others expressed concern over related service delivery infrastructure, including voice over IP (VoIP) session border controllers (SBCs) and load balancers.
  • Attacks Now Exceed 40 Gigabits
    From relatively humble megabit beginnings in 2000, the largest DDoS attacks have now grown a hundredfold to break the 40 gigabit barrier this year. The growth in attack size continues to significantly outpace the corresponding increase in underlying transmission speed and ISP infrastructure investment. The below graph shows the yearly reported maximum attack size.
  • Services Under Threat
    Over half of the surveyed providers reported growth in sophisticated service-level attacks at moderate and low bandwidth levels attacks specifically designed to exploit knowledge of service weakness like vulnerable and expensive back-end queries and computational resource limitations. Several ISPs reported prolonged (multi-hour) outages of prominent Internet services during the last year due to application-level attacks.
  • Fighting Back
    The majority of ISPs now report that they can detect DDoS attacks using commercial or open source tools. This year also shows significant adoption of inline mitigation infrastructure and a migration away from less discriminate techniques like blocking all customer traffic (including legitimate traffic) via routing announcements. Many ISPs also report deploying walled-garden and quarantine infrastructure to combat botnets.

Overall, ISP optimism about security issues reported in previous surveys has been replaced by growing concern over the new threats and budget pressures. ISPs say they are increasingly deploying more complex distributed VoIP, video and IP services that often poorly prepared to deal with the new Internet security threats. More than half of the surveyed ISPs believe serious security threats will increase in the next year while their security groups make do with “fewer resources, less management support and increased workload.”

ISPs were also unhappy with their vendors and the security community. Most believe that the DNS cache poisoning flaw disclosed earlier this year was poorly handled and increased the danger of the threat.

Finally, the surveyed ISPs also said their vendor infrastructure equipment continues to lack key security features (like capacity for large ACL lists) and suffers from poor configuration management and a near complete absence of IPv6 security features. While most ISPs now have the infrastructure to detect bandwidth flood attacks, many still lack the ability to rapidly mitigate these attacks. Only a fraction of surveyed ISPs said they have the capability to mitigate DDoS attacks in 10 minutes or less. Even fewer providers have the infrastructure to defend against service-level attacks or this year’s reported peak of a 40 gigabit flood attack.

As always, this work would not be possible without the support and participation of the Internet security community. The 2008-2009 survey will be released next Fall.

Reblog this post [with Zemanta]