Tag Archives : Internet traffic January 2011

Egypt Loses the Internet

By: Craig Labovitz -

Updated January 31: Added graph and discussion of remaining active paths

Following a week of growing protests and periodic telecommunication disruption, Egypt suddenly lost all Internet connectivity at approximately 5:20pm EST Thursday.

The below graph shows traffic to and from Egypt based on ATLAS data from 80 providers around the world.

Between 3 and 5pm EST, Egyptian traffic rapidly climbed to several Gigabits. At 5:20pm, the all Egyptian transit providers abruptly withdrew the major of Egypt’s several thousand BGP routes and traffic dropped to a handful of megabits per second.

At present, the cause of the outage is unknown though many press reports have drawn parallels to the Internet outages following Iranian political protests during the summer of 2009. Further, the simultaneous failure of Internet across multiple different Egyptian ISPs and diverse physical paths (i.e. satellite, fiber optic, etc) suggests this was a coordinated event rather than a natural failure. Typically, telecommunication companies operate under strict regulatory control in many countries around the world.

As of Monday (January 31), Egypt remains disconnected from the Internet. A week view of traffic in and out of Egypt below.

Normally, Egypt enjoys one of the largest and most robust Internet infrastructures in Africa with a dozen major providers, more than 30% consumer penetration, and multiple high-speed paths to Europe and the rest of the world. Egypt also serves as a major terrestrial fiber optic crossing point for traffic to other countries in Africa and the Middle East. Traffic to other countries using these links through Egypt has not been impacted.

While the Egyptian telecommunication market has enjoyed significant liberalization in the last decade, the Egyptian government Telecommunications Regulatory Authority (TRA) continues to assert a strong level of regulatory control over the telecom licensees. See http://www.tra.gov.eg for more information (although the TRA web site is currently unreachable outside Egypt).

Google Sets New Internet Traffic Record

By: Craig Labovitz -

In their earnings call last week, Google announced a record 2010 third-quarter revenue of $7.29 billion (up 23% from last year). The market rejoiced and Google shares shot past $615 giving the company a market cap of more than $195 billion.

This month, Google broke an equally impressive Internet traffic record — gaining more than 1% of all Internet traffic share since January. If Google were an ISP, as of this month it would rank as the second largest carrier on the planet.

Only one global tier1 provider still carries more traffic than Google (and this ISP also provides a large portion of Google’s transit).

In the graph below, I show a weighted average percentage of Internet traffic contributed by the search / mobile OS / video / cloud giant. As in earlier posts, the Google data comes from 110+ ISPs around the world participating in ATLAS. The multiple shaded colors represent different Google ASN and reflect ongoing global traffic engineering strategies.


Google now represents an average 6.4% of all Internet traffic around the world. This number grows even larger (to as much as 8-12%) if I include estimates of traffic offloaded by the increasingly common Google Global Cache (GGC) deployments and error in our data due to the extremely high degree of Google edge peering with consumer networks. Keep in mind that these numbers represent increased market share — Google is growing considerably faster than overall Internet volumes which are already increasing 40-45% each year. More data on general Internet growth trends is available in some of our earlier papers and blog posts.

While its not news that Google is Big, what is amazing is how much bigger Google continues to get.

A quick analysis of the data also shows Google now has direct peering (i.e. not transit) with more than 70% of all providers around the world (an increase of 5-10% from last year). In fact, the only remaining major group of ISPs without direct Google peering are several of the tier1s and national PTTs — many of whom will not settlement-free peer with Google due to regulatory prohibitions or commercial strategy.

While the business press may debate Google’s future (i.e. can it expand beyond search and continue its earnings growth?), for now Google’s traffic growth continues apace with massive corresponding impact on the network topology, peering arrangements and the overall Internet infrastructure.

– Craig

How Big is Google?

By: Craig Labovitz -

Google’s recent FTTH announcement generated a wave of media coverage and industry discussion. Responses ranged from exuberant local communities racing to sign up to anti-competitive howls from incumbent carriers.

Industry pundits wondered what is Google up to? What will the search giant do with 1Gbps to the home? And more ominously, is Google getting too big?

While this blog post won’t explore the politics / strategy behind Google’s FTTH initiative (except to suggest Google should choose Ann Arbor), we will share some data on Google’s relative size and growth from a global Internet perspective.

Google is big.

And by “big”, I mean really big. If Google were an ISP, it would be the fastest growing and third largest global carrier. Only two other providers (both of whom carry significant volumes of Google transit) contribute more inter-domain traffic. But unlike most global carriers (i.e. the “tier1s”), Google’s backbone does not deliver traffic on behalf of millions of subscribers nor thousands of regional networks and large enterprises. Google’s infrastructure supports, well, only Google.

Based on anonymous data from 110 ISPs around the world, we estimate Google contributes somewhere between 6-10% of all Internet traffic globally as of the of summer of 2009.

The below graph shows the weighted average percentage of all Internet traffic contributed by Google ASNs between June 2007 and July 2009. Most of Google’s rapid growth comes after the acquisition of YouTube in 2007.

Google's Contribution to Global Internet Traffic

Before getting much further, a few words about what we’re measuring. Traffic volumes provide only the most indirect measure of a network’s size or popularity (for example, it takes tens of thousands of Tweets to match the bandwidth of a single HD video). Our anonymous data also does not include internal provider services (e.g. IPTV or VPN) nor data served from co-located caches within provider data centers. Rather, we’re measuring inter-domain traffic, i.e. the traffic between providers (the “inter” in “Internet”).

With all of the above said, inter-domain traffic volumes provide a key metric for understanding Internet topology and the evolution of Internet traffic patterns.

But even traffic volumes tell only part of the story.

The competition between Google, Microsoft, Yahoo and other large content players has long since moved beyond just who has the better videos or search. The competition for Internet dominance is now as much about infrastructure — raw data center computing power and about how efficiently (i.e. quickly and cheaply) you can deliver content to the consumer.

And here again, Google is at the head of the pack.

In 2007, Google used transit providers for the majority of their Internet traffic (including Level(3)). But over the last three years, Google both built out their global data center and content distribution capability as well as aggressively pursued direct interconnection with most consumer networks.

The graph below shows an estimate of the average percentage of Google traffic per month using direct interconnection (i.e. not using a transit provider). As before, this estimate is based on anonymous statistics from 110 providers. In 2007, Google required transit for the majority of their traffic. Today, most Google traffic (more than 60%) flows directly between Google and consumer networks.


But even building out millions of square feet of global data center space, turning up hundreds of peering sessions and co-locating at more than 60 public exchanges is not the end of the story.

Over the last year, Google deployed large numbers of Google Global Cache (GGC) servers within consumer networks around the world. Anecdotal discussions with providers, suggests more than half of all large consumer networks in North America and Europe now have a rack or more of GGC servers.

So, after billions of dollars of data center construction, acquisitions, and creation of a global backbone to deliver content to consumer networks, what’s next for Google?

Well, I’m hoping for delivery of content directly to the consumer via a nice, fat 1 Gbps FTTH pipe.

Google, please choose Ann Arbor.

Who Put the IPv6 in my Internet?

By: Craig Labovitz -

About this time last year, we released a study on the state of IPv6 deployment in the Internet. Our August 2008 paper found diminishingly small traces of IPv6 — less than one hundredth of 1% of Internet traffic.

This year?

In a dramatic reversal of long-term IPv6 stagnation, global IPv6 traffic globally grew more than 1,400% in the last 12 months. Even more remarkable, this growth is due primarily to one application and one ISP.

We’ll explain in a moment, but first some background: Both our 2008 work and this IPv6 study used traffic statistics from 110 ISPs participating in the Internet Observatory. Though the Observatory is capable of collecting native (i.e. not tunneled) IPv6 traffic statistics, only six ISPs out of the 110 in the study currently have routers and collection infrastructure with native IPv6 enabled. As a result, our data generally includes only IPv6 traffic through Teredo and 6to4 tunnels. Further, since only a handful of Observatory participants use monitoring infrastructure with payload visibility, our study sees only the UDP Teredo control traffic (i.e. not the data portion).

The above technical limitations and our somewhat dismal 2008 assessment of IPv6 deployment engendered a bit of criticism. The main critique (such as this posting) seems to be that we significantly under counted IPv6. In particular, many pointed to the Amsterdam Internet Exchange (AMS-IX) switch statistics which show a Gigabit or more of IPv6 traffic (far more than we found in our study). Others pointed to the high rate of IPv6 address allocations as evidence of broader IPv6 deployment.

From the perspective of a year later, we stand by our 2008 IPv6 findings. A July 2009 news server outage confirmed suspicions that AMS-IX IPv6 traffic mainly consisted of file sharing through the free AMS-IX based IPv6 news servers. And a PAM paper earlier this year found both minuscule levels of IPv6 traffic in a tier1 network and confirmed that registry allocations provide a poor indicator of IPv6 usage. As a side note, the PAM paper also found that the small amount of tier1 IPv6 traffic consisted mainly of DNS and ICMP (i.e. test traffic and not real IPv6 usage).

So in August of 2008 real IPv6 Internet traffic was mostly non-existent.

And then things changed…
The above graph shows IPv6 traffic (Teredo and 6to4) as a normalized weighted average percentage of all Internet traffic between July 2007 and July 2009. In July of 2007, IPv6 represented less than 0.002% of Internet traffic. Beginning in August of 2008, tunneled IPv6 traffic begin to grow dramatically followed by an abrupt and even larger jump in April of 2009 (the E. Karpilovsky et al. PAM paper also observed this first 2008 jump in traffic but did not speculate as to the causality).

What happened?

This stark August 19, 2008 warning to the NANOG mailing list by Nathan Ward provides a strong clue:

Sit up and pay attention, even if you don't now run IPv6, or even if you don't ever intend to run IPv6.
Your off-net bandwidth is going to increase, unless you put some relays in.
As a friend of mine just said to me: "Welcome to your v6-enabled transit network, whether you like it or not ;-)".
uTorrent 1.8 is out, as of Aug 9.

Nathan was mostly right. While uTorrent never generated the expected flood of new traffic (at least by IPv4 standards), the introduction of IPv6 P2P succeeded where most previous IPv6 inducement efforts had failed (i.e. liberal peering, high quality IPv6 porn, IPv6 ASCII animation of Star Wars, etc.). In the space of ten months uTorrent helped drive IPv6 traffic from .002% to .03% of all Internet traffic (a dramatic 15x jump).

But the more interesting (and from an infrastructure perspective, far more important) IPv6 traffic increase came on April 21, 2009 with Hurricane Electric’s turn up of a global anycast’ed Teredo relay service. Hurricane Electric enabled 14 Teredo relays in Seattle, Fremont, Los Angeles, Chicago, Dallas, Toronto, New York, Ashburn, Miami, London, Paris, Amsterdam, Frankfurt and Hong Kong.

More details of Hurricane Electric’s infrastructure is available in this May 2009 LACNIC presentation.

Historically, IPv6 connectivity across the Internet has been, well, abysmal. Inefficient routing, multiple IPv6 tunnel encapsulations and overall lack of coordination between Teredo and 6to4 relay providers added latency, loss and played havoc with jitter (i.e. mangling VoIP). Frequently, a traceroute between two providers at the same exchange could traverse multiple countries or continents en route. For added background, see this 2009 Google IPv6 Conference presentation, this 2008 RIPE study and related 2007 study.

By all accounts, Hurricane Electric’s Teredo service significantly improved the IPv6 goodput for the average Internet end user over night. In particular, Microsoft Windows users got a big boost. Though Windows has shipped with a Teredo client (on by default) since XP, Microsoft never provided a public relay service. teredo.ipv6.microsoft.com now uses Hurricane’s 6to4 relays. And the dramatic improvement in Teredo and 6to4 relays seems to have lead to a corresponding jump in IPv6 traffic.

This is good news.

Finally, in the below graph, you can see the impact of both uTorrent and Hurricane’s relay deployment by region. We again show IPv6 tunneled traffic as a weighted normalized percentage of all Internet traffic. The most important take away is that the IPv6 growth after August 2008 is a global phenomena (with Asia at the forefront follow by Europe).
We look forward to revisiting IPv6 traffic in another year as relays improve, meaningful IPv6 content becomes available and more providers offer native IPv6 service.

Editor’s Note: This blog is the fourth in a series of weekly posts leading up to the publication of the joint University of Michigan, Merit Network and Arbor Networks “2009 Internet Observatory Report”. The full technical report goes into detail on the evolving Internet topology, commercial ecosystem and traffic patterns — available this October. Next week: “How Big is Google?”


Reblog this post [with Zemanta]

The Internet After Dark (Part 2)

By: Craig Labovitz -

This blog completes our informal three week study of Internet daily traffic patterns. Using data from the Internet Observatory, we analyzed weekday application traffic across 110 geographically diverse ISPs, including some of the largest carriers in North American and Europe. We believe this report (and upcoming paper) represent the largest study of Internet traffic temporal characteristics to date.

In the first half of this post, we showed unlike European Internet traffic which peaks in the early evening and then drops off until the next day’s business hours, US Internet traffic reaches its peak at 11pm EDT and then stays relatively high until 3am in the morning.

The question is what are Internet users doing after dark?

The answer: long after Exchange and Oracle business traffic slows to a crawl, Internet users turn to the web to surf, watch videos, send IM’s and happily try to kill each other.

We illustrate these trends with graphs of four application categories below.


The top two graphs show the daily average traffic fluctuations of TCP / UDP ports related two popular online game multi-player platforms: World of Warcraft and Steam (which includes many popular first person shooter games like Half Life). The bottom two graphs show common video and instant messaging protocols. As in earlier analysis, we take the average of North American consumer / regional providers traffic over 10 weekdays in July. To make the graph more readable, we show traffic as a percentage of peak traffic levels. All times are EDT.

Some observations:

  • Gamers Come Out at Night: Unlike most Internet applications which peak midday or late afternoon, online game traffic grows by more than 60% after 2pm. Gaming prime time appears to be between 8pm and 11pm EDT weekday nights (corresponding to the traditional and now declining television prime time hours). By comparison, web traffic levels remain relatively constant through the late afternoon and peaks much earlier at 5pm.
  • A Guild that Plays Together Stays Together: Unlike other online game traffic, World of Warcraft’s Battlenet shows a distinct 30% jump exactly at 8pm EDT every evening. In-house WoW level 80 colleagues suggest 8pm is a common time for guilds to set out on quests. Also unlike other game traffic, WoW declines rapidly after 11pm every night. Again, we suspect WoW traffic patterns are related to the more large group, social nature of World of Warcraft.
  • Midnight Video: Of all Internet applications, streaming video protocols reach their traffic peak the latest around midnight EDT every evening. We do not have very good visibility into what Internet users are watching this late, but correlation with large content site traffic patterns (below) provides some clues.
  • Always in Touch: Beginning at 9am EDT at lasting though midnight, Internet users IM constantly. The IM graph above shows traffic reaches 80% of peak by 10am and stays above 80% until midnight (with a 5pm EDT peak — perhaps related to millions of users making dinner plans). Interestingly, email exhibits a very different pattern and plummets by more than 30% immediately after 5pm EDT.

As mentioned earlier, we do not have detailed visibility into what Internet users are watching at midnight but ASN level traffic analysis provides some hints. Predictably, traffic grows dramatically to consumer sites like Google’s YouTube and large CDN / video providers. Also not surprisingly, we see a large jump in traffic to colo / hosting companies with adult content such as a 40% jump to ISPrime (AS23393) between 10pm and 1am EDT. We will explore one of the fastest growing and largest nighttime sites, Carpathia Hosting (AS29748), in an upcoming blog.

Editor’s Note: This blog is the third in a series of weekly posts leading up to the publication of the joint University of Michigan, Merit Network and Arbor Networks “2009 Internet Observatory Report”. The full technical reports goes into detail on the evolving Internet topology, commercial ecosystem and traffic patterns — available this October. Next week: “Who Put the IPv6 in My Internet?”

Reblog this post [with Zemanta]

The Great GoogleLapse

By: Craig Labovitz -

Web sites go down. Circuits fail. Network engineers goof router configs. And few of these outages ever make the nightly news…

But if you happen to be Google and your content constitutes up to 5% of all Internet traffic, people notice.  Network engineers around the world frantically email traceroutes to mailing lists. IRC channels fill with speculation (“definitely was a DDoS attack”, “no, a worm”, “it was ISP xxx’s  fault!”). And end users Twitter (a lot).

So what does it look like when 5% of the Internet disappears on an otherwise uneventful Thursday morning? The below graph shows average traffic across 10 tier1/2 ISPs in North America from Google’s network (ASN 15169). Outage began roughly at 10:15am and lasted through 12:15pm EDT.

Looking at the data, most large transit providers appear to have been impacted (e.g., Level3, AT&T, etc.). Other providers (e.g. large consumer DSL / Cable) showed no drop in traffic from/to Google.

Looking at BGP (below snapshot is from Arbor’s Routeviews Servers) we see a lot of churn in Google’s BGP routes around the outage timeframe — one prefix I choose at random flaps across half a dozen providers before getting withdrawn.

In a recent official company blog post, Google blamed some combination of airplanes and BGP for the outage.

Reblog this post [with Zemanta]

ATLAS 2.0: Observing A Rapidly Changing Internet

By: Danny McPherson -

It’s already been over 2 years ago since we first introduced our Active Threat Level Analysis System – ATLAS, a multiphase project that’s been evolving pretty much constantly ever since.  The first phase of ATLAS focused on capturing data via a globally scoped network of sensors running a number of data capture and analysis tools that would interact with attackers to discover what activities they are attempting, capture full payloads and classify them, and characterize scan and backscatter traffic.   This information was then correlated with a number of other ATLAS system data sources, and wrapped in the ATLAS portal, a public resource that delivers a sub-set of the intelligence derived from the ATLAS sensor network on host/port scanning activity, zero-day exploits and worm propagation, security events, vulnerability disclosures and dynamic botnet and phishing infrastructures.  It includes:

  • Global Threat Map: Real-time visibility into globally propagating threats
  • Threat Briefs: Summarizing the most significant security events that have taken place over the past 24 hours
  • Top Threat Sources: Multi-dimensional visualization of originating attack activity
  • Threat Index: Summarizing Internet malicious activity by offering detailed threat ratings
  • Top Internet Attacks: 24-hour snapshot of the most prevalent exploits being used to launch attacks globally
  • Vulnerability Risk Index: Determines the most dangerous vulnerabilities being exploited on the Internet today

Today we announced ATLAS 2.0, the next generation of ATLAS.  Many of you that follow the ASERT blog, employ our Active Threat Feed (ATF), or work with Arbor and our ASERT team on operational security issues, have seen bits and pieces of ATLAS 2.0 for quite a while now.  In a nutshell, what we’ve done with ATLAS 2.0 is expand well beyond the initial ATLAS capabilities, incorporating new intelligence information, to include:

  • collaboration with over 100 ISPs across 17 countries
  • expanded Fingerprint Sharing Alliance participation
  • real-time ‘coarse’ Internet traffic levels, protocols, and applications
  • topologically diverse global view of Internet routing system security, stability and intelligence
  • topologically diverse DNS system inputs and analysis (e.g., to identify fast flux and other DNS-related threats)
  • attack traffic data flows and trajectory information
ATLAS Fast Flux Bots

ATLAS DNS Fast Flux Bots

It’s all about visibility and baselining up and down the IP protocol stack, operating at each of the various layers, the more information we collect and model, from more globally distributed and diverse locations, in particular with the ever-increasing array of topologically scoped threats, the more likely we are to detect deviations from what’s normal or acceptable.  Once those deviations are detected, they can be analyzed to determine whether they’re legitimate or malicious.  Whether they’re Internet control plane routing system stability [1], global Internet traffic levels [2, 3, 4], exploit activity for a given vulnerability [5], DNS flux activity [6], or botnet command and control log and execution activities [7], establishing broad visibility and understanding what constitutes normal activity enables network operators and engineers to respond most effectively in their operating environment.

We hope that the additional intelligence gained through ATLAS 2.0 will permit Arbor to continue to provide a valuable public resource, and enable Arbor customers and non-customers alike to better prepare for the rapidly evolving global Internet threat landscape.

Reblog this post [with Zemanta]

The Great Obama Traffic Flood

By: Craig Labovitz -

Streaming video traffic coverage of Obama’s inauguration flooded North American backbones today. Traffic increases varied wildly across US providers with some seeing an overall 5% increase in backbone traffic and others jumping more than 40%.

This multi terabit per second flood represents one of the single largest one day spikes in Internet traffic since ATLAS Internet Observatory monitoring began five years ago. Apparently, US presidents are more popular than pro golfers — the inauguration traffic handily beat the last Internet traffic record set during the US Open.

While most of the US infrastructure appears to have withstood the flood, at least two ISPs showed clear failures and traffic drops during the traffic peaks coincident with Obama’s swearing in (traffic levels started to drop quickly beginning with the subsequent poetry readings).

Chiefly the traffic surge centered on Flash (TCP port 1935) and UDP port 8247 (which includes CNN streaming). In the US, most of these increases focused on consumer (DSL / MSO) providers and transit ISPs (especially those interconnecting large CDNs). Flash traffic spiked by more than 60% in most providers and by 400% in a few of the larger cable operators.

The below graph shows both of these ports across 10 of the largest US ISPs participating in Arbor’s ATLAS Internet Observatory traffic sharing initiative (see NANOG presentation for more details).

Great Obama Traffic Flood

While US backbones saw a large inauguration traffic spike, Europeans and Asian viewers appeared less interested in US politics with an under 1% increase in backbone traffic (in fairness, timezone differences also likely had a significant impact). Our Canadian neighbors proved more interested with a 2-5% growth in backbone traffic today.

Though multiple content providers hosted the traffic streams today, Limelight (AS 22822) was one of the clear winners — ATLAS data across the ten US consumers ISPs show a massive increase in AS22822 traffic (median of 160%). Akamai showed a more modest increase of 17%.

The Obama inauguration marks a historic day in US politics and a remarkable day for the popularity of Internet streaming video. We look forward to watching more great things to come.

(Co-authored with Scott Iekel-Johnson)

Reblog this post [with Zemanta]