Tag Archives : “End of Internet” March 2010

How Big is Google?

By: Craig Labovitz -

Google’s recent FTTH announcement generated a wave of media coverage and industry discussion. Responses ranged from exuberant local communities racing to sign up to anti-competitive howls from incumbent carriers.

Industry pundits wondered what is Google up to? What will the search giant do with 1Gbps to the home? And more ominously, is Google getting too big?

While this blog post won’t explore the politics / strategy behind Google’s FTTH initiative (except to suggest Google should choose Ann Arbor), we will share some data on Google’s relative size and growth from a global Internet perspective.

Google is big.

And by “big”, I mean really big. If Google were an ISP, it would be the fastest growing and third largest global carrier. Only two other providers (both of whom carry significant volumes of Google transit) contribute more inter-domain traffic. But unlike most global carriers (i.e. the “tier1s”), Google’s backbone does not deliver traffic on behalf of millions of subscribers nor thousands of regional networks and large enterprises. Google’s infrastructure supports, well, only Google.

Based on anonymous data from 110 ISPs around the world, we estimate Google contributes somewhere between 6-10% of all Internet traffic globally as of the of summer of 2009.

The below graph shows the weighted average percentage of all Internet traffic contributed by Google ASNs between June 2007 and July 2009. Most of Google’s rapid growth comes after the acquisition of YouTube in 2007.


Google's Contribution to Global Internet Traffic

Before getting much further, a few words about what we’re measuring. Traffic volumes provide only the most indirect measure of a network’s size or popularity (for example, it takes tens of thousands of Tweets to match the bandwidth of a single HD video). Our anonymous data also does not include internal provider services (e.g. IPTV or VPN) nor data served from co-located caches within provider data centers. Rather, we’re measuring inter-domain traffic, i.e. the traffic between providers (the “inter” in “Internet”).

With all of the above said, inter-domain traffic volumes provide a key metric for understanding Internet topology and the evolution of Internet traffic patterns.

But even traffic volumes tell only part of the story.

The competition between Google, Microsoft, Yahoo and other large content players has long since moved beyond just who has the better videos or search. The competition for Internet dominance is now as much about infrastructure — raw data center computing power and about how efficiently (i.e. quickly and cheaply) you can deliver content to the consumer.

And here again, Google is at the head of the pack.

In 2007, Google used transit providers for the majority of their Internet traffic (including Level(3)). But over the last three years, Google both built out their global data center and content distribution capability as well as aggressively pursued direct interconnection with most consumer networks.

The graph below shows an estimate of the average percentage of Google traffic per month using direct interconnection (i.e. not using a transit provider). As before, this estimate is based on anonymous statistics from 110 providers. In 2007, Google required transit for the majority of their traffic. Today, most Google traffic (more than 60%) flows directly between Google and consumer networks.


google_peering

But even building out millions of square feet of global data center space, turning up hundreds of peering sessions and co-locating at more than 60 public exchanges is not the end of the story.

Over the last year, Google deployed large numbers of Google Global Cache (GGC) servers within consumer networks around the world. Anecdotal discussions with providers, suggests more than half of all large consumer networks in North America and Europe now have a rack or more of GGC servers.

So, after billions of dollars of data center construction, acquisitions, and creation of a global backbone to deliver content to consumer networks, what’s next for Google?

Well, I’m hoping for delivery of content directly to the consumer via a nice, fat 1 Gbps FTTH pipe.

Google, please choose Ann Arbor.

The End is Near, but is IPv6?

By: Craig Labovitz -

As of this blog posting, exactly 900 days remain until the end of the Internet, or at least the exhaustion of IPv4 registry allocations. And you don’t have to take my word for it, even the normally staid London Times and Fox News proclaimed, “Internet meltdown… The world is heading for a digital doomsday”.

Heady stuff.

Of course, IPv6 (or the new IPv4) was supposed to take care of all of this — billions of billions of new IP addresses, hardened security built in from the start, and an elegant new architecture to replace all of IPv4′s hacks.

So what happened to IPv6?

Well, it has been a strange, long year…

The year began with fears over the “end of the Internet” (due to lack of IPv6 adoption) and ends this month with renewed IPv6 enthusiasm centered around the Olympics and a successful US government IPv6 mandate. In between these two extremes of IPv6 despair and enthusiasm, IPv6 generated a surge of news coverage (see graph below). At its peak this past June, print media around the world published nearly 3,000 articles a month on IPv6 (almost twice as much as the comparatively uninteresting IPv4).

Much of the recent coverage has focused on the summer Olympics this week. Chinese organizers have planned the summer Olympics game as a showcase for IPv6 technology. From a recent article, “… IPv6 will herald the arrival of China as a major center for technological and scientific advancement in a way that will overshadow its own unbeatable record as a world leader…”. Through China’s Next Generation Internet (CNGI) initiative, China has reportedly spent several billion dollars making sure they got a national IPv6 backbone right.

In the US, the recent government deadline for IPv6 compliance also generated a flurry of IPv6 activity: All major vendors publicly declared their IPv6 readiness. Popular press and industry magazines filed thousands of stories on IPv6. US Federal Departments officially declared success and the Internet IPv6-ready this past June 30th.

So has imminent collapse of the Internet has been avoided?

Is the Internet moving full steam ahead towards IPv6?

Maybe.

The truth is that as an industry we don’t have a good measure on the relative adoption success of IPv6 with respect to Internet traffic.

No idea really.

We do have some anecdotal measurements on IPv6 registry allocation and BGP announcements. But, very little data on actual IPv6 usage.

As our small effort to fill this gap, we spent much of the last year looking for IPv6 traffic in the Internet. In cooperation with the University of Michigan and close to 100 Internet providers, we leveraged commercial traffic probes across the world to measure inter-domain IPv6 traffic in the Internet. We believe this is the largest study of IPv6 and Internet traffic in general to date (by several orders of magnitude).

Our dataset covered 87 ISPs including one quarter of the tier1 ISPs and a sizable percentage of the regional / PTT providers in North America and EMEA. In all, we monitored 2,389 peering and backbone routers, 278,268 customer and peering interfaces, and an aggregate 15 exabytes of Internet inter-domain traffic at an average daily rate of 4 terabits per second (we spoke about some of this measurement infrastructure at a recent NANOG). We believe this gave us a pretty good view of overall IPv6 traffic trends in the Internet.

You can view the full technical report at http://www.arbornetworks.com/IPv6research.

What did we find?

Not much. In fact, less than not much — very, very little.

The below shows the percentage of IPv6, both native and tunneled, as a percentage of all Internet traffic. At its peak, IPv6 represented less than one hundredth of 1% of Internet traffic. This is somewhat equivalent to the allowed parts of contaminants in drinking water (my household water comes from the Detroit river).

IPv6 Traffic Graphed as Percentage of IPv4

Now the above graph may not be completely fair since many of the ISPs do not have infrastructure to monitor native IPv6 (more about this later). But our numbers seem to agree with data from a variety of other sources on IPv6 adoption rates.

Some related IPv6 statistics:

Percentage of ASN with IPv6 BGP announcements 3%
Percentage of Internet2 sites with passing IPv6 grade 1%
Percentage of Alexa Top 500 websites using IPv6 0.4%
IPv6 DNS queries as percentage of IPv4 DNS load 0.2%
IPv6 as a percentage of all Internet traffic 0.002%

We are not the first to raise concern over the small amount of IPv6 traffic (see Geoff’s slides last month) — just the first to have Internet wide IPv6 traffic usage measurements.

And the lack of IPv6 traffic is not for lack of trying. Many organizations and individuals offer a range of lures to encourage IPv6 adoption. For example, the next generation research and education backbone in the US, Internet2, offers free transit for IPv6 traffic. And unlike IPv4, many large ISPs have very liberal IPv6 peering policies.

The single greatest lure? For ISPs or large multi-homed enterprises struggling to justify just one more tiny, little IPv4 /16 allocation, the minimum IPv6 allocation is /32 or a staggering 2^64 larger than the entire IPv4 Internet today.

On the less pragmatic side, other IPv6 proponents offer free high quality IPv6 porn. Others yet provide ASCII animation of Star Wars movies (IPv4 users get only black & white — make sure you watch the “Return of the Jedi” version). And, of course, the dancing turtle. Several web sites provide more listings of exotic IPv6 only Internet content.

But none of these efforts have been enough to generate any significant IPv6 traffic.

So, why so little IPv6 traffic?

Well, the biggest issue is money. Specifically, the department of commerce estimates it will cost $25 billion for ISPs to upgrade to native IPv6.

And this massive expense comes without the lure of additional revenue since IPv6 offers diminishingly few incentives nor new competitive features to attract or upsell customers. In many ways, IPv6 in the United States is much like the high definition television federal mandate (but without the mandate or the really crisp looking football games).

The harsh logic of the Metcalfe Effect also applies. With so few web sites and end users currently supporting IPv6, the incremental value to any single new IPv6 end site is limited. For many end users, v6 is an IP version all dressed up but with nowhere to go.

The third major issue is technical. While most vendors passed the OMB IPv6 requirements, it kind of depends on what you mean by “IPv6″ and “requirement”.

For example, some backbone routers “support” IPv6 forwarding, but not in hardware (at least not at line rate). Or IPv6 “support” does not include ACLs nor other critical security features. An ICANN survey of security vendors found that less than one in three commercial firewalls support IPv6.

Maybe you want an IPv6 MPLS VPN backbone? Sorry, not supported.

And even if your router supports IPv6, you might not be able to test or monitor it. Few vendors offer complete IPv6 SNMP / MIB support and even fewer support IPv6 Flow export (in fairness, V9 flow support is included on many Cisco cards today and Juniper has announced IPFIX support sometime in the next year). We blogged about many of these deployment issue earlier this year and Randy gave a presentation on the topic at a recent NANOG. The CAIDA and ARIN IPv6 Survey also has a nice summary on market / business forces limiting ISP IPv6 adoption.

Perhaps the biggest problem is that IPv4 works. And works well.

While IPv4 addresses are still relatively plentiful and cheap, ISPs and end customers have few incentives to migrate to IPv6. Some recent research even suggests IPv4 addresses may be more plentiful than previously believed. This tech report found that less than 4% of allocated addresses are actually occupied by visible end hosts. The authors concluded that most Internet space is likely, in fact, unused (though allocated).

All of this lack of IPv6 adoption has lead to quite a bit of hand wringing in the ISP technical community. While not declaring IPv6 a failure, discussions do wander into questions about “premature extinction of IPv6″ or whether “IPv6 is an exercise in futility”.

Imminent Collapse

Predicting the imminent collapses of the Internet has a long and storied history over the last twenty years. But despite all of these predictions, the Internet has survived. Sure we crashed a few routers, announced some bogus routes, and dropped a few bits here and there. But the Internet survived — even grew a bit and gained some new users. I saw Bob Metcalfe eat a tie.

And the Internet will undoubtedly change and evolve past the impending IPv4 exhaustion.

But how?

Well, the questions is more about market forces than technology. IPv4 address allocations already have a minimal cost ($18,000 for ARIN large allocation). And growing registry management justification requirements and shrinking allocation size have steadily increased the overall cost of address space to ISPs over the last ten years. During the heady Internet technology bubble days, several companies made acquisitions in significant part based on the valuation of large legacy IP allocations.

Many think the price of IPv4 and scarcity will lead to open or if not sanctioned, black markets for address space. And debates continue whether an open market for IPv4 would be good or bad thing for Internet policy. Personally, I think an IPv4 market is inevitable.

The Future of IPv6

It is now clear the original optimistic IPv6 deployment plans have failed.

While the end of the Internet is not near, neither is IPv6. At the current rate of adoption, we are a decade or more away from pervasive adoption of dual stack support for IPv6. As Alain correctly notes in a recent IETF draft, “The IANA free pool of IPv4 addresses will be depleted soon, way before any significant IPv6 deployment will have occurred”.

So IPv6 adoption will take far longer and will look far different than most of us expected back in 1994 when the IAB announced the selection of IPv6. Clearly things need to change, including IETF and vendor exploration of other technologies to facilitate IPv6 adoption such as better NAT interoperability or lighter weight dual stack.

Number of IPv6 News Articles per Month

Still, despite some of the rather anemic IPv6 traffic statistics above, IPv6 is growing. The graph above shows the number of print media articles per month mentioning IPv6 and IPv4 in the first 30 words (source MetaNews). Note that IPv6 is running almost two to one against IPv4. If judged purely by public interest, IPv6 is a winning (by comparison, DNSSEC averages only 50 articles per month and barely peaked at 150 during the DNS crisis. BGPSEC fared even worse).

The below graph shows the aggregate average daily IPv6 (tunneled and native) traffic across 87 ISPs over the last year. Since July 2007, IPv6 traffic has grown by nearly a factor of 5 to an average of 100 Mbps per day. BGP tables show an even larger proportional growth. Though not a landslide of adoption, it is still something.

While it is easy to poke fun at predictions of the “Imminent Collapse of the Internet”, the eventual exhaustion of IPv4 allocations is real. We need to do something. And IPv6 is our best bet.

Aggregate IPv6 Traffic Internet Wide

So, I’ll end with my top four predictions on IPv6 growth:

  1. Islands are beautiful too.
    IPv6 may succeed in the same way multicast failed. And by multicast failing, I really mean multicast succeeded. Though multicast never evolved a business model to justify its originally envisioned Internet wide inter-domain deployment, multicast has been astonishingly successful within the enterprise and ISP service infrastructure. Almost all Fortune 500 enterprises use multicast in some form to broadcast company video or update applications. Similarly, multicast is at the core of most ISP IPTV infrastructure.Like multicast, we are seeing rapid adoption of IPv6 within consumer, IPTV and 3G / 4G mobile providers for management of their internal infrastructure. You can pick your favorite market driver: 3G / 4G Mobile IP, Digital Radio, RFID, Control Networks, etc. But the numbers for globally unique end devices is staggering no matter which trend you choose.

    For example, Comcast has migrated all of their internal control network to IPv6 with plans to manage 100 million cable modems.

    Various estimates place number of network devices that will need to be managed at 12 billion by 2012. Note that these devices may not need global visibility, but providers will need to at least internally provision and manage (and RFC1918 space is not a reasonable option).

  2. Market trumps technology. And politics trumps markets.
    The future of the Internet is not fixed line devices. Nor is it users in the United States.The future of the Internet is likely both mobile devices and emerging Internet countries like China which reportedly surpassed the number of US web users at 253 million last month.While politics and government mandates do not always drive technology (see GOSIP or metric system in the United States), sometimes they do (see metric system in United Kingdom).

    Throughout the world, government mandates are spurring IPv6 adoption. China’s CNGI initiative and billions in spending uses IPv6 as the centerpiece. Similarly, Japan, Korea all have major IPv6 initiatives. The EU called for mass migration to IPv6 by 2010.

    The important bit to realize about governmental focus on IPv6 is that it is not about technology nor is it even really about IPv6. Many governments view IPv6 through the prism of politics. These countries, rightly or wrongly, view ICANN and the large US centric legacy IPv4 allocations as instruments of US control. For China, Japan and many EU nations, IPv6 is really about no less than who will control the future of the Internet.

  3. IPv6 has already succeed.
    You can now get native IPv6 from many providers, including Verizon, Tata (formerly VSNL/Teleglobe), Global Crossing and others. Over half of surveyed providers say they have plans to roll out commercial IPv6 offerings in the next year. As more vendors integrate IPv6 into their products lines, the ISP IPv6 tax has correspondingly dropped. For many, IPv6 comes with latest refresh of hardware which ISPs generally amortize over 5-8 year periods. While it will be many years before the millions of embedded end devices support IPv6, your backbone routers likely already do.Most encouraging of all, there is finally the beginning of real IPv6 content. Or at least you can now use IPv6 to search content (as long as the indexed content is IPv4). At the Philadelphia IETF this year, Google announced support for production quality IPv6 search at ipv6.google.com.
  4.  

  5. And the final reason IPv6 will succeed?
    No one has the stomach to spend another twenty years replacing IPv6 with something else.

Personally though, I just configured our local router for IPv6 so I can watch Michael Phelps (former University of Michigan athlete) win eight golds this week at http://2001:252:0:1::2008:6.

Full disclosure — I worked on the failed TUBA counter-proposal to IPv6 and still harbor a grudge.