The MegaUpload Shutdown Effect

By: Jose -

The popular file sharing site MegaUpload was shut down by the US FBI and Department of Justice on Thursday, January 19, and executives from the company were taken into custody. This story is very well covered by the Wall Street Journal and includes a copy of the indictment for your reading.

As you would expect, this was a wildly popular site with users from all over the world. So much so that even notable celebrities appear in a video discussing MegaUpload, almost endorsing it. Previous work by Arbor Networks showed that content providers and hosting sites like MegaUpload are the new “Hyper Giants”. With enough global data, you can actually see the traffic drop when the shutdown occurs. Based strictly on the traffic rates it appears that the shutdown started just after 19:00 GMT on January 19, with traffic plummeting down over the next two hours. The graphic here shows three main client regions – Asia-Pacific, Europe, and the US.

Over the past 24 hours, the top countries (in aggregate) using MegaUpload were the United States, France, Germany, Brazil, Great Britain, Turkey, Italy, and Spain, although dozens more countries are represented.

As for the traffic drop off, we’re not the only ones to notice. As seen on Twitter, South America experienced a dramatic traffic drop at about the same time, presumably due to this MegaUpload shutdown. Furthermore, we’re seeing reports of a fake MegaUpload site that is supposedly a malware infection site.

Friends of mine from elsewhere in the world have been joking that the Internet seems to be running a bit smoother today. That may be, given how much bandwidth appears to have been freed up.

MegaUpload

Torrent Sites and The Pirate Bay: DDoS Afoot?

By: Jose -

Around the time of the convictions late this week of the folks behind The Pirate Bay (also see The Pirate Bay Trial: The Official Verdict – Guilty on TorrentFreak), a well known BitTorrent tracker distribution site, we started seeing reports of DDoS attacks on other torrent tracker sites. Never one to miss an opportunity to look for massive DDoS attacks against important sites, I went looking in our archives to see if this was indeed the case.

It’s interesting to note that TPB may indeed be a key to illicit torrents and pirated material (although if it’s shut down something will surely replace it). See P2P researchers fear BitTorrent meltdown from the Iconoclast website. According to the piece, TPB is so heavily connected with over half of the torrents tracked there disrupting it may have an impact on all BitTorrent traffic for a while:

Raynor told TorrentFreak that if The Pirate Bay goes down, many of the other trackers might collapse as well. “If The Pirate Bay goes down the load will automatically shift to others. This is because most of the Pirate Bay swarms also include other trackers. When Pirate Bay goes down it would overload others until they fall also. Meaning even more stress and further casualties. This is likely to end in a BitTorrent meltdown.”

This makes sense that someone, perhaps a vigilante or just a “griefer”, is hoping to shut down pirate activities by going after major torrent tracking sites. This bears some analysis.

All in all, except for free-torrents.org getting attacked by a Black Energy botnet run out of China (using the C&C at hack-off.ru), we can’t corroborate this spate of attacks. Free-torrents.org has been getting pounded by this botnet since mid March, 2009, in fact. But none of the other major sites appear to be receiving such packet love.

As such, while this may be happening we’re not seeing it hit the massive scale that we would expect.

Conficker Did Not Melt the Internet

By: Jose -

But it is busy.

Last week’s April 1 trigger date for the new routines in Conficker.C/D (depending on the vendor) was mis-reported by some press agencies as the date many in the CWG said the Internet would melt down. Not quite. The press has been busy with the story as a hype leading to fizzle angle etc. Sure enough, the Internet kept on trucking. Here’s a view from one BGP peer during that time, no big change in traffic from the days prior:

internet_1wk_no_conficker_effect.png

Some folks even said Conficker was sleeping.

Not quite. It’s not DDoSing, but it is apparently doing it’s thing, or “Confickering” as a friend said. It’s dropped a new E variant over P2P. Some reports are seeing it talk to a Waledac domain but not everyone is.

As for why a possible Waledac (aka Storm) connection, we do not know.

So, what happened? It looks like the DNS lockout has been working; I suspect the attackers also noticed the great de-peerings of bad ISPs that have been going on for a while and decided to avoid hosting a C&C in one of those and instead went the P2P route. For details on how the bots know how to talk to other bots over P2P see the LEXSI CERT blog.

We have no additional info to give outside of those above links. If you’re following this story this is a major development.

Other Conficker Stuff

IBM/ISS has interesting stats on Conficker infections based on their P2P models. Some in the press have said these numbers mean 4% of the Internet is infected with Conficker. Not quite. As Vint Cerf pointed out, it helps to know the denominator when you make those sorts of population claims. As I’ll point it, it also helps to know what you’re measuring. We know Conficker has a big population, but we also know our measurements are only a lower limit (based on IP visibility etc).

We’re seeing a rise in TCP/445 scanning. 3wks of ATLAS data for the top 50 scanners is shown below, a clear rise in this time:

445 tcp scans.png

We think this is related to new Conficker activity in this timeframe.

UPDATE April 10

Oh and Mafiaboy, Conficker was not a ruse.

iWorkServices == P2P iBotnet

By: Jose -

If you want iWork 09 and didn’t want to pay for it, you may have grabbed a pirated copy. That may not have been all you got. If you wanted your Mac to be a part of a P2P botnet, then you’re in luck!

It turns out the package you may have downloaded over BitTorrent, a massive 450MB ZIP installer, is really just a huge Trojan horse package that installs a simple P2P bot tool on your box. Running the installer will not install iLife but instead the official sounding “iWorkServices”. This is not what you think it is. The binary has these characteristics:

MD5 (iWorkServices) = 046af36454af538fa024fbdbaf582a49
SHA1(iWorkServices)= 55d754b95ab9b34bdd848300045c3e11caf67ecf
SHA(iWorkServices)= 6b83df2636a4813ef722f3fad7c65b5419044889
file size: 413568 bytes
iWorkServices: Mach-O universal binary with 2 architectures
iWorkServices (for architecture ppc):   Mach-O executable ppc
iWorkServices (for architecture i386):  Mach-O executable i386

When run as root it creats a couple of files and directories to get set up:

/System/Library/StartupItems/iWorkServices
/System/Library/StartupItems/iWorkServices/StartupParameters.plist
/usr/bin/iWorkServices

This will now run whenever your box boots. The installer makes sure that the script is runnable:

chmod 755 /System/Library/StartupItems/iWorkServices/iWorkServices

And the script just launches the binary:

#!/bin/sh
/usr/bin/iWorkServices &

Not very sophisticated. On startup it creates a “dot” directory under /tmp:

/tmp/.iWorkServices

It fires up some connections:

69.92.177.146:59201
qwfojzlk.freehostia.com:1024

It will keep on trying until it connects. It also grabs a list of seed P2P peers from the file itself by decrypting the running file (thwarting static analysis) and managing the known peers as you would expect. It generates a port to listen on as needed (although it’s not quite clear to me how it would handle being behind a NAT device).

The bot software itself appears to be a Kadima-related P2P protocol with the expected commands to manage the peer list, but also to provide a remote shell, download and run arbitrary code, and to give full access to the box:

socks
system
httpget
httpgeted
rand
sleep
banadd
banclear
p2plock
p2punlock
nodes
leafs
unknowns
p2pport
p2pmode
p2ppeer
p2ppeerport
p2ppeertype
clear
p2pihistsize
p2pihist
platform
script
sendlogs
uptime
shell
rshell

What’s more is that there is an embedded Lua interpreter, giving a very sophisticated command language some additional structure.

So, what’s this botnet been up to? DDoS it seems, via a downloaded and executed PHP script. Clever.

Looking to find if anyone else is monitoring this botnet …

Bear in mind that this is just like all of the other OS X malware: you have to willingly install it. It’s much more of a Trojan Horse than a virus or worm.

Related info:

Edited to fix the name of the product this Trojan package masqueraded as.

Fast Flux and New Domains for Storm

By: Jose -

At last week’s FIRST conference in Vancouver I presented on some of our ATLAS fast flux data. The slides aren’t yet available, but the ongoing reports in ATLAS have been reflected to continuously update some of the analysis we did. Some of the new reports include the lifetimes for each network, and the “distinct networks” section, which identifies related domains through shared botnet membership. ATLAS users can also get the updated blocklist of fast flux domains for use in stopping such attacks.

Just in time, too, the Storm Worm has begun using new fast flux domains. Messages look like this:

> Date: Sun, 29 Jun 2008 00:56:18 +0700
> From: hp_ejer@levelton.com
> Subject: You make my world special
-
> My heart belongs to you ht tp:/ /latinlovesite.com/

Here’s a list of all of the domains we’ve identified so far.

theloveparade.com        NS     ns5.lollypopycandy.com
latinlovesite.com        NS     ns5.lollypopycandy.com
youronlinelove.com       NS     ns5.lollypopycandy.com
yourloveletter.com       NS     ns5.lollypopycandy.com
makinglovedirect.com     NS     ns5.lollypopycandy.com
lollypopycandy.com       NS     ns5.lollypopycandy.com

Storm has changed its tactics constantly in the past year and a half, and this “love theme” is nothing new. We’ll see how long this theme lasts.

UPDATE 1 July 2008

Here’s a full list of domains:

superlovelyric.com      NS      ns.verynicebank.com
bestlovelyric.com       NS      ns.verynicebank.com
makingloveworld.com     NS      ns.verynicebank.com
wholoveguide.com        NS      ns.verynicebank.com
gonelovelife.com        NS      ns.verynicebank.com
loveisknowlege.com      NS      ns.verynicebank.com
lovekingonline.com      NS      ns.verynicebank.com
lovemarkonline.com      NS      ns.verynicebank.com
makingadore.com NS      ns.verynicebank.com
greatadore.com  NS      ns.verynicebank.com
loveoursite.com NS      ns.verynicebank.com
musiconelove.com        NS      ns.verynicebank.com
knowholove.com  NS      ns.verynicebank.com
whoisknowlove.com       NS      ns.verynicebank.com
theplaylove.com NS      ns.verynicebank.com
wantcherish.com NS      ns.verynicebank.com
verynicebank.com        NS      ns.verynicebank.com
shelovehimtoo.com       NS      ns.verynicebank.com
makeloveforever.com     NS      ns.verynicebank.com
wholovedirect.com       NS      ns.verynicebank.com
grupogaleria.cn NS      ns.verynicebank.com
activeware.cn   NS      ns.verynicebank.com
nationwide2u.cn NS      ns.verynicebank.com

A Case Study in Transparency

By: Kurt Dobbins -
One of the overriding themes in the Network Neutrality debate, and what triggered much of the recent activity with Comcast and the FCC, has to do with transparency.  Or in the recent words of FCC Chairman Kevin Martin, “Consumers must be completely informed about the exact nature of the service they are purchasing”.  When it comes to transparency about service plans, and the business necessity behind them, I can think of many good examples, but one service provider stands above the rest, an ISP in the UK called PlusNet. In the spirit of transparency, PlusNet is an Arbor customer.

PlusNet offers an array of residential broadband services, called “Broadband Your Way” shown in the following diagram, ranging from a “pay as you go” service for light users – casual users who are typically migrating from a dial service to always-on broadband – to high-end broadband subscribers that enjoy the heavy use of gaming, VoIP, and peer to peer file sharing. PlusNet also has a specific plan for gaming that offers quality broadband with low ping and latency, called “Broadband Your Way PRO.”

 

 

Each of the service options has some form of traffic management associated with it, so each plan can appeal to a different demographic: from a light user that does not use file sharing to a heavy user that wants file sharing and streaming 24×7. Rather than have a one-plan-fits-all service, PlusNet offers consumers a plan that fits their service and economic requirements.

What makes PlusNet really interesting is that they clearly explain each of the service options, and even go on to explain that there is no such thing as “unlimited” broadband bandwidth; i.e., the network needs to be managed during peak busy hours to ensure fairness and to deliver real-time and interactive applications with a good quality of experience. PlusNet employs three methods of ensuring fairness on their network during peak busy hours:

 

1) Traffic Management: For certain plans, maximum bandwidth rates for peer to peer file sharing and other file downloading services are managed during peak busy hours. Each service plan comes with a higher or lower degree of traffic management;

2) Prioritization: For all plans, interactive applications like Web and real-time applications such as VoIP are given priority over non real-time or “background” applications.

3) Changing Human Behavior: For all usage-based plans where there is a monthly Usage Allowance, subscribers are given an economic incentive to use the network during non-busy off-peak hours. Any usage during off-peak hours is considered “free” and does not count against the monthly allowance.

 

PlusNet fully discloses maximum downstream and upstream bandwidth rates for specific application types, such as peer to peer file sharing, as well as what applications are prioritized, for each service option. Because the need for some form of traffic management is driven by the ISP cost model, PlusNet also discloses how UK ISPs pay for bandwidth in order for their customers to understand the business drivers for employing traffic management techniques during peak hours as well as future plans for capacity planning and traffic management. The consumer continues to be informed about the services they are purchasing.

But explanation details in a service plan are not always enough. “Seeing is believing” as they say, so PlusNet even publishes network traffic graphs depicting how the network is used during peak and off-peak hours, clearly demonstrating  the benefits of their traffic management policies. Winning awards is also a nice way to demonstrate better service too!

 By prioritizing interactive applications like Web and Streaming, PlusNet ensures a great customer experience during peak busy hours, as shown in the graph below for the hours between 8pm and 9pm.

 

 

Conversely, by managing peer to peer file sharing during peak hours, in conjunction with encouraging consumers to do file sharing at night (off hours) when it is “free” and not counted against any monthly usage allowance, PlusNet is able to get better utilization of their network bandwidth by time-shifting the bandwidth used for file sharing into the off-peak hours when there is unused capacity on the network. The effects of this are dramatic, as shown in the following graph during the hours 4am to 5am, allowing PlusNet to keep its costs lower by deferring expensive upgrades to bandwidth capacity.

 

 So, regardless where the Network Neutrality debate ends up, one thing is certain: ISPs will be required to inform consumers about the exact nature of the service they are purchasing. ISPs can learn a valuable lesson in transparency by taking a closer look at the PlusNet model.

 

 

 

 

 
 
 

 

 

Ono and ISP Coziness

By: Danny McPherson -

Some of you may have seen the coverage that Ono picked up today because of it’s ability to optimize P2P transaction speeds by enabling more topologically optimal distribution – all while requiring no interaction with the ISP. On one hand, I’m happy about this, as the whole P4P thing, and the topology intelligence dependence doesn’t seem a viable long-term option. However, given where the bottlenecks are in the current system, Ono leaves some room for concern as well.

Specifically, in measurements we’ve seen the peak to trough bandwidth on the fixed broadband access edge in both cable and DSL is around 4x peak-trough (although the x value itself isn’t particularly relevant to this discussion). So, for example, if there were 1Gbps peak utilization, trough utilization would be around 250Mbps. Given that ISPs during the capacity planning process need to plan for peak loads, they’ll typically engineer capacity expansion with peak loads of 1Gbps, plus some variable that accommodates incremental bandwidth utilization increases based on historical growth, as well as projected new markets, subscriber acquisition, etc..

Needless to say, much of this peak load is driven by P2P and other protocols. So, when folks come up with solutions for improving P2P transfer rates, for example, by a professed 207% as with Ono, that 1 Gbps might now be 2.1 Gbps, and the peak-trough ratio may now be 6x or 8x, versus 4x. Arguably, this exacerbates the problem where it’s most obvious, in the access network, and in particular, in the cable access network where downstream bandwidth is shared among multiple subscribers. Now, given these peak burst rates, ensuring fairness among users of the network is even more critical.

Other applications have improved transactions rates considerably as well. For example, most browsers opening multiple connections to web servers in order to download multiple images and text in parallel, or iTunes and Google Maps opening tens of connections in order to obviate TCP’s inherent per-session (v. per-user) congestion control mechanisms in order to optimize aggregate transactions rates. When your single smtp (email) connection is contending for network resources with 300 TCP connections from your neighbors ‘optimized’ application, ensuring fairness among subscribers by the ISP is critical IF contention for resources exists, in particular those of access loop capacities.

The implications of this aren’t felt just on the network from an available bandwidth and packet forwarding quality of service perspective, but also, by devices like NATs and stateful firewalls that need to track all of these connections. Applications that arguably exploit TCP’s per-session Internet congestion-friendly characteristics in order to optimize the local user’s experience are becoming more and more common. More focus on fariness across users, as opposed to fairness across transport connections, is sure to be a critical issue in transport protocol design and network architecture in the coming months.

I believe that if Ono-like solutions enable topologically optimal distribution of content, that’s a good thing. However, there will always be a bottleneck, and ensuring it’s in the place that scales best and is most manageable is critical.

Vuze, TCP RSTs, and Educated Guesswork

By: Danny McPherson -

Triggered by this report (pdf) from Vuze, and Iljitsch’s ars technica article, my friend Eric Rescorla (ekr) posted on his Educated Guesswork blog this morning some bits regarding how many TCP transactions end in RSTs. I’m glad he did this (he saved me the work), as the variances in the data and the methodology employed have been frustrating me since the Vuze report was published. I’ve heard many ISPs taking issue with the report, and several doing so publicly (e.g., AT&T), and while all the appropriate disclaimers are provided by Vuze, a typical consumer might heavily weigh the results in this report when selecting ISPs, or presupposing which ISPs might employ blunt instrumentation in attempts to throttle P2P traffic.

I commend Vuze for attempting to add some actual empirical data points to the P2P throttling discussion, and for making both summarized raw data (ZIP) and their plug-in openly available. I firmly believe empirical evidence is a fine thing, assuming stated “facts” so represented by that evidence are verifiable, and specifically, the methodology used to collect that evidence is indeed measuring the Right Thing. This is where I take issue with the methodology employed to collect this empirical evidence, as do Eric and Iljitsch, and believe the “first results” in the report are misleading, at best.

Given that the objective of the Vuze plug-in, as stated in the report, was to “add relevant data to the traffic throttling debate, and encourage that decisions be made based on facts“, I trust they’ll be updating both their report and methodology to accommodate any misrepresentations that the data might provide.

Blunt Instruments

By: Kurt Dobbins -

In his written statement to the Senate Committee on Commerce, Science and Transportation on Tuesday of this week, FCC Chairman Kevin Martin referred to a particular method of traffic management as a “blunt means to reduce peer-to-peer traffic by blocking certain traffic completely.”

The blunt means was referring to how some Deep Packet Inspection (DPI) platforms manage traffic when placed out of line. If a device is out of line, then one of the ways to control traffic is to directly or indirectly signal the sender to slow down or terminate the communication session. Terminating a session is the low hanging fruit because:

1) It’s easy to send a TCP reset message to the sender;

2) How harmful can it be to reset a peer to peer connection, most peer to peer file sharing clients have hundreds of sessions open!

For DPI, out of line versus in line has been an ongoing debate. Overall, out of line DPI is easier to deploy in a network, and easier to engineer as a product. Because out of line placements “tap” into the optical links or receive a mirrored copy of the packet from another network device, there is less risk in inserting into the network. The DPI processing is not in the critical service delivery path of subscriber traffic. If the DPI element fails, it is not service-effecting. “Hey the out of line DPI element crashed! No worries! The subscriber’s real packets are somewhere else!” By being out of line, there are significantly less performance constraints to engineer into the product because the packet seen by the DPI is only a copy. “One half second of latency? No worries! Dropped packets? No worries!”

Out of line is not the only traffic management option, as FCC Chairman Martin, in his written statement, alludes “… more modern equipment can be finely tuned to slow traffic to certain speeds based on various levels of congestion.” This type of finely tuned traffic management requires DPI to be placed in line. In line DPI can gently slow aggressive peer to peer applications during periods of congestion, while simultaneously ensuring that every active subscriber has an equal share of the bandwidth.  In line DPI can promote fairness, while still providing the freedom of subscribers to access content and services of their choice.

Traffic management made possible by in line DPI should resonate with the Chairman. Why? In 2005 the FCC established four principles to promote the open and interconnected nature of the public Internet:

• Consumers are entitled to access the lawful Internet content of their choice;
• Consumers are entitled to run applications and use services of their choice, subject to the needs of law enforcement;
• Consumers are entitled to connect their choice of legal devices that do not harm the network;
• Consumers are entitled to competition among network providers, application and service providers, and content providers.

The Commission also noted that these principles were subject to reasonable network management, so it seems to support in line traffic management, under periods of congestion, which ensures fair access to available bandwidth, content, and services without blocking or denying service.

But, deploying DPI in line is a whole different ball game. Every “live” packet of every subscriber is directly processed by the DPI platform. In line placement has to be certified for being in the service delivery path of subscribers; it has to have failover and redundancy so there is no service loss in case of failure, and it has to have low latency and no dropped packets in order to preserve the service quality of real time applications like voice and video. However, engineering this type of traffic management, say at 10G Full Duplex bandwidth rates, with low latency, no packet loss, and full failover capability is both costly and difficult. But more importantly, it takes a certain mindset and discipline for in line DPI, as evidenced by the European Advanced Networking Test Center AG (EANTC) P2P test report.

The benefits of DPI for traffic management, while well known to service providers, are not well understood by consumers, and the entire debate about Network Neutrality has been misshaped by the lack of transparency. I’ll talk about transparency in a future blog.

 

Net Neutrality’s Unintended Consequence

By: Danny McPherson -

Most of the discussion surrounding net neutrality to date seems to revolve around fixed broadband consumers complaining that ISPs are unjustly mucking with their traffic, discriminating between different traffic types, and providing prioritization without their consent — and the providers shouldn’t be. Rob took a bit of a consumer-centric view to practically addressing net neutrality in his blog post here, I’m going to take more of a ISP-centric view.

ISP actions have centered around attempts to ensure fairness across subscribers (e.g., where contention may exit because of asymmetries in available bandwidth, or shared medium such as cable environments), and optimize network resources, in particular bandwidth utilization at various points in the network. Generically speaking, these network resources exist in one of three primary locations:

  1. Access network capacity: This relates to the bandwidth available between the subscriber’s cable modem and the MSO’s Cable Modem Termination System (CMTS) with cable-based Internet services, or the subscriber DSL modem and Digital Subscriber Line Access Multiplexer (DSLAM) in DSL networks. One of the primary distinctions between these two types of access networks architectures is that, because cable employs the existing shared HFC cable plant, cable access is “shared” across multiple subscribers. Downstream traffic (network -> subscriber) is multiplexed onto a TV channel that is shared between a group of cable modem subscribers. DSL, on the other hand, is “dedicated” in the sense that each digital subscriber loop is allocated a discrete channel from a set of available channels. The offshot with the cable access model is that during periods of heavy utilization subscribers within a given group may experience performance degradation because of high-bandwidth users in the same group. With both cable and DSL, traditionally, available bandwidth was much narrower upstream (subscriber -> network) than downstream, hence asymmetric DSL (ADSL) services, which are similar in cable offerings. The primary reason for this was so that, unlike a synchronous service, higher frequencies could be allocated downstream, providing subscribers nearly 2x download speeds. This, of course, makes assumptions about what applications subscribers are using (assumes larger download capacities v. upstream P2P traffic, video uploads, etc..).
  2. Internal network capacity: Internal capacity includes traffic from DSLAM or CMTS to on-net access servers, local caches or content distribution infrastructure, often including email and other ISP-offered services, and traffic being carried to points at which the ISP inteconnects with other ISPs and content providers. Also, in both cable and DSL architectures, subscribers can’t communicate directly, so it’s necessary for traffic to be switched locally by the DSLAM or CMTS, or routed between devices further upstream within the broadband ISPs network. Often P2P and other local user-user traffic would be included here.
  3. Transit “Internet” capacity: Because ISPs that offer broadband services typically focus in access markets, they’re often considered eyeball heavy. That is, they’ve got lots of users, but little content. What this means is that most of their users access content that doesn’t reside on their network. In order to provider connectivity and access to this content, they either engage in bi-lateral interconnection agreements with content companies and other types of ISPs, or acquire transit services from ISPs that provide this connectivity. Lower-priced transit services from ISPs may range from ~$100-$300/mbps or more monthly recurring, just for IP access, in addition to transport infrastructure costs, which includes the local loop and often long-haul backbone circuits upon which IP traffic is carried. And they’ve got all the associated capital investment, of course. Scaling this capacity for millions of users is a significant cost. Recall that unlike a utility, your Internet transactions could be with devices just down the street, or across the globe. Infrastructure has to be provisioned to accommodate these any-any connectivity models.

Broadband access has obviously not been what most folks might refer to as a high-margin market, with profits in the 5-10% range (after removing less-fortunate outliers). Given that most of these ISPs are publicly-traded companies and have shareholder expectations to meet, something has to give. Two things broadband ISPs have traditionally been most concerned with are subscriber growth, and minimizing subscriber churn. Growing average revenue per user (ARPU), and ideally, associated profit margins, has been a key driver for services convergence in the form of triple play (voice, video, and data). However, with the higher revenue services comes higher consumer expectations, in particular for services availability, performance and security. Furthermore, minimizing call center volume is key to optimizing profitability, as with traditional data services only yielding 5-10% profits with low ARPU services, a single customer help desk call can often snuff out profitability for 3 years or more!

ISPs simply can’t continue growing subscriber numbers and access speeds without continuing to invest heavily in Internet transit and interconnection bandwidth, internal network capacity, and access infrastructure. Looking for ways to offset the operational costs associated with these various investments is critical. Some ways of doing this include initiatives such as P4P, or investing in infrastructure that time-shifts lower-priority traffic into network utilization “troughs”, providing more real-time protocols such as VoIP with the network performance characteristics they demand. Peak-to-trough access network utilization rates today are often 4x or more, and because networks have to be engineered to perform without capacity problems during peak utilization periods – capacity planning for ISPs becomes a painfully expensive ordeal. This is why traffic management for ISPs is a critical function – liken it to the PSTN’s Mother’s Day problem.

I don’t believe exploiting protocol behaviors by doing arguably unethical things like injecting spoofed spurious TCP RSTs is the right answer, as that’s just going to piss subscribers off, increase call center volumes and subscriber churn, further compromise transparency, potentially break applications and result in non-deterministic network behavior. Unless there are incentive structures in place to encourage users to change behaviors (e.g., that of how their P2P protocols utilize network bandwidth) then nothing is going to change.

Which finally, leads me to my point (yes, there is one). Consumer advocates say ISPs shouldn’t discriminate, shouldn’t attempt to time-shift traffic, and basically, shouldn’t gate subscriber traffic in any way. So, under the auspices of the net neutrality proponents, if you’re an ISP, pretty much all you can do is one thing: change your billing models to change subscriber behaviors, and generate more revenue to fund further build-out of your infrastructure. That is, bill customers differently. Be it either strict metered services (usage-based), or more sophisticated usage-based services that introduce peak v. off-peak, on-net versus off-net, distance-sensitive, domestic versus international! It certainly costs more for a US broadband subscriber to obtain content or engage in a VoIP conversation with someone in London, or Singapore, or Cairo, as opposed to someone across the street. Who bears these costs, costs which have traditionally been “transparent” to subscribers? Think about it, this all you can eat model can’t continue, and users don’t change behavior without being incentivized to do so.

Why do the mobile folks have such high ARPU, and better profitably per subscriber? Because of a slew of billing models they have that make users accountable for network resources they consume. Ever use data roaming services on your mobile handset in another country and discover at the end of the month how incredibly expensive it can be? There’s a good reason for this.

Interestingly, technology hasn’t been at a place where these types of IP-based metered billing models were a viable option. Today, they are, and ISPs are being given little alternative. And trust me, your Internet access bill is NOT going to get cheaper as a result. We’re starting to see basic metered models emerge already.

Net neutrality advocates, be careful what you wish for…

Go Back In Time →