Change All Your Passwords, Right Now!

ASERT team

by Steinthor Bjarnason, Senior ASERT Security Analyst & Roland Dobbins, ASERT Principal Engineer

CloudFlare are probably best known as a DDoS mitigation service provider, but they also operate one of the largest Content Delivery Networks (CDNs) on the Internet. Many popular Web sites, mobile apps, etc. make use of the CloudFlare CDN, which hosts content associated with more than 4 million DNS domains, worldwide.

CloudFlare have developed customized software which allows their CDN to service a huge amount of Web requests every second of every day. It’s a complex, highly-scalable system, and CloudFlare are continually working to enhance its functionality and scalability in order to meet the needs of their customers.

Unfortunately, it appears that there was a serious security vulnerability in one of the software components used in CloudFlare’s CDN system which was exposed as CloudFlare were making some alternations to the service late last year. And as a result, it appears that a great deal of information related to CloudFlare-powered Web sites and applications – potentially including usernames, passwords, chat sessions, and other forms of personally-identifying information (PII) – has inadvertently leaked from CloudFlare and has been cached by search engines such as Google, Bing, DuckDuckGo and Baidu; privately-run ‘Web scrapers’; and similar services.

That means that this heretofore-private information for millions of Internet users – again, potentially including access credentials – has been available to anyone to stumble across, and possibly exploit.

And it doesn’t really matter whether that information was encrypted via SSL/TLS or not.

How Did This Happen?

Whenever one of the CloudFlare CDN reverse-proxies receives an SSL/TLS request, it has to decrypt the request and perform some processing in order to determine how best to service the request. The results of this processing would normally be stored in the operating RAM of the reverse-proxies and then forgotten about (not deleted) as the system moved on to service subsequent requests.  A bug in a component of the CloudFlare CDN service stack resulted in some users not only receiving the results they were expecting, but also random memory contents from the CloudFlare reverse-proxies – which could contain everything from other users requests and responses to sensitive information like decrypted passwords and usernames.

Basically, if user A accessed content from server X, user B could, in addition to the expected results from server Y, see what user A got in his responses from server X.  Normally, this wouldn’t be an issue as Web browsers would ignore the additional content, and user B would only see his expected results.

However, to improve performance and for purposes of statistical analysis, almost all of the big Web search providers like Google, Bing, Baidu, et. al. will cache and store information on almost all Web servers in the world.  This means that when user B did made requests and inadvertently received potentially sensitive information related to user A, the results were often stored in the caches of these popular search engines – and possibly in privately-run Web crawlers, too.

This resulted in the CloudFlare CDN inadvertently leaking random user session information which was subsequently stored by Google and other search providers.  This information is readily searchable, and is relatively easy to locate using simple search tools.  Google has been actively working with CloudFlare to find this kind of inadvertently-leaked-and-cached sensitive content and are actively working to purge it from the Google cache. It is assumed that other prominent search engine operators are working with CloudFlare to do this on their platforms, as well.

What Does This Mean to Me?

As noted previously, many of the most popular Web sites, mobile apps, and related services make use of the CloudFlare CDN service. A programmatically-generated list of the domains for these sites and services may be found here. This handy Chrome Web browser plugin will also alert users when they’re browsing a CloudFlare-powered site, and this Web-based utility is also very useful.

Chances are, you’ve used one or more of the sites, services, and/or apps serviced by the CloudFlare CDN, whether you’ve accessed them directly, or whether they’re a dependency of some other site, service, or app you’ve used. Because of the popularity of the CloudFlare CDN service, it’s for all practical purposes impossible for most Internet users to determine whether or not their Web browsing or app traffic has been served by the CloudFlare CDN.

And that’s where the problem lies.

Change Them All, and Change Them Now!

For most of us, the only truly safe response to this large-scale information leak is to update our passwords for the Web sites and app-related services we use every day. Pretty much all of them.

Google, Apple, and Microsoft do not appear to utilize the CloudFlare CDN – but unless one has the time, energy, and available to do a detailed analysis of one’s historical Web surfing and app usage over the last several months, it’s better to do a password rotation for all other accounts, just to be on the safe side.

And keep in mind that other types of information may’ve leaked as well – everything from chat logs to email contents to financial information and Web browsing habits. So, while it’s always a good idea to keep a weather eye on one’s online information, increased vigilance in the wake of this mass information leakage is warranted.

A Rapid Response to a Big, Complex Problem

Google security researcher Tavis Ormandy of Google’s Project Zero team initially identified this issue and contacted CloudFlare in short order. CloudFlare responded rapidly, and have been working with Google and other organizations to remove as much of the leaked information as possible from Web search engine caches, and have provided a public explanation of how this incident occurred, and the steps they’ve taken to both remediate it and ensure that it doesn’t happen again.

While it’s unfortunate that events of this nature take place, both Google Project Zero and CloudFlare themselves should be commended for taking quick action to resolve the issue to the best of their ability, and in publicly discussing what took place so that other security researchers – and the Internet community at large – can make their own assessments of the potential impact. It is our hope that more organizations will learn from their example, and move to both remediate and disclose pertinent details of security incidents in a timely and forthright manner.

A Special Note for Arbor Cloud Customers

We expect to receive some inquiries as to whether it’s possible that a problem of this sort could arise for sites and services protected by our own Arbor Cloud DDoS mitigation service.

Because Arbor Cloud is focused strictly on DDoS mitigation and is not a CDN service, it doesn’t perform the same types of operation on customer content and application traffic as does a CDN service like CloudFlare’s. So, this type of issue doesn’t apply to the Arbor Cloud service, nor to Arbor Cloud customers.

Additional Insights on Shamoon2

IBM analysts recently unveiled a first look at how threat actors may have placed Shamoon2 malware on systems in Saudi Arabia. Researchers showcased a potential malware lifecycle which started with spear phishing and eventually led to the deployment of the disk-wiping malware known as Shamoon. Their research showcased a set of downloaders and domains that […]

Flokibot Invades PoS: Trouble in Brazil

Curt Wilson

Introduction Threat actors salivate at the thought of an increased volume of credit and debit card transactions flowing through endpoints they have compromised with card-stealing malware. While there are many distinct malware families that scrape unencrypted process memory to obtain cards, some of these malware capabilities overlap with generic information stealing trojans such as Flokibot […]

Non-Government Organization in Support of Government Hopes

Red Team analysis is the process of viewing a situation from the perspective of an adversary thus providing insights beyond those that might otherwise be limited by normative biases. This blog provides an assessment of one company’s foray into the popular APT outing trend in the context of China’s cyber buildup and what that could […]

Dismantling a Nuclear Bot

Dennis Schwarz

A recent tweet mentioned that a new banking malware called “Nuclear Bot” has started to appear for sale on underground marketplaces. Its price starts around $2500 which is more than double the price of another recent entry to the market. This post dismantles a sample of this malware to determine whether we need to take […]

On the Economics, Propagation, and Mitigation of Mirai

ASERT team

By Kirk Soluk and Roland Dobbins In late November of 2016, a new Mirai variant emerged that leveraged a propagation mechanism different from the Telnet-based brute forcing mechanism originally provided in the leaked Mirai source code. This new variant exploits vulnerable implementations of the TR-064/TR-069 protocol used by ISPs to remotely manage their customer’s broadband […]

Analysis of CryptFile2 Ransomware Server

Curt Wilson

Download ASERT Threat Intelligence Report 2016-06 here This report describes several elements of a ransomware staging system using the Nemucod malware to deliver CryptFile2 (aka Hydracrypt.A and Win32/Filecoder.HydraCrypt.C) ransomware, an ongoing threat since at least mid-March of 2016. This report reveals TTP’s (tactics, techniques, procedures) of threat actors, including insight derived from limited interactions via e-mail. […]

Diving Into Buhtrap Banking Trojan Activity

Curt Wilson

Cyphort recently published an article about the Buhtrap banking trojan [], targeting users of Russian and Ukrainian banks as reported in March of 2016 by Group-IB []. Cyphort’s insightful article analyzes the compromise chain from the website eurolab[.]ua, directing users via an apparently injected HTML script src attribute to rozhlas[.]site which served exploit code for […]

FlokiBot: A Flock of Bots?

Dennis Schwarz

In early October, Flashpoint released an analysis of an underground forum advertisement for a new malware family known as FlokiBot. It took some time before a sample was found in the wild, but a researcher known as hasherezade flagged one on VirusTotal in early November. She also wrote an analysis of its dropper here. This […]

Flying Dragon Eye: Uyghur Themed Threat Activity

Curt Wilson

DOWNLOAD FULL REPORT HERE DOWNLOAD INDICATORS OF COMPROMISE (IOCs) HERE This paper documents attempted exploitation activity aimed at Uyghur interests outside of China. Exploitation is being attempted via the usual tactic of spear phishing containing malicious attachments to targets. The exploit code attached used for dropping the malware is older – CVE-2012-0158 – and from […]