Rackspace is in a crater right now, on fire.
So are many of its customers. I’m one of them.
What happened, Rackspace says, was “the result of a ransomware incident.” Damaged, lost or destroyed is its Hosted Exchange business. On that cloud platform, companies and individuals around the world run their email and much else.
It’s quite a saga.
The first report from Rackspace came at 11:49pm Pacific (where I am) on Thursday, posted on its Incidents page:
We are investigating an issue that is affecting our Hosted Exchange environments. More details will be posted as they become available.
Updates got more wordy as the outage continued. About a day later, a long posting said,
We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact. After further analysis, we have determined that this is a security incident.
They also offered a lifeline of sorts:
At no cost to you, we will be providing access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice. At no cost to you, we will be providing access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice. To activate, please use the below link for instructions on how to set up your account and users.
For reasons not worth going into, this was useless to me. But I’m also just one guy, and almost all of Rackspace’s customers are businesses with more to lose.
Getting help from Rackspace quickly became difficult or impossible, while details about the situation were minimal, until this morning (6 December), when they said it was a ransomware incident. Which countless tweets had been suggesting from the start.
Here is what Rackspace said it was doing about it:
Alongside our internal security team, we have engaged a leading cyber defense firm to investigate. Our investigation is still in its early stages, and it is too early to say what, if any, data was affected. If we determine sensitive information was affected, we will notify customers as appropriate.
No FBI? CIA? NSA? DHS? Police? My guess is that at least two of those have been notified because it would make sense for them to be involved. But I’m sure abundant caution on Rackspace’s part is the reason we’re not hearing about any of that.
As for notifying customers, good luck. In a posting two days ago, Rackspace said,
For those who are finding the process challenging and are awaiting support, we ask for your patience as we increase staff to help every customer. Since our last update, we have mobilized roughly 1000 support Rackers to reduce wait times and address ticket queues. We will continue to accelerate and deploy even more resources to further help customers.
Search for Rackspace+wait on Twitter to see how that’s going.
Yesterday morning I chose the “we’ll call you back asap” option at the Rackspace support number, after calling them fruitlessly before that. Since then, crickets. Meanwhile, I’ve been working almost non-stop on moving my email hosting to Hover, which answers the phone quickly and is always helpful.
Of course, I haven’t been able to export anything from Rackspace, and I have growing doubts that I ever will. If the failure is total, many details of my life in the digital world will be lost forever.
One bit of good fortune is that my wife and sister, who both also have searls.com email addresses, were on Rackspace’s basic non-Exchange email service. Since that was still working today, we could export their mail archive from Rackspace as .mbox files, and start new accounts for them on Hover. (Ironically, I moved to Rackspace’s Exchange service because I punched through the basic service’s 25Gb limit on storage, and they sold me on the Exchange service’s better security.)
Ramsomware is bad shit. If you’re Rackspace or one of its customers, there is plenty to fear.
But the larger story here isn’t about Rackspace or its customers. It’s about the cloud business, which is beyond massive.
I’ve been looking for examples of cloud failures that are bigger than this one. Here’s a list from five years ago. Here is one from July of this year. Here is one from August. Most list disruptions lasting hours. This one has been going on for five days with no end in sight.
So let’s imagine that Rackspace and its customers are stuck in that crater, and it just keeps burning. For years. What are the lessons from that?
[Later, on 10 December…] This report by Kevin Beaumont on the Rackspace catastrophe (which continues) is long on technical and administrative details that nobody else seems to be reporting, and is damning to Microsoft as well.
Leave a Reply