SPONSORED: In August 2023, Danish hosting subsidiaries CloudNordic and AzeroCloud were on the receiving end of one of the most serious ransomware attacks ever made public by a cloud services company. During the incident, CloudNordic suffered a complete encryption wipe-out that took with it applications, email services, websites, and databases, and even backup and replication servers. In a memorably frank admission, the company said that all customer data had been lost and would not be recoverable.
To the hundreds of companies Danish media reported as having lost data in the incident, this must have sounded incredible. Surely service providers are supposed to offer protection, not even greater vulnerability? Things were so bad, CloudNordic even offered customers last resort instructions on recovering lost website content through the Wayback Machine digital archive. The company reportedly refused to pay a ransom demanded by the attackers but even if it had paid there is no guarantee it would have made any difference.
Ransomware attacks are a dime a dozen these days and the root causes are various. But the assumption every customer makes is that behind a service provider’s virtual machine (VM) infrastructure is a comprehensive data protection and disaster recovery (DR) plan. Despite the common knowledge that ransomware targets backup and recovery systems, there is still a widespread belief that the same protections will always ride to the rescue and avoid catastrophic data loss. The CloudNordic attack is a warning that this isn’t always the case. Doubtless both companies had backup and data protection in place, but it hadn’t been enough.
“The attack and its outcome is not that extraordinary,” argues Kevin Cole, global director for technical product marketing at Zerto, a Hewlett Packard Enterprise company. “This probably happens more than we know. What’s unusual about this incident is simply that the service provider was open about the fact their backups had been attacked and deleted.”
This is what ransomware has done to organizations across the land. Events once seen as extreme and unusual have become commonplace. Nothing feels safe. Traditional assumptions about backup and data resilience are taking a battering. The answer should be more rapid detection and response, but what does this mean in practice?
The backup illusion
When responding to a ransomware attack, time is of the essence. First, the scale and nature of the incursion must be assessed as rapidly as possible while locating its source to avoid reinfection. Once this is within reach, the priority in time-sensitive industries is to bring multiple VM systems back online as soon as possible. Too often, organizations lack the tools to manage these processes at scale or are using tools that were never designed to cope with such an extreme scenario.
What they then fall back on is a mishmash of technologies, the most important of which is backup. The holes in this approach are well documented. Relying on backup assumes attackers haven’t deactivated backup routines, which in many real-world incidents they manage to do quite easily. That leaves offline and immutable backup, but these files are often old, which means that more recent data is lost. Even getting that far takes possibly days or weeks of time and effort.
Unable to contemplate a long delay, some businesses feel they have no option but to risk paying the ransom in the hope of rescuing their systems and data within a reasonable timescale. Cole draws a distinction between organizations that pay ransoms for strategic or life and death reasons – for example healthcare – and those who pay because they lack a well-defined strategy for what happens in the aftermath of a serious attack.
“Organizations thought they could recover quickly only to discover that they are not able to recover within the expected time window,” he explains. “They pay because they think it’s going to ease the pain of a longer shutdown.”
But even this approach is still a gamble that the attackers will hand back more data than will be recovered using in-house backup and recovery systems, Cole points out. In many cases, backup routines were set up but not properly stress tested. Under real-world conditions, poorly designed backup will usually fall short as evidenced by the number of victims that end up paying.
“Backup was designed for a different use case and it’s not really ideal for protecting against ransomware,” he says. “What organizations should invest in is proper cyber recovery and disaster recovery.”
In the end, backup falls short because even when it works as advertised the timescale can be hugely disruptive.
Achieving ransomware resilience
It was feedback from customers using the Zerto solution to recover from ransomware that encouraged the company to add new features tailored to this use case. The foundation for the Zerto solution is its continuous data protection (CDP) technology, with its replication and unique journaling technology, which reached version 10 earlier this year. Ransomware resilience is an increasingly important part of this suite, as evidenced by version 10’s addition of a real-time anomaly system that can detect that data is being maliciously encrypted.
Intercepting ransomware encryption early not only limits its spread but makes it possible to work out which volumes or VMs are bad and when they were infected, so that they can be quickly rolled back to any one of thousands of clean restore points.
“It’s anomaly and pattern analysis. We analyze server I/O on a per-volume basis to get an idea of what the baseline is at the level of virtual machines, applications and data,” explains Cole. “Two algorithms are used to assess whether something unusual is going on that deviates from this normal state.”
An important element of this is that Zerto is agentless which means there is no software process for attackers to disable in order to stop backup and replication from happening behind the victim’s back.
“It sounds small but it’s a really big advantage,” says Cole. “Many ransomware variants scan for a list of backup and security agents, disabling any they find protecting a VM. That’s why relying on a backup agent represents a potential weakness.”
A second advanced feature is the Zerto Cyber Resilience Vault, a fully isolated and air-gapped solution designed to cope with the most serious attacks where ransomware has infected the main production and backup infrastructure. Zerto stresses that this offers no point of compromise to attackers – replication from production via a ‘landing zone’ CDP mirror happens periodically via an FIPS-validated encrypted replication port rather than a management interface which might expose the Vault to compromise.
The possibility of a total compromise sounds extreme, but Cole points out that the use of this architecture is being mandated for financial services by the SEC in the U.S., and elsewhere by a growing number of cyber-insurance policies. The idea informing regulators is that organizations should avoid the possibility of a single point of failure.
“If everything blows up, do you have copies that are untouchable by threat actors and which they don’t even know exist?,” posits Cole. “In the case of the Cyber Resilience Vault, it’s not even part of the network. In addition, the Vault also keeps the Zerto solution itself protected – data protection for the data protection system.”
Ransomware rewind
The perils of using backup as a shield against ransomware disruption are underscored by the experience of TenCate Protective Fabrics. In 2013 this specialist manufacturer of textiles had multiple servers at one of its manufacturing plants encrypted by the infamous CryptoLocker ransomware. This being the early days of industrial ransomware, the crippling power of mass encryption would have been a shock. Tencate had backups in place but lost 12 hours of data and was forced to ship much of its salvageable data to a third party for slow reconstruction. In the end, it took a fortnight to get back up and running.
In 2020, a different version of CryptoLocker returned for a second bite at the company, this time with very different results. By now, Tencate was using Zerto. After realizing that one of its VMs had been infected, the security team simply reverted this to a restore checkpoint prior to the infection. Thanks to Zerto’s CDP, the total data loss was a mere ten seconds and the VM was brought back up within minutes.
According to Cole, TenCate’s experience shows how important it is to invest in a CDP that can offer a large number of recovery points across thousands of VMs with support for multi-cloud.
“Combined with encryption detection, this means you can quickly roll back, iterating through recovery points that might be only seconds apart until you find one that’s not compromised.”
While loss of service is not the only woe ransomware causes its victims, the inability to run applications and process data is where the immediate economic damage always begins. For the longest time, the only remedy was to keep the attackers out. But when those defenses fail as they surely will one day, it is better to fail in style, says Cole.
“The choice is not just backup as people have come to know it,” he advises. “Continuous data protection and isolated cyber recovery vaults are the future anyone can have now.”
Sponsored by Zerto.
