In this article

What would happen if your company's data, the information, and the systems you need to operate, suddenly became unavailable? Furthermore, what if your backup data was also unavailable or corrupt? Would you be able to continue running your business, or would you have to close shop?

In 2020, a major hospital chain in the US, with over 400 locations, began to see their systems fail over a weekend. Medical staff resorted to pen and paper, including hand-labeling medications. In 2017, the WannaCry ransomware cryptoworm infected organizations worldwide, including the National Health Service in the UK, Boeing, and FedEx. More recently, in 2021, Colonial Pipeline was attacked utilizing a compromised password; the company shut the pipeline down to mitigate physical risk. The company paid a $4.4M ransom to the attackers because Colonial didn't know how deep the attackers reached and were concerned about the business effects of an ongoing outage. Predicted global ransomware costs will top $20 billion in 2021, and their fiscal impacts will rise 15% annually*.

In light of the increasing attacks, companies must develop in-depth strategies to mitigate, detect, and recover from cyber-attacks. Simple backups are not enough; the best possible solution is multi-pronged. First, since most attacks start with social engineering, educate users to recognize suspicious links or attachments. Next, take advantage of your primary storage array's native snapshot capabilities for rapid in-place recovery. Finally, an air-gapped vault copy of backup data is where software can apply analytics to look for unusual behavior in the data. While the backup copy is ultimately the most robust solution, recovery can take some time and is generally the copy of last resort. This article will focus on getting Secure Snaps running on PowerMax arrays.

PowerMax

Part of Dell EMC's PowerMax TimeFinder local replication suite is a feature called Secure Snaps, released in 2017 due to customer requests. A security-focused extension to SnapVX, Secure Snaps allows an expiration date to be set on snapshots, up to 400 days from the time of creation. Copies marked as secure cannot be deleted before the expiration date arrives; the date can be extended, but it cannot shrink. Additionally, Secure Snaps interact with a few other features in PowerMaxOS that lend themselves to keeping your data safe.

  • Tamper-proof clock - Secure Snap retention time works from the system's tamper-proof clock, which cannot advance anywhere near fast enough that using an NTP skew attack is valuable. In other words, even though a bad actor is trying to move the clock forward to cause snaps to expire, the rate at which the array will allow the clock to move makes this useless.
  • Pool reserve capacity (PRC) - When a VMAX3, VMAX AFA, or PowerMax hits 90% full (the default value), it will prevent additional copies from being made and will start invalidating existing snapshot copies. It does this to support primary host data; snapshots are considered secondary and take a back seat to primary volumes. If an attacker were to trigger a massive amount of host writes to try to fill data volumes, this would cause any snapshots to terminate forcibly, leaving you exposed. Secure Snaps ignore the PRC value and stick around until the array is 100% full.
  • Because of the way the array handles host mounting copies via linked targets, these snapshot copies are, by nature, immutable. A linked target has its scratch data area separate from the snapshot.

There are several ways to trigger a Secure Snap in the array: Unisphere, Snapshot Policies (preferred), or symcli.

Snapshot policies

Snapshot policies were introduced in the Q3 2020 release of PowerMax OS, 5978.669.669, also called Hickory. It allows users to create and apply policies to storage groups rather than schedules. As part of setting up a policy, you can tag them as secure, and their time-to-live will follow the retention setting in the policy. A side benefit of moving to Snapshot Policies is you gain the ability to have 4x more snapshots per source volume when compared to traditional SnapVX snapshots: 256 → 1024.

Snapshot Policy setup dialog

SYMCLI

Solutions Enabler has been around for decades and used to be the preferred or only way to manage the array. Its rich feature set aligns well with scripting and day-to-day operations. It has continued to evolve and support new array functionality as it comes out. Creating Secure Snaps with symcli is simple.  

symsnapvx -sid <SID> establish -name <name> -secure -delta <delta_time>  

Additionally, to take an existing snapshot and convert it to secure, give it your snapshot name and do the following:

symsnapvx -sid <SID> -snapshot_name <snapshotname> set secure -delta <delta_time>

Note, once you convert a snapshot to secure, there's no going back without some effort, which I'll discuss below.

Unisphere for VMAX/PowerMax

To create a secured snapshot in Unisphere is simply ticking another checkbox during creation. Go to Storage Groups → Protect, and in the dialog that pops up, make sure expiry type is set to 'none,' go to advanced options and check the "Enable Secure Snaps" checkbox.  

Snapshot creation advanced options dialog

As with all local protection, you need to monitor space consumption, particularly on high change rate (write) workloads or keeping copies for extended periods. Recall from above that Secure Snaps will ignore the storage pool filling up, so you can end up in a bad place if not careful. Appropriate trending and notifications are strongly encouraged. Additionally, once snapshots are marked secure, they cannot be converted back to a standard snapshot. For example, if you accidentally set 400-day retention, the only way out of the situation is to get some documents signed by management at your company and Dell. Once confirmed the appropriate signing levels truly desire this action and data loss documentation is signed, support can remedy the situation, logging in with their secure service credentials and terminate the snapshot.

Conclusion

Nobody is going to solve ransomware in a short article like this. Implementing Secure Snaps on your PowerMax array is just a step on the road to protection. The best mitigation strategy will involve numerous internal security, networking, and governance players. It will take real work to get everyone to the table and develop a defensive plan, and these will not be the same for every customer. Ideally, an isolated environment, sometimes called "air-gapped," severely limits the attack surface and only allows specific traffic from a limited host set into the vault. This Secure Snap approach takes you from zero to a level where you have options.

WWT has strong capabilities in developing and implementing ransomware protection for customers of all sizes. We can help you build your strategy and automate your ransomware protection solution, including a cyber vault. The criminals are only getting better tools, so start planning before it's too late. To continue the conversation, reach out to me or anyone else in our primary storage and data protection practices.

Engage with the global engineering team
Request workshop

Technologies