This is the second part of a three-part blog series on Veeam and Pure Storage FlashBlade. In the previous blog post, we configured a Network File System (NFS) share on a Pure Storage FlashBlade as a Veeam backup repository. In this blog, we’ll focus on configuring SafeMode snapshots to harden the backup files that are residing on the FlashBlade.
Ransomware attacks continue to rise with constantly evolving sophistication and complexity. A key part of a ransomware resilience strategy is backing up data on a regular basis and implementing a strong line of defence against threats targeting the backup data. Adopting industry standards for data protection such as a 3-2-1 rule, offline backups and immutable backup storage are effective techniques to protect backup data sets against malicious attacks. Now let’s discuss how to make your FlashBlade system an immutable backup storage target with SafeMode snapshots.
A storage snapshot is a point-in-time, image-level view of data that is impervious to ransomware. This immutability makes a storage snapshot an ideal layer of defence against ransomware. The problem with storage snapshots is that they can still be removed by rogue admins or attackers if they gain access to the storage array management. In the case of a Pure Storage system, the deleted snapshots are temporarily stored in a ‘destroyed state’ that is similar to a recycle bin. If these snapshots are not recovered in a timely manner, they will be auto-eradicated and can even be manually destroyed prior to the auto-eradication.
The SafeMode snapshots on the other hand, cannot be deleted, modified, or encrypted- either accidentally or intentionally. This prevents the manual and complete eradication (permanent deletion) of data backups that are stored within the FlashBlade. Due to their immutability, the SafeMode snapshots serve as an additional mitigation mechanism against ransomware attacks or rogue administrators.
The SafeMode snapshots are created and managed automatically by Purity, the FlashBlade operating software, independent of administrator control. You can schedule SafeMode snapshot generation of any data residing on the FlashBlade file systems. Primary and backup data can be directly recovered from these snapshots. If original copies are corrupted or destroyed, data can still be recovered from the SafeMode snapshots.
Once the SafeMode snapshots are activated, snapshots for all the file systems get created that incur additional capacity utilisation on FlashBlade. Therefore, prior to SafeMode activation, it is recommended to consult your local Pure Storage Systems Engineer to validate the FlashBlade capacity sizing required to support SafeMode snapshots. Following is a sample workflow:
To make a change to an existing SafeMode policy, follow the high level procedures described below:
Before contacting Pure Support, select at least two individuals who will be listed as SafeMode authorised users. These individuals will be provided with a PIN for safekeeping. Pure Support requests this PIN every time an authorised user calls and requests a change to the SafeMode configuration.
Only an authorised person with the correct PIN is allowed to make changes to the SafeMode configuration via Pure Support. This is the key layer of defence. SafeMode storage snapshots can be deleted if an authorised user with the correct PIN requests Pure Support to do so. Only these authorised personnel can work directly with Pure Technical Support to configure the feature, modify policy, or manually eradicate snapshots.
Once the authorised user list has been provided, Pure Support performs a preactivation health check that involves checking the FlashBlade growth trend to ensure sufficient capacity to enable SafeMode. Note: Pure Support destroys any pre-existing snapshots before enabling SafeMode.
The following table presents the SafeMode sizing examples to assist readers with possible SafeMode storage consumptions.
Pure Support will request the required data retention period. It is recommended to provide a value that allows adequate time to identify and suppress an attack.
Following are the SafeMode configurations that are performed by Pure Support.
Once Pure Support has enabled SafeMode, validation is required.
The screenshot above displays a list of snapshots of the file system . There are 2 distinct snapshot groups presented here:
When attempting to eradicate a SafeMode snapshot, the file system will be moved to a destroyed state.
The destroyed snapshot will be placed in a destroyed snapshot folder as highlighted below. In our test, eradication timeout for the destroyed snapshots was changed from the default value of 14 days to 7 days.
When attempting to eradicate (permanently deleting) a SafeMode snapshot, a warning message will be displayed. as shown below.
It is evident that under SafeMode, the administrators cannot manually remove a filesystem or snapshots from FlashBlade.
A sophisticated ransomware attack does not just encrypt production data. Perpetrators often target the backups to ensure organisations cannot recover. There are cases where the backup servers were compromised by ransomware attacks and became inaccessible. Other possible scenarios are when perpetrators and/or rogue admins gain access to the storage systems and attempt to delete the file systems and snapshots hosting backup data.
To demonstrate the effectiveness of SafeMode, we have assumed that access to both the Veeam console and FlashBlade interface has been gained. From the Veeam console, a ‘Delete from disk’ operation was performed on all backups that are stored on the FlashBlade NFS repository. This results in the deletion of the following:
The following screenshot depicts the status of after all of its content is removed from the corresponding Veeam backup repository. Note that the used space displays 0TB as the file system is storing nothing.
The NFS file system that is used for the Veeam repository was then destroyed from the FlashBlade console.
If other protection mechanisms are compromised, the FlashBlade storage becomes a solid final line of defence. As SafeMode snapshots are immutable and cannot be eradicated manually, the recovery process can be started, as below.
To recover a destroyed file system:
Review the file systems in the destroyed container and click Restore. Note that the destroyed file system moves back to the file system folder.
Based on our simulated attack, restoring the file system is just the first step to recovery, as both the Veeam backup data and the File System (NFS) were destroyed. The next step is to revert the File System to a previous snapshot before the data was deleted/compromised.
Reverting the file system to a previous SafeMode snapshot requires the intervention of Pure Support. The FlashBlade administrators are not allowed to perform a filesystem rollback using GUI or CLI. The company designee needs to contact Pure Support to complete the restoration process.
Note: Purity does not allow snapshot restore by an administrator with SafeMode enabled. An error message is shown whenever Administrators attempt to revert a snapshot.
Let’s consider a few scenarios.
Scenario 1: The backup records in the Veeam configuration database were not destroyed.
Scenario 2: The backup records in the Veeam configuration database were destroyed.
Scenario 3: Veeam Backup & Replication server was destroyed
Adding a FlashBlade NFS Share to Veeam as a Backup Repository is covered in part one.
The simulated attack described above involved deleting the backup records from the Veeam configuration database, but not from the NFS backup repository. Therefore, the following steps are performed.
After the rescan is complete, the restore points will reappear under Backups > Disk (Imported)
Veeam can now perform an Instant Recovery to immediately restore VMs by running them directly from the backup files stored on the reverted NFS backup repository.
The following screenshot shows the restored VMs that are available in the vSphere Client.
Following are a few caveats surrounding SafeMode that customers should be aware of:
It is important to consider the security of the device as well. If the storage device can be easily taken over, then the software layer providing the storage immutability may not be of much help. Fortunately, in the case of Pure Store SafeMode, even if the management GUI was accessed, the rogue administrator cannot factory reset/wipe the storage and manipulate the storage snapshots.
To further increase security, Multi-Factor Authentication (MFA) will be introduced in an upcoming Purity release (v6).
While not required, it is recommended to configure the Veeam Backup & Replication (VBR) Configuration Backup in the same NFS Backup Repository. This simplifies the VBR server recovery when the server is corrupted or unavailable. A new VBR can be deployed and the Veeam configuration backup can be recovered onto the server.
The Pure Storage FlashBlade is an on-premises backup storage that provides immutable storage for data such as Veeam backups. SafeMode configuration provides an additional layer of protection against ransomware/cyber-attack.
In the next blog post, we will discuss how Object Storage presented from the FlashBlade will be used as a backup repository target for Veeam Backup for Office 365.
Disclaimer: For advice regarding retention periods and ransomware security, please speak to your local Pure Storage SE.
Standard Snapshot restoration of Veeam backups (GUI)
To restore a non-SafeMode snapshot:
Following are commands that are used to restore the file system with snapshot via Purity FB CLI.
The FlashBlade administrators can use either CLI or GUI to perform file system restoration as if the array is not operating under SafeMode. Customers will need to contact Pure Support to perform the following steps in FlashBlade under SafeMode.
After a successful file system restore, one can observe that the used capacity on file system is increased to 114.51GB. This indicates that the previously removed objects are restored to the file system.
The next step is to proceed to the Veeam console to complete the recovery of affected virtual machines.