Share

Safeguard your Veeam backups with Pure Storage FlashBlade SafeMode

Written in collaboration with Lawrence Ang and Dilupa Ranatunga of Veeam

This is the second part of a three-part blog series on Veeam and Pure Storage FlashBlade. In the previous blog post, we configured a Network File System (NFS) share on a Pure Storage FlashBlade as a Veeam backup repository. In this blog, we’ll focus on configuring SafeMode snapshots to harden the backup files that are residing on the FlashBlade.

Ransomware attacks continue to rise with constantly evolving sophistication and complexity. A key part of a ransomware resilience strategy is backing up data on a regular basis and implementing a strong line of defence against threats targeting the backup data. Adopting industry standards for data protection such as a 3-2-1 rule, offline backups and immutable backup storage are effective techniques to protect backup data sets against malicious attacks. Now let’s discuss how to make your FlashBlade system an immutable backup storage target with SafeMode snapshots.

A storage snapshot is a point-in-time, image-level view of data that is impervious to ransomware. This immutability makes a storage snapshot an ideal layer of defence against ransomware. The problem with storage snapshots is that they can still be removed by rogue admins or attackers if they gain access to the storage array management. In the case of a Pure Storage system, the deleted snapshots are temporarily stored in a ‘destroyed state’ that is similar to a recycle bin. If these snapshots are not recovered in a timely manner, they will be auto-eradicated and can even be manually destroyed prior to the auto-eradication.

The SafeMode snapshots on the other hand, cannot be deleted, modified, or encrypted- either accidentally or intentionally. This prevents the manual and complete eradication (permanent deletion) of data backups that are stored within the FlashBlade. Due to their immutability, the SafeMode snapshots serve as an additional mitigation mechanism against ransomware attacks or rogue administrators.

The SafeMode snapshots are created and managed automatically by Purity, the FlashBlade operating software, independent of administrator control. You can schedule SafeMode snapshot generation of any data residing on the FlashBlade file systems. Primary and backup data can be directly recovered from these snapshots. If original copies are corrupted or destroyed, data can still be recovered from the SafeMode snapshots.

Setup SafeMode

Overview

Once the SafeMode snapshots are activated, snapshots for all the file systems get created that incur additional capacity utilisation on FlashBlade. Therefore, prior to SafeMode activation, it is recommended to consult your local Pure Storage Systems Engineer to validate the FlashBlade capacity sizing required to support SafeMode snapshots. Following is a sample workflow:

  1. Customer contacts Pure Support, upon which Pure Support contacts the Pure Storage account team to validate customer identity.
  2. Customer supplies the authorised usernames to Pure Storage, to generate and provide a unique identification PIN for each user.
  3. Pure Support performs a health check on FlashBlade and validates the capacity headroom available for SafeMode snapshots.
  4. Pure Support enables SafeMode and sets the eradication timeout. The default eradication timeout is 14 days. The eradication timeout can be modified upon request of authorised users.
  5. Pure Support enables SafeMode snapshot schedule.
  6. Customer validates the operations of SafeMode.

To make a change to an existing SafeMode policy, follow the high level procedures described below:

  1. Customer authorised user contacts Pure Support.
  2. Customer authorised user supplies name and PIN over the phone.
  3. Pure Support incorporates the requested changes to SafeMode.
  4. For requests to eradicate file systems or snapshots, Pure Support eradicates only the destroyed items.

Enabling SafeMode Snapshots via Pure Support

Authorised Users

Before contacting Pure Support, select at least two individuals who will be listed as SafeMode authorised users. These individuals will be provided with a PIN for safekeeping. Pure Support requests this PIN every time an authorised user calls and requests a change to the SafeMode configuration.

Only an authorised person with the correct PIN is allowed to make changes to the SafeMode configuration via Pure Support. This is the key layer of defence. SafeMode storage snapshots can be deleted if an authorised user with the correct PIN requests Pure Support to do so. Only these authorised personnel can work directly with Pure Technical Support to configure the feature, modify policy, or manually eradicate snapshots.

Health Check

Once the authorised user list has been provided, Pure Support performs a preactivation health check that involves checking the FlashBlade growth trend to ensure sufficient capacity to enable SafeMode. Note: Pure Support destroys any pre-existing snapshots before enabling SafeMode.

The following table presents the SafeMode sizing examples to assist readers with possible SafeMode storage consumptions.

SafeMode Enable

Pure Support will request the required data retention period. It is recommended to provide a value that allows adequate time to identify and suppress an attack.

Following are the SafeMode configurations that are performed by Pure Support.

Validating the operation of safemode

Once Pure Support has enabled SafeMode, validation is required.

The screenshot above displays a list of snapshots of the file system . There are 2 distinct snapshot groups presented here:

  1. Standard policy-based snapshots: These are created and managed by FlashBlade administrators. In the image, these snapshots are represented by the policy name.
  2. SafeMode snapshots: These snapshots are not managed by FlashBlade administrators. Creation and eradication of SafeMode snapshots are fully automated by SafeMode policy that is defined by the authorised designee. As observed, there is no policy name (-) assigned to these SafeMode snapshots.

When attempting to eradicate a SafeMode snapshot, the file system will be moved to a destroyed state.

The destroyed snapshot will be placed in a destroyed snapshot folder as highlighted below. In our test, eradication timeout for the destroyed snapshots was changed from the default value of 14 days to 7 days.

When attempting to eradicate (permanently deleting) a SafeMode snapshot, a warning message will be displayed. as shown below.

It is evident that under SafeMode, the administrators cannot manually remove a filesystem or snapshots from FlashBlade.

Simulating an attack

A sophisticated ransomware attack does not just encrypt production data. Perpetrators often target the backups to ensure organisations cannot recover. There are cases where the backup servers were compromised by ransomware attacks and became inaccessible. Other possible scenarios are when perpetrators and/or rogue admins gain access to the storage systems and attempt to delete the file systems and snapshots hosting backup data.

To demonstrate the effectiveness of SafeMode, we have assumed that access to both the Veeam console and FlashBlade interface has been gained. From the Veeam console, a ‘Delete from disk’ operation was performed on all backups that are stored on the FlashBlade NFS repository. This results in the deletion of the following:

  • Records about the selected backups from the Veeam Backup & Replication console and configuration database.
  • Selected backup files from the backup repository.

The following screenshot depicts the status of after all of its content is removed from the corresponding Veeam backup repository. Note that the used space displays 0TB as the file system is storing nothing.

The NFS file system that is used for the Veeam repository was then destroyed from the FlashBlade console.

Recovering with SafeMode

If other protection mechanisms are compromised, the FlashBlade storage becomes a solid final line of defence. As SafeMode snapshots are immutable and cannot be eradicated manually, the recovery process can be started, as below.

  1. Delete compromised/encrypted data.
  2. Reinstall and reconfigure Veeam backup server. (Only if Veeam server environment is compromised and inaccessible)
  3. Point backup software at data that is stored in the SafeMode snapshot, with assistance from Pure Support.
  4. Begin the recovery process at the speed of FlashBlade.

Recovery of FlashBlade destroyed File System

To recover a destroyed file system:

    1. Sign in to FlashBlade
    2. On the left navigation pane, click Storage > File Systems > Destroyed. Note that the view is expanded.


Review the file systems in the destroyed container and click Restore. Note that the destroyed file system moves back to the file system folder.

    1. Once the destroyed file system is recovered, it is visible in the file systems view.

Based on our simulated attack, restoring the file system is just the first step to recovery, as both the Veeam backup data and the File System (NFS) were destroyed. The next step is to revert the File System to a previous snapshot before the data was deleted/compromised.

Reverting the file system to a previous SafeMode snapshot requires the intervention of Pure Support. The FlashBlade administrators are not allowed to perform a filesystem rollback using GUI or CLI. The company designee needs to contact Pure Support to complete the restoration process.

Note: Purity does not allow snapshot restore by an administrator with SafeMode enabled. An error message is shown whenever Administrators attempt to revert a snapshot.

Restoring with Veeam

Let’s consider a few scenarios.
Scenario 1: The backup records in the Veeam configuration database were not destroyed.

  • Revert the SafeMode snapshot.
  • Restores from Veeam can immediately be started from the Veeam Console.

Scenario 2: The backup records in the Veeam configuration database were destroyed.

  • Revert the SafeMode snapshot.
  • Perform a Veeam rescan on the NFS backup repository.
  • Restores from Veeam can immediately be started from the Veeam Console.

Scenario 3: Veeam Backup & Replication server was destroyed

  • Revert the SafeMode snapshot.
  • Install VBR. NOTE: If the Veeam Configuration Backup is available, import the configuration backup. If backup files were deleted from the configuration backup before the last configuration backup occurred, a rescan may be required before VM restores can begin.
  • Add the FlashBlade NFS share as a Veeam NFS Backup Repository.
  • Perform a Veeam rescan on the NFS backup repository.
  • Restores from Veeam can immediately be started from the Veeam Console.

Adding a FlashBlade NFS Share to Veeam as a Backup Repository is covered in part one.

Simulating a restore

The simulated attack described above involved deleting the backup records from the Veeam configuration database, but not from the NFS backup repository. Therefore, the following steps are performed.

  • Revert the SafeMode snapshot.
  • Perform a Veeam rescan on the NFS backup repository.



After the rescan is complete, the restore points will reappear under Backups > Disk (Imported)

Veeam can now perform an Instant Recovery to immediately restore VMs by running them directly from the backup files stored on the reverted NFS backup repository.

The following screenshot shows the restored VMs that are available in the vSphere Client.

Additional Information

  • FlashBlade includes the SafeMode snapshots feature at no extra charge.
  • To reconfigure SafeMode after enabling it, please contact Pure Support.
  • Pure Support will only eradicate destroyed items.
  • Besides protecting snapshots, SafeMode also prevents the File Systems from being destroyed for a selected time period. Even if no snapshots are present, the share will be moved to the ‘Destroyed File Systems’ bucket for a defined time period before being destroyed.
  • SafeMode is a global setting, and once enabled, all file systems are protected.
  • SafeMode is part of a comprehensive security strategy, hence it should not be solely relied upon to prevent or thwart a ransomware attack.

Caveats

Following are a few caveats surrounding SafeMode that customers should be aware of:

  • FlashBlade administrators are not allowed to perform a filesystem rollback using GUI or CLI. The authorised designee will need to contact Pure Support to revert any SafeMode Snapshots.
  • SafeMode snapshot retention cannot exceed 400 days.
  • SafeMode does not work with replication (either as the source or destination) — customers should not turn on SafeMode on the FlashBlade array with file system replication running on. This limitation will be lifted in Purity //FB 3.1.
  • SafeMode does not have WORM (Write Once Read Many) capabilities

Security

It is important to consider the security of the device as well. If the storage device can be easily taken over, then the software layer providing the storage immutability may not be of much help. Fortunately, in the case of Pure Store SafeMode, even if the management GUI was accessed, the rogue administrator cannot factory reset/wipe the storage and manipulate the storage snapshots.

To further increase security, Multi-Factor Authentication (MFA) will be introduced in an upcoming Purity release (v6).

Veeam Backup Configuration

While not required, it is recommended to configure the Veeam Backup & Replication (VBR) Configuration Backup in the same NFS Backup Repository. This simplifies the VBR server recovery when the server is corrupted or unavailable. A new VBR can be deployed and the Veeam configuration backup can be recovered onto the server.

Summary

The Pure Storage FlashBlade is an on-premises backup storage that provides immutable storage for data such as Veeam backups. SafeMode configuration provides an additional layer of protection against ransomware/cyber-attack.

In the next blog post, we will discuss how Object Storage presented from the FlashBlade will be used as a backup repository target for Veeam Backup for Office 365.

Disclaimer: For advice regarding retention periods and ransomware security, please speak to your local Pure Storage SE.

Appendix

Standard Snapshot restoration of Veeam backups (GUI)

Standard Snapshot restoration of Veeam backups (GUI)
To restore a non-SafeMode snapshot:

    1. To expand the view and detail available file system snapshots, select the file system.
    2. Based on when the incident occurred, select the appropriate snapshot that is available for the file system. Note that the time and date stamps are included in the snapshot name.

    1. Click Restore that is next to the appropriate snapshot.

  • A file system can be restored or rolled back from the most recent snapshot of that file system using the purefs copy command.
  • To restore a file system from a previous snapshot, include the source snapshot name that indicates the restore point (SOURCE) and the target file system name (TARGET).
  • To indicate that the file system is being restored from the specified snapshot, make sure to include the –overwrite option
  • To discard the data on the existing target file system (but not existing data in the specified source snapshot) use the –discard-non-snapshotted-data option.
  • Similar to the file system and snapshot eradication, file system rollback is categorised as a privileged operation under SafeMode. FlashBlade administrators are not allowed to perform a filesystem rollback using GUI or CLI. The company designee will need to contact Pure Support to complete the below procedure.

Standard Snapshot restoration of Veeam backups (CLI)

Following are commands that are used to restore the file system with snapshot via Purity FB CLI.

The FlashBlade administrators can use either CLI or GUI to perform file system restoration as if the array is not operating under SafeMode. Customers will need to contact Pure Support to perform the following steps in FlashBlade under SafeMode.

After a successful file system restore, one can observe that the used capacity on file system is increased to 114.51GB. This indicates that the previously removed objects are restored to the file system.

The next step is to proceed to the Veeam console to complete the recovery of affected virtual machines.

Tags: Backup, Backup Management, Data Centre Backup, flashdrive, Ransomware, safemode, Security, Storage Backup, Veeam

Featured

Subscribe to our blog