5 key considerations when designing hyper-converged infrastructure

What is hyper-converged?

When you ask vendors what hyper-converged means you will get a different answer from each and often a different answer from the same vendor as their solution offerings change. As vendors aim to differentiate in the market, they will try to make artificial boundaries to exclude their competition from being evaluated in the same class.

I consider traditional architecture to be compute and storage managed separately. This then gave rise to converged architecture, where storage and compute is separate, but has the ability to be orchestrated together to deliver the illusion of a single holistic platform. In my opinion, hyper-converged is where storage is delivered from the same distributed virtualised physical platform as the compute; this leads to the emergent concept of a redundant array of independent nodes (RAIN).

The minimum components that a hyper-converged platform will have is more than one physical node containing hosting software for pooling and sharing of:

• CPU Resources
• Memory Resources
• Networking Resources
• Storage Resources

There are often other services that tightly integrate with these base services, such as unified management, orchestration, backup, security, and replication.

Most solutions consist of a hypervisor and a software-defined storage (SDS) offering, for example, VMware and VSAN, Hyper-V and Storage Spaces Direct, VMware and OmniStack, KVM and DRBD.

Problems at the Start

There were many teething problems with the early products, and it was common to have data corruption, split brain issues (where a cluster would split in two and both halves would be active), and general stability issues. Performance of the early solutions was less than ideal, and these solutions were used predominantly in test/dev or SMB environments. Over time, these issues were addressed and more players entered the market. Now SDS is taking a serious slice of the shared storage market.

The problems with performance are disappearing with the rise of flash-based storage, and with higher bandwidth networks, other problems have been addressed, with better error correction, better shared file systems and smarter cluster software.

The initial problems have left some with hesitation in taking on modern hyper-converged platforms, however, just because the original problems have been largely addressed, does not mean that all is clear skies ahead.

Current solution considerations

1. Flash storage

Flash storage has removed much of the performance problems with SDS, and the software has got better at detecting problems. However, in many solutions, there are also greater demands placed on the media, which causes stress. In a traditional SAN, the vendor has tight control over the type of media, the endurance, performance, brand and firmware. In a SDS solution there is often little to no control. It has become very important for reliable operation to review the SDS vendor’s Hardware Compatibility guides – but, don’t assume that because an SSD is supported by one vendor it will work with another. Although on occasion, we are finding that a SDS provider will certify a server, however there is no way to get a certified configuration. (This has been common with SSD’s lately, with rapid changes from SSD providers, and slow certification processes).

Things to watch for when selecting flash media:

• Capacity/Performance
• Endurance
• Certification with SDS

2. Usable capacity

Calculating how much capacity will be delivered can be a challenge. It can be difficult to determine what you will end up with and these numbers can be almost impossible to compare between vendors.

The storage efficiency of solutions can be much lower than expected and if designed incorrectly, the storage efficiency can be embarrassing. When designing hyper-converged solutions more effort is required on the storage side, as people who are contemplating this are often well versed in the hypervisor side of the equation. When looking at server resources it is important to know how much CPU and memory will be consumed on each node. Some vendors minimise this with dedicated cards, others have a small footprint, but generally you don’t get something for nothing. If features like inline deduplication, compression, or encryption are required these will chew up resources.

When designing for hypervisor capacity it is usual to design for N+1 fault tolerance – most SDS designs are done to an N+2 design, or RAIN+RAID. With all SDS solutions there will be a minimum number of nodes to achieve a level of redundancy, but most also have variable efficiency depending on number of nodes.

Another thing to watch out for is unusable space within the nodes – this can be cache disks required, RAID overhead within a node, slack space, reserved space for flash rebuild etc. – and these may be great to have. But, if the system is losing 50% of a nodes capacity in this and then loosing another 70% of capacity across nodes – 100TB of flash disks has capacity for 15TB of data! More of a smaller disk may yield a better result.

Lastly, each vendor will have a “you shouldn’t run the system at more than xx%”. This should be used to work out the actual usable capacity of the system, not the plate value.

Some things to be cognizant of when doing initial design are:

• Required fault tolerance level
• Overhead within a node
• Number of nodes needed to achieve desired storage efficiency
• Amount of storage the SDS vendor requires to be left free
• Amount of CPU and Memory dedicated to storage

3. Network considerations

The network in a hyper-converged solution is critical. As per a traditional virtualisation stack it carries VM networks and storage traffic, but on top of this, the network also carries all of the back end SAN traffic. What would be one write to a SAN, turns into multiple transactions with nodes replicating, moving, checking and redirecting. A 10GB/s network won’t be fast enough to write 1GB/s of data, and new deployments are now using 100Gb/s networks.

It is important to note that the networking considerations for storage networking, including high port buffers, low contention, low latency, and of course high bandwidth are requirements for hyper-converged systems. Some SDS platforms have routable connections between nodes in a cluster, allowing clusters to easily span data centres. Others will not allow this and require L2 connections or separate clusters per data centre. This is important when looking at metro storage cluster type solutions.

4. How does it fail?

It is important to note not only how a solution works, but how it will fail. We may design for N+2 redundancy in a solution, but what constitutes a node failing?

• In Storage Spaces Direct; a failed disk takes the node out of service
• In VSAN; a failed disk (cache or capacity) takes the disk group out of service
• SimpliVity uses RAID on the nodes
• Nutanix treats all disks as individual storage elements
• StoreVirtual uses RAID on the nodes

In all cases if the hypervisor on the node fails (server failure, SW Crash, lightning strike) the SDS node will also fail. If a cluster loses more than the allowed number of nodes the general response is to take all storage offline, this prevents corruption of data, and split-brain situations. Many solutions can have nodes reside in multiple geographic locations to provide some additional protection, the design of witness nodes etc. is crucial in this case.

5. How do you remove nodes?

When you add a new system to an environment it is important to look at how nodes are added and how you grow the solution. This is covered well by most hyper-converged systems. It is also important to note how the vendors handle swapping a node to replace a failed node or during planned lifecycle refresh and at the end of the solution’s life. Can the solution be downsized? Make sure you know how the storage capacity and performance is effected; removing the 6th VSAN Node reduces storage by more than 50%. Removing or swapping nodes can cause large amounts of IO and can leave the storage in a more vulnerable state for days or weeks of restripe time.

Who is best suited to use?

From what I see, there is a progression from traditional to virtualised storage. At the moment hyper-converged solutions will be most suited to highly virtualised environments with one hypervisor. The scope is increasing with recent changes allowing more diverse environments and complex topologies.

A large proportion of current hyper-converged platforms are 2-4 node ROBO type deployments. Although, this is starting to move to larger deployments.

SDS is being promoted by storage vendors and software vendors such as VMWare, Microsoft, Many Linux Distro’s and application vendors. The future of storage is looking like a battle between hyper-converged platforms, and may involve moving nodes between disparate vendors’ platforms. In this case we may see a rise in composable platforms (such as HPE’s Synergy platform) that allows resources to be dynamically moved between platforms.

To learn more about hyper-converged infrastructure, feel free to reach out to me on LinkedIn.