Hammerspace Hyperscale NAS Targets Fast File Services For AI On GPU Servers

Hammerspace’s Hyperscale NAS technology sits on an industry-standard server between a customer’s GPU-based front-end servers and whatever storage they use from hard drive to NVMe flash as a way to scale file services as needed, particularly for massive data stores associated with AI workloads.

Hammerspace, which develops global data file systems that work across data centers and large public clouds, Wednesday introduced its new Hyperscale NAS as a way to provide high-performance, highly-scalable file services to burgeoning AI workloads.

Hammerspace’s Hyperscale NAS is nothing less than a new storage architecture, said Eric Bassier, senior director of solutions marketing for the Los Altos, Calif.-based company.

“It is effectively combining the properties of an HPC file system with an enterprise NAS system for the first time ever,” Bassier told CRN. “It is the best architecture for general AI training, GPU computing, and other forms of unstructured data process.”

There is a broad-based adoption of different forms of GPU processing going on in every major industry, and one of the AI infrastructure challenges is around assembling and sourcing data, Bassier said.

“Hammerspace can help with that, as well as with managing these large sets of unstructured data that are typically distributed across multiple sites, and maybe stuck in different storage silos,” he said. “What we've really learned in the last year working with some large customers is there is an unmet need for a storage architecture that can provide the extreme parallel performance to feed GPUs. And that's what we're introducing now.”

When it comes to high-performance storage, enterprises can generally use one of two types, scale-out NAS and HPC (high performance computing) systems, Bassier said.

“Neither one meets all of their needs,” he said. “Scale out NAS, although it's widely deployed in the enterprise and meets all the enterprise feature sets, it simply can't provide the throughput and low latency performance to feed these GPU clusters. And while HPC file systems have the performance, they're incredibly complex to manage, and require proprietary software and unique networking requirements. That’s prevented HPC parallel file systems from really being a viable option in the enterprise.”

To power enterprise AI initiatives in GPU computing requires a new architecture, which Hammerspace is introducing with its Hyperscale NAS, which combines a parallel file system with enterprise NAS, Bassier said.

The fact that hyper-tuned, purpose-built supercomputer parallel file systems like Lustre, GPFS, and Weka exist is kind of proof that enterprise NAS, the traditional enterprise file environments, have not been able to address performance needs, said Hammerspace CEO David Flynn (pictured).

“Their very existence is proof that there is a an opportunity and a missing piece,” he said. “How do we get that level of performance, but with the ease of use, with the reliability, with the enterprise rich feature set of NAS because frankly, it won't scale if it requires the kind of care and feeding that those exotic file systems take in the world of supercomputing. You have to have engineers on staff. You have to tweak and tune them. They fall apart and break easily.”

Hammerspace Hyperscale NAS sits on a separate standard server between GPU-based servers running standard Linux and the customers’ choice of storage from hard drives to NVMe flash storage, Bassier said. Hyperscale NAS leverages the NFS 4.2 client in the Linux kernel to do a couple of important functions including separate out the metadata and the data path and establish a direct data path between the Linux clients and the underlying storage.

“Any client can directly read and write data to any volume on any one of those storage systems,” he said. “Everything is standards-based. Everything is plug and play. In some ways, it looks like a parallel file system architecture with the benefits of extreme parallel performance and being able to scale performance linearly just by adding more GPU nodes or adding more storage nodes. But unlike those parallel file systems, everything's standards-based. There's no proprietary software. And you have your choice of any type of networking.”

No other storage vendors have built a hyperscale NAS system similar to what Hammerspace has done because building file systems is incredibly difficult, Flynn said.

“There's a risk in using new protocols,” he said. “If you don't have somebody like our [Chief Technology Officer] Trond Myklebust, who's the kernel maintainer of this, it's rather impossible to do what we do. Folks couldn't do what we did because we basically we acquired the talent that is very limited in the industry. There's only one Linux, and there's only one maintainer of this part of Linux. And he's our CTO.”

What Hyperscale NAS Means To Channel

Hammerspace Hyperscale NAS gives channel partners an all-new architecture to meet the needs of generative AI training and GPU computing, Bassier said.

“Hammerspace is providing that software, but everything else can be the hardware of their choice,” he said. “And that is Hyperscale NAS in a nutshell, for the first time combining the performance of HPC file systems with standards-based simplicity and the reliability, availability, and serviceability of enterprise NAS.”

Hammerspace currently has hundreds of channel partners engaged with many enterprise and web-scale accounts, Flynn said.

“For this technology, we have found that some of the cloud-focused partners, the cloud-first partners, are better than the ones that are focused on on-prem infrastructure,” he said. “The reason for that is the cloud-first partners understand the challenges of getting file data into the cloud. You need folks who are actually trying to solve hybrid cloud or multi-data center things which require data orchestration, or partners who are trying to solve building hyper-scalable infrastructure HPC.”

Hammerspace has proven to be a good partner to the channel, said Dave Donald, CEO of Keeper Technology, an Ashburn, Va.-based solution provider which has worked with the company for over two years.

Donald told CRN that Hyperscale NAS is a big deal for his company’s customer base.

“It really demonstrates operations at scale,” he said. “This is a key concern for our customer base, including the Depart of Defense and intel agencies. Having scalable front-end processing with NFS storage is exactly what our customers need. And none of our customers start out at the hyperscale level, but it’s important to know they can grow into it.”

Donald said one very large high data velocity client of his company is working with 30 petabytes to 40 petabytes of data in billions of files, and can really use Hammerspace Hyperscale NAS.

“Such customers are using Quantum StorNext now,” he said. “But Hyperscale NAS is a differentiator when they want to consolidate other NAS systems, and scale to the cloud. Now they can do so without migrating from StorNext. Hyperscale NAS can sit alongside Quantum StorNext. No vendor likes to see someone in their space. But for clients, it’s important to have multi-vendor implementations.”

Hammerspace, which was founded in 2018, has yet to be cash flow positive, and that is by design and now only temporary, Flynn said.

“Technically speaking, if we wanted to grow organically and stay cashflow positive, we could,” he said. “But it would come at the cost of the rate at which we can capture market share. So it's always a balance between going into the negative to hire and ramp up the sales team in advance or growing organically. If we wanted to operate cashflow positive, we could today, but that would mean we don't put as many new resources in play as we can when we have access to investor capital. … By the end of the year, where we couldn't go negative if we wanted to, we're going to be having so much cash flow that it won't require much more.”