SwiftStack, vendor of multi-cloud data storage and management offerings, announced on Tuesday a new customer-proven edge-to-core-to-cloud solution that supports large-scale Artificial Intelligence/Machine and Deep Learning (AI/ML/DL) workflows.
SwiftStack has recently deployed the solution stack in two autonomous vehicle use cases. The solution includes an integration of SwiftStack with Valohai’s deep learning platform-as-a-service to provide machine orchestration, version control, and AI/ML pipeline management.
SwiftStack’s AI/ML solution delivers massive storage parallelism and throughput, needed for ingest, training and inferencing; a scale-out global namespace for access to data whether on-premises or in one or more clouds; data services such as tagging, search and metadata management to support AI/ML workflows; and Kubernetes and TensorFlow support. Additionally, the solution extends to public clouds to take advantage of cloud-bursting and economies of scale, while data is secured on-premises.
SwiftStack is highly-parallelized storage ideal for AI/ML/DL pipelines and workflows. SwiftStack enables these multi-stage workflows where data is continuously ingested, transformed, harnessed to develop and train new models, and then used to derive inferences. Each of these stages have distinct storage requirements for bandwidth, mixed read/write handling for both small and large files, scale, concurrency, meta-data labeling.
“Infrastructure challenges are the primary inhibitor for broader adoption of AI/ML workflows,” said Amita Potnis, Research Director at IDC’s Infrastructure Systems, Platforms and Technologies Group. “SwiftStack’s multi-cloud data management solution is the first of its kind in the industry and effectively handles storage I/O challenges faced by edge-to-core-to-cloud, large-scale AI/ML data pipelines.”
“With this integration users can quickly adopt an AI/ML platform, easily use multi-cloud workflows and frameworks, and scale to petabytes of storage and hundreds of gigabytes of bandwidth,” said Eero Laaksonen, CEO of Valohai. “This gives them the ability to create a deep learning infrastructure, even at an enormous scale, in a fraction of the time.”
“The SwiftStack solution accelerates data pipelines, eliminates storage silos, and enables multi-cloud workflows, thus delivering faster business outcomes,” said Jason Blum, CTO at GPL Technologies, an NVIDIA and SwiftStack elite partner. “SwiftStack provides us with the flexibility, technology leadership and breakthrough economics to build tailored solutions for our customers.”
GPL Technologies has created multiple ways to implement the solution, with NVIDIA DGX-1 GPU server(s), NVIDIA GPU Cloud, and other system hardware.
The emergence of AI/ML workloads has created a new set of challenges for organizations, while the rise of GPU computing is enabling massive parallelism and several petaflops (floating-point operations per second) of computational power. It has been difficult for these environments to build and manage a storage infrastructure that provides appropriate scale and concurrent performance.
“Traditional storage architectures are not designed for these new distributed workloads and fall short of performance, scale, and value, so storage services needed to be rethought to accommodate AI/ML pipelines,” said Shailesh Manjrekar, head of product and AI/ML solutions marketing at SwiftStack. “Successful customer deployments are proving that we have created a solution that enables them to put their GPU cycles to work, bring cloud and AI to the data and scale affordably as the workflow grows.”
