Transferring larger files between storage providers can be painful. Traditional tools copy one file at a time and do not parallelize data transfers. Users are frustrated with long wait times and satisfying application use cases become a problem. Storage administrators want to see arrays filling data pipes and hitting vendor throughput claims.
M-Stream, a new feature of the SME Enterprise File Fabric, unlocks the potential in existing Distributed File Systems, Network Attached Storage (NAS) and S3 & Swift API compatible object storage systems.
The Copy Throughput Dilemma
Copying files between storage providers has two inherent problems.
First is the data flows through the device that initiated the copy, creating a bottleneck and tying up the client device. Below is an example of a typical transfer from a NAS Filer to an S3 Compatible Object Store.
The second problem is copy operations start at the beginning of a file and proceeds sequentially to the end of the file. This approach limits the transfer speed for large files. Modern storage systems contain hundreds of spindles each capable of 50MB/s of throughput. Reading files sequentially limits the client to working with a single drive at a time, even if the file is chunked across numerous drives.
Announcing M-Stream: Accelerated Data Transfers
To alleviate this problem built out a product feature that we call M-Stream, which we are today releasing in beta to a limited set of design partners and which will shortly be made publicly available as part of our general File Fabric product.
M-Stream breaks files into chunks and transfers these chunks in parallel. This parallel transfer is enabled in both uploading and downloading of file/object data.
M-Stream Accelerates Latent Storage
M-Stream also works over latent Internet connections to S3-compatible clouds enabling high speed transfers of data cross-country or cross-continent.
M-Stream for Windows & Mac OS
M-Stream is also built into the Storage Made Easy clients for Windows and Mac OS, enabling accelerated transfers to desktop clients in the office or on the road.
Breaking Barriers for Media, Healthcare, and Genomics
4K Cameras, MRI imaging and 30X full human genome files break the definition of large file data. As 100GB files become the norm, new technology is needed to seamlessly enable users to take advantage of object storage and all flash arrays.
M-Stream enables storage systems to fully saturate network pipes and removes the client “in-the-middle” bottleneck. M-Stream is a feature of the Enterprise File Fabric and uses the native capabilities of CIFS, NFS, S3, Swift API, and other filesystem protocols without any the requirement for software or agents on the storage servers.
More Information
The below video shows a demonstration of M-Stream technology during the OpenStack Sydney Summit
by 
Douglas Soltesz
Latest posts by Douglas Soltesz (see all)
- LucidLink Technology Preview Setup Guide - September 6, 2021
- The features and benefits of using LucidLink with the Enterprise File Fabric - September 2, 2021
