Enterprise File Fabric™
The Enterprise Storage Podcast is back with a new season of Storage Masters!
For the first episode of season 2, Andres Rodriguez, CTO at Nasuni, and Jim Liddle, CEO at SME, discuss the challenges of adapting to a new data ecosystem in which companies want REST API access to data, but there’s still a lot of on-premises end users storage that require more traditional protocols.
by 
The Amazon S3 API has become the de-facto standard for object storage API access. As such, many software products adopt the S3 API first. The Storage Made Easy File Fabric provides an S3 compatible API endpoint for any storage backend. This means that non-S3 storage providers such as Dropbox, Azure, Google Drive, OneDrive, Sharepoint and others can now be accessed via the S3 API.
The Amazon Web Services (AWS) Command Line Interface (CLI) is a command line tool for accessing AWS compatible services. AWS CLI is one of the most feature rich S3 tools publicly available (Free!!!) .
This article will outline the setup AWS CLI with the SME File Fabric and non-S3 compatible storage providers.
Continue reading “How to use the AWS CLI with almost any Storage”
by Over the past week I’ve been busy creating Python scripts to automate certain processes here in the SME DevOps team.
One of my main objectives has been to create a script that interacts with the SME API to check the health of specific functions, for this task I decided to opt for the language of ‘DevOps’ Python. Whilst I do not profess to be a Python expert in any way I felt that this is a language is one that I have become accustomed too and am able to ‘get by’ with. The nice thing with Python are the extensive libraries that are readily available, and the fact that they are fairly easy to use (of course, only if you have some background experience with Python).
by Hi, I’m James, a Senior Developer at Storage Made Easy. As a developer, I often look for ways to reduce coupling in the tools and technologies that we use, and I see this from the low level code that we write, to the higher-level architectural choices we make.
Continue reading “My Favourite Feature: Cloud S3”
by In the past we have written about Amazon S3 and how, aas the 100 pound gorilla, of the Cloud Storage world, it’s S3 API has become almost a de facto interface for developers. This is one of the reasons that we originally added an S3 API protocol adaptor to our service.
Many start up’s, small businesses and even enterprises choose initially to use S3 for storage. This can be fine initially, however, when the volume increases the monthly bill can become an OPEX issue and small companies (and Enterprise) are looking for ways to slash their costs in any way they can. Aside from this other companies have stringent issues about where data is stored (for clarification, Amazon S3 is PCI DSS 2.0 compliant, SAS 70 Type II certified, and VPV ISO 27001 certified) or wish to store it within their own data centre or site.
As SMEStorage supports over 35 Clouds and SaaS services, you could very easily turn Google Docs, Box, Windows Azure or DropBox into an S3 Platform, or you could just add your own NetGear or PogoPlug appliance, or other private storage implementation.
The SME hosted service, and Cloud Appliance, provides the ability access to any Saas or cloud storage mapped to your account via multiple protocols. These include FTP, WebDav and also S3. These work even if the backend provider does not support the protocol natively. The SME protocol adaptors will do the protocol translation to the native storage provider protocol. One of the benefits of this that the users don’t need any special software to be able to access the Clouds. They can use any FTP, or WebDav client, or in the case of S3 any S3 client or code in which the host endpoint can be changed.
To demonstrate compatibility with the S3 API and tools we will now look at how to use AWS s3curl with a smestorage account.
To use s3 curl you will need to modify s3curl.pl and change the end point to
‘s3.smetorage.com’ (US Server) or ‘s3eu.storagemadeeasy.com‘ (EU Server) e.g my @endpoints = ( ‘s3.storagemadeeasy.com‘);
Your id is your smestorage account user name and you can obtain your secret key by logging into SMEStorage.com going to “My Dashboard” (from the sidebar) and copying the API key from the“Tech Info” section where the “API secret Key” resides.
Now you are all set to use s3curl. For example to list all the buckets you can use
./s3curl.pl –id smestorageusername –key API secret key http://s3.storagemadeeasy.com
For s3curl command line options please see the README file that is part of the s3curl package. Also note that the secure way to use s3curl is to use the .s3curl file in your home directory to pass the id and and key.
by We’ve added bit.ly integration for the generation of short file links for Cloud files mapped to a users SME account.
The long file link is a secure file link that we generate, but we have always had the option to shorten this with TinyURL. This will still be the default option, but for users who want to take advantage of bit.ly for tracking SEO or statistics against links, they can by entering their bit.ly key from within the App integrations menu on the right sidebar after login to the website. After doing this bit.ly will be used to generate short links from any access client.
Bit.ly integration works on free, personal or business accounts.
by 
We are really pleased to announce that we have added OpenStack Swift object Storage support to the SME Open Cloud Platform. Swift is a sub project of OpenStack and provides a highly scalable redundant unstructured data store. Swift is 5 separate services, object, container, account, auth and proxy. Although each of these can be scaled separately, in practice they run together.
Never heard of Swift? it’s the underlying distributable object store that supports RackSpace Cloud Files. It’s akin to Amazon’s S3 implementation but unlike implementations such as Eucalyptus, which clone S3 API’s, but are not sponsored by Amazon, openStack and Swift has RackSpace firmly onboard, and have proven scale.
As Swift is used by Rackspace Cloud Files. Swift RackSpace claim it is production-ready code that is scalable to massive levels (100-petabyte clusters and 100000 requests per second). Swift sacrifices C for A and P from a CAP theorem perspective. Although most operations happen synchronously consistency is sacrificed in failure scenarios.
From our perspective we have seen ISP’s and larger SMB users of our on-premise Cloud Gateway appliance expressing interest in SME supporting this, and we supply this as VMWARE Appliance (OR XEN, KVM) or as a dedicated hardware appliance for smaller companies who wish to embrace their own private Cloud infrastructure.
As with our S3 API endpoint support SME will overlay a more traditional file store on top of Swift layered with the business functionality we provide in our Cloud File Server, which includes virtual drives and clients for Mac, Windows and Linux, and feature rich mobile clients for iPad, iPhone, Android and BlackBerry, as well as value added features to Swift such as Webdav and FTP support.
Setting up Swift with SME is easy. First you need to add a new Cloud Provider and then the Cloud Wizard will be invoked. The first step is to enter your OpenStack details:
When entering the endpoint URL you should be sure to include the Port. An example URL is: http://<IP Address>:11000/v1.0.
Next you will need to choose which containers you want to work with and which should be the default container for any uploads to smart folders.
Once you have done this you will be ready to start the meta-sync which pulls in and caches all the information about containers and files.
If you have any issues connecting please refer to this advanced post on using SME with OpenStack 1.60 and SWAuth.
Once complete you will be able to access/amange your OpenStack files from the SME Web clients, as well as using a Cloud Drive on Windows, Mac or Linux, and mobile clients for Android, iOS, and BlackBerry, and the plethora of other tools and clients that SME provides. We’v e posted some screenshots below of this.
Web File Manager
iOS OpenStack
Android OpenStack
Firefox Plug-In OpenStack
Chrome OpenStack Plug-In
Mac Cloud Drive OpenStack
The OpenStack Swift API’s also get embedded for use within our own feature rich multi-cloud API framework in which we add many business driven features.. You can find details about that on our developer page
by I read a post by Gladinet yesterday that talked about why, in their opinion, an S3 compatible API is not a good idea, and to be honest I was surprised at their viewpoint. If an S3 compatible API is such a bad idea, why do Eucalyptus offer it ? Why do Mezeo ? Why do Google Storage ? I’ll tell you why, because right now Amazon S3 is the big gorilla in Cloud Storage and to make it easy for developers, and tool vendors, to move to/use their platforms they are offering the nearest thing there is to a standard (apart from WebDav….and I’ll come back to that), which is an S3 compatible interface, or way into their storage clouds.
If you extend that thinking further you can see why it would be useful to users, developers, and Organisations, to use an S3 API above other Storage Clouds, that don’t currently have it. For example using an S3 endpoint with RackSpace or Google Docs could solve some key integration problems if you already use S3 and want to manage files against these clouds.
The issue really is about standards, and the lack of standard API’s in Cloud. There is an old, but still relevant ,post from Gartner on this that I tend to agree with. Having standardised API’s from day one is not good because it stifles innovation, however the basic object cloud storage is pretty basic in terms of an API and for lots of users, developers, and companies, being able to standardise around the dominant API, has in certain circumstances, many benefits.
This brings me back to WebDav. Dav was the first real (and still the only) standard for working with files over http. I’m amazed by how many vendors don’t support it as an entry point to their cloud ie. Google Docs, Amazon S3, RackSpace Cloud Files, DropBox etc etc. Someone made a comment to me recently that WebDav “was so 20th century”. It made me smile as WebDav is still one of the most implemented integration points for Applications. Consider the Apple App Store. It’s two years old and there are a myriad of Applications that support WebDav and also have to support the clouds that don’t support WebDav, and if your cloud is not on the list, then, well it’s tough ! Need a more recent example? The new Nokia N8 Smartphone. It supports WebDav as a native client and users have already picked up that they can use our service to access their cloud data on the device. When we launched our ubiquitous WebDav over any cloud (whether it support it natively or not), CloudDav interface, we did it because we had the same frustrations. We launched this on August 25th. In only 3 months it is responsible for 30% of our traffic and it has lead to a huge increase in our user base ! That is an amazing statistic and shows that users really want standardisation when it enables them to access their data the way they want to, and of course this is our raison d’etre 😉
Interestingly, in the past Gladinet also had a positive point of view about the S3 API, at least in earlier blog posts, so I’m not sure what lead to their change of heart, but one thing is for sure, I expect more storage companies, not less, to continue issuing compatible S3 endpoints as entry points into their storage clouds.
by We’re happy to roll out support for another API that is becoming ubiquitous. In this case it is the Amazon S3 API. This is already used in solutions such as Eucalyptus Walrus, and we have now also added it as entry point for files accessed through our Cloud Gateway.
What does this mean ? It means you can use the S3 compatible API to access your files on Google Docs, or Google Storage, or RackSpace Cloud Files, or MIcrosoft SkyDrive, or any of the many other Clouds that we support.
For those interested, moving from a file based structure to a flat file object structure was achieved using the depth first algorithm within the implementation mapping to our API.
Lets look at a practical implementation of this using a tool that supports S3 compatible endpoints, CloudBerry explorer. We’ll use this to access our Google Docs, although we could have chosen DropBox, or any other number of clouds to access.
After launching the free version of CloudBerry explorer Choose File-Walrus Accounts
Choose to add an account and fill in the endpoint as ‘s3.storagemadeeasy.com’ and the QueryID as your SME username. You can obtain the API key from logging into the SME website and going to “My Account” from the sidebar and copying the API key from where it says “API secret Key”
Now you can choose to access the service using the CloudBerry exlorer:
and you can browse, download, upload files:
You can find out more on the SME Wiki
by We’ve recently been doing some integration work with a large engineering company that has been focused on integrating multiple clouds with their Portal framework which in this case is Ext JS. Ext (pronounced “Eee Eks Tee”) is a JavaScript library for building interactive web applications using techniques such as Ajax, DHTML and DOM scripting. It has gained quite a following and is a popular choice for a Portal infrastructure.
The main advantage of that ExtJS has over other frameworks is its component-oriented model which leads to a consistent and easy to use API to build widget-based interfaces. It also now features integrated touch sensitive web apps using HTML5 which makes it an even stronger Ajax tools and Portal framework
The integration that we have done for the engineering company is going to be key to enabling data access for storage clouds when users sign into the Portal. Clouds such as S3, and internal storage clouds, will be mapped into the on-premise SMEStorage Cloud Gateway appliance. Authentication is being handled by integration with an internal identity management server so that single sign-on can be accomplished.
The real power of the on-premise appliance and the rich API is that not only can users use the Portal to access files, they can also use the white label version of the Windows / Mac / Linux desktop tools , and also the Android, BlackBerry, iPhone, IPad mobile clients.
If you are interested in integrating ExtJS with an on-premise Cloud Gateway then please contact us
by **This application is now deprecated
The SME multi-cloud API has been used in a variety of tools. It is what is used in our iPad/iPhone, BlackBerry, Windows, Linux and Mac clients. From a language perspective this means Objective C, Java (J2ME), .net, Delphi, PERL, Ruby and of course HTML and PHP. The API is very rich and targetting this API means you can address over 10 storage clouds by binding to one API.
To see how easy it was to use for other users we gave the API to a company and asked them to create an Adobe Air mockup that used the API. What they came back with in a short space of time was impressive.
by We have a new version of our Multi-Cloud API Manual with added functionality. You can download the manual on our Google Code site.
SMEStorage Multi-Cloud API Manual_0_7_3 Revised with example
by 