Posted on

Building an Audio Streaming or Video Streaming service with the File Fabric Multi-Cloud API

Our CEO recently published a blog post on how the Storage Made Easy multicloud API can be used to build streaming audio and video Apps that unifies audio and video content from disparate clouds into a single namespace which could be used as part of an audio or video streaming service. The post provides links for downloads of the Apps built (also contained at the foot of this post).

You can check out the associated video below.

You can download the Audio streaming App here
You can download the video streaming App here

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

Step-by-step guide to authenticate with the Enterprise File Fabric API using Python

Over the past week I’ve been busy creating Python scripts to automate certain processes here in the SME DevOps team.

One of my main objectives has been to create a script that interacts with the SME API to check the health of specific functions, for this task I decided to opt for the language of ‘DevOps’ Python. Whilst I do not profess to be a Python expert in any way I felt that this is a language is one that I have become accustomed too and am able to ‘get by’ with. The nice thing with Python are the extensive libraries that are readily available, and the fact that they are fairly easy to use (of course, only if you have some background experience with Python).

Continue reading “Step-by-step guide to authenticate with the Enterprise File Fabric API using Python”

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

Google Docs upload currently not working

Our Cloud monitor has noticed an issue with Google Docs uploading over the resumable upload API that we use to upload files to Google Docs. Currently for any upload Google Docs is returning a response of:

<errors xmlns=’http://schemas.google.com/g/2005′><error><domain>GData</domain><code>InvalidEntryException</code><internalReason>Sorry, there was an error saving the file. Please try again.</internalReason></error></errors>

We’ve reported the issue to Google and are waiting for a response. We will post here when Google resolve the issue.

Update: Google have responded:

The thread can be found here.

Update: Google have now resolved the issue

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

Achieving S3 API compatibility with OpenStack Swift using SME

One of the things that is close to our hearts at SMEStorage is interoperability and, wherever possible, enabling open solutions. As outlined in our previous post, we recently added support for OpenStack Swift object storage. Of course there are other Object Storage frameworks, such as Amazon S3, but this is closed source, unlike Swift. There are however open source implementations of the Amazon S3 API’s including Walrus from Eucalyptus. There are also many tools, solutions, exiting code and frameworks that use S3 compatible API’s, and no doubt there will be occassions when it would be useful to enable users to connect to SWift using Amazon S3 API’s.

As SMEStorage has a compatible S3 API abstraction it turns out that is very easy. Let’s demonstrate how easy below:


After launching the free version of CloudBerry explorer Choose File-Walrus Accounts, and then choose to add an account.

The endpoint for the s3 compatible interface with SMEStorage is s3.storagemadeeasy.com. The username is the SMEStorage Account username, and the secret key can be found for each user on the “My DashBoard” page under a section called “Tech Info“.

Once we have entered the details we can test the connection:

and finally we can connect over the S3 API to work with our OpenStack Swift files:

As you can see this was easy and takes less than a couple of minutes to access and work with OpenStack Swift files using S3. We expect it to be very useful for ISP’s and SMB’s who want to deploy their own private cloud solution using OpenStack and enables interoperability and prevents lock-in.

Update: Note that in the Swift Diablo release that OpenStack also supports the ability to turn on Amazon S3 API’s.

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

Why an S3 compatible API ? A rebuttal

I read a post by Gladinet yesterday that talked about why, in their opinion, an S3 compatible API is not a good idea, and to be honest I was surprised at their viewpoint. If an S3 compatible API is such a bad idea, why do Eucalyptus offer it ? Why do Mezeo ? Why do Google Storage ? I’ll tell you why, because right now Amazon S3 is the big gorilla in Cloud Storage and to make it easy for developers, and tool vendors, to move to/use their platforms they are offering the nearest thing there is to a standard (apart from WebDav….and I’ll come back to that), which is an S3 compatible interface, or way into their storage clouds.

If you extend that thinking further you can see why it would be useful to users, developers, and Organisations, to use an S3 API above other Storage Clouds, that don’t currently have it. For example using an S3 endpoint with RackSpace or Google Docs could solve some key integration problems if you already use S3 and want to manage files against these clouds.

The issue really is about standards, and the lack of standard API’s in Cloud. There is an old, but still relevant ,post from Gartner on this that I tend to agree with. Having standardised API’s from day one is not good because it stifles innovation, however the basic object cloud storage is pretty basic in terms of an API and for lots of users, developers, and companies, being able to standardise around the dominant API, has in certain circumstances, many benefits.

This brings me back to WebDav. Dav was the first real  (and still the only) standard for working with files over http. I’m amazed by how many vendors don’t support it as an entry point to their cloud ie. Google Docs, Amazon S3, RackSpace Cloud Files, DropBox etc etc. Someone made a comment to me recently that WebDav “was so 20th century”. It made me smile as WebDav is still one of the most implemented integration points for Applications. Consider the Apple App Store. It’s two years old and there are a myriad of Applications that support WebDav and also have to support the clouds that don’t support WebDav, and if your cloud is not on the list, then, well it’s tough !  Need a more recent example? The new Nokia N8 Smartphone. It supports WebDav as a native client and users have already picked up that they can use our service to access their cloud data on the device. When we launched our ubiquitous WebDav over any cloud (whether it support it natively or not), CloudDav interface, we did it because we had the same frustrations. We launched this on August 25th.  In only 3 months it is responsible for 30% of our traffic and it has lead to a huge increase in our user base ! That is an amazing statistic and shows that users really want standardisation when it enables them to access their data the way they want to, and of course this is our raison d’etre 😉

Interestingly, in the past Gladinet also had a positive point of view about the S3 API, at least in earlier blog posts, so I’m not sure what lead to their change of heart, but one thing is for sure, I expect more storage companies, not less, to continue issuing compatible S3 endpoints as entry points into their storage clouds.

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

SME adds S3 compatible API access for many clouds including Google Docs, Google Storage, RackSpace + more

We’re happy to roll out support for another API that is becoming ubiquitous. In this case it is the Amazon S3 API. This is already used in solutions such as Eucalyptus Walrus, and we have now also added it as entry point for files accessed through our Cloud Gateway.

What does this mean ? It means you can use the S3 compatible API to access your files on Google Docs, or Google Storage, or RackSpace Cloud Files, or MIcrosoft SkyDrive, or any of the many other Clouds that we support.

For those interested, moving from a file based structure to a flat file object structure was achieved using the depth first algorithm within the implementation mapping to our API.

Lets look at a practical implementation of this using a tool that supports S3 compatible endpoints, CloudBerry explorer. We’ll use this to access our Google Docs, although we could have chosen DropBox, or any other number of clouds to access.

After launching the free version of CloudBerry explorer Choose File-Walrus Accounts

Choose to add an account and fill in the endpoint as ‘s3.storagemadeeasy.com’ and the QueryID as your SME username. You can obtain the API key from logging into the SME website and going to “My Account” from the sidebar and copying the API key from where it says “API secret Key”

Now you can choose to access the service using the CloudBerry exlorer:

and you can browse, download, upload files:

You can find out more on the SME Wiki

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

Coming Soon: Access your Cloud Files from any cloud using S3 API

We’ve been surprised by the huge success of the work we did with CloudDav which enables you to access your Cloud Files from any of the storage clouds we support using WebDav, even if the underlying Storage Cloud itself does not support it.

With that in mind we’ve been working on something even more innovative and that is enabling you to access any of your Cloud Files using the Amazon S3 API. This will mean that you will be able to use existing Amazon S3 tools, scripts etc that support the ability to specify your own endpoint to directly access Cloud Storage on Cloud such as Google Docs, DropBox, RackSpace, SkyDrive etc. We think this will prove just as useful to many developers, individuals and companies as our CloudDav layer has been.

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

Integrating SME multi-cloud API with Simple Cloud API from Zend

We were excited when we read about the Simple Cloud API from Zend. An interoperable Cloud API accessible via PHP was something we were keen to see if we could integrate it’s storage API with our own Multi-Cloud API which is based on REST but which has different language examples and bindings, including PHP.


After examining the SimpleCloud API it is clear to us that the storage API piece, as is, was limited compared to our own multi-cloud storage API whose aim is to provide an abstraction layer with file system functionality coupled with service functionality, such as the ability to backup email, backup contacts from social clouds, provide cloud sync and failover capabilities, encryption facilities etc. The Simple Cloud API is focused on providing access to clouds, with simple Create, Read, Delete functionality, and does not necessarily abstract the way the clouds are accessed or manipulated. 


There are two ways in which we could integrate with Zend. We could take advantage of their libraries to add support for clouds supported by the SimpleCloud API that we do not support, or we could integrate in a way that enables the SimpleCloud API to take advantage of the 10 plus clouds we support, by providing our own SimpleCloud SMEStorage plug-in.


It looks as there will be a fair bit of work to take advantage of the SimpleCloud API in our architecture.  The Current version of the SimpleCloud API doesn’t allow the streaming downloading of files. This would require us to create a set of classes that will extend the Simple Cloud classes and would have new methods related to streaming downloading and other methods that will return some information to use with streaming upload. There are other issues also, such as limitations on listing all buckets in S3. It only works with the bucket that the user provides in the settings.


Despite all of this the SImpleCloud API will be here to stay so we intend to move forward with our integration to it, and it is on our roadmap. 


Stay tuned !

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

SME Cloud Storage API now supports Microsoft Live SkyDrive

One of the added benefits of the Microsoft Live SkyDrive support that we recently announced is thatit becomes accessible via the SMEStorage rich multi-cloud API becomes available for developers who want to access Microsoft Live SkyDrive programmtically. On the API’s Google Code site, as well as a detailed manual, there are PHP, PERL, and .Net examples and sample applications, and soon there will follow fully featured .Net & Cocoa libraries. Remember also, that the sames API also supports Amazon S3, Box.net, RackSpace Cloud Files, MobileMe, WebDav, Gmail Clouds, Email Clouds, FTP Clouds as well as Google Docs and Evernote. Expect to see support for at least two more storage clouds before the end of the year.


Many of the above storage clouds not only lack a proper file system (file support, but no folders, with only basic operations) but some, such as MobileMe, SkyDrive, GMail etc, lack any sort of API at all. That’s why we built the SMEStorage Multi-Cloud File System. We built it to enable us to have a universal API to support the storage clouds we plugged into our platform for our own purposes. Indeed it is what all our client tools are based on.


Given that is it is proven and more feature rich than the alternatives we made the decision to make it available as a generic storage cloud API for those wishing to use the SMEStorage platform as a Gateway to support multiple Cloud platforms in their applications or services. It supports multiple connections and multiple users and it does not impact the actual files themsevles as they are still stored on the original cloud service. SMEStorage merely keeps information about the metadata of the files and enables a file system, with services, on top of the remote storage cloud.


In the case of SkyDrive the lack of an API means our implementation is tightly coupled to Microsoft’s web implementation. Any major changes could render the API temporarily inoperable, but we are committed to make the necessary changes to continue support of this Cloud Provider as we move forward in our pursuit of supporting all major storage clouds.

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

The need for a multi-cloud storage API

photo-2During a lightening talk at London CloudCamp we talked about the need for a multi-cloud storage API (slides below). We then presented the options currently available, and of course the SMEStorage multi-cloud API is one of several. We do believe we bring some nice things to the table with out API which include:


– REST based providing an SPI to multiple clouds
– Provides value added services over and above just upload/dowload, and file system view. There are too many to list, but these include file encryption, Email backup, ETL from one cloud to another cloud, content filtering, faxing and much, much more
– Ability to provide a single view of disparate data in different clouds – a cloud data unification within a single file system view. 


If you are a developer take the time to check out the API and let us know how we can do more to help and what features you are looking for.

The Need For Multi Cloud Storage Api

View more presentations from SMEStorage.

Facebooktwitterredditpinterestlinkedinmailby feather
Posted on

Multi Cloud Storage API Updated

We have just update our multi-cloud API manual and published it on Google Code. We also have added some more code examples in PHP to supplement those in .Net, and added a PERL libaray example with a test PERL app as well.

The multi-cloud API now supports Amazon S3, RackSpace Cloud Files, Box.net, MobileME, Gmail, Any Email, FTP, Google Docs, Evernote, and any WebDav enabled cloud. Coming soon will be Microsoft SkyDrive, Twitter, Deicious, and Walrus from Eucalyptus.

Facebooktwitterredditpinterestlinkedinmailby feather