Step-by-step guide to authenticate with the SME API using Python

Over the past week I’ve been busy creating Python scripts to automate certain processes here in the SME DevOps team.

One of my main objectives has been to create a script that interacts with the SME API to check the health of specific functions, for this task I decided to opt for the language of ‘DevOps’ Python. Whilst I do not profess to be a Python expert in any way I felt that this is a language is one that I have become accustomed too and am able to ‘get by’ with. The nice thing with Python are the extensive libraries that are readily available, and the fact that they are fairly easy to use (of course, only if you have some background experience with Python).

Continue reading →

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Google Docs upload currently not working

Our Cloud monitor has noticed an issue with Google Docs uploading over the resumable upload API that we use to upload files to Google Docs. Currently for any upload Google Docs is returning a response of:

<errors xmlns=’http://schemas.google.com/g/2005′><error><domain>GData</domain><code>InvalidEntryException</code><internalReason>Sorry, there was an error saving the file. Please try again.</internalReason></error></errors>

We’ve reported the issue to Google and are waiting for a response. We will post here when Google resolve the issue.

Update: Google have responded:

The thread can be found here.

Update: Google have now resolved the issue

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Achieving S3 API compatibility with OpenStack Swift using SME

One of the things that is close to our hearts at SMEStorage is interoperability and, wherever possible, enabling open solutions. As outlined in our previous post, we recently added support for OpenStack Swift object storage. Of course there are other Object Storage frameworks, such as Amazon S3, but this is closed source, unlike Swift. There are however open source implementations of the Amazon S3 API’s including Walrus from Eucalyptus. There are also many tools, solutions, exiting code and frameworks that use S3 compatible API’s, and no doubt there will be occassions when it would be useful to enable users to connect to SWift using Amazon S3 API’s.

As SMEStorage has a compatible S3 API abstraction it turns out that is very easy. Let’s demonstrate how easy below:


After launching the free version of CloudBerry explorer Choose File-Walrus Accounts, and then choose to add an account.

The endpoint for the s3 compatible interface with SMEStorage is s3.storagemadeeasy.com. The username is the SMEStorage Account username, and the secret key can be found for each user on the “My DashBoard” page under a section called “Tech Info“.

Once we have entered the details we can test the connection:

and finally we can connect over the S3 API to work with our OpenStack Swift files:

As you can see this was easy and takes less than a couple of minutes to access and work with OpenStack Swift files using S3. We expect it to be very useful for ISP’s and SMB’s who want to deploy their own private cloud solution using OpenStack and enables interoperability and prevents lock-in.

Update: Note that in the Swift Diablo release that OpenStack also supports the ability to turn on Amazon S3 API’s.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Why an S3 compatible API ? A rebuttal

I read a post by Gladinet yesterday that talked about why, in their opinion, an S3 compatible API is not a good idea, and to be honest I was surprised at their viewpoint. If an S3 compatible API is such a bad idea, why do Eucalyptus offer it ? Why do Mezeo ? Why do Google Storage ? I’ll tell you why, because right now Amazon S3 is the big gorilla in Cloud Storage and to make it easy for developers, and tool vendors, to move to/use their platforms they are offering the nearest thing there is to a standard (apart from WebDav….and I’ll come back to that), which is an S3 compatible interface, or way into their storage clouds.

If you extend that thinking further you can see why it would be useful to users, developers, and Organisations, to use an S3 API above other Storage Clouds, that don’t currently have it. For example using an S3 endpoint with RackSpace or Google Docs could solve some key integration problems if you already use S3 and want to manage files against these clouds.

The issue really is about standards, and the lack of standard API’s in Cloud. There is an old, but still relevant ,post from Gartner on this that I tend to agree with. Having standardised API’s from day one is not good because it stifles innovation, however the basic object cloud storage is pretty basic in terms of an API and for lots of users, developers, and companies, being able to standardise around the dominant API, has in certain circumstances, many benefits.

This brings me back to WebDav. Dav was the first real  (and still the only) standard for working with files over http. I’m amazed by how many vendors don’t support it as an entry point to their cloud ie. Google Docs, Amazon S3, RackSpace Cloud Files, DropBox etc etc. Someone made a comment to me recently that WebDav “was so 20th century”. It made me smile as WebDav is still one of the most implemented integration points for Applications. Consider the Apple App Store. It’s two years old and there are a myriad of Applications that support WebDav and also have to support the clouds that don’t support WebDav, and if your cloud is not on the list, then, well it’s tough !  Need a more recent example? The new Nokia N8 Smartphone. It supports WebDav as a native client and users have already picked up that they can use our service to access their cloud data on the device. When we launched our ubiquitous WebDav over any cloud (whether it support it natively or not), CloudDav interface, we did it because we had the same frustrations. We launched this on August 25th.  In only 3 months it is responsible for 30% of our traffic and it has lead to a huge increase in our user base ! That is an amazing statistic and shows that users really want standardisation when it enables them to access their data the way they want to, and of course this is our raison d’etre 😉

Interestingly, in the past Gladinet also had a positive point of view about the S3 API, at least in earlier blog posts, so I’m not sure what lead to their change of heart, but one thing is for sure, I expect more storage companies, not less, to continue issuing compatible S3 endpoints as entry points into their storage clouds.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

SME adds S3 compatible API access for many clouds including Google Docs, Google Storage, RackSpace + more

We’re happy to roll out support for another API that is becoming ubiquitous. In this case it is the Amazon S3 API. This is already used in solutions such as Eucalyptus Walrus, and we have now also added it as entry point for files accessed through our Cloud Gateway.

What does this mean ? It means you can use the S3 compatible API to access your files on Google Docs, or Google Storage, or RackSpace Cloud Files, or MIcrosoft SkyDrive, or any of the many other Clouds that we support.

For those interested, moving from a file based structure to a flat file object structure was achieved using the depth first algorithm within the implementation mapping to our API.

Lets look at a practical implementation of this using a tool that supports S3 compatible endpoints, CloudBerry explorer. We’ll use this to access our Google Docs, although we could have chosen DropBox, or any other number of clouds to access.

After launching the free version of CloudBerry explorer Choose File-Walrus Accounts

Choose to add an account and fill in the endpoint as ‘s3.storagemadeeasy.com’ and the QueryID as your SME username. You can obtain the API key from logging into the SME website and going to “My Account” from the sidebar and copying the API key from where it says “API secret Key”

Now you can choose to access the service using the CloudBerry exlorer:

and you can browse, download, upload files:

You can find out more on the SME Wiki

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Coming Soon: Access your Cloud Files from any cloud using S3 API

We’ve been surprised by the huge success of the work we did with CloudDav which enables you to access your Cloud Files from any of the storage clouds we support using WebDav, even if the underlying Storage Cloud itself does not support it.

With that in mind we’ve been working on something even more innovative and that is enabling you to access any of your Cloud Files using the Amazon S3 API. This will mean that you will be able to use existing Amazon S3 tools, scripts etc that support the ability to specify your own endpoint to directly access Cloud Storage on Cloud such as Google Docs, DropBox, RackSpace, SkyDrive etc. We think this will prove just as useful to many developers, individuals and companies as our CloudDav layer has been.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather