Step-by-step guide to authenticate with the SME API using Python

Over the past week I’ve been busy creating Python scripts to automate certain processes here in the SME DevOps team.

One of my main objectives has been to create a script that interacts with the SME API to check the health of specific functions, for this task I decided to opt for the language of ‘DevOps’ Python. Whilst I do not profess to be a Python expert in any way I felt that this is a language is one that I have become accustomed too and am able to ‘get by’ with. The nice thing with Python are the extensive libraries that are readily available, and the fact that they are fairly easy to use (of course, only if you have some background experience with Python).

Continue reading →

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Turning any Storage Cloud into an Amazon S3 compatible Private, Public or hybrid Cloud

In the past we have written about Amazon S3 and how, aas the 100 pound gorilla, of the Cloud Storage world, it’s S3 API has become almost a de facto interface for developers. This is one of the reasons that we originally added an S3 API protocol adaptor to our service.

Many start up’s, small businesses and even enterprises choose initially to use S3 for storage.  This can be fine initially, however, when the volume increases the monthly bill can become an OPEX issue and small companies (and Enterprise) are looking for ways to slash their costs in any way they can. Aside from this other companies have stringent issues about where data is stored (for clarification, Amazon S3 is PCI DSS 2.0 compliant,   SAS 70 Type II certified, and VPV ISO 27001 certified) or wish to store it within their own data centre or site.

As SMEStorage supports over 35 Clouds and SaaS services, you could very easily turn Google Docs, Box, Windows Azure or DropBox into an S3 Platform, or you could just add your own NetGear or PogoPlug appliance, or other private storage implementation.

The SME hosted service, and Cloud Appliance,  provides the ability access to any Saas or cloud storage mapped to your account via multiple protocols. These include  FTP, WebDav and also S3. These work even if the backend provider does not support the protocol natively. The SME protocol adaptors will do the protocol translation to the native storage provider protocol. One of the benefits of this that the users don’t need any special software to be able to access the Clouds. They can use any FTP, or WebDav client, or in the case of S3 any S3 client or code in which the host endpoint can be changed.

To demonstrate compatibility with the S3 API and tools we will now look at how to use AWS s3curl with a smestorage account.

To use s3 curl you will need to modify s3curl.pl and change the end point to

s3.smetorage.com’ (US Server) or ‘s3eu.storagemadeeasy.com‘ (EU Server) e.g my @endpoints = ( ‘s3.storagemadeeasy.com‘);

Your id is your smestorage account user name and you can obtain your secret key by logging into SMEStorage.com going to “My  Dashboard” (from the sidebar) and copying the API key from  the“Tech Info”  section where the “API secret Key” resides.

Now you are all set to use s3curl. For example to list all the buckets you can use

./s3curl.pl –id smestorageusername –key API secret key http://s3.storagemadeeasy.com

For s3curl command line options please see the README file that is part of the s3curl package. Also note that the secure way to use s3curl is to use the .s3curl file in your home directory to pass the id and and key.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Bit.ly integration added for Cloud File links

We’ve added bit.ly integration for the generation of short file links for Cloud files mapped to a users SME account.

The long file link is a secure file link that we generate, but we have always had the option to shorten this with TinyURL. This will still be the default option, but for users who want to take advantage of bit.ly for tracking SEO or statistics against links, they can by entering their bit.ly key from within the App integrations menu on the right sidebar after login to the website. After doing this bit.ly will be used to generate short links from any access client.

Bit.ly integration works on free, personal or business accounts.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

OpenStack now supported for SME Open Cloud SaaS Platform and Cloud Appliance

We are really pleased to announce that we have added OpenStack Swift object Storage support to the SME Open Cloud Platform. Swift is a sub project of OpenStack and provides a highly scalable redundant unstructured data store. Swift is 5 separate services, object, container, account, auth and proxy. Although each of these can be scaled separately, in practice they run together.

Never heard of Swift? it’s the underlying distributable object store that supports RackSpace Cloud Files. It’s akin to Amazon’s S3 implementation but unlike implementations such as Eucalyptus, which clone S3 API’s, but are not sponsored by Amazon, openStack and Swift has RackSpace firmly onboard, and have proven scale.

As Swift is used by Rackspace Cloud Files. Swift RackSpace claim it is production-ready code that is scalable to massive levels (100-petabyte clusters and 100000 requests per second). Swift sacrifices C for A and P from a CAP theorem perspective. Although most operations happen synchronously consistency is sacrificed in failure scenarios.

From our perspective we have seen ISP’s and larger SMB users of our on-premise Cloud Gateway appliance expressing interest in SME supporting this, and we supply this as VMWARE Appliance (OR XEN, KVM) or as a dedicated hardware appliance for smaller companies who wish to embrace their own private Cloud infrastructure.

As with our S3 API endpoint support SME will overlay a more traditional file store on top of Swift layered with the business functionality we provide in our  Cloud File Server, which includes virtual drives and clients for Mac, Windows and Linux, and feature rich mobile clients for iPad, iPhone, Android and BlackBerry, as well as value added features to Swift such as Webdav and FTP support.

Setting up Swift with SME is easy. First you need to add a new Cloud Provider and then the Cloud Wizard will be invoked. The first step is to enter your OpenStack details:

When entering the endpoint URL you should be sure to include the Port. An example URL is: http://<IP Address>:11000/v1.0.

Next you will need to choose which containers you want to work with and which should be the default container for any uploads to smart folders.

Once you have done this you will be ready to start the meta-sync which pulls in and caches all the information about containers and files.

If you have any issues connecting please refer to this advanced post on using SME with OpenStack 1.60 and SWAuth.

Once complete you will be able to access/amange your OpenStack files from the SME Web clients,  as well as using a Cloud Drive on Windows, Mac or Linux, and mobile clients for Android, iOS, and BlackBerry, and  the plethora of other tools and clients that SME provides. We’v e posted some screenshots below of this.

Web File Manager

iOS OpenStack

Android OpenStack

Firefox Plug-In OpenStack

Chrome OpenStack Plug-In

Mac Cloud Drive OpenStack

The OpenStack Swift API’s also get embedded for use within our own feature rich multi-cloud API framework in which we add many business driven features.. You can find details about that on our developer page

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Why an S3 compatible API ? A rebuttal

I read a post by Gladinet yesterday that talked about why, in their opinion, an S3 compatible API is not a good idea, and to be honest I was surprised at their viewpoint. If an S3 compatible API is such a bad idea, why do Eucalyptus offer it ? Why do Mezeo ? Why do Google Storage ? I’ll tell you why, because right now Amazon S3 is the big gorilla in Cloud Storage and to make it easy for developers, and tool vendors, to move to/use their platforms they are offering the nearest thing there is to a standard (apart from WebDav….and I’ll come back to that), which is an S3 compatible interface, or way into their storage clouds.

If you extend that thinking further you can see why it would be useful to users, developers, and Organisations, to use an S3 API above other Storage Clouds, that don’t currently have it. For example using an S3 endpoint with RackSpace or Google Docs could solve some key integration problems if you already use S3 and want to manage files against these clouds.

The issue really is about standards, and the lack of standard API’s in Cloud. There is an old, but still relevant ,post from Gartner on this that I tend to agree with. Having standardised API’s from day one is not good because it stifles innovation, however the basic object cloud storage is pretty basic in terms of an API and for lots of users, developers, and companies, being able to standardise around the dominant API, has in certain circumstances, many benefits.

This brings me back to WebDav. Dav was the first real  (and still the only) standard for working with files over http. I’m amazed by how many vendors don’t support it as an entry point to their cloud ie. Google Docs, Amazon S3, RackSpace Cloud Files, DropBox etc etc. Someone made a comment to me recently that WebDav “was so 20th century”. It made me smile as WebDav is still one of the most implemented integration points for Applications. Consider the Apple App Store. It’s two years old and there are a myriad of Applications that support WebDav and also have to support the clouds that don’t support WebDav, and if your cloud is not on the list, then, well it’s tough !  Need a more recent example? The new Nokia N8 Smartphone. It supports WebDav as a native client and users have already picked up that they can use our service to access their cloud data on the device. When we launched our ubiquitous WebDav over any cloud (whether it support it natively or not), CloudDav interface, we did it because we had the same frustrations. We launched this on August 25th.  In only 3 months it is responsible for 30% of our traffic and it has lead to a huge increase in our user base ! That is an amazing statistic and shows that users really want standardisation when it enables them to access their data the way they want to, and of course this is our raison d’etre 😉

Interestingly, in the past Gladinet also had a positive point of view about the S3 API, at least in earlier blog posts, so I’m not sure what lead to their change of heart, but one thing is for sure, I expect more storage companies, not less, to continue issuing compatible S3 endpoints as entry points into their storage clouds.

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

SME adds S3 compatible API access for many clouds including Google Docs, Google Storage, RackSpace + more

We’re happy to roll out support for another API that is becoming ubiquitous. In this case it is the Amazon S3 API. This is already used in solutions such as Eucalyptus Walrus, and we have now also added it as entry point for files accessed through our Cloud Gateway.

What does this mean ? It means you can use the S3 compatible API to access your files on Google Docs, or Google Storage, or RackSpace Cloud Files, or MIcrosoft SkyDrive, or any of the many other Clouds that we support.

For those interested, moving from a file based structure to a flat file object structure was achieved using the depth first algorithm within the implementation mapping to our API.

Lets look at a practical implementation of this using a tool that supports S3 compatible endpoints, CloudBerry explorer. We’ll use this to access our Google Docs, although we could have chosen DropBox, or any other number of clouds to access.

After launching the free version of CloudBerry explorer Choose File-Walrus Accounts

Choose to add an account and fill in the endpoint as ‘s3.storagemadeeasy.com’ and the QueryID as your SME username. You can obtain the API key from logging into the SME website and going to “My Account” from the sidebar and copying the API key from where it says “API secret Key”

Now you can choose to access the service using the CloudBerry exlorer:

and you can browse, download, upload files:

You can find out more on the SME Wiki

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather