Writing to an erasure coded pool in Ceph Rados

CephLately we’ve been working very closely with RedHat’s Ceph Rados storage and it’s librados API, where we’re seeking an ever closer integration with the backend storage to utilise many of Ceph’s benefits.

However lately, we hit a issue where one of our customers had configured their Pool to be erasure coded. Erasure coding is a form of data protection and data redundancy whereby the original file or object is split up into a number of parts, and distributed across a number of storage nodes, either within the same data-centres or across multiple multiple data-centres and regions.

This is not an uncommon practice within the storage space, however, when our implementation was tested on an EC pool, we observed some nuances between how you can write to an erasure coded pool, and non erasure coded pool.

Since the Storage Made Easy Appliance acts as a gateway or access layer to your backend storage, when we’re handling files we prefer not to keep the full file in memory, as this consumes the valuable memory on the machine. For example, when a user uploads a file, as the data is streamed from the client, we stream the data we receive straight onto the backend storage.

We were utilising the same approach for Ceph Rados, through it’s rados_append API function, which is also similar to the rados_write API method with offsets.

When we were dealing with non erasure coded pools, we found this to be the perfect balance in terms of memory usage and upload speed. However, when we transitioned to an erasure coded pool, our second append operation would return “Error 95: Operation not supported”. Despite looking around for a while, there seemed to be little information around on how you could solve this issue.

Researching around, the leading solution people were suggesting was to install a replacated caching pool in-front of the erasure coded pool, however given we don’t control our customers storage, we opted not to stick with this resolution. We approached the Ceph mailing list with this issue, and we were pointed towards writing in multiples of the stripe width. Striping is the process of “storing sequential pieces of information across multiple storage devices–to increase throughput and performance”.

As we found out, the trick is to to write in a multiple of the stripe width. For example, if you have an object that is 10000 bytes in size, and your stripe width is 4000 bytes, the most efficient way would be to first append / write the first 8000 bytes, then followed by the remaining 2000 bytes. However, you have to be aware that no further writes or appends can be made to this object after you have written data that is not a multiple of the stripe width. In such cases, you will have to re-write the whole object from the beginning.

To find out the Stripe width of your erasure coded pool, you can make use of the rados_ioctx_pool_required_alignment API method, that will return the stripe width in bytes.

We hoped an believed that this should be an implementation detail managed by the Ceph cluster, but it’s unfortunately not. Hopefully this information will help anyone who may run into this issue!

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

How do you solve a problem like DropBox ? (aka securing corporate dropbox accounts)

How do you solve a problem like dropbox?

DropBox has been in the press quite a lot lately with regards to passwords breaches and also with surreptitious behaviour with regard to  machine security on Mac. Although DropBox has started to reassure users with ‘how secure we are‘ type information Corporate IT departments will again feel they have cause for concern with regards any internal corporate use of DropBox.

So the key question we are trying here is ‘just do do you solve a problem like Dropbox‘ ? Indeed this is a slightly unfair question in that it is using DropBox to make a point and the reality is we could have picked on one of several cloud storage services as Corporate IT has misgivings with anything Cloud when it relates to files.

Continue reading →

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Securing Storage Made Easy with Let’s Encrypt

Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. Let’s Encrypt is a service provided by the Internet Security Research Group (ISRG). Lets-Encrypt-SME

Read more at About Let’s Encrypt

Storage Made Easy recommends all traffic to be secured with encryption, as a matter of fact, by default we enforce the user of HTTPS communication. That said the software ships with self-signed certificates, to get you started, and when you first connect you will be greeted by an Invalid Certificate message in most browsers as self-signed is fine for pre-production, setup and testing but real certificates need to be added for production. In this post I will show you how to setup a free of charge, trusted certificate with our product.

Continue reading →

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

Webinar: An Introduction to the Storage Made Easy Enterprise File Share and Sync Fabric


SME Webinar

With the world growing increasingly data-rich, customers are looking for tools to access data at any time, whilst also being as secure as possible.

Storage Made Easy provides a comprehensive control and management solution with the most enriched cloud computing features to enable secure cloud collaboration across data points for corporations.

Continue reading →

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather