Here at Storage Made Easy we can quite often get involved in helping a customer clone, or move, their data from one storage provider to another. This is often a one time operation, and we have a number of in-house apps and scripts that we are able to use but today I’d like to discuss a third party free solution called Rclone.
As good as Rclone is, it does have its quirks, one of which is that it neither Rclone sync nor Rclone copy seem to be able to create empty directories on Amazon S3 object storage service , even when we used the create-empty-src-dirs flag. In some cases our customers need those empty directories, which means that we had to find an alternative way to ensure that they were created.
The solution was relatively straightforward as we have many in-houe Apps and tools to handle this. In this case the solution was to use our own Windows stand-alone S3 drive to finish the job. The solutions was automated to follow these steps:
- Pre-Configure the S3 drive to talk to the destination S3 storage account.
- Use Rclone sync to copy the files and the folders that contain files from the source storage to the S3 destination storage.
- Repeat the operation in dry run mode, and capture the output in a log file.
- Find the lines in the log file that look like this:
2019/09/17 16:18:49 NOTICE: TopLevel/User3/Company3/2002/Apr: Not making directory as dry run is set
Each of these lines corresponds to a directory that needs to be created.
- Extract the paths from these log file lines.
- Use the Windows mkdir command line to create the missing directories on the stand-alone drive through the Windows command line.
- The S3 drive will create the empty folders (actually, zero length objects with keys containing path segment delimiters) on the S3 storage to which it is connected.
Readers who are Rclone users might wonder why we used our own drive to create the empty directories on S3 instead of using Rclone’s mkdir command. The answer is that the Rclone mkdir command simply would not create empty directories on S3 either in the version we were using. Its likely this will be resolved in future no doubt but for now a combination of the two Apps served us well.
You can check out our S3 Drive here:
https://storagemadeeasy.com/standalonetools/S3-Drive/
Although Storage Made Easy is a software company primarily known for the variants of its Enterprise File Fabric (either for Object Storage, Governance and Compliance, or Media and Entertainment) we often get involved in some fairly meaty data migrations so if you are a company looking to move a lot of data (and we have dealt with Gigabytes right through to Petabytes) then you may well want to contact us to help out.
by 
