How to use the AWS CLI with almost any Storage

  The Amazon S3 API has become the de-facto standard for object storage API access.  As such, many software products adopt the S3 API first.  The Storage Made Easy File Fabric provides an S3 compatible API endpoint for any storage backend.  This means that non-S3 storage providers such as Dropbox, Azure, Google Drive, OneDrive, Sharepoint and others can now be accessed via the S3 API.

The Amazon Web Services (AWS) Command Line Interface (CLI) is a command line tool for accessing AWS compatible services.  AWS CLI is one of the most feature rich S3 tools publicly available (Free!!!) .

This article will outline the setup AWS CLI with the SME File Fabric and non-S3 compatible storage providers.

Installation

AWS CLI is compatible with Mac, Linux, and Windows with Python 2 version 2.6.5+ or Python 3 version 3.3+.  This article requires that AWS CLI is already downloaded and installed.  For detailed instructions on Installation see the links below:

Amazon’s Guide can be found here:  http://docs.aws.amazon.com/cli/latest/userguide/installing.html

Storage Made Easy detailed guide (with pictures):
https://blog.storagemadeeasy.com/the-comprehensive-guide-to-installing-aws-cli-on-windows-mac-or-linux/

S3 Prerequisites from File Fabric

AWS CLI and other S3 tools require three pieces of information:

1. Endpoint (URL of S3 protocol gateway)
   1. US SaaS – https://s3.storagemadeeasy.com
   2. EU SaaS – https://s3eu.storagemadeeasy.com
   3. IaaS or OnPrem File Fabric – Ask your administrator
2. Access Key (Username)
   1. Use File Fabric User login IE: username@organization
3. Secret Key (Password)
   1. This must be obtained via the Web GUI of File Fabric

Locating the S3 Secret Key

To obtain the S3 Secret Key:

1. Login to File Fabric
2. Open the Menu Icon in Top Right of Browser Window
3. Click “My Personal Data”
4. Expand “Tech. Info”
5. Click the reveal icon under “API secret key”

Configuring AWS CLI to connect with File Fabric

On a system that is installed with AWS CLI run the following to setup the File Fabric:

aws configure --profile=SME

Note:  The profile does not need to be “SME”, it can be any value.

AWS Access Key ID [None]: dsoltesz@mycompany
AWS Secret Access Key [None]: 6b2a362943a95a0e345az6c8b38519d0
Default region name [None]: US-East-1
Default output format [None]:

Note:  Fill in your Access and Secret Keys.  Use “US-East-1” as the default region

Listing File Fabric Providers with AWS CLI

Now that the credentials are inputed, AWS CLI can be used by specifying the endpoint and profile in the command line.  The following example uses the US SaaS endpoint (URL) for the Storage Made Easy File Fabric.

# aws --endpoint=https://s3.storagemadeeasy.com --profile=SME s3 ls s3://SME
 PRE My DropBox files/
 PRE My Google Drive files/
 PRE My OneDrive files/
 PRE My SME files/
 PRE Shared Documents/

Here’s what that command did, it listed all connected backend providers in the bucket called SME.  By default, the File Fabric creates a virtual bucket called SME.

Now Dropbox, Google Drive, OneDrive and any other provider can be accessed via the S3 API.  Let’s copy a folder with subfolders from Dropbox to a my local desktop.

Listing the folders in Dropbox with the command below shows three folders.

aws --endpoint=https://s3.storagemadeeasy.com --profile=SME s3 ls "s3://SME/My DropBox files/"
 PRE Images/
 PRE Marketing Docs/
 PRE Cat Photos/

Copy Local Folder to Dropbox via S3 API

To copy a local folder with subfolders to a new Dropbox folder called Marketing Images :

aws --endpoint=https://s3.storagemadeeasy.com --profile=SME s3 cp Marketing_Images/ "s3://SME/My DropBox files/Marketing Images/" --recursive

Conclusion

The AWS CLI is just one of many tools that can be utilized with S3 API capable storage.  Storage Made Easy File Fabric makes any storage S3 compatible.  With AWS CLI data movement can be scripted, synced, migrated on Windows, Linux and Mac desktops.

For more information see these posts:

AWS CLI aliasing for Windows, Linux and Mac

by

The following two tabs change content below.

• Bio
• Latest Posts

Douglas Soltesz

Director Product Solutions at Storage Made Easy

Doug's focus is in Object and Cloud Storage APIs, Data Governance, Virtualization, and Containerization.

Latest posts by Douglas Soltesz (see all)

• LucidLink Technology Preview Setup Guide - September 6, 2021
• The features and benefits of using LucidLink with the Enterprise File Fabric - September 2, 2021