The Amazon S3 API has become the de-facto standard for object storage API access. As such, many software products adopt the S3 API first. The Storage Made Easy File Fabric provides an S3 compatible API endpoint for any storage backend. This means that non-S3 storage providers such as Dropbox, Azure, Google Drive, OneDrive, Sharepoint and others can now be accessed via the S3 API.
The Amazon Web Services (AWS) Command Line Interface (CLI) is a command line tool for accessing AWS compatible services. AWS CLI is one of the most feature rich S3 tools publicly available (Free!!!) .
This article will outline the setup AWS CLI with the SME File Fabric and non-S3 compatible storage providers.
AWS CLI is compatible with Mac, Linux, and Windows with Python 2 version 2.6.5+ or Python 3 version 3.3+. This article requires that AWS CLI is already downloaded and installed. For detailed instructions on Installation see the links below:
Amazon’s Guide can be found here: http://docs.aws.amazon.com/cli/latest/userguide/installing.html
Storage Made Easy detailed guide (with pictures):
S3 Prerequisites from File Fabric
AWS CLI and other S3 tools require three pieces of information:
- Endpoint (URL of S3 protocol gateway)
- US SaaS – https://s3.storagemadeeasy.com
- EU SaaS – https://s3eu.storagemadeeasy.com
- IaaS or OnPrem File Fabric – Ask your administrator
- Access Key (Username)
- Use File Fabric User login IE: username@organization
- Secret Key (Password)
- This must be obtained via the Web GUI of File Fabric
Locating the S3 Secret Key
To obtain the S3 Secret Key:
- Login to File Fabric
- Open the Menu Icon in Top Right of Browser Window
- Click “My Personal Data”
- Expand “Tech. Info”
- Click the reveal icon under “API secret key”
Configuring AWS CLI to connect with File Fabric
On a system that is installed with AWS CLI run the following to setup the File Fabric:
aws configure --profile=SME
Note: The profile does not need to be “SME”, it can be any value.
AWS Access Key ID [None]: dsoltesz@mycompany AWS Secret Access Key [None]: 6b2a362943a95a0e345az6c8b38519d0 Default region name [None]: US-East-1 Default output format [None]:
Note: Fill in your Access and Secret Keys. Use “US-East-1” as the default region
Listing File Fabric Providers with AWS CLI
Now that the credentials are inputed, AWS CLI can be used by specifying the endpoint and profile in the command line. The following example uses the US SaaS endpoint (URL) for the Storage Made Easy File Fabric.
# aws --endpoint=https://s3.storagemadeeasy.com --profile=SME s3 ls s3://SME PRE My DropBox files/ PRE My Google Drive files/ PRE My OneDrive files/ PRE My SME files/ PRE Shared Documents/
Here’s what that command did, it listed all connected backend providers in the bucket called SME. By default, the File Fabric creates a virtual bucket called SME.
Now Dropbox, Google Drive, OneDrive and any other provider can be accessed via the S3 API. Let’s copy a folder with subfolders from Dropbox to a my local desktop.
Listing the folders in Dropbox with the command below shows three folders.
aws --endpoint=https://s3.storagemadeeasy.com --profile=SME s3 ls "s3://SME/My DropBox files/" PRE Images/ PRE Marketing Docs/ PRE Cat Photos/
Copy Local Folder to Dropbox via S3 API
To copy a local folder with subfolders to a new Dropbox folder called Marketing Images :
aws --endpoint=https://s3.storagemadeeasy.com --profile=SME s3 cp Marketing_Images/ "s3://SME/My DropBox files/Marketing Images/" --recursive
The AWS CLI is just one of many tools that can be utilized with S3 API capable storage. Storage Made Easy File Fabric makes any storage S3 compatible. With AWS CLI data movement can be scripted, synced, migrated on Windows, Linux and Mac desktops.
For more information see these posts:
Latest posts by Douglas Soltesz (see all)
- How to use Ansible to automate VMware OVAs - July 24, 2019
- Ansible: Time series data in S3 API without HEADing metadata - July 17, 2019