Using Ansible with On-premises S3 Object Storage

At Storage Made Easy we work/partner with many Object Storage vendors.

When a project required data automation and the use of an on-premises object storage buckets, I turned to Ansible and our storage partners.  However, I found that the aws_s3 module for Ansible wasn’t as friendly and well documented when working with S3 object storage that isn’t either AWS or Ceph.

The project was simple.  Pull data from a number of secure locations worldwide and send to various S3-compatible object stores.

Using the documentation and examples provided for the aws_s3 ansible module, I quickly had some code running that was able to copy data to an AWS bucket.

Example from:

- name: Simple PUT operation
    bucket: mybucket
    object: /my/desired/key.txt
    src: /usr/local/myfile.txt
    mode: put

Once that was working, I tried to copy data to some of our various partners:
Cloudian, Scality, SwiftStack, IBM Cloud Object Storage, Minio

Per the documentation, I should have had to only add the URL endpoint of these storage providers and additionally, whether to trust self-signed certificates or if the data should be encrypted with SSE-S3 (Server Side Encryption).

   s3_url: "{{ s3_endpoint }}"
   validate_certs: true
   encrypt: false

However, this code was consistently throwing errors related to the endpoint (s3_url) being invalid.  This was solved by adding the flag for Rados Gateway, which is the Ceph solution for using the S3 API.

   rgw: true

It’s not documented, but anytime the S3 target is not AWS, the rgw flag must be set to true when specifying an endpoint via s3_url.

This code worked for some of our partners like Cloudian, but most others still failed, now with an error related to ACLs.  It turns out that every time Ansible writes an object, it’s also setting the ACL of the object which is an S3 API call that many 3rd party providers don’t yet support.

Disabling the setting of ACLs took much digging (trial /error and reading the source code), since it’s not documented.  The solution was to specify an empty list for the permission flag.

   permission: []

The end function for the playbook to upload a set/list of files to an on-premises object storage system like Minio, Scality or SwiftStack looks like this:

    - name: Upload to Non AWS S3 Storage
        bucket: "{{ backup_bucket }}"
        object: "{{ item.path }}"
        src: "{{ item.path }}"
        mode: put
        aws_access_key: "{{ access_key }}"
        aws_secret_key: "{{ secret_key }}"
        validate_certs: (true/false)
        encrypt: (true/false)
        rgw: true
        permission: []
        s3_url: "{{ s3_endpoint }}"
      loop: "{{ files_to_backup.files }}"
        label: "{{ item.path }}"

I hope this article saves you hours of research (trial and error) and enables devops to use AWS S3 alternatives.


Facebooktwitterredditpinterestlinkedinmailby feather
The following two tabs change content below.

Douglas Soltesz

Director Product Solutions at Storage Made Easy
Doug's focus is in Object and Cloud Storage APIs, Data Governance, Virtualization, and Containerization.