Skip to content
Storage Made Easy Blog

Storage Made Easy Blog

Enterprise File Fabric™

Posted on November 20, 2020 by Steven Sweeting

Jibe: Real-time Content Monitoring of Files and Objects for Compliance and Search

The Enterprise File Fabric provides a global file system through which users and applications can transparently access corporate file systems and object storage no matter where they or the storage is located. To provide universal browsing, search, and other services the File Fabric scans and smartly indexes file and object metadata. In this way, no data is copied or replicated, and the metadata is used to build out the high-performance global file system.

Unlike other global file system solutions, the File Fabric does not prevent direct access to the storage – what we call bi-model access. Data can be accessed and updated concurrently through the File Fabric and directly through native file and object-APIs. When objects are updated through native APIs the File Fabric metadata should also be updated to keep indexes and other metadata synchronized.

An earlier blog article outlined three general approaches for keeping metadata up to date, including a specific example using Amazon S3 object notifications. See Using AWS Lambda to Automatically Sync Metadata With S3 Events. This article describes a similar approach we have developed for file systems using the same framework we call Jibe.

 

Jibe Metadata Sync Architecture

File System Metadata Notifications

The File Fabric supports several different file systems. Essentially any file system that can be mounted on Linux, or accessed over SMB, NFS, FTP or WebDAV can be connected to the File Fabric. The metadata is loaded using native storage APIs, traversing object buckets, and folder trees. Once indexed the storage is accessible through the File Fabric global file system.

How do we keep the File Fabric metadata up to date when changes are made to the underlying file system?  There are several ways in which we could detect changes in real-time. Some approaches are vendor or operating system specific, and some are specific to the protocol. For SME we settled on using SMB folder notifications. Although it is a protocol-specific mechanism, it provides a lot of control and of course is supported by any file server that supports the SMB protocol.

SMB Notification Mechanism

Jibe makes an SMB Change Notify request to a file server to be notified of specific changes to that directory tree. The SMB server returns a list of file system events that match that request. Jibe makes the request again to obtain subsequent changes. If there are no changes queued the request blocks until changes are available – a “long polling” algorithm.

Implementation Considerations

As we’re using an event stream rather than logs or queues if Jibe is not actively running, changes may be missed; there is no log file to use for recovery for example. We solve this by deploying redundant instances and making any updates that require a restart to each node in turn.

SMB notifications don’t distinguish files from folders. Jibe may try to sync an event as a file and based on the context may subsequently retry that event as a folder.

When a folder is renamed or moved change notifications are only received for the renamed object, not for child items and subfolders. In this case, Jibe will synchronize folder contents as well.

Optimizations include merging multiple events for the same file and ignoring well-known temporary files.

Jibe Configuration

Jibe is a Python application that can run standalone, as a Lambda function, or as Docker image. Settings for Jibe include event “sources” – log files or event streams providing storage change events.

    {
        "endpoint": "https://storagemadeeasy.com",
        "login": "adminuser@example.com",
        "password": "*****",
        "sources": [
            {
                "name": "Provider Name",
                "smb_share": "ExampleShare",
                "smb_dir": "",
                "smb_server": "smb.example.com",
                "smb_username": "smbuser",
                "smb_password": "smbpass"
            }
        ]
    }

Policy Enforcement Use Case

The ability for the File Fabric metadata to be updated from native file and object storage events in real-time enables the File Fabric to take actions as it becomes aware of the new file and objects.

Consider real-time policy enforcement to prevent exposure of personal data and intellectual property or automatically creating archiving copies of file//objects to protect from Ransomware threats.

  1. File added to storage –  Using S3 API or file share
  2. Jibe Sync detects storage event –  Updates File Fabric metadata
  3. File scanned for sensitive data –  Also updates content search index
  4. Policy-based enforcement –  Quarantines file and send notification
  5. Quarantined file no longer accessible –   (from S3, file share, or user apps)
Jibe Policy Enforcement
Enterprise File Fabric – Real-time policy enforcement

More Information

For more information about Jibe in your environment contact support@storagemadeeasy.com

 

*Image by Gerd Altmann from Pixabay 

Facebooktwitterredditpinterestlinkedinmailby feather
The following two tabs change content below.
  • Bio
  • Latest Posts

Steven Sweeting

Latest posts by Steven Sweeting (see all)

  • Using Storage Made Easy File Fabric with Exoscale Object Storage - July 6, 2021
  • Jibe: Real-time Content Monitoring of Files and Objects for Compliance and Search - November 20, 2020

Related posts:

  1. Using AWS Lambda to Automatically Sync Metadata With S3 Events
  2. GoodBye Google Search Appliance – hello federated enterprise content search !
  3. Getting started using the Enterprise File Fabric with BlockChain for Secure Governance and Compliance of Company Files
  4. Partner Webinar: One-Stop Content Access Across Cloud & On-Premise Storage
CategoriesAmazon S3, AWS, CIFS, Enterprise File Fabric, Jibe TagsAmazon S3, Enterprise File Fabric, Jibe, metadata, python application, S3 API, SMB, Storage Made Easy

Post navigation

Previous PostPrevious Customer Case Study – Data Management and Collaboration for Ceph and File Based Data Storage
Next PostNext Why companies should consider seamless recovery of Office365 and MS Teams file data as part of their CyberSecurity and DR Strategy

Categories

SME Web CIFS

File Fabric Collaterals

White Papers
Video Gallery


Proudly powered by WordPress

We are using cookies to give you the best experience on our website.

You can find out more about which cookies we are using or switch them off in settings.

Powered by  GDPR Cookie Compliance
Cookie Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

You can adjust all of your cookie settings by navigating the tabs on the left hand side.

Functional Cookies

Functional Cookies should be enabled at all times so that we can save your preferences for cookie settings.

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.

Analytics Cookies

This website uses:

Google Analytics to collect anonymous information such as the number of visitors to the site, and the most popular pages.

Leadlander to help build intelligence about visits to our blog to provide accurate and better information for our sales teams about site visitors.

Keeping these cookies enabled helps us to improve provide more targeted services and helps us improve our site.

Please enable Strictly Necessary Cookies first so that we can save your preferences!

Privacy Policy

See our Privacy Policy and Cookie Policy for more information about the information we collect and your rights.