Jibe: Real-time Content Monitoring of Files and Objects for Compliance and Search

The Enterprise File Fabric provides a global file system through which users and applications can transparently access corporate file systems and object storage no matter where they or the storage is located. To provide universal browsing, search, and other services the File Fabric scans and smartly indexes file and object metadata. In this way, no data is copied or replicated, and the metadata is used to build out the high-performance global file system.

Unlike other global file system solutions, the File Fabric does not prevent direct access to the storage – what we call bi-model access. Data can be accessed and updated concurrently through the File Fabric and directly through native file and object-APIs. When objects are updated through native APIs the File Fabric metadata should also be updated to keep indexes and other metadata synchronized.

An earlier blog article outlined three general approaches for keeping metadata up to date, including a specific example using Amazon S3 object notifications. See Using AWS Lambda to Automatically Sync Metadata With S3 Events. This article describes a similar approach we have developed for file systems using the same framework we call Jibe.

 

File System Metadata Notifications

The File Fabric supports several different file systems. Essentially any file system that can be mounted on Linux, or accessed over SMB, NFS, FTP or WebDAV can be connected to the File Fabric. The metadata is loaded using native storage APIs, traversing object buckets, and folder trees. Once indexed the storage is accessible through the File Fabric global file system.

How do we keep the File Fabric metadata up to date when changes are made to the underlying file system?  There are several ways in which we could detect changes in real-time. Some approaches are vendor or operating system specific, and some are specific to the protocol. For SME we settled on using SMB folder notifications. Although it is a protocol-specific mechanism, it provides a lot of control and of course is supported by any file server that supports the SMB protocol.

SMB Notification Mechanism

Jibe makes an SMB Change Notify request to a file server to be notified of specific changes to that directory tree. The SMB server returns a list of file system events that match that request. Jibe makes the request again to obtain subsequent changes. If there are no changes queued the request blocks until changes are available – a “long polling” algorithm.

Implementation Considerations

As we’re using an event stream rather than logs or queues if Jibe is not actively running, changes may be missed; there is no log file to use for recovery for example. We solve this by deploying redundant instances and making any updates that require a restart to each node in turn.

SMB notifications don’t distinguish files from folders. Jibe may try to sync an event as a file and based on the context may subsequently retry that event as a folder.

When a folder is renamed or moved change notifications are only received for the renamed object, not for child items and subfolders. In this case, Jibe will synchronize folder contents as well.

Optimizations include merging multiple events for the same file and ignoring well-known temporary files.

Jibe Configuration

Jibe is a Python application that can run standalone, as a Lambda function, or as Docker image. Settings for Jibe include event “sources” – log files or event streams providing storage change events.

    {
        "endpoint": "https://storagemadeeasy.com",
        "login": "adminuser@example.com",
        "password": "*****",
        "sources": [
            {
                "name": "Provider Name",
                "smb_share": "ExampleShare",
                "smb_dir": "",
                "smb_server": "smb.example.com",
                "smb_username": "smbuser",
                "smb_password": "smbpass"
            }
        ]
    }

Policy Enforcement Use Case

The ability for the File Fabric metadata to be updated from native file and object storage events in real-time enables the File Fabric to take actions as it becomes aware of the new file and objects.

Consider real-time policy enforcement to prevent exposure of personal data and intellectual property or automatically creating archiving copies of file//objects to protect from Ransomware threats.

1. File added to storage –  Using S3 API or file share
2. Jibe Sync detects storage event –  Updates File Fabric metadata
3. File scanned for sensitive data –  Also updates content search index
4. Policy-based enforcement –  Quarantines file and send notification
5. Quarantined file no longer accessible –   (from S3, file share, or user apps)

More Information

For more information about Jibe in your environment contact support@storagemadeeasy.com

 

*Image by Gerd Altmann from Pixabay 

by