No Comments

    Leave a Reply

    seventeen − 7 =

    Secure Delivery Managed File Transfer Using RoboMQ

    Managed File Transfer (MFT) is a reliable and guaranteed delivery file transfer service with all the management, error handling, and recovery features built in for a secure and encrypted file transfer. It supports managed file transfers across any of the file transfer protocols and various file systems (Windows, Linux/Unix, midrange systems, S3, Object Stores, Cloud Storage, etc). The file transfer is secure and encrypted. It’s tracked throughout the transfer as well as in-transit transformation, providing a reliable enterprise-grade MFT platform for managing the secure transfer of data from a source location, to one or more destinations through public, private and hybrid networks.
    figure

    Introduction

    RoboMQ MFT service allows you to transfer files between file systems using a variety of file transfer protocols. The Managed File Transfer platform is built on the strength of RoboMQ Message Oriented Middleware (MOM) providing guaranteed delivery and reliable data transfer. It utilizes the RoboMQ broker for receiving and tracking events, identifying each step of the transfer and in-transit transformation.

    • Support for all file transfer mechanisms and protocols
    • End to end tracking for the file transfers and transformations
    • Ability to break files into messages and integrating the data with APIs. This approach offers integration with non-file based APIs and systems
    • Microservices based approach for data transformation, enrichment, and validation
    • Robust error handling, monitoring, alerts, and Error Analytics

    Setup and Configuration

    The MFT service needs to talk to your RoboMQ tenant, therefore you will need to provide your vhost or tenant ID,  username, password and your preference for secure transfer over ssl.

    To transfer files from one location to another, MFT service requires access to four folders, let’s call them  upload,  inbound,  outbound  and  destination  (you can use any name you want):

    |— destination
    |   |—- archive
    |   |__ error
    |— inbound
    |   |— archive
    |   |__  error
    |— outbound
    |  |— archive
    |   |__  error
    |__ upload
    |—- archive
    |___  error

    Those folders don’t need to be under the same directory. They can, and often do, reside on different servers, file systems or S3 buckets.

    Note that you will need to provide credentials of the accounts for accessing those file systems, and make sure those accounts have read and write permissions on the above specific folders.

    In case you need some files under those folders, but don’t want them to be transferred, you can configure a filter to ignore files with names that match a certain pattern.

    At this point, the MFT service has been configured and is ready to work.

    Workflow

    The following picture depicts a typical MFT file transfer flow.

    Secure Managed File Transfer MFTFIG 1: HIGH LEVEL SCHEMATIC OF A FILE TRANSFER FLOW

    MFT has a file listener that scans the upload folder on a configurable interval. Once an expected file is detected, it will be copied to one or more inboundfolders (multiple MFT services can work together in case the file needs to be duplicated and processed at multiple destinations).

    If a file requires specific validation or transformation, an optional MFT Microservice can perform transformation, validation, or data enrichment on the file in the inbound folder. After successfully processing the file, this component moves the file to the outbound folder.

    Finally, another MFT Microservice will pick it up from the outbound folder and put it into the destination.

    Note that just like with any RoboMQ integration, there could be a whole chain of Microservices performing a variety of functions. This includes transformation, validation, data enrichment, or application of any business logic while the file is in transit from source to destination.

    At each step of the processing, the original successfully processed files are archived in the corresponding archive folder.

    Error Handling, Alerts, and Monitoring

    Upon failure, the current file will be moved to a corresponding error folder. At this point the RoboMQ error handling process kicks in. It has multiple options for tactical, as well as strategic handling of error cases.

    An error message (includes file information) will be sent to an error queue and a notification message (describes what the error is and how it happened) will be sent to a notification queue.

    • An SMS, an email, or a phone call can be triggered to the operations team, or the responsible party, based on the setup of the severity rules and notification mechanism
    • A ticket or a case can be created in any of the case management platforms being used, like ServiceNow, Jira, Salesforce or any other
    • All the errors are saved into a real-time database, which provides an analytical dashboard view of the errors for the determination of long term trends, root cause analysis, and corrective actions
    Error Analytics Dashboard FIG 2: ERROR ANALYTICS DASHBOARD FOR ANALYSIS AND CORRECTIVE ACTIONS

    For more details on RoboMQ Managed File Transfer, please checkout out our Managed File Transfer page.

    Did you enjoy this article? Subscribe below for RoboMQ updates:


    No Comments

      Leave a Reply

      4 × 5 =

      You Might Also Like

      Expandable File Based Integration

      Traditional Managed File Transfer (MTF) systems have become outdated. Their limited functionality, accompanied by an inability to scale alongside larger business needs, has paved the way for a new Managed File Transfer workflow that boasts a microservices architecture with…

      Customize Workflows With File to API Integration

      Traditional managed file transfer (MFT) solutions are good at one thing: performing the core file exchange and transmitting file data. Since MFT came of age in the mid-2000s, the landscape has rapidly changed toward cloud-hosted applications and IT infrastructures have…

      Automate Employee Onboarding with Workday to Active Directory Integration

      Automate Employee Onboarding with Workday to Active Directory Integration

      Workday is quickly becoming the most popular financial and Human Capital Management (HCM) application in the industry. Much like ADP’s clientele, employers use Workday to manage their workforce, onboarding tasks, changes in employment, and the financial aspects of their…

      No Comments

      Leave a Reply

      four × four =