Incoming funds automation in Mass Payment

This RFC describes the solution to integrate the automation of the Incoming Funds into Ebury Mass Payments environment.

Reference Documents

Reference Document Location
EPIC01 EMP - Incoming Recon efficiency improvements (EPIC)
IFGUIDE Incoming funds processes information
DIAGRAMS Diagrams for this RFC

Abbreviations, acronyms, and definitions

  • Autoclient: A SWIFT component responsible for receiving messages from the SWIFT network
  • Core: Current Ebury production environment, not belonging to Ebury Mass Payments
  • EMP: Ebury Mass Payments
  • MT Message: A SWIFT Message Text, in the scope of this RFC, MT940 and MT942
  • SQS: Amazon Simple Queue Service
  • Publisher: EMP Delivery Service
  • Handler: EMP MT94x Gateway
  • BAE: Bank Account Entry (BOS model)

Problem Description

Currently, the reconciliation process in EMP is manually uploading an XML file or a preload movement, by going through the following processes.

Current EMP processes

EMP users want to skip the manual process for a big number of entries, which will be created in the system from MT messages received via SWIFT.

This RFC means to update the reconciliation process adding automatic processes for Barclays accounts, covering 90% of the incoming recon volume.

Desired EMP processes

Background

The creation of incoming recon entries via SWIFT is a process where, across MT files, the bank movements are processed and created in the system.

On average, EMP has about 1800 Incoming Funds per month. About 90% of them are related to Barclays accounts. This RFC will be focused on these entries.

In order to implement the solution, we need to solve 2 issues:

  • How to acquire the MT files from Barclays;
  • How to consume the MT files from Barclays;

In Ebury Core both processes are handled by a set of services and components that are not available to Ebury Mass Payments, as you can see in the diagram below:

Services in Both platforms

Solution

Problem 1: Acquire MT Files from Barclays

Ebury Core Autoclient is already receiving MT files for both Ebury Core and Ebury Mass Payments. These files are made available to FX Suite By design, Ebury Core FX Suite is reading and storing the MT files that belong to EMP, but it does not process them.

This is done in a 2-step process: "Acquire + Store" and "Read + Process"

"Acquire + Store" reads the file from Autoclient, and stores the files in FX Suite database for later consumption, and a backup file in a S3 bucket. For each file a new entry is added on SQS, pointing to the file to be processed, after it's written in the DB. It's meant to be a "dumb" process, responsible for ensuring that Ebury Core has a consistent copy of MT files. No business logic is expected in this process.

"Read + Process" will then read SQS and for each entry it'll fetch the file from FX Suite DB, read it and decide between processing the file inside FX Suite or discarding the file for not being one of the configured accounts. Here is where we apply business logic to decide if a file should be processed and BankAccountEntries should be created or if the files should be discarded.

This proposal is to change the "Read + Process" part of FX Suite, as it's already reading the files to apply business logic. If it finds accounts that belong to Ebury Mass Payments, it would then divert this file to a S3 bucket belonging to Ebury Mass Payments, and in parallel also push a message to a SQS for Ebury Mass Payments

Acquire MT files

A few steps must be done to implement this solution:

  • Configure Autoclient to retrieve MT940/MT942 files for EMP Bank Accounts;
  • Add logic to Ebury Core's EMP-Publisher to identify which Bank Accounts belong to Ebury Mass Payments;
  • Change the Ebury Core's FX Suite "Read + Process" to push the EMP BAE messages to a specific SQS, to be used by the EMP Delivery Service;
  • Creation of the EMP Delivery Service, capable of reading this SQS and doing API calls to a Gateway service in the EMP infrastructure;
  • Creation of a Gateway service in the EMP infrastructure, capable of receiving BAE messages and sending to a specific BOS SQS, to be used by the BOS;

Having the EMP Delivery Service separate from FX Suite and reading from SQS allows us to scale it independently, and allows easier recovery if there's a communication problem between the Delivery Service (producer) and the Gateway Service (consumer). In this case, FX Suite in Ebury Core wouldn't be affected.

That would allow EMP to receive only it's interesting Bank account entries, while maintaining infrastructure independence from Ebury Core.

Variations to this solution

A variation from this solution is to implement the the EMP Delivery Service as a Celery Daemon in Ebury Core, inside FX Suite codebase.

Caveats

The communication between the producer and consumer would be done by the producer pushing an API call using a mTLS encryption layer between the Publisher and the Handler.

The tricky part is the authentication between the services, as currently there's no authentication service available in the EMP environment.

So the communication must be done following one or more of the alternatives below:

  • Enforce communication channels using AWS networking rules;
  • Encrypt communication between services using public/private keys(mTLS);
  • Change from push to pull (consumer polling producer for new messages, and consumer authenticates with producer) - performance would suffer in this case;

Alternatives

We've explored 4 different ways to read MT files from Barclays, detailed in this document.

Option 1: Install a new instance of Swift Autoclient for EMP

Autoclient is a complex system to install and maintain, and also it's expensive. Given the timeframe of the project, where we expect EMP to be out of BOS in 2 years, it would not be worthy to install Autoclient for EMP just for this specific functionality.

Option 2: Use Barclays File Gateway to receive the MT942 files

Barclays confirmed that they can provide MT files through a product called File Gateway, which is basically a Host to Host connection between us and them. This would require the creation and maintenance of a new service to synchronize files between Ebury and Barclays, that would be used only in the EMP environment.

Ebury uses Barclays File Gateway indirectly, by the Quantum system. Quantum handles all the infra and connection between us and Barclays, and Ebury do not have access to the files, as the process is completely internal to Quantum.

Option 3: Receive the MT942 files from Barclays through email

As we already have an integration with Barclays through e-mail on Ebury Core, for creation of automatic Broker Deals, we explored this possibility, but Barclays does not provide the service of MT files through e-mail.

Problem 2: Consume MT files from Barclays

After FX Suite on Ebury Core delivers the files on a S3 Bucket for Ebury Mass Payments, it can also post a message to SQS for Ebury Mass Payments, where FX Suite on Ebury Mass Payments can read asynchronously and process the files.

This process already exists on FX Suite, but it's disabled in the EMP environment. There's one caveat, as the SQS message nowadays contains the id of the file in the Database, and we'll need to do a small change in FX Suite to make it read from S3, instead of the DB.

The change can be achieved and implemented along with a Switch that can be enabled only in the EMP environment. The rest of the process would be the same, with FX Suite parsing and processing the MT File, and adding a BankAccountEntry in another SQS that would be read from BOS for post-processing.

BOS will then pick up this SQS message, the same way it does in the Core environment, and add a BankAccountEntry object for reconciliation.

Consume MT files from Barclays

In summary, these are the steps necessary to implement the solution:

  • Changes FXSuite to capture all discarded MT messages and parse them to a new Queue which Publisher will consume from
  • Add a new SQS that will be used by FX Suite to send the serialized BAE
  • Create a new service on Ebury Core environment, EMP swift message publisher (from now on, Publisher), that will consume BAE messages from FXSuite and send them to EMP environment if they belong to EMP funds
  • Create a new EMP Gateway Service (Handler), exposing an API endpoint to receive BAE messages and send it to BOS SQS - this service will trick BOS into believing it received the BAE Message the same way Ebury Core did
  • Test the whole process

Alternatives

No other alternative was found following this architecture.

Tactical Solution

Considering the urgency and time constraints of this project, a tactical solution has been selected to address the pressing need for system integration. One of the main factors driving this decision is the need to free up resources on EMP, as they are currently occupied with manually uploading files into the system. By implementing a lightweight approach, we can overcome the challenges associated with the legacy systems (BOS and FXSuite), within a shorter timeframe. This approach will help the EMP manually process and could also benefit one of the most promising profitable areas(EMP) in their daily process. While pursuing this tactical solution, it is important to acknowledge that we will introduce a certain level of technical debt into the existing system. Additionally, the workflow will be modeled after Ebury Core's approach, but adjustments will be necessary due to differences in infrastructure between the environments.

Integrating legacy systems inherently involves working with outdated technologies, outdated coding practices, and limited documentation. These factors contribute to the accumulation of technical debt, as the tactical solution may require temporary patches, custom configurations, or non-standardized approaches to achieve the desired integration.

While it is essential to acknowledge the potential increase in technical debt, we have carefully considered the trade-offs and made this decision based on the urgency of the integration project and the benefits it brings to our organization. The tactical solution allows us to quickly establish connectivity and overcome the operational challenges we currently face.

We can point out the benefits of the tactical solution for this approach: - Time savings: Implementing a tactical solution allows for faster integration, enabling us to address the pressing need for system integration within the given time constraints. - Resource optimization: By freeing up resources on EMP Team, which are currently occupied with manual file uploads, we can allocate them to more valuable and strategic tasks, improving overall operational efficiency.

Concerns

We are connecting two softwares that are already in a deprecated state FXsuite on Core, and BOS on EMP environment. An ideal project we will need an external parser process outside the current deprecated FXsuite code. The decision to add a custom parser on FXSuite raises legitimate concerns that must be carefully considered, particularly in relation to future projects involving the parser MTMessage. By introducing this custom parser, we need to anticipate and account for potential issues that may arise when working with new projects that rely on the MTMessage parser. These concerns become especially pertinent when considering the ongoing Request for Change (RFC) to migrate MTMessage to the ISO-Message format.

Moreover, the process of migrating the MTMessage parser to the ISO-Message format, as indicated in the ongoing RFC, may introduce additional considerations and challenges. The future RFC may be reviewed and modified to accommodate the changes required on the EMP Recon project, as a result the approach on EMP can be reused as it is today avoiding future re-work on EMP side.

By addressing these concerns, we can better prepare for the future challenges associated with the custom parser and the migration of MTMessage to the ISO-Message format. This approach will help us navigate potential re-implementation efforts, maintain system compatibility, and ensure a smooth transition for both existing and upcoming projects relying on the parser.

Caveats

Adding new bank accounts into the current Autoclient will increase the volume of received messages.

We will be creating a single point of failure for both EMP and Core. If Autoclient fails for some reason, both Ebury Core and Ebury Mass Payments will stop receiving BAE messages. In this case, EMP operators would have to revert to their current manual process.

There's a seasonal factor where, by the end of the year, Ebury receives lots of forwards, so we expect to have a delay in processing on both Ebury Core and EMP, as Autoclient and FX Suite in Ebury Core would be overloaded.

In Ebury Core there's a component called Transformer, that parses MT Messages to a serialized json, This component is in the Core Environment, so it makes necessary a deploy on EMP, as a result we choose to use Transformer component on the Ebury Core environment. Taking advantage of the Transformer process to parse MT Messages to use in the EMP Environment. Another component called Categorizer, that uses Machine Learning technology to add more metadata into the MT files, allowing a more accurate description of the incoming funds, due to the number of different types of funds. It's not clear yet if this component will be needed for parse EMP BAE messages, but in any case, it's use is optional, and if necessary we could use this component inside Core environment before send the BAE Message to EMP later on.

In the event that we encounter challenges or issues with the custom parser integration on FXSuite, an alternative approach could be considered. We can explore the option of sending MT Messages directly to the EMP environment and performing the parsing process there. However, this alternative approach would require deploying a transformer component on the EMP to handle the message transformation and parsing tasks.

By considering this alternative approach, we can potentially overcome any obstacles associated with the initial custom parser integration on FXSuite, ensuring the successful parsing of MT Messages within the EMP environment and address any potential risks involved.

Operation

We expect to reduce the amount of manual work from the operations team for the incoming funds reconciliation, since the process will automate 90% of the movements.

On tech operations, as we're introducing new processes to receive and process the files inside Ebury Mass Payments, we'll need to add new alerts and runbooks to monitor the automated process and handle possible exceptions.

Security Impact

Autoclient currently receives data from both Ebury Core and Ebury Mass Payments, and we expect to divert only EMP BAE to the EMP environment.

The communication between all the new services will need to be configured, which means opening ports and configuring network routes.

Authentication services are not available in the EMP environment, we choose use mTLS.

Auditing MT94x Messages

Auditing for the MT messages in the EMP environment will be handled natively by ebury-audit library, the same way it does for Ebury Core.

Auditing capability is turned ON in the EMP environment, and any Create/Update/Delete signal in the MT94x models will be reflected in the audit database.

This is controlled by a hard-coded setting inside fxsuite.settings.py, which already contains the MT940 and MT942 messages.

Given that this is a hard-coded setting, the same models will be audited in both platforms.

Performance Impact

It is expected a performance impact in the BOS in the EMP environment.

It's expected an increase in volume in Swift Autoclient in Ebury Core environment, as it'll also be receiving entries from Ebury Mass Payments accounts. It's important to note that currently Autoclient is already receiving data from EMP accounts, but not from all of them.

Developer Impact

Developers from the Money Flow teams will have to support new services in 2 different architectures at the same time.

Deployment

The deployment pipeline of BOS will need to be reviewed for EMP, as we have to enable new services/celery queues for the new processes.

Dependencies

[1] Autoclient configuration

[2] Autoclient Balancer Github repository

[3] SWIFT Service Github repository

[4] Transformer Swift Github repository

[5] Categorizer Github repository

[6] Possible ways to read the MT files from Barclays

[7] Diagrams used to create this RFC

[8] BOS settlements.consumers.sqs_bank_account_entry_consumer.SQSBankAccountEntryConsumer

[9] FX Suite services.publishers.BankAccountEntryPublisher

[10] Ebury Audit repository

[11] RFC for auditing