Auto-Trading with BarX in the EMP environment
Problem Description
Currently, trade booking in the Ebury Mass Payments (EMP) environment is done manually by the users, by going through the following process:
EMP users want to skip the manual booking process, by requesting the system to automatically book broker deals for specific currencies, and under specific configurations.
The business already decided to use Barclays (BarX) as their liquidity provider for this objective.
Background
In order to run the auto-trading feature for EMP, we need to connect FX Suite with the liquidity provider, which is BarX in this case.
This connection is done through the usage of the FIX protocol, which is currently not available to EMP due to the following factors:
- The EMP environment was created manually, before acquisition from Ebury, when they still were FrontierPay;
- The EMP environment does not have Quick Fix Service (QFS) installed;
The fix connection is available to Ebury Core, due to differences in the infrastructure and services configuration. So, currently we have this picture in Ebury Core:
And in Ebury Mass Payments we have this infrastructure and services:
Solution
In order to enable the auto-trading feature in the platform, a few things must be done:
- A staging/pre-production environment must be created in order to test BarX connection (BarX possesses a sandbox environment capable to provide test FIX sessions);
- QFS must be configured to connect with BarX servers (new config files and adjustments in dockerfiles);
- Stunnel must be included in the EMP environment (needed for the FIX connection, TLS 1.2 encryption);
- QFS must be installed in the EMP environment;
- EMP Network must be adjusted (firewall, routing, port opening) to allow this connection;
- FX Suite must be able to connect to QFS (via Pub/Sub protocol using Redis);
- FX Suite must be configured to use BarX as a liquidity provider for specific currencies and within limits;
- Test trades must be done by the users in staging to confirm the system is behaving correctly (There is not possible to do test in production for a single account);
- Redis capacity/dimension must be checked, so we can be sure it'll be able to handle the volume of Pub/Sub messages expected (which we don't know yet);
The expected final EMP infra design will be this:
We'll also need to:
- Deploy Jenkins in Ebury Mass Payments infrastructure;
- Create a new IaaC pipeline to create the ECS instance where the QFS service will run;
- Create a new pipeline to deploy/update the QFS service into the ECS instance;
- Create a new playbook to deploy updates in the QFS service (updates in FIX services historically are complicated).
- Develop a new Business Continuity Plan (BCP) to handle the scenario where BarX quotes are not available;
- Create an EMP staging platform connected to BarX sandbox in order to do tests
Alternatives
Wait for the new pricing architecture
There is an RFC approved to change the current architecture of the pricing stack, which will rely on Kafka and QFC instead of QFS, but this is a work in progress, and also this change is focused in smartTrade (another liquidity provider), not in BarX.
- The new architecture relies on QuickFix Connect (QFC) - BarX supports is not implemented there yet;
- The new architecture needs Kafka, which EMP does not possess (for now);
- The new architecture is untested and unproved - while it has great potential, its problems are still unknown;
- We do not desire to use EMP business as a test/playground/sandbox for the new architecture;
We don't know how long this change in architecture will take, and we also don't know how long it'll take until we implement BarX support inside QFC.
To be clear, BarX connection is not legacy, and it's actively used as a backup to smartTrade. The only legacy part we have here is QFS.
Caveats
As BarX in Ebury Core is part of a Business Continuity Plan, it also must have a BCP for Ebury Mass Payments.
With BarX being the only automated booking facility on the Mass Payment platform, after its implementation the MP business will be able to increase its capacity. If there was a failure of the BarX feed then, as per the existing process, the connection would be disabled and MP would defer to manual booking as it does currently. The MP business have acknowledged this and will ensure that they are in a position to meet demand if it was necessary to revert to manual booking. If it was felt this was too large a constraint on their ability to scale and dependant on timescales, they would explore the option of adding a SmartTrade implementation to their environment which would be a separate piece of work outside of the current scope.
New infrastructure environments for staging and production must be created before we deploy this solution, as currently the EMP infrastructure is not handled by IaaC or Terraform Modules. So this project has a dependency on another project from infra.
The current gap between the EMP infrastructure and Core infrastructure is large, and there's a plan ongoing right now to bring the EMP environment up to date. This means installing Jenkins, Prometheus, Grafana, Vault and other components not directly related to this project.
EMP already has a signed contract with BarX enabling us to create a new FIX session with them. BarX is waiting our contact to start working and validating the solution with us.
Operation
Impacts on Support, SRE, TFT, Pricing.
As QFS is a critical part of the pricing infrastructure, so new alerts in Alert Manager and their respective runbooks must be created to monitor the EMP QFS instance.
A new index in Kibana will need to be created, so we can isolate the logs from QFS from other system logs.
FX Suite needs Prometheus for metrics and alerts regarding quote and booking flows.
Security Impact
The communication between QFS and BarX will need to be configured, which means opening ports and configuring network routes.
BarX requires at least TLS 1.2, so the QFS connections are stablished through stunnel.
Performance Impact
We expect to have a performance impact in the current EMP Redis server, as it'll be used as a Publisher/Subscriber point for quotes and booked trades between FX Suite and QFS.
FX Suite capacity will need to be evaluated as well, as we expect it to handle BarX trades.
Developer Impact
Developers from the pricing team will have to support 2 different architectures at the same time: a new one in Ebury Core using QFC along another one in Ebury Mass Payments using QFS.
This new architecture can be used in the future for Ebury Mass Payments, if the business decide to enroll in the SmartTrade service as well, but this is outside the scope of this RFC.
Deployment
The release pipeline of QFS is already complex in Ebury Core, where there's a lot of automation in place.
We expect it also to be complex in EMP, but how complex will depend upon which solution option we decide.
A new deployment plan will need to be created to support specifically the EMP environment, as there will be only BarX as the liquidity provider. We won't be able to use the same plan that exists for Ebury Core, due to the fact that Ebury Core relies upon having multiple liquidity providers (BarX and smartTrade) to allow a smaller downtime during maintenance.
QFS is very stable (only 6 releases during 2021, and only 1 during 2022). However, during releases, the FIX connection is shut down, so BarX will need to be disabled in FXSuite before the release, and EMP users will need to be notified about the scheduled downtime. In Ebury Core, the release takes less than five minutes, so assuming we'll have the same level of automation in EMP, we can consider it'll take about the same time.
Dependencies
References
[1] Pricing and Trading - EMP - SignOff
[2] How to cover trades in BarX, EMP
[3] smartTrade Internal Book RFC
[4] QFS Deployments for Ebury Core (Confluence), same link post-confluence migration (VPN required).
[5] SRE 3-Point estimation (initial review, in progress)
[6] EMP Parity with prod document
[7] QFS Github repository
[8] QFC Github repository
[9] EMP Parity Plan RFC (in progress)