Flexible Broker Deal allocation
Prerequisites
None.
Reference Documents
| Reference | Document Location |
|---|---|
| TL108PRD | TL-108 - PRD |
| EVENDA01 | Event-driven architecture |
| MIGBOS01 | Migrating BOS |
| TREDTA01 | Treasury Domain - Target Architecture |
Problem Description
Through the analysis of several FX initiatives, it has become apparent that the current backend logic that handles the integration between BOS and Quantum needs to be reviewed. The initial design did not accommodate scenarios where Broker Deals can be booked from different sources or reallocated.
The main problem with that is that the current solution inserts elements in a buffer, and those are sent periodically which means that, when the periodic task takes those elements, they will be sent with the information at the execution time, instead of sending the different updates as they happen.
The other main challenge with the current implementation is that, once the element is inserted in the buffer (for updates), we know that it has changed, but we lose track of what has changed. This limits the logic we can use for scenarios in which we need to know exactly what was modified, creating that way some flows in which the reporting is inaccurate and need manual intervention.
The goal of this is to provide a solution that enables the flexibility required to deliver more complex FX initiatives.
Background
Whenever a broker deal is created, updated, or deleted an element is included in the 'TMSBuffer' table. A periodic task (QusSendInstrument) will then pick the unprocessed elements there and proceed classifying them. Once classified, they are passed onto the serializers. These are the ones containing the logic of what must be sent in every field.
All serialized elements are then stored in one (or more) records under the table 'QusMessage'. This will publish the information in Kafka, that will be picked up by the Treasury Service.
Treasury Service will parse the instruments and store them in the database. Later, it will publish those instruments to another Kafka topic that will consume the Quantum Gateway service.
Quantum Gateway will take the information, generate the QXT files with it properly presented, and then will proceed uploading the file to Quantum's SFTP.
Some problems with the current solution are that: - given that the information is sent through a periodic task and not in real time, there might be scenarios in which two 'updates' happens in the window between executions and, even though the information sent is correct, we are losing traceability of the intermediate changes. - the logic lives completely in BOS, making it slightly harder to test and release. - even though it is based on events, it doesn't entirely follow the event-driven architecture.
Solution
To tackle this problem, we are going to follow the target architecture for the Treasury domain, therefore, we are going to implement a new service, the ‘Treasury Instrument Aggregation Service’.
Treasury Instrument Aggregation Service
We need this new service to process multiple Kafka messages to extract the information published by BOS for the different tables involved.
This new service will take that information and dump it in the database. The goal of this is to have a local copy of the BOS database (to have a self-contained service as defined in the 'Migrating BOS' RFC) for the tables required to construct the QXTs.
At this point we can proceed developing the serializers as done in BOS. For now, we want to mirror the current behavior, so we can have a testing phase in which you can check the new service running parallely to the BOS process. This way we can compare both outputs before activating this flow.
For the aggregation and serialization of the elements we might be able to use the capabilities of 'ebury-hela'.
Treasury Service
From this side, we will need a new topic and a new consumer. The mission of this one would be to parse the information, enrich it when needed, and publish it in the format that Quantum Gateway will need.
Quantum Gateway
Here we'll need again a new topic and a new consumer. Once the information reaches this point, and it is parsed, the service will need to prepare it in a way the Quantum API processes it correctly.
Here, we will need to implement a retry solution that will allow the service to try and send the information through the API several times in case of failure.
Here's a graphical representation of how the new solution would work:
Service Ownership
| New Service | Service Name | Service Owner |
|---|---|---|
| No | BOS | FX Teams |
| No | Treasury Service | TRE Team |
| No | Quantum Gateway | TRE Team |
| Yes | Treasury Instruments Aggregator Service | TRE Team |
Alternatives
The other available option would be to modify and extend how BOS currently works. For that, we would have to modify the current process that takes the elements and sends the elements, 'QusSendInstrument'.
To do so, we would need to still use this periodic task, so it would process the elements, send them to their proper serializer, and having those serialized published to a Kafka topic. After that, the Treasury Service would have to be modified to pick them up, process them and send them to Quantum Gateway to be sent as QXT files to Quantum.
This approach is not appropriate as it wouldn't be aligned with the Target Architecture for the Treasury Domain. On top of that, it wouldn't allow us to completely decouple the logic from BOS, as the serializer logic would still be living and managed in BOS.
Caveats
This solution would require an extra effort for two things:
- it requires setting up a new service
- it establishes the basis for decoupling the logic for the rest of the instruments from BOS
Operation
This new service will consume messages from topics published from internal services (BOS). Then it will interact, through Kafka, with some more services from the Treasury Domain (Treasury Service, and Quantum Gateway).
Security Impact
The proposed solution will read information from BOS (through Kafka messages) in order to generate the instruments to be sent to Quantum. As it requires access to sensitive data, it requires the same access restrictions used on the BOS side.
Performance Impact
For the new service, it will be considered the fact that an update over an existing element could happen at the same time it might be required to read from that element, however, this won't happen over the same row, so no locks must be controlled there.
From the Treasury Service's side, we don't expect this to have an impact as it would use different pods for processing the information coming from this new service.
In Quantum Gateway, performance might be affected by the Quantum API response times, however, this usage could be mitigated by using different pods, or by parallelizing the API calls.
Developer Impact
The broker deal communication from BOS to Quantum will change as it will pass through the new service. Once it goes through this new service, even though it will go through the Treasury Service (and Quantum Gateway), it will not use the same flow, as the end goal is to use the Quantum API instead of generating the QXT files to be uploaded to Quantum's SFTP.
Data Contracts
There will be another data contract between the Treasury Service and Quantum Gateway (that will not collide with the existing one). Also, we will need a data contract between the new Treasury Instrument Aggregator Service and the Treasury Service.
As both of them are new, we don't have to worry about compatibility.
Data Sources
The input data will come from BOS. Once it is received in the Treasury Instrument Aggregator Service, it will be compiled (similar to the serialization step done in BOS), and then prepared to be sent to the Treasury Service, Quantum Gateway, and finally to Quantum via API.
Deployment
This new service will be active for some time during the 'comparator' phase to check that the output matches with the information being sent through the current process.
Once we want to activate it, we will need to disable the sending of broker deals through the 'QusSendInstrument' task, and proceed to activate this service. For that, we will make use of the 'ebury-manifests' capabilities, and therefore, no help from the SRE Team will be needed for the activation step.
Dependencies
None.
Based on RFC Template Version 1.1