Publishing multi-payment events
Multi-payment processing creates new payment records and updates some information that must be shown to the user. For Ebury 2.0 we must publish domain events that will be consumed by Payment Query Service to update the read model and to use it as the source for queries by EBO (and API) to display multi-payment related payments.
Link to the "Comments" part of RFC
Abbreviations, acronyms, and definitions
- execution date: a date on which a payment is sent towards its beneficiary via an intermediary bank
- delivery date: a date on which a payment arrives in the beneficiary's account
- MP, multi-payment, or mass payment: a trade that can contain more than one payment for multiple beneficiaries.
Problem Description
As described in Smart Date service integration, multi-payment file has a single field named "Value Date" that contains the execution date. FrontierPay treats this execution date as the delivery date and manually updates the execution date when the file is uploaded. We are adding a parameter to indicate if "Value Date" contains execution date or delivery date and to calculate execution date automatically. By doing so we want to store both the calculated execution date and delivery date for traceability and reporting needs. We need a place to store the delivery date.
The initial idea was to avoid updates in BOS and to publish a domain event and update the Payment Query Service read model. But this service is not ready yet and we need to move forward with Smart Date service and BOS integration.
Background
Solution
Create a new table with one-to-one relation to the BeneficiaryPayment table. We called it PaymentDelivery in the spirit of splitting BeneficiaryPayment into multiple "domains"
Set up a Change Data Capture process in PostgreSQL that captures all changes at the database level and produces Kafka events. This process is decoupled from the multi-payment processing and completely asynchronous. It will not have a direct impact on performance, but the CDC process may generate additional load on the server by mining database logfiles.
The downside is that Kafka events produced are about row-level changes, not domain events. An additional step of enriching the message will be needed. That also means that there will be two events captured: one for BeneficiaryPayment and one for PaymentDelivery. Debezium connector can be instructed to enrich CDC events with transaction metadata and the two events could be grouped.
Alternatives
Storing delivery date in the BeneficiaryPayment table
Store delivery date in the BeneficiaryPayments table and use Change Data Capture for producing Kafka events.
Seems to be the most straightforward solution, but there is a resistance to modifying the already bloated BeneficiaryPayment model.
The database upgrade process needs special care to avoid downtime. Memory usage might increase for all code using BeneficiaryPayment regardless if they need the field or not.
Publishing events to Kafka explicitly using Kafka REST Proxy
Events can be published in Kafka using the HTTP protocol and JSON format. We could publish the domain events that are ready for consumption by the Payment Query Service.
Connecting to the proxy and publishing event is a synchronous task and will have an impact on
multi-payments performance. POST /topics/topic_name endpoint supports bulk mode and several
records may be posted at once to minimize performance impact.
A compensation event might be needed if we create 9 payments and publish events to Kafka and the 10th payment fails. The database transaction might roll back and lead to inconsistent data in the BOS database and Payment Query Service.
This solution will decrease the availability because of the tight coupling to additional components (Kafka and Kafka REST Proxy).
Publishing events to Kafka explicitly using other libraries
Using Kafka by introducing new dependencies in BOS:
-
Ebury Event library uses fastavro dependency for Kafka message serialization that requires Python3. It can't be used for BOS running on Python 2.7
-
Confluent Python Client supports Python >= 2.7 and supports Avro messages using a different (less performant) library. This approach has one less component (Kafka REST Proxy) and should improve performance and availability. Other pros and cons are the same as for the Kafka REST Proxy option.
This approach conflicts with our understanding of Ebury 2.0 Ambassadors: Ambassadors should be the tool to integrate deprecated service (BOS) with Ebury 2.0 instead of integrating them directly. Comments are welcome.
Publishing events to Kafka asynchronously through Celery
Instead of calling an external Kafka REST Proxy, put a message into the Celery queue (in the same BOS database).
If a full Kafka event is stored, it will create an additional load to the database by storing additional data. If only the payment ID and delivery date are stored, the celery worker will be required to fetch records from the database to enrich the Kafka event.
Storing a message in the Celery queue is an additional SQL statement and the overhead on the the multi-payment process is comparable with storing the delivery date in PaymentDelivery table.
Celery queue cleanup process has caused incidents before, see 20210128_B: Outage - BOS & EBO & API
Caveats
- Change Date Capture produces events about row-level changes, not domain events
- Payment Query Service does not exist and domain events have not been defined
Performance Impact
Storing delivery date in another table will require:
- an extra SQL operation to store it
- in addition to the delivery date, some extra space for the ID field and the database index
- an extra join in the database
Developer Impact
Remember to select_related the new table to avoid lazy fetching.
Data Consumer Impact
N/A
Deployment
Creating a new table will not cause any downtime.
Dependencies
None