How To Add a New Field to Payments of API Async Mass-Payments

Prerequisites

None

Reference Documents

Reference Document Location
MPF01 Async mass payments

Problem Description

To extend their business case or improve their integrations, our customers wish that our API POST /mass-payments [MPF01] endpoint support new fields.

Lately we have added beneficiary and account id fields to payments so that beneficiaries are identified by id.

In the future we need to add a new field where our customers can store any data they wish and that they can use to match payments created in mass payments with data in their internal systems (field external_reference_id).

New fields that are independent of payment instructions (see here) are straightforward to add in our systems by attaching them to the mass-payment data model, but fields that can have different values depending on each payment in the mass-payment require more implementation steps.

This document focuses on the latter, provides technical details about how BOS is processing a mass payment CSV file and list the implementation steps needed.

Background

Mass payments are uploaded in BOS as CSV files. When submitting a multipayment in BOS, the parsing is done in trades.models.multipayment.Multipayment.parse_multipayments_file_bulk. A FileParsingTemplate is used to parse the CSV. Each FileParsingTemplate has a set of FileParsingTemplateRule objects, stored in the database. FileParsingTemplateRule is a mapping between a field name and an index in a CSV row. FileParsingTemplate is selected based on the CSV header, more specifically based on the number of items in the header (i.e. CSV columns). Both flows (BOS API endpoint and BOS views) are using the same strategy to find the parser.

Next, the parser is validated. What this means is that the parser must support the following mandatory fields.

    MANDATORY_DESERIALIZED_FIELDS = [FIELD_CURRENCY,
                                     FIELD_PAY_AMOUNT,
                                     FIELD_SWIFT,
                                     FIELD_IBAN,
                                     FIELD_ACCOUNT_NUMBER,
                                     FIELD_VALUE_DATE,
                                     FIELD_PAYMENT_REFERENCE,
                                     FIELD_DIRECTION
                                     ]

Solution

When adding new fields to mass payments we must maintain backwards compatibility which means that all new fields should be optional. In order to add new fields, we have to create new FileParsingTemplate records to the database. When doing that, we need to make sure that the mandatory fields used in validation are also submitted, but if they are not used, we can submit them as empty (e.g. when API converts json to CSV), so the parser can be validated successfully. BOS only checks their presence.

In theory, with each new optional field that we add, we double the number of FileParsingTemplate records in the database because we need to account for all combinations of optional fields. This happens because we can’t introduce optional fields in a CSV file and maintain backwards compatibility. Moreover, we need to make sure that there is no conflict between FileParsingTemplate. There can be only one FileParsingTemplate that can handle a certain number of columns.

Luckily, sometimes we can reuse existing templates. API gets the mass payment instructions from customers as a json payload, where fields are named, so it knows which fields will be used. API then converts the json payload into a CSV file and uploads it to BOS. During this process, API will add to the CSV any optional fields and set them to empty, therefore for mass payments submitted from the API, we can try to use a smaller set of FileParsingTemplate records. We identify the FileParsingTemplate that is used only by the API and then we simply increment the number of columns supported by it with a BOS migration and, if needed, add a FileParsingTemplateRule. This operation must be done in two steps however to avoid incompatibilities. First, clone the FileParsingTemplate and its FileParsingTemplateRules, increment the number of columns and add a new FileParsingTemplateRule to it if needed. Then, after the required changes have been done to API webapp to support the new field, the original FileParsingTemplate and its FileParsingTemplateRules can be removed.

If in the future we want to allow customers to submit mass payments with the new field from EBO for example, then we need to make sure that we have the right combination of FileParsingTemplate in the database.

We need to follow the below steps for adding a new optional field to each payment of a mass-payment submitted with the API.

  1. Get a report of FileParsingTemplate records in the production database to see if there would be a conflict when adding new ones.

  2. Create BOS migrations to clone existing FileParsingTemplate or update existing ones. When cloning, make sure to clone the associated FileParsingTemplateRules and assign them to the new template.

If there is a FileParsingTemplate that is used only by the API and supports all the required fields, we will be able to extend this template. However, in order to preserve backwards compatibility, the operation needs to be done in two steps. In this step, clone the FileParsingTemplate along with all its FileParsingTemplateRule, incremenet the number of supported columns and add a FileParsingTemplateRule if needed.

If we want a mass payment with the new field to be uploaded from EBO or BOS, clone all the existing templates that are used by them and add the new field. You will basically add the new field to all combinations of supported fields.

  1. If we want to store the new field on the payment object, create a migration to add the new field to the payment records.

Identify teams that own consumers of any domain or change events that include this new field and inform them about the schema change.

Also identify teams that replicate BOS data into other systems (e.g. data warehouses), such as the analytics and the data teams. Inform them about the proposed schema change.

  1. Parse the new field in BOS multipayments logic. Add the field to the parser. Add any validations required.

  2. Send the new field to the payment object during the processing of the mass payment. In settlements.tasks.create_payments._get_payment_from_dict, send the new field to the BeneficiaryPayment object from payment_dict. If this field affects the processing in some way, make sure to start any needed background processing. This step could involve creation of new celery queues.

  3. Add the new field to the CSV builder in API webapp.

  4. Add new field to API webapp POST /mass-payments schema. E2E test the endpoint. Update the public docs with this field.

  5. Consider writing performance tests for the POST /mass-payments endpoint if needed, depending on the impact or size of the new field.

  6. If the FileParsingTemplate is only used by the API, it is the time to create a BOS migration to remove the original FileParsingTemplate with its rules, that we cloned in a previous step.

  7. Add the new field to each received payment in GET /mass-payments/<mass-payment-id>/payments-received response. In API webapp, convert back from CSV to json. E2E test the endpoint. Update the public docs with this field.

  8. If we want to store the new field on the payment object and show it to customers, update the payment serializer in BOS to output the new field.

  9. If we want to store the new field on the payment object and show it to customers, add new field to each payment in API webapp GET /payments. E2E test the endpoint. Update the public docs with this field.

  10. Add new field to API webapp POST /multipayments to maintain consistency with the mass-payments endpoints. E2E test the endpoint. Update the public docs with this field.

  11. Consider writing performance tests for the POST /multipayments endpoint if needed, depending on the impact or size of the new field.

  12. If we want to submit independent payments or trade payments with the new field, send the fied to the payment object from BOS add payment to a trade view to support creating independent or trade payments with the new field.

  13. If we want to submit independent payments or trade payments with the new field, add the new field to API webapp POST /payments. E2E test the endpoint. Update the public docs with this field.

  14. Consider writing performance tests for the POST /payments endpoint if needed, depending on the impact or size of the new field.

Service Ownership

New Service Service Name Service Owner
No BOS (multipayments) Many
No API WebApp MBL Team
No EBO ONL Team

Alternatives

Redesign the CSV upload and field parser.

The CSV does not dictate the parsing logic, it's just a data interface.

Several upload endpoints (parameterized based on the supported field set) would pre-process the data and convert from CSV to an internal structure, which is then validated. An endpoint knows the CSV columns based on its type/parameters, not the CSV column number.

Caveats

A new FileParsingTemplate could conflict with an older one.

Operation

N/A

Security Impact

N/A

Performance Impact

Might increase size of the mass payment payload.

Developer Impact

N/A

Data Contracts

If new fields are stored in the database (e.g. on payment records), consumers of any domain or change events that include this new field must be identified and made aware of the proposed schema change.

Data Sources

N/A

Deployment

N/A

Dependencies

N/A



Based on RFC Template Version 1.1