EMP - Smart Payment Execution Date
This RFC describes the solution to integrate the Smart Payment Execution Date service into Ebury Mass Payments environment.
Abbreviations, acronyms, and definitions
- Ebury Core or Core: Current Ebury production data, infrastructure and services.
- EMP, Ebury Mass Payments or Mass payments: Current Ebury mass payments production data and services.
- IaC: Infrastructure as Code.
- PQS: Payment Query Service.
- UI: User Interface
Problem Description
Smart Date service has been implemented following RFC: Smart Payment Execution Date.
However, even though this service was exclusively built for EMP business, it was implemented under Ebury Core infrastructure. With the need of having both environments independent of each other, and running the same services, the Smart Date service needs to be part of the EMP system and without any dependencies to Ebury Core system.
Background
Smart Date service calculates when a payment needs to be executed in order to be delivered in the specified date by the user. Payment Query Service is used in the multipayment delivery date workflow, to store and retrieve information about the payments expected delivery dates, to be displayed in EBO. There is no dependency between both services, however, both are needed to have the delivery date functionality.
Currently, this service is deployed, but not in use, on Ebury Core in production, but is available on Ebury Core staging,
where it can be tested for the first use case, salary payrolls. PQS can be enabled, so it can be queried from EBO, by
activating EBO waffle switch: enable_payment_query_service. Only users with permission
EBO_MULTIPAYMENT_SELECT_DELIVERY_DATE can see the delivery date choice on EBO.
With the migration of Smart Date service to EMP environment all the infrastructure needed to have the service running on EMP needs to be implemented. Also, PQS needs to be available in the EMP environment, as Smart Date Service and execution date calculation can't be used without it.
On Ebury Core the services, Payment Query Service and Smart Date Service, are already deployed on kubernetes.
Current design on Ebury Core:
Solution
With the upgrade of EMP infrastructure, Smart Date Service and Payment Query Service, will be replicated on EMP, and deployed on staging and production. Then some configurations will need to be added.
The Redis Server configured as an in-memory store for Smart Date, configured with a persistence strategy will be used to
store countries holidays and cutoff times. To populate it, a kubernetes job will be created on EMP, which will run once
per month, to fetch and store the holidays. Also, the cutoff times are uploaded from a csv file, which has a column
entity that needs to have its values updated to the entities on EMP.
There are some variables that need to be added to EBO and some configuration settings that need to be added to each instance of the service to point to the right services on EMP.
To make the service available to users, permission EBO_MULTIPAYMENT_SELECT_DELIVERY_DATE should be given to the
selected users, which is currently done trough shell. A functionality to add permissions to users from the UI may be
added to facilitate the process. Also, EBO waffle switch enable_payment_query_service needs to be enabled.
With the services available and configured on EMP, a test phase should take place in staging. After the testing phase is finished the services can be made available in production for a small group of testing users.
There is also to create a monitoring plan in case it is detected some malfunctioning in any of the services.
Adjustments needed in BOS and EBO to integrate the services are implement in the current version available on Ebury Core, as described on RFC: Smart Date Service Integration Phase B, Adjusting BOS and Adjusting EBO.
Alternatives
Use service on Ebury Core infrastructure
A possible alternative is to use the current service deployed on Ebury Core, where the service is ready to start being tested by a group of users.
In parallel, the infrastructure to deploy the service on EMP environment would be set up. When finished, the services would be deployed and the database needed for the PQS service migrated. As in regards the Redis Server, it could also be migrated, or it can be created and populated again, as it will only store the holidays.
In one hand, this would make it possible to have the service available faster, as everything is already deployed.
On the other hand, while the service was not fully available on the EMP environment, it was still dependent of the infrastructure on Ebury Core, which is what we want to avoid with this EMP system upgrade. It is also adding some extra work, as after the infrastructure is set up on the EMP environment a migration of the service already being used on Core should be executed.
Caveats
Currently, the EMP infrastructure upgrade is still a work in progress, and this solution totally depends on it.
Operation
Impacts on Support, SRE and Payments team.
The addition of the functionality to assign permissions to users from the UI will remove the need to request to support team to add those permissions.
New Kibana index will bre created to isolate the logs of the Smart Date Service. Also, a new Grafana dashboard and Prometheus alerts will be created for monitoring the service. As new alerts are going to be added their respective runbooks must be created.
Security Impact
The Kafka events are encrypted in-transit (TLS) and at rest (MSK). The read model database is also encrypted at rest.
Authentication and authorisation against Smart Date Service and Payment Query Service API is not needed for the current version of the service. If a new authentication solution is proposed on ow to protect internal APIs the relevant services, including Smart Date and PQS need to be revisited.
In order to have access to this service, and for the value date choice to show up, the users need to have the
EBO_MULTIPAYMENT_SELECT_DELIVERY_DATE permission.
Performance Impact
Payment Query Service read models will be optimised for the multi payment delivery workflow use case. It is expected to provide better performance than BOS endpoints.
On the other hand, adjustment to BOS endpoints will include querying additional model. Solution should optimize additional data retrieval to minimize impact on performance.
The biggest performance impact is expected on EBO side due to the sequential process of retrieving payments first from BOS, extracting payment ids from the response, retrieving expected delivery dates from PQS and finally merging all data together.[2]
Developer Impact
Developers from payments team will have to provide support to a new service available in the EMP environment.
As this is a service to be used exclusively for EMP the current service and infrastructure on Ebury Core can be removed.
Deployment
The new services will only be used by other Ebury applications and do not need to be made available publicly. They will
be deployed into a namespace in the Kubernetes cluster with label networkArea: internalservices (formerly, natonly in
the legacy infrastructure). There the workloads will have only access to natonly VPC where the required services
infrastructure will be deployed using Terraform.
The code will be deployed using Jenkins CI.
Dependencies
It depends on the conclusion of the EMP infrastructure upgrade, to be paired with Ebury Core, which is a work in progress at the moment of this RFC. Which will provide a reliable infrastructure based on Kubernetes and will deploy the services on EMP environment.
Also, it is required that BOS, EBO and Kafka Connect are running on the new infrastructure.
Payment Query Service depends on Debezium, Kafka Connect CDC stream and local database.
It requires a valid API key for Calendarific API.
References
[1] Smart Payment Execution Date RFC
[2] Smart Payment Execution Date Integration RFC
[3] Smart Date Service Integration Phase B RFC
[4] Smart Payment Execution Date - Mass Payments Requirements RFC
[5] Payment Query Service presentation
[6] EMP infra presentation
[7] EMP Infrastructure Parity Plan
[8] Ebury Smart Date Service