Valuation report automation
The data managed by ICE applications need to be consumed by key processes at Ebury. This document covers the mechanisms we'll use to automate the extraction and delivery of Dynamic Forwards data as a long-term solution. To understand better the current document, it's recommended to read the DFO Services blueprint.
Reference Documents
| Reference | Document Location |
|---|---|
| DFO0001 | DFO Services blueprint |
| DFXE0001 | Dynamic Forward Extraction (interin solution) |
| EPIC01 | DFO valuation report (EPIC) |
| TA0001 | Tech analysis on valuation report automation |
Problem Description
There are processes at Ebury that require Dynamic forward data, which are managed by a third party application, ICE. ICE has an API, which will be used in order to implement an integration using the provided endpoints, automate data extraction and delivery to Ebury's teams.
Background
Dynamic Forwards and Options is an Ebury product which uses ICE as the tool to manage it. For reporting Dynamic Forwards metrics we need to fill our data lake/warehouse with data coming from ICE.
Applying the solution of the current RFC, the short-term tactical solution of ICE Dynamic Forward Data Extraction will be deprecated.
Solution
The proposed solution would be for the DFO services to invoke and download valuation report data, using the PostgreSQL database where is required, so finally would upload the retrieved data on the S3 storage bucket where the DIO team has access. The solution is not introducing any breaking change for the DIO team, since the delivered file will have the same schema with the one of the interim solution.
ICE API - Valuation Report Workflow
Sequence Diagram
Invoke valuation report
To request a new Valuation report generation, the endpoint "InvokeValuationReport" has to be called. The report date and file format has to be specified in this request. The format of the final report is determined by the request parameter "FileFormat" (possible values could be XLS, XLSX, CSV, XML). The endpoint is asynchronous and will trigger the report creation on ICE backend, responding once the command is accepted, before the generation is completed. The response contains a reportId, which will be used in next steps. - The reportId cannot be retrieved by ICE API (so has to be persisted) - A report cannot be retrieved by ICE API, if was not requested by ICE API (e.g. by UI)
Get report status
As mentioned in the previous step, requesting the generation of a new valuation report is an asynchronous process, so we cannot know the actual status of the report by the response. Here comes the second endpoint, "GetReportStatus", which we’ll use to get the actual status of the report. The "report_id" part of the 1st step response is required to use this endpoint. This endpoint has to be polled until having a final state for the report, success or failure.
Download report
Once the status of generation is successful, this endpoint can be used to retrieve the actual data of the report (including its file type). The report will be in the format from the InvokeValuationReport call. The generated report can be downloaded many times without extra charge. Report ID is needed here too, as responded from the 1st step.
DFO Services - Automate valuation report workflow
Please not that this section contains some technical details in order to explain better the flow. In any case, possible status values or model fields are just for explanation and are not strict. Values could be different in actual implementation.
Sequence Diagram
Initialise a daily report
- A command is sent by cronjob (living in same repository as DFO-TE) to DFO-TE through Kafka, to trigger the daily report process. Such command should be sent once per day according to the requirements, however even if sent more times, DFO-TE should have a command validation (to avoid over charging)
- Command is consumed by DFO-TE, so the daily report process is triggered
- Validation is been performed on DFO-TE and if the command is valid, then a new record for the daily report should be created in DFO persisting system, as unhandled report
Invoke valuation report
For each unhandled report following actions should be performed: - Periodic task inside DFO-TE picks up unhandled daily report - DFO-TE requests the generation of a new valuation report for the specific past day from ICE-GW, through HTTP request - ICE-GW invokes a valuation report from ICE-API through HTTP request - ICE-API replies to ICE-GW with an asynchronous response that contains the reportId. This response doesn't mean the process has finished, but that the process is accepted and has just started in ICE - ICE-GW responds back to DFO-TE with the reportId - DFO-TE persist the reportId and updates the started datetime in the unhandled report, so the record should change state from unhandled to incomplete
Get report
For each incomplete report following actions should be performed: - Periodic task inside DFO-TE pick up incomplete daily record from DB - DFO-TE tries to download the report: - DFO-TE gets the report for specific reportId, from ICE-GW, through HTTP - ICE-GW gets status for the reportId from ICE-API - If the report status is not final (pending, in progress, postponed, etc): - ICE-GW responds with "NOT READY YET" to DFO-TE - DFO-TE terminates the process - If the report status is final (finished, finished with errors, etc): - ICE-GW sends request to ICE-GW, to download the report content - ICE-API responds to ICE-GW with the report content - ICE-GW responds to DFO-TE with the report content - DFO-TE packs report contents in a file and uploads it on S3 bucket (if something breaks during the S3 upload, the process should exit, letting the DFO periodic task retry it on a next iteration - downloading an existing report from ICE is not a paid action) - DFO-TE updates the daily record state to complete
Service Ownership
| New Service | Service Name | Service Owner |
|---|---|---|
| No | DFO Trade Engine Service | TEG Team |
| No | ICE Gateway | TEG Team |
Alternatives
There are no alternatives because this is the long-term solution to deprecate the temporary one.
Caveats
- The proposed solution depends on ICE adding non-live-trades to the report generated by the API
- Since the operation is asynchronous, there's always an edge case where the report would not be delivered on S3 in time
Operation
The DFO Trade Engine service will be triggered by a K8s cronjob once per day, to generate valuation report in ICE API, through ICE Gateway, to upload it on S3 once is ready.
Security Impact
Secrets management
Any secret (e.g. API, S3, DB credentials) that may be used in DFO Services should be configured and managed in Vault.
Authentication/Authorization
Cronjob <-> DFO Trade Engine
Since such communication is performed by Kafka messages, the component that must be authorised is the corresponding Kafka producer service, to send events or commands to specific topics.
DFO Trade Engine <-> S3
We are going to use S3 protocol to upload valuation report files on the storage bucket. The authentication will be by user credentials provided by SRE Team.
DFO Trade Engine <-> Database
The proposed solution will require access to DFO database. The authentication will be by postgres user credentials.
DFO Services <-> Other Ebury Domains
As described in DFO Services blueprint
DFO Trade Engine <-> ICE Gateway
As described in DFO Services blueprint
ICE Gateway -> ICE
As described in DFO Services blueprint
Performance Impact
As described in DFO Services blueprint
Developer Impact
As described in DFO Services blueprint
Data Sources
- Input data sources: data will be fetched from the ICE API
- Transformation requirements: the input data will not be transformed/processed; will be delivered as is
- Output data: the target location of the transformed data is DIO Team S3 storage bucket
Deployment
As described in DFO Services blueprint
Dependencies
There are no dependencies to start implementing this RFC.
Based on RFC Template Version 1.1