Dynamic Forwards & Options Services

This document describes the Dynamic Forwards & Options services and explains how data derived from an external financial instruments' provider, ICE, are going to be fetched and distributed in our internal system’s services, by the DFO Trade Engine Service.

Reference Documents

Reference Document Location
EPIC01 DFO Services (EPIC)
ICE Intercontinental Exchange (ICE)
GWSPTLB Gateway - Ebury Blueprints
S2SAUTH Service-to-Service Authentication and Authorisation
DFXE0001 Dynamic Forward Extraction
K8SDEVHB K8s Developers Handbook
DIAGRAMS Diagrams for this RFC

Problem Description

There are some processes at Ebury that require Dynamic Forward data that are managed by the external system, ICE. Since our services require access to specific resources of the ICE platform, these data have to be available in our workloads.

Background

ICE, the Intercontinental Exchange, is a global platform that operates financial and commodity marketplaces and exchanges. Its operations, also include futures and cash exchanges, as well as services for a wide range of financial instruments. It is providing an API, which allows to programmatically interact with the resources that exist on the ICE platform.

As also described in the ICE Dynamic Forward Data Extraction RFC, Dynamic Forward is an Ebury product which uses ICE as the tool to manage it. For reporting Dynamic Forwards metrics we need to fill our data lake/warehouse with data coming from ICE.

Solution

The proposed solution is to introduce a single point of interaction with the ICE API, encapsulating this in a service, the ICE Gateway, which will be used as a middleware between the external system and the other services, managing the integration between them, providing at the same time a layer of abstraction. This will also allow us to work in the internal system without being affected by the external system (and the opposite). Apart from that, it will also provide a centralised way of logging and monitoring, so we can easily identify possible issues, errors and bottlenecks related to the ICE system. On the other hand, the DFO Trade Engine service will consume the resources from the ICE Gateway to respond to the inbound requests.

Services

  • DFO Trade Engine Service: this is the actual service that handles the cross-domain communication between our services, which means other services will request resources, by Kafka Command Events and the service will respond with the corresponding Kafka Domain Events batch. The service requests from ICE Gateway, through HTTP requests. The daemon of this service will be a Kafka consumer.
  • ICE Gateway: It serves between the DFO Services and ICE API, providing HTTP synchronous entrypoints as an HTTP Proxy. It's responsible to transfer data from ICE API to our system, and the opposite.

Internal system communication

The DFO Trade Engine service will be adapted to our system, so it can serve any internal service following our main strategy on inbound requests between different domains. Data will be transferred in an asynchronous way through the Kafka events, without being restricted by the external system endpoint type (synchronous or asynchronous). However, Kafka events should not be the only way of data transfer, since infrastructure components could also be used, for example data uploading on S3 storage bucket. Finally, a PostgreSQL database (component of infrastructure as well) will be used to enhance DFO Trade Engine service for use cases that require data persisting in storage.

DFO Services Architecture

External system communication

The ICE Gateway will consume the resources from the ICE API, performing the main required operations as authentication, authorization and data format transformation, to achieve proper communication, ensuring that all requests are authenticated, authorised and the data are also validated matching the format as required from ICE.

DFO Services Abstract

Service Ownership

New Service Service Name Service Owner
Yes DFO Trade Engine Service TEG Team
Yes ICE Gateway TEG Team

Alternatives

The only alternative could be to access ICE API directly from any location within our internal system, which would not be proper since the external system:

  • Has specific security and access restrictions that have to be covered
  • Has endpoints with complexities as investigated in POC, that must be addressed
  • Requires data to be converted in specific format/structure (in our case, most in XML)
  • Most endpoints are synchronous, which would be a limit according to our architecture

Apart from that, it would also be very difficult to monitor and log all issues and exceptions that would be caused from the interaction with the external system. Additionally, in case that there are updates related to the external system's API, it would require a lot of effort and changes from our side, in all repositories that consume it.

Caveats

  • The Gateway service will depend on the external system, ICE, so any issue or problem with it will also affect the Gateway service itself.
  • Because of the above dependency, any change on the external API also means that we need to update the gateway service, to adapt to these updates.

Operation

Based on the current requirements, as also described in the use case section, but also on the current tier in ICE platform, we expect this operation to be executed once per day, so DFO Service will fetch data periodically.

Security Impact

Secrets management

Any secret (e.g. API credentials) that may be used in DFO Services should be configured and managed in Vault.

Authentication/Authorization

DFO Services <-> Other Ebury Domains

Since such communication is performed by Kafka messages, the component that must be authorised is the corresponding Kafka producer service, to send events or commands to specific topics.

DFO Trade Engine <-> ICE Gateway

Authentication and authorisation between the services of DFO will be implemented following our security practice Service-to-Service Authentication and Authorisation.

ICE Gateway -> ICE

Most of the endpoints provided by the ICE API are secured by authorization. This means that to access them, the auth token has to be part of the request header. Two of the most common and important flows, between the ICE Gateway and the ICE API, may be found below in the flow diagram.

  • Authentication flow: is important to retrieve the auth token, needed for the rest of the calls. In this operation, the credentials have to be posted to the endpoint, which will respond back with the auth token.

  • Authorised request flow: where an ICE API resource has been requested from the Gateway. In this operation, since the endpoint is secured by authorisation, the auth token (we retrieved from the previous operation) has to be sent in the request headers. Finally, the endpoint responds with the requested data.

DFO ICE Gateway workflows

Performance Impact

It is expected a performance impact in the communication between ICE Gateway and ICE API, as the performance of ICE will always be the limiting factor.

It is also expected a reasonable failure rate of ICE API requested actions, on the provider's side. Οur reaction may vary on a case-by-case basis, considering the implications on retrying the same request again immediately or later. For example, there are endpoints we can safely retry, like getting a single resource. However, there are endpoints that may require a totally different approach, like the generation of valuation report, which is billable and also includes a retry logic in the provider's side, that should automatically reschedule the failed process.

The service will be monitored emitting metrics and providing real-time alerts. There should also be a dashboard, so we can easily identify and track any possible performance issue that may arise.

Developer Impact

  • Make implementations according to the programming stack, Kafka producer and consumer, FastAPI.
  • Understand the Repository Pattern in contexts of Rest API integration
  • Test code in contexts of unit and integration testing
  • Handle errors/exceptions and smoothly pass the information through the layers
  • Work with the ICE API, the external system and adapt to possible changes
  • Be familiar with logging and monitoring on the services

Data Sources

  • Input data sources: data will be fetched from the ICE API
  • Transformation requirements: the input data will be transformed from XML to JSON
  • Output data: the target location of the transformed data is the Kafka stream

Deployment

Services will be deployed to Kubernetes, with Jenkins fully automated processes (CI/CD) so different application versions can be deployed after the pull requests are merged. Both the Kubernetes infrastructure and the deployment process are documented and can be found in the K8s Developers Handbook.

Dependencies

There are no dependencies to start implementing this RFC.



Based on RFC Template Version 1.1