Architecture to manage account details

This document is an overview of the architectural decisions made for the Stages 2 and 3 of the Account Details Project. The goal is to provide visibility and gather feedback from the wider Technology Department at Ebury.

Problem Description

Ebury needs a way to offer their account creation capabilities to our clients. Existing workflows involve manual intervention from the Operations team in BOS, despite Ebury can create certain accounts automatically by means of a 3rd-party provider: Form3.

The Account Details Project implements the required software components to allow our API Clients to directly request (and obtain) an account programmatically. Similarly, Online users will be able to automatically obtain an account on their name from EBO. Existing workflows through BOS will still be supported.

In addition to the former functionality, the Account Details Project incorporates additional improvements to existing workflows (describing them is out of the scope of this document). Furthermore, the project builds upon the opportunity to take some responsibilities out of BOS and encapsulate them inside a new micro-service: the Account Details Service (ADS).

The design of ADS is interesting in several aspects:

• ADS offers a REST API.
• ADS connects to an external provider which, in turn, asynchronously updates the accounts it creates.
• ADS generates domain events.
• BOS requires some sort of database synchronization with ADS.

In the remainder of this document, we'll detail the components that comprise the envisioned solution (and their relationships) so that the underlying design decisions can be ratified by the Technology Department - or changed accordingly.

Background

Let us define some terminology used in the Account Details Project that is relevant to this document.

• Segregated Accounts. They refer to internal bank accounts Ebury holds on behalf of specific clients that require an scheme Ebury cannot offer widely (e.g. BACS accounts). They're modeled in BOS as a BankAccount that refers to a BankAccountMask.

• Virtual Accounts. Client-named accounts Ebury offers through our information systems. These can be created manually or automatically via Form3. They're modeled as a BankAccountMask in BOS.

• Pool Accounts. Ebury-held accounts our clients may use to operate with Ebury. These accounts are still managed in BOS according to the current scope of the project. They're modeled as a BankAccount in BOS.

• Stage 2 of the Account Details Project. Allow EBO/API users to retrieve their account details from ADS. Similarly, BOS users retrieve account details from ADS.

• Stage 3 of the Account Details Project. Allow EBO/API users to create accounts via ADS. Similarly, BOS users create account details through ADS.

Note that each stage involves additional endpoints and BOS pages, but their analysis isn't relevant for the objectives of this document (i.e. they don't imply architectural changes).

Solution

Architecture

The image below shows the architectural components required to create accounts via ADS, and the relationships among them. Account Details-specific components are shown in blue. Arrows show the direction in which the communications flow. Dashed lines represent components and flows outside Ebury (i.e. that happen in a 3rd-party provider, Form3 in this case).

ADS

ADS is the service that implements the functionality to list, create, and update segregated and virtual accounts. It exposes a REST API to offer these capabilities to other Ebury services. Multiple instances of this service run simultaneously for fault tolerance and increased performance via load balancing.

The information ADS manages is stored on a relational database. When accounts are created or updated, ADS publishes the corresponding AccountCreated or AccountUpdated events in the appropriate topic. In this way, other Ebury systems outside the bounded context of ADS may receive these events and react accordingly. Note that this allows ADS to remain highly cohesive because it only manages account details. Other services interested in the lifecycle of account details are loosely coupled with ADS via the Events subsystem.

When an Operations user creates an account in BOS on behalf of the client (manual account), ADS stores the changes immediately in the database. However, when such account is given on-demand by an external provider (automatic virtual account), the Clients themselves can also request it by means of EBO or Ebury's API. In this case, ADS must ask Form3 for the creation of the account.

Form3 API returns immediately the new account with a pending status. When Form3 either confirms (confirmed status) or rejects (failed status) the new account, it publishes an event in the SQS queue of our choice to communicate the update. In our current use cases, this confirmation immediately follows the creation of the account. These events are consumed by the next component.

ADS API

We will extend the API with a number of new account-related endpoints. Below is a diagram displaying the new endpoints, note that the Stage 1 endpoints are omitted (you can find them here):

We're allowing listing, creating and modifying different types of accounts, as well as activation/deactivation and making an account default. It's worth noting that the /accounts endpoint will return all types of accounts by default, with the option of filtering for specific types via a query parameter.

Another thing that needs to be highlighted are the settlement accounts. These are physical accounts and each virtual account is routed to a settlement account through the BIC and scheme. Currently, in many of our services this relation is implemented in different ways, or the settlement account is hard coded (e.g. in the case of sepa-service). Settlement accounts are part of internal accounts.

Deprecated with Internal Account modeling. Through the /settlement-accounts endpoint, the idea is to enable API consumers to find out what is the settlement account for a specific virtual account. To enable this, the proposal is to allow the GET endpoint to take a virtual account and the scheme as query parameters. ADS will check if that virtual account exists, take the BIC of its issuer and use that along with the scheme to find out which settlement account corresponds to it. The BIC and scheme will form a composite unique constraint on the settlement accounts.

With this approach, we will have a uniform solution that can be used by other services while making configuration changes smoother because the change is required only in one place.

Form3 Accounts Consumer

This daemon reads Form3 events from the SQS queue. It updates the corresponding account in the database with its new status and publishes an AccountUpdated event in the appropriate Kafka topic. Multiple instances of this daemon may run simultaneously.

Account Details Consumer

Despite ADS is the owner of account details, different periodic tasks that currently run in BOS expect them to be available in the BOS database. E.g. the AssignedAccountsDataReport daemon iterates over the instances of the BankAccountMask model (the equivalent of "account details" in BOS).

These tasks might be moved or refactored in the future, but in the meantime we need a means to synchronize ADS state with the BOS database.

We take advantage of the AccountCreated and AccountUpdated events triggered by ADS to update the database in BOS. Hence, the Account Details Consumer is a daemon that reads those events from Kafka and updates the BOS database accordingly.

The number of simultaneous consumers that can be run is restricted to the number of partitions in the Kafka cluster.

Kafka topics

ADS publishes events on the following topics:

• ebury.events.account-details.segregated-accounts.v1
• ebury.events.account-details.virtual-accounts.v1

Form3 Accounts Consumer publishes events on the following topic:

• ebury.events.account-details.virtual-accounts.v1

In both cases, the key of the published event is the UUID of the account. This makes all events related to a same account to be stored in the same Kafka partition. In turn, this guarantees that the consumers subscribed to those topics process the events of a same account in the same order they were published. E.g. this assures that we cannot try to update an account before it's been created. The latter might not hold true under rare events like changing the number of partitions of the Kafka cluster.

Account Details Consumer subscribes to the aforementioned topics:

• ebury.events.account-details.segregated-accounts.v1
• ebury.events.account-details.virtual-accounts.v1

Topic names assume this structure:

$domain.$message_type.$scope.$resource.$version

Use cases

Below you can find a detailed description of the main use cases that ADS supports. It's not an exhaustive list.

Create account manually

Ebury has the capability to provide a client with an account in a given currency. However, such account cannot be created automatically and it requires manual action - because the issuer that gives us such capability is not integrated in Form3.

1. Clients contact their dealer to issue their account. The latter saves the request in Salesforce. This triggers a notification sent to the Operations team.

2. The Operations user creates a manual virtual account in BOS.

3. BOS calls ADS endpoint POST /virtual-accounts to create the account.

4. ADS saves the account and, once the transaction's been committed, it triggers an AccountCreated event.

5. ADS returns an HTTP 201 Created response that contains the account details in the body.

6. BOS updates the view for the Operations user.

7. Clients eventually know from their dealer that their account is ready.

After step (4), an hypothetical Notifications service might send the Clients an email or any other kind of message to let them know right away their account has been created.

Create account automatically

Ebury has the capability to provide a client with an account in a given currency. Such account can be created automatically via Form3.

1. An Online user asks for a new account from EBO (*).

2. EBO calls API to create the new account.

3. API calls ADS endpoint POST /generate to create the automatic virtual account.

4. ADS calls Form3 to create the account.

5. ADS receives the response from Form3. It contains all the account details but the status is pending.

6. ADS saves the account in the database and, once the transaction's been committed, it publishes an AccountCreated event.

7. ADS returns an HTTP 202 Accepted response that contains the account details in the body.

8. API returns the appropriate response to EBO.

9. The Online user sees their account details in EBO.

10. When the new account is in confirmed or failed status, Form3 publishes an event in an SQS queue.

11. Form3 Accounts Consumer reads the Form3 event.

12. Form3 Accounts Consumer saves the account in the database and, once the transaction's been committed, it publishes an AccountUpdated event.

(*) This use case may also be initiated by an API User calling the API, or an Operations user working with BOS. The described flow applies in all cases.

Note that steps (10-12) happen asynchronously after Form3 returns the response in (5).

After step (12), an hypothetical Notifications service might send the Clients an email or any other kind of message to let them know their account has been confirmed (or failed); the Webhooks system might notify API Clients their account is ready (or not); an hypothetical Websockets server might issue a message to update a reactive interface; and the likes.

Database synchronization

As explained above, we exploit the Events subsystem to keep the BOS database in sync with the account details in ADS. The Account Details Consumer subscribes to the topics of interest so that it receives the events triggered in the use cases above.

1. Account Details Consumer receives AccountCreated or AccountUpdated event from Kafka.

2. Account Details Consumer updates the BOS database accordingly.

List account details

1. An Online user asks for their account details from EBO (*).

2. EBO calls API to retrieve the list of account details.

3. API calls ADS to retrieve the list of account details.

4. ADS reads the requested information from the database.

5. ADS returns an HTTP 200 Ok response that contains the list of account details in the body.

6. API returns the appropriate response to EBO.

7. The Online user sees their account details in EBO.

(*) This use case may also be initiated by an API User calling the API, or an Operations user working with BOS. The described flow applies in all cases.

Note that the API call is more complex because it requires a BOS interaction to retrieve pool accounts. However, it doesn't impact the architecture described here.

Alternatives

Account Details Consumer triggers background task

Instead of immediately updating the BOS database when processing an event, the Account Details Consumer could trigger a background Celery task to do the actual job. Or, similarly, it could publish a message in an SQS queue and a different daemon would update the database (but Celery builds an appropriate task queue abstraction that simplifies this).

The main advantage of this approach is that the consumer quickly dispatches the task and gets back to listening for additional events, thereby reducing the event consumption delay on a per-consumer basis. Since only one consumer in a Kafka's Consumer Group reads messages from a given partition of the topic, this alternative becomes more relevant if the number of partitions in the cluster is very low. Otherwise, we could just improve the throughput by launching more consumers (up to the number of partitions).

Another benefit lies on Celery's task abstraction, given that it facilitates retrying the task a finite number of times following an exponential back-off. The former is especially useful to handle rare events like changes in the number of partitions of the cluster. Under such circumstances, the event delivery order is not guaranteed. A simple "retry" algorithm like the one that comes with Celery may cope with most of these situations. On the other hand, these kinds of situations shouldn't happen often and our code must handle them anyways.

Finally, Celery also introduces its own challenges. Unless we restrict ourselves to a single worker, we cannot assure that events are processed in the same order they were produced. I.e. we're making Kafka deliver events in order, but the actual database updates could occur out-of-order.

Summing up, and assuming we'll have an appropriate (and relatively stable) number of partitions in Kafka, we'll go with the synchronous approach because its simplicity outweighs it potential disadvantages. In any case, we could always move to this option without disrupting existing use cases.

Form3 Accounts Consumer triggers background task

Instead of immediately updating the status of an account (and sending the corresponding event), the Form3 Accounts Consumer could trigger a background Celery task to do the actual job.

While this approach would be perfectly fine in this case too, the problem it solves is less important to address than in the previous section:

• Contrary to Kafka, the number of simultaneous consumers we may launch for SQS doesn't have a hard limit. So we can improve the throughput by just launching more consumers.

• Contrary to BOS, the Account Details Project doesn't have at this moment any backend for Celery tasks. If it's not really needed, we prefer to keep things simple and minimize our infrastructure requirements.

In case this is useful at any point in time, we might implement this option without disrupting existing ADS clients or event consumers.

Send only one event per automatic account

In the approach depicted here, every account created via Form3 triggers two different events: AccountCreated when the account is created in Form3 with pending status; and AccountUpdated when Form3 notifies the updated status.

Instead, we might send just one AccountCreated event once the final status has been received from Form3. While this could fit our current use cases, we cannot be certain that it applies for future ones.

Thus, we prefer to play on the safe side and communicate via events the actual things which are happening to the accounts. E.g. think of a central log of domain events that allows for keeping track of what's being happening in the platform. If we miss an update event from Form3, such a log would easily show that there was an AccountCreated event but no corresponding AccountUpdated.

Publish events in the background

As explained before, both ADS and the Form3 Accounts Consumer publish events only after the database transaction has been committed. This means that we might change our system's state but the event that should communicate this to other services could fail.

We might cope with temporary failures (e.g. network hiccup or Kafka REST Proxy overload) by simply publishing the event in the background using Celery or a similar technology. However, this doesn't preclude dealing with events that eventually fail to be published (see Caveats section below).

Kafka topic names

We have a hard constraint on the structure of the topic name: the Account Details Project needs to publish both AccountCreated and AccountUpdated events on the same topic. The granularity of the topic names must fit this requirement.

Otherwise, we'd have to deal in our code with out-of-order processing of the events related to a same account. For the sake of simplicity and maintainability, it's better to let Kafka take care of it.

The remainder of the proposed topic structure is more amenable to changes. While we believe it exposes some nice features, this is to be discussed elsewhere.

Caveats

Since the Account Details Project heavily builds upon event-based, distributed systems, the usual limitations apply. In particular, we note here that the system must deal with failures when publishing and consuming events.

On the publication side, events that cannot be sent must be logged somehow in order to replay them either manually or automatically (if appropriate). In its simplest form, this can be accomplished by inspecting application logs. Alternatively, failed events can be stored in the database and a periodic task could be in charge of handling them.

Regarding consumption, the version we add to the Kafka topic minimizes the likelihood of bugs due to a contract mismatch between sender and receiver. In any case, we might protect against processing errors by using a dead letter topic (analogous to a dead letter queue in a message queue system).

Operation

Nothing specific to the Account Details Project. We're using existing infrastructure like Kafka or well-known managed technologies like SQS.

However, given that the Kafka cluster has been created recently and this is one of the first projects that uses it, some infrastructure changes will be required. These are to be discussed elsewhere.

Security Impact

Nothing specific to the Stages 2 & 3 of the Account Details Project.

Performance Impact

We expect the Account Details Project to have a positive impact on the performance of several workflows. In addition to an improved user experience, listing accounts will take shorter when it's done from ADS than from BOS.

Developer Impact

N/A

Data Consumer Impact

N/A

Deployment

N/A

Dependencies

• Infrastructure changes to consume Kafka events from BOS VPC
• Consensus on the structure of Kafka topic names
• Consensus on the number of Kafka partitions

References

• ADS API (requires being connected to our VPN)
• Form3 API
• Apache Kafka
• Amazon SQS
• Celery