PSD2 for EMP clients on Ebury API

This document reflects the proposal of including support for EMP clients on PSD2 feature in the public Ebury API.

Problem Description

Due to European regulations Ebury has to provide Strong Customer Athentication (SCA) service. See PSD2 Strong Customer Authentication RFC for further details on PSD2.

Right now, Ebury API allows Ebury Core clients to use PSD2 features, but due to the implementation used this is not extensible to Ebury Mass Payments (EMP) clients. This RFC propose a solution to allow EMP clients to use PSD2 features.

Background

First, it is needed to understand how the Ebury API works on the Ebury Mass Payment environment, and how the PSD2 feature is currently working on the Ebury Core environment.

Current Ebury API on Ebury Mass Payment environment

There is just one Ebury API instance, connected with two backends (BOS, FXS, …): Ebury Core and Ebury Mass Payments backends. To choose which backend is used on every connection, each Ebury API client has a backend field on DynamoDB, so every time a contact login into Ebury API, the contact sends the client_id parameter, API checks Client DynamoDB table to know which backend is going to be selected using the client_id provided by the user. This backend field is stored on Redis with the access token, so every time this contact uses this access token on API-Webapp the request is going to be sent to the same backend. Take in mind that this API client_id is different than BOS client_id.

A special case is 2FA feature: verify service has just one instance for both Ebury Core and Ebury Mass Payment environments. Verify service is contacted in order to know if the contact that tries to login has enabled the 2FA

There are some special cases:

  • Authenticate and verify 2FA templates: those endpoints render the HTML templates for login and verify the 2FA. All backends are using the Ebury Core templates.

  • Metadata endpoints: those endpoints doesn't require login, so they don't have backend info available. In those cases they connects always with Ebury Core backend.

Ebury API will remove multibackend support after EMP migration was done.

See how this is working on code for further details about the implementation.

Current PSD2 on Ebury API

PSD2 is a European regulation. Ebury uses a third party integration to accomplish this regulation. This third party is equensWorldLine (eWL).

Ebury API has 3 different flows on PSD2:

  • AIS (Authorization flow): Get a consent to check accounts, account details, balances, transactions, …

  • PIS (Payments flow): Payment management flow.

  • FCS (Funds Confirmation flow): Check if an account has enough balance.

Take in mind that the request of these flows arrives to Ebury API in a different order that the client does the requests to eWL due to eWL as an intermediate saves some requests and sends them to Ebury API in a different order.

AIS and PIS flow work in a similar way between eWL API and Ebury API, but FCS flow is a completely different flow that will be analyzed separately.

AIS and PIS endpoints are open to the Internet but only eWL API is expected to use them, direct clients must not use them. Those endpoints are not on Ebury API public documentation or OpenApi specification.

FCS flow has an IP whitelisting filter (as it hasn’t any kind of authentication), so just eWL is able to connect to it.

Let's explain in details each flow:

AIS flow

It uses a set of endpoints in API-Auth, the ones under /sca/<resources> endpoints, in order to get an access token to access API-Webapp. Let’s call these API-Auth endpoints as SCA endpoints.

  • Those endpoints connect with BOS and Verify. Those endpoints have the backend and the brand fields hardcoded to the default values on connections with BOS (the default values are Ebury Core ones). These values cannot be dinamically calculated due to the client_id parameter is not sent, so Client table of DynamoDB is not hit, so API doesn't know the backend and the brand of the user.

In 2FA cases, the brand data to be used on Verify is extracted from a BOS endpoint. To use that BOS endpoint the online_url (that it needed to get the user brand) and the backend are hardcoded with the default values, that's mean that right now only Ebury Core Clients that belong to Ebury Core brand are allowed to use 2FA.

On API-Webapp this flow just uses some GETs endpoints related to knowing the info of the clients accounts (GET account, GET balances, GET transactions,...). Take in mind that after getting an access token, this token is valid in all API-Webapp, not just some endpoints.

  • Those requests connects with BOS, but in this case the backend field is read from the access token stored on Redis, as it is done on the usual API usage.

PIS flow

PIS flow uses a set of endpoints in API-Auth, the ones under /sca/<resources> endpoints, in order to get an access token to access API-Webapp. Let’s call these API-Auth endpoints as SCA endpoints. 2FA feature is exactly the same as AIS flow.

On API-Webapp this flow manages payments on the BOS side (create, get and cancel payments in bulk). These endpoints are under /psd2/payments/ endpoints. Take in mind that after getting an access token, this token is valid in all API-Webapp, not just some endpoints.

This flow follows the same SCA flow explained in the AIS Flow section, so it is not going to be duplicated here. The differences between the flows are some parameters included on the sca session token that are not relevant for this analysis, they are used by equensWordLine (eWL).

FCS Flow

FCS flow uses just one endpoint in API-Webapp. This endpoint is a special case, as it doesn’t require an access token. This endpoint is /funds-confirmations. It contacts BOS in order to get information about funds in an account. It doesn’t require an access token, so the backend param is not read from Redis, so the BOS client uses the default one (the Ebury Core one).

This endpoint uses IP whitelisting to access it. Right now only equensWorldLine is allowed to access.

See how this is working on code for further details about the implementation of the different flows.

Summary of issues

After the explanation about how it is built, several problems are obeserved here that needs to be solved if Ebury Mass Payments wants to provide PSD2 service to their customers.

Issues that need to be addressed are:

  • PSD2 endpoints have hardcoded the Ebury Core backend and Ebury brand in connection with BOS, so this feature is not working for clients that are not part of Ebury Core backend, or clients that have a brand different than Ebury. In other words, PSD2 is only working for Ebury Core clients that are from the Ebury brand.

  • PSD2 endpoints have hardcoded the Ebury brand in connection with Verify.

  • PSD2 templates have the Ebury brand hardcoded to be retrieved from s3.

  • PSD2 requests don't include any dynamoDB client parameter, so we cannot match with any existing API clients.

  • FCS flow doesn’t hit Redis, so no info about backend is available.

  • Bearer token on SCA endpoint is saving Brand on backend fields on DynamoDB and Redis.

  • As the last step on SCA (destroy session), auth-webapp is using the brand as backend too.

Solution

The issues can be summarised on:

  • Problems to select the backend in a SCA request.

  • Problems to select the backend on FCS request.

  • Problems to select the brand to connect with BOS.

  • Problems to select the brand to connect with Verify.

  • Problems to select the brand to render the templates.

  • Problem mixing fields on dynamoDB and redis.

  • Problem mixing fields on destroy session step.

They will be analysed separately.

Problems to select the backend in a SCA request

On a usual request, the contact includes the API client_id on login call, so the API can check on the Client table of DynamoDB the selected backend.

On SCA the API client_id is not available, so the backend can not be selected using the Client table on DynamoDB.

Proposed solution

Ebury API has just two backends and most of the requests are for Ebury Core one. Ebury API can try with the Ebury Core backend, and in case the response is 1103 'Invalid client identifier.' or 1101 'The email or password is incorrect' retry using the second backend. If the second backend receives the same error, send it to the client as the user is not on the system.

In the case that the contact will be on both system, Ebury Core one will return the data. Another option is to always retry on both system and raises a log message to notify if the contact is on both environment.

It would be nice to save the final choosed backend on DynamoDB on scaSessionData model to know which backend API is using to troubleshoot issues and have it available for future calls (e.g to create the access token the backend is stored on DynamoDB too).

Take in mind that Ebury API will remove support for multibackend after EMP migration, so this solution will be temporal.

This proposed solution will require an API-Auth task to include the retry on BOS connections and save the backend used on DynamoDB.

Pros: no new params are needed to be sent by the final user (or equensWorldLine). Once EMP migration will be done all will continue working without any change.

Cons: EMP clients are going to have big times than Ebury ones. Errored requests are going to have big times too.

Problems to select the backend on FCS request

On a normal transactional request, Redis stores the backend with the access token. (Funds Confirmation) FCS endpoint doesn’t have an access token, so no backend field is available.

Some special points here:

This endpoint just receives an IBAN and an amount: on code BOS allows the same IBAN on Ebury Core and EMP BOS backend but from business POV this situation is not going to be raised. An IBAN must be assigned just to a single client, so the only case the same IBAN can be on both systems if in the case the client is a client of both platforms, and both platforms offer this IBAN to the client, that is not possible.

This endpoint has an IP whitelist. Right now EMP clients are going to connect to eWL using the same URLs, and eWL is going to connect to Ebury API using the current machines, so it is not needed to add new IP’s on the filter.

Proposed solution

As in the previous section, API can just retry: try first with Ebury, and if the response is 404 Not found, API can just retry on EMP. In case it raises a 404 again, this means that the IBAN is not on the Ebury Core or EMP system.

Take in mind that Ebury API will remove support for multibackend after EMP migration, so this solution will be temporal.

This proposed solution will require an API-Webapp task to include the retry on FCS endpoint.

Pros: no new params are needed to be sent by the client (and eWL). Once EMP migration will be done all will continue working without any change.

Cons: EMP clients are going to have big times than Ebury ones. Errored requests are going to have big times too.

Problems to select the brand to connect with BOS

BOS uses the brand on several call from API to BOS, they will be analysed in order to check if the brand is really needed:

  • POST /api/v1.0/contact: brand is used in BOS if SWITCH_ONLINE_LOGGING_SEGREGATION is enabled (here). This was done due to some requirements from EBO. Due to this, the contact is only retrieved if email, password and brand match with the ones on the BOS database. But on API this requirement is not needed, so this requirement can be deleted from BOS just for Ebury API users. The user can be distinguished between EBO and API user using the request.origin field on the BOS view. This point requires a task for BOS to remove this requirement for API, and a task for API-Auth to avoid sending the brand on that request.

  • GET /api/v1.0/clients: brand is used in BOS if SWITCH_ONLINE_LOGGING_SEGREGATION is enabled and if brand is sended (here). It is an optional parameter to be used as a filter, so if it is sended, the clients will be filtered also by brand, but if it is not sended, all clients will be returned, independently of the brand. It is a decision for the product team if we want to filter by brand on API clients too. Right now, Ebury API is sending the brand here too, so API is filtering by brands too.

    • Do API Clients have several brands? If this is not the case, removing the brand from this call will help with API PSD2 implementation for EMP. This point will require one task on API-Auth to avoid sending the brand on that request.

    • In case the brand cannot be removed here from Ebury API call due to business requirements we will search for a way to get it. In that case, the BOS ClientUser model has a property (here) to retrieve the online_brand for a contact, so including this field on the response of POST api/v1.0/contact and making it available for next call will solve this issue. This point will require a task on BOS to include the online_brand on the response of api/v1.0/contact endopointt and a task on API-Auth to retrieve it from the api/1.0/contact call, store it if needed and use it on the api/1.0/clients one.

Proposed solution

Removing the brand from BOS login just for Ebury API clients does things easily: API doesn’t need to send it on /contact request.

On the /clients endpoint, brand is just optional, so we can remove the parameter from the Ebury API request to BOS and that’s all. In that case, in case that the contact belongs to more than one client brand, info from all the clients from all brands of the contact is going to be retrieved. This point needs to be agreed with the product team.

Pros: Less params to send to BOS from Ebury API.

Cons: Changes needed on BOS. Returns all clients from a contact without taking in mind the brand.

Problems to select the brand to connect with Verify

Similar to the previous point, Verify connection is using the brand indirectly. It is used on the next places:

  • First, the brand data is retrieved from the BOS endpoint /api/v1.0/brand sending the EBO url as parameter.

  • After this, verify_username and verify_password are extracted from the BOS response to set up the connection with Verify.

Summarising, Verify requires an user and password that are stored on the brand data.

Right now, the /brand BOS endpoint requires EBO url to search the brand data on the database, but Ebury API is always statically sending the Ebury Core EBO url, it isn't dynamically calculated.

Proposed solution

As a proposed solution, the /brand BOS endpoint can accept the brand code, and use it as an optional filter too. So Ebury API will always send it.

On the other hand, Ebury API doesn’t store the brand right now. Analysing the model ClientContact from BOS, there is a method called get_online_brand (here) that retrieves the brand of a contact. This field can be included on the BOS response for /contact endpoint, so Ebury API can store the brand code on DynamoDB on the scaSessionData object in order to be used later to get the verify_username and verify_password on /brand request.

The brand code needs to be propagated on API-Auth every time the VerifyClient is initialised with username and password (generating the 2FA code - here, and verifying the 2FA sended by the client - here).

To take in mind: The get_online_brand method returns the brand of the client, in case the contact's clients have just one. If the contact's clients have more than one brand, and one of them is Ebury one, Ebury brand is returned, othercase the first one is returned.

This proposed solution will require one task on BOS side to accept the brand code as filter on /brand endpoint, another BOS task to include the brand code on the /contact response and a task on API-Auth to retrieve the brand code from /contact and use it on /brand.

Pros: Brand will be dynamic.

Cons: Changes on BOS side. If the client has several brands, just the EBP or first one are going to be used (Do Ebury API contacts have more than one brand?)

Problems to select the brand to render the templates

There are two different templates used: one to render the login page (here) and one for the 2FA verification page (here).

The brand is used to retrieve the template from s3. On Oauth2.0 requests the templates are always showing the background from Ebury (independantly of the environment), so showing here the Ebury ones too should not be a problem.

It is required to review s3 to know how many templates we have: if the brand is not found, the template is not properly rendered, so there is high probability that we are wrongly rendering it for EMP clients on OAuth 2.0 authentication.

Proposed solution

Keep Ebury brand to retrieve the template from s3, as OAuth2.0 is also always using Ebury one.

No tasks required here.

Pros: Less complexity (Ebury API doesn’t have information at that point to retrieve any info about the brand or the contact that it is going to use this endpoint).

Cons: Ebury template is always used.

Problem mixing fields on dynamoDB and redis

Bearer token on SCA endpoint is saving Brand on backend fields on dynamoDB and Redis (here).

Proposed solution

SCASessionData object will have available the backend field, so we can populate it with the correct value instead of having it wrongly hardcoded with the brand. Fixing it on dynamo will solve Redis too, as Redis is relying on Dynamo object

This proposed solution will require an API-Auth task to store the proper field on dynamo.

Pros: Correct data stored on Dynamo

Cons: -

Problem mixing fields on destroy session step

The last SCA endpoint requires to connect with BOS on api/v1.0/clients, but the brand is sended as backend field (here).

Proposed solution

This endpoint is mentioned in a previous section (here). The proposed solution is the same as that section.

Summary of the required changes

This is a summary of the required changes.

To make PSD2 works with EMP clients, these changes will be needed:

  • API-Auth task to include retry on BOS connections and to save the selected backend on DynamoDB on scaSessionData model.

  • API-Webapp task to include retry on BOS connection on FCS Ebury API endpoint.

  • BOS task to remove the brand usage from an Ebury API user on /contact endpoint.

  • API-Auth task to avoid sending the brand on the /contact BOS request.

  • API-Auth task to avoid sending the brand on the /clients BOS request (In case the product team agrees, check here for full information).

  • BOS task to include the brand code as a filter on /brand endpoint.

  • BOS task to include the brand code as response on /contact endpoint.

  • API-Auth task to retrieve the brand code from /contact and use it on /brand request.

  • API-Auth task to correct the field on DynamoDB on token generation.

Alternatives

As an alternative, Ebury API can receive the API client_id. Using this client_id, the Client table on DynamoDB can be hit, and retrieve the backend and brand values, as it is done on the usual usage of Ebury API for EMP clients (here).

This will solve the problems with the brand and backend fields on most cases, but it requires that the final user (or equensWorldLine as intermediate) sends this parameter to Ebury API, so there will be an impact on final clients that uses PSD2 (for both, Ebury Core and EMP clients).

Also, this require to onboard PSD2 clients as Ebury API clients: sending them API credentials too (client_id and client_secret).

An important point here is that right now, Ebury API haven't any client using PSD2, so any breaking change (e.g. make mandatory the client_id) won't have any impact on current clients.

If this solution wants to be taken into account it will be analysed deeper the exact parameters and changes that will require. To take this solution is a product decission.

The issues that are not covered by this alternative are:

  • The problem to select the backend on FCS endpoint will remain: in that case it is just and endpoint, so the option can be to develop the retries over the backend.

  • The problems on mixing fields: this is a current problem independantly of the environment used (Ebury Core or EMP) that should be fixed.

Caveats

The retries solution to solve the backend problem will be temporal: Ebury API will remove support for multibackend once EMP migration was done.

The retries solution on SCA implies that the email of the contact just live in one backend. If an email is going to be on both systems, Ebury Core one is going to be selected. This point require to understand how the EMP migration will be done: if during that migration there will be a slot of time that emails are live in both systems, API should search a mechanism to prioritise by default one backend or another (probably an environment variable).

Similar case happens with the backend problem on FCS endpoint: by bussiness definition one IBAN should not be on both systems, so the retries solution will work. But it is needed to understand how EMP migration will be done, if the same IBAN is going to be live on both system during a slot of time, API should search a mechanism to prioritise by default one backend or another (probably an environment variable).

Operation

All the PSD2 operations are performed using Ebury API side, so no operations directly needed here. The flow of the payments created will be the same as right now for Ebury Core on PSD2.

There is a special case: funds confirmation flow. This flow requires that Client servicing department manual give consent through the web interface to an user to use this endpoint. This is the same process than they are currently doing for Ebury Core clients.

Security Impact

As a security impact, EMP clients that uses Ebury API will be allowed to use PSD2, that includes verification on payments flow.

Performance Impact

The proposed solution will have a negative impact on the performance: adding the retries will do the requests a bit slower as it is going to internally do two calls instead of one in the worst scenario.

Initially this negative impact will be done just for EMP clients or failed requests, due to Ebury Core one is the going to be the first attempt.

On SCA requests, once the first requests found the contact on Ebury or EMP environment, this backend will be selected and next calls will be done directly to the choosen environment.

Developer Impact

The proposed solution will affect ebury-api-auth (auth-webapp service), ebury-api-webapp (app-webapp service) and bos.

Data Contracts

None

Data Sources

None

Deployment

The deployment must be done on several steps:

  • First, BOS changes need to be deployed before deploy API-Auth ones:

    • BOS has to remove the mandatory brand on /contact before API-Auth stops to send it.

    • BOS has to include the brand code on the /contact response before API-Auth starts to use it.

    • BOS has to include the brand code filter as optional on /brand before API-Auth starts to send it.

  • Second, API-Auth changes can be safely deployed.

API-Webapp changes are all related to the retry on the Funds Confirmation (FCS) endpoint, so it can safely deployed without dependencies.

An important point here: these changes are related to Ebury Core and EMP environment, but the code will be deployed on all the existing environments: BOS need to be deployed on all environments (like sandbox) before starts the deploy of API-Auth.

Dependencies

None