Ebury Transactional API Gateway: the motivation

Current API architecture and networking infrastructure is a bit convoluted.

Replacing (some parts of) the API layer by a good API gateway can simplify them a lot.

This document contains the motivation and a first sketch of the envisioned solution.

The technical description of every problem that needs to be solved by the gateway product are in a separate document with the requirements.

Problem Description

There are some specific problems with the API architecture.

There are a lot of services, and their responsibilites are not clear. Boundaries between them are not well established, and some problems are solved several times. E.g., rate limiting is enforced at several services.

Networking connections are increasing, mostly because webapp is actually working as a gateway.

Some of the parts are not testable neither executable locally. Troubleshooting, instrumentation, and debugging are difficult with such a complex setup.

Background

API services were designed to be a public facade to a subset of the BOS api. It was for legacy reasons that don't apply anymore to new services. We can start thinking of better, less constrained, solution.

As they were never designed to expose several internal services, you can find also some design gaps in the code of the webapp. E.g. What if an environment lacks some of the services? What if a user of an specific brand needs to be connected to a different backend?

Solution

Solution is to deploy a service that acts as a gateway.

Most of the features solved in the API layer can be solved with already existing products if they are properly configured.

Some of the features will need persistence. The gateway will need to store the access tokens, for example.

In the future we will probably face not forecasted problems, so it will be important that the gateway is extensible with plugins or support services.

Alternatives

We could write our own nginx configuration, but it is likely that established products have better implementations, easier to maintain.

We could keep the current services chain/mess with adhoc services, but the related problems are already discussed.

Caveats

No solution will work out of the box. We will need to select a product and configure it accordingly.

Operation

Once we select a production-ready product that supports all of our requirements, we will need to instrument a git-based configuration update/rollback mechanism, or to package the gateway as a portable docker image.

After that, it will be a normal service that will receive updates to support new features.

Security Impact

This is a very special service because it will be the responsible for accepting most of the external traffic, so most of the security measures must be taken here.

We expect the requirements related to security to be approved by the security team, and also they to be aware of every security-related change.

Performance Impact

Having one service instead of several chained ones will reduce the latency, the amount of logs, the internal network traffic, and the amount of memory.

Using a gateway product will be much more performant that having adhoc nginx configurations. The difference will be even more noticeable when compared with python services.

Developer Impact

Simplifying this architecture will ease the developer workflow. Among other reasons, because the ability to test the infrastructure in local will make the process much faster, and because solving the common problems in one place will save time to work on the business logic.

Data Consumer Impact

This gateway will not produce business data.

Deployment

First, we can deploy the gateway as a transparent proxy to the existing working architecture.

Once the configuration of the gateway is instrumented to be maintained in git, we can start evolving it task by task.

Step by step, the gateway will then absorb features that represent whole services, so some of them can be deprecated and furthermore, eliminated.

We cannot just deploy the gateway and drop app-webapp. Legacy apis must be kept behind a transformation layer until eventually refactored. Think that currently we have a mix of apis: some are well design and some are poorly. We can't just expose that to end user. We might want to do a refactor an improve our internal APIs but we are going to have to mantain compatibility at least with already published versions of the API.

Details

We will fit in the gateway everything we can map as configuration, to avoid having small little services reinventing the wheel. The services that does not fit as a gateway configuration will be kept. No service will be re-written.

The service auth-webapp has some functionality that covers some standard authorization mechanisms. This functionality can be solved inside the gateway with some configuration and with little or code. The service will keep the functionality that does other things, like the PSD2 endpoints.

The service mtls covers functionality we expect from a gateway, so it could be deprecated once we have one.

The services app-proxy, webhooks-proxy, and auth-proxy would be deprecated once the authentication is moved to the gateway: they are only checking the validity of a token, some ip filtering, path rewriting, etc.

The service app-webapp will be kept as a public facade for some existing services until they have their own facades or the transformation is solved in the gateway.

Dependencies

No dependencies.

References

No references.