mTLS with Kong

This document describes the infrastructure changes required to move the client certificate validation in mutual TLS (mTLS) authentication from the mtls service to Kong.

Reference Documents

Reference	Document Location
[1]	API Gateway motivation
[2]	Channels IAM

Problem Description

One of the reasons for using an API gateway is that it can absorb non-business functionalities that are needed by multiple internal services. These functionalities can easily be applied to any service that we want to publicly expose, by means of API gateway configuration instead of implementing them as services and hosting them multiple times.

One such functionality is mTLS. mtls proxy, which implementes the client certificate validation logic, was planned to be decommissioned when the API gateway was in place [1]. Kong is now in place as our API gateway, we can proceed to decommission mtls proxy and move certificate validation to Kong.

Background

We have an mtls proxy based on nginx which terminates TLS, validates client certificates against known certificate authorities (CA) and forwards a header X-Client-Certificate-Subject-DN containing the certificate subject DN to upstreams ({auth,app,webhooks}-proxy services) The certificates are bound to access tokens during the authentication flow (by auth-webapp). The upstreams ({auth,app,webhooks}-proxy) further check that the submitted client certificates match the ones bound to the access tokens.

The current mtls proxy is reachable with a separate dns: trusted.ebury.io (prod), trusted.ebury.rocks (staging) or trustedsandbox.ebury.io (sandbox).

We can now decommission mtls proxy. In the future, once authentication is moved to Kong + Identity Provider (IdP) [2], the other proxies and parts of auth-webapp can be decommissioned too [1].

Solution

Kong is currently running behind an AWS ALB enhanced with AWS WAF. In order for Kong to validate the client certificate, it needs to terminate TLS.

We introduce an additional NLB in front of Kong, which allows TLS passthrough. We will point trusted domains (trusted.ebury.io in prod) to this NLB. We can enhance the NLB with AWS Shield. This new NLB will point directly to the Kong ECS instance. Kong will have the ability to terminate TLS. However requests to api.ebury.io will be TLS terminated by the ALB, while requests to trusted.ebury.io will be TLS terminated by Kong.

Customers will not be affected by this change. If they visit api.ebury.io with a browser, the browser will not prompt for a certificate.

We cannot configure a WAF for the NLB. We can instead attach it to the ALB created by the Kubernetes Ingress Controller, after Kong and before the traffic reaches our services in Kubernetes. We also need to add a WAF in front of all other services that we are accessing from Kong, such as PSPP API.

Our sandbox Kong instance already points to CloudFront + WAF (because sandbox is still hosted in ECS), so sandbox won't require an extra WAF.

A future IdP will have to terminate TLS too to implement mTLS certificate binding.

Service Ownership

New Service	Service Name	Service Owner
No	Kong	API Team

Alternatives

WAF at Kong level

An API protection mechanism can be installed at Kong level.

As Kong is based on openresty (nginx), opensource WAFs such as ModSecurity can be installed as nginx modules.

Also, third parties companies like Wallarm, provide API protection services. It can be installed separately in ECS but this approach looks the same as having an extra ALB with WAF capabilities. It can be installed in K8s, integrated within the Kong Inngress controller or as a sidecar container. However, our Kong is not deployed in Kubernetes yet.

Single entrypoint

A single entrypoint to Kong will be an NLB, which allows TLS passthrough. We can enhance the NLB with AWS Shield. We will move the WAF after Kong.

We will use a single domain for both regular API traffic and mTLS traffic having multiple domains was not a business requirement, but a technical decision at the time mtls proxy was developed. Which means that during a migration window, we will point both api.ebury.io and trusted.ebury.io to the NLB in front of Kong. After the migration window we can decommission the trusted.ebury.io domain.

Because our clients will connect to the same domain no matter if they use mTLS or not, regular traffic and mTLS traffic will have the same TLS policies (same TLS versions, ciphers etc). At the moment mtls proxy is enforcing TLS 1.3, but we will need to relax the requirement to 1.2+.

When clients visit api.ebury.io with a browser, on some platforms the browser might ask for a certificate. This will be a drawback if we decide to use the same Kong setup in front of other Ebury services that have a web UI.

Caveats

See Security Impact.

Operation

No changes to how Kong runs.

Security Impact

For mTLS traffic, WAF will analyze traffic after Kong, which means Kong itself and all used plugins are not protected by the WAF rules.

Kong only provides api gateway capabilities: rate limiting, auth mechanisms, ip-based protection, content encryption, request & response transformations and others. No business logic runs in Kong and Kong does not and should not directly access business data in any way. WAF will kick in after Kong and will check requests that pass through Kong, which could have their original form or could have been changed by Kong, depending on the endpoint and enabled capabilities.

Performance Impact

None overall. For mTLS traffic, client certificate validation is moved from mtls proxy to Kong.

Developer Impact

None

Data Contracts

None

Data Sources

None

Deployment

We can deploy in stages with DNS weights, gradually moving traffic from mtls proxy to Kong.

Dependencies

None