Kubernetes Ingress

Background

Besides services and workloads communicating between each other and working together, a problem already reviewed in the networking blueprint, traffic from outside the clusters needs to be routed to the proper services.

There are three channels: - EBO UI - Ebury external APIs - Operations UI

Each channel will have its own set of allowed sources, authentication, protections, performance enhancements, etc. thus needling different pathways to the cluster workloads.

EBO UI

EBO is currently deployed in publicfrontal VPC on ECS, behind Cloudfront and with WAF attached to load balancer.

EBO current ingress

Ebury external APIs

API services are currently deployed on Kubernetes cluster, but API Gateway (Kong) is running on ECS.

WAF is not attached to the balancer because Kong has its own WAF capabilities.

In addition, it was well established during Kong implementation that we will not use Kong as ingress controller, because that would mean a huge increase in Kong Licenses needed. It was agreed that Kong will forward requests trough an Load Balancer to an Nginx Ingress controller, and this is how it is implemented.

API current ingress

Operations UI

Bosporus is currently deployed on Kubernetes cluster, so far it is using the same ingress as legacy channel.

Operations UI ingress

Legacy

There is a forth "channel", which is the legacy one. That is, BOS and the services currently deployed in the backoffice VPC (Sherlock, Cornado, FXSuite and Verify). This "channel" is currently reachable by:

  • Ebury offices
  • VPN Generic
  • Legacy authenticated proxy

Solution

Provide infrastructure bits and Kubernetes objects and controllers, allowing secure incoming traffic in the different channels in the catalogue.

Target Ingress architecture

Projects will be able to choose from the offer in the catalog which ingress is best suited for their needs by configuring its ingressClass accordingly.

Ingress traffic through means different than ingress offering exposed here will not be allowed.

Load Balancer provisioning

Currently, load balancers are created dynamically by EKS when we define Kubernetes Service object of the type LoadBalancer. Although that was handy for initial development, it is now showing limitations. Load balancers will be provisioned as infrastructure, and ingress (or other services) will be bound to existing balancers.

This is where we draw the line between what is part of infrastructure provisioning and what is defined as Kubernetes objects and managed by the cluster or its controllers.

Additional connections

  • We need support as well the legacy channel.
  • We need to support interaction with applications not migrated to Kubernetes yet.

This means we need an additional Load Balancer, and the whole picture of components will look like:

Intermediate Ingress architecture

Service mesh

Service Mesh is not yet in place. So far, service discovery is being done with FQDN and traffic goes through NAT Gateway, even if the traffic is internal. This has several implications in the Allowed CIDRs blocks in the different Load Balancers that are not shown in the diagrams for simplicity.

Alternatives

  • Use Kong also for EBO traffic, or even all traffic. We are not sure if it is possible, and in case it is possible, if it would bring any advantage. In any case, it would need to be evaluated in depth by API team, as the Platform team does not have at the moment much knowledge on the internals and configuration of Kong.

  • Deploy the current Kong (Enterprise) in the Kubernetes cluster and use it as ingress controller, discarded because of the Kong license model.

  • Use a separate Kong (OSS) deployed in the Kubernetes cluster as ingress controller, or use a different API gateway provider.

  • Deploy the current Kong (Enterprise) in the Kubernetes cluster, but route all the traffic to single Nginx ingress controller (same as the current alternative, but with Kong being deployed in the cluster, so the usage of Kong custom resources in the service Helm chart is feasible). It would require significant work.

Caveats

  • Kong is deployed outside the Kubernetes cluster, so services that require to expose endpoints (like webhooks) will not be able to use Kubernetes Custom Resources for configuring its ingress. The configuration for those endpoint would need to be done in two separate places, in the service Helm chart alongside with service deployment and in the Kong configuration, with some extra mapping and routing configuration needed.

Operation

Load Balancers will be created as part of infrastructure, with Terraform code in GitHub repository that maintains live environments, deployed and maintained by the platform teams.

All the new components (i.e. Ingress and API Gateway) deployed in Kubernetes will be managed and deployed by the platform teams.

The services behind will be deployed and updated by development teams with tools provided by platform.

Security impact

As with everything in Kubernetes, it brings new threats but also remove some of the existing ones. Load balancers that before were in different subnets will be now in the same public subnets, but still protected by security groups (i.e. Ingress service inside the cluster listens in a specific port in the Kubernetes worker nodes, and rule is created on each node security groups allowing inbound traffic only from the Load Balancer).

In any case, we will not be removing any security feature (i.e. WAF, VPN requirement) in the services exposed to the internet.

Performance impact

Higher latency may be experienced as more hops are included in the picture.

Developer impact

All applications in the Channels area will need to support deployment in Kubernetes for all environments types (i.e ED8K and Staging/Production), publishing Helm charts with their deployment specification, and environment and secret configuration adapted as well.

All the needed activities to achieve that end are detailed in the Developers handbook

Data contract

N/A

Deployment

As we are changing systems that are both live and under continuous deployment, we need to guarantee smooth transition from workloads on ECS to workloads on Kubernetes, while maintaining the same security levels. This will mean several intermediate states between the current ingress infrastructure (pictures shown in the background section) and the desired target infrastructure.

Each ingress type can be handled independently, the specific details about different stages for each Ingress will be part of different documents, explaining how we plan to maintain compatibility and steer traffic, as it will need different approaches for each different ingress.