Kafka for development environment in Kubernetes

Use vanilla Kafka installed with a Helm chart in each development workspace in order to simulate the behavior of AWS managed Kafka service (MSK). The setup will include a customized image of Kafka Connect with Debezium connector installed.

Problem Description

When developing new services, or migrating services to Kubernetes, development and test environments will need to support the usage of Kafka and Debezium for integration and E2E tests.

Background

As the roadmap for moving Ebury transactional platform into Kubernetes progress, services are being migrated or created from scratch being Kubernetes native.

At the same time, Ebury architecture is also evolving into and event driven architecture powered by Apache Kafka. Furthermore, some of the services follow CDC Pattern, with Kafka Connect and Debezium connector (specifically, BOS database changes being streamed as events in Kafka).

The implementation in staging and production environments consists on a MSK cluster and a ECS Task for Debezium connector. It is not possible to replicate this set up locally, so we are currently simulating it in ED2K with a heterogeneous set of containers orchestrated with docker-compose.

ED8K

ED8K (Ebury development kit for K8s) is outlined here. It is composed of a Kubernetes cluster supporting multiple tenants (development workspaces), and an in-house built cli (ed8kctl) for creating workspaces and interacting with them

Several services available for the workspace are started automatically inside the workspace when it is created, like localstack or mongo, while others have a single instance for the whole cluster, managing multitenancy at application level (like vault)

Solution

Include https://bitnami.com/stack/kafka/helm in the set of charts that are installed automatically on a workspace at creation time.

The inclusion of kafka will be activated by default, as the service is core part of Ebury 2.0

Alternatives

  • Develop our own chart with community images for Kafka and Zookeeper.
  • Use Helm chart for Confluent Platform (https://github.com/confluentinc/cp-helm-charts).
  • Starting a small MSK cluster for each developer workspace would produce a environment closer to production, but it would be costly if done that way by default. We can configure some environments with MSK if needed, but not as a general rule.
  • Deploy a single multi tenant Kafka cluster for all the development and test environments, as we have already done for vault. That would need additional configuration supporting multitenancy at application level, not straightforward with either MSK or our own Helm Chart, specially if we want to avoid noisy neighbors problem.

Caveats

The set up in development workspaces would differ from the setup in production. This is mitigated by the fact that there will be a staging environment fully replicating the production setup, so final tests could be run there.

The configuration for Debezium connector is currently done with terraform and coupled with infrastructure, so it would need to be duplicated, with high risk of drift in configuration. Kafka connect configuration details will be addressed in a separate document.

Operation

DXP team will be in charge of managing templates form the different flavors of workspaces, while development teams will be in charge of creating their own workspaces based on the templates.

Security Impact

Image version will be aligned with the version running in MSK cluster in production. Whenever a update is proposed for the clusters running in staging an produciton, initial tests shall be done in ED8K environment.

Nothing beyond what is already stated in the blueprint for k8s development

Performance Impact

N/A

Developer Impact

Developers will create workspaces on demand with ed8kctl, optionally starting Kafka.

Data Consumer Impact

N/A

Deployment

N/A

Dependencies

N/A

References

N/A