Migrating BOS

The Ebury Back Office System (BOS) has fulfilled its original purpose - and much much more. It has enabled the company to grow to the size it is today and supports its global operations.

As BOS has grown, it has become increasingly difficult to maintain, upgrade and add new functionality. It has fulfilled its purpose and the technology used to build it is now outdated and unsupported.

It needs to be replaced piecemeal with new services using modern technology and best practices - in a way that minimises disruption to the business.

It needs to be incrementally upgraded and re-architected to support Ebury over the next 5 years and beyond

Migrating BOS to Right-Sized Services

When migrating away from legacy monolithic applications, the best practices are :-

• Small incremental changes minimising risk
• Short-lived branches merged often into the main trunk of development
• Parallel running and testing of new functionality in production
• Switchable functionality with rollback.
• Continuous resourcing of migration to the best technologies and architectures.

Updating legacy software systems is a common problem for the IT industry and there are well defined patterns and best practices to migrate to a more sustainable and evolvable architecture.

BOS has been implemented using Django, a 3-layer web framework.

The user interface layer, the application layer and the database layer are tightly coupled and migration needs to be performed on all three layers.

• Migrating Services The application logic can be migrated into new services - whilst retaining the legacy user interface and accessing the legacy database.

• Migrating Databases A service will own the data it is responsible for - whilst still obtaining data from other services it requires to fulfil its function. Each service has its own database and there is a process of migrating data from the legacy BOS database into the new service.

• Migrating User Interfaces The new user interface needs to be able to selectively incorporate views from the legacy system (Django, Marionette) and the new UI (React) to provide the user with a single application front end.

Self-Contained Service Target Architecture

New services are self-contained and include :- * A synchronous gRPC interface supporting channels * An asynchronous Kafka AVRO interface for :- * receiving commands from other services and sending responses. * sending commands to other services and receiving responses. * consuming relevant domain events * producing domain events * A logical database owned by the service

Migrating Services

The Branch by Abstraction Pattern

The Branch by Abstraction pattern is a standard pattern for moving an interface internal to a monolith into new external service (1)

• Identify the seams surrounding cohesive functionality inside the Monolith -- belonging to one Domain
• Create a well defined abstraction for the functionality
• Implement an interface for the abstraction inside the Monolith using existing code
• Incrementally refactor all clients of the functionality to use the abstraction.
• Create a new external service replicating the functionality
• Create an interface in the monolith that calls out to the new external service
• Parallel run legacy and new interfaces in production and use a feature switch.
• Remove the legacy implementation

This moves the application logic into an external service - but retains the legacy database and does not impact the legacy user interface.

Migrating Databases

Split The Code

Split the code first. Create a new external service.

In moving the functionality it should become apparent which data belongs to the service and which data is required by the service.

Data belonging to the service is initially accessed directly from the BOS database - but using the latest tech stack (e.g. Python 3.x, SQLAlchemy, etc) and patterns (E.g. The Data Repository Pattern, The Unit of Work Pattern, etc) (4)

Data required by the service is initially accessed via a Service API in BOS.

The new service must only access the data it owns directly from the database.

Logically Separate Databases

De-couple the Service data from the BOS DB :- * Replace foreign key relationships with natural keys or UIDs. * Refactor schema to reflect the service domain data model. * Place the service data in a separate schema.

BOS data and the Service data are logically separated - but share the same database.

Provision a Service Database

Provide the Service with its own database initialised as a replica of the Service Schema from the BOS DB.

The service is now self-contained

Decouple Services

Synchronous Service APIs are a necessary during the transition to the new architecture - but result in tight coupling between all services.

As more and more types of domain events are published on Kafka, the service listens to the events on Kafka and makes local copies of the data it needs.

Over time the service has less and less need to request data directly from other services.

This reduces coupling and improves reliability and performance.

Migrating User Interfaces

The Bosphorus architecture is described in the RFCs Channel Architecture & API Styles and Frontend Architecture for BOS Operations Dashboard. This section considers the sequence of actions required to implement and migrate a backend service in a way which is transparent to the Backend for Frontend (BFF).

New Channel Endpoints

All new channel endpoints must be defined in the context of a full service that is to be migrated. The The Branch by Abstraction Pattern detailed at the beginning of this blueprint still applies. An abstract interface is defined with the first implementation of the interface being inside BOS with the legacy codebase.

Create New Service

Using the The Branch by Abstraction Pattern, a new external service is created and tested with a feature switch.

The Channel API is treated in the same way as an internal client.

Call the New Service

The API endpoint is moved to the new service and called directly.

Legacy clients internal to BOS access the service via the outbound gateway.

References

1. Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith. Sam Newman.

2. Refactoring Databases: Evolutionary Database Design. Scott Ambler.

3. Working Effectively with Legacy Code. Michael C. Feathers.

4. Architecture Patterns with Python: Enabling Test-Driven Development, Domain-Driven Design, and Event-Driven Microservices. Bob Gregory and Harry Percival.