Ledger Service Proof of Concept
Reference Documents
Background
The Back Office System (BOS) Application Ledger is currently tightly coupled with the code in the monolithic application.
This project is a Proof of Concept (PoC) that aims to demonstrate how the Application Ledger functionality can be replicated in a service outside of BOS.
Problem Description
Legacy Technology
Python 2.7 and Django 1.11 reached the end of their life in 2019. There is no support security patches or updates for these technologies making them susceptible to performance issues and security vulnerabilities.
The ledger implementation technology is locked into the legacy technology of the monolith and cannot be independently upgraded to benefit from newer tools and technologies.
Scalability Challenges
The current ledger is part of a monolithic application. The ledger capability cannot be scaled in isolation. therefore the entire monolith must be scaled vertically to improve the performance of the ledger. This is expensive and limited.
Maintenance Challenges
The ledger code is part of the monolithic application codebase. Over time it has become increasingly complex and tightly coupled with the application code - making it more challenging to maintain and extend.
Collaboration Challenges
The work of multiple developers working on the same monolithic codebase needs to be carefully coordinated. There is a risk of unintended interference with the work of others and unexpected side effects. Without well defined boundaries, this limits the pace at which changes can be made.
Lack of Agility
The maintenance and collaboration challenges lead to a lack of agility in implementing new requirements. This limits the pace at which functionality supporting new products can be implemented - which subsequently impacts revenue.
Limited Testing
The lack of well defined boundaries for the ledger functionality limits and compromises the effectiveness of automated testing. A clearly defined API for the ledger functionality enables full functional and performance testing - leading to improved reliability and performance.
An API also enables contract testing - making frequent releases safer in a microservices environment.
Limited Application Level Concurrency Control
User level concurrency control is not currently part of the ledger capability.
There is a separate optional mechanism (Duplicate Movement Chains) which is used at source on an ad-hoc basis to identify and reduce the impact of clashing concurrent user actions.
User level concurrency control should be an inherent part of the Ledger Service to ensure the application level consistency of the ledger in all use cases. The user level concurrency control mechanism also needs to be accessible by all clients of the Ledger Service in a distributed microservice architecture.
Requirements
- A stand-alone Ledger Service providing a common application ledger capability to all of Ebury’s applications and services.
- A technology stack that will be supported and easily maintained in the long term.
- A service that scales easily.
- A well defined interface that supports :
- idempotent requests
- optimistic concurrency control
- transactions spanning multiple movements
- pending and posted movements
- An extensible data model that has:
- immutable data
- double entry bookkeeping
- extensible accounts and ledgers.
- user defined schema-less metadata.
- A query interface that supports :
- efficient balance queries based on posted and value times.
- queries of user defined metadata
- An immutable, append only transaction log
- Verifiable data lineage for auditability.
- Supported backup and restore procedures
- A CDC stream that can be connected to Kafka
Scope
This project is a Proof of Concept that the application ledger functionality can be migrated out of BOS.
How an Application Ledger Service may be implemented and any choice of software / services for a production implementation is outside the scope of this proposal.
Supporting queries from BOS to a new Application Ledger Service is not part this proposal.
The main goals are :
- Demonstrate that the ledger capability can be replicated outside of BOS.
- Demonstrate that the balances of ledgers implemented by the new Ledger Service are the same as those maintained by the Ledger Service internal to BOS.
- Demonstrate that the Events generated by the new Ledger Service are equivalent to events generated for the audit service and the treasury service by BOS.
- Discover any issues in mapping legacy workflows onto a new Ledger Service.
Solution - A Ledger Service
Setup a new Ledger Service that can be accessed by any tool or service.
Run the Ledger Service in parallel with BOS replicating the BOS Ledger transactions.
Demonstrate that the balances and history reported by the Ledger Service are accurate with respect to the BOS Ledger.
Phase 1 : Define and use an Internal Ledger API
Currently, facades and service interfaces exist in the BOS codebase for Ledger related functionality - however not all Ledger related activity is performed via these interfaces.
In this phase, all the functionality for creating ledger entries in BOS needs to be encapsulated behind a single well defined API.
Phase 2 : Replicate Internal Ledger API calls to an External Service
All calls made to the API creating new ledger entries are packaged and communicated asynchronously to a new service that manages the proof of concept.
At periodic intervals, the ledger balances are also communicated asynchronously to the new service.
Existing Ledger functionality in BOS remains unchanged - but a stream of activity and balance checkpoints are sent to a new service so that it can replicate the Ledger functionality.
Phase 3 : Replicate the Ledger Functionality in the Ledger Service
The new Ledger Service implements the Ledger functionality.
The new service is run in parallel with the ledger internal to BOS.
The Proof of Concept Orchestrator maps the BOS ledger activity into calls to the new Ledger Service.
Phase 4 : Capture and Publish Ledger Changes
The Ledger Service publishes events encoding the ledger state changes onto Kafka.
The Proof of Concept Orchestrator reconciles the calls made to the Ledger Service with the events on Kafka. It also monitors the balances of the new ledger service relative to the balance checkpoints received from BOS.
Phase 5 : Demonstrate Equivalence
Demonstrate that the balances of ledgers implemented by the new Ledger Service are the same as those maintained by the Ledger Service internal to BOS.
Demonstrate that the Events generated by the new Ledger Service are equivalent to events generated for the audit service and the treasury service by BOS.
At this point, BOS still uses its internal ledger - however we have proved that the ledger functionality can be replicated outside of BOS.
Alternatives
Using the BOS CDC Stream
Ledger activities could be reverse engineered from the BOS database CDC stream and replicated in the new ledger service. This would be less obtrusive - however, demonstrating that BOS code can be refactored to create entries in a new ledger service is part of the proof of concept.
Using the Data Warehouse
The data warehouse is constructed from the BOS CDC stream. Consequently, as for the CDC stream, it would not demonstrate that BOS code can be refactored to use a ledger service.
Caveats
The Proof of Concept aims to demonstrate that ledger entries created in BOS can be replicated accurately in a new service. It assumes that once the ledgers are replicated in a new service, queries for balances and ledger data in BOS can be redirected to a new service.
Supporting queries from BOS to a new ledger service is not part of the proof of concept.
Operation
As a proof of concept, this is intended to be run in production for a limited period before being decommissioned. The team owning the service will be the only users of the service.
Security Impact
The project replicates BOS ledger entries in another service. However, as a proof of concept, the only users of the service are the team building it. There are no internal or external interfaces.
Performance Impact
Details of internal ledger API calls in BOS are serialised and communicated asynchronously to the PoC Orchestrator. However, these do not happen at a rate that would impact the performance of BOS.
Developer Impact
The proof of concept is local to the team developing it and does not impact other developers.
Data Contracts
None of the existing data contracts are changed.
Data Sources
The second phase constructs a data source that is used to drive a parallel prototype ledger service. There are no other consumers of this data source and no other data sources are used.
Deployment
As detailed above, the proof of concept will be deployed in five phases.
Dependencies
This is a stand alone proof of concept. There are no prerequisites for the project and there are no dependents either.