AsyncAPI for documentation in Events based architecture
Ebury has adopted an Events based architecture (aka EDA) as described here.
This RFC proposed creating the documentation of the events architecture and the messages currently existing in our ecosystem and those to come using the AsyncAPI specification.
Further usages based on the documents created needs to be analized separately.
Problem Description
Software today is becoming increasingly interconnected. In a microservices ecosystem where a bunch of the participant will publish and/or consume, the ability to have machine/human readable documentation is needed to keep trace of all the expected participants.
As per today, we have BOS publishing 7 different types of messages, and Webhooks project allows external consumers to subscribe to those messages. ADS will publish 2 different messages and consume another pair. But this is just the beginning.
As the number of messages interchanged through the services will grow, a correct documentation will help us not only controlling what we have, but also to expose our capabilities.
Background
In an Event driven architecture, there is an underlying contract between multiple parties. One of the parties is publishing and others are consuming, and they all need to know the shape of the exchanged messages, so they can validate it and extract the information as desired.
From the external side, in the same way we document our REST APIs, we can consider Events as an API that can be documented. So, with same reasoning the industry has been supporting documentation in APIs, there's no big need of repeating here the list of advantages.
On the internal side, for the sake of control and monitoring, we need to be able to have a full list of the messages we are managing.
OpenAPI (fka Swagger) has raised as one of most widely used specifications for documenting http APIs. AsyncAPI took in origin this as a base to start due to the number of similarities, but extending where the difference appears.
Solution
AsyncAPI provides the ability to create a documentation which is understandable by humans and machines.
AsyncAPI is protocol agnostic, currently supporting the most common protocols such as Kafka, SNS/SQS (our current cases), AMQP, MQTT, others and growing.
The initiative is supported by an active community providing a set of tooling that allows to extend the capacities documenting to validation or testing.
Specification
AsyncAPI covers the contractual representation of the API messaging system. We could define our topics and payloads with something like this.
asyncapi: 2.0.0
info:
title: BOS trades application
version: '0.1.0'
servers:
kafka-rest-proxy:
url: kafka-rest-proxy
protocol: kafka
channels:
ebury.bos.trades-models-spot_deal.SpotDeal.created:
publish:
message:
payload:
$ref: 'path/to/producer-schema.avsc/#SpotDeal'
ebury.payments.beneficiaries.events:
consume:
message:
payload:
$ref: 'https://github.com/Ebury/beneficiary-services/blob/main/schemas/ebury.payments.BeneficiaryDomainEvent.avsc'
Avro is the selected schema definition language for Kafka messages in Ebury, and the schema files for defining message payloads can be referred to as shown above.
Note: AsyncAPI can be defined in YAML or JSON.
Every service will have its own asyncapi.yaml file living in the same repository as the service. To avoid message
definitions repeated across all the services, plain JSON files can be setup in a commons repository where all the common messages live in. Then each asyncapi.yaml can reference them where appropriate.
The asyncapi.yaml file must be present in the service repos even if the service does not provide/use an actual API (but directly consumes events from Kafka, for example). The purpose is to find all producers and all consumers of a channel (Kafka topic) by searching through all the service repos. It would be possible to build a Jenkins pipeline that generates visualization (directed graph) of the producers and subscribers using the information in the AsyncAPI definitions across all service repos.
For mono-repos, hosting multiple services (such as beneficiary-services) the AsyncAPI file definition shall follow the naming convention asyncapi-${SERVICE_NAME}.yaml
Testing
The tool microcks provides a mocking and testing platform where executing contract examples or Postman collections.
Validation
The tool asyncapi-validator provides a way to validate messages based on the defined schema.
Other tools
The full list of tools can be accessed in the official site.
Alternatives
There are no much real alternatives in what comes to definition of an EDA. AsyncAPI appeared from the lack of options in the industry and was the first covering this scenario. I really haven't found other real option, and if they exist, it seems unlikely to be as mature and backed up as AsyncAPI.
CloudEvents specification covers a way for describing events data in a common way. But it main constraint is focusing more in the message whilst AsyncAPI aims to describe how the applications are connected. Both can be complimentary and maybe could be interesting using CloudEvents specification to define the envelope of our messages.
Only human readable definition can be faster completed, but we would be missing extra features for testing and validation.
For organizing the documents there are other options:
- Instead of having them all in a single separate repository, we could have them located in each of the services publishing/consuming.
- Having just one document specifying the whole services architecture is not recommended. It has problems not only with the increasing size, but also it would be less flexible for versioning messages.
Caveats
AsyncAPI is not a magic bullet – there will always be competing standards and additional, case-specific solutions. That being said, AsyncAPI is very powerful at what it does and is a very good solution for any message-driven use case.
We would need to think on how to force all our internal services creating the specification document.
External clients connected to Webhooks can't be included as consumers?
It is possible to combine AsyncAPI with Kafka's schema registry by following steps detailed in Avro's parser plugin. Using the parser, AsyncAPI documents can be created referencing schema from the registry.
const parser = require('@asyncapi/parser')
const avroSchemaParser = require('@asyncapi/avro-schema-parser')
parser.registerSchemaParser(avroSchemaParser)
const asyncapiWithAvro = `
asyncapi: 2.0.0
info:
title: BOS trades application
version: 0.1.0
servers:
kafka-rest-proxy:
url: kafka-rest-proxy
protocol: kafka
channels:
ebury.bos.trades-models-spot_deal.SpotDeal.created:
publish:
message:
schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
payload:
$ref: 'https://schemas.example.com/user'
`
await parser.parse(asyncapiWithAvro)
Operation
A repository will be created in Bitbucket by any user with permissions.
Generating documentation needs to be automatized in jenkins.
An accessible URL to access documentation needs to be supported by devOps team.
Security Impact
N/A
Performance Impact
N/A
Developer Impact
Developers needs to be in charge of creating/updating documentation together with any change in messages.
Data Consumer Impact
Versioning of messages is needed for not breaking compatibility with consumers.
Deployment
N/A
Dependencies
Events are already live, documentation can start at any point with the messages currently exchanged.