Auto-generated client libraries from OpenAPI specs

This document proposes to automatically generate and publish client libraries for Ebury services that expose a REST API. Our Continuous Integration and Continuous Deployment (CI/CD) pipelines shall automate these tasks based on the OpenAPI specification of the service.

Problem Description

Services expose the functionality they implement by means of a remote API. The team building a given service is currently in charge of developing (at least) one client library to facilitate the consumption of the offered capabilities in other services or applications. Instead of wasting development resources in creating, updating, and publishing client libraries, this document encourages to automate this process.

In particular, we describe how to automatically generate client libraries for REST APIs using OpenAPI Generator. As long as teams maintain an OpenAPI specification for their services, libraries for most programming languages can be created and published to the appropriate registry automatically.

Background

One of the main benefits of REST APIs is that they can be easily consumed by third parties. However, the developer experience improves significantly if the API has a well-maintained client library in the programming language of the client application. Otherwise, each client application would have to solve the same kinds of problems; e.g. the network and serialization libraries to use, or the exception hierarchy and corresponding error handling, among others.

While this applies to both public-facing and private APIs, the impact on our internal workflow might be significant as we transition from monolithic applications to micro-services1. Splitting the monoliths helps avoid development bottlenecks, and therefore it allows technical teams to scale better. However, in such a new scenario, multiple services must collaborate among themselves in order to implement a business case. If we have N Python services using the API of another one, we want them to use the corresponding client library instead of duplicating the work N times. Hence, at this moment teams build client libraries for services as they are required.

But the number of libraries required per service could start to grow rapidly. As we move into a polyglot environment with applications and services written in different languages (e.g. Javascript) or flavors (e.g. synchronous, threaded, or asynchronous Python), the overhead of creating and maintaining these libraries threatens to (partly) jeopardize the scalability gains achieved by adopting micro-services.

In order to put this overhead in perspective, note that a team member may require about two weeks (one Scrum iteration) to create a single client library for a small service. But the main problem is that every future change in the service may require the team to update the library accordingly and to publish the new version in the appropriate registry. Since this is a manual step at this moment, it's error-prone and easy to forget. Moreover, creating, updating and publishing libraries are not value-added tasks, just technical requirements.

In order to let our developers focus on tasks that provide our clients with real value and avoid unnecessary overheads, we encourage all teams to automatically generate client libraries for the micro-services they build.

The solution depicted in this document is intended for services exposing a REST API. In the remainder of this section we briefly review the building blocks required for our proposal: OpenAPI specs and the OpenAPI Generator tool2.

OpenAPI

The OpenAPI Specification defines a standard document format to describe RESTful APIs. OpenAPI specs are JSON objects that can be represented in either JSON or (the more human-friendly) YAML formats.

The benefits of maintaining OpenAPI specs are multiple. They document the capabilities offered by the service; they serve as a contract between provider and consumers; and we can take advantage of the ecosystem of tools around them (documentation formatters; API clients; automatic generation of client libraries, server stubs, and mock servers; and many more). Nonetheless, having OpenAPI specs is a recommended practice at this moment.

There are multiple ways to maintain the OpenAPI spec of a service. Some common ways to do this (in increasing order of preference):

  • Manually. Have the spec file under revision control in the service's git repository.
  • Generated from code annotations. Use framework plugins that let you insert OpenAPI definitions as part of your docstrings or using some kind of annotation. The spec is generated automatically from such annotations.
  • Generated from code. Use a framework able to extract OpenAPI specs from your code. E.g. FastAPI leverages Python 3's type hints to automatically generate the OpenAPI spec.

OpenAPI Generator

Once we have the OpenAPI spec for the service, we can leverage OpenAPI Generator to create our client libraries.

OpenAPI Generator is an open source Java application that generates clients, servers, and documentation from OpenAPI specs. The project maintains public Docker images to run the application.

It comes with built-in generators for more than 50 programming languages (including Javascript, Typescript, Java, Go, or Python). The auto-generated code should be enough for most purposes. But in case it's needed, we could fine-tune the output by means of writing our own templates (easier) or creating a custom generator (more involved and less likely to be required).

The client libraries created via the default Python generator have these nice features:

  • Support of Python 2/3.
  • Client-side validation. I.e. if you pass the wrong params to an API call, the library will raise an error instead of sending the request over the network. This includes some type validations like regex matching for string params. It can be disabled via client code if needed.
  • Support of synchronous and threaded clients.
  • Support of asynchronous clients. We can generate additional client libraries with asyncio support.

Solution

Description

Following our current best practices, each service exposing a REST API must have a corresponding OpenAPI spec. We propose that client libraries must be automatically generated from such spec using OpenAPI Generator.

The CI/CD pipeline must ensure that the client libraries can be correctly generated in every build. Otherwise, the build shall fail.

The CI/CD pipeline must publish the new version of the client libraries each time the service is deployed successfully. I.e. the pipeline must be in charge of uploading the libraries to the corresponding registries (e.g. Ebury's PyPI in the case of a Python library).

Benefits

  • Lower development overhead. Client libraries are automatically generated instead of manually developed.

  • Lower management overhead. We don't need additional git repositories to host the code of client libraries any more. Furthermore, when a new version of a service is deployed, the corresponding client libraries are automatically generated and published.

  • As a consequence of the latter, this solution is less error prone because it eliminates human intervention in the process. Overall, the whole system gets more consistent as every service in production has its corresponding client libraries ready to be used.

Implementation

The solution above has been already implemented to generate the account-details-client Python library for the Account Details Service.

In the remainder of this section, we describe the approach we followed and how it can be further developed.

Docker images for OpenAPI Generator

Custom Docker images based on OpenAPI Generator's official image can be created as required. The granularity for these images may range from a single image to generate all Ebury client libraries, to as many images as client languages and flavors are used.

So far, we have one Docker image that can be used to generate client libraries in Python with simultaneous support of versions 2 and 3, and both synchronous and multi-threaded clients:

# openapi_json=<path-or-url-to-openapi-spec>
# project=<project-name>
# version=<lib-version>
# output=<output-path>

docker-compose run --rm openapi_generator \
    -i ${openapi_json} \
    -p ${project} \
    -v ${version} \
    -o ${output}

# docker-compose.yml (snippet)
services:
  openapi_generator:
    image: public.ecr.aws/ebury/eburypartners/openapi-generator-python:1.1.0
    entrypoint: ["/bin/generate-lib.sh"]
    dns:
      - 10.5.0.3
    networks:
      - ed2k
    user: ${USER_ID}:${GROUP_ID}
    volumes:
      - "../:/local"
    depends_on:
      - application

REST services

In order to benefit from this proposal, services must take three simple steps: expose the OpenAPI spec, provide a Makefile target, and add a var in their Jenkinsfile.

Expose spec

Each service that implements a REST API must expose the corresponding OpenAPI spec in a well-known URL (e.g. /openapi.json). An alternative implementation could maintain the spec as a file under revision control. However, we recommend to automatically generate the spec from our code if possible.

Makefile target

The service must declare a generate-lib target in the project's Makefile. The CI/CD pipeline will run this target to create the library. The build will fail if the client library cannot be generated for any reason.

.PHONY: generate-lib
generate-lib: ##@development Generate client library from OpenAPI spec.
generate-lib: openapi_json?= http://account-details.localhost/openapi.json
generate-lib: project?= account-details-client
generate-lib: version?= local
generate-lib: output?= api-client
generate-lib:
    ${DC} run --rm openapi_generator -i ${openapi_json} -p ${project} -v ${version} -o /local/${output}

Note that the value of default arguments refer to the local development environment. In order to generate the client library for the production service, we must provide the corresponding URL for the OpenAPI spec as shown next.

Jenkinsfile arguments

When a service wants to generate and publish a library automatically, the corresponding Jenkinsfile just has to include the openapi_json argument with the URL of the production spec.

Note that the former guarantees that we release the library for the service version currently running in the production environment.

@Library('jenkins-devops@7.39') _

// It calls to https://github.com/Ebury/jenkins-devops/blob/master/vars/eburyPipelinePythonService.groovy
String projectId = 'account-details'
def daemons = [:]
def kwargs = [
        'system_tests' : false,
        'openapi_json': 'https://account-details.eburypartners.com/openapi.json'
    ]
def slackChannels = ['#releases-account-details-service']


eburyPipelinePythonService(projectId, daemons, kwargs, slackChannels)

Jenkins pipelines

For each kind of service, the corresponding Jenkins pipeline must support client library generation and publishing in a conditional way. I.e. to maintain compatibility with services not exposing an OpenAPI spec, the generation and publishing stages only run if the argument openapi_json has been given.

The remainder of this section shows how this has been done for Python services:

Generate library in every build

  • Define new stage.
  • Enter stage only if argument openapi_json has been given.
  • Use Makefile targets server and generate-lib to generate the library. Note that the former is required because the service is exposing the OpenAPI spec that the generator is consuming.
stage('Generate Client library') {
    when {
        beforeAgent true
        expression {
            kwargs.containsKey('openapi_json')
        }
    }
    steps {
        sh 'make server generate-lib'
    }
    post {
        always {
            sh 'make clean'
        }
    }
}

At the moment, the objective of this stage is to make sure that we don't break the auto-generation of the library.

Publish library after deployment

  • Define new stage.
  • Enter stage only if argument openapi_json has been given and a new release has been deployed.
  • Use Makefile target generate-lib to generate the library from the spec exposed in the production environment.
  • Upload library to the appropriate registry. In the case of a Python library, use the devpi module.

Note that the current implementation (see below) uploads the library to two different registries: One of them is used by BOS and it should be deprecated soon.

stage('Upload Client library') {
    when {
        buildingTag()
        beforeAgent true
        expression {
            kwargs.containsKey('openapi_json')
        }
    }
    steps {
        sh "make generate-lib openapi_json=${kwargs['openapi_json']} version=${env.TAG_NAME} output=${env.PROJECT}-api-client"
        dir("${env.PROJECT}-api-client") {
            devpiRun {
                script {
                    devpi.use("${env.DEVPI_URL}")
                    devpi.login('jenkins')
                    devpi.use('jenkins/dev')
                    devpi.upload()
                    devpi.pushLocalRepo("${env.LIBRARY_NAME}", "${env.TAG_NAME}", 'jenkins/staging')
                    devpi.pushRemoteRepo("${env.LIBRARY_NAME}", "${env.TAG_NAME}")
                }
            }
        }
    }
}

Alternatives

Keep doing this manually

As already explained throughout this document, this option wastes development resources and the overhead will just keep growing as we develop more services and adopt more programming languages.

Automate using gRPC tooling

Services using gRPC can define their API using protocol buffers as the IDL. Specific tools can then be used to automatically generate client libraries following the same workflow described in this document.

However, for services exposing a REST API, the solution described here is the one that applies.

Automate using other OpenAPI generators

OpenAPI Generator has been chosen because it is the SDK generator that supports more languages and it is backed by a large community. However, other generators are available and might be used without impacting the overall solution described here.

Caveats

The Python code that OpenAPI Generator issues by default has some shortcomings. In decreasing order of importance:

  • Client-side validation of server responses cannot be disabled. This might be a problem in some cases. E.g. say we have a service whose response includes an enumerate field. In version 1 the allowed values are A and B. Version 2 adds C as an allowed value. As soon as version 2 is released, clients using the library for version 1 will break if they receive a response that includes C as the value of such field.

  • Some client-side validations cannot be disabled. In particular, trying to pass a request parameter that does not exist in the current library version is not allowed.

  • Error responses from the API trigger a generic ApiException. While the exception includes all the information about the particular error, it would be nicer if they triggered specific exceptions per error type.

Note that all of these drawbacks can be fixed by customizing the generator.

Generators for other languages may have similar shortcomings - and they could also be fixed by means of fine-tuning the generated code.

Operation

N/A

Security Impact

N/A

Performance Impact

No performance impact on services.

CI/CD builds include two conditional extra steps. When they apply, the build takes a bit longer but it is negligible with respect to the overall build time.

Developer Impact

Overall, this solution has a positive impact on development because it removes the need of manually creating, updating, and publishing client libraries.

Services using the described solution must implement the three simple steps referred above. As long as the service is using a supported pipeline (Python service) and supported client language (Python library), the impact is very low.

In order to support additional service types, developers must coordinate with the DevOps team to build the pipeline. The impact in this case is also very low.

In order to support client libraries in additional languages, developers must create a new Docker image based on the official one from OpenAPI Generator and coordinate with the DevOps team. Again, this has low impact.

When the need for customizing the auto-generated code arises, developers must coordinate with other teams also using that generator. Alternatively, the team might use a dedicated OpenAPI Generator image if they want to include changes that might break other services' workflow (and coordinate with the DevOps team if required).

Data Consumer Impact

N/A

Deployment

This solution is already deployed for the described use case.

Dependencies

N/A

References

N/A

1. In this document, the term micro-service doesn't refer to the size of the service. In general, the scope of the service should account for the high cohesion and loose coupling principles of software systems design.

2. Note that OpenAPI Generator is a switchable component of the described solution. Other SDK generators could work too.