BOS Releases Rotation Policy
Rotation policy for teams to manage releases that are related to any kind of development in BOS.
Problem Description
The release process in BOS is mainly handled by an experienced member within the Developer Experience team. This is performed automatically through Jenkins and it’s split into two different pipelines (the entire workflow is detailed in this blueprint).
Having a single entry point to manage releases in probably the most popular service at Ebury is not scalable due to different reasons:
- The knowledge to perform a release in BOS is not spread to the wider engineering teams.
- The knowledge to perform a hotfix in BOS is not spread to the wider engineering teams.
- Coordination problems in case of illness, holiday, etc, of the release manager.
- Impact on the DXP team's daily work and, as a consequence, in the metrics.
- Lack of developers' visibility for monitoring their tickets when they are deployed in the production environments.
Background
It’s already known that the BOS monolith is the biggest core service in Ebury. The complexity associated with the service causes a special treatment in every workflow that implies any kind of interaction with it.
The BOS releases always were a mess for two main reasons:
- The process contained a lot of manual steps.
- The coordination/communication with the different teams.
A consequence of this was that it wasn’t possible to release more than one time per day in the happy scenario, usually once every two or three days. However, this changed a lot after implementing the workflow automatisation, where basically it’s just required to press a button in SECO, confirm data, and merge and tag the BOS release PR. After that just monitor the workflow as in any other service.
Currently, there are two releases per day, one before 12:00 CET and another one after that. If the code is merged between 12:00 - 17:00 CET, then it’s going to be deployed the next day in the morning release.
On the other hand, If the code is merged after 17:00 and before 12:00 CET (the next day), then it’s going to be deployed in the afternoon release.
Morning release:
- 17:00: SECO & PR release are created
- 08:30: Staging deployment
- 10:00: Merge and tag to master (Trigger deployments)
- 10:10: Automatic Demo envs deployment
- 10:10: Automatic Ebury env deployment
- 11:10: Ebury Mass Payment env deployment
Afternoon release:
- 12:00: SECO & PR release are created
- 12:30: Staging deployment
- 15:00: Merge and tag to master (Trigger deployments)
- 15:10: Automatic Demo envs deployment
- 15:10: Automatic Ebury env deployment
- 16:10: Ebury Mass Payment env deployment
Note: The time is based on the Spanish timezone and the usual time workflow (this could suffer delays that are communicated through the #bosdevelopments Slack channel) so please be aware that those times may not be suitable for people outside the European time zones.
Solution
Implement a rotation system for teams to manage BOS releases.
In order to provide a smooth transition, the Developer Experience team already delivered a number of workshops and training material after the project was finished back in August 2021, and we can firmly confirm that the new release process is working seamlessly (there are some exceptions that are required manual intervention, but this is not common and we already collected the details in this document).
However, the Developer Experience team will provide more dedicated training for developers to enable them to:
- Manage BOS releases from scratch.
- Manage hotfixes for BOS.
- Troubleshooting.
- Coordinate the requested change to production with the related teams.
- Adapt their features deployment according to the release calendar.
- Clarify details regarding the release process and staging/production stuff.
Workflow
It will really depend on each team. However, we need to ensure that a responsible person is assigned to manage the release process.
For that reason, it will be required to configure a rotation system for teams to manage releases in BOS. Initially, this assignment process will be manual. However, it might be possible to think about an automatic assignment solution if the manual process is not working well.
We have prepared a spreadsheet to facilitate the rotations: BOS Releases - Coordination
Responsibilities
It will require the team leaders to choose, at least, two team members per team to be involved in this pair-working/training process and the full management of the release process in the future. The attached spreadsheet must be visited by the team leads on a regular basis to make sure everyone is aware of the people in charge of the release process during the current week.
All engineers working with anything related with BOS must understand at least how to perform a release in the service they are putting code into. Senior engineers and above must be able to actually perform the entire release process in BOS.
Alternatives
Create a release team for the BOS service, not necessarily be developers, just a group of people in charge of it.
Caveats
From the Security team perspective there is no regulatory requirement/control that would prevent a developer managing their own releases. Adopting industry best practices such as a requirements checklist, acceptance testing criteria or oversight is key.
We need to ensure we have the required reviewers within the PR that allows us to put in place a best practice but balanced approach to keep things fluid, avoid unnecessary delay and with minimal user impact.
Operation
During the first two months after this policy is approved, the Developer Experience team, specifically Daniel Gordillo, will work in pairs together with developers as more practical training sessions so that when reaching the end of this phase, the knowledge should have been spread to all developers.
This should be the way to go after the estimated end date, developers will manage the releases and will continue spreading the knowledge to new joiners so that there are always responsible engineers to perform a release in BOS looking for a more self-service model as in any other service in Ebury.
Security Impact
Developers will require access to the tools used to deploy code in BOS:
- Jenkins: It requires access to jobs that deploy code in PROD (just for the BOS service) in the different BOS PROD environments.
- SECO: As of now, access to manage releases BUT it might require some permission in JIRA to create/edit “Releases” in case to amend some data or JIRA ticket.
- Bitbucket: Include the people involved in the “Able to merge” group to allow update the “dev” branch through commit when a hotfix is merged to the “master” branch.
Performance Impact
N/A
Developer Impact
The performance of the developer in charge of managing the release might be affected. In order to properly track the effort required to manage the deployments during the related week, we strongly recommend creating a Jira ticket based on the “Release” type. This ticket will be used to link the tasks related to each release that has been coordinated by the assigned developer.
Data Consumer Impact
N/A
Deployment
The rollout of this rotation policy will take place in two different phases:
- One month of shadowing training together with Daniel Gordillo
- Two months of teams taking care of the entire release process in BOS, supervised by Daniel Gordillo and the already mentioned experts in case of Dani’s absence.
Dependencies
The DXP team will be available to provide support to engineers in case anything is going wrong regarding the release process. They can be notified via Slack through #dxp or #bos-release-management