Bitbucket to GitHub migration
Migration plan to move our repository workspace from Bitbucket to GitHub
Problem Description
Considering the number of incidents related to the Bitbucket service unavailability, we consider the migration of our repositories workspace to the GitHub service.
This unavailability affects directly to the availability in our services, in the daily developers work (because cause failures in the CI pipeline) and also in the deployments (releases can't be done)
Background
Bitbucket is used for a long time in Ebury but the business plan was updated one year ago when we started with the Code-Ownership initiative where we want to include some verifications in our SDLC like don't merge PR without code-owner approval, a minimal of approvals, reset approvals when we do a commit in a feature branch, ...
Solution
We want to migrate all our repositories from Bitbucket to GitHub for several reasons:
- Availability. The number of incidents is a lot compared with GitHub apart from the time resolution.
-
Mar-May Bitbucket GitHub Difference Nº Incidents 23 8 15 Incident time (h) 90.7 11.03 79.7
-
- Code review experience. For the developers and code owners GitHub offers better features.
- Auditing. GitHub offers an auditory tool for Enterprise business plan.
Alternatives
Other alternative analysed was GitLab but there you can see the analysis: - Pros: - Best DevOps experience including CI/CD flexible as Jenkins - Optional work-flow for Kubernetes - Cons: - Migration from JIRA, Bitbucket, Jenkins not viable - Unestimated effort to migrate but so high - High cost (Premium business plan expensive than GitHub Enterprise plan)
Caveats
The migration process requires extra process to sync the meta-data in the repositories (branch restrictions, repository access, ...) that aren't cloned using the Git mirror functionality but it's going to migrated using request to the Bitbucket API to obtain this data.
After the migration, we'll change the direction of synchronization from GitHub to Bitbucket but to be used as rollback during some time (as contingency plan and avoid any disruption to the developers in case anything goes wrong). If we want to use Bitbucket as contingency permanently this is a separate project and out of scope of this one.
Code Owners in GitHub will be covered in a different epic to make sure everything is translated into GitHub properly. However, for this specific migration, we will ensure the default reviewers to be updated in order to have the same functionality as we had in Bitbucket as a starting point.
Operation
The migration is organized by the PSO team but the implementation of this RFC is mainly delegated on the DXP team (probably documentation references and CODEOWNERS files update could be done by teams or related Code-Owners)
Security Impact
The migration is going to improve the security in our repositories accesses because it's going to be mandatory set a 2FA in our GitHub account and the integration with other services is going to be done using SSH keys (user and password isn't allowed if the accounts have set 2FA)
Apart from that, the current Bitbucket team structure, branch restrictions, repositories access is going to migrated to maintain the current configuration/restrictions.
Performance Impact
The performance not should be affected directly, indirectly is affecting considering that the GitHub availability is better so it's going to create less blocks in our daily work.
Developer Impact
It's part of this Epic give training to the developers about how to use GitHub and the main commands/actions used in their daily work. Apart from that, this migration should have zero impact for the development team.
Data Consumer Impact
NA
Deployment
The plan to execute the mentioned migration is the next:
[Phase 0] Analysis
During this phase we'll work on the analysis of repositories and dependencies that are required to be migrated to GitHub. This phase will be used to perform PoC of the migration and the synchronisation method that will be used.
- Create sandbox workspace to don't use real Ebury workspace.
- Test project be used as PoC of the migration (to avoid security issues)
- Detect third parties services connected to Bitbucket
- Define migration plan for the different scenarios in the repositories:
- Non-Bitbucket references
- [Read] Bitbucket references in documentation
- [Read] Bitbucket references with URL's (to clone the repo)
- [Write] Bitbucket references to users, API endpoints, others.
- Snippets
[Phase 1] Repositories migration and synchronization
In this phase we'll investigate how to synchronise the repositories between Bitbucket and GitHub and will migrate all the repositories between Bitbucket and GitHub.
- Create five different Jenkins pipelines in a new repository:
- One to manage the project creation, this step is required before the rest to avoid issues.
- Another one, to maintain synchronised all repositories from Bitbucket to Github every certain time (configured with as cron task)
- One more related to sync but based on webhooks, to sync specific projects everytime that those receive a commit change
- One to manage the creation of meta-data for each repository where we're gonna use the Bitbucket API to obtain this data and later, create terraform files for each repository with this info (that will be uploaded to another repository)
- The snippets require a different way to migrate them because in GitHub doesn't exist snippets so we'll create a repository to include all of them.
- Create checks to confirm the migrations:
- We'll check the last hash between repositories
- Create Jenkins job to obtain references between repositories to know in what order should be moved.
[Phase 2] Adapt Bitbucket dependencies and workflows in GitHub
During this phase we'll adapt the new migrated repositories in GitHub to remove the dependencies they have with Bitbucket, basically, the scenarios related to "Read" Bitbucket references.
Also, we'll adapt all the workflows that are attached to Bitbucket into GitHub. In other words, we need to modify the processes and pipelines that are linked to Bitbucket in order to make them work in GitHub at the same time that still working in Bitbucket.
It requires analyse the different scenarios one by one to check how to proceed (only 10-12 repositories affected)
Apart from that, we'll adapt all the third parties required at this point like JIRA, Jenkins, ...
[Phase 3] Onboard a Pilot team
In this phase we'll assign a Pilot team to migrate their daily work to GitHub. A requirement for the pilot team in this phase will be to provide them access to GitHub and revoke the access in Bitbucket accordingly.
We'll have prepared the proper pipelines to synchronize this repository from GitHub to Bitbucket in case of rollback.
[Phase 4] Execute the migration plan to GitHub
This is the final phase where we'll switch the workload from Bitbucket to GitHub. In other words, the entire tech department will start working using GitHub.
Dependencies
We don't have direct dependencies but we need to implement the solution taking into account that both source-code repository hosting should work at the same during some period of time.