Code Ownership

This document provides guidelines regarding the concept of Code Ownership in Ebury.

Problem Description

Clearly defined processes relating to a codebase (contributions, releases, documentation, bugfixes, etc.) are essential in developer communities. Not only to have smooth workflows, but also to ensure that the number of contributors could safely scale up, as the project is growing.

Furthermore it is important to have clear responsibles on a project level with a clear vision both on the project itself, and how it evolves within the bigger, global vision as much in short, as in a long term.

Though Ebury had guidelines for various Development Processes, the interpretation was much dependent on team culture, thus inconsistent across the company.

Furthermore there was no appointed group of responsibles on a project level, which was particularly problematic for big projects, in terms of maintaining coherence, consistency across the project, and to head towards a unified vision.

Areas that weren't covered by company guidelines (like: Update Policy) were fully left for the teams to decide -- occasionally leaving heavy backlogs.

All in all, there was a need for better supervision, guidance and control over the Ebury development processes.

Background

Inner Source (or Inner Sourcing) is a term coined by Tim O’Reilly in 2001, referring to the “use of Open Source techniques within the corporation”. The idea is to take the lessons learned from developing Open Source software, and apply them on the company's internal software development processes.

Ebury code contributions generally work along the guidelines of Open Source projects with open collaboration.

Open Source software development clearly recognizes the value in the ownership, that main contributors have a tendency to grow over their projects. According to Daniel H. Pink (Drive book), autonomy, mastery, and purpose are the three true elements that motivate us, and the companies must promote intrinsic motivation to mirror the impact of open software development in their employees.

Martin Fowler classifies code ownership in three categories: strong, weak and collective code ownership. Out of which, generally the latter ones are commonly used in Open Source projects.

The Ebury Code Owners policies are based on these grounds, best suited to the Ebury Inner Sourcing company culture.

Solution

Our solution relies on developers' motivation to work on particular technologies and business solutions, by encouraging to take ownership of codebases aligned with their individual or collective interests.

The concept of Code Ownership was introduced in Ebury, clearly defining the roles and responsibilities within the company-wide project-level workflows.

Definition

Code Ownership in Ebury is within the guidelines of Martin Fowler's definition of "weak Code Ownership" combined with "collective Code Ownership".

Practically, there is a well-defined set of Code Owners per codebase, including but not limited to the number of Contributors. However, the agreement of one or more Code Owners is required for changes on the codebase.

A Code Owner is someone with a solid expertise, and a feel of responsibility and commitment for the application code. Holder of a generic, deep knowledge about the project, often someone who has been closely following its evolution for a while. Contributors who are best placed to guide the direction as the project is developed and maintained.

Code Ownership is not based on the volume of contributions, and should not be seen as a reward for sustained contributions.

Large projects may have multiple layers of Code Owners, such as

  • General Code Owner
  • Scoped Code Owner.

For smaller projects Code Owners combine the responsibility of both a General a Scoped Code Owner.

General Code Owner

General Code owners typically have

  • a high-level, global view of the project combined with in-depth understanding
  • a vision on how the project integrates in current and future company architecture
  • a good knowledge on solutions implemented within the project
  • a good knowledge about related projects, interactions and interfaces

Furthermore General Code Owners may have

  • a particularly strong business knowledge
  • particularly strong standards and skills relating to code quality
  • other global, application-level concerns (DB interactions, etc.)

Scoped Code Owner

Scoped Code owners typically have

  • a particularly strong business knowledge in the given scope
  • strong understanding and concerns of the given scope (service-level, module-level, etc.)

Responsibilities

All code owners have the following responsibilities:

  • actively participating in shaping the vision for the future version of the application (codebase)
  • being involved in the technical decisions related to their scopes
    • technical changes
    • future architectural changes
    • ongoing/upcoming projects
    • etc.
  • review, approve or reject contributions upholding architectural integrity
    • detecting major issues (design issues, inconsistencies across the project, etc.)
  • participate in shaping the Code Ownership Process

General Code Owner

Furthermore, a Global Code Owner should

  • track changes in the project from a high-level perspective
  • prevent/block attempts against global vision and goals
  • act as specific code owner for modules without owners
  • keep global documentation (codebase description, etc.) up-to-date
  • Code Reviews insights
    • the change should integrate with the global vision
    • should be consistent with the rest of the codebase
    • reuse already-existing solutions whenever possible
    • using application-level internal standards
    • etc.

Scoped Code Owner

A Scoped Code Owner may be responsible for a subset of a huge project, or even just a single module. In any case, his/her responsibilities include:

  • vision corresponding to the scope (service-level, etc.)
    • including cross-application or cross-team efforts
  • keeping track of ongoing major changes within the scope
  • identifying and prioritizing technical debt
    • including outdated dependencies
    • issues signaled by the Security Team
    • transparent internal changes (major refactors, etc.)
    • etc.
  • ensure that scope-level documentation is added and maintained by the Contributors
    • typically Developer's Notes or a local README is added alongside new, non-trivial solutions
    • code comments for small but complex solutions
    • Confluence Docs when convenient
  • participation in incident post-mortem activities relating to their codebase
    • ideally: participate the post-mortem meeting directly
    • alternative: urgent follow-up on post-mortem decisions before unfavorable actions may advance
  • establishing a well-defined release process for the codebase

NOT the responsibility of Code Owners

After having come across a couple of misunderstandings and misconceptions, we found that it's also important to highlight functionalities, that are NOT the responsibilities of Code Owners.

Most of these concerns are related to Code Reviews, namely:

  • Since the Code Owners are the "final barrier"-s of the Code Review process...
    • ...therefore they should NOT review code contributions that have not yet been approved on a team level (i.e. by min. 50% of the team members [1]). The Code Ownership process is not to replace team-level verification processes, but to provide an additional layer on top of those.
  • Since the Code Owners are responsible to maintain the integrity of the project...
    • ...therefore they are NOT responsible to deep-scan code contributions for bugs or hidden mistakes, that were supposed to be filtered out by the developers team
  • Since the Code Owners are holders of generic knowledge...
    • ...therefore they are NOT the primary contacts for an outage.
    • Clearly Code Owners are more than happy to help in critical situations, and may be particularly helpful when generic or service-level knowledge is needed. However outages rather require Developers who have been recently working on the code, Release Managers, etc.. Code Owners however should be involved in decisions for the long-term resolution of the issue (typically: postmortems)
  • Since Code Owners are only responsible to establish a Release Process for their codebase...
    • ...therefore they should NOT participate in the individual code releases

Responsibilities of involved parties

In order to have the Code Ownership processes well established, it also requires the rest of the company being aware of their involvement and the processes where their participation is advised or required.

It is important that Code Owners would get informed, get involved in decision-making whenever any impact towards their application (code) is foreseen.

Please note that Code Owners have to represent ownership over the codebase. They would stand against any changes that may not be beneficial for the application (codebase). It's better to shape your plans alongside with their recommendations and advises, instead of having them reject or request to alternate fully worked-out plans.

Planning

Code Owners should be consulted in early stages of changes relating to their applications (codebases). This means that they should be rather contacted at the time of planning a change -- instead of letting them know about it as a fully implemented solution. This way: - it's more than likely to get a (quick) approval - no time is "wasted" on a solution that Code Owners may not agree to

We already have well-established channels for communication with Code Owners: RFCs and project-level Slack channels.

However we provide a few more alternatives/hints, in case any of these may seem better suited in a particular use case.

Major changes

Clearly Code Owners should be made aware of major changes impacting their application (codebase), and involved in related decision-making.

Mid-size changes

In case of small projects (typically a small epic or a group of related changes) it's worth to contact the Code Owner, asking his/her opinion on the early plans. The feedback received may help to avoid delays on the project, going for the right solution from the start.

Depending on the volume of the issue and the complexity, the Code Owner may be contacted in an async written form (Slack, email, etc.), or live.

Small changes

As for individual Jira tickets, clearly it would be an overkill to contact Code Owners with each and every one of them. Not to mention that a good number of tickets either relate to a (small) project (detailed above), or would be quite straightforward to implement. There's no need to bother Code Owners with that.

However, a number of tickets require heuristics, may not be trivial or leave the Developer in hesitation across multiple implementation options. On these occasions it may be particularly advised to briefly present early ideas to a Code Owner (preferably the same one who would also be invited to approve the final implementation).

In this case (except unusual cases) the Code Owner should be contacted in a written form. The best for this purpose is to present the planned solution in the Developer Notes of the Jira ticket. Of course in case of doubt or questions, when interactivity is needed, Slack may be more convenient.

Please try to avoid that Code Owners would learn about your solution in the first time as fully implemented code.

Code Reviews

Though already mentioned before, we prefer to highlight again in this section: Developers are responsible to land their own code. It is the Developer who has to ensure that the required number of Code Owners would review his/her code on time.

Please acknowledge the fact that Code Owners are often under a particularly high workload. They may only be able to respond or schedule a request with a delay. Therefore it's important that as a Developer you would send your requests well in advance to Code Owners -- in order to have them complete it on time. Flooding Code Owners with PRs a day before the end of the sprint is likely to result in those tasks not to be completed within the scope of the corresponding sprint.

Practical advice for Code Owners

Having learned a few lessons, we thought to share a couple of practical suggestions with (new) Code Owners here.

Distribution of the load

Given the (occasionally particularly) heavy load Code Owners may be facing, we suggest that Code Owners would coordinate among themselves in order to fairly distribute the load. A simple example could be to take the average number of Code Reviews per sprint and divide it by the number of COs. This allows for everyone to be able to better plan with his/her time, knowing the expected load. Whenever a CO is contacted with review requests above the threshold, they should kindly redirect the request to a member who has been less loaded so far.

Of course, distribution may be proportional instead of equal. Typically if a CO has additional responsibilities, only allowing for a partial participation. Or in case a subset of COs are heavily involved in other, Code Ownership-related duties (design discussions across projects, etc.)

In case of a heterogeneous set of Code Owners for a project, you may also divide the load by area of expertise. Furthermore, you may want to introduce "split" reviews, where multiple Code Owners may review the same change -- however each of them looks at a different part of it (business code, tests, etc.).

Involvement

Whenever it is in your reach, try to get involved in early stages of any changes relating to the application (code). It's more economic to block, or 're-shape' changes on a design level, instead of the final stage, when it's implemented code already.

Reflecting your work

We strongly suggest that whenever Code Ownership implies a significant workload, you would feel free to create a Jira ticket to track and reflect your work.

Typically you could permanently have Jira tickets, from added at Sprint Planning time to highlight on the workload generally expected from your Code Owner duties. For example:

  • as a BOS Generic Code Owner: 2-3 points
  • as a BOS Service-level Code Owner: 1-2 points
  • as both a BOS Generic and Service-specific Code Owner: 3-5 points

This allows for traceability of your work, furthermore it's particularly helpful for your team (Lead), understanding the proportion of time that is regularly spent on this activity per sprint.

Administrative aspects

This chapter defines formal processes relating to the Code Ownersip role.

Appointment

Code Owner candidates could be Developers, who

  1. have already had major contributions to the application (code)
  2. have in-depth knowledge about the application (code)
  3. have aligned vision about the future of the application and the repository
  4. are experienced working on the application code
  5. are core members of the set of Contributors of the codebase
  6. have at the minimum level a Senior Developer title (applies to service-level Ownership and above)

The Code Owner candidates are proposed by the Leads of the teams that work with the corresponding application codebase. The proposal has to be justified for each candidate.

Code Ownership is always an individual assignment, never collective. However, for some cases when multiple repositories fall under the same expertise and share the same structure, therefore have the same developers as Code Owners, groups can be used to reduce the burden of managing multiple similar repositories.

These groups must not be GitHub Teams at management level (i.e. not SRE, PAY, TEG, etc), but must form a specific collection of individuals selected as reviewers as a GitHub Team.

The naming convention should be *-codeowners, e.g. terraform-codeowners, graphql-codeowners, etc.

Codeowner groups, as GitHub teams, should not be sub-teams of a management level team (i.e. terraform-codeowners should not be a sub-team of SRE), but within an area (Platform, Channels, etc).

Also, not everything in our repositories is application code, we also define infrastructure as code and configuration as code. In the case of infrastructure, and specifically for Terraform, we have two types of repositories, modules specifying generic components and repositories specifying live infrastructure. While the first ones can be considered as code libraries with individuals as owners, the repositories specifying live environments can have teams as code owners, as environments are owned by teams and not individuals.

The minimum number of Code Owners per codebase should be 3 people, allowing for a good consensus mechanism to reach decisions. Generally, its preferred to have an odd number of code owners.

There must be clear policies for potential (synchronized) absences. Especially in case all Code Owners may be absent.

Confirmation

The appointment is communicated in the shape of a new Pull Request against the codebase's CODEOWNERS file, with the Code Owner added to the corresponding scope.

The pull request must explain the reasons for the choice.

The decision takes effect when the PR is merged.

When groups are used in CODEOWNERS files, it's required to get formal agreement from the candidates, the majority of the Code Owners and the team leads before changing the member list.

Replacement

There could be a number of reasons why one may decide to step down from the Code Owner's role, such as

  • Code Ownership assignment in another repository with similar or greater impact
  • internal re-organizations may move the Code Owner to focus on another application (codebase)
  • one may become a Code Owner for more repositories than his/her capacity
  • the company, represented by the VP Engineering, suggests to leave this responsibility
  • the code owner leaves the company

The successor should be chosen according to the Appointment section. Ideally both the Team Lead and the leaving Code Owner should agree on the appointed replacement.

Not strictly related to the blueprint, but we would like to mention two additional roles relating to application codebases.

This both helps to distinguish between the roles, and clarify what belongs to each scope.

Keep in mind that a person may "wear many hats". A Code Owner is also an Administrator, and of course a Developer as well.

Contributor

As the name suggests, Contributor is a Developer who contribute to the application code.

Each repository has to contain instructions for Contributors to the project. (The default policy is usually that all Developers are welcome to propose changes and contribute to the project via Pull Requests.)

Administrator

Someone who is authorized to handle permissions for the project, with an assignment to handle access level for Contributors, Code Owners or and Administrators.

All the Code Owners are also Administrators. However the Administrator role is not limited to Code Owners: Team Leads and System Administrators are often Repository Administrators as well.

Documentation

Application codebases must include the following documentation.

  • README.md: Guidelines on how to install and deploy the project, which MUST contain sections on Vision and Getting Started, and MUST include either:
  • a section on Contributing,
  • a separate CONTRIBUTING file.

The following is a guide, authors are encouraged to tailor the content of each section to their project as a required.

  1. Vision: Scope and the vision of the project
    • Helpful for Contributors, suggesting what kind of changes may be in-line with the global vision
  2. Getting Started:
    • Prerequisites
    • Installation
    • Usage e.g. How to run tests, links to documentation, whatever is helpful

  3. Contributing: Guidelines on how Contributors should add changes to the project

    • Where and how to report bugs
    • How to contribute code changes
    • How the code is reviewed
    • CI workflow (PR, UAT, etc.) phases and the quality gates

    Authors can optionally split Contributing into it's own file, taking advantage of the Github CONTRIBUTING file magic https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/setting-guidelines-for-repository-contributors

     - See [template](https://mozillascience.github.io/working-open-workshop/contributing/)
 - See [example](https://github.com/InnerSourceCommons/InnerSourcePatterns/blob/master/CONTRIBUTING.md)
    • CODEOWNERS: The list of Code Owners (and their scopes)
    • Following the GitHub format for CODEOWNERS
    • Should also indicate the main contributing teams

Implementation

Code Ownership will be gradually introduced on all (300+) repositories in Ebury. The status of the integration can be followed on this spreadsheet).

Integration steps are:

  1. Identification of the main Contributors and the responsible team per repository
  2. Identification of the most relevant repositories per team
  3. Help the team through the integration process (responsibilities, documents, etc.) for one repository
  4. The team will complete the integration process for the rest of their repositories

The integration process is prioritized by the business weight of the repository.

Within the implementation process corresponding access rights have to be adjusted following the Repository Access Policy. New Code Owners' permissions are set by current Administrators (identified here) and have to be communicated towards the impacted teams.

[1] https://fxsolutions.atlassian.net/wiki/spaces/PRODUCTS/pages/967737380/Microservices+-+Approvals#Microservices-Approvals-PRapproval