Measuring Cycle Time

Problem Description

We want to introduce a new metric that allows us to measure the delivery time of a team so that features can be validated in production as soon as possible. Additionally, improving these times will have a positive impact on incident resolution time.

Background

The metric of productivity we have been using until now is based on the number of tickets delivered by a team in a sprint. This can measure the capacity of a team to break down the issues in manageable and small tasks in order to add value as soon as possible. However, this metric could be biased if a team is creating too many tasks for the same amount of time compared to the others that create less. You can always create more tasks in order to achieve a target without taking into account the value delivered by them. The idea behind this number of items per FTE metric was to reduce the risk of delivering large amounts of work at once or having it remain in progress. We have reached a level of maturity where we need to monitor more closely the process of developing, reviewing and releasing tasks.

Solution

We propose to measure the Cycle Time: the time it takes to bring a code change from a developer’s laptop through to production. The faster your cycle time, the quicker you can introduce new features to your end-users.

The Cycle Time we want to measure takes into account the development, review and delivery phases of the software development process. Ultimately, the goal is to quantify and understand the speed at which an engineering team can deliver working software. The faster your cycle time, the quicker you can introduce new features to your end users.

The way to aggregate this value will be to take into account the 90th percentile of all cycle times of the tasks delivered to the end-user.

How are we going to track Cycle Time?

We need to record when a work has started on a task and when it was completed. This means we can use Jira to track it. We can calculate the time passed from the moment a task is moved to "In Progress” to the time it is released. We want to focus on items that deliver direct value to the customers. To calculate the exact time when the issue is released in production we will look at the release date of the fix version associated with the issue. Cycle time will be the difference of those two dates in calendar time.

Clarifications on the measure

• Are we going to take into account calendar days or only working days? It is calculated as a date difference, so all the days in between are taken into account.

• What happens if a ticket goes directly from Backlog/To Do to Done without going through In Progress? The last transition before In Progress will be taken as the start date for development. That is to say when it was moved to To Do or when it was created in the case of Backlog. For this reason we must be precise in reflecting the current status of an issue in Jira.

• Are all resolutions taken into account? Tickets that are resolved as Won't Do or Won't Fix will not count for the calculation.

• What about those who do not have fixVersion assigned to them? We will only count tickets that are associated to a version with a Release date. For example one-off actions or spikes will not be counted.

• If an issue starts in progress but later returns to the Backlog. When it goes back into progress what is the cycle start time of that issue? It starts counting from the first move to In Progress, as the time spent on solving it started on that day.

Alternatives

Lead time is a similar performance indicator which takes into account the whole life of the issue. The problem with this metric is that the analysis time has much more impact than the execution and deployment time, so it is very difficult to improve times with that part alone. We consider this metric good for Bugs and Postmortem Action Points as we want them to be resolved as soon as possible given their priority. In addition, they should be previously analysed once they have been created.

Commit Cycle time is another way of measuring the cycle by taking into account the date of the first commit associated with that task. Like relying on Jira transitions, it has its advantages and disadvantages that we can mitigate by changing the way we work. As it only starts to count when the first commit is made, even if work starts earlier, it can cause bad behavior by trying to delay that time as long as possible.

Pull Request Cycle time measures the time elapsed between PR created and when it's merged. Code Review is only one part of the cycle. The chosen metric encompasses more things we can improve.

Caveats

N/A

Operation

N/A

Security Impact

N/A

Performance Impact

The new proposed metric should help the team to improve both review and release time and frequency.

Developer Impact

The developer should only be accurate in the issue status in Jira, as he has been doing so far.

Data Consumer Impact

N/A

Deployment

N/A

Dependencies

N/A

References

Accelerate by Jez Humble, Gene Kim, Nicole Forsgren (IT Revolution Press, March 2018)

DORA State of DevOps report (DORA Google, 2019)