Evidence of Requirements Implementation (ERI)
This proposal replaces the current UAT process with Evidence of Requirements Implementation (ERI) and provides a simplified, compliant, well-defined and developer friendly process for our software projects.
Problem Description
The current UAT process is not well-defined and we are making full-bureaucracy with video the common denominator. This hurts development and is pushing engineers towards bad practices like adding unrelated changes in the same branch to avoid the amount of paperwork that's required even for simple small technical changes (or even typos).
Effects of the current UAT process on our engineering practices:
- Adding unrelated changes to feature branches to avoid the extra bureaucracy required to split things properly.
- Sometimes the "unrelated change" is a fix to something blocking dev. We never want to wait for the feature branch approval, linted, CI'd, etc, to unblock dev.
- If a feature breaks something we may need to git revert. Having other changes makes that more difficult, which is exactly what you don't want when fighting fires.
- Ignoring simple changes and improvements because of the process burden involved with it.
- Extremely big branches that try to fit-all into a single review to avoid managing more tickets and therefore having to do more UATs.
We are also calling UAT something that's not really User Acceptance Testing. So as part of this proposal is to rename this process into Evidence of Requirements Implementation (ERI) to reduce confusion.
Background
Regulatory-wise Ebury is like a bank, and as such the company must meet regulatory audit requirements. The requirement to comply in question for branches at the moment is: leads must approve all changes prior merge.
The current way to convey approval is via email, and must include a UAT video for the change. The template goes something like this:
Hi <bos lead>[, <product people interested if any>],
We need your approval to release this code:
Description: <copypaste from the ticket>
JIRA: <link to JIRA ticket>
UAT: <link to drive link>
Do you agree?
In this, we are conflating two things: product and leads approval.
If the change is purely technical or even a preliminary internal middle-step prior to the product visible implementation (because the feature got split into smaller chunks for easier review) this adds absolutely no value and makes developers waste time on finding creative ways to fit the process.
Also, in this process we are not considering the code-owners. Team leads will still need to approve the changes at release time but for code reviews we don't think the leads should be the bottleneck since the code-owners often can do the job better and reply faster.
Solution
Split between user affecting changes and internal ones and require a different flow for each.
Also leave leads out of the approval for purely technical changes, leads will still need to approve releases but for technical changes make the code-owners the authoritative entity for ERI approval in the Pull Request itself by just approving the PR. This approval is enough for auditing purposes as Bitbucket is storing time and date of approval and because our code review guidelines include taking into account requirements as part of the review criteria.
For user affecting changes, the requester of the story should approve the change with a similar email like we had up to this point.
User affecting changes definition
Any change that's visibly affecting the behavior of the UI/UX -- this means that for example frontend refactors/cleanups that don't affect behavior don't count as user affecting changes as they are transparent to the user.
Changes requested explicitly by product (or an internal requester), even if they could be potentially transparent to end users (e.g. performance improvements). In this case the requester must receive some clear proof that their expectations met so they can allow the ticket to move forward.
For branches with user affecting changes that require approval from product
Send an email with an attachment that's a clear proof for the task. We can't enforce people from product to use Bitbucket so email is a good common denominator, we can probably simplify it though to not include the description or anything that's already listed on the ticket:
Hi [<product people interested>, ]
We need your approval to release this code:
JIRA: <link to JIRA ticket>
ERI: <link to drive link>
Do you agree?
What kind of attachment should you send as the ERI? The one that makes the recipient of the change convinced enough, so use your best judgement. Here are some recommendation for common cases:
| format | use case |
|---|---|
| Video | UI/UX flow visible changes |
| Screenshot(s) | Showing before and after for a simple cosmetic change |
| Google Doc/Spreadsheet/logs | Performance improvements, spike, metrics |
| RFC | For a change or a new proposed RFC |
| Pull Request link | When the change is purely technical and the diff is clear enough |
For purely internal / technical changes
Code-owners would just state approval in the Pull Request.
Developers may at their best judgement add screenshots/videos and include them in the description of the Pull Request.
The tests, documentation and notes should be enough proof so that leads consider the change approval.
Alternatives
- Keep it as is, get people fed up and leaving.
- Force product people to approve changes at Bitbucket Pull Requests. This in principle sounds tentative but the approval may come from a lot of different departments and we probably don't want to tightly couple the PRs because of the credentials and the difficulties these people may find using Bitbucket.
- Force approval statements on the JIRA tickets themselves. This somewhat shares the caveat with the approach above that we may not want to require JIRA access to the product approval. The other caveat is that for technical leads, the PR approval approach simplifies things.
Caveats
None, as long as we confirm we are fulfilling the auditing requirements.
Operation
Code-owners will require access to Bitbucket in order to stamp the Pull Requests but this is currently the case.
Security Impact
It improves auditing because we can automatically query when code-owners approved a given Pull Request via Bitbucket APIs.
Performance Impact
This directly impacts developer performance for the better, see notes in the next section.
Developer Impact
- Developer Performance will improve as the amount of overwork for changes reduces to just the Pull Request.
- Code-owners can more effectively use the Bitbucket dashboard to follow their required approvals than emails which are currently pretty busy with notifications and a myriad of communications.
- Leads can focus on other stuff more important than rubber stamping and avoid becoming a bottleneck for developers.
- This change helps towards the end goal of smaller focused branches which improve the quality of the reviews and the response times in case of incidents.
- Technical leads involved in the normal review process can now streamline the approval in a single step.
- Newcomers to the platform will feel less friction at the time of submitting their first changes.
Data Consumer Impact
N/A
Deployment
Once agreed we must communicate the new flow to developers and leads and we can start going this way.
Dependencies
None