Aquasec Supply Chain at Ebury

This is a proposal to deploy Aquasec into Ebury’s SDLC applications and services to implement security on every step of the SDLC.

The goal of this RFC is to fulfil the requirements documented in the SSDLC RFC previously published on the Blueprints repository.

Prerequisites

The prerequisites to deploy Aquasec are the following:

  • An Aquasec active licence with the Supply chain module enabled
  • A GitHub enterprise organisation

Reference Documents

Reference Document Location
Jira Epic SECENG-378

Problem Description

Software vulnerabilities exist extensively in the applications we use and develop on a daily basis. A vulnerability can affect the confidentiality, integrity and availability (CIA) of the application and its data. Protecting the CIA of our applications and systems is critical for Ebury to avoid potentially significant financial and reputation damage as a result of a breach.

In the SSDLC RFC, a new approach that incorporated security on each step of the Secure Software Development Lifecycle (SSDLC) was proposed. After evaluating Snyk, Aquasec and GitHub Advanced Security, it was decided to opt for Aquasec due to the wide offer of services they proposed. Now that the licence has been purchased, it is possible to upgrade the current development life cycle to an SSDLC approach.

Background

Aquasec was purchased due to the fulfilment of all the requirements needed by Ebury to upscale to an SSDLC. These requirements were extensively defined in the SSDLC RFC, but a brief description of each component is included below:

  • IDE Integration. As an optional configuration, it was necessary to purchase a solution that offered an IDE plugin to conduct several scans on the developer’s laptops. These scans will show potential vulnerabilities and misconfigurations that will be detected on later stages, empowering developers to fix them before they are persisted on any Source Code Management (SCM) platform.

  • Static Application Security Testing (SAST). The SAST will conduct automatic source code reviews to find potential security vulnerabilities. SAST tools can also be integrated in a methodical process to detect security vulnerabilities before merging code to repositories, or even when the code has already been deployed.

  • Container Image Scanning. Container images will be scanned automatically and vulnerabilities affecting the software versions will be collected and reported.

  • Dependency Scanning. Applications make use of third party libraries and dependencies. The dependency scanner will conduct a Software Composition Analysis (SCA). This will build a Software Bill of Materials (SBOM) with all the dependencies a software uses. Aquasec will scan these versions and collect a list of vulnerabilities that affect them.

  • CI/CD Platform Integration. Integrating SSDLC solutions into a CI/CD platform may help automating the previous security validations explained above. Then, new discoveries could be seen directly on the CI platform, or directly on the security dashboard.

  • Security Dashboard. The vulnerabilities and misconfigurations detected by the solution’s scanners are required to be centralised in a single dashboard, where the information can be tracked down.

Except for the IDE integration scans, the remainder scans will report and centralise the information in the security dashboard. The information collected in the dashboard gives the opportunity to introduce them in a vulnerability management process. This process will evaluate the risks and set a due date for the vulnerability to be fixed. This process will be described in a vulnerability management policy and detailed procedures will then be socialised with the Tech team.

Solution

The current licence purchased allows the scanning of 250 repositories in total. Given that Ebury’s GitHub organisation has more than 600 code repositories, as a first step of the deployment it is necessary to choose what repositories will be included in the new SSDLC methodology. This process will be conducted gradually and only a few teams will be enrolled at the beginning of the integration (ONL, LMI, TRE and MOB/API).

Unfortunately, Terraform repositories and modules will not be part of the SSDLC due to the number of repositories that exist, which are not feasible to include due to the Aquasec licence. For these repositories, Security Engineering has created a different approach using GitHub actions and TfSec.

To deploy Aquasec, there are two main phases that must be done: SCM integration and CI/CD integration. There is an optional phase that will require developers to install a plugin on their IDEs so they can benefit from Aquasec scans before committing changes to repositories. Guidelines are included in this RFC, but the Security Engineering team will show teams how the plugin can be used and empower developers to fix security vulnerabilities.

1. SCM Integration

In terms of integrating with SCM solutions, Aquasec eases the enrollment and deployment of the previous features by simply adding small changes to the GitHub organisation. First of all, in order to work, Aquasec needs read-only access to Ebury’s GitHub organisation. Permissions can be granted to all repositories or only a list of them. Since Ebury will choose what repositories are enrolled in, it is recommended to grant access only to the repositories in scope. To choose what repositories will be scanned, Security Engineering decided to start with those repositories who have been updated in the last six months. After these repositories have been imported, other repositories that are still used but no longer maintained could be included. When the permissions have been granted, the organisation will appear in Aquasec’s dashboard.

repositories_aquasec

Figure 1 - Aqua Security Supply Chain (EU) GitHub application permissions.

ebury_sandbox_aquasec

Figure 2 - Ebury-sandbox organisation enrolled into Aquasec.

Once the read-only permissions have been granted to Aquasec, the enrollment of repositories can be initiated. The integration of a repository can be done directly from a SCM or by deploying a GitHub action. On the first phase of the Aquasec deployment, repositories will only be fetched using the SCM integration.

When a repository has been scanned, Aquasec shows valuable information in a dashboard with multiple sections related to the functionalities offered by the service.

ebury_sandbox_sample_repo_aquasec

Figure 3 - Aquasec dashboard with the list of repositories that have been scanned.

ebury_sandbox_overview

Figure 4 - Aquasec dashboard reporting multiple vulnerabilities affecting the selected repository.

Applying the changes to GitHub required to start conducting scans on source code repositories will not have any noticeable change on the company applications or services, hence why this will be the first step to establish an SSDLC at Ebury.

2. CI/CD Integration

Integrating Aquasec into our current CI/CD processes is not an easy job. Although deploying Aquasec into known Jenkinsfiles should be easy, it is known that Jenkinsfiles are not standarised and teams can manage and customise their own pipelines. Also, different repositories that belong to a team might have different Jenkins files and it is difficult to centralise the Aquasec deployment in a single place. Finally, Jenkins is a risky service that runs sensitive applications and has an excessive amount of privileges so the Security Engineering team will instead use GitHub Actions to integrate with Aquasec.

To conduct this phase, there is a dependency on the SCM Integration phase. First of all, it is necessary to know the list of repositories that will be enrolled, not meeting the limitation imposed by the number of repositories that can be scanned. Once the list is compiled and the repositories are enrolled on Aquasec, the CI/CD integration can be started.

The CI/CD integration will be based on deploying a prepared GitHub action in the workflows folder in each repository that is part of the list collection. The action will be submitted to the repositories in an automated manner, making use of a script that will create a branch, add the action and submit the changes in the form of a Pull Request (PR). In this way, it is ensured the changes follow the normal Ebury process, being reviewed before being merged to master branches.

An example of the action working can be seen below:

ci_cd_aquasec

Figure 5 - Aquasec CI/CD action running on a test repository.

The CI/CD integration will perform all the scans conducted when the repository is enrolled into Aquasec, including secret scanning, vulnerabilities, version scanning, SCA and SBOM. However, the integration will work depending on how the GitHub action has been configured. For instance, it can run when a status change is made on a branch, on PRs or while pushing to master.

No noticeable changes will happen by integrating into the CI/CD processes using GitHub actions, apart from requiring a PR to be approved and merged to the master branch of the targeted repositories. Nevertheless, for the action to be able to submit the results to Aquasec, it will be necessary to create two secrets within the GitHub organisation.

3. (Optional) IDE Installation

Aquasec offers a plugin for multiple IDEs. Developers can decide whether they want to install these plugins in their IDEs and benefit from the improvements it can bring to the development phases. Installing these plugins will benefit developers on earlier stages, reducing the amount of effort required to fix certain security vulnerabilities or misconfigurations. Nevertheless, the plugin needs to be run by the developers and it is necessary for them to be keen to find these issues before committing changes to source code repositories.

The reason for the plugins to be optional is because the same security vulnerabilities and misconfigurations can be detected on later stages of the SSDLC (i.e.: when the source code is pushed to GitHub).

Guidelines about how to install and use the plugin can be found on the official documentation.

Terraform Repositories and Modules

Due to the elevated number of IaC repositories and the limitation on the number of repositories, it is not feasible to scan all of them. A potential solution to this issue would be to promote mono-repository structure, but this is out of the scope of this RFC. Hence, the Security Engineering team has implemented a custom process to integrate IaC scanning on our repositories.

Aquasec uses TfSec as its IaC scanner. TfSec has an open source version that can be integrated into the CI/CD process using GitHub actions. The action will be deployed on all the repositories with Terraform code and will run on monitor mode without failing the process. The results of TfSec will be part of the GitHub status checks and the findings can be seen there. On a later stage, the GitHub action will be modified to block code merges when certain misconfigurations are detected, requiring teams to fix them before deploying them to the cloud environments.

The only problem of this approach is the lack of centralisation in a security dashboard. To overcome this problem, a custom parser can be created to grab the information using the GitHub API. Then, the results will be imported on a dashboard deployed and maintained by the Risk & Security team. The ultimate goal of a security dashboard is to help manage vulnerabilities and misconfigurations, addressing them in a timely manner.

The script to deploy GitHub actions automatically will be accessible in the seceng repository. The GitHub action that will be used can be seen here

Assurance Policies

Aquasec has the ability to monitor or make PRs and build processes fail, depending on the requirements established by the assurance policies. These policies are fully customisable, but Aquasec provides a built-in list of policies to assist on its deployment.

At the beginning, a monitor only approach will be used. In the future, when the deployment is more mature, it is recommended to block code merges in certain situations. The reason to make processes fail and require changes is because otherwise, security will not be improved, becoming an optional action that can be circumvented, putting Ebury and its assets at risk.

The Security Engineering team will work on the assurance policies that should be implemented, but it is agreed that other teams will be included in the discussion to be aligned and find the best policies for Ebury.

An example assurance policy that will send audit messages when severities scored as high or critical are detected. The action can be duplicated or modified to fail PRs and builds, should the processes not be compliant with the requirements established in it.

audit_policy_aquasec

Figure 6 - Aquasec assurance policies options.

audit_policy_aquasec_2

Figure 7 - Aquasec assurance policies built-in controls.

Success Criteria

The success criteria of deploying Aquasec will depend on three different factors:

  • The main repositories have been enrolled in Aquasec or the limit of 250 repositories has been reached;
  • The main repositories have been integrated in the CI/CD process on GitHub
  • Assurance policies have been agreed with technology teams and deployed on Aquasec.

Alternatives

After purchasing Aquasec, there is no alternative apart from continuing with a common, and non-secure SDLC approach. That will mean vulnerabilities will be detected in the latest stages of the development cycle, after the applications have been released to production. An increase in the complexity of resolving new and existing vulnerabilities will be produced. Furthermore, should these vulnerabilities be exploited by attackers, data breaches may happen, impacting the entire company.

Caveats

Due to the number of repositories that can be integrated into Aquasec, it is not possible to scan all the repositories that exist in the organisation. Thus, only selected repositories will be included into Aquasec, having a big number of repositories left behind with undetected misconfigurations and vulnerabilities.

Furthermore, in the last couple of months, Ebury’s GitHub organisation repositories have increased dramatically. Should teams continue creating new repositories instead of following a more centralised-like repository or mono-repository approach, we will not be able to successfully deploy an SSDLC completely. At the moment there is no corrective action planned, but subsequent guidance on this will be provided.

Operation

The Security Engineering team will take ownership of the SSDLC solution. The Risk & Security department will conduct the detection, reporting, notification and tracking of vulnerabilities, communicating teams when a risk should be remediated. However, the SSDLC solution aims to help and empower developers to fix their own code, preventing vulnerabilities from reaching production environment. Thus, SECENG may help teams, but will not directly fix the vulnerabilities unless it is necessary.

Security Impact

Security will be improved due to the enhancement of access controls which will limit what is exposed to each user. By adopting this proposal and incorporating robust access control mechanisms, Ebury can significantly enhance its overall security posture and protect valuable products. Several advantages are listed below:

  • Enhanced protection of intellectual property: Proper access control measures help safeguard sensitive data and intellectual property, reducing the risk of data leaks, theft, or misuse.

  • Mitigation of insider threats: Implementing adequate access restrictions can prevent employees with malicious intent from accessing, manipulating, or deleting sensitive data, source code, or configurations, thereby protecting the organisation from potential harm.

  • Reduced attack surface: Adhering to the principle of least privilege and granting only necessary privileges to users can minimise the attack surface, making it more challenging to exploit vulnerabilities and access critical resources.

  • Improved monitoring and auditing capabilities: Implementing robust access control measures enables better tracking of user activities, facilitating the detection of suspicious behaviour and timely response to potential security incidents.

  • Compliance with legal and regulatory requirements: Protecting intellectual property and sensitive data by implementing access control best practices helps ensure compliance with industry regulations, avoiding financial penalties and reputational damage.

Performance Impact

Performance should not be impacted due to deploying Aquasec.

Developer Impact

Developers will not be impacted by Aquasec as long as the assurance policies are agreed between Security and Technology.

Regarding the IDE plugins, since it will be an optional feature, developers should decide to opt-in, installing the plugin and running it. The Security Engineering team will be keen to help should they require any assistance during the process.

Data Contracts

Aquasec’s implementation will not impact any data sources.

Deployment

The deployment of the new GitHub structure will follow the action plan proposed. The deployment of Aquasec will be conducted by the Security Engineering team. The approach will be divided in three main phases, considering the repository limit we have in our licensing agreement. The phases are the following:

  1. SCM integration
    • Integrate Ebury’s GitHub organisation into Aquasec
    • Select up to date repositories, below the 250 repositories limit
    • Terraform repositories will be discarded
  2. CI/CD integration
    • Deployment of GitHub Action on the selected repositories
  3. Assurance policy creation
    • Define assurance policies and set them on monitor mode
    • Over time, agree with Technology teams and decide which assurance policies should block PRs and build processes.

Due to the limit of 250 repositories, Terraform repositories and modules will not be incorporated into Aquasec. For these repositories, Security Engineering has created a different approach using GitHub actions and TfSec.

Dependencies

ONL, TRE, LMI and API teams will be part of the initial limited rollout. There are no technical dependencies that affect Aquasec’s deployment.

References

  • SSDLC RFC: https://github.com/Ebury/ebury-blueprints/blob/master/docs/processes/ssdlc-process.md
  • Aquasec Supply Chain: https://www.aquasec.com/products/software-supply-chain-security/
  • Trivy IDE plugin documentation: https://aquasecurity.github.io/trivy/v0.40/ecosystem/ide/
  • GitHub Actions: https://github.com/features/actions