Secure Software Development Lifecycle at Ebury

This is a proposal to extend Ebury's existing Software Development Lifecycle (SDLC) to integrate security into each step.

Prerequisites

The prerequisites to transition to a Secure SDLC will be the following:

• An open mindset from all the teams to help each other and improve the current SDLC.
• Willing to adapt the current team's workflows to integrate security.
• Centralise the code repositories on one platform (Github).

Reference Documents

Reference	Document Location
Jira Epic	SECENG-77

Problem Description

Software vulnerabilities exist extensively in the applications we use and develop on a daily basis. A vulnerability can affect the confidentiality, integrity and availability (CIA) of the application and its data. Protecting the CIA of our applications and systems is critical for Ebury to avoid potentially significant financial and reputation damage resulting from a breach.

We currently adopt a reactive approach to detecting and resolving vulnerabilities, whereby we are responding to security issues once they are detected either during pre-release testing or in production by third parties. This is a manual, slow, painstaking and error-prone process that is an unnecessary risk to Ebury.

Background

In the last years, we have seen how vulnerabilities affecting well-known and extensively used software be targeted by attackers. In Ebury, the security of its applications and the data managed by them has become also an important discussion. Thus, applications that expose functionality directly to the clients as well as databases that stores the data, or the infrastructure running Ebury as a whole must be secure and reliable.

Before a software product is deployed to production, it goes through well-defined phases such as requirement gathering, design, development, verification and maintenance. A Secure SDLC aims to maintain the same phases of the SDLC but introducing security on each of these phases. Processes can be deployed to help ensuring a secure SDLC approach is followed. Nevertheless, security needs to be in the developer's minds who will implement the requirements of an application to achieve the best results. Overall, the final goal of implementing a Secure SDLC is to be able to detect security issues before deploying to production, reducing the risk of discovering security vulnerabilities on the latest stages.

Solution

We propose a change of mindset in which each phase of the SDLC must contribute to the security of Ebury. This is achieved in different ways depending on the stage of the SDLC, but there is a common factor on all of them: security should be in the team's minds.

The security solutions that could be used in the future are out of the scope of this RFC, and only high level details, requirements, recommendations and good practises will be described. A subsequent RFC will be released with technical and specific descriptions of the implementation when a solution is agreed.

Secure SDLC Phases

Figure 1 - SDLC phases.

There are many ways to describe the phases of the SDLC. No matter what stage is called, it is possible to implement security in each of them. Most of these additional security tasks can be automated which can ensure secure applications are developed and deployed with minimal disruption to developers and existing processes.

1. Requirement Gathering

In this first phase, requirement definition should consider potential vulnerabilities that could affect a certain functionality the application will implement. For instance, if a login panel needs to be implemented, security vulnerabilities affecting this features should be taken into consideration, such as login brute forcing, account lockouts, password resets and user enumeration. Thinking about potential attacks may facilitate finding solutions, such as the implementation of a CAPTCHA to block automated attacks.

Well-known methodologies like OWASP can be followed. Although there are several others, the most common vulnerabilities can be listed in the following OWASP links:

• https://owasp.org/www-project-top-ten/
• https://owasp.org/www-project-api-security/

2. Design

This phase translates the requirements defined in the previous step into a plan about how everything should look like in the application. Functional requirements focus on what should happen, while security requirements focus on what should not happen. For example, if a user logs into the application, the user should be uniquely identified, have a valid session token with an expiration time and that only resources assigned to that user are accessible.

3. Development

When the development phase is started, it all switches to ensure the code is well-written from a security point of view. To do so, secure coding guidelines will be distributed and code reviews will be done during the development lifecycle. These code reviews can be done manually or automatically using technologies such as Static Application Security Testing (SAST).

4. Verification

The verification phase is where applications are tested to ensure they meet the original requirements and follow good design principles. In combination with common testing, the verification phase is a good place to introduce a variety of security tests such as parameter fuzzing, that could detect improper error handling, inadequate parameter sanitisation and information leakage.

5. Maintenance and Evolution

After the application has been released, and even if a Secure SDLC has been followed, new vulnerabilities or previously undetected vulnerabilities may arise affecting the components involved in the application. Therefore, it is recommended to conduct external penetrations tests against released applications in an annual or biannual basis.

Secure SDLC Best Practises

The previous sections provide an overview of the new tasks that should be included in Ebury's current SDLC. However, that will not be possible unless good practises and the correct mindset are adopted. We propose the following best practises to help the teams get along with the new Secure SDLC:

1. Educate Developers

The best way to remove a vulnerability from its roots is directly not to code it. That is a really interesting goal but there are previous actions to do before. We propose the following actions:

• Creating secure coding guidelines for developers
• Providing developers with security awareness training
• Delivering application security / secure coding training
• Set expectations around how quickly issues should be remediated (remediation SLAs)
• Establish clear lines of communication to help resolve security issues

It isn't mandatory to do all of them, but the combination of them all will help developers as a whole.

2. Have Clear Requirements

In the requirement gathering phase, tasks should be easy to understand. Development teams need to clearly list what functionalities will be developed. Then, a security mindset will be applied to identify where security issues could be and plan how to mitigate them before they even exist. Vulnerabilities detected in the development phases should be easy to act on, saving a lot of time and efforts.

3. Maintain a Growth Mindset

A Secure SDLC will change how multiple teams work and interact with each other, so it is important for everyone involved to be open-minded. The most important is for the security team to have a mindset of empowering developers to secure their own applications.

4. Team Up

Multiple teams need to work together to bring the most value out from the Secure SDLC. For instance, DevOps teams will need to work together with the security team to provide the best advice and help to development teams.

5. Focus on the Big Problem First

Due to all the third-party software involved in a project, vulnerabilities will continuously be discovered. It may not be feasible to remediate all the vulnerabilities identified, so focusing on the most important issues and actionable fixes would be the best approach.

Secure SDLC Tools

While a Secure SDLC starts with changing how we think about security, there are many SDLC tools we can use to help improve and automate security. These SDLC vendors commonly offer a combination of tools in packages, according to their licensing model. These are typically divided into two sections: Supply chain and Runtime protection. This RFC will only cover supply chain tooling.

The software supply chain is anything and everything that touches an application or plays a role, in any way, in its development throughout the entire SDLC. Software supply chain security is the act of securing the components, activities, and practices involved in the creation and deployment of software. To do so, the tools covered in the following subsections aim to facilitate developers and security engineers tackle potential security issues before they reach production.

From the Security Engineering team an evaluation has been conducted of three different solutions:

• Snyk
• Aquasec Supply-chain
• Github Advanced Security

Due to pricing, Snyk was discarded, leading us to the evaluation of Aquasec Supply-chain module and Github Advanced Security. The goal of this RFC is to clearly state the mandatory requirements an SSDLC solution should have and how easy it could be adopted by Ebury. A comparison between both solutions has not been included in this RFC, although it can be seen in the Security Engineering wiki. When a decision is made, a subsequent and more specific RFC will be written explaining the solution adopted.

The following sections detail where the vendor tooling can be integrated into our SDLC ordered by where they occur in the lifecycle.

IDE Integration

Supply chain tools can be integrated into developer's IDEs to conduct scans on their code they are implementing. For instance, these scans can look for bad coding practises that could lead to security vulnerabilities.

Utilising an IDE plugin can empower developers to discover and remediate security vulnerabilities at the earliest stage of the SDLC. This helps to minimise the number of vulnerabilities that are committed into a code repository. However, to achieve that, developers should be proactive, and scan their code before pushing recent changes to their repositories.

It should be noted these tools support IDE integration, but not all the tools support every IDE. For instance, Snyk and Aquasec support integration with Visual Studio Code and Jetbrains. However, Github Advanced Security only supports integration with Visual Studio Code.

Static Application Security Testing (SAST)

A SAST can be considered a tool to automatically conduct source code reviews to find potential security vulnerabilities. SAST tools can also be integrated in a methodical process to detect security vulnerabilities before merging the code to the repository, or even when the code has already been deployed. Both scenarios will trigger a security review and vulnerability management processes to solve potential vulnerabilities.

A SAST scan takes place early in the SDLC process, after the code has been committed, commonly, to a code repository. These scans look for well-known misconfigurations or coding practises that could lead to these vulnerabilities. The process of identifying these vulnerabilities may happen by performing a data flow analysis, following how a variable is processed in a code base. Additionally, some of these scanners can be improved with custom queries to look for certain vulnerabilities. For instance, if a developer is known to repeat certain vulnerabilities, custom queries can help detecting them. In addition, these scanners can also look for secrets that could be committed to a code repository.

Since these tools are automatic scanners, they usually come with a dashboard to aggregate, visualise and process all the potential issues that have been detected. Security dashboard are really useful due to the possibility of managing vulnerabilities directly from there. For instance, it is known these solutions might report false positive security issues. False positives must be discarded and the dashboard come with functionalities to do that.

Depending on the solutions, it would be possible to run an offline scan of a project directly from the developer's IDE. This will overlap with the IDE integration discussed above. Should this be supported, vulnerabilities could be fixed before committing new changes to the repository. Nevertheless, this feature is not supported by every solution and may depend on the IDE used by the developer.

Container Image Scanning

Container images can be scanned automatically and vulnerabilities will be collected in the solution's dashboard. Similarly to the SAST solution, these vulnerabilities will require actions such as vulnerability management or evaluations to see if safer version can be used instead.

Dependency Scanning

Our applications, like most modern software, make use of third party libraries and dependencies. Libraries are regularly affected by security vulnerabilities, and it is important to identify and resolve these. While the direct dependencies may be obvious in an application, all the transitive dependencies that each external library use are not. This creates a complex list of direct and transitive dependence that each application relies upon, which makes detecting and resolving vulnerabilities in dependencies a challenging problem.

To aid managing and resolving dependency vulnerabilities, supply chain solutions conduct Software Composition Analysis (SCA). SCA tools inspect package managers, manifest files, source code, container images, and more. The identified libraries are compiled into a Software Bill of Materials (SBOM), which are then compared against a variety of databases, including the National Vulnerability Database (NVD), to map each library with any known vulnerabilities affecting it.

CI/CD Platform Integration

Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project. It’s a primary DevOps best practice, allowing developers to frequently merge code changes into a central repository where builds and tests then run automatically.

At Ebury, Jenkins is the main CI platform used. Most vendor solution can be integrated with Jenkins, via Jenkins plugins, but not every solution will support this feature. These plugins can be installed with administrative access and quickly included in existing pipelines by just adding some new steps to them. In case of wanting to use a different CI utility, these tools support integrations with Github Actions, CircleCI, TeamCity or Gitlab CI, among others.

Integrating SSDLC solutions into a CI platform may help automating previous security validations such as container image scanning and/or dependency scanning. Then, new discoveries could be seen directly on the CI platform in the form of an HTML report, or directly on the security dashboard SSDLC solutions offer. Similarly to previous sections, these vulnerabilities should be analysed and resolved via vulnerability management processes.

CI integrations offer the possibility to monitor or block detected vulnerabilities, depending on the risk of a security issue. The security engineering team is against setting blockers to developers and therefore, we encourage using the monitor mode. Furthermore, considering the previous steps discussed, vulnerabilities should be remediated before reaching the CI platform.

Security Dashboard

The vulnerabilities detected using the tools describe above will be aggregated and displayed directly in a security dashboard provided by the solution. From that dashboard, actions could be applied to help conducting vulnerability management actions. Therefore, vulnerabilities may offer the option to be processed in several ways. For instance, if a vulnerability is known to be a false positive, the dashboard will allow to discard the issue once it has been checked.

In the case of dependency scanning and to better understand how libraries are linked, security dashboards may offer graphs or tree views to visualise how third-party modules are linked with each others. However, this possibility is not supported by all the vendors evaluated.

Success Criteria

Three vendors have been assessed to find the solution that fits the best at Ebury, commercially and technically. These solutions were required to have the following features:

• Static Application Security Testing (SAST)
• Software Composition Analysis (SCA) / Software Bill of Material (SBOM)
• Container image scanning
• Integration with CI solutions
• IDE Integration
• Security Dashboard
• Programmatic (API) access for future automations

Security vendors commonly use licensing models based on modules, bundling multiple of these tools into a single package. It may be that not all the features we require are not in a single module so we may need to purchase multiple modules to satisfy the requirements above.

Alternatives

Even though it is totally recommended to invest in an SSDLC solution, some of the previous tasks can be done using open-source projects. For instance, detecting container image vulnerabilities and conducting dependency scans can be done using public tools. This will require further actions to visualise and process the findings since it will require to develop a dashboard internally.

On the other hand, Ebury could continue with a common, and non-secure SDLC approach. That will mean vulnerabilities will be detected in the latest stages of the development cycle, after the applications have been released to production. An increase on the complexity of resolving new and existent vulnerabilities will be produced. Furthermore, should these vulnerabilities be exploited by attackers, data breaches may happen, impacting the entire company.

Caveats

As it was discussed in the previous sections, developers should take the initial step of following secure best coding practices. To achieve that, multiple workshops should be delivered to educate them on secure coding practices, the risks vulnerabilities may introduce to Ebury and how to prevent them at the early stages of development. In addition, adequate tools should be provided to them to ease the process.

New policies should be created to evaluate the security posture of an application on each step of the secure SDLC. To do so, well established procedures could be modified in the future. These changes will be conducted in a transparent way for the developers without impacting any other team. However, depending on how restrictive the policies that evaluate the security posture of a certain step are defined, additional time might be required by development teams to meet these policies. Due to a Secure SDLC approach being followed since the early stages of the development, time to meet the policies will be taken into consideration when the projects are planned.

Operation

The Security Engineering team will take ownership of the SSDLC solution and will assist teams in the deployment of this into their existing processes.

Security Impact

Security will be impacted in a positive way. Automations will be created to ensure security is improved and enforced across the entire SDLC.

Performance Impact

The performance of the deployed applications and systems will not be impacted. However, it is possible that build times would be extended slightly due to the additional steps security tooling is performing. Where possible, these checks will be performed in parallel with other stages so that they are least disruptive.

A positive performance impact will be achieved by shifting the security further left in the SDLC, this will reduce the development time spent resolving each vulnerability, reducing the load on the development teams.

Developer Impact

Developers and team leads will be the most affected during the transition to a Secure SDLC. The following key areas may impact a developer's day-to-day work:

• Optional installation of supported IDEs
• Optional installation of vendor plugins to detect security vulnerabilities
• Addition of new CI/CD pipeline stages
• Remediating security issues discovered in their repositories
• Secure coding training

Additionally, team leads should plan accordingly considering the new tasks the Secure SDLC will append to development efforts, especially at the beginning of the transition.

Finally, after some time following a Secure SDLC, it is expected to be done automatically, without requiring developers to spend a lot of time coding securely or fixing vulnerabilities.

Data Contracts

The Secure SDLC proposal will not affect any data sources.

Deployment

Initially, the Secure SDLC will be implemented in a safe environment that only the Security Engineering department has access to in order to minimize disruption. All the components involved will be thoroughly tested before raising the changes with other teams. Once everything has been successfully tested, it will be presented to development teams before a gradual roll-out across the entire organization. The goal is to integrate and make use of the existing platforms used at Ebury to integrate security into the SDLC. To do so, automated third-party tools might be integrated to aid in the detection of third-party library vulnerabilities, as well as potential insecure coding practices.

Dependencies

No matter the chosen SSDLC vendor solution, the following dependencies exist due to how Ebury operates on a daily basis:

• IDE Integration: Common IDEs should be supported, such as Jetbrains or Visual Studio Code.
• Code repository: Ebury hosts its code on Github so SSDLC vendor solutions should be capable of integrating with Github.

Next steps

This RFC only addresses the need for Ebury to implement a Secure SDLC (phase 1). However, the following phases will each be defined in their own RFC at a later date. It should be noted that some of these phases can be deployed independently, and do not need to be performed phase by phase.

• Phase 1: Address the need of implementing an SSDLC;
• Phase 2: Deployment of an SSDLC solution;
• Phase 3: Deployment of a runtime security enforcement solution;
• Phase 4: Deployment of a Cloud Security Posture Management (CSPM);
• Phase 5: Enhance software development testing.

References

• https://snyk.io/learn/secure-sdlc/
• https://www.synopsys.com/blogs/software-security/secure-sdlc/
• https://www.eccouncil.org/cybersecurity-exchange/application-security/what-are-the-five-phases-of-the-secure-software-development-life-cycle/
• https://arxiv.org/pdf/1609.04886.pdf
• https://www.aquasec.com/products/software-supply-chain-security/
• https://snyk.io/product/snyk-code/
• https://snyk.io/product/open-source-security-management/
• https://docs.github.com/en/get-started/learning-about-github/about-github-advanced-security