How to deliver secure code more frequently

Security is (or should be) everyone’s responsibility throughout a software delivery pipeline. Application security and continuous delivery need to combine to create a holistic secure DevOps solution. Let’s talk through how to do that, starting with some definitions to get us all on the same page.

Continuous Integration/Continuous Deployment (CI/CD) Pipeline

CI/CD is a collection of practices to deliver code changes more frequently and reliably. A CI/CD pipeline automates these practices using tools like HCL Launch.

Continuous Integration is a development practice where code developed by multiple developers for the same features is integrated frequently to surface errors earlier. The process includes these steps:

• A developer writes a code in a development box.
• A developer pushes the code to a version control repository.
• An automation server runs a build to verify the check-in and runs all the unit and integration tests.
• An automation server shares build and test results with the development team.
• If there are failures the team works on fixing them.

Continuous Deployment is a practice to release every good build to users as soon as it’s ready if these conditions are met:

1. If the build and unit/integration tests are successful, the build is deployed to a UAT server.
2. If UAT is successful, the build is deployed to a production server.

AppSec

AppSec (Application Security) is the process of finding, fixing, and preventing security vulnerabilities in the software development cycle. As 85% of security incidents are observed at the application layer and with the number of threats steadily increasing every year, it is important to have AppSec in the software development lifecycle. Application security measures are enforced to identify and remediate the security issues in the early phases of the SDLC, reduce overall development cost and deliver secure code. Widely used application security testing methodologies are:

• Static Analysis Security Testing (SAST) – A white-box testing that analyzes your source code from inside to detect and report security vulnerabilities. Problems can be identified during the development phase itself, providing immediate feedback to the developers. Tools or IDE plugins exist to identify security flaws in the source or the compiled versions of code in a non-running state or without code being executed. SAST reports would contain all required details including filenames, providehclhc line numbers to remediate issues, and identify false positives. Common and popular SAST tools include HCL AppScan Source, HCL AppScan on Cloud, Veracode, Checkmarx, SonarQube, etc. SAST tools generate reports with the required information for developers to remediate the issues.
• Software Composition Analysis (SCA) – A mechanism to identify insecure open-source libraries bundled in the application and ensure these libraries meet basic security standards and do not introduce risk to the organization. SCA tools can also detect software licenses and deprecated dependencies. Some of the popular SAC solutions available are HCL AppScan On Cloud, BlackDuck, Veracode, CheckMarx, WhiteSource, etc.
• Dynamic Application Security Testing (DAST) – A black-box testing that tests applications from outside. It looks for security vulnerabilities by simulating external attacks on an application while the application is running. It attempts to penetrate an application from the outside by checking its exposed interfaces for vulnerabilities and flaws. A few advantages are its capability to identify runtime problems that SAST cannot, technology-independent, low false positives, identifies server configuration issues. Some of the popular DAST tools available are HCL AppScan Enterprise, HCL AppScan on Cloud, Veracode, Netsparker, Checkmarx, etc.
• Interactive Application Security Testing (IAST) – Works from within an application through instrumentation of the code while the application is running to detect and report security vulnerabilities. IAST is generally deployed in the QA environment when the deployment is used for automated/manual functional testing. IAST reports provide detailed information including line numbers which can help developers in remediating the issues easily and identifying false positives. Some of the popular IAST tools available are HCL AppScan Enterprise, HCL AppScan on Cloud, Synopsis, Contrast Assess, etc.

CI/CD pipeline with AppSec

An AppSec pipeline follows the principle of DevOps, sometimes called DevSecOps. Security practices are included in the CI/CD pipeline to deliver a secure code more frequently.

Continuous Integration (CI) with AppSec

• A developer writes a code in a development box.
• A developer performs security testing using SAST IDE plugins.
• A developer pushes the code to a version control repository.
• An automation server runs a build to verify the check-in and runs all the unit and integration tests.
• An automation server runs a Static Analysis Security Testing job (Automated)
• An automation server runs an SCA job (Automated)
• An automation server shares integration, unit, and security test results with the development team.
• If there are failures the team works on fixing them.

Continuous Deployment (CD) with AppSec

• If the build and tests are successful, the build is deployed to a UAT server.
• Perform DAST and/or IAST (Automated)
• If UAT testing is successful, the build is deployed to a Production Server.

Steps to integrate AppSec into CI/CD pipeline

SAST into the pipeline – It is recommended to automate SAST scans to run for every check-in. Leverage the incremental scanning capability of the SAST tool to reduce the scanning time.

Application onboarding is a one-time combined effort to be performed by the security expert and the development team to onboard applications into the CI/CD pipeline. Steps include

• List the files/packages to be scanned. You might want to omit dependencies.
• Create a config file as needed by the SAST tool.
• Run a scan and push the results to the central repository.
• Automate the step to push the results to a central repository.
• Triage the scan results and mark false-positives in the repository.
• Baseline the scan. New issues in the subsequent scans get highlighted in the central repository. This avoids re-triaging unresolved and reappearing issues and lets you focus only on the delta.
• Introduce a process to push the results of every scan to a central repository before applying build gating (making a build fail if specified criteria are not met) and it is important to have issues merge capability in the issues repository.

Some important things to keep in mind about SAST scans include:

• It is crucial to create a scan rule/configuration and filters specific to the application to get the most accurate and actionable results. This depends on the set of issues you are interested in (ex: OWASP top 10) and requires the knowledge of onboarded application and able to understand sink, source and taints from the SAST results.
• Relatively, SAST reports more false-positives since the SAST tool does not understand the context the code runs in. A person who is aware of the context of the application, application users, information processed by the application, validation mechanisms used should be able to triage security issues, identify and mark false positives.
• Automating a scan is the last in the process after baselining the scan, triaging issues, and identifying false-positives. Either command-line options or plugins are used with an automation server to automate scans.
• Let scan completion initiate a process of creating tickets in the defect tracking tool automatically. It is recommended to use SAST tool features like fix-group (issues that require a single remediation task) to create a single ticket for a bunch of SAST issues. This can contribute to a simpler triage process and faster remediation.
• Stop a build/pipeline depending on the policy defined. Ex: Stop the build if – ((number of critical issues > 0) || (number of high issues > 5))

DAST into the pipeline – You would want to run DAST scan for every code check-in or on a particular day in a week or on reaching a particular stage of Agile Sprint. It is recommended to leverage the incremental scan capability of the DAST tools and reduce the scanning time.

Some important things to know about DAST scans include:

• Application onboarding is a one-time activity that would require identifying the starting URL of the application, generate and include recorded login sequence to the scan, and tuning scan configuration for accuracy and speed. This can vary between applications and hence the person responsible for configuring a scan need to have a context of the application.
• It is equally important to scan web services (ex: REST APIs) which would require including traffic captured through proxy servers during functionality testing using API client tools like Postman, SOAPUI, etc.
• It is crucial to have the starting URL of the application unchanged every time the pipeline runs. This would require an automation server to deploy the application to the same box every time the pipeline runs. Otherwise, every run expects the login sequence regenerated and added to the scan.
• Let scan completion initiate a process of creating tickets in the defect tracking tool automatically.
• Stop a build/pipeline depending on the policy.

Applying application security to your CI/CD pipeline helps organizations solve these common issues:

• Reduced manual intervention – Automation reduces human error by taking the manual work out of tedious processes.
• Early detection of security issues – By introducing security testing earlier in the SDLC, threats, and vulnerabilities can be detected and addressed faster.
• Continuous vulnerability triage – Automated scanning and ticket creation help security engineers and developers decide which must be addressed immediately and who is responsible for resolving the problem.
• Responsibility clarification – Automation takes the uncertainty out of security in software deployment. Shifting security can confuse who is responsible for what, but automated scans can present remediation options for the party responsible at that stage of development.

For more information on our secure DevOps solutions, check out the HCLSoftware DevOps portfolio.