Keep Security in Your CI/CD or Pipeline at Your Own Risk

There are continuous integration/continuous delivery (CI/CD) pipelines doing so many automated things for us in today's world. The need for security and general good practices in implementation is of paramount importance. 

It takes real effort on the enterprise to continuously avoid security breaches that can potentially destroy the established trust in the company. This article will discuss some of the valuable considerations to keep at the forefront, so they are not lost due to competing priorities. 

Authentication and authorization

Two items are often misconstrued, so let's clarify their differences before moving on. Authentication (Authn) is a matter of confirming that the user is who they say they are. Authorization (Authz) allows said users to access specific resources. These two elements in tandem constitute identity and access management (IAM). It is essential to start with appropriate IAM configurations for user and service accounts, and understanding this difference will aid in account creation. 

Principal of least privilege (PLP) becomes even more important the closer to production you get. Understandably, developers may need more access for experimentation in the sandbox than the customer-facing environments. To that end, limiting network access to specific hosts and within the org/VPN will limit the scope of potential external exposure. This further improves your security posture from the start.

Infrastructure and configuration

Outside of a compromised account, misconfigured infrastructure is one of the largest vulnerabilities that can leave your organization open to attack. Start with making sure accounts like root and administrator are not being used unless absolutely necessary. Separate the accounts that run your application and configure the system. 

The best practice for deploying infrastructure is using Infrastructure as Code (IAC) and keeping it in source control. This approach will allow for the shortest feedback loops to infrastructure experimentation. The same applies to Configuration Management (CM). In addition, let's lock down unnecessary ports, disable unused/unnecessary services and create a reminder to audit the system deployment (IAC/CM) regularly to make sure drift in configuration or implementation isn't leaving you exposed. Audit tools like Terrascan and AWS Inspector can assist with this.

Infrastructure & secret management

Secret management is vital to overall application, configuration and pipeline security. Let's cover the basic best practices. Using a secret store is critical, be it Vault, any of its competitors or your cloud providers variant like AWS Secret Manager, Azure Key Vault or GCP's Cloud Key Management. 

Abstracting your API keys, SSH creds, Personal Access tokens and passwords and being able to rotate them out is crucial. Separating them per account and/or per environment and/or per service will limit blast radius if a compromise occurs. Compromised dev credentials should not impact prod. Renounce putting plain text credentials into build scripts, configurations or checking them into code repositories. Using something like Mozilla SOPs will enable you to add encrypted secrets in the git repo if you must. This is for secrets that don't need to be managed per environment or called to the keystore. Otherwise, abstract and call a secret management tool to access your API keys, servers, services, etc.

Less security, more best practice

Here are some auxiliary elements that will improve the software development lifecycle (SDLC) journey as you create CI/CD. These elements impact security success, just like setting up proper Auth/Autz, setting up IaC and CM, and properly managing secrets.

Creating a naming convention and keeping it consistent, such as "task_project_environment," will help keep you from making unintended changes to specific projects or environments.

Maintain your build configuration files in source control. All of your .gitlab-ci.yml, Jenkinsfile and .circle-ci/config.yml living in source control will allow you to version, audit and optimally maintain them. If a submitted change breaks a pipeline, reverting a commit will be much easier than tracking down an engineer or manually troubleshooting what went awry. At this point, we are on the doorstep of GitOps — a practice of putting Infrastructure as Code, using a merge request process and putting CI/CD in source control. This emerging technology term, like many others, is not strictly defined the same way by everyone. The principals in its application can be of value for many teams. 

Mind the installed plugins in your implementation. They go out of date often, sometimes have critical vulnerabilities and even cease to be useful in some cases. A periodic audit of your plugins may enable you to stay out of tomorrow's data breach news cycle. This is especially true for Jenkins implementations but applies to other CI/CD tools as well. In the case of Jenkins specifically, an enterprise solution and implementation such as CloudBees can simplify your deployment process and safeguard you from risk. 

Logging and monitoring

Log everything and know where that data goes, especially for production. Logging and application monitoring will allow for a myriad of improvements to be made to your app, all because of the information it provides. Are you hitting out of memory errors, I/O limitations? Is your CPU pegging? 

Logging and monitoring will offer you the insight to not only tune the application but to know when you're subject to attack. To subvert poor code living in your source code, implementing static/dynamic code analysis tools like SonarQube is a great option. An added method to curtail attacks is to ensure your firewalls are enabled and maintained, both on the host machine and any potential worker nodes.

Finally, if refining how you utilize CI/CD is on the agenda for 2021 and beyond, consider these fundamentals. 

• Bring your organization to a place where you can relish knowing your IAM is on point.
• IAC and CM are modular and in source control.
• There is a secrets management tool implemented and being used.

Let's make it easier to implement changes, not harder. Limit the risks of doing so by having all the security bases covered.