Mitigating Log4Shell With the Help of Synopsys Blackduck and a DevOps Culture

In the early hours of Friday, December 10, 2021, Information Security professionals started receiving notifications about a new exploit coming from one of the world's most commonly used logging frameworks, named Log4J. This exploit would allow a malicious user to execute whatever code they wanted to on a vulnerable system, simply by sending it a specially crafted message. As the day progressed to the weekend, organizations began a mad scramble to identify their exposure points.

Below, we'll discuss how development teams were able to quickly identify whether their applications were vulnerable. This allowed them to pivot into mitigation mode, update their software, and give them the peace of mind that they wouldn't fall victim to this exploit that is taking the world by storm.

What is Log4J and why do we have a problem?

Simply put, applications need to log things. One will typically see messages like "08:45:23 User john.doe logged in" or "15:43:21 Employee 45 accessed the medical record for patient X." When events happen in an application, they are often captured and written to a log file for accountability and auditability purposes. One of the most commonly used tools for applications written in the Java programming language is a library called log4j.  

Users often include log4j in their applications for its ability to create consistent logging messages and logging severity levels. It is easy to set up and provides a solid framework for capturing messages from events. However, in 2013, a feature was added to the log4j library that allowed it to potentially reach out to an external server to look up information that it didn't already have available. It would then execute the information that was returned, with the intent being that log4j would simply place that data into the log file.

It was discovered and disclosed that this lookup functionality had a severe flaw. If someone was able to get a very specific message into the logs, the lookup would automatically get triggered. This would then have the unfortunate side-effect of executing the payload that was returned from the lookup, even if it was something malicious.  

End users don't typically have direct access to logs to inject their own information. Those users do, however, have access to input fields such as "Username" and "Password" when attempting to log in to an application. As seen in the log example earlier, this data is often taken directly from that field and placed into a log message. Researchers found out that by placing that specially crafted message into fields such as these that are often logged, they could easily get their code to execute on the target system.

Implementation of Blackduck

One of the main goals of our development teams is to write secure software, both for our own internal use, as well as for our customers. A key requirement in all of this is understanding the components that make up the applications we build. An easy way to do this is by maintaining a Software Bill of Materials, or SBOM. SBOMs in and of themselves aren't anything new, but gained much more media attention due to 2021 attacks on the software supply chain and recent executive order

One tool that we often use to maintain this accurate bill of materials is a tool owned and developed by Synopsys called Blackduck. Blackduck is known as a static composition analysis (SCA) tool. Its main goal is to scan through an application and capture all of the known libraries that are used to build and run said application. The tool then identifies the security and license risks associated with using that library.

Providing tool availability

To ensure that teams were able to effectively utilize Blackduck, a decision was made to make the scanning process as consistent as possible between projects. The Blackduck scanning capability was placed into a Docker container and offered up to each team to use freely.

To try and embrace having security built into the DevOps culture, our InfoSec and Application Security teams worked with the developers to integrate the scanning into their CI/CD pipelines. Shifting the visibility of issues to earlier in the development process is a key way to ensure that projects aren't being released with vulnerable libraries. By simply adding a step into their existing build flow, teams would be able to quickly see their SBOM and any currently known risks associated with their project. 

Synopsys alert

An additional feature that was then leveraged was a tool called Synopsys Alert. This functionality takes the notifications from the Blackduck tool and presents them in a format that is more easily digestible by development teams. This can be done via instant messaging tools like Slack or Microsoft Teams, or through more traditional avenues like email. Adding Alert to a project allowed developers to have a real time view of risks that show up.

Mitigating the Log4J vulnerability

Due to the realtime aspect of Synopsys Alert and the teams' ability to bake Blackduck into their pipelines for continuous monitoring, the morning that the vulnerability was disclosed was a flurry of quick actions to mitigate the areas known to have vulnerable versions of log4j within them. Rather than having to play the guessing game of wondering who or what might be affected, alerts were quickly sent to teams via Alert, tipping them off to a potential issue.

Logging in to the Blackduck instance quickly gave our Application Security teams a birds-eye view into all of the known vulnerable projects and the versions they currently relied upon. Several teams had already taken it upon themselves to respond to their alerts and perform the upgrade guidance. Those that hadn't were easily notified due to the visibility that was available to the security team.

Due to the quick response by the development teams and their desire to make security a first-class citizen within their Software Development Lifecycle, teams were able to have access to their SBOM and determine where the vulnerable version of Log4J sat within their project. This allowed them to update accordingly, long before the mass exploitation of the vulnerability began.

Conclusion

The importance of maintaining an accurate SBOM for applications is ever-increasing.  With constant attacks on the software supply chain and newly disclosed critical vulnerabilities such as the one found here in log4j, having an inventory of software components is invaluable. Additionally, performing actions such as scanning during the build pipeline can help to increase the visibility of issues as they crop up. This saves both time and money as it minimizes re-work by the development team and keeps the operations side of the house from scrambling to understand where the vulnerabilities lie.