How we mitigated the log4j vulnerability "Log4Shell" within 48 hours

Posted by André Christ on December 11, 2021

vsm-log4shell-vulnerability

Executive Summary / TL;DR

On Dec. 9, 2021, a remote code execution (RCE) vulnerability in Apache log4j 2 was identified as being exploited in the wild. Investigation revealed that this exploitation was incredibly easy to perform. Due to the broad usage of the popular Java library, many IT Systems and SaaS providers are now at severe risk. By submitting a specially crafted request to a vulnerable system, an attacker is able to instruct that system to download and subsequently execute a malicious payload. LeanIX mitigated the risk on our side by upgrading all crucial systems as of Saturday, Dec 11th. Analysis of our logs did not show any indications of exploitation. This rapid response – 48 hours – was possible thanks to the coordinated efforts of our passionate engineering team and the full visibility across our own (Micro)services provided by LeanIX Value Stream Management (VSM).

Identifying all affected services

A crucial first step for our team was to identify which of our software artifacts were affected by the vulnerability. Since we maintain an automated inventory of all services we use, we could query our internal VSM inventory for the affected log4j versions:

Screenshot LeanIX VSM: Identifying all affected services

This data is always accurate because we automatically receive the dependencies to underlying libraries from data produced in our CI/CD toolchain when building our services (see more details on our CI/CD integrations here).

Understanding the full impact on teams and products

With 150 employees in our product department organised in 5 tribes and 20+ teams, it was important to understand the full picture and see which services in which teams were affected:

Screenshot LeanIX VSM: Affected Teams

In order to identify any potential customer impact, we also needed to understand exactly, which of our products was impacted by the vulnerability:

Screenshot LeanIX VSM: Affected Products

Upgrading log4j library and monitoring the progress

To mitigate the vulnerability, our engineers implemented a strategy focused on upgrading the affected software versions. Once the logging library was updated and the service was rebuilt via the CI/CD pipelines, the updated versions automatically showed up in the audit trail.

Screenshot LeanIX VSM: Monitor Updates

Once deployment was confirmed (increase of deployment count), the status of the service was set back to "Green". In total, libraries in 7 services had to be upgraded.

Screenshot LeanIX VSM: Progress

Learn more

If you are interested to learn more, check out the following resources: