We in the Security world were just given an unwanted Christmas gift. Call it the 2021 Christmas Security Fruitcake from Hell (c). The Apache log4j vulnerability also called log4shell.
The JNDI vuln was originally talked about and made public in 2016 at Black Hat in Las Vegas : (click here for Youtube Video) There are those who state that it may take up to 5 years for the talks to hit mainstream. Well, it's 2021 and 5 years ago the talk was held. Maybe they're right ?
This is a big one and not something you need to ignore. I will predict this vulnerability alone is going to cost in the billions to resolve in the long run. But more so will impact every company of any size.
The main block of information is located here on the Apache Website. It was initially thought that 2.15.0 was the best path towards remediation, then 2.16.0 and on Friday night (2021-12-17)/Saturday morning (2021-12-18) there was an article released that stated there was a new path as discovered by Security Company Blumira. Thi sis not looking good at all.
Early in the morning of Saturday (2021-12-18) Apache releases 2.17.0.
Personally, I must commend the maintainers and Apache for their swift addressing of this vulnerability. However, it's snowballing out of their control.
Friday 2021-12-17, The Record posts: Google: More than 35,000 Java packages impacted by Log4j vulnerabilities
More than 35,000 (thirty five thousand)! Oh my GOD!
Google posted this on their Security.googleblog.com website about this vulnerability article and information. Supposedly, they have already cleaned up nearly 4,000 of these within 1 week. Ok, so there is hope.
Take a look at "Understanding the Impact of Log4j" on the googleblog website. Google is stating that it could take up to years to fix everything. Wow. That's mind boggling!
Why is everyone screaming about this log4j vulnerability? Easy. It's one of the most popular java libraries so widely used it is thought to exist in over a few billion devices, software and embedded hardware devices. This vulnerability will blow away any of the previous vulns statistics for years to come.
Suggested Remediation(s) that are considered Stop GAP measures and Temporary:
- Upgrade/Update your firewall. Call your FW Vendor and see if they have a patch available to block or filter out any of these attacks. (see Downsides of Filtering below)
- Upgrade/Update your IPS immediately. Call your IPS vendor to get the latest patches installed. If your FW vendor has an IPS built into it, get it updated ASAP.
- Update your WAF ASAP. Contact your WAF vendor to see what they have ready to be installed now.
- Focus on Internet facing devices (hardware and servers) and software. Get these upgraded ASAP. Do not fumble around with anything lesser than a full upgrade to the latest version of the Library. In this incident, it has already been proven that shortcuts do not solve this problem. They may actually introduce more issues.
The above steps will give you a slight respite as you move forward with full remediation.
Downsides of Filtering on your Firewall, IPS and/or WAF:
This vuln has already begun mutating. This means that there will be millions of fuzzing and obfuscated strings used in these attacks and not just the straight clear text strings. All of these will be aimed at bypassing these filters. Then comes in the more sophisticated State Sponsored attacks that will be difficult to stop. Then the Malware mob (as I call them) will be utilizing this vuln to deliver Malware onto your systems.
True Remediation will take time:
The only true remediation is to update ALL Devices and software that utilizes the Log4j library. If you have 100 Software Applications you developed and running, you will need to update ALL of them. If your development team utilizes any of the libraries that Google identified within the Maven repository, you will need to update these as well into your applications. This may hold you back from updating ALL applications waiting on announcements of updates, but keep moving forward. Focus on those applications that you can fix immediately.
But start on the fundamentals. Get control over your assests and your Applications!
Other Required Tools:
Ok, this is a good time to talk about a CMDB (Computer Management Data Base). Do you have one ? Does it also provide SBOM (Software Bill of Materials) capabilities?
Having a CMDB is a step forward. It can help pinpoint what systems have the Log4j.jar or other library files are installed on them. Make sure that the CMDB agent is installed on ALL systems, including Production, QA and Development systems. This will help decipher what needs to be updated quickly.
An SBOM can help find which applications have what libraries installed. Great for this type of security issue. Is the tracking of what libraries used in your SDLC ? It needs to be and it needs to be updated each time you change libraries or create new applications.
Having a notable SIEM that can be tweaked to alert you to any intrusion based upon Log4j is critical. Don't focus on the cost, focus on what will it cost if you are breached and exposed to the world? You need some sort of alerting system and auditing system to show what machines, who touched and date and times. Going through individual logs is 1800's style of incident management. Drag your leadership into the 22nd century and get them to approve and buy a real SIEM package!
Food for Thought and Best Practices:
- Place into your SLDC that ALL outdated and no longer used Libraries be removed from the developers systems. There should never be any libraries that are not approved by Security floating around.
- Centralize your development libraries. No need to have thousands of libraries scattered all over your developers laptops, systems or storage. Centralization helps to keep track of them also.
- Upgrade to the most latest Library. Patching requires new libraries, remove old, replace. If there is a fear that a new updated library will break an application, then TEST it and prove your fears to be true. Many libraries available are all backward compatible which by nature negates that fear.
- Think about securing your Applications and libraries. Scan them periodically with the latest techniques to ensure your libraries and applications are safe. Just one scan before going into production is nowhere near enough.
- Set up a vetting process for your Security Department to approve libraries and software you want to incorporate into your applications. This should be a baseline for any SDLC.
- Keep aware of the Open Source community and notifications from them. They find these luns all the time. Listen to them.
- Listen to Industry Experts.
- Perform a Fire Drill periodically. Mimic what needs to be done to fix these vulns. Make people especially Developers and Mangers of aware that there are procedures in place.
- Follow your own procedures! Don't have one? Write it !
More to come as this unfolding Security Incident continues to evolve. I'll post more articles against this parent article.