We’re now at about the two week point after news of the vulnerability in the Apache Foundation’s Log4j logging tool for Java, dubbed Log4Shell, splashed into the headlines.
For a catch up on what this story is all about and a guide for how to kickstart your mitigation efforts, check out our post from 12 December before continuing to read.
It’s ok, we’ll wait.
Hopefully by now you’ve been able to identify which of your applications are vulnerable and have started patching. You can use our repo of confirmed vendors and components to save some time and start patching your most critical apps and software.
This post will look more at developments in the story and provide advice on how to mitigate your risk for the next wave of attacks that will come after this.
Let’s get started.
Where Do We Stand Now?
The Log4Shell vulnerability has caught mainstream attention, increasing the pressure for organizations to get on top of their patching.
While it’s still too early to really know how organizations are doing, there are some indicators of progress.
According to research by cloud security company Wiz, along with Ernest and Young, that looked at enterprise cloud data, they found that 45% of vulnerable resources had been patched by the 10th day after the announcement.
While 45% might not sound great, it beats the average which is usually down in the 30s. The problem is that this research doesn’t tell the whole story when it comes to figuring out if we are any safer.
Patching Can Be a Rocky Road
To put it lightly, this patch rollout hasn’t been the smoothest. After getting folks to upgrade from version 2.15, version 2.16 apparently had a DoS bug that was later shown to have a Remote Code Execution (RCE) bug, so that wasn’t great.
Version 2.17 is now out and we recommend going to get the patch now, though some CISOs may decide to wait a tick or two on this one.
With this steady stream of well publicized bug-filled updates, an org may have started patching early on like they should, but now find themselves having to repeat this process over a couple of times.
It is highly likely that many security professionals are holding back and waiting for a more secure version to come out before investing their resources in starting to patch the (tens of) thousands of apps that are using Log4j.
While some reticence to act now may be understandable, attackers appear to be taking full advantage of the bug and surrounding confusion to squeeze this lemon to the last drop.
Preparing for the Next Wave of Attacks
Like most cybersecurity incidents where a truly critical vulnerability is exposed, the vast majority of the initial criminal/hacking activity appears to have been focused on cryptocurrency mining.
A marker of this vulnerability is the ease of exploiting it on unprotected systems, requiring neither privileged access or special expertise. But as more sophisticated attackers have more time to play with it, they are becoming more ambitious in their targeting.
Already in the first few days of this story, reports were already coming out that state actors like China, North Korea, Iran, and others were already pouncing on Log4Shell as part of their espionage campaigns. We should assume that the Five Eyes and any other hackers who draw a government check should probably be/is doing the same.
With the government groups leading the way as they often do, it’s unsurprising that we are already seeing more advanced criminal groups incorporating Log4Shell into their work as well. Reports are out now that the Conti ransomware group has developed a full attack chain using this vulnerability. We can expect others to follow their example soon enough, and with Christmas break around the corner, they will have plenty of time to try out this exploit.
These developments that we can see are worrying, but wait, there’s more. What about the ones we do not see yet?
We have to figure that hackers of all sorts are taking this opportunity to lay the groundwork for future attacks. In practice this means that they are:
- Gaining persistence on compromised servers
- Collecting valuable data like legitimate credentials lists
- Learning how their targets respond to see who is on top of their game
The result is that while some no-goodniks are making for a fast payday with ransomware and other kinds of attacks, others are playing the long game. Some are probably doing both.
And for good reason. This software is going to keep getting intense focus from attackers because:
- It is now known to be so widely used so more available targets. Straddling the line between responsible disclosure and code red, there are now multiple lists on repos telling everyone what is vulnerable. This is a good thing because it makes for faster patching ops, but it is also a target list for attackers in the same way that the CVEs normally are.
- The rush for fast releases means that there will be a plethora of vulnerabilities in the new versions.
- Many orgs simply are still probably unaware of all the spots where they’re using Log4j, so they are likely still unpatched. Even if they have a Software Bill of Materials (SBOM) from each of their vendors, this is not an automated process and it will take valuable time to manage.
So what should organizations be doing to protect themselves?
Mitigating Risks to the Access Layer — Assuming Breach
While much of the focus of this story has been rightly on the need to support the open source community in maintaining the projects that are so heavily used across every industry apparently, it is also a real marker that we are entering a new era for security posture.
This shift starts with accepting that we really are in the era of assuming breach. Given the ease of compromise and the scale of available targets, it is not hyperbole to say that attackers could well be inside of whichever targets they have the time or desire to be in.
Responding appropriately means adopting a Zero Trust posture where we verify every request, every time.
On a high level that means using tools that authenticate identities and reduce access to resources down to the absolute minimum levels necessary (Principle of Least Privilege). If an attacker does manage to gain a foothold in your environments, segmentation via access controls is your best bet to reduce the blast radius of the damage that they may be able to cause.
Implementing a Zero Trust approach is not a single tool or product, though there are plenty of tools that will help you achieve it more effectively. It is a long term strategy that requires a change in how we think about security in the cloud/WFH era.
But for those seeking more immediate advice for mitigating their risk from Log4Shell, check out a couple of useful resources and tips here below.
5 Steps for Log4Shell Mitigation Now and in the Future
Upgrading to the latest version of Log4j is a must for all. Here are a couple more good ideas.
Understand What is at Risk
In an attempt to improve their posture, the Cybersecurity and Infrastructure Security Agency (CISA) is expecting a full accounting from every federal agency of all their software that takes data from the internet and whatever is vulnerable to Log4j. Which as we have established, is basically everything. They are also expected to remediate whatever has been impacted.
Oh, and they want it done by today, December 23, 2021.
While everyone wants to be done with this nightmare of an accounting task before the Christmas break, this is going to be a tough one for all organizations. Knowing where to start can save you a lot of time.
We recommend using a couple of the many free resources out there to identify which apps and components are verified as vulnerable.
At this point everyone and their father’s brother’s nephew’s cousin’s former roommate has put out a Log4j vulnerability checker/scanner. Ours is pretty great too so give it a whirl if you haven’t already.
For some reason that somebody needs to do a deeper dive on, the Dutch authorities are really good at the cybers. Check out their repo for some of the most comprehensive updates, especially if you don’t spreek Nederlands. Also, the good folks over at CISA have a good repo going so be sure to check theirs out as well.
Have an Active SBOM
Software Build of Materials (SBOMs) are going to have to be a big part of the story moving forward. Having a good idea of what you have in your code is the first step towards patching.
However, the SBOMs are not going to be the silver bullet that we need quite yet. As a number of experts have told SCMedia’s Joe Uchill, the scale of the number of applications in an organization and the lack of an infrastructure that knows how to handle the workload of taking those SBOMs and using that info for search is still far from where we need it to be.
Turn on MFA
As noted before, hackers are likely to come away with boatloads of creds that they can use to gain access. Make their lives a little bit harder by making sure that everyone has some good ‘ol Multi-Factor Authentication (MFA) on their accounts.
Basic security hygiene is often the first and best defense against malicious actors.
It is probably a good idea to also start a push for your org to change out passwords that might’ve been compromised. Maybe this is the time to even go passwordless? It’s a new year and the opportunities for better security are wide open.
Monitor for Indicators of a Breach
These next few days and weeks are going to be critical as the ne’er do wells try to get their hooks in deep.
If you see suspicious activity, call in an incident response team earlier than later. The longer you wait, the harder it will be to pull them out so it’s better to start as soon as possible.
Watch for Version Updates
As we’ve noted throughout this post, stay a step ahead of the herd by updating to the latest version. Keep on top of the Log4j page from the Apache Foundation for updates. They are probably going to keep pushing out updates with fixes for the next while, so be sure not to miss out.
You can also follow them on Twitter to make sure that you don’t miss the next patch.
Preparing for a Not So Jolly Holiday Break
On Monday, CISA came out with an additional set of warnings before the Christmas break.
Their Public Service Announcement (PSA) posting offered key advice for how to respond to an incident that may occur over the holiday when organizations are likely to be on skeleton staff. Most of which gels pretty well with our advice above.
“In the heightened threat environment that we see during holidays, proactively strengthening your cyber defenses can help ensure your friends, family, and staff can enjoy some well-deserved time off,” said CISA Director Jen Easterly. “Ultimately, good cybersecurity is not only about technology – it’s about people. It’s important to take the appropriate steps to protect ourselves so we can all enjoy some peace of mind (and eggnog!) this holiday season.”
Attacks over holiday breaks have become a standard part of the cybersecurity rhythm, so this warning comes as no big surprise.
But with a shiny new vulnerability to play with, we can expect hackers to make this a holiday season to remember.