Lessons and Defenses Learned From SolarWinds ‘Sunburst/Dark Halo’ - Atomicorp - Unified Security Built on OSSEC

The widely reported December 2020 hack of the SolarWinds Orion network performance monitoring system employed a sophisticated series of takeover steps that included backdoors, expired domains, the use of Orion itself as a vector, compromised credentials, and malware implants, all to steal data and compromise systems. The attack, referred to as Sunburst, Sunburst Backdoor, and Dark Halo, may have begun with undetectable malicious code, but subsequent stages were anything but undetectable. In the lab, I reconstructed Sunburst and monitored it with OSSEC. I uncovered multiple warning signs that looked like all-too-common ‘blip’ occurrences, but they were not. These so-called “false negatives” were actually the start of something malicious.

In my previous blog, I discussed fending off lateral attacks in Windows environments using Open Source Security (OSSEC) and provided some results on how I was able to detect and stop Sunburst lateral movement elements. This blog focuses more on which software vulnerabilities were exploited during the SolarWinds hack, and how organizations can use OSSEC to detect and stop Sunburst/Dark Halo and similar attacks.

The SolarWinds Sunburst Attack Exploited Supply Chain, Software, and Infrastructure Vulnerabilities

From the initial vendor code modification to the information gathering and exfiltration, this was a well-engineered attack. It exploited a long-feared supply chain weakness, allowing the responsible hacker group or state-sponsored entity to create their own zero-day vulnerability inside a trusted channel/product and set the stage for deeper infiltration and exfiltration. 

Here are the software and infrastructure vulnerabilities we saw:

• The initial dynamic link library (DLL) injection
  • The attackers inserted malicious code into SolarWinds.Orion.Core.BusinessLayer.dll, a code library belonging to the SolarWinds Orion Platform, in a method named RefreshInternal. This malicious code created a ready-made “forward operating base” for the attackers to connect to remotely and perform reconnaissance and data exfiltration.
  • The DLL was digitally signed using the legitimate SolarWinds code-signing certificate, then pushed via the SolarWinds update channel.
  • The DLL was installed in the Orion product as part of an update. Since Orion is recommended to be installed using a local server administrator account, the attacker started with elevated privileges.
  • The backdoor, after ensuring that the installation went undetected for 14 days and that the environment was free from potential detection tools, initiated a connection to a predefined C2 server to report some basic information about the compromised system and begin receiving commands.

• The exploitation of the SAML tokens and AD federations, using an attack dubbed “Golden SAML”
  • The Golden SAML technique involves the attackers first gaining administrative access to an organization’s ADFS server and stealing the necessary private key and signing certificate.
  • When the user attempts to access a service and when the service redirects the request to ADFS for authentication, the attacker would forge a SAML response using the stolen key to gain unauthorized access.
  • ADFS servers are considered “tier-zero” infrastructure and are therefore usually well protected, requiring high privileges for access. However, the threat actors in this case had a major advantage. The attackers had compromised a trusted system – the SolarWinds IT monitoring solution itself.

• The rampant use of Powershell and other LOLBINs as admin –
  • The attack utilized Powershell scriptlets to instantiate remote scheduled tasks and perform reconnaissance
  • Powershell was executed with the “NoProfile”  “ExecutionPolicy bypass”  and “EncodedCommand” switches
  • Remote file copying was performed using Admin c$ shares
  • The adversaries used CMD / C to execute administrative Powershell scripts (exshell.psc1), zip files for exfiltration, and execute other external utilities (sqlceip.exe)

So how could we have detected or stopped the initial DLL injection? Sadly, this was highly unlikely. This was a very sophisticated detection that goes beyond a zero-day exploit. They created their own backdoor. The adversaries broke into a trusted code repository, inserted code that was not caught in regression / unit testing or by any other software development controls. Once the code passed acceptance testing and was signed with SolarWinds’ certificate, the odds of something flagging that library were near zero. It was SolarWinds’ responsibility to secure its trusted supply chain, and it failed. It is unfair to expect that organizations should have been able to protect against a malicious injection into a trusted infrastructure product. If that is true, then we have to envision a world in which no one is allowed to run any software as an administrator ever again.

However, the adversary’s ability to leverage its custom back door to run rampant through the enterprise could have been better detected with proper defense-in-depth.

What Is Defense-in-Depth?

Defense-in-depth is a term that encompasses multiple considerations. In the classic military sense, defense-in-depth means that the defenders constructed multiple obstacles for the attackers to overcome in sequence. When the attackers overcame the outer barrier, the defenders retreated to the next barrier, and so on. Successful military defense-in-depth tactics used a diversity of barriers, both visible and hidden, to thwart smart planners and prevent one attacker tactic from being successful against all barriers.

Defense-in-depth in cybersecurity is similar. It requires:

1. A complete defensive architecture (physical, logical) that addresses all attack surfaces and environments
2. A layered defense that places multiple successive physical or logical barriers in the path of the adversary
3. The use of multiple diverse technologies to prevent the attacker from overcoming all defenses
4. A robust detection and response capability that allows defenders to engage the adversaries so they may not make their incursion unchallenged

Building a successful defensive architecture with defense-in-depth in the current computing paradigm is challenging. It requires a moderate to significant time and labor investment in developing internal policies, procedures, standards, guidelines, and initiatives. It also requires a moderate to significant investment in cybersecurity technologies to allow the organization to successfully detect, protect, respond, and recover from a cyber attack.

Detection, Active Response, and Analysis – What Atomic OSSEC Brings to Defense-In-Depth

Well-resourced adversaries will always have the opportunity to employ zero-day attacks against your network, and there should be no expectation that these can be reliably stopped. However, once inside your network, the adversaries need to be able to perform reconnaissance, move laterally, escalate privilege, access data, and exfiltrate assets. The more obstacles for them, the better your ability to stop the spread and damage.

Unless the adversary strings together multiple zero-days, or unless they find a way to jackpot domain admin credentials right off the bat, they still must use well-known techniques to execute their mission. This is where a defense-in-depth detection strategy comes into play, and can be harnessed effectively.

Detection requires artifacts that can be captured and analyzed for malicious activity. In the physical world, this can be movement on a camera, a hit on a motion detector, or the activation of door sensors. In the computer world, we have logs. Anytime a file, process, or object is created, modified, deleted, or disabled, there is a log capability that can capture that action.

Effective detection requires logging to be ubiquitous, diverse, and meaningful. Ubiquitous logging means logging from all areas of your network, from the perimeter to the core of your network, and every device in between. You can’t detect an APT if there are areas of your network where they can operate unseen. Diverse logging means logging multiple sources on the same system (i.e., operating system, application, interpreter, protocol, database). This allows you to see operations against all the areas of your system that comprise the attack surface, and the areas that following a successful attack the attacker may operate in. Meaningful means that the logs you generate have enough context and information that a smart system or operator could determine that something bad was occurring. Yes, that means increase your logging (especially on Windows – see my previous post).

So how does this make Atomic OSSEC more powerful than other solutions?

Atomic OSSEC has a very powerful log decoding capability. It has the native ability to decode over 130 log formats out of the box, and you can easily write XML-based decoders for any crazy log format you have. This means that your ubiquitous and diverse logs can be easily understood by a single technology. It is also lightweight and does not require the installation of complex or resource-intensive clients.

Atomic OSSEC agents are smart. They know how to perform significant log reduction and aggregation. This reduces their output to a fraction of other products. And this also helps them play well with other intrusion detection products that charge based on bandwidth.

Finally, they have a powerful ruleset that helps them to extract meaning from all of this data. Using our cutting-edge honeypots and testing facilities, and applying decades of experience in developing attack detection rules, our senior analysts have developed thousands of rules for Atomic OSSEC. These rules do not rely on filenames, or hashes, or any other indicator that could easily be obsoleted by a change in tools or operators. Atomic OSSEC rules are based on the behavior of systems under active attack. They focus on what happens on a database server when a SQL injection attack occurs, or when an adversary launches a Powershell command against a domain controller.

Once it is determined that unwanted activity is occurring, there are multiple and configurable mechanisms for alerting and notifying so that your team may execute your response.

However, human response times do not often translate into effective containment actions. What if it’s 3 AM? A weekend? A holiday? The day the senior analyst hikes the Appalachian Trail?

This is where Atomic OSSEC active response comes into play. Active response can be enabled for any rule, and the response can be any action or collection of actions you can think of. You can kill a process, change permissions, shun an IP address, log out a user, or all of these at once. You can even isolate a computer from the rest of the network. All of this can happen even before you get the notification that something has occurred. The sky’s the limit.

Adversaries will almost always have the upper hand. Well-resourced adversaries can, and WILL, penetrate even the best-defended networks given sufficient motivation. But penetrating a network is not the same as extracting value from a network. Embracing a defense-in-depth detection philosophy (ubiquitous, diverse, and meaningful) will give you instant notice of adversary activity on your network, provided you can effectively manage and analyze your logs.

Atomic OSSEC can help. It can detect (and stop) malicious activity based on preset and ML-generated rules. It also enables you to analyze logs, leverage global threat intelligence, and drill into data and patterns for more strategic security as part of DevSecOps, where security is not an afterthought in business computing. With our lightweight and smart agents, our deep detective ruleset, and our active response capabilities, we can help you easily institute and embrace that defense-in-depth philosophy without overburdening your assets or your staff.  

Learn more about Atomic OSSEC defense-in-depth intrusion detection functionality.

Get a demo.

Visit my previous article, “How to Defend Against Lateral Movement in Windows With OSSEC,” if you missed it.