When workloads moved to the cloud, a huge burden was lifted from the enterprise in infrastructure and operational overhead. This transition also brought with it the “shared responsibility” model, where cloud providers took on much of the responsibility previously relegated to expensive engineering teams.
The shift of getting resources “aaS”, or as a service, shows no signs of slowing. But what does this blurred perimeter and shared responsibility do to previously demarcation lines of security and boundary? It has added a new challenge to an increasingly complex world of interconnected systems. Security teams are struggling to adapt to the new complex attack surface and in some cases, customers are flying blind without the visibility they need to secure their digital assets.
When things get “cloudy”, you need a strategy to proactively look for signs of malicious activities or indicators of compromise (IOCs) before they gain a deep foothold within your organization's environment. And although the practice of threat hunting is not new, the methodologies are dramatically different than they were even just a few years ago. After we cover some basics, let’s review some recent headline-making security incidents and how modern threat hunting is a necessary component to any mature security program.
Active pursuit vs. passive detection: The bear and the snare
Threat hunting is a term that is sometimes overused, so a simple analogy might help us start the discussion. Imagine setting up a base camp somewhere in the wilderness. Within this camp, you have the critical assets needed to brave the elements and survive. What you have is also valuable for animals that may be near your perimeter.
Bears, tigers, wolves… other people, all pose different risks and might employ different adversarial tactics or techniques. To protect yourself and your assets you can employ two strategies, a defensive, detection-based strategy that will alert you if something is, say entering your tent or rummaging through your food stores. Taken further, maybe a monitored fence surrounding your camp might be appropriate. These detection countermeasures would be a great first step. Once that is complete, it makes sense to understand your adversary, and even anticipate and withstand threats before any damage occurs.
This mindset can help build a modern and effective security program that employs both proactive and reactive strategies. Having advised hundreds of organizations of all levels of maturity, I find it effective to use a crawl-walk-run approach to ensure fundamentals are covered before diving into perhaps more exciting areas. This “cybersecurity first principles” approach is proven, and helps frame risk, response and what security is doing to help, not hinder business. In fact, there is a great book I recommend everyone in the space read with the same title. In rough order,
Ensure your architecture is built on modern security practices such as zero trust and least privilege.
Employ modern security point solutions to create a layered defense-in-depth approach for identifying vulnerabilities, and actively protecting your data and systems.
Aggregate telemetry across your systems, services and security solutions to provide visibility and cross-correlation of activity with security information and event management (SIEM)
Set up a threat-hunting motion to look for indicators of compromise when your defenses fail, or as we will discuss, when the defenses of your software suppliers and vendors fail.
Hunting hypotheses
Focusing on step four in this article, let's return to our base camp scenario. If you were to leave camp to actively scout and learn what threats might surround you, how would you go about it? What “indicators” would you use to identify adversaries? Footprints, broken branches, disturbed soil, unusual sounds? Think of these as your start hypotheses.
A hypothesis provides a clear starting point, guiding threat hunters to look for specific indicators or behaviors that align with the hypothesis. Without it, the threat-hunting process can become aimless, wasting valuable time and resources. By formulating a hypothesis, threat hunters can narrow their focus, making their investigation more efficient. It helps them avoid getting lost in the vast amount of data and instead concentrate on the most relevant areas.
A hypothesis-driven approach encourages a systematic method of investigation. Threat hunters follow a logical sequence—starting with the hypothesis, gathering evidence, analyzing findings, and concluding. This structured process increases the chances of detecting real threats. Remember, you may not know what’s lurking out in the woods or darkness. If you already knew that, it would be easy to set up a static detection strategy.
True threat hunting isn't just about finding what's already known; it's about discovering new, hidden, or emerging threats.
Threat hunting in a SaaSy world
Perhaps where this analogy breaks down is, we don’t have a clearly defined perimeter anymore. In fact, our “base camp” more likely resembles a scene out of Mad Max or a campsite at Burning Man. Our supply chain and software interdependencies are so complex it can make your head spin. Much of the software we bring into the deepest parts of our environment is open-source, built by developers we’ve never met in countries we’ve never been to. How is that secure? I dig deeper into that topic in The ultimate race condition: Securing open source infrastructure. But it’s not just an open-source problem. As we’ve seen this year, even the titans like Microsoft have fallen, leaving their customers completely vulnerable.
As described in the CISA report, “In May and June 2023, a threat actor compromised the Microsoft Exchange Online mailboxes of 22 organizations and over 500 individuals around the world. The actor—known as Storm-0558 and assessed to be affiliated with the People’s Republic of China in pursuit of espionage objectives—accessed the accounts using authentication tokens that were signed by a key Microsoft had created in 2016. This intrusion compromised senior United States government representatives working on national security matters.”
In short, the attack exploited a vulnerability related to a cryptographic key mistakenly included in Microsoft's systems. This allowed the attackers to generate signed access tokens, which enabled access to email accounts and other sensitive customer information.
I bring this up because this is an attack vector that not many expected. And it would be very difficult to have a rule in place to cover every possible scenario, including the compromise of systems that are assumed secure like backend email and cloud communication systems. After all, isn’t the point of the “shared responsibility” such that you don’t have to lie awake at night worried that AWS, Google or Microsoft are ensuring their systems are safe?
And this supply-side risk isn’t an outlier. Consider the MOVEit Transfer software issue where a managed file transfer tool was exploited by a ransomware group. They took advantage of a zero-day vulnerability, allowing them to access and exfiltrate sensitive data from numerous organizations worldwide. Okta, the identity provider faced several security incidents, including phishing attacks, a breach, and the theft of its GitHub source code. Then their support portal was found to have issues as well. All these were linked to attacks on high-profile customer environments and downstream third-party vendors like BeyondTrust, Cloudflare, and 1Password.
Let’s not stop there. The well-known SolarWinds incident was a sophisticated supply chain attack in which hackers, believed to be associated with a state-sponsored Russian group, compromised SolarWinds' Orion software. The attackers inserted malicious code into a routine software update, which was then distributed to thousands of SolarWinds customers. This allowed the attackers to gain unauthorized access to the networks of numerous organizations, including U.S. government agencies, Fortune 500 companies, and other high-profile entities. The breach went undetected for several months.
All of these highlight vulnerabilities in supply chain security and the need for enhanced monitoring and a threat-hunting environment. No matter how good you think you are, you may be vulnerable for reasons outside of your control. Specifically what role do threat hunters play?
Proactive detection of anomalous activity: Threat hunting involves actively looking for signs of compromise that may not be detected by standard security tools. In the case of supply chain attacks, threat hunters can identify unusual behaviors or patterns in network traffic, such as unexpected outbound connections, data exfiltration attempts, or irregular communication with known legitimate software. By continuously hunting for threats, organizations can detect attacks earlier in the kill chain, allowing for a quicker response to mitigate damage. In the case of SolarWinds, Microsoft or Okta, earlier detection could have limited the attackers' access and prevented them from reaching critical systems.
Improved detections: When indicators of compromise are discovered, threat-hunting teams feed these findings back to the detection team. This changes the threat from “unknown” status, to “known” status. This results in a net increase in visibility into all aspects of the network, including endpoints, cloud environments, and communications. They can then develop new hypotheses based on the tactics, techniques, and procedures (TTPs) used by attackers, continuously adapting to evolving threats.
Operationalizing threat intelligence
Key to a successful threat-hunting practice is threat intelligence (TI). TI captures and categorizes these TTPS and enhances the effectiveness and efficiency of your team’s security operations by establishing a proactive, systematic approach for detecting and responding to potential threats. I recently co-hosted a webinar on threat hunting with subject matter experts from our partner, eSentire. Ryan Westman, Sr. Manager of Threat Intelligence, described the process of operationalizing TI as a continuous loop of five discrete steps.
Direction: Conduct research regularly to stay updated on the latest threats, attack methods, and industry trends. This involves understanding the evolving threat landscape, which helps you identify what specific threats are most relevant to your organization. Staying informed enables you to align your threat-hunting activities with the current and emerging risks specific to your industry and environment. Sumo Logic’s Threat Lab Team also conducts this research and continuously releases new detections relevant to new attack vectors.
Collection, organization, and process: Gather threat intelligence data from various sources and organize it effectively. Prioritize these threats based on factors such as their severity, potential impact on the organization, and likelihood of occurrence. This prioritization helps in focusing on the most critical threats that may require immediate attention or rapid response to minimize risk.
Analysis: Perform a thorough analysis to understand how specific threats or vulnerabilities could potentially exploit weaknesses in your systems. Develop detection strategies and techniques that can identify these threats early. This step involves creating detailed profiles of threat actors, understanding their tactics, techniques, and procedures (TTPs), and determining the best way to detect their activities. Ensure that your analytics tools can search back into historical data without rehydration of data as this will quickly become cost prohibitive.
Dissemination and collaboration: Share the results of your threat analysis with the threat-hunting team and other relevant stakeholders. This information should be communicated clearly and effectively to enable hunters to recognize advanced threats more accurately and to apply the intelligence in their day-to-day activities. Collaboration ensures that everyone is working with the same information, which enhances the overall effectiveness of the threat-hunting process. One advantage of the Sumo Logic platform is the unlimited user licenses and the ability to easily create shareable dashboards and reports helping to “democratize” data already ingested.
Continuous feedback and improvement: Implement a feedback loop that includes constant monitoring of detection strategies and the identification of false positives. Use this feedback to refine and update threat intelligence and detection methods continually. By doing so, you increase the precision of your threat-hunting capabilities, reduce false alarms, and improve the ability to recall and identify genuine threats in a timely manner. This ongoing refinement helps to adapt to new threats as they emerge and ensures that your threat-hunting practices remain effective and relevant.
The data lake: A treasure trove for threat hunters
Before you can even begin to consider building a threat-hunting program, you need a centralized repository for hunters to dive into. I’ve heard it referred to as “finding a needle in a needle stack”. And the stack of data is growing exponentially. Without it, you’re a hunter with no tracks to follow, no footprints to examine, and no signs to tell you which way the danger lurks.
Think of a security data lake as the central hub where all the evidence gathers. Every log file, every network flow, every user activity—is aggregated at scale into this data store in the cloud. Centralizing logs into a security data lake enables efficient querying and analysis. Platforms like Sumo Logic provide powerful search capabilities that allow threat hunters to quickly sift through vast amounts of data to find relevant information. This speed and efficiency are crucial when responding to potential threats, as it reduces the time needed to identify and analyze suspicious activity, allowing for faster detection and response.
In some ways, it would be akin to arming our traditional tracker that is used to operating on foot, with a drone equipped with a powerful zoom camera, infrared, microphone, and other useful sensors. (Are you tired of this analogy yet?)
Ultimately, threat hunting is a game of intelligence, and having a centralized data lake is crucial for understanding trends and recognizing deviations. When a subtle, slow-moving threat is making its way through your network, having months or even years’ worth of logs at your fingertips allows you to trace its path and understand its methods.
It’s also a foundational component on which your detections are built because data lake isn’t just a passive storage pool; it’s the power source for automation. With the right integrations, you can set up automated alerts, responses, and even remediation actions based on the data flowing into your lake. It’s like setting up an automated trap that catches the bear the moment it enters your camp.
Strengthen your defenses with feedback loops
We touched on this earlier, but a centralized log analytics platform fosters collaboration among security teams. With a unified view of the data, team members can share insights, cross-check findings, and coordinate their efforts. This shared visibility ensures that everyone is on the same page, reducing the risk of gaps in coverage and enhancing the overall effectiveness of the threat-hunting process. Collaboration is key to a cohesive and efficient response to security incidents.
Centralizing data in a security data lake also sets the stage for automation. With the integration of machine learning and advanced analytics, organizations can automate the detection of known threats and unusual behavior patterns. Automated alerts and responses based on predefined criteria allow security teams to focus on more complex, targeted threat-hunting activities, knowing that common threats are being automatically managed, so you can rest easy in your base camp.
Learn how to go threat hunting from your command line.
Complete visibility for DevSecOps
Reduce downtime and move from reactive to proactive monitoring.
