Beyond signatures: Modernizing SecOps
Using AI behavioral detection to outpace advanced attackers
On their own, conventional cybersecurity models are no match for the volume, variety, and speed of threats facing organizations today. Security teams that rely on legacy, reactive models will continue to play catch-up against adversaries who are using a growing range of readily available, advanced tools and AI capabilities for launching attacks and evading detection.
Modern security operations teams must operate within a new cybersecurity model that can rapidly detect and respond to an increasing volume of indiscriminate attacks, whilst having the capability to quickly orientate and defend against sophisticated, targeted attacks from concerted threat actors with a defined and well-researched objective.
Organizations can now supplement their existing SecOps processes and tools with behavioral anomaly analysis and detection capabilities that previously were too expensive, time-consuming, and labor-intensive to implement. By identifying activities that deviate from established normal patterns, behavioral analysis — along with contextual orientation techniques — helps organizations surface, respond to, and investigate threats before they can cause damage.
What makes behavioral anomaly detection different?
Behavioral analysis is not new. Implementing point solutions for analyzing single-system behavior and isolated use cases, such as classifying phishing emails, has long been a common and successful cybersecurity strategy. But until recently, the costs and resource demands of implementing behavioral analysis on a large scale have been too great for most organizations.
With increased availability of AI and ML techniques, that barrier has decreased significantly. Organizations can now use AI- and ML-based tools to analyze and classify significant amounts of data — including activity logs, app usage, and network traffic — from various systems and environments. They can establish a baseline of “normal activity” and then continuously monitor new data for deviations from that baseline.
Traditionally, cybersecurity operations teams have relied on signature- and rule-based tools that use previously observed attack patterns. They look for evidence of malicious activity in analytics from live event data and historical log data gathered during post-breach investigations. But the complex multi-vector nature of modern attacks and the operating procedures of sophisticated adversaries render static detection and mitigation techniques almost obsolete.
Behavioral analysis is an important alternative to traditional approaches. Deployed alongside established capabilities, including security and information management (SIEM) and extended detection and response (XDR), this approach enables efficient, expedient detention and mitigation of both modern and legacy threats to enterprise infrastructure and corporate assets.
Behavioral analysis works by identifying new and unusual patterns in network, system, agent, and user behavior. It doesn’t rely on previously observed and recorded patterns of behavior or indicators of attack. To put it another way, behavioral detection is designed to detect new and novel threats instead of attack artifacts — which helps security teams pivot from responding to the aftermath of cyberattacks to preventing and pre-empting them in the first place.
Signature-based methods vs. behavioral analysis
Signature- or rule-based tools …
Look for known threats (patterns, malicious IPs / domains, malware behaviors)
Are reactive in nature (they primarily identify attacks after compromise)
Can miss insider threats, zero-day attacks, and other modern threats
Behavioral analysis …
Looks for anomalies and exceptions in typical behavior (abnormal system, agent, user, and network activity)
Is designed to be proactive (it detects possible attacks before or soon after they occur)
Is affordable, accessible, and scalable thanks to advances in AI
Advantages over other approaches to behavioral detection
Organizations that embrace behavioral analysis as a core security strategy enjoy these three advantages over previous approaches:
1. Augment traditional signature-based detection.
Signature- and pattern-based observability approaches are reactive. Organizations that only leverage these techniques are limited to identifying attack profiles after they have been previously discovered and analyzed by others in the form of packaged signatures or indicators of compromise (IOCs).
By contrast, behavioral analysis is proactive. It looks for unusual activity in its early stages, finding indicators of attack (IOAs), such as phishing attempts, suspicious network activity, or uncharacteristic user interactions with systems and services. This early, anomaly-based detection allows cybersecurity teams to get ahead of adversaries and prevent attempted compromises before they cause harm.
2. Gain a holistic and affordable defense ecosystem.
Traditional tools focus on a narrow aperture — such as the system, user, or network — and do not incorporate context. Behavioral analysis, on the other hand, integrates data from multiple sources to provide context-rich analysis. Internal and external context information can help ensure that not all unusual behavior is flagged as malicious. Behavioral analysis tools can determine within specified degrees of likelihood whether an activity is malicious or benign once a model has been trained on a system or network.
Historically, this type of context-rich analysis has been expensive to deploy. It required significant computing power and labor-intensive workflows for storing, processing, and interrogating large volumes of disparate datasets. Advances in AI-enabled data engineering and event correlation have made context-rich analysis much more affordable than previous approaches in human-driven analytics.
3. Enhance adaptability to modern threats and workflows.
Behavioral analysis models learn continuously and adapt to new threats — and new workflows. For example, let’s say you implement an AI agent. Over time, the model will learn the autonomous behaviors of that agent and define them as normal. If the AI agent is compromised or deviates from established patterns of permitted activity, the model will detect this anomalous behavior and, if appropriate, suspend the agent or limit the actions it can perform without human intervention.
Behavioral detection also accounts for new ways of working. For example, you might expand your business globally and set up new branch offices in new countries. After an initial period of tuning, the model will learn that remote logins from those locations at certain times of day are not fraudulent in nature, but reflective of a new, normal pattern.
Behavioral anomaly analysis strategies
Many organizations already have some behavioral analysis capabilities embedded in endpoint, security, or identity platforms. But not all teams understand how to make the most of these distributed capabilities, and security fatigue can lead to lapses. The following best practices can help your team implement and use behavioral detection effectively.
1. Define scope and integration.
Whether you plan to gain greater value from existing capabilities and the data generated by your environments or are considering a new solution, identify potential use cases first. For example, you might want to focus on identifying potential insider threats, discovering lateral movement or data exfiltration attempts. You could even define your use cases more narrowly. For instance, you might want to detect “impossible travel,” when a user seems to log into the network from two distant locations within an unreasonably short timeframe. Or you might want to stop repeated data loss prevention (DLP) violations, when employees move or share confidential information in ways that breach internal security policies.
2. Train and optimize models.
Behavioral analysis can be automated and orchestrated over multiple systems and security enforcement points. Make it a priority to train and optimize the AI models and integrations that enable this automation. Initial training helps the model understand what “normal” looks like. Continuous tuning and retraining helps you keep pace with evolving threat behaviors.
You could assist and accelerate the training process by intentionally seeding malicious attack patterns instead of waiting for malicious behavior to take place. Over time, models will rely on learned behaviors and assign confidence scores for their predictions of malicious versus benign anomalies.
3. Leverage AI for advanced defense.
Historically, behavioral analysis has relied on various ML algorithms to assess and surface anomalies. Employing fine-tuned, locally augmented generative AI (GenAI) models and agentic AI can help further streamline the detection of anomalies within a protected environment. With these more advanced capabilities, behavioral analysis solutions reduce the load on human investigators, cross-reference anomalies against broader knowledge bases, and automate next-step defenses, including the ability to apply upstream control changes, quarantine hosts, and suspend user accounts.
4. Pay attention to interoperability.
Choose a behavioral analysis solution that will integrate with your existing tools. Otherwise, you’re likely to spend excessive time and money to rip and replace solutions you already have. For example, a behavioral analysis solution should integrate with — and complement — existing detection and response capabilities, including SIEM tools. Traditional SIEM tools analyze events in real time, correlating event data from multiple sources to determine and surface predetermined attack patterns, whereas modern SIEM systems include aspects of behavioral analysis to assess user and device behavior over time, continuously refining an internal reference model of IOAs.
5. Anticipate false positives.
Not every anomaly is malicious. But as with many AI-driven approaches, there is the risk of both false positives and false negatives. Minimizing wrongly classified behaviors might require a form of human intervention, including in-the-loop collaboration. In particular, you’ll need to tune models to reduce noise and enhance behavior classification. You can also implement intelligent correlation and enrichment sources specific to the protected environment and operating model of your business to add additional context to the discovery of malicious and out-of-character behavior.
Also keep in mind that models need time to stabilize and adapt, and require human oversight for nuanced cases. Success hinges not only on technology, but also on skilled teams operating with well-defined processes to interpret and act on detection insights.
6. Define metrics for success.
Establish key performance indicators (KPIs) to measure the value of behavioral analysis investments over time. For example, you could measure the time to detect initial access attempts and the prediction of optimal attack intervention techniques. Additional KPIs might measure the resulting reduction of dwell time for attackers or assess the potential impact of a mitigated attack if it had been successful.
Are you using the right anomaly detection strategy?
Traditional signature- and rule-based defenses alone are insufficient for defending against rapidly evolving security threats and determined adversaries. Incorporating behavioral anomaly detection into your security operations strategy can help you establish a more proactive approach to holistic cyber defences by spotting potential threats before they do damage.
Cloudflare uses various ML and AI techniques to operate a global connectivity cloud and deliver leading-edge services to organizations. User and entity behavior analytics (UEBA) is part of the Unified Risk Posture offering — a suite of capabilities that brings together security service edge (SSE), web application, and API security solutions in a single platform. Embedding behavioral detection and risk scoring capabilities within an SSE or secure access service edge (SASE) platform enables you to capitalize on their benefits while reducing management complexity and improving operational efficiency.
This article is part of a series on the latest trends and topics impacting today’s technology decision-makers.
Dive deeper into this topic.
Discover strategies for staying ahead of the latest threats and safeguarding AI implementations in the Modernizing security for the AI era ebook.
Author
James Todd — @jamesctodd
Field CTO, Cloudflare
Key takeaways
After reading this article, you will be able to understand:
Why traditional, reactive cybersecurity tools are falling short
How behavioral analysis augments signature-based detection
6 best practices for implementing behavioral analysis