MACD: Multi-Agent Collaborative Approach for Cybersecurity Defense Strategy Generation

Abstract

Cybersecurity defense strategy generation transforms threat intelligence into actionable defense measures against sophisticated multi-stage cyberattacks. Existing approaches lack multi-dimensional coordination of technical, tactical, and threat actor expertise, with limited benchmarks for evaluating defense strategy quality. To address these gaps, we introduce MACD (Multi-Agent Collaborative Defense), a novel framework that orchestrates specialized AI agents to generate ATT&CK-aligned defense strategies. MACD deploys three expert agents for technical defense, kill chain phase analysis, and APT profiling, coordinated through a synthesizing agent, while leveraging retrieval-augmented generation to mitigate hallucination risks in threat mapping. Additionally, we construct CyberDefBench, a comprehensive benchmark combining real-world APT cases and synthetic scenarios with dual-layer annotations for reactive and proactive defenses. Experimental results demonstrate that MACD achieves 84.6% technique mapping accuracy and 72.3% defense coverage, significantly outperforming baseline methods and validating the effectiveness of multi-agent collaboration for cybersecurity defense.

1. Introduction

The rapid evolution of cyber threats has reshaped the information security landscape, presenting unprecedented challenges for organizations worldwide. Modern cyberattacks have grown increasingly sophisticated, with Advanced Persistent Threats (APTs) demonstrating complex multi-stage tactics that evade traditional security measures [1,2]. These sophisticated attacks impose severe consequences, with the average data breach cost exceeding $4.45 million and the time to identify and contain breaches extending beyond 200 days [3,4]. This escalating threat environment necessitates a paradigm shift from reactive security measures to proactive, intelligence-driven defense strategies. Effective defense strategy generation requires deep technical expertise across multiple security domains, rapid analysis of evolving threat patterns, and synthesis of comprehensive countermeasures, yet traditional manual approaches face significant scalability bottlenecks as threat volumes increase exponentially [5]. Thus, leveraging artificial intelligence to augment human cybersecurity expertise and enable automated generation of adaptive defense strategies has become both an urgent need and a research priority in this field [6,7].

Existing approaches to cybersecurity defense strategy generation have evolved from rule-based systems to intelligent reasoning frameworks, broadly categorized into four main paradigms as shown in

Table 1. Comparison of cybersecurity defense strategy generation methods.

Method	Overview	Advantages	Disadvantages
Rule-Based	predefined rules and signatures for known attack patterns	Interpretable; easy to deploy; quick response	Limited to known attacks; manual updates needed
ML/DL-Based	Train models on datasets for automatic threat detection	Handles large-scale data; adapts to new threats	Needs labeled data; adversarial vulnerability
Reinforcement Learning	RL agents learn optimal defense in simulated environments	Dynamic strategy generation; models interactions	Long training time; sim-to-real gap
LLM-Based	Large models with RAG for contextual strategy generation	Strong reasoning; fast generation; adaptable	Hallucination risk; trustworthiness issues

. Traditional methods, including rule-based matching, machine learning anomaly detection, and reinforcement learning-based policy optimization, have advanced the field but face fundamental limitations such as an inability to generalize to novel threats, scarce labeled data requirements, and substantial simulation-to-reality gaps [8,9,10]. Most recently, large language model (LLM)-based approaches leverage the powerful reasoning capabilities of foundation models to produce contextual defense strategies, often incorporating Retrieval-Augmented Generation (RAG) with authoritative knowledge bases such as MITRE ATT&CK to map unstructured threat descriptions to standardized defense recommendations [11,12].

Despite these advances, current methods face critical limitations that constrain their real-world effectiveness. LLM-based approaches, while demonstrating strong reasoning capabilities, suffer from hallucination issues that generate plausible but incorrect strategies, as single-agent systems struggle to maintain sufficient knowledge depth across all required security dimensions simultaneously. Comprehensive cybersecurity defense requires expertise spanning technical vulnerability mitigation, attack progression analysis through kill chain phases, and threat actor behavioral profiling, yet single agents inevitably face knowledge depth insufficiency or perspective bias when attempting to cover all domains. More critically, existing methods lack structured collaboration mechanisms to systematically integrate these diverse security perspectives into coherent strategies. These limitations indicate the need for multi-agent architectures that balance specialized knowledge depth with systematic collaboration mechanisms.

To address these limitations, we propose MACD (Multi-Agent Collaborative Defense), a novel framework that orchestrates multiple specialized AI agents for comprehensive defense strategy generation. Our approach recognizes that effective defense requires expertise across multiple dimensions and achieves this through a multi-agent architecture that decomposes the complex defense generation task into specialized subtasks, enabling each agent to develop deep expertise while a coordinator agent integrates knowledge across dimensions to ensure strategic consistency. MACD operates through a four-phase process that systematically transforms threat queries into actionable strategies: Phases 1–2 preprocess threat intelligence and leverage RAG technology to accurately map natural language descriptions to ATT&CK techniques while mitigating hallucination risks. Phase 3 constitutes the core innovation, deploying three specialized agents (Technical Defense Expert, Phase Defense Expert, and APT Defense Expert) that generate dimension-specific strategies in parallel, with a Coordinator Agent synthesizing these perspectives through deduplication, conflict resolution, and prioritization into a unified, ATT&CK-aligned defense plan. Phase 4 validates feasibility and optimizes effectiveness. This architecture achieves knowledge depth through specialization, efficiency through parallelization, and coherence through intelligent coordination.

In this paper, we make several key contributions:

• We construct CyberDefBench, a comprehensive benchmark combining real-world APT cases and synthetic scenarios with standardized evaluation metrics for assessing defense strategy quality.
• We propose MACD (Multi-Agent Collaborative Defense), a novel multi-agent framework that orchestrates three specialized AI agents coordinated through a Coordinator Agent to systematically generate comprehensive, ATT&CK-aligned cybersecurity defense strategies.
• Through extensive experiments on CyberDefBench, we demonstrate that MACD significantly outperforms baseline methods across multiple metrics, validating practical applicability and interpretability.

2. Related Work

2.1. Cybersecurity Defense Strategy Generation

The growing complexity of cyber threats has driven the evolution of defense strategy generation from traditional rule-based approaches to intelligent automated systems. Early methods relied on expert-defined rules and pattern matching for security policy generation [13], but these approaches struggled with configuration errors, limited scalability, and inability to adapt to emerging threats. To overcome these problems, machine learning and deep learning techniques have been widely used in attack detection systems, using methods such as K-Nearest Neighbors, decision trees, and deep structures like BiLSTM and CNNs to achieve better detection accuracy across different attack types [9,14]. Knowledge graph-based methods have further improved threat intelligence by enabling logical analysis and automated defense strategy prediction through link prediction [15]. More recently, reinforcement learning frameworks have emerged as powerful tools for autonomous defense, with hierarchical multi-agent architectures demonstrating remarkable effectiveness in coordinating defensive actions while maintaining operational stability [16,17]. The latest advancement integrates large language models into security automation, enabling systems to transform unstructured threat intelligence into prioritized assessments and generate context-aware mitigation strategies [18,19], bridging the gap between automated threat analysis and human-interpretable decision support.

Recent advancements in large language models have shown promising results in cybersecurity defense strategy generation, demonstrating strong reasoning capabilities and rapid response to emerging threats. However, LLM-based approaches face critical challenges, including hallucination—generating plausible but incorrect strategies—and context forgetting in long conversations, which undermine the trustworthiness and reliability of generated defense recommendations. To address these limitations, this paper focuses on developing a multi-agent collaborative framework that mitigates individual LLM weaknesses through specialized agent coordination and systematic validation mechanisms.

2.2. Multi-Agent AI Systems

The emergence of large language models has catalyzed significant advances in autonomous agent research, with LLMs demonstrating human-level intelligence through vast web knowledge acquisition [20,21]. Single-agent systems have evolved from basic instruction-following to sophisticated reasoning and action capabilities, with frameworks like ReAct enabling agents to synergize reasoning traces with task-specific actions through interleaving thought processes and environmental interactions [22]. Further enhancements have focused on empowering agents with action learning abilities, allowing them to expand action spaces and develop skills through experiential learning rather than operating within fixed constraints [23,24]. However, single-agent architectures face inherent limitations when addressing complex, multifaceted problems requiring diverse expertise across multiple domains. This recognition has driven the development of multi-agent AI systems, where networks of autonomous agents collaborate through decentralized coordination to tackle tasks beyond individual agent capabilities [20]. Multi-agent frameworks have demonstrated substantial value across various domains, with applications in healthcare showing improved diagnostic accuracy, treatment planning, and interdepartmental coordination through specialized agent collaboration [25].

In cybersecurity contexts, threats exhibit multidimensional characteristics spanning technical vulnerabilities, attack progression through kill chain phases, and adversarial behavioral patterns. Single-agent approaches struggle to simultaneously maintain comprehensive expertise across these diverse security dimensions while ensuring coherent strategy integration. To address these limitations, this paper proposes a multi-agent collaborative framework that orchestrates specialized agents for technical defense, phase-based analysis, and threat actor profiling to generate unified and validated cybersecurity defense strategies. This work represents a principled adaptation of multi-agent paradigms to the cybersecurity domain, where agent roles are explicitly defined by the inherent layered structure of the ATT&CK framework rather than generic task decomposition.

3. Problem Formulation

Cybersecurity defense strategy generation transforms threat intelligence into comprehensive and actionable defense measures through automated analysis and synthesis. We formalize this task by defining $\mathcal{Q}$ as the space of threat queries, where each query $q\in \mathcal{Q}$ represents a textual description of observed threats, suspicious activities, or attack indicators reported by security analysts. The output space $\mathcal{S}$ comprises defense strategies, where each strategy $s\in \mathcal{S}$ is structured as a collection of defense recommendations:

$$s=\{d_1,d_2,\dots ,d_n\},$$

(1)

with each defense item $d_i$ containing the measure description, applicable threat techniques, technical implementation details, deployment prerequisites, and operational guidance. The core objective is learning a mapping function $f:\mathcal{Q}\to \mathcal{S}$ that generates comprehensive defense strategies from threat descriptions:

$$s=f\left(q\right),q\in \mathcal{Q},s\in \mathcal{S}.$$

(2)

Realizing this mapping function f faces two fundamental challenges that collectively determine the effectiveness of defense strategy generation systems.

• Challenge 1: Accurate Threat Intelligence Understanding. The first challenge is accurately identifying the underlying attack techniques or tactics from diverse natural language threat descriptions. Threat queries q may manifest in various forms, including concise and unambiguous technical indicators, ambiguous descriptions that confuse multiple similar techniques, noisy reports embedding real threats within extensive irrelevant information, or false positive scenarios where legitimate business activities superficially resemble attack patterns. Systems must accurately map these unstructured descriptions to specific techniques $t\in \mathcal{T}$ within established frameworks such as MITRE ATT&CK, which requires deep understanding of threat characteristics and precise knowledge of framework taxonomies.
• Challenge 2: Strategic Defense Generation for Attack Disruption. The second challenge is generating strategic defense measures that effectively disrupt attack progression. Effective defense strategies should not only provide reactive measures against currently observed attack techniques but also proactively anticipate and prevent subsequent steps in the attack chain, thereby disrupting attack progression before adversaries achieve their objectives. Formally, given an attack chain $\mathcal{C}=\langle t_1,\dots ,t_k,t_{k+1},\dots ,t_n\rangle$ where technique $t_k$ is observed, the generated strategy s should include both current-stage defenses ${\mathcal{D}}_{t_k}^{\mathrm{current}}$ and next-stage defenses ${\mathcal{D}}_{t_{k+1}}^{\mathrm{next}}$, i.e., $s={\mathcal{D}}_{t_k}^{\mathrm{current}}\cup {\mathcal{D}}_{t_{k+1}}^{\mathrm{next}}$, ensuring the strategy both mitigates identified threats and preemptively blocks subsequent attack chain evolution.

4. Methodology

4.1. MACD Framework Overview

The MACD framework addresses the multi-dimensional nature of cybersecurity threats through a systematic pipeline that transforms unstructured threat descriptions into comprehensive, actionable defense strategies. As illustrated in Figure 1, the framework operates through four sequential phases, each designed to handle specific aspects of the defense generation process. The first phase establishes a structured knowledge base by extracting and vectorizing MITRE ATT&CK techniques, mitigations, kill chain phases, and APT profiles, creating the semantic search infrastructure for subsequent threat analysis. The second phase bridges the gap between informal threat queries and formal ATT&CK specifications through a RAG-enhanced process that combines LLM-based query refinement, vector-based candidate retrieval, and confidence-scored technique confirmation to ensure accurate threat-to-technique mapping while mitigating hallucination risks. The third phase constitutes the core innovation of our approach, deploying three specialized expert agents (each instantiated from the same LLM without fine-tuning, with specialization achieved through role-specific prompts rather than training) that independently generate dimension-specific defense strategies from technical, kill chain, and threat actor perspectives in parallel, with a Coordinator Agent then consolidating their outputs through post hoc integration without exercising supervisory control over the expert agents to ensure strategic consistency. The fourth phase validates generated strategies against feasibility constraints through a dedicated LLM-based reasoning step, and ranks recommendations by effectiveness using a multi-criteria scoring function. This modular architecture enables systematic decomposition of expertise, parallel reasoning across defensive dimensions, and explicit coordination for consistency. The entire framework operates as a stateless inference-time system with no continuous learning loop, while the RAG-enhanced mapping ensures that all generated strategies remain grounded in authoritative ATT&CK framework standards rather than relying solely on LLM parametric knowledge.

4.2. Data Preprocessing

Data preprocessing establishes the ATT&CK knowledge foundation by extracting, structuring, and indexing information from the MITRE ATT&CK framework. We download the latest ATT&CK datasets covering enterprise, mobile, and ICS domains, parsing attack techniques ($\mathcal{T}$), mitigations ($\mathcal{M}$), kill chain phases ($\mathcal{K}$), and APT group profiles ($\mathcal{A}$) along with their inter-relationships. Each technique $t_i\in \mathcal{T}$ is represented as $t_i=\{\mathrm{id},\mathrm{name},\mathrm{description},\mathrm{tactics},\mathrm{platforms}\}$. We construct bidirectional mappings between techniques and mitigations, enabling rapid lookup during strategy generation. To enable semantic search, we employ a sentence transformer model (all-MiniLM-L6-v2) to generate dense vector embeddings for all techniques and mitigations. Specifically, we encode the concatenated textual description of each technique into a vector representation ${\mathbf{v}}_t\in {\mathbb{R}}^d$ where $d=384$. These vector embeddings, structured metadata, and relationship mappings are stored in an indexed database to support efficient retrieval during real-time threat analysis.

4.3. Threat Understanding & Mapping

Threat understanding and mapping connects natural language threat descriptions to formal MITRE ATT&CK techniques through a three-step RAG-enhanced process. Given a threat query $q\in \mathcal{Q}$, the system first performs query rewriting using an LLM to extract key cybersecurity concepts and transform the query into ATT&CK-aligned search terms. This produces domain-specific keywords and a rewritten query $q^{\prime }={\mathrm{LLM}}_{\mathrm{rewrite}}\left(q\right)$ optimized for semantic retrieval. For example, when a security analyst reports “suspicious PowerShell activity executing encoded commands,” the system reformulates this into technical terms matching ATT&CK technique descriptions. This rewriting step enables the system to handle diverse ways practitioners describe threats in natural language, ensuring robust retrieval even when queries use colloquial security terminology.

The second step performs knowledge retrieval using pre-computed technique embeddings. We compute cosine similarity between the query embedding ${\mathbf{v}}_{q^{\prime }}$ and all technique vectors $\{{\mathbf{v}}_{t_1},\dots ,{\mathbf{v}}_{t_n}\}$ to identify the top-K semantically similar candidates: $C=\mathrm{TopK}({\left\{\langle {\mathbf{v}}_{q^{\prime }},{\mathbf{v}}_{t_i}\rangle \right\}}_{i=1}^n,K)$. To prevent incorrect mappings caused by semantic ambiguity or insufficient context, we implement an LLM-based confirmation step. Each candidate technique is independently evaluated by prompting the LLM to assess match quality on a 0–10 scale, identify specific matching characteristics, and provide justification. The technique with the highest confirmation score above a threshold (typically 6) is selected as the confirmed mapping $t^{*}={argmax}_{t\in C}{\mathrm{LLM}}_{\mathrm{score}}(q,t)$. This RAG-enhanced approach combines semantic similarity with reasoning-based validation, significantly reducing hallucination risks compared to pure LLM-based mapping while maintaining flexibility to handle diverse threat descriptions.

4.4. Multi-Agent Strategy Generation

The core innovation of MACD lies in its multi-agent collaborative strategy generation mechanism, which addresses the multi-dimensional nature of cybersecurity defense through specialized agents that independently reason about complementary aspects of threat mitigation. Once a threat query q is mapped to a confirmed technique $t^{*}\in \mathcal{T}$, we deploy three expert agents—each embodying distinct defensive perspectives—to generate dimension-specific strategies in parallel: Agent 1 (Technical Defense Expert) focuses on technique-specific countermeasures by analyzing the technical characteristics of $t^{*}$ and retrieving associated MITRE mitigations $M_{t^{*}}=\{m\in \mathcal{M}\mid (t^{*},m)\in \mathrm{Relationships}\}$, generating strategies $s_{\mathrm{tech}}$ covering detection methods, prevention controls, configuration hardening, and monitoring approaches tailored to the specific attack vector. Agent 2 (Phase Defense Expert) adopts a kill chain perspective by retrieving the primary tactic $\tau \in t^{*}.\mathrm{tactics}$ and corresponding kill chain phase information $k_{\tau }\in \mathcal{K}$, generating layered defense strategies $s_{\mathrm{phase}}$ that address early detection, phase transition prevention, containment, and recovery, explicitly considering how the attack fits into the broader adversarial campaign lifecycle. Agent 3 (APT Defense Expert) takes a threat actor-centric view by identifying APT groups $A_{t^{*}}=\{a\in \mathcal{A}\mid t^{*}\in a.\mathrm{techniques}\}$ that employ technique $t^{*}$, generating strategies $s_{\mathrm{apt}}$ focused on behavioral detection patterns, proactive threat hunting procedures, deception techniques, and intelligence-driven defenses informed by APT tactics, techniques, and procedures (TTPs).

Each agent operates independently using carefully designed prompts (see Appendix A) that provide the original threat query q, the mapped technique details, and dimension-specific context (mitigations for Agent 1, kill chain phase for Agent 2, APT profiles for Agent 3), ensuring that generated strategies remain grounded in both the specific threat scenario and the broader ATT&CK knowledge base. The prompts explicitly instruct agents to contextualize their recommendations to the particular attack scenario rather than producing generic defenses, as illustrated by the directive: “Your strategies should be contextualized to the actual threat scenario, not just generic defenses for the technique.” This contextualization is crucial for generating actionable strategies that address the specific characteristics of the observed threat, such as the particular attack vector, target environment, and operational constraints mentioned in the threat query.

The independent reasoning of specialized agents offers several advantages: (1) depth of expertise, as each agent can focus exclusively on its dimension without cognitive overload from managing all perspectives simultaneously; (2) parallel generation efficiency, enabling faster overall strategy production; and (3) diversity of coverage, reducing the risk that critical defensive angles are overlooked due to perspective bias. However, this parallelism introduces the challenge of ensuring consistency and coherence across independently generated strategies, which we address through Agent 4 (Coordinator Agent).

Agent 4 serves as an intelligent integrator that synthesizes the outputs $\{s_{\mathrm{tech}},s_{\mathrm{phase}},s_{\mathrm{apt}}\}$ from the three expert agents into a unified, coherent defense plan $s_{\mathrm{final}}$. The coordinator performs four critical functions: (1) deduplication and conflict resolution, identifying redundant recommendations across agents and resolving logical inconsistencies; (2) prioritization, ranking strategies by effectiveness for the specific threat scenario using criteria such as immediacy of impact, implementation complexity, and expected risk reduction; (3) contextualization verification, ensuring all recommended strategies remain relevant to the original threat query q and are not generic ATT&CK guidance disconnected from the scenario; and (4) gap analysis and enhancement, identifying missing defensive controls not covered by the expert agents and augmenting the strategy accordingly. The coordinator is prompted with all three expert outputs and the original threat context, generating a structured defense plan organized into categories (immediate actions, detection strategies, prevention controls, monitoring and alerting, threat hunting, response and recovery) with explicit priority levels and source attributions. This integration step transforms independently generated defensive perspectives into a comprehensive, operationalized defense strategy that security teams can directly implement.

4.5. Strategy Validation & Optimization

The strategy validation phase ensures that the coordinator-generated defense plan $s_{\mathrm{final}}$ is practically deployable before optimization. We first conduct feasibility checks to verify that each defense measure $d\in s_{\mathrm{final}}$ can be deployed in the target environment by evaluating four dimensions: (1) technical compatibility, which assesses whether the measure is applicable to the target operating systems, platforms, and existing security tooling identified in the threat query context; (2) resource requirements, including estimated personnel effort, required tooling, and acceptable deployment time windows; (3) operational impact, evaluating whether deployment would disrupt business continuity or introduce new attack surfaces; and (4) prerequisite dependencies, verifying that required configurations or upstream controls are already in place. Measures failing these checks are either removed or flagged for manual assessment. This feasibility validation is implemented as an LLM-based reasoning step, where the model is prompted with both the defense measure details and the environmental context extracted from the original threat query, ensuring that validation judgments remain grounded in the specific deployment scenario rather than generic applicability assumptions.

The optimization phase ranks validated strategies using a multi-criteria scoring function that balances threat coverage, implementation urgency, risk reduction potential, and deployment complexity:

$$\mathrm{Score}\left(d\right)=w_1·\mathrm{Coverage}\left(d\right)+w_2·\mathrm{Urgency}\left(d\right)+w_3·\mathrm{Effectiveness}\left(d\right)-w_4·\mathrm{Complexity}\left(d\right),$$

(3)

where $\mathrm{Coverage}\left(d\right)$ measures the breadth of threats addressed, $\mathrm{Urgency}\left(d\right)$ assigns higher scores to measures targeting earlier kill chain phases where intervention cost is lower, $\mathrm{Effectiveness}\left(d\right)$ is assessed based on the historical defensive efficacy ratings of ATT&CK mitigations against their associated techniques, and $\mathrm{Complexity}\left(d\right)$ aggregates prerequisite tools, configuration steps, and required expertise level. The weights $w_1$–$w_4$ default to equal values ($w_i=0.25$) but are adjustable by security teams to reflect organizational priorities, such as elevating $w_2$ during active incident response. The final output organizes strategies into immediate deployment, short-term implementation, and long-term planning priority tiers, with each strategy annotated to identify the originating agent, targeted ATT&CK techniques, and expected security improvement, enabling security teams to make informed decisions and communicate response plans effectively.

5. CyberDefBench Benchmark Construction

5.1. Dataset Construction Methodology

To rigorously evaluate defense strategy generation systems against the two core challenges identified in Section 3—accurate threat intelligence understanding and strategic defense generation—we construct CyberDefBench based on real-world attack chain progressions. We systematically extract 40 attack chains from MITRE ATT&CK Procedure Examples and authoritative threat intelligence reports (Mandiant, CrowdStrike, CISA), covering 12 distinct APT groups across Enterprise, Mobile, and ICS domains, and formalize each as a directed sequence $\mathcal{C}=\langle t_1,t_2,\dots ,t_n\rangle$ where $t_i\in \mathcal{T}$ represents ATT&CK techniques following the temporal progression observed in actual incidents. Our dataset construction comprises two key components: Threat Intelligence Generation simulates diverse threat descriptions to evaluate accurate understanding capabilities, and Defense Strategy Annotation provides dual-layer ground truth defenses to assess strategic generation capabilities.

Threat Intelligence Generation. For each attack chain $\mathcal{C}$, we select an observation point $t_k$ (where $1\le k<n$) representing the currently detected technique, then generate realistic threat scenarios across four difficulty categories that systematically probe threat understanding capabilities. Simple scenarios provide unambiguous indicators directly mapping to $t_k$, establishing baseline accuracy. Confusing scenarios deliberately mix characteristics of $t_k$ with a similar technique $t^{\prime }$ from the same kill chain phase, requiring fine-grained disambiguation. Noisy scenarios embed real threat indicators within 60–70% irrelevant information such as routine IT issues or benign activities, simulating information overload. False-positive scenarios describe legitimate business activities that superficially resemble attack patterns, testing discrimination capability. The core of our scenario generation relies on LLM-based prompt engineering, with category-specific prompt templates designed to ensure systematic and reproducible scenario generation across all difficulty levels, as summarized in

Table 2. Core prompt directives for four-category scenario generation.

Category	Concise Prompt Template
Simple	“Given the following ATT&CK technique reference: [ID, name, and description of $t_k$], generate a concise threat report containing clear and unambiguous observable indicators of this technique, ensuring the described behaviors directly and exclusively correspond to the ATT&CK technique definition.”
Confusing	“Given the following ATT&CK technique reference: [ID, name, and description of $t_k$] and [ID, name, and description of $t^{\prime }$], generate a threat report that intermixes their observable behaviors without explicitly naming either technique, such that determining the primary exploitation method requires fine-grained disambiguation.”
Noisy	“Given the following ATT&CK technique reference: [ID, name, and description of $t_k$], generate a threat report where indicators of this technique are embedded within extensive irrelevant contextual information (e.g., routine IT operations, benign system events, help desk tickets), ensuring the real threat indicators constitute no more than 30–40% of the overall content.”
False-positive	“Given the following ATT&CK technique reference: [ID, name, and description of $t_k$], generate a report describing a legitimate IT or business operation conducted by authorized personnel that produces observable behaviors superficially resembling this technique, but originates entirely from sanctioned activities with no malicious intent or unauthorized access involved.”

. For instance, to generate Noisy scenarios, we first identify the target technique $t_k$ and extract its formal ATT&CK description, then construct prompts instructing the LLM to embed technique-specific indicators within distractor information, such as “Generate a threat report where PowerShell execution indicators are mixed with routine IT troubleshooting logs, network maintenance activities, and user software installation events, ensuring the real threat constitutes only a small portion of the content.” This prompt-driven approach enables systematic control over scenario difficulty while maintaining a realistic operational context.

Defense Strategy Annotation. For each scenario anchored at observation point $t_k$ within attack chain $\mathcal{C}=\langle t_1,\dots ,t_k,t_{k+1},\dots ,t_n\rangle$, we construct dual-layer ground truth that captures both reactive and proactive defense requirements. Current-Stage Defenses ${\mathcal{D}}_{t_k}^{\mathrm{current}}=\{m\in \mathcal{M}\mid (t_k,m)\in \mathrm{Relationships}\}\cup \{d\in \mathcal{D}\mid (t_k,d)\in \mathrm{Relationships}\}$ comprise mitigations and detections directly addressing the observed technique $t_k$, extracted from authoritative ATT&CK framework relationships—these represent immediate containment measures that security teams must implement to neutralize the current threat. Next-Stage Defenses ${\mathcal{D}}_{t_{k+1}}^{\mathrm{next}}$ comprise mitigations and detections targeting the next technique $t_{k+1}$ in the attack chain, representing strategic measures that break the chain by preventing adversary progression to subsequent stages. Each defense item strictly adheres to ATT&CK standardized specifications. This dual-layer annotation enables fine-grained evaluation of whether systems can not only respond to visible threats but also anticipate and disrupt attack evolution, distinguishing reactive defense from strategic foresight.

5.2. Dataset Examples

Table 3. Representative threat scenarios from Operation Ghost (APT29 campaign) across four difficulty categories (observation point: T1546.003, next technique: T1078.002).

Category	Threat Description
Simple	“We discovered suspicious WMI event subscriptions on several domain controllers that were not part of our baseline configuration. The subscriptions trigger execution of encoded scripts whenever specific security events occur, persisting through system reboots. Investigation shows they were created using administrator credentials outside normal change windows.”
Confusing	“Our monitoring detected unusual persistence mechanisms across the finance department servers. Some appear to be WMI-based event filters executing payloads on certain triggers, while others look like scheduled tasks running at similar intervals. Both were created recently using elevated privileges, making it difficult to determine which persistence method is primarily being exploited.”
Noisy	“This week started with our quarterly system health checks—IT ran diagnostics on all domain controllers, generating thousands of WMI queries for performance monitoring. The new SIEM deployment also created baseline WMI subscriptions for security event correlation. On Wednesday, we noticed some unusual WMI event filters that seemed outside our standard monitoring configuration, created during the maintenance window. Meanwhile, the help desk dealt with printer spooler service crashes requiring multiple restarts, and our backup system triggered its scheduled tasks for nightly data replication across servers.”
False-positive	“Our IT security team deployed a new endpoint monitoring solution this week that uses WMI event subscriptions to track security-critical events across all domain-joined systems. The tool creates permanent WMI filters and consumers to monitor login attempts, privilege escalations, and policy changes. These subscriptions were installed using domain administrator credentials as part of our authorized security enhancement project.”
Current-Stage Defenses (for T1546.003):
Mitigations: M1040, M1026, M1018
Detection: DET0086
Next-Stage Defenses (for T1078.002):
Mitigations: M1032, M1027, M1026, M1018, M1017
Detection: DET0210

illustrates representative examples from CyberDefBench across the four difficulty categories, anchored within Operation Ghost, an APT29 campaign from 2013 to 2019 targeting European government entities. We construct scenarios based on the documented attack chain where the observation point is T1546.003 (WMI Event Subscription for persistence) and the next technique is T1078.002 (Valid Accounts: Domain Accounts), reflecting the adversarial progression in which APT29 first establishes WMI-based persistence on compromised hosts and subsequently leverages stolen administrator credentials for lateral movement across the network. For the Simple scenario, the threat description provides unambiguous WMI persistence indicators, enabling straightforward technique identification. The Confusing scenario deliberately mixes WMI Event Subscription and Scheduled Task characteristics, requiring systems to disambiguate between T1546.003 and T1053.005. The Noisy scenario embeds real WMI threat indicators within extensive irrelevant context about routine system administration and monitoring alerts, testing robustness to information overload. The False-positive scenario describes legitimate WMI-based monitoring deployed by IT operations that superficially resembles malicious persistence, evaluating discrimination capability. For all scenarios, the Current-Stage Defenses include mitigations M1040 (Behavior Prevention on Endpoint), M1026 (Privileged Account Management), M1018 (User Account Management), and detection strategy DET0086 targeting T1546.003. The Next-Stage Defenses target T1078.002 and include mitigations M1032 (Multi-factor Authentication), M1027 (Password Policies), M1026 (Privileged Account Management), M1018 (User Account Management), and M1017 (User Training), alongside detection strategy DET0210 to identify anomalous use of domain credentials and unauthorized lateral movement attempts.

5.3. Evaluation Metrics

We evaluate defense strategy generation systems using two primary metrics that assess the two core challenges identified in Section 3. Technique Mapping Accuracy measures whether systems can correctly identify the actual attack technique from threat descriptions, corresponding to Challenge 1 (accurate threat intelligence understanding):

$${\mathrm{Acc}}_{\mathrm{map}}={\displaystyle \frac{1}{|{\mathcal{Q}}_{\mathrm{test}}|}}\sum _{q\in {\mathcal{Q}}_{\mathrm{test}}}⊮\left[t_q^{*}=t_q^{\mathrm{gt}}\right]$$

(4)

where $t_q^{*}$ denotes the predicted technique and $t_q^{\mathrm{gt}}$ represents the ground truth technique for query q. Ground truth labels $t_q^{\mathrm{gt}}$ are directly derived from ATT&CK Procedure Examples documented in our annotated attack chains, and correctness is determined by exact technique ID matching, eliminating subjective judgment. This metric evaluates threat intelligence understanding across the four difficulty categories (Simple, Confusing, Noisy, False-positive). Defense Coverage evaluates whether generated strategies comprehensively cover the ground truth defenses, corresponding to Challenge 2 (strategic defense generation for attack disruption):

$${\mathrm{Cov}}_{\mathrm{def}}={\displaystyle \frac{1}{|{\mathcal{Q}}_{\mathrm{test}}|}}\sum _{q\in {\mathcal{Q}}_{\mathrm{test}}}{\displaystyle \frac{|s_q\cap {\mathcal{D}}_q|}{|{\mathcal{D}}_q|}}$$

(5)

where $s_q$ represents all generated defenses (mitigations and detections) and ${\mathcal{D}}_q={\mathcal{D}}_{t_k}^{\mathrm{current}}\cup {\mathcal{D}}_{t_{k+1}}^{\mathrm{next}}$ denotes all ground truth defenses extracted from the ATT&CK framework, including both current-stage defenses for the observed technique $t_k$ and next-stage defenses for the subsequent technique $t_{k+1}$ in the attack chain. All ground truth defenses ${\mathcal{D}}_q$ are automatically extracted from ATT&CK’s structured mitigation-technique and detection–technique relationship mappings, ensuring annotation objectivity without manual labeling subjectivity. This dual-layer evaluation enables assessment of whether systems can not only respond to visible threats but also anticipate and disrupt attack evolution.

6. Experiments

6.1. Experimental Setup

6.1.1. Baseline Methods

To comprehensively evaluate the effectiveness of our MACD framework, we compare against three representative baseline approaches that reflect current practices in automated defense strategy generation. To ensure fairness, all methods operate over identical input scenarios and ATT&CK knowledge sources.

• Single-Agent LLM (SA-LLM). A single GPT-4 agent directly generates defense strategies from threat queries without ATT&CK context or specialized knowledge retrieval.
• RAG-Only. This approach retrieves top-K most similar ATT&CK techniques and directly returns their associated mitigations and detections without LLM-based reasoning or contextual adaptation.
• Deep Research. An advanced LLM reasoning system (Claude Deep Research) that autonomously performs iterative web search and synthesis to generate defense recommendations without structured ATT&CK grounding.

6.1.2. Evaluation Metrics

We employ the two evaluation metrics defined in Section 5: Technique Mapping Accuracy (${\mathrm{Acc}}_{\mathrm{map}}$), measuring the correctness of threat-to-technique identification, and Defense Coverage (${\mathrm{Cov}}_{\mathrm{def}}$), assessing the completeness of generated strategies relative to ground truth ATT&CK mitigations and detections.

6.1.3. Experimental Environment

All experiments are conducted on a server equipped with an Intel Xeon Gold 6248R CPU and 128 GB RAM. The MACD framework uses GPT-4 as the backbone LLM for all agents, with temperature set to 0.3 for deterministic generation. The sentence transformer model (all-MiniLM-L6-v2) is used for vector embeddings with dimension $d=384$. For RAG retrieval, we set $K=5$ candidate techniques with a confirmation score threshold of 6.0. Our evaluation dataset comprises 160 scenarios derived from 40 real-world APT attack chains, with each chain generating 4 scenarios (Simple, Confusing, Noisy, and False-positive) at different observation points. Each scenario includes ground truth annotations for both current-stage defenses (targeting the observed technique $t_k$) and proactive defenses (targeting the next technique $t_{k+1}$ in the attack chain). Each experiment is repeated three times with different random seeds, and we report mean values with standard deviations.

6.1.4. Overall Performance Comparison

Table 4. Overall performance comparison on CyberDefBench. Best results are in bold. Statistical significance is assessed via Welch’s t-test comparing each baseline against MACD (* p < 0.05, ** p < 0.01).

Method	${\mathbf{Acc}}_{\mathbf{map}}$ (%)	${\mathbf{Cov}}_{\mathbf{def}}$ (%)	p-Value
Single-Agent LLM	68.2 ± 2.8	51.4 ± 3.2	p < 0.01 **
RAG-Only	75.8 ± 1.9	58.6 ± 2.5	p < 0.01 **
Deep Research	77.5 ± 2.4	64.2 ± 2.9	p < 0.05 *
MACD (Ours)	84.6 ± 1.6	72.3 ± 2.1	—

presents the overall performance comparison across all methods on CyberDefBench. MACD consistently outperforms all baselines across both metrics, achieving 84.6% technique mapping accuracy and 72.3% defense coverage. The Single-Agent LLM baseline exhibits significantly lower performance (68.2% mapping accuracy, 51.4% defense coverage), demonstrating that access to structured ATT&CK knowledge is critical for accurate threat understanding and comprehensive defense generation. The RAG-Only approach achieves moderate mapping accuracy (75.8%) through semantic retrieval but suffers from poor defense coverage (58.6%) due to lack of contextual reasoning and strategy synthesis. Deep Research demonstrates strong general reasoning capabilities (mapping accuracy 77.5%, defense coverage 64.2%) but cannot match MACD’s structured multi-agent collaboration grounded in authoritative ATT&CK framework knowledge.

6.1.5. Performance Across Scenario Categories

Table 5. Technique mapping accuracy (%) across scenario categories (mean ± std over three independent runs).

Method	Simple	Confusing	Noisy	False-pos
Single-Agent LLM	75.0 ± 2.0	58.3 ± 3.0	65.0 ± 2.5	52.5 ± 3.5
RAG-Only	82.5 ± 1.8	70.0 ± 2.5	72.5 ± 2.0	65.0 ± 3.0
Deep Research	85.0 ± 2.0	71.2 ± 2.8	69.4 ± 2.3	62.5 ± 3.2
MACD	90.0 ± 1.5	82.5 ± 2.5	81.8 ± 2.0	70.0 ± 3.0

breaks down performance by scenario category, revealing distinct strengths and challenges for each method. MACD demonstrates particular advantages in Confusing (82.5% vs. 71.2% for Deep Research) and Noisy (81.8% vs. 69.4%) scenarios, where multi-dimensional reasoning and noise filtering are critical. The Single-Agent LLM baseline performs reasonably on Simple scenarios (75.0%) but degrades significantly on Confusing (58.3%) and False-positive (52.5%) categories, highlighting its lack of structured knowledge for disambiguating similar techniques. The RAG-Only baseline achieves stable performance across categories but plateaus due to limited contextual adaptation capabilities. Notably, all methods struggle with False-positive scenarios, with MACD achieving 70.0% accuracy compared to 62.5% for the best baseline, indicating that distinguishing legitimate activities from malicious behaviors remains challenging even with sophisticated multi-agent reasoning.

6.1.6. Ablation Study

To validate the contribution of each agent in MACD, we conduct ablation experiments by systematically removing individual agents from the framework, as shown in

Table 6. Ablation study results showing the impact of removing individual agents.

Configuration	${\mathbf{Acc}}_{\mathbf{map}}$ (%)	${\mathbf{Cov}}_{\mathbf{def}}$ (%)	$\mathbf{\Delta }$ Acc	$\mathbf{\Delta }$ Cov
w/o Agent 1 (Technical)	83.1	60.5	−1.5	−11.8
w/o Agent 2 (Phase)	81.2	65.1	−3.4	−7.2
w/o Agent 3 (APT)	81.1	66.7	−3.5	−5.6
w/o Agent 4 (Coordinator)	84.2	65.9	−0.4	−6.4
MACD (Full)	84.6	72.3	–	–

. Removing the Technical Defense Expert (Agent 1) causes the largest performance drop in defense coverage (−11.8%), confirming that technique-specific mitigation knowledge is fundamental to comprehensive defense generation. Removing the Phase Defense Expert (Agent 2) primarily impacts defense coverage (−7.2%) and mapping accuracy on complex attack chain scenarios, validating its role in kill chain reasoning and proactive defense. The APT Defense Expert (Agent 3) contributes moderately to both metrics (−3.5% mapping accuracy, −5.6% coverage), particularly for threat hunting and behavioral detection strategies. Removing the Coordinator Agent (Agent 4) results in a 6.4% decrease in defense coverage due to redundant and inconsistent recommendations from uncoordinated expert outputs, demonstrating the necessity of strategy integration while maintaining high mapping accuracy since technique identification occurs before coordination.

7. Conclusions

This paper addresses the challenge of automated cybersecurity defense strategy generation by proposing MACD, a multi-agent collaborative framework that orchestrates specialized AI agents to generate comprehensive, ATT&CK-aligned defense strategies. Recognizing that effective defense requires expertise across technical mitigation, attack progression, and threat actor profiling, MACD deploys three expert agents coordinated through an intelligent integration mechanism while leveraging RAG-enhanced threat mapping to mitigate hallucination risks. We construct CyberDefBench, a comprehensive benchmark combining real-world APT scenarios with standardized evaluation metrics. Experimental results demonstrate that MACD significantly outperforms single-agent, retrieval-only, and deep research baselines, with particularly strong performance on complex multi-stage attack scenarios. Ablation studies validate the complementary contributions of each specialized agent. Future work will explore real-time deployment and extending coverage to emerging attack vectors such as supply chain and IoT threats.