LLM-Guided Dynamic Security Testing of Android Applications: A Comparative Study Across Selected Models

Abstract

The rapid growth of publicly available digital services increases the need for scalable security assessment. This is particularly important for software directly used by end users, such as Android applications. Due to staff shortages and financial constraints, small and medium-sized enterprises are often unable to test their applications for vulnerabilities. One possible support mechanism is the use of large language models (LLMs) to assist testers during such assessments. The aim of this study was to investigate the possibility of using an LLM as an interactive guide for dynamic application security testing (DAST) of Android applications. For this purpose, five LLM-based systems were compared: Gemini 2.5 Flash, GPT-oss 120B, Llama 3.3 70B, Qwen 3 32B, and Trinity Large Preview accessed via OpenRouter. The models were evaluated on intentionally vulnerable Android applications using weighted step-level scoring and three selected exploit guidance scenarios. In the main guidance experiment, Gemini achieved the highest combined Fully Discovered and Partially Discovered (FD + PD) detection rate of 79.1% in the representative run, while repeated runs for selected models showed limited aggregate variability. The results also indicate that more detailed prompts improve the quality of generated guidance. The findings suggest that LLMs can serve as interactive guides for DAST testing of Android applications, although they should be treated as supporting tools rather than standalone security-testing systems.

1. Introduction

In an era of rapidly expanding digital services, security is essential. However, the scale of modern software development makes continuous security testing difficult to perform consistently. This is particularly evident in the case of services developed for smartphones and applications developed for the Android operating system [1]. Such applications may complement web applications or constitute the primary interface through which users interact with a service. While security testing of web applications is well established, this is less so for mobile applications because of their specific characteristics [2]. This gap is also influenced by a lack of suitably trained personnel and insufficient knowledge of testing methodologies. Despite the numerous tools available for dynamic application security testing (DAST) on mobile devices, specific knowledge of what to test and how to test it is required. This is where Large Language Models (LLMs) can help as guides, indicating the next steps in testing. The aim of this study was to examine the usefulness of existing LLMs as guides in dynamic application security analysis, particularly in situations where an application’s security must be assessed quickly. Therefore, intentionally vulnerable Android applications (Sieve, DIVA, InjuredAndroid, InsecureBankv2, and PIVAA) were selected and analysed to determine how the evaluated models (Gemini, GPT, Llama, Qwen, and Trinity) can guide the tester through subsequent stages of security testing, including tool selection, command construction, and interpretation of results. The main contributions of this study are as follows:

• We developed scenarios to test the security of Android applications using selected LLM-based systems from different provider ecosystems.
• We assessed the extent to which the models help identify vulnerabilities in applications and, for selected high-impact scenarios, whether they assist in constructing reproducible exploits.
• We examined how prompt quality affects vulnerability detection results.

The results showed that certain models can assist users in the security testing of Android applications. Furthermore, the results provide an initial indication of the extent to which such assistance can be provided and which models may be better suited for this purpose.

The rest of this paper is organised as follows: Section 2 describes the fundamentals of the research subject. Section 3 covers the state of the art in the use of LLMs for detecting software vulnerabilities. Section 4 describes the experimental design and research scenarios. Section 5 presents the results and their analysis. Section 6 discusses the findings and identifies limitations of the study. Finally, Section 7 summarises the article and presents further possibilities for extending the research.

2. Background

2.1. Secure Development Lifecycle and the Role of DAST

SDLC (Software Development Lifecycle) [3] is a process for creating software that meets customer expectations. The SDLC comprises several stages. An example of this process is shown in Figure 1.

It consists of six stages:

1.

Analysis—detailed requirements are gathered and analysed to ensure a clear understanding of the project’s scope and objectives.

2.

Designing—abstract ideas transform into concrete plans (creating the software’s architecture and design).

3.

Coding—transforms the design into a functional application.

4.

Testing—checking that the software works as intended, detecting and fixing bugs, performance issues, or security vulnerabilities before deployment.

5.

Deployment—the application is deployed in a production environment where end users can access the software.

6.

Maintenance—supporting the software after deployment by fixing bugs or updating features.

When developing software using SDLC, organisations typically adopt a specific model, such as Agile, Waterfall, or DevOps. Each SDLC model focuses on different aspects of the development process and can be iterative or non-iterative. The team’s choice of model depends on the team’s competencies, organisational culture, and project requirements. SDLC ensures that software is developed in accordance with best practices. One of the benefits of using SDLC is the quality assurance it provides. With multiple stages of testing and validation, SDLC provides a guarantee of quality by helping to identify and resolve issues early in the development process. The focus on quality assurance ensures that teams can deliver high-quality software that meets customer expectations. One component of such quality is security, and developer verification activities should include security-oriented validation before release [4]. In this context, DAST (Dynamic Application Security Testing) is one of the relevant testing approaches [5]. DAST is a dynamic application security testing method. It involves analysing the operation of an application in real time: the application running on a server or in a test environment is examined by interacting with it directly. DAST focuses on detecting problems during application use and should be performed during the testing phase of the SDLC cycle, i.e., before the application is made available to users. Therefore, this is a crucial aspect in ensuring high software quality and security.

2.2. Security Aspects of Android Applications

Android enforces application isolation through Linux kernel primitives combined with platform mechanisms. Each application typically executes within an application sandbox under a distinct UID, which restricts direct access to other applications’ data and system resources [6,7,8].

Access to privileged functionality is mediated by a permission model and by controlled Inter-Process Communication (IPC) mechanisms, including Intents, Binder-based services, Broadcast Receivers, and Content Providers [7,8,9].

While this architecture provides a strong baseline, it also creates mobile-specific exposure points and trust boundaries. This baseline is reinforced by platform-level security mechanisms such as application sandboxing, permission enforcement, verified boot, and other system hardening features provided by Android [10]. In practice, vulnerabilities often stem from incorrectly exposed components and IPC endpoints, insecure local storage, misuse of cryptographic APIs, and inadequate network security configuration. Common examples include Activities or Services left exported=true (making them callable by any other app), misconfigured ContentProviders that leak sensitive data, and WebView instances that execute untrusted scripts. Such recurring risk categories are reflected in the Open Worldwide Application Security Project (OWASP) Mobile Top 10; the latest formal mobile-specific release is the 2024 edition [11]. While such taxonomies identify common risk classes, they do not confirm exploitability in a specific application.

Many Android issues are conditional on runtime state, user interaction, or device/environment configuration, and therefore require execution-time validation. For instance, certain vulnerabilities manifest only after a user logs in with elevated privileges, when a lifecycle callback such as onResume() is triggered, or when the device switches network context. Static inspection cannot capture these conditions, so dynamic tests must exercise representative user flows and app states to confirm exploitability.

2.3. Aspects of DAST in Android Applications

DAST evaluates security properties by interacting with an application at runtime. It involves observing externally visible behaviour (e.g., network traffic, IPC interactions, and filesystem effects) and actively manipulating inputs and state to test exploit hypotheses [12]. In this work, we use the term DAST broadly to include interactive, tester-driven runtime security assessment commonly used in mobile application testing.

For Android, DAST is particularly relevant because application behaviour is event-driven (lifecycle callbacks, UI events) and mediated by IPC and system services, which cannot be fully characterised through static inspection alone.

A typical Android DAST toolchain combines (i) traffic interception and modification (e.g., HTTPS interception proxies) [13], (ii) component enumeration and exploitation for exported IPC endpoints (e.g., Drozer) [14,15], and (iii) runtime instrumentation for dynamic tracing and method hooking (e.g., Frida) [16]; collectively, these techniques support validation-oriented testing by allowing an assessor to move from a suspected weakness (such as an exported activity or insecure storage) to evidence that the weakness is exploitable under realistic runtime conditions.

DAST workflows are iterative and hypothesis-driven. The tester must interpret intermediate evidence, assess its strength, and decide which branch of the testing strategy to pursue next. This is non-trivial in mobile contexts: recent work shows that significant portions of application logic may remain unexercised without targeted exploration and guidance, even when dynamic analysis is employed [17].

Moreover, adversarial runtime protections, often referred to as anti-runtime-analysis techniques (e.g., anti-instrumentation, anti-hooking, emulator or root detection, and debugger checks), can substantially reduce the effectiveness of dynamic tools [18].

Such checks are particularly relevant in security-sensitive Android applications, including banking and fintech apps, where app integrity, device integrity, emulator detection, root detection, and anti-instrumentation mechanisms may be used to identify potentially unsafe runtime environments. Depending on the implementation, such mechanisms may cause the application to terminate, restrict functionality, or alter its behaviour when instrumentation or tampering is detected.

Handling such protections during authorised assessment may require app-specific configuration, controlled test-device preparation, or complementary static analysis, motivating research into assistance mechanisms for complex DAST workflows.

2.4. Characteristics of DAST Artefacts Used by LLM

The use of LLMs is increasingly popular today, as they simplify and automate specific tasks. The user creates a command (in the form of a prompt), and the LLM tries to provide the best answer. This interaction model is rooted in in-context and few-shot prompting capabilities that have been widely discussed in the LLM literature [19]. Undoubtedly, this approach can be used for security testing. Moreover, while in static application security testing (SAST) the user pastes code into the chat to request a security analysis, in DAST this is not as obvious. In the context of DAST, the user can provide certain artefacts, and the LLM will propose a solution based on them. In this context, artefacts should be understood as data obtained from DAST that, once sent to the model, are analysed, summarised, and used to plan the next steps. Such artefacts may include:

• HTTP requests and responses;
• DAST tool reports;
• Session and authentication data;
• DAST scanner configuration files;
• Application logic;
• Application structure and architecture;
• Specific test results;
• Errors (including server errors).

As described earlier, DAST is a black-box approach to testing running applications, during which vulnerabilities are identified, analysed, and validated. Web proxy tools are often used for this purpose, as they capture HTTP traffic (both requests and responses). Such traffic can serve as input data for an LLM during the initial stage of attack scenario creation, enabling the LLM to plan the attack and immediately detect security vulnerabilities in the analysed HTTP requests and responses [20].

Another key artefact is the reports generated by DAST tools. These typically contain a great deal of information, much of which is often unnecessary from a security testing perspective. The role of LLMs in this case is to extract the most important elements from such a report and plan the next steps [21]. It should be emphasised that DAST tools indicate the potential presence of a vulnerability, which must then be verified, so the LLM can determine how to do so in a given case. Of course, it is also important to emphasise the significance of results relating to a specific test case. When receiving results from a DAST tool designed to verify a specific type of vulnerability, an LLM can perform a precise analysis and develop an action plan.

Session and authentication management mechanisms are further elements that an LLM can use to suggest next steps [22]. Analysing the correctness of both mechanisms can reduce the risk of threats such as JWT token misuse, session fixation, incorrect session timeouts, or inconsistent login workflows.

A key factor influencing DAST tool results is the tool’s configuration. Assessing the configuration allows us to determine whether the scanner is set up correctly, which can lead to optimising its performance. Furthermore, it is also worth tailoring the configuration to a specific application, as well as to the scope of the scan in general.

Based on information about the application’s logic, an LLM can reconstruct the full workflow [23,24,25]. This is important for testing business logic vulnerabilities, which traditional scanners often fail to detect. Information about the application’s structure may include API endpoints, system modules, and frontend/backend components. Furthermore, by understanding the application’s architecture, the LLM can assist testers performing DAST in analysing, among other things, where authentication and access control take place and where data is validated.

Error messages often reveal clues about vulnerabilities. An LLM can detect potential vulnerabilities such as SQL injection or, more generally, a lack of input validation. Furthermore, server errors often include a stack trace that reveals details about the backend framework, the application structure, file paths, database names, and the versions of the libraries in use [26]. This can also be used to narrow down test cases, as identifying the specific technology the application was written in allows us to rule out vulnerabilities unrelated to it.

3. Related Work

We review the literature at the intersection of LLMs and security testing, focusing on Android and on workflows adjacent to mobile DAST. Recent survey literature indicates that the intersection of software security and large language models is rapidly expanding, covering vulnerability detection, secure code analysis, and security-oriented automation workflows [27,28]. We organise the discussion into four areas: (i) LLM-based Android security analysis, (ii) agentic or semi-automated vulnerability discovery and validation, (iii) LLMs as penetration testing assistants, and (iv) mobile DAST constraints and recent empirical context.

3.1. LLM-Based Analysis of Android Applications

Early research on LLMs for Android security has largely emphasised static or semi-static analysis. Kouliaridis et al. [29] evaluated nine LLMs (GPT-3.5, GPT-4, GPT-4 Turbo, Llama 2, Zephyr Alpha, Zephyr Beta, Nous Hermes Mixtral, MistralOrca, and Code Llama) against Android vulnerability categories aligned with OWASP Mobile Top 10. Their evaluation used over 100 vulnerable code samples, including obfuscated ones, and revealed substantial variability across models, with GPT-4 emerging as the top performer.

However, their study focused exclusively on static code analysis of isolated snippets rather than runtime validation, leaving questions about exploitability and false positive rates in realistic assessment scenarios. In particular, the dataset comprised over 100 artificially constructed vulnerable code fragments, which do not reflect the interdependence of components or lifecycle behaviour present in real-world apps. Consequently, while the results highlight the potential of LLMs to recognise syntactic vulnerability patterns, they do not speak to the models’ ability to reason about context or dynamic state.

Their results suggest that LLMs can provide explanatory value but do not consistently match specialised analysers for all vulnerability classes.

More recent work investigates scaling LLM reasoning to large Android codebases via context management and decomposition. Qian et al. proposed LAMD, which applies context extraction and tiered reasoning for Android malware classification [30]. Although the target problem is malware classification rather than vulnerability detection, the work highlights a recurring challenge for LLM security analysis. Naively providing large volumes of code can degrade signal-to-noise ratio and lead to incorrect conclusions. Other studies explored narrow tasks such as detecting hardcoded secrets in Android apps using LLM-based reasoning [31].

Importantly, contemporaneous empirical evidence also questions whether current LLM vulnerability signals reflect deep security semantics. Weissberg et al. show that vulnerability predictions can correlate strongly with shallow code metrics, motivating scepticism and careful validation when using LLMs for security decisions [32]. This supports the argument that runtime confirmation and human oversight remain necessary, particularly when moving from hypotheses to exploitability claims.

3.2. Agentic Vulnerability Discovery and Validation for Android

A parallel research direction integrates LLMs into agentic pipelines that orchestrate multiple analysers and attempt automatic validation. Wang and Zhou proposed A2, an agentic system for discovering and validating Android vulnerabilities that combines tool outputs with LLM reasoning and emphasises exploit validation to reduce false positives [33]. Their approach demonstrates that LLM reasoning can be integrated with existing analysis techniques, such as static analysers and automatic proof-of-concept generation, to validate findings within a specialised automated pipeline. Such systems provide an important reference point for what is achievable under high automation. However, alternative lightweight approaches, such as reinforcement-learning agents or heuristic-driven fuzzers, remain less explored in mobile DAST settings.

Critically, their evaluation focused on end-to-end automation metrics (success rate, time to exploitation) rather than on whether human testers find LLM guidance helpful and actionable in real assessment scenarios. In contrast, our work evaluates a fundamentally different deployment model: conversational LLM assistance for human-driven Android DAST. This paradigm is accessible to small and medium-sized enterprises (SMEs) that lack agent orchestration infrastructure but could leverage LLM guidance to improve tester effectiveness.

Another category comprises agents used to analyse the security of an application’s source code. Examples of such solutions can be found in the articles [34,35].

3.3. LLMs as Penetration Testing Assistants

LLM-assisted penetration testing has been studied primarily in web and infrastructure contexts. Deng et al. introduced PentestGPT, which uses structured task management to mitigate context loss in multi-step workflows [36]. Subsequent evaluations indicate that contemporary LLMs still struggle with end-to-end autonomy despite planning structure [37]. From a broader decision-support perspective, LLM-guided testing can also be viewed as a process in which heterogeneous observations are interpreted over time to support dynamic testing decisions; this is only loosely analogous to data-driven monitoring and dynamic decision-support approaches in other engineering domains [38]. Contemporary LLM-based systems often fail in enumeration and exploitation phases, especially when they are required to utilise intermediate evidence correctly. Complementary work explores planning-based agent architectures to improve coherence and long-horizon performance [39].

Notably, prior work has not systematically evaluated how prompt quality affects guidance effectiveness, particularly in the context of human-driven Android DAST workflows. By prompt quality we mean (i) the specificity of environment details provided to the LLM (e.g., OS version, device/emulator type, available tools), (ii) the structure and completeness of task descriptions (objective, current findings, next actions), and (iii) the choice of decoding parameters such as temperature. Security-oriented prompts often require more explicit context than typical coding prompts, because hallucinations or missing assumptions can lead to false positives or dangerous omissions.

While general prompt engineering principles are well-documented for software engineering tasks [40], their application to security workflows remains underexplored. Factors such as prompt specificity, inclusion of technical context, and decoding parameters (e.g., temperature) may significantly influence both the coverage and precision of LLM-generated testing plans.

These results are relevant but not directly transferable to Android DAST. Android assessments require mobile-specific toolchains (ADB, Drozer, Frida, interception proxies), reasoning about component exposure and IPC, and validation of runtime conditions under a sandboxed operating system. Therefore, our work focuses on whether LLMs can improve tester effectiveness in mobile DAST, rather than whether they can autonomously complete a penetration test.

3.4. Mobile DAST Constraints and Recent Evidence

Mobile DAST faces structural constraints that complicate both automation and LLM guidance. Dynamic testing must exercise relevant execution paths. Reachability gaps remain significant even with dynamic analysis techniques [17], and anti-runtime-analysis mechanisms can further reduce the effectiveness of dynamic tooling [18].

At a broader level, a systematic literature review of DAST discusses ongoing challenges in coverage, false positives, and integrating intelligent guidance into dynamic testing workflows [41].

To support reproducible security research, the community maintains intentionally vulnerable Android applications with documented ground truth. Examples include InsecureBankv2 [42], DIVA (Damn Insecure and Vulnerable App) [43], and Sieve [44], which cover multiple OWASP Mobile Top 10 categories, such as insecure data storage, misuse of cryptography, improper platform usage and insecure communication. For instance, InsecureBankv2 implements a vulnerable authentication API and weak SSL certificate validation; DIVA includes misconfigured exported components and hard-coded keys; Sieve demonstrates improper random number generation and insecure storage of encryption keys. These applications enable controlled evaluation of detection rates and false positives, which is essential for assessing the quality of LLM guidance.

From a practical standpoint, the OWASP Mobile Application Security Verification Standard (MASVS) and the OWASP Mobile Application Security Testing Guide (MASTG) define mobile verification requirements and testing techniques that commonly underlie industrial Android DAST processes [45,46]. Institutional assessments and widely cited industry threat reports, such as the ENISA Threat Landscape 2025 and Elastic’s Global Threat Report 2025, document the increasing volume and sophistication of cyber threats [47,48]. Although these reports do not focus on mobile DAST directly, they underscore the importance of robust and repeatable security testing practices to keep pace with a rapidly evolving threat landscape.

3.5. Summary of the Research Gap

The literature indicates three key findings. First, LLMs can support selected Android security tasks, yet their reliability varies and may partly reflect superficial signals rather than vulnerability semantics [29,32]. Second, agentic systems can automate parts of discovery and validation, but typically require substantial orchestration and rarely evaluate interactive guidance quality [33]. Third, LLM-assisted penetration testing benefits from structured prompting while still exhibiting weaknesses in long-horizon, evidence-driven workflows [36,37].

What remains underexplored is the use of LLMs as interactive guides for Android DAST performed by a human tester. Prior work does not provide controlled measurements of overall guidance effectiveness, detection rates across specific vulnerability categories, the influence of prompt quality and decoding parameters, or the extent to which LLMs support practical exploit construction. More broadly, the reliability of AI-assisted analytical pipelines depends not only on the model but also on the quality, structure, and preprocessing of input data, a concern that is also discussed in data-driven monitoring and decision-support contexts [38]. This study addresses these gaps through controlled experiments on intentionally vulnerable Android applications with documented ground truth.

4. Materials and Methods

To assess the effectiveness of LLMs in DAST testing, the experiments were designed to answer the following research questions:

• RQ1: Is it possible to use an LLM as a guide for dynamic application security testing (DAST) on Android, and if so, to what extent?
• RQ2: Does prompt quality affect the number of vulnerabilities found?
• RQ3: Do LLMs help generate reproducible proof-of-concept guidance for selected exploit scenarios?

Through RQ1, we wanted to determine whether a user with limited knowledge of Android application penetration testing methods could use an LLM as an interactive handbook to assess the security of selected applications. For this purpose, five models were each tasked with guiding a tester through the DAST assessment of five vulnerable applications using a standardised two-turn interaction protocol (Section 4.2).

Through RQ2, we intended to examine how prompt quality affects the number of vulnerabilities found. Quality in this context refers to the amount of application-specific information provided to the model in the initial prompt. To isolate this variable, we tested three prompt specificity levels on a single application, Sieve, while keeping all other experimental parameters constant (Section 4.3).

Finally, in the context of RQ3, we investigated whether, for selected high-impact vulnerabilities, an LLM could help generate a reproducible proof-of-concept in a controlled testing environment. The RQ3-related scenario is described in Section 4.4.

4.1. Common Experimental Setup

4.1.1. Target Applications

Five intentionally vulnerable Android applications with documented ground truth were selected: Sieve [44] (8 test cases), DIVA [43] (10 test cases), InjuredAndroid [49] (5 test cases), InsecureBankv2 [42] (13 test cases), and PIVAA [50] (7 test cases). These applications were selected based on three criteria: (i) availability of documented ground-truth vulnerabilities, (ii) coverage of diverse OWASP Mobile Top 10 categories, and (iii) frequent use as benchmarks in prior Android security research [29,33]. For each application, the set of known vulnerabilities was decomposed into ordered chains consisting of one to four expected testing steps, which serve as the ground truth for evaluating model guidance. For RQ2, only Sieve was used as a focused prompt-sensitivity case study, because its eight test cases span diverse vulnerability categories relevant to Android DAST, including IPC exposure, SQL injection, and security misconfiguration. Therefore, the RQ2 results are treated as indicative evidence rather than as a general claim across all Android benchmark applications. For RQ3, we did not evaluate a full vulnerability set per application. Instead, we selected three exploitation-focused scenarios designed to represent common high-impact Android weaknesses and to require end-to-end, reproducible proof-of-concept (PoC) guidance. Scenario S1 targets a local SQLite injection in DIVA, whereas scenarios S2 and S3 target InsecureBankv2 (exported activity authentication bypass and Android backup misconfiguration, respectively). Each scenario is defined by a fixed attacker model (tester with ADB access; no root assumed unless explicitly stated), an initial evidence artefact (e.g., logcat SQLite errors, drozer component enumeration results, manifest configuration), and a six-step exploit checklist used for response evaluation (Section 4.4).

4.1.2. LLM Selection and Access

Five LLM-based systems were used as sources of guidance in the experiments and were accessed through three API providers. They were selected to represent commonly available model families and provider ecosystems rather than to perform an architectural benchmark.

Table 1. Large language model (LLM)-based systems used as guidance sources in the experiments.

Label	Model	Provider	Architecture	Total	Active
Gemini	Gemini 2.5 Flash	Google AI Studio	Undiscl.	Undiscl.	Undiscl.
GPT	GPT-oss 120B	Cerebras	MoE	117 B	5.1 B
Llama	Llama 3.3 70B	Cerebras	Dense	70 B	70 B
Qwen	Qwen 3 32B	Cerebras	Dense	32.8 B	32.8 B
Trinity	Trinity Large Preview	OpenRouter	MoE	∼400 B	13 B

summarises the models and access providers used in the study.

The evaluated set comprised Gemini 2.5 Flash, GPT-oss 120B, Llama 3.3 70B, Qwen 3 32B, and Trinity Large Preview. The API identifiers used in the experiments were gemini-2.5-flash, gpt-oss-120b, llama-3.3-70b, qwen-3-32b, and arcee-ai/trinity-large-preview:free.

Because the focus of this study is comparative testing performance rather than architectural benchmarking, we report the model names, providers, and provider-reported parameter information used in the experiments rather than relying on detailed architectural comparisons. Provider-reported technical characteristics may differ in scope and level of disclosure across models and services. Gemini 2.5 Flash was accessed through Google AI Studio and the Gemini API service. Note that GPT-oss was accessed through the Cerebras API service, whereas the model itself is an open-weight model released by OpenAI and is distinct from OpenAI’s proprietary hosted GPT models. Trinity Large Preview was accessed via the OpenRouter API service.

4.1.3. Testing Infrastructure and Shared Configuration

All models were accessed through a locally deployed custom web application used as a querying interface for the selected cloud API services. The application was implemented using React 19.2.0 and React DOM 19.2.0, and built with Vite 7.2.4. It enabled parallel querying of all five models under matched high-level experimental conditions, with provider-specific defaults retained. The interface sent requests to Google AI Studio/Gemini API, Cerebras API, and OpenRouter API endpoints. Since these were cloud API services rather than locally installed inference software, no local software version number was applicable for the provider-side services. For RQ1 and RQ2, each session used the same general-purpose system prompt, instructing the model to act as a technical security analyst. For RQ3, a modified exploitation-focused system prompt was used, explicitly requesting concrete commands, payloads, and step-by-step instructions for reproducible exploit verification. Both system prompts are provided in Appendix A.1. Across all experiments, the temperature was fixed at $0.7$, selected as a pragmatic balance between response consistency and diversity, consistent with settings used in prior LLM-assisted penetration testing work [36]. The effect of temperature on guidance quality was not investigated in this work and constitutes a potential variable for future experiments. All remaining generation parameters (top_p, top_k, frequency_penalty, etc.) were left at provider default values for all models.

Output token limits were configured per provider to ensure complete, non-truncated responses while respecting API constraints: 15,000 tokens for the three Cerebras-hosted models (GPT-oss, Llama, Qwen), 6000 tokens for Trinity via OpenRouter, and 50,000 tokens for Gemini via Google AI Studio. No response truncation was observed during the experiments. The heterogeneity of provider-level parameters, including output token limits, is a deliberate design choice: the study evaluates models as deployed through the standard API access rather than under artificially unified conditions, reflecting realistic usage by organisations without specialised inference infrastructure. Consequently, the reported results should be interpreted as a comparison of practically available model-service configurations rather than as a fully controlled comparison of model architectures under identical decoding settings. Two of the evaluated models produced explicit reasoning content before the final answer in the API responses used during the experiments. For consistency of scoring, such reasoning content was removed before evaluation so that only the final guidance text was assessed. No additional reasoning-specific parameters were configured explicitly; provider defaults were retained.

Table 2. Inference configuration per model used in the experiments. Context window denotes the maximum input length supported by the provider endpoint used at the time of the experiments; max output tokens denotes the limit configured in the querying application.

Label	Context Window	Max Output Tokens	Temperature
Gemini	1,048,576	50,000	0.7
GPT	131,072	15,000	0.7
Llama	128,000	15,000	0.7
Qwen *	32,768	15,000	0.7
Trinity	128,000	6000	0.7

* Qwen 3 32B supports 32768 tokens natively; extended context of up to 131072 tokens is available via YaRN scaling but was not explicitly configured.

summarises the inference configuration for each model.

The model parameters and context-window values in Table 1 and Table 2 are based on provider-reported specifications available at the time of the experiments.

At the tool-category level, the testing infrastructure remained constant across all three research questions. The experiments used Android Debug Bridge (ADB) for device communication, Drozer for IPC testing [14,15], Frida for runtime instrumentation [16], an HTTPS interception proxy for traffic analysis [13], an Android emulator, and a physical Android device. Exact minor versions of these tools were not fixed as an experimental variable, because the study evaluated LLM-generated testing guidance rather than the performance of specific tool releases. Commands and tool usage followed the guidance generated in each session and the tester’s available local environment. Where local software versioning was not applicable or exact version information was not retained, relevant official documentation or repository pages are provided in the references where available, together with the access date. The concrete sequence of executed actions was not predefined by the researchers: it was derived from each model’s guidance and from the runtime evidence collected during testing. At a methodological level, this reflects a dynamic decision-support setting in which observations from a running process are used to guide subsequent decisions, which is conceptually related to data-driven monitoring and decision optimisation approaches [38]. When a suggested action was infeasible due to missing details or incorrect assumptions, the step was evaluated according to the scoring rubric rather than being silently corrected.

4.2. RQ1 Scenario: Two-Turn DAST Guidance

To simulate a realistic human-in-the-loop Android DAST workflow, we employed a structured two-turn interaction protocol.

In the first turn, an initial prompt described the target application type, the testing environment (Windows with PowerShell), and requested a complete DAST testing plan with specific tools and commands. The prompt structure was kept consistent across all five applications, with only the application description adapted per target (full prompts in Appendix A).

After receiving the model’s DAST plan, the tester executed the suggested steps against the target application. The outcomes, namely which steps succeeded, which could not be performed due to insufficient detail, and what evidence was collected, were then fed back in a structured second turn requesting explicit vulnerability analysis and next-step guidance.

For the representative cross-model comparison, this protocol was applied for each model-application combination (5 models × 5 applications = 25 sessions). The conversation transcripts were evaluated against the ground-truth step chains using the scoring methodology in Section 4.5.

To assess the stability of the RQ1 results across independent model sessions, the RQ1 protocol was repeated two additional times for the three models that remained available on their original provider endpoints: Gemini 2.5 Flash, GPT-oss 120B, and Trinity Large Preview. These repeated runs used the same application descriptions, system prompt, temperature ($T=0.7$), and scoring procedure as the original RQ1 session. Llama 3.3 70B and Qwen 3 32B were not repeated because their original provider endpoints were no longer available under the same serving configuration after the initial experiments; re-running them through different serving configurations would not constitute a directly comparable repetition.

4.3. RQ2 Scenario: Prompt Quality Levels

To investigate the effect of prompt specificity on vulnerability detection, we designed three prompt variants of increasing detail, all targeting Sieve. Each variant used the same two-turn protocol (Section 4.2) and scoring methodology (Section 4.5), but varied the contextual information about the target application (full prompt texts in Appendix A):

• Simple prompt: a minimal functional description, where the application stores user data locally, without mentioning security mechanisms, authentication, or data sensitivity;
• Medium prompt: added explicit security context: the application manages sensitive credentials and is protected by user-defined authentication;
• Advanced prompt: a full operational description identifying the application as a password manager with master password, PIN, offline operation, and local backup/restore functionality.

Each of the five models was tested at all three levels (5 models × 3 levels = 15 sessions), with each session covering the same eight Sieve vulnerability test cases.

4.4. RQ3 Scenario: Exploit Generation

RQ3 evaluates whether an LLM can assist a human tester in producing a complete and reproducible proof-of-concept (PoC) exploit for selected Android weaknesses. In contrast to RQ1/RQ2 (coverage of multiple vulnerabilities), RQ3 focuses on end-to-end exploit construction quality for three high-impact scenarios:

• S1 (DIVA): Local SQLite SQL injection. Evidence artefact: UI search field and logcat SQLite errors triggered by special characters, suggesting unsanitised query construction.
• S2 (InsecureBankv2): Exported activity authentication bypass. Evidence artefact: drozer enumeration indicating exported post-login activities with permission:null.
• S3 (InsecureBankv2): Android backup misconfiguration. Evidence artefact: android:allowBackup="true" and sensitive data stored in SharedPreferences and local databases.

4.4.1. Interaction Procedure

Each scenario was tested independently with a single exploitation-focused prompt describing (i) the application context, (ii) the evidence artefact motivating the vulnerability hypothesis, and (iii) the requirement for a step-by-step dynamic exploitation strategy with a reproducible PoC. Unlike the two-turn protocol used in RQ1 and RQ2, RQ3 employed a single-turn interaction: the model received one prompt containing the scenario description and evidence artefact, and its response was evaluated without follow-up. This design choice reflects the RQ3 objective of assessing whether a single, well-structured prompt with concrete evidence can elicit a complete exploit procedure, rather than measuring iterative guidance quality. All remaining inference and tooling settings followed the common configuration described in Section 4.1.3. The tester did not “repair” missing commands; insufficient operational detail was treated as an evaluation signal.

4.4.2. Exploit Checklist and Step-Level Scoring

For each scenario, we defined a fixed six-step exploit checklist reflecting practical exploitation stages (identification, payload/commands, procedure, confirmation of impact, runtime evidence, reproducibility). Each step was rated using the same colour rubric as in Section 4.5: green ($1.0$), yellow ($0.6$), orange ($0.3$), red ($0.0$).

Unlike RQ1/RQ2 (where chains have one to four steps depending on the vulnerability), RQ3 uses a uniform six-step structure to explicitly capture “PoC completeness” and reproducibility requirements.

4.4.3. Exploit Guidance Score (EGS)

To aggregate the six-step ratings into a single numeric metric, we computed the Exploit Guidance Score (EGS) as a weighted sum:

$$\mathrm{EGS}={\displaystyle \sum _{i=1}^6}{w}_i{c}_i,$$

(1)

where $\mathrm{EGS}$ denotes the Exploit Guidance Score, $i\in \{1,\dots ,6\}$ denotes the exploit-checklist step index, $w_i$ denotes the positional weight assigned to step i, and $c_i\in \{1.0,0.6,0.3,0.0\}$ denotes the colour-based quality score assigned to step i according to the green/yellow/orange/red rubric.

The positional weights used in the RQ3 exploit checklist were defined as follows:

$$(w_1,w_2,w_3,w_4,w_5,w_6)=(0.25,0.20,0.18,0.15,0.12,0.10).$$

(2)

This weighting reflects that early exploit stages (e.g., identifying the correct attack surface and providing executable commands) are necessary to enable any exploitation attempt, while later steps (especially reproducibility) refine the strength of the PoC. The weights decrease approximately linearly from identification to reproducibility, with the first two steps jointly accounting for 45% of the total score.

4.4.4. Gate Step and Exploit Success (ES)

Because an exploit can only be considered valid if the model provides at least one “decision-critical” verification instruction, we introduce a gate step concept. A gate step is the checklist item that directly validates exploit impact in the running application (e.g., confirming SQL injection success, verifying authentication bypass, or demonstrating extraction of sensitive data from a backup). The gate step index is scenario-dependent:

• S1 (SQL injection): gate step = Step 4 (confirming injection success).
• S2 (exported activity bypass): gate step = Step 4 (verifying authentication bypass while logged out).
• S3 (ADB backup): gate step = Step 5 (identifying sensitive data in extracted backup artefacts). In cases where a model’s response structure merged steps 4 and 5, the gate was evaluated at the step containing the verification content.

We define a binary exploit success indicator:

$$\mathrm{ES}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{the}\mathrm{gate}\mathrm{step}\mathrm{is}\mathrm{green}(\mathrm{fully}\mathrm{correct}\mathrm{and}\mathrm{actionable}),\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(3)

4.4.5. Final Exploit Guidance Classes (FE/PE/NE)

We map the pair $(\mathrm{EGS},\mathrm{ES})$ into three exploit guidance outcomes:

• FE (Fully Exploitable) if $\mathrm{ES}=1$ and $\mathrm{EGS}>0.75$.
• PE (Partially Exploitable) if ($\mathrm{ES}=1$ and $\mathrm{EGS}\le 0.75$) or ($\mathrm{ES}=0$ and $\mathrm{EGS}>0.40$). The first sub-case covers scenarios where the gate step is correct but surrounding guidance is weak; the second covers correct overall direction without decision-critical verification.
• NE (Not Exploitable) otherwise ($\mathrm{ES}=0$ and $\mathrm{EGS}\le 0.40$; insufficient or incorrect guidance).

This design is intentionally stricter than the FD/PD/ND mapping used in RQ1/RQ2, because RQ3 targets reproducible PoC generation rather than vulnerability discovery. In particular, missing or non-actionable verification guidance (gate step) prevents classifying a scenario as fully exploitable even if earlier steps are generally correct.

4.5. Scoring Methodology

To quantify guidance quality, we developed a weighted step-level scoring methodology. Each vulnerability test case is represented as an ordered chain of N steps. For each step i, the evaluator assigns a quality score $c_i$:

• Green ($c_i=1.00$): fully correct and actionable;
• Yellow ($c_i=0.60$): described but lacking detail or specific instructions;
• Orange ($c_i=0.30$): mentioned but too generic to be directly actionable;
• Red ($c_i=0.00$): incorrect or blocking.

Earlier steps carry more weight than later ones. This design choice reflects the structure of DAST workflows: the initial step in a vulnerability chain typically determines whether the tester can engage with the relevant attack surface at all (e.g., selecting the correct tool or identifying the target component), whereas later steps refine and confirm the finding. A model that provides a correct entry point enables the tester to proceed with partial manual effort, even if subsequent guidance is incomplete; conversely, an incorrect first step leaves no actionable path forward regardless of later quality. The positional weights $w_i$ depend on chain length N (

Table 3. Positional weight vectors for guidance chains of lengths from 1 to 4.

N	Weights $({\mathit{w}}_1,{\mathit{w}}_2,\mathit{\dots },{\mathit{w}}_{\mathit{N}})$
1	$\left(1.00\right)$
2	$(0.60,0.40)$
3	$(0.45,0.30,0.25)$
4	$(0.35,0.27,0.21,0.17)$

).

The weight vectors were constructed so that (i) values decrease monotonically with step position, (ii) the first step always carries roughly twice the weight of the last, and (iii) weights sum to unity. Within these constraints, the specific values were set to decrease approximately geometrically. Because the mapping from scores to detection classes uses coarse thresholds, the weighting scheme is intended to reflect the relative importance of earlier and later testing steps rather than to provide a fine-grained continuous performance measure. We do not apply a hard-fail rule for weak intermediate steps: in practice, testers can often recover from partially specified guidance by consulting tool documentation or building on earlier correct steps through manual exploration.

The final score for each vulnerability test case was computed as follows:

$$S={\displaystyle \sum _{i=1}^N}{w}_i{c}_i,$$

(4)

where S denotes the final guidance score for a vulnerability test case, N denotes the number of steps in the corresponding vulnerability chain, i denotes the step index, $w_i$ denotes the positional weight assigned to step i, and $c_i$ denotes the quality score assigned to step i according to the colour-based rubric.

The score was mapped to three detection classes: FD (Fully Discovered, $S>0.75$), PD (Partially Discovered, $0.40<S\le 0.75$), and ND (Not Discovered, $S\le 0.40$). A test case was considered detected if classified as FD or PD.

This methodology was applied identically to RQ1 and RQ2. RQ3 used a separate success criterion described in Section 4.4. All step-level ratings were assigned manually using the predefined rubric and the same evaluation checklist across all sessions. The evaluation was conducted by a two-person team consisting of a senior security researcher with academic and penetration testing experience and a junior/intermediate penetration tester. To reduce subjectivity, the evaluators discussed the assigned ratings and resolved ambiguous cases by consensus before computing the final FD/PD/ND and FE/PE/NE outcomes. Formal inter-rater agreement and blinded scoring were not applied and are therefore treated as limitations of the study.

5. Results

5.1. RQ1: LLM as a Guide for Android DAST

Following the experimental setup and scoring methodology described in Section 4.1 and Section 4.5, we evaluated whether guidance generated by the five LLM-based systems could help a tester identify vulnerabilities across 43 test cases in the five target applications. This subsection first illustrates the scoring on two contrasting examples, then presents the aggregate results.

5.1.1. Illustrative RQ1 Scoring Examples

To illustrate the application of the scoring methodology, we present two contrasting cases.

High-Performing Case: Gemini on InsecureBankv2

Figure 2 shows the step-level evaluation of Gemini’s guidance for all 13 vulnerabilities in InsecureBankv2. Out of the 13 test cases, Gemini achieved FD on 9, PD on 2, and ND on 2. For example, the Exploiting Android Vulnerable Broadcast Receiver vulnerability consists of three steps: (1) use Drozer to enumerate exported broadcast receivers, (2) analyse code related to the broadcast receiver, and (3) use ADB/Drozer to send a crafted broadcast. Gemini received green on step 1, yellow on step 2, and green on step 3, yielding $S=0.45\times 1.0+0.30\times 0.6+0.25\times 1.0=0.88$ (FD). In contrast, for User Enumeration (2 steps), step 1 was scored red and step 2 yellow, giving $S=0.60\times 0.0+0.40\times 0.6=0.24$ (ND); the model failed to guide the tester towards the initial identification step.

Low-Performing Case: Trinity on Sieve

Figure 3 presents the evaluation of Trinity’s guidance on Sieve (8 vulnerabilities). Trinity achieved FD on 0, PD on 1, and ND on 7. The only partially discovered vulnerability was Content Provider—Data Leakage (3 steps), where Trinity received yellow on step 1, yellow on step 2, and orange on step 3, yielding $S=0.45\times 0.6+0.30\times 0.6+0.25\times 0.3=0.525$ (PD). For Exported Services (3 steps), all three steps were scored orange, giving $S=0.45\times 0.3+0.30\times 0.3+0.25\times 0.3=0.30$ (ND); the model mentioned the relevant concepts but never provided specific commands or tool invocations. For SQL Injection—Data Extraction (3 steps), all steps received red, resulting in $S=0.0$ (ND).

5.1.2. Aggregate RQ1 Results

Table 4. RQ1 results for the representative run: number of vulnerabilities classified as FD, PD, and ND per model and application ($n=43$ per model). FD—Fully Discovered, PD—Partially Discovered, ND—Not Discovered.

Application	Gemini			GPT			Llama			Qwen			Trinity
	FD	PD	ND	FD	PD	ND	FD	PD	ND	FD	PD	ND	FD	PD	ND
Sieve (8)	4	2	2	0	2	6	0	0	8	0	0	8	0	1	7
DIVA (10)	2	6	2	1	6	3	2	3	5	4	4	2	2	5	3
InjuredAndroid (5)	0	5	0	2	3	0	0	5	0	2	2	1	5	0	0
InsecureBankv2 (13)	9	2	2	3	3	7	4	3	6	4	4	5	8	1	4
PIVAA (7)	2	2	3	1	2	4	0	5	2	1	2	4	0	3	4
Total	17	17	9	7	16	20	6	16	21	11	12	20	15	10	18

presents the RQ1 detection results for all five models across the five applications. The table reports the representative run used for the main cross-model comparison. In total, 43 vulnerability test cases were evaluated per model (the number in brackets next to each application indicates the number of security use cases under consideration).

Table 5. RQ1 detection rates for the representative run per model across all 43 vulnerability test cases.

Model	FD	PD	FD + PD
Gemini	39.5%	39.5%	79.1%
GPT	16.3%	37.2%	53.5%
Llama	14.0%	37.2%	51.2%
Qwen	25.6%	27.9%	53.5%
Trinity	34.9%	23.3%	58.1%

summarises the detection rates for the representative run as percentages.

Gemini achieved the highest FD rate (39.5%) and the highest combined FD + PD rate (79.1%), meaning that for approximately four out of every five vulnerability test cases, Gemini provided guidance that was at least partially useful to the tester. Trinity ranked second in terms of FD count (15 out of 43, i.e., 34.9%), but also exhibited a high ND count (18, i.e., 41.9%), indicating polarised behaviour: when Trinity appeared to align better with a given vulnerability type, its guidance was often fully actionable, but for other vulnerability types it failed to provide any useful direction. Qwen and GPT each left 20 out of 43 test cases (46.5%) in the ND category while Llama left 21 (48.8%). However, their FD/PD distributions differed notably: Qwen produced 11 FDs and 12 PDs, whereas GPT produced only 7 FDs but 16 PDs, and Llama 6 FDs but 16 PDs. This indicates that when Qwen did provide guidance, it tended to cover more steps correctly (reaching FD), while GPT and Llama more often gave partial guidance that identified the right direction but omitted critical details needed for full confirmation.

Repeated-run stability results for the subset of models that remained available on their original provider endpoints are reported separately in

Table 6. RQ1 inter-session detection stability per application across three independent runs for the models that remained available through their original provider endpoints.

Model	Application	n	Run 1 *	Run 2	Run 3	Mean Det. Rate ± SD
Gemini	Sieve	8	4/2/2	3/3/2	4/2/2	75.0 ± 0.0%
	DIVA	10	2/6/2	2/5/3	2/5/3	73.3 ± 5.8%
	InjuredAndroid	5	0/5/0	1/4/0	0/5/0	100.0 ± 0.0%
	InsecureBankv2	13	9/2/2	8/3/2	7/3/3	82.1 ± 4.4%
	PIVAA	7	2/2/3	1/3/3	2/2/3	57.1 ± 0.0%
GPT	Sieve	8	0/2/6	0/2/6	1/2/5	29.2 ± 7.2%
	DIVA	10	1/6/3	1/5/4	1/6/3	66.7 ± 5.8%
	InjuredAndroid	5	2/3/0	1/4/0	2/3/0	100.0 ± 0.0%
	InsecureBankv2	13	3/3/7	3/4/6	3/3/7	48.7 ± 4.4%
	PIVAA	7	1/2/4	1/2/4	1/2/4	42.9 ± 0.0%
Trinity	Sieve	8	0/1/7	0/2/6	0/1/7	16.7 ± 7.2%
	DIVA	10	2/5/3	1/5/4	2/4/4	63.3 ± 5.8%
	InjuredAndroid	5	5/0/0	4/1/0	4/1/0	100.0 ± 0.0%
	InsecureBankv2	13	8/1/4	7/2/4	8/1/4	69.2 ± 0.0%
	PIVAA	7	0/3/4	0/2/5	0/3/4	38.1 ± 8.2%

Note: Each run column reports FD/PD/ND counts. The detection rate is computed as $(\mathrm{FD}+\mathrm{PD})/n\times 100$. SD denotes the sample standard deviation across the three run-level detection rates. * Run 1 corresponds to the representative session reported in Table 4.

and 

Table 7. RQ1 aggregate inter-session detection stability across all five applications.

Model	n	Run 1 *	Run 2	Run 3	Mean Det. Rate ± SD
Gemini	43	17/17/9	15/18/10	15/17/11	76.7 ± 2.3%
GPT	43	7/16/20	6/17/20	8/16/19	54.3 ± 1.3%
Trinity	43	15/10/18	12/12/19	14/10/19	56.6 ± 1.3%

Note: Each run column reports FD/PD/ND counts aggregated across all five applications. The detection rate is computed as $(\mathrm{FD}+\mathrm{PD})/43\times 100$. SD denotes the sample standard deviation across the three aggregate run-level detection rates. * Run 1 corresponds to the representative session reported in Table 4. Llama 3.3 70B and Qwen 3 32B were not included in the repeated-run stability analysis because their original provider endpoints were no longer available under the same serving configuration after the initial experiments; re-running them under different serving configurations would not constitute a directly comparable repetition.

.

The repeated-run analysis suggests limited inter-session variability for the three models that could be re-evaluated on their original provider endpoints. Table 6 reports application-level stability, while Table 7 summarises the aggregate results across all five applications. At the aggregate level, Gemini achieved a mean detection rate of 76.7 ± 2.3%, GPT achieved 54.3 ± 1.3%, and Trinity achieved 56.6 ± 1.3%. Application-level standard deviations ranged from 0.0 to 8.2 percentage points, suggesting that the main trend for these models was not driven solely by a single anomalous generation. The repeated sessions also preserve the main trend observed in the representative run: Gemini remained the strongest model in terms of combined FD + PD detection rate, while GPT and Trinity showed lower aggregate detection rates. However, because Llama and Qwen could not be repeated on their original endpoints, no inter-session stability claim is made for those two models.

5.1.3. Summary of RQ1 Findings

The results suggest that LLMs can provide useful guidance for Android DAST, although performance varies substantially across models and applications. In the representative run, Gemini provided at least partially useful guidance for 79.1% of test cases (FD + PD), yet even the best model achieved an FD rate of only 39.5%. The repeated-run analysis for the three models that remained available on their original endpoints confirmed the same general trend, with aggregate mean detection rates of 76.7 ± 2.3% for Gemini, 54.3 ± 1.3% for GPT, and 56.6 ± 1.3% for Trinity. The ND rates observed in the representative run indicate that a substantial proportion of vulnerabilities still received no actionable guidance. Therefore, LLMs can meaningfully assist a human tester but should be treated as a supplementary aid rather than a standalone methodology.

5.2. RQ2: Effect of Prompt Quality on Vulnerability Detection

Following the prompt quality protocol described in Section 4.3, each of the five models was evaluated on Sieve’s eight vulnerability test cases at three prompt specificity levels (simple, medium, and advanced). The same scoring methodology (Section 4.5) was applied, classifying each vulnerability as FD, PD, or ND.

The objective of this experiment was to determine whether the level of contextual information provided in the initial prompt influences the quality of guidance generated by the LLM during Android DAST. The testing protocol remained identical to the one used in RQ1, ensuring that differences in results originate solely from the prompt content. The simple prompt contained only a short functional description of the application. The medium prompt additionally indicated that the application stores sensitive credentials and that access to stored data is protected by user-defined authentication configured during setup. The advanced prompt described the application more precisely as an offline password manager with master password protection and local backup functionality.

5.2.1. Illustrative RQ2 Prompt-Sensitivity Examples

To illustrate how prompt specificity affects the quality of generated guidance, we present three examples of Gemini’s step-level evaluations for the Sieve application. Each figure corresponds to one prompt specificity level. The colour coding follows the scoring rubric defined in Section 4.5: green denotes a fully correct and actionable step, yellow indicates partially correct guidance lacking sufficient technical detail, orange represents generic guidance, and red corresponds to incorrect or missing instructions.

Simple Prompt

Figure 4 presents the evaluation results obtained using the simple prompt configuration. With only minimal application context available, Gemini proposes several relevant testing directions but frequently omits the concrete commands required to execute them. As a result, multiple steps are evaluated as yellow or orange. For instance, vulnerabilities related to exported components or content providers are often identified conceptually, yet the model does not consistently provide complete Drozer or ADB commands necessary to reproduce the testing chain.

Medium Prompt

When additional security context is introduced, the quality of the generated guidance improves. As illustrated in Figure 5, Gemini provides more precise testing instructions and identifies a larger number of relevant attack steps. In comparison with the simple prompt configuration, the proportion of green steps increases, particularly for vulnerabilities associated with exported activities and content providers. Nevertheless, some steps still lack sufficient operational detail and therefore remain classified as partially discovered.

Advanced Prompt

The advanced prompt yields the highest-quality guidance. As shown in Figure 6, Gemini produces more complete testing sequences and more frequently references specific tools such as Drozer and ADB. Most vulnerability chains consist of green or yellow steps, indicating that the proposed procedures can be followed directly by the tester with minimal additional interpretation. Compared with the previous prompt variants, the guidance becomes noticeably more structured and operational.

5.2.2. Aggregate RQ2 Results

The aggregate analysis compares the three prompt specificity levels across all five evaluated models. The results are presented first as FD/PD/ND counts and then as percentage-based detection rates to make the effect of prompt quality easier to compare between models.

As shown in

Table 8. RQ2 results: number of vulnerabilities classified as FD, PD, and ND per model and prompt specificity level on Sieve ($n=8$ per model-level combination). FD—Fully Discovered, PD—Partially Discovered, ND—Not Discovered.

Prompt	Gemini			GPT			Llama			Qwen			Trinity
	FD	PD	ND	FD	PD	ND	FD	PD	ND	FD	PD	ND	FD	PD	ND
Simple	2	3	3	0	4	4	0	2	6	0	3	5	0	2	6
Medium	4	3	1	0	7	1	0	4	4	0	3	5	1	6	1
Advanced	5	3	0	7	1	0	0	6	2	1	6	1	4	4	0

, higher prompt specificity generally reduces the number of ND outcomes and shifts more cases into the FD or PD categories. The corresponding percentage-based detection rates are reported in

Table 9. RQ2 detection rates per model and prompt specificity level on Sieve ($n=8$).

Model	Prompt	FD	PD	FD + PD
Gemini	Simple	25.0%	37.5%	62.5%
	Medium	50.0%	37.5%	87.5%
	Advanced	62.5%	37.5%	100.0%
GPT	Simple	0.0%	50.0%	50.0%
	Medium	0.0%	87.5%	87.5%
	Advanced	87.5%	12.5%	100.0%
Llama	Simple	0.0%	25.0%	25.0%
	Medium	0.0%	50.0%	50.0%
	Advanced	0.0%	75.0%	75.0%
Qwen	Simple	0.0%	37.5%	37.5%
	Medium	0.0%	37.5%	37.5%
	Advanced	12.5%	75.0%	87.5%
Trinity	Simple	0.0%	25.0%	25.0%
	Medium	12.5%	75.0%	87.5%
	Advanced	50.0%	50.0%	100.0%

.

The results show that prompt specificity affects vulnerability detection performance in this experimental setting. Across most models, the number of fully discovered vulnerabilities increases when more detailed application context is provided. For example, Gemini improves from two fully discovered vulnerabilities with the simple prompt to five with the advanced prompt. GPT exhibits an even stronger effect, achieving seven fully discovered vulnerabilities under the advanced configuration after producing none under the simple prompt. Trinity also shows a substantial improvement, increasing from zero fully discovered vulnerabilities with the simple prompt to four with the advanced prompt. Qwen was the only model for which the medium prompt did not improve detection over the simple prompt (FD + PD remained at 37.5% for both levels), although the advanced prompt still produced a substantial increase to 87.5%.

At the same time, the number of ND outcomes decreases as prompt specificity increases. This trend suggests that richer contextual descriptions help the models identify relevant attack surfaces and produce more operational testing steps. Overall, within this Sieve-based experiment, the results suggest that providing detailed application context improves the usefulness of LLM-generated guidance in Android DAST scenarios. Due to the small sample size ($n=8$ vulnerability test cases per model-level combination), formal statistical testing was not applied; the observed trends are reported descriptively. Although the analysis is limited to a single application, Sieve’s test cases span multiple OWASP Mobile Top 10 categories, including IPC exposure, SQL injection, and security misconfiguration. Therefore, the improvement from the simple to the advanced prompt observed across all five models should be interpreted as indicative rather than general evidence that prompt specificity influences guidance quality in Android DAST.

5.3. RQ3: LLM-Assisted Exploit Generation

While RQ1 and RQ2 focused on vulnerability identification and testing guidance, RQ3 investigates whether LLMs can assist a tester in constructing a complete and reproducible exploit for selected Android vulnerabilities. In this experiment, the LLM is expected not only to suggest testing directions but also to provide operational guidance enabling the tester to verify exploitability in practice.

Three representative exploitation scenarios were selected, covering different classes of Android security weaknesses and two widely used intentionally vulnerable applications: DIVA and InsecureBankv2.

• S1: SQL Injection in DIVA. The application uses a local SQLite database to retrieve user records through a search field. The behaviour of the application when special characters are inserted (e.g., a single quote) suggests that the query may be constructed through unsanitised string concatenation. SQLite errors observed in logcat indicate the possibility of SQL injection.
• S2: Exported Activity Authentication Bypass in InsecureBankv2. Runtime analysis using drozer reveals that several activities responsible for post-authentication functionality (e.g., dashboard, transfer, statements) are exported without permission protection. This raises the possibility of launching these activities directly via ADB or drozer without performing authentication.
• S3: Exploiting Android Backup Misconfiguration in InsecureBankv2. The application’s manifest contains the configuration flag android:allowBackup="true". This configuration may allow extraction of application-private data through the ADB backup mechanism without requiring root access, including credentials stored in SharedPreferences and local databases.

These scenarios were selected to provide diversity in exploitation mechanisms rather than to form a statistically representative benchmark. They cover three distinct DAST-relevant weakness classes: input-driven local data extraction, unauthorised access through exported components, and extraction of private application data through platform-level backup misconfiguration.

For each scenario, model responses were evaluated using a six-step exploit checklist covering identification of the attack surface, exploitation commands, confirmation of impact, and exploit reproducibility. Each step was rated according to the same colour rubric used in previous experiments (green, yellow, orange, red). The final exploit guidance outcome was classified into three categories. These labels refer to the completeness and reproducibility of the model-generated exploit guidance, not to the intrinsic exploitability of the underlying vulnerability:

• FE (Fully Exploitable)—the model provides sufficient guidance to construct a complete and reproducible exploit.
• PE (Partially Exploitable)—the model provides correct direction but lacks sufficient operational detail for full exploit confirmation.
• NE (Not Exploitable)—the guidance is incomplete or incorrect and does not allow exploit verification.

5.3.1. Illustrative Examples

To illustrate the evaluation methodology, we present three representative examples corresponding to the tested scenarios.

Scenario S1: SQL Injection in DIVA (Trinity Example)

Figure 7 shows the evaluation of Trinity’s response for the SQL injection scenario. The model correctly identifies the vulnerable input field and suggests SQLite-compatible injection payloads, though with incomplete specificity. However, later stages of the exploit procedure are described only partially and the explanation of exploit reproducibility is incomplete. As a result, although the attack direction is correct, the exploit cannot be reliably confirmed and the scenario is classified as PE.

Scenario S2: Exported Activity Authentication Bypass in InsecureBankv2 (Llama Example)

Figure 8 presents the evaluation of Llama’s response for the exported activity authentication bypass scenario. The model partially identifies relevant post-login activities and describes the concept of launching activities directly. However, the commands required to launch the activity via ADB or drozer are incomplete or incorrect. Consequently, the tester cannot reliably verify the authentication bypass and the exploit is classified as NE.

Scenario S3: Android Backup Exploitation in InsecureBankv2 (Gemini Example)

Figure 9 illustrates the exploit guidance generated by Gemini for the Android backup misconfiguration scenario. The model correctly explains how to verify the allowBackup flag, execute the ADB backup command, convert the backup archive to a readable format, and analyse extracted files for sensitive data. The exploit procedure is complete and reproducible, resulting in a FE classification.

5.3.2. Aggregate RQ3 Results

Table 10. RQ3 results: exploit guidance outcomes across three scenarios.

Model	S1 (DIVA)	S2 (IBv2)	S3 (IBv2)
Gemini	FE	FE	FE
GPT	FE	FE	FE
Llama	PE	NE	FE
Qwen	FE	FE	FE
Trinity	PE	FE	FE

summarises exploit guidance outcomes across all three scenarios for each evaluated model. For compactness, InsecureBankv2 is abbreviated as IBv2 in Table 10 and 

Table 11. RQ3: Exploit Guidance Scores (EGS) and Exploit Success (ES) per model and scenario.

Model	S1 (DIVA)		S2 (IBv2)		S3 (IBv2)
	EGS	ES	EGS	ES	EGS	ES
Gemini	0.960	1	0.960	1	0.960	1
GPT	0.960	1	0.960	1	0.860	1
Llama	0.532	0	0.357	0	0.820	1
Qwen	0.960	1	0.960	1	0.960	1
Trinity	0.640	0	0.820	1	0.900	1

.

Table 11 reports the underlying Exploit Guidance Scores and Exploit Success indicators from which the exploit guidance classifications were derived.

For consistency with the previous research questions,

Table 12. RQ3 results: number of scenarios classified as FE, PE, and NE per model ($n=3$ scenarios per model).

Model	FE	PE	NE
Gemini	3	0	0
GPT	3	0	0
Llama	1	1	1
Qwen	3	0	0
Trinity	2	1	0

presents the aggregated number of FE, PE, and NE outcomes per model.

Within this limited set of controlled scenarios, the results indicate that several models can produce complete exploit-verification guidance when provided with concrete vulnerability evidence. Gemini, GPT, and Qwen generated FE-classified guidance for all three evaluated scenarios. Trinity achieved FE-classified guidance in two scenarios but produced partially exploitable guidance in the SQL injection scenario. Llama demonstrated the highest variability, producing one fully exploitable scenario, one partially exploitable scenario, and one case where the exploit could not be reproduced.

5.3.3. Summary of RQ3 Findings

The results suggest that, in the three controlled exploit scenarios examined in this study, selected LLMs were able to assist testers in constructing exploit procedures for Android vulnerabilities. These findings should be interpreted as exploratory evidence rather than as a general measure of LLM exploit-generation capability in Android DAST. In successful cases, the models provide concrete commands, appropriate payloads, and clear instructions for verifying exploit impact. However, exploit generation quality varies between models, and incomplete verification steps remain a common limitation preventing reliable reproduction of the exploit. It must be clearly emphasised that the aim of RQ3 is not to encourage unethical use of LLMs. The experiments and tests were conducted on deliberately vulnerable applications and were defensive in nature.

6. Discussion and Limitations

6.1. Discussion

The results show that LLM-generated guidance can support Android DAST, but the level of support is uneven across models, applications, and vulnerability types. This variation should not be interpreted solely as a consequence of model size or provider-reported architecture. For example, Trinity achieved the second-highest FD count (15 out of 43, i.e., 34.9%), whereas GPT-oss achieved a lower FD rate (16.3%) but a higher PD rate (37.2%), indicating that it often identified the correct testing direction without providing sufficient operational detail. These differences suggest that, in human-driven DAST, the practical usefulness of an LLM depends not only on the model itself, but also on whether its guidance is specific, actionable, and aligned with the available testing artefacts. The repeated-run analysis further supports this interpretation for Gemini, GPT-oss, and Trinity, as the aggregate detection rates remained stable across three independent sessions, with standard deviations between 1.3 and 2.3 percentage points. This cautious interpretation is consistent with findings that LLM-based vulnerability predictions may partly reflect shallow code-level signals rather than deep vulnerability semantics [32].

The RQ2 results suggest that prompt specificity had a substantial effect on detection rates in our Sieve-based experiment. With the advanced prompt, three models achieved 100.0% FD + PD on Sieve, compared to 25.0%–62.5% with the simple prompt. This implies that organisations should invest effort in crafting informative prompts, including application functionality, authentication mechanisms, and data sensitivity, rather than relying solely on model selection.

RQ3 provides exploratory evidence that selected models can produce complete exploit procedures with concrete commands when supplied with specific vulnerability evidence. Although the scenario count is small ($n=3$), the scenarios were deliberately selected to cover distinct and practically relevant Android DAST situations: local input-driven data extraction through SQLite injection, unauthorised access through exported application components, and extraction of private application data through an Android backup misconfiguration. Thus, RQ3 should not be interpreted as a broad statistical evaluation of LLM exploit-generation capability, but rather as a focused scenario-based assessment of whether LLMs can support reproducible PoC-style exploitation guidance for representative high-impact Android weaknesses.

In this setting, Gemini, GPT-oss, and Qwen produced guidance classified as fully exploitable for all three scenarios. A recurring limitation was the inconsistent recommendation of Drozer for IPC testing. Although Drozer was available, models frequently substituted generic ADB commands, likely reflecting limited Drozer-specific content in training corpora. This contributed to weaker outcomes for vulnerabilities requiring IPC-specific tooling. Overall, the findings suggest that LLMs can help address the research gap identified in Section 3: interactive, human-in-the-loop DAST guidance appears feasible and practically useful in the tested setting, but its quality remains model- and context-dependent and does not replace specialised tooling knowledge.

An important issue worth discussing is the ethical aspect of using LLMs. Generative AI may not necessarily create new attacker capabilities, but it can transform existing techniques by increasing their effectiveness and reducing the amount of work required. LLMs can analyse and summarise technical documentation and suggest search methods an attacker might not have considered. This reduces the cognitive load associated with reconnaissance and can accelerate the selection phase, during which potential vulnerabilities are identified. The misuse of LLMs in the context of offensive cybersecurity may lower the barrier to entry into the world of cybercrime, enabling individuals without in-depth technical knowledge to develop complex concepts (e.g., code generation or system analysis). Furthermore, the scalability and automation enabled by LLMs facilitate the execution of attacks. It must be clearly emphasised that unauthorised hacking is unacceptable and may constitute a criminal offence.

6.2. Limitations

The study has several limitations that should be considered when interpreting the results. First, all experiments used intentionally vulnerable applications with documented ground truth. These do not implement anti-analysis protections or complex business logic, so the reported detection rates likely represent an upper bound for real-world scenarios.

Second, the experiments used a fixed temperature of $0.7$ without exploring the effect of decoding parameters on guidance quality. Third, the two-turn interaction protocol may underestimate model potential, as a real-world tester would engage in a longer iterative dialogue. Fourth, generation parameters beyond temperature were not unified across providers, which limits strict cross-model comparability. However, since the research questions concern the general feasibility of LLM-guided Android DAST rather than a controlled model ranking, results are interpreted with respect to individual guidance quality rather than as rankings derived from controlled ablations.

Fifth, GPT-oss and Qwen 3 natively produce reasoning traces that were stripped before evaluation. While this ensured consistent scoring, it may have affected how providers allocated output tokens between reasoning and final guidance, potentially influencing response quality in ways not captured by the evaluation.

Sixth, model availability and serving configurations may change over time. Trinity is a preview release, Gemini is accessed through a proprietary API that may be updated without notice, and some third-party endpoints used during the experiments may later be modified or deprecated. Exact reproduction of the results cannot therefore be guaranteed.

Some limitations arise from the benchmark applications used in the research. Therefore, the current results should not be generalised directly to production Android applications. Benchmark applications often contain vulnerabilities that are less sophisticated than those found in production Android applications. However, the research focused on the DAST approach, which is more effective at identifying issues that can be exploited at runtime. Assessing applications that use code obfuscation, root/emulator detection, or anti-instrumentation using DAST tests alone is not the most effective approach. In such cases, it is usually recommended to combine DAST with SAST (static application security testing), manual review, and application-specific threat modelling. Complex vulnerabilities arising from the application’s business logic or mechanisms such as authentication or session management are particularly relevant here. In such cases, much depends on the content of the tester’s prompt. The more precise the description, the more accurately the LLM can suggest how to test a given mechanism. From the perspective of someone who understands the application’s logic, this approach is even more valuable, as such a description can contain details that allow the LLM to perform a more targeted analysis. The RQ2 prompt-quality analysis was limited to Sieve only. Although Sieve includes several vulnerability categories relevant to Android DAST, the conclusions about prompt specificity should be treated as indicative. Testing the same prompt levels across all five target applications would provide stronger evidence and is left for future work.

Another limitation is the potential subjectivity of manual step-level scoring. Although a predefined rubric and a consistent evaluation checklist were used, the colour-based distinction between partially correct, generic, and incorrect guidance may still involve evaluator judgement. The evaluation was performed by a two-person team consisting of a senior security researcher with academic and penetration testing experience and a junior/intermediate penetration tester. Ambiguous ratings were resolved by consensus; however, formal blinded scoring and inter-rater agreement metrics were not computed. Replication with a larger pool of independent evaluators would strengthen the generalisability of the conclusions.

Finally, although the RQ1 representative results were supplemented with repeated runs for three models, the sample sizes remain limited overall: 43 test cases for the representative RQ1 evaluation, three repeated sessions only for Gemini, GPT-oss, and Trinity, 8 test cases per model-prompt combination for RQ2, and 3 scenarios for RQ3. These sample sizes are sufficient to identify descriptive trends but too small for formal statistical testing. This limitation is particularly relevant for RQ3, where only three exploit-generation scenarios were evaluated and two of them were based on InsecureBankv2. Although these scenarios were deliberately selected to represent distinct Android DAST-relevant weakness classes, the RQ3 findings should be interpreted as exploratory rather than as a general benchmark of LLM exploit-generation capability.

7. Conclusions and Future Work

In this article, we investigated whether and to what extent selected LLMs can serve as guides for DAST testing of Android applications. For this purpose, applications were selected and tested using guidance obtained during interactions with the models. The results suggest that the LLMs under investigation can be helpful in such testing, but they are not sufficient for users with no prior security testing knowledge. The experiments showed that a user with even minimal security testing knowledge can make better use of the guidance provided by the LLM. A DAST tester must have a basic understanding of the Android platform (e.g., the structure of .apk files or how app components interact) and fundamental security concepts such as input validation, authentication, and session management. Additionally, it is helpful to be familiar with tools useful for conducting such tests (e.g., the Android emulator, network proxies, adb, or drozer). However, a reliable interpretation of the results still requires an understanding of the Android runtime environment and an awareness that not all vulnerabilities can be reliably detected using DAST alone (e.g., hardcoded credentials). Furthermore, the results suggest that more detailed descriptions of the test problem can improve response accuracy and increase the specificity of the suggested actions, such as commands for particular tools, for confirming vulnerabilities and, in selected scenarios, verifying their exploitability.

In the future, we plan to expand our research in this area. First and foremost, the experiments will be expanded to include new models. Additional tests will examine the effect of temperature on the number of vulnerabilities found, response variance at different temperatures, and the impact of the number of interaction rounds (i.e., longer dialogue) on the results obtained. Future work will also repeat the prompt-specificity experiment across all target applications to assess whether the Sieve-based trend generalises to a broader Android DAST benchmark. Furthermore, it is worth considering an LLM as a real-time mentor. One possible direction is the integration of the model with an IDE or DAST tool, whereby the model can observe in real time which actions the tester is performing (commands, logs, and results) and proactively suggest the next steps without requiring additional queries, which could potentially speed up testing. Such future systems could also draw on ideas from data-driven monitoring and temporal decision optimisation, where heterogeneous process observations are fused to support dynamic decision-making [38].

Future work will also extend the RQ3 exploit-generation evaluation to a larger and more diverse set of Android vulnerabilities and applications, so that scenario-based observations can be validated under broader experimental conditions.