How to Conduct AI-Assisted (Large Language Model-Assisted) Content Analysis in Information Science and Cyber Security Research

Abstract

The advent of Large Language Models (LLMs) has revolutionised natural language processing, providing unprecedented capabilities in text generation and analysis. This paper examines the utility of Artificial-Intelligence-assisted (AI-assisted) content analysis (CA), supported by LLMs, as a methodological tool for research in Information Science (IS) and Cyber Security. It reviews current applications, methodological practices, and challenges, illustrating how LLMs can augment traditional approaches to qualitative data analysis. Key distinctions between CA and other qualitative methods are outlined, alongside the traditional steps involved in CA. To demonstrate relevance, examples from Information Science and Cyber Security are highlighted, along with a new example detailing the steps involved. A hybrid workflow is proposed that integrates human oversight with AI capabilities, grounded in the principles of Responsible AI. Within this model, human researchers remain central to guiding research design, interpretation, and ethical decision-making, while LLMs support efficiency and scalability. Both deductive and inductive AI-assisted frameworks are introduced. Overall, AI-assisted CA is presented as a valuable approach for advancing rigorous, replicable, and ethical scholarship in Information Science and Cyber Security. This paper contributes to prior LLM-assisted coding work, proposing that this hybrid model is preferred over a fully manual content analysis.

1. Introduction

Generative Artificial Intelligence (AI) has advanced rapidly since late 2022, when ChatGPT 3.5 was first released to the public (30 November 2022) [1,2]. Universities have struggled to respond to the use of tools such as OpenAI’s ChatGPT in assessments, amid concerns that students may outsource work and therefore fail to master core knowledge and skills or demonstrate independent learning [3,4,5,6]. Alongside these integrity risks, it is equally important to examine how AI can be used responsibly to support scholarship—for example, to brainstorm, locate literature, assist with coding and data cleaning, improve accessibility, and scaffold writing—within clear pedagogical guidelines and assessment designs [7,8,9,10].

AI, especially large language models (LLMs), can also assist researchers across the project lifecycle [11,12,13,14,15,16,17]. Tools, such as Rayyan and related platforms, can assist in the dual coding and screening of systematic literature reviews [18,19]. For scientific writing and analysis, LLMs can enhance writing quality and reduce task time, with the most significant gains observed for writers with lower baseline skills [20,21].

This paper proposes that Artificial Intelligence (AI), particularly large language models (LLMs), can serve as a valuable tool for conducting content analysis (CA). Drawing on the disciplines of Information Science (IS) and Cyber Security, this paper illustrates, through a review of the literature, how AI-assisted CA can enhance methodological rigour, while acknowledging that the approach can be applied across a wide range of disciplines. The discussion begins by outlining the relevance of CA as a methodology for Information Science (IS) and Cyber Security, followed by an overview of its core principles. It then provides a set of guidelines to ensure validity and reliability when employing AI-assisted CA. To demonstrate the practical application of this approach, this paper presents examples of a CA of disinformation, illustrating how researchers can systematically integrate AI tools into their analysis.

1.1. Rationale

Advances in AI and natural language processing (NLP) make it timely to articulate how AI-assisted content analysis (CA) can strengthen empirical inquiry in IS and Cyber Security. Traditional CA provides a transparent and systematic approach for describing the manifest and latent features of texts [22], yet the volume, velocity, and heterogeneity of contemporary digital corpora—threat reports, incident tickets, logs, social media streams, forums, and policy documents—strain purely manual approaches common in social science methods. AI-assisted content analysis (CA) offers a potential solution to the challenges posed by ‘big data’ by enabling researchers to process large and complex datasets while maintaining methodological rigour. Importantly, such approaches can enhance the reliability and validity of research by creating reproducible audit trails, thereby supporting transparency and replicability. At the same time, AI-assisted CA retains the critical role of human oversight, ensuring that construct validity and contextual interpretation are preserved [23].IS and Cyber Security, where many researchers originate from technical or applied traditions and may have limited formal training in qualitative/quantitative social-science methodologies [24]. AI-assisted CA offers a pragmatic bridge: it encodes the procedural discipline of CA—clear units of analysis, explicit coding rules, and reliability checks—while utilising algorithms to manage scale and to standardise repetitive tasks, thereby reducing researcher burden and variance [25].

Crucially, the promise of AI-assisted CA hinges on rigorous governance to mitigate construct drift, dataset bias, and spurious correlations. Best practice includes preregistered coding protocols, reliability assessment for human and model-assisted labels, and documentation of the process [23,26,27]. Such guardrails ensure transparency and replicability—core scientific values that remain unevenly implemented in Information Science and Cyber Security—while allowing these fields to exploit AI’s strengths for scale and consistency without sacrificing theoretical fidelity or interpretability [23].

1.2. Differences Between CA and Thematic Analysis (TA) and Discourse Analysis (DA)

Content analysis (CA), thematic analysis (TA), and discourse analysis (DA) represent three prominent approaches to studying textual and communicative materials, differing in their epistemological stance, analytic focus, and intended outcomes [28,29,30]. CA is traditionally systematic and rule-based, oriented toward coding manifest or latent content into predefined categories to generate replicable and, often, quantifiable results [22]. TA, by contrast, emphasises the identification and interpretation of patterned meanings or themes across datasets, offering flexibility but typically less emphasis on replicability or quantification [31]. DA differs further, treating language not as a neutral medium but as a constitutive social practice that shapes identities, power relations, and ideologies [32]. Because CA is built on coding frameworks, explicit decision rules, and reliability testing, it aligns more directly with the affordances of AI, particularly LLMs [33,34]. AI-assisted CA can automate classification, support consistent application of coding rules, and scale analyses across large corpora, all while maintaining the possibility of reliability checks against human coders [26]. In contrast, the more interpretive and reflexive orientations of TA and DA pose challenges for automation, as they rely heavily on researcher judgment, contextualisation, and theoretical framing. These two approaches tend to also be more qualitative than quantitative. For these reasons, this paper focuses on the promise of AI-assisted CA.

1.3. Current Study

This paper examines the utility of AI, particularly LLMs, as a methodological tool for conducting CA. Situated within the field of Information Science and Cyber Security, the discussion illustrates how AI-assisted CA can strengthen methodological rigour while recognising that the approach is broadly transferable across disciplinary contexts. This paper synthesises insights from prior research and introduces a new applied example that outlines the practical strengths for implementation. In addition, it offers a critical appraisal of the strengths and limitations of AI-assisted CA, evaluating both its promise and its challenges for future research practice.

2. Defining Content Analysis

CA is a systematic, rule-governed method for making valid and replicable inferences from texts (or other meaningful material, such as photographs or other visual materials) to the contexts of their use [22,26]. At its core, it transforms qualitative materials, such as social media posts, news articles, interview transcripts, policy documents, or images, into analysable data [35,36,37]. The process involves deciding on the content that requires analysis, sampling from this content, coding, and finally drawing inferences [38,39]. Disciplines that more commonly utilise this methodology include psychology, media and communications, health, political science, and, more recently, information science [40,41] and Cyber Security research [42,43].

CA can be either quantitative or qualitative. Quantitative content analysis seeks to measure manifest or latent variables and to test hypotheses using counts, cross-tabulations, or statistical modelling [29,44]. It relies on a priori categories, carefully trained coders, and formal reliability estimation (e.g., Krippendorff’s α or Cohen’s Kappa) to measure coder agreement [22]. In contrast, qualitative content analysis is oriented toward interpreting meaning and context. It may use inductive coding to build categories from the data or integrate theory more flexibly, while still maintaining transparency through audit trails and explicit decision rules [45,46].

There are four primary purposes for conducting CA. First, it maps and describes the distribution of topics, frames, or sentiments across large corpora (e.g., how often a newspaper takes a conservative compared to a liberal perspective) and over time [47,48]. Second, it tests theory by operationalising abstract constructs—such as moral foundations, stigma, or strategic framing—into observable categories that can be associated with predictors or outcomes [49]. Third, it compares groups and contexts (e.g., focus groups, cultural groups) [39,50,51]. Fourth, it supports mixed-methods inference, such as triangulation [52] or as a starting point to guide future studies [53].

Notably, CA can be theoretically or atheoretically driven [54]. A theoretical content analysis starts from existing theory, models, and hypotheses and uses them to define categories and decision rules a priori [55]. In contrast, an atheoretical approach, often referred to as inductive content analysis, begins with minimal a priori categories and allows patterns and themes to emerge directly from the data [30,56].

As aforementioned, content analysis distinguishes between coding manifest and latent content. Manifest content refers to what is explicitly present and readily observable, such as the presence or frequency of specific words, phrases or images [57,58]. Methodologically, manifest coding prioritises operational clarity and replicability. Categories are often defined a priori with clear inclusion rules, which allows researchers or automated procedures, such as Generative AI, to code. This approach typically achieves higher intercoder reliability because coders evaluate observable features with limited discretion. The trade-off is that manifest coding can underrepresent constructs whose meaning is context-dependent. Latent content, in contrast, involves inferring the meaning behind what is written, such as identifying a ‘moral panic’ code or an attitude position that is not stated verbatim [59,60]. Because latent coding is designed to capture meaning, connotation, and underlying patterns in text, coders are required to interpret segments in light of the broader context, theoretical constructs and discourse conventions. Reliability is, therefore, more challenging for latent coding [53]. The strength of latent coding lies in construct validity, which is essentially how well a theoretical concept is measured.

CA has notable strengths: versatility across media types, alignment with theory testing, and compatibility with both qualitative and quantitative paradigms. Its limitations arise when categories are not clearly defined (needed for validity and reliability), coder training is insufficient (needed for reliability), or the contextual nuance is lost through over-reduction [47]. In brief, content analysis provides researchers with a useful tool to transform meaning-rich materials into evidence that can test theories or develop new ones.

Similarities and Differences Between Content, Thematic, and Discourse Analyses

Qualitative and mixed-methods researchers frequently draw upon content, thematic and discourse analyses for examining textual and communication materials. While these approaches share some broad similarities, it is important to distinguish the differences with clarity (see also summary in

Table 1. Similarities and differences between content, thematic and discourse analyses.

Dimension	Content Analysis	Thematic Analysis	Discourse Analysis
Epistemological orientation	Positivist/post-positivist.	Flexible—positivist/constructionist/reflexive.	Constructionist/post-modernist/critical.
Focus of Analysis	What is present in the text, frequency and distribution of categories, manifest and latent features.	Patterns of meaning—identifying, developing, and interpreting themes across the dataset.	How language works—discursive practices, rhetorical strategies, ideological functions.
Coding process	Structured codebook with explicit rules; intercoder reliability checks.	Iterative coding and theme generation; phases include familiarisation, coding, theme development and refinement.	Close reading of extracts; interpretative identification of discursive repertories; metaphors and strategies.
Reliability and Validity	Intercoder reliability measured by Krippendorff’s α or Cohen’s Kappa; Validity addressed through clear coding scheme that measure constructs and includes criterion checks.	Reliability conceptualised as transparency and reflexivity, can use interrater reliability or member checking and peer debriefing; requires an audit trail.	Reliability achieved through rigorous, systematic methods, clear research questions and intersubjectivity shared by other researchers; validity concerns include the accuracy of the reflection.
Outputs	Quantitative summaries; descriptive statistics; cross-tabulations; visualisations of patterns.	Thematic maps, rich narrative accounts of themes, supported by illustrative extracts; tabulations; visualisations of patterns.	Critical interpretations of discourse practices; accounts of ideology, power, and identity construction.
Strengths	Scalability, replicability, ability to combine with statistical analysis.	Flexibility, accessibility, capacity to capture patterned meanings.	Deep contextual insight; capacity to interrogate power, identity and ideology.
Limitations	Can overlook nuance; risk of reductionism, especially if categories are poorly defined.	Risk of superficiality if rigour is not applied; themes may reflect researcher bias.	Limited replicability; findings are typically not generalisable.

). The choice of method should be guided by the researcher’s aims, objectives, and specific research questions, as well as by the epistemological paradigm underpinning the study—for example, a positivist orientation versus a postmodern or constructionist perspective [61,62].

At a general level, all three methods provide systematic frameworks for moving from raw textual data toward more abstract interpretations of meaning. Each involves processes of coding, whether this is operationalised as assigning categories to manifest or latent content (content analysis), clustering codes into themes (thematic analysis), or identifying discursive repertoires and linguistic strategies (discourse analysis) [63]. Moreover, all approaches rely on transparent documentation of coding decisions, reflexivity about the researcher’s interpretive role, and explicit links between analytic claims and textual evidence [63,64].

One of the primary distinctions lies in their epistemological foundations. CA is traditionally rooted in a positivist/post-positivist orientation, aiming to produce replicable, reliable measures of manifest and latent variables within texts [22]. In contrast, thematic analysis is more flexible and can be applied across positivist, critical realist, or constructionist paradigms, with reflexive thematic analysis emphasising researcher subjectivity and meaning-making [31,65]. Discourse analysis, particularly in traditions such as critical discourse analysis or discursive psychology, is explicitly constructionist, treating language as a social practice that constructs realities and reproduces power relations [32].

Another key distinction between these qualitative methods lies in their analytical focus. CA focuses on the content (what is written) and the frequency and distribution of categories or codes, often producing quantitative summaries alongside qualitative interpretation [22]. In contrast, thematic analysis seeks to identify patterns of meaning across a dataset, offering rich, narrative accounts of themes and how they relate to one another and to broader research questions [65]. Discourse analysis, by contrast, investigates how language is used to perform actions, construct subjectivities, and embed ideologies, often attending to grammar, metaphors, intertextuality, and rhetorical strategies [32].

The procedures for carrying out these analyses also differ. CA typically employs a structured coding frame, explicit decision rules, and intercoder reliability checks, sometimes incorporating computational tools for large-scale text analysis [22]. Thematic analysis can be more interactive, iterative, and flexible, with phases including data familiarisation, code generation, theme development, and refinement. Like CA, inter-coder reliability can be important; however, in the more modernised version of thematic analysis, reliability is conceptualised in terms of reflexivity and transparency [31,66]. Prescriptive procedures are not required for discourse analysis, which emphasises the close reading of extracts, contextualisation within broader social and institutional structures, and theoretically informed interpretation of how discourse enacts power and identity [32].

3. The Utility of CA for Information Science and Cyber Security Research

Information Science and Cyber Security are closely linked through their shared focus on information as both a resource and an asset to be protected. Information Science examines the creation, organisation, retrieval, and use of information within socio-technical systems [67,68,69], while Cyber Security seeks to safeguard that same information and the systems that process it from unauthorised access, disruption, or misuse [69]. Their connection is evident in overlapping concerns with privacy, trust, data governance, and human behaviour. However, as argued by von Solms and Niekerk (2013) “traditional information security to include not only the protection of information resources, but also that of other assets, including the person him/herself [69]. In information security, reference to the human factor usually relates to the role(s) of humans in the security process” (p. 97). It is argued here that as emerging technologies such as cloud computing, AI, and digital infrastructures continue to blur disciplinary boundaries, collaboration between Information Science and Cyber Security has become essential for designing environments that are not only efficient and usable but also secure and trustworthy.

In Information Science, CA may enable research on how information is organised, communicated, and used across platforms and communities. Researchers can assess metadata and case studies, information behaviour acquired through interviews, and observational studies, and examine policies (e.g., organisational, defence, intelligence and government).

With respect to Cyber Security, CA could be used to examine phishing emails, security awareness materials, threat-intel reports, policy and regulatory guidance. More broadly, CA could be used to support threat and vulnerability landscape mapping, policy and governance analysis, human factors and security culture and malicious communication analysis.

Examples of how CA has been used in Information Science and Cyber Security research are shown in

Table 2. Examples of Content Analysis in Information Science and Cyber Security Studies.

Study	Materials	Methods	Key Findings
Trends in information behaviour research 1999–2008: A content analysis [70]	749 articles.	CA on human information literature using the searchers ‘information needs’ and ‘information uses’.	Overtime, scholarly researchers increased and practitioners decreased.
Content analysis of cyber insurance policies: how do carriers price cyber risk? [71]	235 documents.	CA on the losses covered by cyber insurance policies and those excluded; the questions carriers pose to applicants in order to assess risk; and how cyber insurance premiums determined.	Most important firm characteristics used to computer insurance premiums were the firm’s asset value base rate, rather than specific technology or governance controls.
What security features and crime prevention advice is communicated in consumer IoT device manuals and support pages? [72]	Manuals and associated support pages for 270 consumer IoT devices produced by 220 different manufacturers.	CA on examined security features.	Manufacturers do not provide enough information about the security features of their devices.
Twenty-five years of cyber threats in the news: A study of Swedish newspaper coverage (1995–2019) [73]	1269 newspaper articles.	CA examined threats along several dimensions: modality, ambiguous themes, how threat has changed over time and event orientation.	Swedish papers cover multiple threats, hacking has multiple meanings, coverage has changed over time.
An investigation of the impact of data breach severity on the readability of mandatory data breach notification letters: Evidence from U.S. firms [74]	512 data breach incidents from 281 U.S. firms across 2012–2015.	CA examined data breach severity attributes and readability measures.	Data breach severity has a positive impact on reading complexity, and a negative impact on numerical terms.
Scamming higher ed: An analysis of phishing content and trends [75]	2300 phishing emails from 2010 to 2023.	CA examined persuasive and emotional appeals, topics, linguistic features and metadata	There has been a shift in the topics in phishing emails; persuaded using sources of authority, scarcity and fear appeals.

.

3.1. Steps Involved in a Traditional CA

3.1.1. Reliability and Validity

Ensuring that research is both valid and replicable requires strict adherence to established methodological procedures. This is particularly critical in Information Science and Cyber Security, where many researchers are trained in technical or applied domains but often lack systematic exposure to methodological traditions common in the social sciences [24]. The absence of such training risks undermining the rigour of empirical inquiry, leading to studies that may be difficult to replicate, insufficiently transparent, or methodologically inconsistent. Without careful engagement with established research methods, findings in these fields risk being interpreted as ad hoc or lacking in scientific credibility [22]. Accordingly, cultivating methodological literacy is not merely a matter of academic thoroughness but a prerequisite for producing research that withstands critical scrutiny and contributes meaningfully to interdisciplinary scholarship.

3.1.2. Research Questions, Coding Frames and Unit of Analysis

As stated in this paper, CA is a systematic and replicable method for examining communication materials to identify patterns, themes, and meanings [26], which follow a series of steps (see

Table 3. Steps involved in carrying out a CA.

Step	Description
Define research questions	Specify focused questions that the analysis will address.
2. Select Content	Identify the corpus and choose an appropriate sampling strategy.
3. Select sample of content	A sample of the corpus.
4. Determine units of analysis	Decide the textual unit (e.g., word, sentence, paragraph, document).
5. Develop coding scheme	Construct clear, mutually exclusive and exhaustive categories (deductive and/or inductive).
6. Train coders and code	Provide coder training; apply coding rules systematically and consistently.
7. Assess reliability	Evaluate intercoder agreement (e.g., Cohen’s K) and refine the scheme if needed.
8. Analyse coded data	Conduct descriptive/quantitative summaries and/or qualitative pattern interpretation.
9. Interpret and report	Link results to questions and theory, discuss implications, limitations, and transparency of procedures.

). The process typically begins with the formulation of clear research questions that guide the analysis, followed by the development of a coding framework. Next, the content that will be examined is considered. Defining the unit of analysis—whether words, sentences, paragraphs, or entire documents—is the next crucial step, as it shapes the scope and granularity of the findings [22]. Researchers must then select an appropriate sampling strategy to ensure the representativeness of the material under investigation.

3.1.3. Coding Scheme

Once materials are collected, researchers construct a coding scheme based on either deductive categories derived from theory or inductive categories emerging from the data itself [76]. Coders then systematically assign codes to segments of text, ensuring that coding rules are applied consistently across the dataset. To maintain rigour, inter-coder reliability checks are often conducted, using measures such as Cohen’s Kappa (0.41–0.60 moderate, 0.61–0.81 substantial and 0.81–1.0 almost perfect) or Krippendorff’s, α $\ge$ 0.80 for confirmatory studies, and 0.67–0.80 $\ge$ for explanatory work, to assess the level of agreement among independent coders [77].

3.1.4. Analysis and Interpretation

After coding is complete, the data are analysed to identify frequency distributions, co-occurrences, and patterns that reveal the underlying structure of the communication. Depending on the epistemological orientation, the analysis may emphasise quantitative aspects—such as the frequency of word categories—or qualitative interpretations that explore meanings and latent content [53]. The final stage involves interpreting the findings in light of the research questions, theory, and broader context, and presenting them in a way that enhances understanding of the phenomena under study [22].

4. Employing AI-Assisted CA

In the last few years, researchers have increasingly used large language models (LLMs) to automate pieces of the CA workflow—building or adapting codebooks, coding text at scale, and serving as ‘second coders’ for reliability checks. Several empirical studies report that for certain well-specified tasks (e.g., sentiment, simple categorical codes), modern LLMs can reach agreement levels comparable to human coders, though performance varies by construct and model, and reliability on subtle, latent constructs (e.g., sarcasm) remains uneven [78,79]. For example, Bojić et al. (2025) employed multiple LLMs with 33 human annotators and found that humans and leading LLMs were similarly reliable for sentiment and political leaning; however, both struggled on sarcasm [78].

One of the obvious benefits of AI-assisted CA is a saving on researchers’ time and resources [80,81,82]. Morgan (2023) found that ChatGPT conducted a thematic analysis in 2 h, which compared to 23 h when human coders conducted the same task [80]. Similarly, Fuller et al. (2024) reported that ChatGPT was able to analyse 470 free-text survey responses within 10–15 min, whereas human researchers required an average of 27.5 min to complete the same task [81].

A natural concern is whether the substantial reduction in analysis time achieved through AI-assisted content analysis (CA) comes at the cost of accuracy. As noted earlier in this paper, current evidence suggests that AI-assisted CA can achieve levels of reliability comparable to human coding, and this reliability is expected to improve as the technology continues to advance [83]. For example, Bijker et al. (2024) employed ChatGPT to analyse forum posts in which individuals shared their experiences of reducing sugar consumption and found that inter-rater reliability with a human coder was almost perfect [83]. Nevertheless, the accuracy of AI coding remains contingent on the type and complexity of data being analysed, with some constructs—particularly latent or nuanced categories—posing greater challenges. Importantly, methodological refinements, such as providing contextual information and iteratively tailoring prompts, have been shown to improve reliability [84].

AI-assisted content analysis (CA) has the potential to generate novel insights into data beyond those typically achieved through manual approaches. For instance, AI systems can be prompted repeatedly to yield alternative interpretations of the same dataset, thereby exposing different analytical angles [85]. Moreover, they can be directed to apply specific theoretical or methodological lenses, enabling researchers to explore data through multiple perspectives in a systematic and scalable manner [85]. Hitch (2024) has suggested that AI could be used as a tool to supplement human coding as part of a reflexive, collaborative approach to analysis [85]. Recent empirical work illustrates these strengths while highlighting the importance of methodological safeguards. Yan et al. demonstrated that ChatGPT not only improved coding efficiency and supported initial data exploration but also provided granular quantitative insights—even for users with less methodological experience—underscoring the need for transparent explanatory mechanisms and iterative validation in human–AI workflows [86]. Similarly, Turobov et al. (2024) found that ChatGPT can serve as a capable assistant in thematic analysis, improving methodological efficacy when used in a validated, transparent, and reliability-conscious manner [87].

Examples of studies that have employed AI-assisted content analysis are shown in

Table 4. Steps involved in carrying out AI-assisted CA.

Study	Materials	AI Role in CA	Findings & Conclusions	Type AI
Beyond manual media coding: Evaluating large language models and news agents for news content analysis [88]	200 news articles on U.S. tariff policies.	7 LLMs assessed under a unified zero-shot prompt.	Strong reliability. Structured prompting and agentic reasoning improved accuracy. Human-in the loop validation may be essential to ensure reliability.	not stated
Comparing large language models and human annotators in latent content analysis of sentiment, political leaning, emotional intensity and sarcasm [78]	100 curated textual items.	33 human annotators; 8 LLM variants	Both humans and most LLMs exhibit high inter-rater reliability in sentiment analysis and political leaning assessments. LLMs demonstrate higher reliability than humans. Both groups struggled with sarcasm detection.	GPT-4, Gemini, Llama 3.1-70B and Mixtral 8 x7B
Using large language models for narrative analysis: A novel application of generative AI [82]	138 short stories written by young people about social media, identity formation, and food choices.	Analysed by humans researchers and two different LLMs (Claude and GPT-01).	LLMs were fast and comparable to human researchers. LLMs provided additional insights that enhanced the analysis.	ChatGPT (version not stated)
LLM- Assisted Content Analysis: Using large language models to support deductive coding. [79]	4 publicly available datasets.	Compared GPT 3.5 with human researchers.	GPT 3.5 can perform deductive coding at levels similar to humans. GPT4.5 can help refine prompts for deductive coding.	GPT 3.5
LLM-Assisted qualitative data analysis: Security and privacy concerns in gamified workforce studies [89]	23 Interview transcripts.	Compared LLaMA, Gemma and Phi.	LLaMA excelled in detailed security insights. Gemma focuses on compliance. Phi focuses on ethical transparency Useful as a preliminary step to enhance efficiency of more traditional qualitative analysis methods.	LLaMA, Gemma, and Phi.
Coding latent concepts: A human and LLM-coordinated content analysis procedure [90]	1000 public comments.	Compared human researchers with GPT-4o and GPT-3.5-turbo-1106.	Fine tuned GPT 3.5-turbo with smaller datasets can surpass GPT 4o’s performance.	GPT3.5 and GPT-4o

. As shown in Table 4, the studies employed a range of LLMs—although some do not report which models. Bojić et al. (2025) reported that LLMs and humans struggled with coding for sarcasm [78]. Inter-coder reliability was typically considered by carrying out a statistical test, such as Krippendorff’s alpha or Cohen’s κ.

4.1. Responsible AI

There has been much discussion in recent times about Responsible AI. Responsible AI refers to the principles and governance practices that ensure AI systems are developed and used in ways that are lawful, ethical, transparent, safe, and accountable [91,92]. The OECD Principles on AI (OECD, 2019) emphasise inclusive growth, human-centred values, transparency, robustness, and accountability, while the European Commission’s Ethics Guidelines for Trustworthy AI (EU, 2019) outline seven core requirements, including human agency, technical robustness, data governance, and societal well-being [91,92].

In empirical settings, Responsible AI considerations often surface when unintended harms become evident. For example, Buolamwini and Gebru’s well known study on facial recognition revealed high error rates for women and people with darker skin tones, underscoring the need for fairness audits and bias mitigation [93]. Similarly, studies on predictive policing algorithms highlight risks of amplifying structural inequalities unless explicit governance interventions are applied [94]. These cases demonstrate that Responsible AI is not simply aspirational but necessary to prevent harms in high-stakes contexts [94].

Arguably, Responsible AI also reinforces research-ethics obligations when CA involves informing real world decisions, such as policy around securing a nation state. It also may involve careful consideration of texts containing personal or sensitive information. Privacy-respecting data governance—including careful sampling, minimisation, lawful basis for processing, and attention to contextual integrity—is essential, particularly when scraping or analysing user-generated content [95]. Equally important is the accountability of employing AI-assisted CA and concerns around validity and reliability. Researchers need to clearly set out the data used, the procedures employed in their studies, the development of a codebook and evidence of intercoder reliability [26]. In summary, Responsible AI provides the normative and procedural infrastructure that allows AI-assisted CA to scale analysis while safeguarding validity, fairness, and reproducibility. Embedding these practices moves AI-assisted CA beyond convenience automation toward theory-consistent, ethically defensible, and auditable research that can withstand interdisciplinary scrutiny in Information Science and Cyber Security.

4.2. Hallucination Phenomenon and Error Mitigation

One important limitation of AI-assisted CA is the hallucination phenomenon, a well-documented challenge in the use of LLMs. As Ahmadi observes (2024), “as these language juggernauts sift through immense volumes of data, the line between accurate interpretation and creation generation can blur” (p. 1) [96]. In practice, generative AI systems can produce content that appears linguistically fluent and conceptually plausible but is ultimately nonfactual [97,98,99]. This poses a particular risk for researchers, as outputs may be accepted at face value without sufficient domain expertise to recognise inaccuracies or fabrications. For AI-assisted CA, such hallucinations have direct implications for both reliability and validity: reliability may be undermined if the same input generates inconsistent outputs across iterations, while validity is threatened when fabricated or distorted information is coded and treated as empirical evidence. Addressing this limitation requires rigorous oversight, triangulation with human coders, and transparent reporting of verification procedures.

4.2.1. Verification Playbook

It is also important to consider the broader issue of ‘model fallibility’ [100]. LLMs are neutral coders but probabilistic systems that generate outputs based on statistical associations rather than grounded understanding. This necessitates the introduction of verifiable controls to safeguard the integrity of AI-assisted CA. Recommended practices include conducting cross-run agreement checks to ensure stability across repeated model queries, using evidence-anchored prompts that require citation of specific text spans from the source corpus, and enforcing a strict rule that no model assertion is accepted unless it links directly to a verifiable snippet in the underlying data. To operationalise these safeguards, researchers can employ a ‘verification playbook’, which documents common failure types, such as hallucinated references, misclassification of nuanced categories, or omission of contextual qualifiers and prescribes targeted countermeasures for each. Such measures not only strengthen the reproducibility and credibility of AI-assisted CA but also make explicit the limitations of LLMs, underscoring the need for human oversight and methodological transparency.

4.2.2. Errors Related to Cyber Security and Information Science

In the domain of Cyber Security, AI-assisted analysis faces unique risks stemming from the highly dynamic and adversarial nature of the field. Edge cases include fabricated indicators of compromise (IoCs) inserted into threat reports and possibly synthetic vulnerability identifiers that mimic established formats without grounding in authoritative registries. If unverified, such artefacts risk being coded as legitimate, undermining both the validity of findings and the reliability of subsequent analyses. This demonstrates the importance of the human-in-the-loop for CA—especially in these disciplines.

4.3. Hybrid Approach

Currently, a hybrid approach that combines the strengths of humans is recommended, given the current state of generative AI and large language models (LLMs). As demonstrated in this paper, the combination of AI and human expertise may improve accuracy and reduce the time required to conduct this type of research. Such an approach allows researchers to leverage the efficiency of AI-assisted content analysis (CA) while ensuring that its application remains responsible and ethically grounded [82]. While LLMs can rapidly process large volumes of text, generate coding suggestions, and reveal alternative interpretive frames, human coders remain essential for safeguarding construct validity, identifying hallucinations, and ensuring that coding decisions align with theoretical frameworks and empirical realities. This balance mitigates risks associated with over-reliance on automation and ensures that AI outputs are subject to critical scrutiny, verification, and contextual interpretation. In this sense, hybrid approaches not only embody ethical and responsible practice but also enable CA to benefit from the scalability of AI while preserving the interpretive depth and validity that underpin rigorous social science research. There may, of course, be some caveats to choosing AI-assisted CA. In cases where complex themes, irony, and sarcasm require coding, a manual approach may be preferred.

5. Stages Recommended for an AI-Assisted CA

Drawing from the research summarised in this paper, a workflow hybrid pipeline with humans leading the research has been outlined (see

Table 5. Deductive approach—AI-Assisted Content Analysis.

	Step1: Define research questions Who: Human researcher/s Process: Develop research questions based on theory and previous empirical research.
	Step 2: Select content Who: Human researcher/s Process: Decide on the population of text or data that will be analysed (e.g., zines, books, newspaper articles, social media posts).
	Step 3: Sample Who: Human researcher/s Process: Decide on a sample from the corpus.
	Step 4: Determine units of analysis Who: Human researcher/s Process: decide on whether to code words, sentences, paragraphs, visuals, memes, documents, etc.
	Step 5: Develop coding scheme and codebook Who: Human researcher/s and LLM/s Process: Human researcher/s first generate categories or codes with definitions informed from research questions, previous research and theory. They also outline the inclusion and exclusion criteria. LLM/s generate own set of categories or codes after being feed research questions, relevant theory, and key findings from the literature. Also outlines the inclusion and exclusion criteria. Human researcher/s make the final decision on the coding scheme considering also the LLMs output. Human researcher/s set up the codebook with categories/codes, definitions and inclusion/exclusion criteria (ideally with examples and counterexamples)—typically in a spreadsheet.
	Step 6: Train coders and code and verification playbook Who: Human researcher/s and LLM/s Process: Human researcher/s and LLMs code independently with the same category/code definitions. Start with a sample and compare results. Human oversight of AI codes. Consider refining definitions. Repeat process until a final set of definitions is obtained. Humans have oversight of the final codes. Both human researcher/s and LLMs code full dataset. Employ a verification playbook to record LLMs failure types, hallucinated references and misclassification of nuanced categories. Can compare with other LLMs to increase reliability.
	Step 7: Assess reliability and verification playbook Who: Human researcher/s Process: Carry out a statistical analysis comparing the human researcher/s and LLMs findings (e.g., using Krippendorff’s α or Cohen’s Kappa).
	Step 8: Analyse coded data Who: Human researcher/s Process: Carry out the qualitative or statistical analysis on the codes. Qualitive analysis involves a pattern summary and quantitative involves carrying out appropriate statistical analysis.
	Step 9: Interpret and report Who: Human researcher/s and can also use LLM/s (with caution) Process: Independently interpret the data. Support LLMs interpretation by feeding research questions, theory, previous key findings, and results. Compare interpretations. Check LLMs interpretation by finding evidence in the academic literature and be mindful of potential hallucination bias. Human researcher/s to make the final decision on the interpretation and write the report.

and

Table 6. Inductive approach—AI-Assisted Content Analysis.

	Step1: Define research questions Who: Human researcher/s Process: Develop research questions based on gaps in the literature.
	Step 2: Select content Who: Human researcher/s Process: Decide on the population of text or data that will be analysed (e.g., zines, books, newspaper articles, social media posts).
	Step 3: Sample Who: Human researcher/s Process: Decide on a sample from the corpus.
	Step 4: Determine units of analysis Who: Human researcher/s Process: decide on whether to code words, sentences, paragraphs, visuals, memes, documents, etc.
	Step 5: Develop coding scheme and codebook Who: Human researcher/s and LLM/s Process: Human researcher/s read and re-read the data to gain a deep understanding of its context and nuances. An initial set of codes and definitions are developed that address the research questions. LLMs are feed the research questions and determine a first set of codes and definitions. Human researcher/s compare findings, re-think the categories or codes and cluster where appropriate. Human researcher’s and LLMs then independently undertake a second round of coding to develop subcategories if appropriate and more fine-grained codes (considering inclusion and exclusion criteria). Again, human researcher/s examine sets of findings and make the final decision on the coding scheme. They also set up the codebook with categories/codes, definitions and inclusion/exclusion criteria (ideally with examples and counterexamples)—typically in a spreadsheet.
	Step 6: Train coders and code Who: Human researcher/s and LLM/s Process: Human researcher/s and LLMs code independently with the same category/code definitions. Start with a sample and compare results. Consider refining definitions. Repeat process until a final set of definitions is obtained. Both human researcher/s and LLMs code full dataset. Employ a verification playbook to record LLMs failure types, hallucinated references and misclassification of nuanced categories. Can compare with other LLMs to increase reliability.
	Step 7: Assess reliability Who: Human researcher/s Process: Carry out a statistical analysis comparing the human researcher/s and LLMs findings (e.g., using Krippendorff’s α or Cohen’s Kappa). Alternatively, reliability could be strengthened through coder triangulation, reflexivity, and iterative consensus-building.
	Step 8: Analyse coded data Who: Human researcher/s Process: Carry out the qualitative or statistical analysis on the codes. Qualitive analysis involves a pattern summary and quantitative involves carrying out appropriate statistical analysis.
	Step 9: Interpret and report Who: Human researcher/s and can also use LLM/s (with caution) Process: Independently interpret the data. Support LLMs interpretation by feeding research questions, theory, previous key findings, and results. Compare interpretations. Check LLMs interpretation by finding evidence in the academic literature and be mindful of potential hallucination bias. Human researcher/s to make the final decision on the interpretation and write the report.

). In this recommended approach, humans lead research question formation and codebook development and LLMs support the generation of categories/codes, the coding, and their definitions and the interpretation.

5.1. Deductive AI-Assisted CA

5.1.1. Research Questions, Population Texts, Samples and Units of Analysis

In a deductive approach (see Table 5), the process begins with the formulation of research questions by human researchers, grounded in theoretical frameworks and prior empirical evidence. Researchers then identify the population of texts to be analysed and select an appropriate sample from this corpus in line with methodological requirements. The next step involves determining the units of analysis—such as words, sentences, paragraphs, documents, or visual artefacts—which are defined by the human researchers as the basis for coding.

Once the units are established, both human researchers and LLMs contribute to the development of the coding scheme. Human researchers initially generate categories or codes with accompanying definitions, informed by research questions, theory, and prior studies. They also specify inclusion and exclusion criteria. In parallel, LLMs are prompted with the same research questions, theoretical perspectives, and key findings from the literature to generate their own proposed categories, definitions, and criteria. The human researchers critically evaluate these outputs and make the final decision on the coding scheme. The human researchers then construct a formal codebook, typically in a spreadsheet, containing the categories, definitions, and inclusion/exclusion rules, ideally supplemented with illustrative examples and counterexamples.

5.1.2. Coding and Reliability

Coding is carried out by both humans and LLMs through an iterative process of training and refinement. A subsample of data is independently coded using the same scheme, and the outputs are compared to identify discrepancies and refine category definitions. This cycle continues until satisfactory consistency is achieved. Once the scheme is stabilised, the entire dataset is coded. Reliability is subsequently assessed statistically by comparing agreement between human coders and LLM outputs, using established measures such as Krippendorff’s α or Cohen’s κ.

5.1.3. Synthesising or Quantifying and Interpreting

The next stage of analysis depends on whether a qualitative or quantitative approach is chosen. Qualitative analysis concentrates on identifying and synthesising thematic patterns within the data, while quantitative analysis employs suitable statistical techniques to examine relationships, frequencies, or distributions across codes. Ultimately, the interpretation and reporting of findings are primarily carried out by human researchers. Although LLMs can be cautiously used to generate alternative interpretive perspectives—when provided with research questions, theoretical frameworks, and results—such outputs must be critically validated against the academic literature to reduce the risk of hallucination. In the end, responsibility for interpretation and reporting lies with human researchers to ensure the validity and integrity of the analysis.

5.2. Inductive AI-Assisted CA

5.2.1. Immersion, Code Development and Definitions

The Inductive AI-Assisted CA proposed here is very similar to the Deductive approach (Table 6). It differs at Stage 5, where human researchers begin by reading and re-reading the dataset to develop a deep understanding of its context and nuances. From this immersion, they generate an initial set of codes and definitions that address the research questions. In parallel, large language models (LLMs) are provided with the research questions and tasked with producing a preliminary set of codes and definitions. The outputs from humans and LLMs are then compared, enabling researchers to reconsider, cluster, and refine the categories where appropriate.

5.2.2. Developing a Codebook

Subsequently, both human researchers and LLMs conduct a second round of independent coding, which may involve the development of subcategories or more fine-grained codes. At this stage, inclusion and exclusion criteria are further clarified. The human researchers critically examine the combined sets of findings, make final decisions regarding the coding scheme, and resolve any discrepancies. The definitive codebook is then produced, typically in a spreadsheet, and includes the categories or codes, their operational definitions, and explicit inclusion and exclusion criteria, ideally supplemented with illustrative examples and counterexamples.

5.2.3. Reliability and Reflexivity

Stage 7 in this workflow differs slightly from the deductive AI-assisted CA process. While reliability may still be assessed quantitatively using statistical measures, it can alternatively be enhanced through qualitative strategies such as coder triangulation, reflexivity, and iterative consensus-building. These approaches place greater emphasis on collective judgment and critical reflection, thereby strengthening the robustness of the findings.

5.3. Practical Implications

The steps set out in Table 5 and Table 6 are a general guide. In practical terms, researchers may need to adjust these steps according to the type of data being examined. Piloting for more complex data may be important, and researchers may wish to obtain confidence scores and take these into account when assessing the reliability. Careful planning of human time, taking into account the complexity and corpus sizes, will also need to be considered.

5.4. Apply in Information Science and Cyber Security Research

While the hybrid workflows outlined above are broadly applicable across disciplines, AI-assisted content analysis is argued to hold promise for researchers in Information Science and Cyber Security, where the scale, complexity, and velocity of textual data demand innovative methodological solutions. Both disciplines frequently involve working with large, fast-moving, and heterogeneous textual corpora, including technical reports, threat intelligence feeds, logs, social media discussions, and policy documents [101]. Traditional, fully manual CA is often infeasible in these contexts due to the scale and velocity of the data. By leveraging large language models (LLMs), researchers can expedite the coding of massive datasets, identify emergent patterns, and even simulate multiple interpretive frames, thereby supporting richer and more timely analyses [86]. For Information Science, AI-assisted CA can enhance understanding of how information is produced, shared, and trusted across digital networks, while in Cyber Security it provides tools to monitor online threat landscapes, detect persuasive techniques in malicious communication (e.g., phishing, disinformation), and analyse the diffusion of security-relevant information [15,102,103].

5.5. Limitations

There are notable limitations that also require critical reflection. Most prominently, the hallucination phenomenon—where LLMs generate fluent but factually incorrect or unsupported content—poses acute risks in these domains [99]. In Cyber Security, hallucinations could manifest as fabricated technical indicators of compromise, misattributed attack vectors, or non-existent vulnerabilities, leading to flawed or even harmful intelligence reporting. In Information Science, hallucinations may distort analyses of trust, misinformation, or communication flows by producing spurious themes or invented references. Such errors can undermine both reliability—if repeated runs generate inconsistent categories—and validity, when fabricated content is mistakenly treated as empirical evidence.

Additional challenges include the risk of algorithmic bias, particularly when LLMs are trained on data that underrepresents technical jargon, non-Western threat actors, or niche information ecosystems, thereby skewing the resulting codes or themes [104]. The opacity of LLM decision-making also raises concerns for transparency and replicability, key principles of scientific inquiry [104]. Finally, over-reliance on AI tools may encourage researchers to under-engage with the interpretive and theoretical dimensions of CA, which are essential for contextualising findings in socio-technical settings.

At present, given the limitations of AI-assisted CA for highly nuanced constructs with low inter-rater reliability, human content analysis is still preferable. However, that is likely to change in the near future as AI continues to develop. Additionally, for legal and safety concerns related to some data addressing legal or safety research questions, human coders may be preferred. As a further note, there are nuances between the different types of LLMs. Whilst a discussion of these differences is beyond the scope of this paper, future reviews and AI-assisted content analyses may consider these differences.

6. Practical Example

For demonstration purposes, a small deductive AI-assisted CA was run on a subset of a database of disinformation and genuine news posts about the pandemic sent during the time of the pandemic [105] that had previously been coded in a thematic analysis of persuasive themes informed by the Elaboration Likelihood Model (ELM) [106]. ChatGPT 5.0 was used to code two of the themes previously coded by two researchers. The data included anonymised Facebook posts.

The first theme was ‘emotional appeals’. ChatGPT was prompted with the exact definition that the researchers used to code for this theme: “An approach used to affect the emotions of others (for example, but not excluding other emotional appeals, fear, sadness, joy, humour, love, beauty, security)”. It was told to search for phrases or words. ChatGPT was also asked to report its confidence levels in its decision, categorising them as low, high, and medium, and provide text extracts to support its answer. In its first iteration, it picked up 62.5% of the same coding as the researchers. This elicited a Cohen’s Kappa of 0.45, which was moderately reliable. The ones that were not coded as finding this theme were all said to have low confidence. The ones that had not been coded were compared with the original coding by the researchers, and it was found that ChatGPT had missed picking up emotions associated with anxiety, stress and depression. It was then asked to recode to also code for text which mentioned anxiety, stress and depression. This time it reached 95% agreement. This elicited a Cohen’s Kappa of 0.90, which is almost perfect inter-rater reliability. The one that ChatGPT identified as having this theme, which the coders did not, was re-examined, and it was found that the coders had missed this text. The text that had been coded was also compared with the texts the coders identified, and it was found that the researchers and coders were considering the same text extracts.

The second theme was repetition. The definition this time was “repeated phrases (can be verbatim or similar)”. It was also prompted to exclude generic terms like ‘COVID’, ‘vaccine’, ‘United States’, or trivial function phrases. There were two discrepancies; ChatGPT coded two of the posts as repetition that the coders had not. A closer inspection revealed that ChatGPT had chosen a trivial phrase for ‘in other words’ but a relevant one for the other ‘1151 claims’ that the coders had missed.

This example demonstrates the importance of keeping the ‘human-in-the-loop’ with AI-assisted CA as well as the utility of this tool (given identified themes missed by the coders). It may, therefore, serve to improve reliability. Notably, this was with a small dataset and two relatively uncomplicated themes. Nonetheless, it does highlight the utility of this in content analysis coding.

7. Conclusions

This paper has advanced the case for adopting a hybrid approach to AI-assisted content analysis (CA), with particular application in the fields of Information Science and Cyber Security. Compared to disciplines such as the social sciences, researchers in these areas are often less familiar with CA as a methodological tool. Nonetheless, the approach offers considerable potential to address a wide range of research questions, from the analysis of information flows to the study of online harms and cyber threat communications.

At the same time, concerns regarding validity and reliability must be taken seriously—even, and perhaps especially, in these technically oriented disciplines. To address this, this paper presented a framework for conducting AI-assisted CA that places human researchers firmly in the driver’s seat. The model sets out a hybrid workflow process, with clearly defined steps to guide researchers in conducting research that is valid, reliable, replicable, transparent, and ethical. Particular attention was given to issues of Responsible AI, including the risks of over-reliance on automated tools, which may discourage researchers from engaging with the interpretive and theoretical dimensions of CA. The discussion also underscored the problem of hallucination, whereby large language models generate plausible but nonfactual content, posing risks to both the reliability and validity of findings. Caveats around also raised regarding the use of AI-assisted CA, particularly in cases where complex themes, such as irony and sarcasm, necessitate a manual approach.

Future methodological work should prioritise benchmarking LLM agreement on complex constructs, developing robust approaches for multi-coder reliability, and establishing transparent protocols for managing version drift. By foregrounding these challenges, this paper positions itself not as a fixed recipe but as a foundation for a broader research agenda that advances the rigour and reproducibility of AI-assisted content analysis. The proposed framework should not be viewed as a definitive solution but rather as a foundation that will evolve alongside technological advances. As AI models improve, workflows for integrating them into CA may need to be redeveloped and refined. To preserve validity and integrity, future adaptations must remain anchored in Responsible AI practices. These include rigorous reliability testing, transparent documentation, triangulation with human coding, and the critical evaluation of outputs against existing theory and literature. By embedding such practices, researchers in Information Science and Cyber Security can leverage the advantages of AI-assisted CA while safeguarding the standards of robust and ethical scholarship.