Empowering End-Users with Cybersecurity Situational Awareness: Findings from IoT-Health Table-Top Exercises

Abstract

End-users in a decision-oriented Internet of Things (IoT) healthcare system are often left in the dark regarding critical security information necessary for making informed decisions about potential risks. This is partly due to the lack of transparency and system security awareness end-users have in such systems. To empower end-users and enhance their cybersecurity situational awareness, it is imperative to thoroughly document and report the runtime security controls in place, as well as the security-relevant aspects of the devices they rely on, while the need for better transparency is obvious, it remains uncertain whether current systems offer adequate security metadata for end-users and how future designs can be improved to ensure better visibility into the security measures implemented. To address this gap, we conducted table-top exercises with ten security and ICT experts to evaluate a typical IoT-Health scenario. These exercises revealed the critical role of security metadata, identified the available ones to be presented to users, and suggested potential enhancements that could be integrated into system design. We present our observations from the exercises, highlighting experts’ valuable suggestions, concerns, and views, backed by our in-depth analysis. Moreover, as a proof-of-concept of our study, we simulated three relevant use cases to detect cyber risks. This comprehensive analysis underscores critical considerations that can significantly improve future system protocols, ensuring end-users are better equipped to navigate and mitigate security risks effectively.

1. Introduction

In modern healthcare settings, devices are integrated with various smart applications, effectively linking these devices to the Internet and integrating them into the Internet of Things (IoT) [1,2]. Interestingly, the multi-layer feature of IoT architectures [3] enables highly distributed solutions, promotes productivity, usability and simplifies the use of IoT systems. However, the increasing dependence on data generated by a large number of heterogeneous IoT devices also creates multiple security issues [4,5,6,7,8]. Hence, the successful use of these IoT applications in healthcare depends on accurate decision-making, comprehensive risk assessment, real-time security, and performance evaluations [9]. Among these, the importance of identifying risks involved in a multi-layer IoT architecture is crucial and an emerging research area [10].

In an IoT health system, security attacks can have serious consequences for human life or cause significant financial damage [11]. It is a significant concern that end-users (for example, general users, practitioners, and/or automated systems in an IoT-Health architecture) may not be able to perceive any potential risks or degraded security as a result of attacks [12]. End-users need to be equipped with cybersecurity situational awareness and understand the security status at each layer in IoT environments. To improve this awareness, end-users need to have access to security information in real-time. Cybersecurity situational awareness [13,14] provides IoT-Health end-users with valuable insights to build trust in secure systems and respond accurately and promptly to cybersecurity risks arising from vulnerabilities. Remarkably, in contrast to many other scenarios, security in a digital health context is not necessarily a binary secure/insecure decision. Data from less trustworthy systems transferred over insecure channels can still add valuable context for diagnosing a patient and determining treatment. However, diagnosing and monitoring a patient’s health status should never solely rely on such insecure data. Therefore, to facilitate proper decision-making from an end user’s view, it is crucial to extend IoT applications with precise situational awareness with respect to the status of the overall system, including all data sources and processing steps.

One possible approach to address this gap was developed by Jaigirdar et al., that suggests adding security metadata to provenance information [15]. Security metadata are the relevant information to represent security evidence at any layer of an IoT architecture. Such metadata can include evidence on active security controls, security protocols used for communication, evidence of system configuration status and update, information on software status running on a device, etc. [15]. The proposed security-aware IoT provenance model, named Prov-IoT, determines the information needed to continuously estimate the risk of IoT data being compromised at different steps of data propagation. The Prov-IoT model establishes the theoretical foundations for adding security-relevant features in IoT data propagation. Further, the authors proposed a proof-of-concept-based experimental analysis [16] with designed security metadata to demonstrate the feasibility of their proposed model. However, the potential impact of integrating user experience and security metadata into real-world case studies to elevate system security awareness through risk analysis remains largely untapped. This presents a significant opportunity for exploration and innovation in the field of security management. Therefore, it is pivotal to inspect systems to identify available and suitable security metadata and whether the system can, in practice, provide/report those. It is also essential to understand different types of security metadata from various viewpoints to further explore how they are helpful in empowering users’ cybersecurity situational awareness and designing future trustworthy systems. With these goals, in this paper, we focus on answering two research questions to get user perspectives and understanding on identifying and using security metadata.

• RQ1. What aspects of security metadata are useful to end-users in an IoT-Health architecture?
• RQ2. How can they help empower users’ cybersecurity situational awareness and design future systems?

The aim of this study is to identify a set of tangible security metadata that provide significant security evidence in a typical, idealized, realistic IoT-Health scenario (as depicted in Figure 1). It also covers pertinent issues and suggestions from the system design viewpoint, while a technical analysis of healthcare scenarios can identify the security controls used, for the anticipated focus on the end-user, it is important to include human perceptions and people’s estimation of risks. Therefore, we undertook table-top exercises [17] with ten security, health, and information and communication technologies (ICT) experts to get their perceptions of the IoT-Health system’s security status and methods for better risk estimation and security awareness for end-users. We conducted the table-top exercises by asking domain experts to work on three tasks to get their understanding and insights on different attacks, security solutions and security metadata in a designed IoT-Health architecture. Moreover, leveraging the security metadata identified during our table-top exercises, we simulated three IoT-Health use cases to highlight the critical role of this metadata in uncovering security risks. This demonstration underscores its significance in enhancing our security posture.

While the security of Internet of Things applications has been subject to long-term research and many approaches to improve security have been developed, this study contributes by adding users’ perspectives to security by identifying what kind of security metadata can be made available and can potentially improve the users’ cybersecurity situational awareness. Further, it highlights experts’ concerns and proposals for important design decisions to be made in the development of IoT applications for digital health. In the following sections, we will first provide some details on the study background, then illustrate the table-top exercises with experts and present the findings. Further, we analyze the findings to identify security metadata currently available but often not used and suggest potential additional ones to be used for better risk estimation and system security awareness. Finally, the paper concludes with relevant discussions and future implications.

2. Background

Cybersecurity issues related to user experience have been explored in human–computer interaction (HCI). For example, risks involved in smartphone usage at the workplace [18], users’ privacy violations via activity tracking technologies [19,20], and challenges associated with wearable devices [21] are all closely related to how users interact with their devices and applications. Moreover, cybersecurity awareness has been widely discussed by both HCI and information science researchers from socio-user perspectives [22,23,24]. Research in this domain often focuses on how humans can protect an organization’s information systems through behavioral management and education. However, we argue that educating and training end-users in individual layer-wise security vulnerabilities (for example, risks/vulnerabilities associated with smartphones, gateways, or user experience in using devices) is inadequate for providing users with a real-time view of the overall security status of a system [3]. Instead, information on security properties of step-wise data propagation is crucial in identifying any potential weaknesses and high-security risks in the system [9,12]. Thus, it is vital to provide such information to enable human end-users to understand and analyze it.

Healthcare data are particularly sensitive; even minor inaccuracies can pose serious risks to patients, potentially resulting in fatal outcomes. Moreover, the expansive attack surface in remote healthcare monitoring and telehealth architectures makes this issue even more critical [25]. Researchers indicated gaps in situational awareness for cybersecurity and urged for more focused research in this vital area [26]. To support end-users in making correct decisions about the reliability of the available data, we need to establish run-time analysis of the security status for processes and devices in these systems.

Provenance has been studied widely in the past in archival science to identify the authenticity of manuscripts, financial reports, patient logs, etc. [27,28]. The provenance of a data item includes information about the processing steps and source data items that lead to its creation and current representation. Inspired by provenance-based models [29,30], the Prov-IoT model [15] presents a security-aware provenance graph that includes security-relevant information by the need of the end-user to see not only the actual processing steps but also the layer-wise security status in the IoT application. Although the Prov-IoT fundamentally motivates the need for adding run-time security metadata to provenance graphs, an analysis of what security information is available in current systems, what can potentially be made available, as well as the human perception and ability to process this security metadata remains open.

While end-users may be empowered with certain training programs and information security awareness education, in the multi-layer decision-oriented IoT system, it is necessary to understand what kind of security information and situational evidence can be used for understanding the current risks for manipulations and attacks potentially corrupting data and changing system behavior. These run-time security metadata need to be propagated through the system layers and visualized to end-users to empower them with the overall system’s exact security and situational status. To the best of our knowledge, this study is the first to consider this research agenda in empowering end-users for cybersecurity situational awareness by considering step-wise security status and relevant information.

3. Research Method

We initiate this study with a standard IoT-Health scenario, collaborating with experts to optimize it for the best possible outcomes. To fulfill the research aim of this study, we (1) collect experts’ insights in the idealized IoT-Health setting and (2) identify security metadata in the scenario. Two table-top exercises were conducted with ten experts and advocates from cybersecurity, information and communications technology (ICT), digital health, law, and policy. We conducted a simulation focused on the sensing layer—responsible for collecting and transmitting medical data—to validate the exercise’s recommendations and demonstrate a real-world use case for the relevant security metadata. This section describes our methods in detail.

3.1. IoT-Health Scenario

In the designed scenario, patients can be equipped with various e-health devices (for example, EEG, ECG, and insulin monitors) that collect patients’ sensitive data. Devices can also include different fitness trackers that are not certified for medical use but still provide information on a person’s overall health. We also consider linking these data to patients’ electronic health records (EHR). Patients’ medical information is collected via a smartphone. The transfer to health applications is abstracted as a gateway. Data are then assumed to be processed and aggregated in an application running on a cloud server. This processing step can include various analytical mechanisms, potentially using machine learning or artificial intelligence. In the end, practitioners, system experts, or clinical users such as doctors receive the processed data and make decisions according to the required application. Experts were asked to identify potential attacks, suitable security controls and metadata that can demonstrate the use of these security controls. We conducted a qualitative analysis of the data from the table-top exercises. These data include recordings of the sessions, notes, and electronic whiteboard content.

3.2. Structure of Table-Top Exercise

We use the table-top exercise to conduct this study as we are interested in collecting expert knowledge on security metadata relevant to human understanding. Table-top exercises are particularly advantageous because they require minimal administrative and technological resources. They have been used widely for cybersecurity exercises, evaluation scenarios, and cybersecurity education for quite a long time [31,32,33]. We are interested in the technical aspects, such as security controls in place and other security-related information worth considering. Thus, we planned to ask domain experts for their insights and opinions on system security states and awareness. We opted for expert discussions, as this approach enabled our research to be grounded in current practice and the discourse and exchange provided rich and in-depth analyses of cybersecurity aspects of the idealized IoT-Health scenario and similar contexts. We provided the IoT health scenario (Figure 1) to participants.

We conducted the table-top exercises in two rounds. The exercises were conducted in Zoom sessions using the Miro whiteboard platform [34]. Each exercise lasted for 90 to 120 min. Participants took part in completing the tasks of the exercise via discussions and Miro whiteboard activities. Participants were informed of the purpose of the study, and written consent was obtained prior to the exercise. All discussions were recorded with the participants’ permission. The exercise was guided by three tasks to be performed by the experts, which are outlined as follows.

• Task-1: Discuss the scenario with your peers to identify the possible points of attack in the overall scenario. (Assume yourself as an attacker)
• Task-2: What kinds of security controls would need to be in place to prevent these attacks? (Switch each participant’s role from an attacker to a protector of the system, and rethink the cases)
• Task-3: What kind of security evidence/security metadata can be gathered to prove that they (the identified security controls) are in place?

Participants spent on average 30 to 40 min on each task. The third author hosted both sessions, and the first author monitored the overall discussion, guided the participants to use the Miro whiteboard, and took notes where necessary. We maintained an ‘open-ended’ discussion with the participants so that we could gather as much as information we could from the experts. Where needed, we elaborated terms and phrases such as security metadata or evidence related to security controls. Accordingly, we discussed relevant examples from the scenario to widen the participants’ understanding of related attacks and associated security controls. We also illustrated the security-aware provenance graph concept with the participants.

3.3. Experts Diversity

With the exploratory nature of our study in mind, we selected ten experts of different expertise with sufficient knowledge of IoT architectures, cybersecurity scenarios, and current cybersecurity technologies. The criteria for selecting participants for our study were that they needed at least five years of experience in cybersecurity, the field of Information and Communications Technology (ICT), and their particular areas of expertise (e.g., cybercrime, security in digital health, general cybersecurity). We used our professional networks and communication channels to recruit the participants. We shared the project’s explanatory statement with the potential participants to make them aware of the aim and structure of the table-top exercises. Once we receive interests from the participants, we discuss those among the team members (all four authors) to remove any bias in selecting the participants.

Table 1. Experts profiles.

ID	Role	Industry Sector	Expertise
1	Professor	Information and Communication Technology	Critical infrastructure security, remote condition monitoring, mobile and sensor network, information processing.
2	Senior Lecturer	Criminology	Cybercrime, fake news, information operation.
3	Senior Lecturer	Software Systems and Cybersecurity	Data security and privacy, trusted computing, secure networked system.
4	Head of research and capacity building–industry and academia	Cybersecurity Policy and Strategy	Public health, information security management, pharmacology, neuroscience.
5	Professor	Computer Security and Reliability	Cybersecurity, digital evidence, trusted computing, network security.
6	Researcher-industry and academia	Secure Information Technology	Trusted computing, formal methods, distributed systems.
7	Lecturer	Cybersecurity	Information security, privacy, Blockchain.
8	Professor	Computer Security and Reliability	Cryptography, information security, network security.
9	Digital Health expert-industry and academia	Digital Health and IT	Health informatics, usability, software design and evaluation.
10	Professor	Software Systems and Cybersecurity	Security, digital health, cryptographic protocols, trusted computing.

provides a summary of the profiles of the participants. It is worth mentioning that the concept of a security-aware provenance (in the Prov-IoT model [15]) graph was new to all participants, and they were able to work through the exercise without being influenced by previous related research.

3.4. Data Collection and Analysis

We collected a total of four hours of recorded audio data, information from the Miro whiteboard, and many pages of discussion notes. Every task during the table-top exercise introduced various topics for us to investigate and analyze, which required spending more time on individual contributions. We then performed inductive analysis on the collected transcripts, while the analysis was targeted at identifying security metadata, the inductive approach enabled us to include additional concepts and topics introduced by the experts.

The table-top exercise began with three tasks for the experts to complete. In Task-1, they exchanged views on security attacks in the designed scenario, and then they discussed relevant security controls for those attacks in the next task (Task-2). After covering the security threats and relevant security controls to a significant extent, the participants started discussing the security metadata (security-relevant evidence) that can be added to provenance graphs (Task-3). The participants worked on the Miro whiteboard for the three tasks, and we collected the outcome after each discussion. An example of a screenshot from the output of the Miro whiteboard is added in Figure 2. The green sticky notes represent participants’ responses on Task-1, the light-blue sticky notes represent their responses on Task-2, and the responses on Task-3 are recorded in yellow sticky notes. We collected several screenshots of the Miro whiteboard.

Codes and design themes/core concepts emerged from our data throughout several rounds of coding leveraging the NViVo qualitative data analysis software [35], by the first author. The first author generated the themes first, mostly according to the tasks described and added any new insights as future works/points to discuss further. All four authors met weekly to discuss potential codes and justified the themes generated and addressed any disagreement occur among them. Further, all four authors agreed upon the final set of codes, the relevant comments and keywords.

3.5. Ethics Approval

This study was conducted with reference to the project Internet of Things (IoT) in digital health table-top exercise and was approved by the Human Research Ethics Committee of Monash University.

4. Findings

This study reveals valuable expert insights from the tasks discussed, highlighting critical information for system designers. Our findings are twofold: first, we identify essential security metadata from the experts’ discussions (detailed in Section 4.1). Second, we present their crucial concerns and innovative proposals (outlined in Section 4.2), which can significantly enhance system design and security. Finally, we ended our study with three case-study based simulations.

4.1. Providing Security Metadata

Table 2. List of security metadata identified from experts’ discussion.

Attack Points	Security Controls	Topic-Based Security Evidence	Relevant Security Metadata
Smartphone, Electronic Health Record (EHR), Fitness Tracker, Bluetooth, Gateway	Access control, Authentication device identity, Security of apps, Anti-malware software, Authentication mechanisms	Trusted computing, Encryption, Policy and human-centric	Programmable system on a chip (PSoC6), Trusted Platform Module (TPM), secure boot, secure attestation, secure enclave, trusted room, trustable logs, HTTPS, digital signature, restricted key management, time-specific data (timestamp), demographic information, device configuration information, law/regulation information
		End-to-end system	Protocols (with version and parameters): 802.1AR, S/MIME, PGP, TLS, etc., multi-factor authentication (2FA, MFA), AI-based authentication, authentication chip, antivirus, OS version and update status, encryption validation, telemetry data, safety info
At any point (general)	Auditing service	Professional auditing service	Configuration target, output from auditing service
Cloud	Cloud security, Certificate authority (CA)	Cloud security	Contractual specifications or policies, security engineering auditing, CA-related evidence
Device-specific	Device standard or update	Device standard	Industry/government standards, HIPAA/FDA/TGA regulations

effectively highlights the key findings of our study, organized by significant themes. From Task-1, the overarching theme identified was ‘attack points’, where the participants discussed possible attacks on smartphones, electronic health records, gateway, cloud etc. ‘Security controls’ was the theme for Task-2, where participants discussed on various security mechanisms and existing security practices. For Task 3, we identified two compelling themes: ‘topic based security evidence’ and ‘relevant security metadata’. The “topic-based security evidence” theme captured broader discussions on security, while “relevant security metadata” focused on crucial information that can be effectively recorded in provenance graphs. The following paragraphs will provide concrete examples to illustrate the depth and progression of this vital discussion.

While analyzing security attacks in smartphones, EHRs, bluetooth, fitness trackers, and gateways used for communication, the participants shared views on different access control mechanisms as prospective security controls. According to Expert-7,

“I think the simplest solution is a digital signature. In every five minutes, data will be aggregated and assigned either by the device itself or from the smartphone, and then it will pass on to the nearest endpoint for the doctors…”

Other experts (expert-6, 8, and 10) raised the concern that digital signatures cannot be easily checked if there is no end-to-end security relationship between the devices and end-users. As a result, we divide our findings from the ongoing discussion into two categories. We first illustrate relevant security metadata observed for trusted computing, encryption mechanisms, policy, and human-centric perspectives. Later, we show relevant security metadata identified for the end-to-end system, while discussing trusted computing, trusted platform modules (TPM) and relevant information for trusted computing were considered. Experts emphasized the need to include information related to trusted execution, secure boot, secure enclave, and secure attestation in provenance graphs to ensure reliability and prevent hacked devices from inserting fake security metadata. Expert-3 noted,

“So basically, I think all the health data need to be processed in the trusted room. Furthermore, if they are not being processed, they have to be staying in encrypted form always so that it will be the cipher-text, and there will be no use for other apps or attackers. Hence, the key for encrypting this host data must always be stored in this secure enclave.”

While explaining trusted enclaves and secure boot in various applications (apps) in smartphones, Expert-1 mentioned

“When you want to open up that app, that app should not function, it should immediately alert that your phone is not secured.”

Experts also pointed to adding demographic information, metadata relevant to encryption, and policy/regulation perspectives while discussing policy and human-centric security evidence (see in the fourth column of the first row of Table 2), while discussing policy and human-centric security evidence, Expert-4 mentioned,

“You can have your point of view, you can do background checking, like police checks and clearance. Furthermore, from a technological point of view, you can do significant activity monitoring for behavioral analytics over time.”

Relevant security metadata identified during end-to-end system discussion are included in the second row of the fourth column of Table 2, while experts argued on authentication mechanisms for implanted devices, some proposed including IEEE standard 802.1AR for secure device identity. Additionally, two-factor authentication (2FA) and multi-factor authentication (MFA) were highlighted repeatedly as crucial components. The discussion also covered the importance of S/MIME (Secure/Multipurpose Internet Mail Extensions) and PGP (Pretty Good Privacy) for providing cryptographic security services for electronic messaging applications. Expert-5 added in this regard, “I obviously would recommend documenting information related to PGP for data transfer.” Interestingly, AI-based authentication was raised by Expert 6. However, others countered that due to various undisclosed security vulnerabilities, an AI-based authentication system might not be the best option. Furthermore, the discussion included details about application-level protocols, TLS version, mode, and parameters, operating system version, and flags indicating whether specific data pass through certain security controls, such as firewalls or intrusion detection systems. All of these were deemed relevant pieces of security evidence.

The experts’ opinion on auditing services is recorded in the second row of Table 2. Some experts noted auditing services as another security control for identifying attacks occurring at different points. They considered various configuration targets and output from auditing services to be added to provenance graphs as security metadata.

Although a majority of the experts indicated the cloud services to be the most trusted, they also talked about cloud security-related information as prospective security metadata. These contain cloud contractual specifications, information on security or crypto-engineering auditing, various policies or specifications, and certificate authority-related evidence, as recorded in the third row of Table 2.

During the exercise, information related to device standards and updates was discussed as potential security metadata to identify device-specific attack risks. To monitor device authenticity, experts suggested the inclusion of information on either industry or government maintained standards for various devices in the IoT-Health network (for example, The Therapeutic Goods Administration (TGA), Health Insurance Portability and Accountability Act (HIPAA), U.S. Food and Drug Administration (FDA)), as included in the last row of the table.

4.2. Providing Concerns and Proposals by Experts (for System Design)

While discussing each task, the experts raised several concerns and proposed useful solutions for future system designs. This section summarizes additional concerns raised by the experts in

Table 3. Concerns for system design.

Concerns	Discussion Points	Representative Quotes
- Cloud security. - Certificate authority (CA).	- TLS, channels and signatures. - Trust the cloud provider. - Trusted computing. - Assuming CA with close environment. - Security engineering/system audits.	“Avoiding cloud is the best thing. Cloud security is contractual security. …As long as you believe the contract, it’s fine.” Expert-5 “I think ‘clouds’ are monitored continuously and managed by professional security experts. Expert-1 “If you are actually assuming that the cloud operator is malicious, you havebigger problems.” Expert-6 “However, I think it is very challenging for attackers to launch side channel attacks inside the cloud right now. I think they still have very strong protection.” Expert-3
- Device standard and update.	- Industry and Government maintained standards. - Upgrade.	“With some antivirus software on the device, we are running into a very tough update issue, …as soon as any kind of software decides this device is infected. We need to remediation processes.” Expert-5 “Whenever you work in the health industry, you should maintain a certain security, for the hardware and for the software.” Expert-7
- Human centric. - Social Engineering.	- ransomware. - awareness. - law/regulation.	“How to define a situation when somebody approaches me looks like a doctor or health professional? I will believe them and maybe even transfer some of the credentials or if they want to replace my device…” Expert-7 “…even for a secure system, the patient might not have any awareness, …For example, they might simply not be able to keep their phone in a safe place. Furthermore, for the doctor, it will be the same.” Expert-2

and illustrates their proposals.

Cloud services were assumed by most of the experts as ‘mostly secured’ with a lower probability of not patching vulnerabilities and properly monitoring their systems. However, other experts argued that cloud services require a lot of trust in the service provider and that many examples have shown vulnerabilities. They also pointed out that a malicious cloud provider would be highly problematic. The first row in Table 3 includes the experts’ comments while they discussed cloud and related security.

Further, while discussing the significance of identifying a device, or security threats relevant to devices, experts emphasized maintaining industry and/or government maintained standards with a timely upgrade of relevant hardware and software. Remarkably, the experts did not always agree and identify best practice approaches. For example, there was no clear conclusion on the topic of virus scanners and anti-malware software, while Expert-7 pointed out anti-malware software needs to be always updated, Expert-5 argued against it,

“I think the only thing that we can assume is that any kind of protection software on smartphones breaks the security of the smartphone. I have not seen any proof of anti-malware software strengthening security. However, I have seen many issues with this…"

Another significant concern by the experts was human-centric issues related to accessing a smart healthcare environment. In this regard, they discussed ransomware attacks, lack of awareness from both the doctor and patient side, and various social engineering attacks, while discussing these, one expert commented,

“So if you’ve got a compromised individual anywhere in the chain, then he can gain system access, and that does not require any vulnerability…”

Expert-4

Along with various concerns, some experts proposed a ‘specially prepared device and system environment’ for minimizing security threats in the designed healthcare scenario and indicated information related to those should be added as security metadata. According to Expert-5,

“I would build something like a Virtual Private Network (VPN) connection through a vendor. Maybe we can channel through them (the vendor) without them able to process it. However, this vendor specifically knows what kind of traffic should go in and out. He can decide whether the traffic is okay or not”.

Expert-6 explained the idea of using VPN in the healthcare system as follows.

“I like the idea that certain data, especially health data can only flow through certain channels. Furthermore, that happens to be a VPN or some private network. In that case, we can skip encryption and make sure that health data pass through these channels is ending up in a secured channel from an information flow perspective. They should not just be going through public networks”.

As another potential solution, some experts proposed utilizing mobile phone operators to limit the communication pathways to allow the devices to reach certain endpoints. One example considered by them was 5G slices for limited communication pathways between the devices. Moreover, experts considered a specially designed smartphone with restricted device configuration and specific access control mechanisms for any unwanted intruder, designed to deal with health data only. We argue that these concerns and proposals reveal important insights for a security-aware provenance-based system design.

5. Simulation

Based on the discussion during the table-top exercise, we identified that IoT devices are particularly vulnerable to security threats, especially in the absence of an end-to-end security solution. Consequently, we selected the sensing layer of the designed scenario, a crucial component responsible for collecting and transmitting medical data, to conduct a simulation. This simulation aimed to validate the applicability and importance of the suggestions provided during the table-top exercise. These specific suggestions were included in the simulation to see how well they could detect threats in real-time. By simulating cases such as code manipulation and device impersonation, the simulation aimed to confirm the experts’ recommendations, showing how security metadata can improve awareness and help respond quickly to cybersecurity risks.

5.1. System Setup

The simulation was conducted in a multi-layer IoT setup as illustrated in Figure 3. We also included the system architecture diagram in Figure 4. In the sensing layer, an ECG monitor continuously captures health data from a patient and transmits it, along with its security metadata, to the patient’s smartphone. Upon receiving the security metadata, the smartphone validates it and synchronizes the results with the cloud. We leveraged two innovative Python projects to effectively simulate the behaviors of both the ECG monitor and the smartphone. With the integration of Google Firebase and Vue.js, end users or inspectors can effortlessly access real-time security situational awareness of the ECG monitor via an intuitive web interface.

In alignment with the experts’ discussion outlined in Section 4.1, we selected the security metadata for the ECG monitor. Specifically, we utilized the program integrity hash to ensure the integrity of the running code and the measurement hash to verify the device’s configuration information (as advised in Table 2, digital signature). To ensure the device’s authenticity, we incorporated an attestation digital signature of the hash. TPMs, as referenced in the table, were employed for these functions, assuming that the TPM internally and securely performs the hash and signature operations. Additionally, we supplemented the security metadata with information such as Node Name, Transmit Destination, and Industry Standards. We completed all configurations regarding public key infrastructures prior to the commencement of the simulation.

5.2. Simulation Use Cases

We carried out experiments that simulated three critical scenarios, demonstrating how potential attacks can trigger significant and unexpected changes in security controls.

Case 1. In the first case, malicious users targeted running codes to launch attacks using code injection techniques to modify the running application.

Case 2. In the second case, malicious actors can take advantage of weaknesses in a device’s input handling systems, potentially launching Denial-of-Service (DoS) attacks. These attacks [36] can compromise configurations and severely disrupt essential medical services, posing significant risks to patient safety and operational integrity.

Case 3. In the third case, we simulate a node cloning attack where attackers create a similar node to send data to the smartphone, impersonating a legitimate device to mislead the system or gather sensitive information.

5.3. Results

Case 1. As illustrated in Figure 5, except during system upgrades, the hash values of the code should remain the same. Therefore, any alteration of the running code could trigger a mismatch of the hash value, and end-users can get a real-time update about this problem.

Case 2. As illustrated in Figure 6, unless medical requirements change, the hash value of the configuration should remain the same. The TPM performs the hash function based on the current tampered configuration of the device. Any mismatch in the hash value can be quickly identified.

Case 3. As shown in Figure 7, even though a forged node may have the same code, firmware, and configuration as an authentic device, it cannot access the private key stored within the Trusted Platform Module (TPM). As a result, any failure in signature verification will be documented in the security metadata.

These cases are designed as a proof-of-concept for providing cybersecurity awareness with evidence-based/real security metadata. It is worth mentioning here that large scale IoT applications might process data from a high number of devices, which may questions the scalability of applying security metadata in different IoT domains. However, we would argue that many applications, such as the scenario presented in this paper, require data related to individual people or specific objects. As our primary objective was to identify any risks involved in such scenarios, we provide the example cases to clarify the applicability of our approach. In future, we would like to integrate it with existing healthcare infrastructures.

6. Discussion

This paper explores a healthcare scenario within the Internet of Things (IoT), leveraging expert opinions through three focused tasks using table-top exercises for insightful feedback and discussion. Our aim was to identify useful security metadata for potential risk estimation and appropriate actions. The identified security attacks, security controls/mechanisms, and potential security metadata trigger various opportunities for empowering end-users, concerns for current systems, and future implications for developing and operating IoT applications.

6.1. Significance of Security Metadata

We presented compelling security metadata and topic-based security evidence that were highlighted during table-top exercises. Experts examined the topic from various attack vectors and security-awareness perspectives, leading to a rigorous discussion of a comprehensive set of security controls. They not only identified essential security metadata but also proposed additional relevant metadata based on existing and partly standardized technologies that remain underutilized in current systems.

Experts provided positive feedback on the concept of adding security metadata in the provenance graphs by noting “…it is timely and important…” and mentioned that inclusion would be helpful to make the system more trustworthy and transparent. The identified security metadata are significant and can be made available to the end-user. The experts recommended a list of security controls as tangible security metadata to be reported in security-aware provenance graphs. Thus, this exercise shows the viability of the security-aware provenance graph by showing that substantial security metadata are available for use.

Furthermore, the simulation results revealed vulnerabilities in IoT devices, particularly within the sensing layer of the healthcare scenario, which collects and transmits sensitive medical data. By simulating three types of attacks (code manipulation, denial-of-service, and node impersonation), this study highlighted how security metadata can help detect and reduce these threats. For example, changes in hash values were useful for identifying code tampering and configuration changes, while digital signatures from trusted platform modules (TPMs) helped detect unauthorized devices pretending to be legitimate nodes. These results show that security metadata are an important tool for real-time monitoring and improving cybersecurity awareness, allowing users to make better decisions about data integrity and system trustworthiness.

6.2. Empowering End-Users with Cybersecurity Situational Awareness

Users often do not have any information on many security mechanisms and controls already in use and that security metadata can, in principle, be added while designing security-aware provenance graphs of a system, while the use of HTTPS as security metadata are visualized in browsers, this is not the case in many other applications. Furthermore, end-users are often not informed about the use of encryption mechanisms, digital signatures, key management, 2FA, MFA, TLS modes and parameters and/or OS versions, device configurations, protocols or flags used, anti-malware software, policies/specifications that were identified as security metadata in the exercises (details are included in Table 2). These pieces of information, for instance, having 2FA or MFA in a system or generating encrypted data versus plaintexts, provide an understanding of the system risk/trust and help end-users make appropriate risk estimations and decisions while using the system. To illustrate, if an end-user (for example, a health practitioner) is informed of having ‘MFA’ based user login system, which is protected with ‘restricted key management’ and ‘secure attestation’, he might not question the data integrity or system authentication. Thus, he should be able to proceed with their further medical treatment without any worry of thinking whether there is a data manipulation or incorrect authentication. In this way, the list of security metadata identified by this study would be helpful in empowering end-users with cybersecurity situational awareness. Moreover, researchers talked about software versions while discussing users’ security awareness and indicated outdated versions as ‘not recommended’ [37]. During table-top exercises, experts discussed software versions and update status as potential security metadata. Accordingly, our study adds valuable recommendations for users’ situational awareness.

The effectiveness of security awareness and behavior training has been discussed by various researchers [23,37,38,39]. During the table-top exercises, some experts raised concerns regarding end-users understanding of security metadata and situational awareness. We strongly believe that targeted training is crucial for enhancing users’ understanding of visualized security metadata. For maximum effectiveness, it is vital that this security metadata are integrated into system visualizations in a clear and accessible manner. By doing so, end users will be empowered to draw accurate conclusions and make informed decisions based on the information they see. This study, therefore, aims to complement existing cybersecurity awareness initiatives, ultimately fostering a more effective and reliable architectural framework for users. Finally, we understand that the true effectiveness of security metadata for enhancing situational awareness cannot be fully exercised without feedback from non-expert users. Testing the usability of clinicians, patients, or administrative personnel would enable the authors to assess the intelligibility of the metadata, determine whether it can aid in decision-making, and evaluate its impact on trust and behavior. Therefore, we raise the importance of performing a usability study with non-expert users and keep it as a future work.

6.3. Design for Security-Aware Provenance-Based Approach and Implications for Future

During the exercise, the experts discussed various areas of concern (illustrated in Section 4.2). For example, why security-relevant information on system updates is necessary, how human-centric issues need attention, and how to control and monitor security updates. These concerns also indicate the significance of the security status of different parts of a system. Moreover, experts recommended establishing a dedicated healthcare setup to enhance security and trustworthiness. As a result, this study presents new design opportunities for collaboration between academia and industry, particularly in graphical user interface (GUI) and human–computer interaction (HCI) fields.

During the exercise, one option discussed was the inclusion of metadata relevant to policies, updates, and industry or government-maintained standards. One critical aspect is how to prevent compromised IoT devices from pretending to be in good operating condition and producing misleading security evidence. Adding evidence on standards and policies used at each layer of the architecture would provide a situational security status to end-users.

The experts recognized various security evidence as necessary metadata for system security, which currently are not used, for example, technology related to hardware-based security, such as trusted computing [40]. Moreover, the experts mentioned that if the system is designed differently (some suggestions and proposals are provided in Section 4.2), it will enhance the system’s security. Therefore, this study provides experts’ insights on potential security metadata to be used in future systems. We believe that this inclusion would enhance the system’s security status, and the relevant evidence would enrich the system with meaningful security information.

Finally, the findings from this study can be applied and extended to other areas beyond digital health. An example could be smart home services, such as water meter monitoring in an apartment building or smart grid environments. Furthermore, it can also help empower end-users knowledge in the context of smart office or smart city applications that can become more resilient by visualizing security-aware provenance graph information [16].

7. Conclusions

At present, the lack of information on the security context of IoT systems limits end-users abilities to identify risks involved in the system and interpret if the data they see are trustworthy. It is unlikely that users can have suitable situational awareness and know whether cyber-attacks are possible or have even occurred while the data were propagating through IoT systems. We conducted table-top exercises with ten domain experts to identify relevant security and situational information in an exemplary IoT-Health system. Our findings, analysis and simulation illustrate a set of security metadata that can improve end-users run-time situational awareness while using IoT applications, while users currently rely mainly on trust in the systems operating status and the use of security controls, this additional information created transparency. We understand that the discussion from the table-top exercise may not represent all possible evidence of security awareness for this scenario (as other technical evidence can be included). Nevertheless, the table-top exercise generated a substantial list of security evidence from an end-user perspective. It may ultimately be the first step towards giving users more autonomy over cybersecurity issues in the next IoT systems. Moreover, we also acknowledge that we included experts’ insights only in identifying security metadata, omitting the concerns and understandings of real users (patients, clinical users). Our future works will involve clinical users or patients to establish the usability and effectiveness of the proposed security metadata in various scenarios of IoT-Health architectures.