IoT Security-Quality-Metrics Method and Its Conformity with Emerging Guidelines

Abstract

This study proposes a security-quality-metrics method tailored for the Internet of things (IoT) and evaluates conformity of the proposed approach with pertinent cybersecurity regulations and guidelines for IoT. Cybersecurity incidents involving IoT devices have recently come to light; consequently, IoT security correspondence has become a necessity. The ISO 25000 series is used for software; however, the concept of security as a quality factor has not been applied to IoT devices. Because software vulnerabilities were not the device vendors’ responsibility as product liability, most vendors did not consider the security capability of IoT devices as part of their quality control. Furthermore, an appropriate IoT security-quality metric for vendors does not exist; instead, vendors have to set their security standards, which lack consistency and are difficult to justify by themselves. To address this problem, the authors propose a universal method for specifying IoT security-quality metrics on a globally accepted scale, inspired by the goal/question/metric (GQM) method. The method enables vendors to verify their products to conform to the requirements of existing baselines and certification programs and to help vendors to tailor their quality requirements to meet the given security requirements. The IoT users would also be able to use these metrics to verify the security quality of IoT devices.

1. Introduction

Security becomes more important with the proliferation of Internet of Things (IoT) devices. In fact, many security breaches on IoT devices have already been reported, hence the increased need for IoT security [1,2,3,4]. Such breaches involve, for example, the “Mirai” malware and its subspecies spreading across cyberspace targeting IoT devices, including IP/web/network cameras, digital video recorders, home routers, smart speakers, and network printers [5,6].

Researchers of IoT security have made significant progress in mitigating security threats and vulnerabilities, such as remote attacks via wireless connectivity such as Wi-Fi, Bluetooth, or Zigbee [7,8,9], and securing an architecture to meet security requirements [4,10]. However, because these functions and mitigation technologies are not self-developed by IoT vendors in most cases, but are externally procured components, IoT vendors are required to assess the security quality of the communication components they adopt. Having said that, in reality, IoT security researchers have not yet clarified standard initiatives that IoT vendors easily adopt to ensure the development of secured IoT devices.

Unlike legislation on safety and environmental issues, laws, regulations, and international standards for IoT security have not been established yet. ISO 27400 (guidelines on IoT security and privacy) [11] is still under development.

Many developers and researchers have duly addressed information security issues via ISO 27001 [12] or have discussed a new cybersecurity certification method [13]. Although ISO 27001 outlines the management and protection of information assets [14,15], security-quality management for the development of IoT products is necessary throughout the product lifecycle and needs to be defined, as is the case with secure software development [16].

Security measures are necessary; however, to guarantee the quality, it is necessary to define initiatives and visualize them as processes throughout the development cycle.

Pino et al. explained that the software development process is a critical factor for delivering quality software systems [17]. This implies that the quality of the software is influenced by the quality of the development process. This strategy is equivalent to those being implemented in other branches of engineering and industries [18]. Jones reported that most successful projects utilize similar patterns of planning, estimation, and quality-control technologies [19].

A similar paradigm must be followed for IoT devices controlled by the software. As described in ISO 30141:2018, the Internet of Things (IoT)—Reference Architecture [20,21], IoT devices are in an important position to connect cyberspace with the real physical space. Consequently, when IoT devices are exposed to attacks, both cyberspace and real physical space face security risks.

Security measures have been implemented for devices for information systems since security issues have been pointed out for years. In contrast, IoT devices, for which there have been few indications of security issues, have spread across the market with few defenses against security risks in cyberspace. In addition, electronics vendors, which have no experience with IoT security risks, have been developing IoT devices without awareness of security risks. For attackers, it is easy to target IoT devices through the wireless communication route such as Wi-Fi or Bluetooth, or the firmware update function. The IoT devices are easily available, and the weaknesses are easier to find, as compared with the information system devices protected. Thus, ensuring the quality in security of IoT products is important and requires a clear standardized development process for IoT vendors. Moreover, the processes must be defined for implementation throughout the product lifecycle and need to be understood by the IoT consumers.

The IoT security-quality transparency model with six areas of the product lifecycle is devised for developing and supporting secure IoT products. Additionally, the IoT security-quality metrics are compiled for each area using the GQM approach, by referencing the requirements of various IoT security regulations and guidelines, and the opinions of security experts.

A primary contribution of the proposed method is for IoT vendors to understand the characteristics of the requirements of IoT security regulations and guidelines. Additionally, the validity of the proposed metrics against the publicly released IoT security requirements from the market and regulations can also be verified.

Thus, we consider this paper to contribute to the improvement of the situation in which many IoT vendors are unable to consider security as a quality requirement and incorporate it into specific secure product development processes simply by presenting the ideas in the guidelines.

The remainder of this paper is organized as follows: Section 2 summarizes the background and necessity of this research by describing no prior work of IoT security from the quality perspective. Section 3 outlines the authors’ previous work on IoT security quality. Section 4 describes the authors’ proposed universal method for IoT security-quality metrics and items that the IoT vendors will be able to design. In Section 5, the authors apply the proposed method to the requirements of emerging IoT security regulations and guidelines and confirm the possibility of evaluating the validity of the proposed metrics. Section 6 demonstrates the sample evaluation results of applying the proposed method to IoT devices. Finally, Section 7 discusses the results of this study, Section 8 provides the future directions, and Section 9 summarizes the conclusion.

2. Background and Necessity for This Study

From the discussion above, the issues with regard to IoT security can be categorized into the following two questions.

2.1. Question 1: Does Any Existing Literature or Standard Covering Cybersecurity-Quality-Control Measures for IoT throughout the Product Lifecycle Exist?

Many security experts have addressed the guidelines for IoT security management from the viewpoint of the basic principles, approaches, threats, and countermeasures.

To assess the cybersecurity of systems, researchers have developed methods such as the Evaluation Assurance Level with Common Criteria certification based on ISO 15408 [22] and EDSA (Embedded Device Security Assurance) certification based on IEC 62443 [23]. However, these certifications are extremely professional: ISO 15408 focuses on quality assurance and does not specify what initiatives to take, whereas IEC 62443 is specific to critical infrastructure in the industrial control system, which generally does not apply to IoT. Furthermore, both approaches do not present a simple way of describing the quality of cybersecurity in IoT products (for vendors and/or general consumers with no knowledge of security).

Although benchmarks and assessment methods for information security have been proposed [24,25], both fall short from a web-specific and a lifecycle perspective when utilized for product security in IoT.

A template to consistently describe the service level of a cloud service has also been proposed [26]; it is, however, specific to cloud services rather than IoT.

Similarly, the importance of IoT security has been pointed out [14]; however, the article only mentions the certification scheme and does not cover the entire lifecycle, service, or system. IoT security has also been previously discussed [27,28]; unfortunately, the discussions are limited to the security of communication protocols.

The majority of the literature includes high-level guidelines and baseline requirements for IoT security for IoT vendors in 2020, discussion of security for IoT with AI (artificial intelligence) [29,30], or with the cloud [31], and the user’s quality of experience (QoE) [32,33] in 2021. Our extensive literature search failed to produce any literature concerned with benchmarks or suggestions for secure development for IoT vendors.

Thus, no simple, standard way of describing the status of cybersecurity readiness from the perspective of product security in terms of the quality of IoT devices was found.

2.2. Question 2: Does Any Reason for Visualizing the Cybersecurity Control Measures Exist?

Most electronic-device vendors producing IoT devices are familiar with ISO 9001, the international quality assurance standard that clarifies the process of product development to standardize the quality throughout the life of a product. Vendors predominantly follow the defined production process and do not perform anything outside the process for cost-efficiency. To prevent noncompliance, it is common to define (and visualize) processes for designing safe products and selecting components with a low impact on the environment.

Similarly, the modalities for product security should be defined in existing processes. In addition, the Information-Technology Promotion Agency of Japan (IPA) reported that approximately half of the IoT vendors have specific policies; moreover, over 70% of them have no concrete standards for their security responses in product development [34]. This implies that the reason behind the lack of concrete action might be that IoT vendors have no clear understanding of who would be responsible for the security; moreover, they do not recognize security measures as their responsibility even if they know the significance thereof. Because it is difficult to add security countermeasures at the implementation stage of the development process, engineers need to devise and apply effective countermeasures at inception. Therefore, the confirmation of effective functioning of the countermeasures at the verification stage is essential. If a new vulnerability emerges, even after the product release, it must be fixed. Therefore, it is necessary to define actions to be carried out at each phase of the product lifecycle. Thus, it would not be possible to develop a secure IoT product unless the relevant processes and practices are clearly defined in the corresponding development process.

In this regard, this paper is a continuation of our previous study [35] in which we reviewed the relevant literature and clarified prospective items in the development of the security-quality metrics discussed herein.

We considered the need for a methodology that would allow IoT vendors to tailor security-quality metrics in addition to existing quality metrics for their products. This would, in turn, indicate to consumers the level of security quality of the IoT devices they develop. We attempted to derive quality metrics for IoT devices based on the literature and perspectives reviewed in our previous work. We referenced the goal/question/metric (GQM) methodology commonly used in the quality field.

The following section is an overview of the previous study [35].

3. Past Research on IoT Security Quality

This section summarizes our previous study conducted, in which we reviewed the relevant literature and clarified prospective items that could be used in the development of the security-quality metrics. We conducted the works in Steps 1 and 2 using the research method described in Figure 1; the results of the research on developing prospective items are discussed in our paper [36].

3.1. Research Method

We conducted a systematic literature review [36] using a combination of keywords such as “IoT”, “security”, and “quality metrics” to find related work. The systematic literature review, with respect to the Wohlin guidelines, was conducted in three steps: (1) Planning the literature review; (2) Conducting the review; and (3) Reporting the review.

In addition, the survey methodology adopted by The European Union Agency for Cybersecurity (ENISA) was adopted as the reference model [37]. This research method starts with a literature survey. The proposal then follows and is succeeded by proof of the effectiveness of the proposal, as shown in Figure 1.

Many security guidelines are formulated on the basis of similar sequences: screening the literature in the relevant fields, selecting items that fulfill the objectives of the guideline(s) to be developed, and reviewing the draft(s)—by experts and the public—before finalizing the guideline(s). In fact, ENISA’s Baseline Security Recommendation for IoT [37] includes items from the National Institute of Standards and Technology (NIST) Cybersecurity Framework v1.1 [38] and the GSMA IoT Security Guidelines [39]. For example, there is a section about threat analysis that is cited in many studies. Therefore, this research method involving a literature survey is well-suited to this study.

3.2. Definition of Scope (Step 1)

To commence the research, we defined the scope, as illustrated in Step 1 in Figure 1 of the research method. An IoT system is complex and comprises many IoT devices and cloud services. Therefore, to simplify the discussion, we focused on IoT devices that are primarily intended for consumer usage. The cybersecurity of cloud services is handled in the ICT (software) industry; there is no such culture as far as hardware is concerned. Historically, most electronic-device vendors are familiar with the physical or electrical safety aspects of quality, but few have ever connected a device to the Internet, a cyberspace fraught with malicious attacks. Consequently, the cybersecurity of IoT devices requires urgent attention.

3.3. Results of Literature Review (Step 2)

As mentioned earlier, we conducted a systematic literature review, as described Step2 in Figure 1, to screen for other researchers with similar research interests. However, we could not find any studies or standards from the systematic review.

The International Standard for Software Quality (SQuaRE, ISO/IEC 25000 series) places “security” (one of the subcategories of functionality) as a quality category for system software. SQuaRE lists “security” as a major nonfunctional requirement in terms of system safety. However, these standards only highlight ideas at the conceptual level together with examples to be considered. Although some ideas and items can be used as references, none of the security-quality-control items are clearly elaborated.

In the security evaluation based on GQM, Abdulrazig et al. [40] discussed the misuse of Web applications. However, this should not be misconstrued as a discussion on the security of IoT devices. Further, Yahya et al. [41] discussed the security assessment of cloud storage. Thus, it is worthwhile to define IoT security-quality metrics based on GQM that IoT vendors could use.

In this study, potential security-quality metrics for IoT devices were selected from the literature for each phase of the previously studied product lifecycles. Quality-control practices were then defined to reflect the opinions of security and quality experts on the parameters that should be considered from the perspective of IoT device users.

4. Itemizing IoT Security-Quality Metrics

Our literature review found no specific work on IoT quality from a security perspective. Therefore, before discussing IoT security-quality metrics, in Step 3 in Figure 1, it is necessary to define the IoT security quality.

4.1. Definition of IoT Security

Because the IoT system consists of electronic devices, it is composed of a hardware device consisting of electronic circuits, sensors, and occasionally actuators, as well as software that controls the functions of the electronic device. Consequently, the performance of every product to verify the quality cannot be comprehensively evaluated. Therefore, it is common practice to guarantee the quality of all products by ensuring that all of the predefined development and production processes conform to the required standards; thus, an assessment of the performance of samples alone is sufficient. Essentially, the collective quality should comprise both process and product quality.

Thus, the quality of cybersecurity of an IoT device may be defined as a combination of the quality of the product-development process and that of the cybersecurity performance of the product.

To outline the security development process, items indicating how to design, build, and support the product should be listed. These items include the results of the process review and the maintenance program based on the development lifecycle. Further, to outline the cybersecurity performance of the product, the results of the security assessment should be listed. To demonstrate IoT cybersecurity performance (mostly in software), the said items should reflect the static and/or dynamic security testing of IoT devices.

4.2. Requirements of IoT Security Quality

Before defining the aforementioned items, it is necessary to clarify the goals and the aspects to screen for.

First, the items must describe the development process in security transparently (for instance, the security policy of an IoT vendor and the organization’s standardized security development process).

To accurately describe the product quality, items properly describing the cybersecurity capability are also required. The results of the product security evaluation should be listed. Importantly, the items should include those responding to market needs as well as those complying with international standards and guidelines.

A crucial source of consumer feedback is aftersales support. The security support program should partly comprise product cybersecurity quality. Activities such as security monitoring, receiving vulnerability feedback, and rolling out updates should be listed as items.

Furthermore, items should be easily comprehensible from the consumer’s perspective; this is important for gaining the user’s trust.

Thus, the requirements of IoT security quality are summarized in

Table 1. Requirements of IoT security quality.

Requirements	Aspect
R1: Describing the development process transparently	1: Security policy of an IoT vendor
	2: Quality of security development process
R2: Describing the cybersecurity capability properly	Quality of product cybersecurity performance
R3: Responding to the market needs and/or requirements	1: Covering the requirements by law or regulation
	2: Following the recommendations of international standards and guidelines
R4: Security support program (post-market)	Security monitoring, receiving the vulnerability input, update, etc.
R5: Any items gaining the user’s trust	-

.

4.3. Transparency Model of IoT Security Quality

To comprehensively identify items of IoT security quality, we defined a transparency model of IoT security quality that describes the nature of items as presented in Figure 2—prior to Step 3—by integrating the definition of IoT security quality and its requirements. This definition of IoT security quality would satisfy the requirement R1, which ensures transparency of the entire product development process.

This model provides a framework for the IoT security-quality metrics. The model is derived by mapping the security development lifecycle, which was released by many organizations such as NIST [42], Microsoft [43], Synopsys [44], and PwC [45], onto the V-shaped product development process that many IoT vendors follow. Nevertheless, by clarifying the relationship between the V-shaped product development process and the security development lifecycle, each of the members involved in IoT product development will know which security-quality metrics they should be responsible for.

When considering IoT security-quality metric items, this novel model not only allows each metric to be assigned to the appropriate area of responsibility but also makes it easier to determine the areas efficient to implement in the future as new requirements emerge.

The “transparent” model for IoT security quality is as follows:

• The “security by design” area comprises two parts, namely, the process quality and corresponding product quality [46]. Those in charge of product planning and those in charge of determining basic specifications are mainly responsible for this area.
• The “security assurance assessment” area involves the evaluation results. Those in charge of product development and those in charge of quality assurance are responsible for this area.
• The “security production” phase entails the items of security management during production. Those in charge of manufacturing the product are responsible for this area.
• The “security operation” phase encompasses aftersales security monitoring and response to incidents. Those in charge of customer support, maintenance, and product security incident response team (PSIRT) are responsible for this area.
• The “compliance with law, regulation, international standard” area implies that the public or industry requirements have been fulfilled. Compliance with industry standards and regulations is relevant to all areas. All members, not just the product manager, are responsible for this area.

Based on this model, perspectives that should be regarded as the state of IoT security—frequently alluded to in the literature survey—are listed.

4.4. Draft Proposal Development of Security-Quality Metrics

Based on the transparency model (Figure 2), IoT security-quality metrics items were examined, and a draft proposal was subsequently developed (Step 3 in the research method).

4.4.1. Software Quality

Because the functionality and security are controlled by software, perspectives of the software quality were referenced [42,43,44,45].

Software-quality control has traditionally been a challenge because an established method for assessing software quality did not exist. Previously, attempts such as visualization by using a bug curve and the number of defects identified have been tried as a method for quantifying software quality. On the other hand, in terms of software reliability, some studies observed that consistency, availability, and maintainability (less downtime) resulted in improved quality. However, when the security perspective is considered, the quality of the product appears to depend on transparency.

4.4.2. GQM Method

The GQM paradigm [47] is a three-tier measurement framework and modeling method in software engineering; the first, second, and third tiers represent the goal, question, and metric, respectively.

Metrics are constructed with reference to the “GQM” method in terms of what to achieve (goal), what to evaluate to achieve the goal (question), and what to use as the evaluation method (metric).

4.4.3. Setting Goals for Each Area

Based on the IoT security-quality requirements discussed in Section 4.2, the goals for each area of the transparency model were set, as listed in

Table 2. Goals for each area of transparency model.

Area	Goals
1-A. Security by Design (Corporate Policy & Development Process Standard)	G1A-1: To provide secure products which gain the trust of customers.
	G1A-2: To define the corporate standard of secure development processes so that all products provided can be manufactured with security throughout the product lifecycle.
1-B. Security by Design (Security measures, Secure Development)	G1B: To develop secure products based on the defined development standard from the planning stage of the product lifecycle.
2. Security Assurance Assessment	G2: To evaluate and confirm that secure products are developed as designed.
3. Security Production	G3-1: To carry out production with a secure production operating system to avoid containing security risks.
	G3-2: To secure the supply continuity.
4. Security Operation	G4: To take prompt actions to minimize the damage to customers when a security risk becomes apparent in the provided product.
5. Compliance with Law, Regulation, and International Standard	G5: To provide products complying with laws, regulations, and international standards of the destination market.

.

We set high-level goals for each area/phase. Furthermore, we excluded product-specific indicators from the goals because these would vary for each product and industry.

The Security by Design area under Area 1 is subdivided into two main areas. The goal of 1-A is to establish a basic policy for providing a secure product that would earn the trust of consumers and define a basic process for implementing the policy. This allows users to trust in management’s commitment to developing secure products. G1A-1 corresponds to the first aspect of R1 and R5 of Table 1. G1A-2 corresponds to the second aspect of R1.

The goal of 1-B, G1B, is to develop a product that considers security throughout the lifecycle of the product in accordance with the basic policy and process in 1-A.

The goal of Area 2, G2, is to ensure that the product developed in Area 1-B is secure, as designed.

Area 3 is a perspective specific to IoT and is absent from general software development. Because IoT products consist of both software and hardware, they are assembly-processed similar to software products. The production process entails such actions as physical assembly, serial number labeling, and the setting of device-specific IDs and passwords for security. In certain cases, the hardware components required for production may be procured externally and assembled manually. Thus, during production, after verifying the security of the product, supplementary actions are taken to finalize the product prior to shipping to the market. There are security risks involved in this process, and the goal is to eliminate or reduce those risks in this area. Each of G1B, G2, G3-1, and G3-2 correspond to the requirement of R2.

The objective of Area 4 is to provide a unique security response that is different from traditional quality assurance. Traditional quality assurance operates such that if a product performs to a certain standard, it is shipped. However, unless a product that does not meet the standard is found in the marketplace (i.e., unless the personnel are notified of a problem by users), the quality assurance personnel do not check and monitor the status of the products in the market themselves. On the other hand, in the world of security, even with the best efforts to develop a secure product, the level of security perceived to be secure is changing every day as attack techniques constantly evolve. Security risks will gradually increase from the time the product is shipped. It is therefore necessary to monitor changes in the circumstances surrounding the product even after it is shipped. Accordingly, the goal, G4, is to have a response system in place to check and correct any product-related security issues discovered and be ready to respond at any time. G4 corresponds to the requirement of R4.

In the traditional approach, if a quality issue occurs after shipment, the cause of the problem may be identified and addressed. In contemporary scenarios, however, a security problem is different from traditional quality assurance because these problems are manifested by a malicious attack and must be dealt with via nonconventional means.

The goal of Area 5, G5, is to comply with the IoT security laws and regulations with which increasing numbers of countries and regions have been demanding conformity in recent years. In some cases, product sector-specific guidelines are provided in some markets; they are required as industry standards to comply. Although this objective should naturally be considered at the design stage, its content is subdivided into different areas. This is because security-related laws and regulations have recently come into force, and the requirements are related to the entire product lifecycle. Significant regional differences also exist. G5 corresponds to the requirements of R3-1 and R3-2.

4.4.4. Setting Sample Questions and Metrics for Each Goals (Step 3–4)

Based on the GQM method mentioned above, the questions and metrics were formulated for each area from the perspectives clarified in the previous research on the aspects of “What do you want to know about IoT security quality?” and “How do you want to make sure?”.

From the perspective of IoT consumers, the questions are intended to reveal what security measures are being taken and how secure the products being made are.

From the perspective of the IoT vendors, identifying what to do and when to do it in the development process is necessary.

The metrics were devised with the following points in mind:

(a)

Do the metrics make sense to IoT vendors?

(b)

What are the criteria for the metrics?

(c)

Will they interfere with the existing development process?

For (a), clarifying the reason for performing metrics makes it easy to understand. For (b), the metrics are formulated based on “what and when”, whereas for (c), the metrics are clear and can be incorporated into existing design processes.

First, the primary questions were listed. The secondary questions were then used to set up a more specific perspective and provide supplementary confirmation.

The metrics at this stage are set as simple assessments, such as the presence or absence of documentation and whether assessments were performed.

The reason for a simple evaluation is that a clear basis or objective indicator to classify the content of each response does not exist. When the organization is sufficiently mature to implement advanced initiatives, these questions and metrics sets are likely to evolve into an advanced form of evaluation by establishing complex questions and metrics with approximately three levels, such as well done, partially done, and nothing done.

The quality and security experts then review and evaluate the validity of the draft questions and metrics in Step 4 of the research method to refine the list of questions and metrics.

The details of the approach to set the IoT security-quality metrics and the results of the examination of the questions and metrics for each area are presented in Appendix A. However, the results of this study are based on the elimination of field-specific product perspectives as much as possible. As such, these results should be considered an example of questions and metrics for IoT in general. If there is a field-specific item necessary to assess, it can be modified to be field-specific by adding such field-specific questions and metrics.

5. Evaluation of Various IoT Security Guidelines with the Sample Metrics

In this section, the characteristics of the requirements presented in IoT security regulations, guidelines, and certification programs are examined. This examination is based on the common security-quality metrics for IoT products reviewed by the quality and security experts as a part of Step 5 in the research method to examine the effectiveness of the metrics. The results are presented in the form of bar charts, respectively.

Table A7. Sources of references examined.

Name of Source	Doc Type	Year	Country	Issued by	Org Type
Telecom Business Act [48]	Law/Regulation	2020	Japan	Ministry of Internal Affairs and Communications	Government
State Bill 327 [49]	Law/Regulation	2020	USA	State of California	Government
House Bill 2395 [50]	Law/Regulation	2020	USA	State of Oregon	Government
Consumer IoT Security Consultation [51]	Law/Regulation	2020	UK	Department for Digital, Culture, Media & Sport	Government
EN 303 645 v2.1 [55]	Baseline Standard	2020	EU	European Communications Standards Institute (ETSI)	SDO
NISTIR 8259 [52], 8259A [53]	Baseline Standard	2020	USA	National Institute of Standards and Technology (NIST)	SDO
Baseline Security Recommendations for IoT [37]	Baseline Standard	2017	EU	ENISA (European Union Agency for Cybersecurity)	Government
The C2 Consensus on IoT Device Security Baseline Capabilities [54]	Baseline Standard	2019	USA	Council to Secure the Digital Economy (CSDE)	Industry
IoT Common Security Requirements Guidelines 2021 [56]	Certification	2020	Japan	Consumer Connected Device Security Council (CCDS)	Industry
ioXt 2020 Base Profile ver.1.0 [57]	Certification	2020	USA	ioXt Alliance, Inc.	Industry
Methodology for Marketing Claim Verification: Security Capabilities Verified to level Bronze/Silver/Gold/Platinum/Diamond, UL MCV 1376 [58]	Certification	2019	USA	UL LLC	Industry

in Appendix B shows the source of references examined for evaluating the effectiveness of the IoT security-quality metrics of this study.

5.1. Regulations

The following four regulations are compared with the IoT security-quality metrics: Terminal Conformity Regulation under Telecommunications Business Law by Ministry of Internal Affairs and Communications of Japan [48], California Senate Bill No. 327 [49], Oregon House Bill 2395 [50], and the consultation on regulatory proposals on consumer IoT security of the UK [51].

Figure 3 illustrates the area of the transparency model under which each regulatory requirement falls. The percentages on the vertical scale indicate the ratio between the number of requirements of each regulation that correspond with the IoT security-quality metrics items and the total number of IoT security-quality metrics items in each area. The number on the horizontal axis is the total number of metrics set for each area. Thus, the percentage for each area is the ratio of the number of metrics matched to the total number of metrics. The same is the case for Figure 4 and Figure 5, shown below.

The requirements of these regulations are minimal, as can be observed in Figure 3. It is also clear that Area 1-A of vendor attitude (such as security policy) or Area 3 of assessment (such as vulnerability assessment) are not required. Therefore, the IoT security-quality metrics cover the range of regulatory requirements sufficiently well to ensure compliance.

As observed in Figure 3, all these regulations focus on Area 1-B and Area 3. The UK regulation requires a security operation after product sales.

5.2. Baseline Guidance

The following four standards and guidelines from the United States and Europe that are presented as baselines are examined here. These are NISTIR 8259 [52] and 8259A [53] and C2 Consensus on IoT Device Security Baseline Capabilities [54] of the US, and Baseline by ENISA [37] and ETSI EN 303 645: Cyber Security for Consumer Internet of Things: Baseline Requirements [55].

Figure 4 describes the results of the area of the transparency model that each baseline requirement fits into. The vertical scale indicates the same units as that in Figure 3. Values over 100% indicate that there are a greater number of requirements than the total number of IoT security-quality metrics items in each area.

The distributions of the two standards from the US are similar, and the trend of the requirements can be considered to follow the same direction. Certain functional requirements for devices that were not set in the IoT security-quality metrics were found in these two US standards.

The approaches taken by the two European guidelines differ from that of the “baseline”. The ENISA baseline offers broad coverage, and many requirements appeared in all areas. In particular, the security function requirements in the area of security-by-design are very extensive, and hence incomparable to the sample metrics. On the other hand, the distribution of ETSI requirements resembles that of the two from the US, and the approach to baselines is also considered close.

Figure 4. Bar chart of requirements distribution of IoT security baseline guidance.

Figure 4. Bar chart of requirements distribution of IoT security baseline guidance.

5.3. Certifications

Several private IoT security certification programs have been released in the market. We examined the following four sets of requirements. The first is from the certification program of CCDS [56] in Japan, and the second is from the ioXt alliance [57] in the US. Finally, we analyzed the two different grades (Bronze and Diamond) of the IoT Security Rating of UL [58], also in the US.

The result for the area of the transparency model to which each certification requirement belongs is described in Figure 5. The vertical scale represents similar concepts as those in Figure 3 and Figure 4, and the meaning of the values that are greater 100% is also the same.

Except for the requirements of UL Diamond, the other programs have a similar number of requirements, and these are covered (i.e., they are below the 100% line) by the metrics.

We also found that the requirements in the security functions of Area 1-B of UL Diamond are quite strict compared to the same level of ENISA baseline requirements. This implies that the ENISA baseline requirements are a very high-level set of requirements, despite being baselines.

Figure 5. Bar chart of requirements distribution of IoT security certifications.

Figure 5. Bar chart of requirements distribution of IoT security certifications.

6. Evaluation of IoT Devices with the Proposed Method

We evaluated the IoT devices by the proposed method. We selected two commercial dashcam recorders (Product A and B) with almost the same functional product specifications as IoT devices. These are products provided by Original Design Manufacturing (ODM) vendors.

6.1. Target IoT Devices

The two products are similar in the following aspects:

• They are consumer products that can be purchased online and in stores.
• A full-HD high-definition recording is the main selling point.
• GPS location recording.
• Wi-Fi (wireless) connectivity with a smartphone.
• 16 GB storage space.
• Easy to install and start using by powering from a cigar socket.
• Downloadable applications for smartphones and PCs that can be connected to and functionally linked with a dashcam.

As mentioned above, the two dashcams are very similar in terms of functionality. The only differences observed from the specification are the following points.

• Product design: shape and color.
• Price: Product A is cheaper than Product B.

Simply speaking, as they are almost the same in terms of functionality, most users will choose Product A because of the price difference, unless they prefer the design of Product B too much.

However, as a user, the following points not readily apparent from the functional specifications are of concern. The points are the policy for handling personal information such as recorded image information, GPS information, information about the user, and the access restriction function for connection functions.

6.2. Evaluation with the Proposed Method

As far as we were able to confirm through interviews with ODMs, we evaluated and compared the security perspective with the metrics of the proposed method.

The results of the evaluation are shown in Figure 6.

The comparison results show that Product B has more product security measures in all areas than Product A, and we can infer that the security quality of Product B is better. This difference is probably reflected in the price difference.

Both companies had policies for handling personal information. However, there was a difference in the authentication of the Wi-Fi connection: Product A had the factory default access point name as the product name and no password (blank). Product B, on the other hand, had the same factory setting with the product name as the access point name, but the password was set to be unique for each device. This perspective is critical, as the requirements affect compliance with California law. Although the specification that anyone can use the device immediately without a password is appealing, the default setting that only the purchaser of the device can access is safer. Even in Product B, we found that there are few efforts in Area 2 and Area 4. The security-conscious IoT vendor of Product B has yet to demonstrate security verification or postshipment support.

6.3. Evaluation Result

The proposed method demonstrated that it could illustrate the differences in the security quality of IoT devices. In addition, it could increase the transparency of the security quality of IoT devices.

It is not easy for general users at present to make this kind of comparative evaluation since they have only the product specifications released by IoT vendors to judge.

However, IoT vendors will want to appeal to users the security measures they have invested in during the development and maintenance of IoT devices. At that time, this method can be a tool to support improvement and raise the level of security measures, since it visualizes areas where security measures are lacking.

7. Discussion of Findings

As described in Section 3, the IoT security-quality metrics are examined from a product lifecycle perspective; quality items are articulated in a manner inspired by GQM methods common in the quality industry, and the metrics that were reviewed by quality and security experts are produced.

Originally, this methodology was designed to help IoT vendors to produce their own IoT security-quality metrics. However, it has also proved to be a useful tool for developing categories of guidelines and certification programs to understand which requirements are lacking or missing from the product lifecycle. In practice, international standards by themselves are insufficient for practical implementation; hence, it is necessary to tailor the contents of international standards to suit the development target, development process, organization, and environment. As discussed previously [59], GQM is used for this tailoring. In the future, as IoT vendors’ product security efforts advance, improvements will be required; the validating GQM (V-GQM) is proposed as a method for reviewing or improving each element of GQM [60]. The review or improvement should occur as soon as the values of the metrics are collected. The use of such a method is expected to make it easier to implement reviews and improvements.

As mentioned in Section 4, all requirements are not distributed evenly throughout the product lifecycle. All regulations are focused on Areas 1-B and 3, whereas only the UK focuses on the maintenance phase of Area 4. Additionally, ENISA suggests incorporating the items in all areas (especially items on high demand) into the policy, process, and security functions at the design phase. Other baselines focus on security functions and operations rather than the level of ENISA. Most certifications focus not only on security functions but also on security assurances.

This approach was also useful in helping IoT vendors to understand how regulatory requirements, guidelines, and certification program requirements are distributed across the product lifecycle and where they are focused. The results of Section 5 reveal that not all sets of requirements are the same and that there are differences in requirements. This may help IoT vendors tailor their IoT security-quality metrics according to the particular requirements specified by consumers. If any deficiencies are found, IoT vendors can make improvements by eliminating them from the quality metrics to meet the security-quality objectives early in the lifecycle of the product being developed, thereby saving time and effort.

In addition, we believe that this method will also serve as an indicator of the product security standard for consumers. From the results of Section 6, we also verified that this method could illustrate that there is a clear difference in security quality, which is difficult to indicate in the difference in product features in functional specifications. To date, a way to communicate the quality of IoT security has not existed. Nevertheless, we foresee that this novel approach will become a quality communication tool between product vendors and consumers.

A limitation of the proposed method is that it is conceived from a framework that assumes a conventional V-shaped development model. Therefore, we have not been able to evaluate its applicability to recent development methods such as an agile development [61,62] and DevOps [63,64].

8. Future Directions

There are three related areas that we would like to pursue in future work. The first possibility is to categorize the metrics that show the countermeasure capabilities of IoT products and those that demonstrate the efforts of IoT vendors. The current metrics belong to either or both of these areas. We plan to examine how to categorize metrics to easily distinguish between the quality of security in IoT products and the quality of the IoT management process at a glance. The second area that may warrant further research involves investigating methods to visualize the coverage of metrics. Herein, the authors selected the bar chart for this purpose; however, comparatively simple methods for visualizing the coverage may be available. Thirdly, although we believe that IoT vendors can adopt this proposed method because of their knowledge of the product manufacturing culture of electronic-device vendors, we also consider it necessary to ask several IoT vendors to adopt this method to verify whether their IoT products are developed securely. In addition, when security support by IoT vendors becomes common practice, and security threats are evolving day by day, we will need to add and refine the basic set of metrics in detail and would need to consider its refinement cycle in the future.

9. Conclusions

This study proposes a method for tailoring security-quality metrics for IoT devices to ensure the quality of IoT security, and the method demonstrates the validity to evaluate the characteristics of the emerging requirements and suggestions of relevant laws, regulations, guidelines, and certification programs in IoT security based on the produced metrics. In addition, the proposed method demonstrates its capability to reveal the difference of security quality behind the product functional specification of IoT devices.

The authors developed the six areas of the transparency model of IoT security quality to outline the entire product lifecycle with reference to the GQM methodology. The draft was reviewed by quality and security experts who reflected upon the findings and incorporated them into the final set of metrics.

To validate the metrics, they were analyzed against the requirements of various IoT security regulations, guidelines, and certification programs. Most of the regulations, guidelines, and certification programs only describe what needs to be actioned without designating an entity to put into practice and without clarifying the purpose of action to perform; this results in ambiguity over the extent of duty and may lead to nothing being accomplished. With this metric, however, the rationale is clear from the GQM. Thus, it is easier for the entity or IoT vendors to self-assess the security-quality metrics items needed for their security goal.

Although many guidelines are available for the development of secure software, no practical framework follows the lifecycle of a hardware-oriented product that is easy for device vendors to understand. The presentation of the metrics for each area as a framework enables IoT vendors to easily incorporate security initiatives into their existing processes.

The proposed method and the metric for IoT security quality were examined and found to be useful for (1) developing categories of guidelines and a set of requirements to review the balance of items throughout the product lifecycle, (2) enabling IoT vendors to determine the focal areas of guidelines and a set of requirements to meet, and (3) enabling consumers to use these IoT product security-quality metrics to communicate the security risks to the product vendors.