Accuracy of Two CE-Marked Blood Glucose Monitoring System Based on EN ISO 15197:2015

Abstract

Blood glucose monitoring systems (BGMS) are essential for the management of diabetic patients. Although International Organization for Standardization (ISO) 15197:2015 criteria require rigorous monitoring of BGMS devices before commercialization, manufacturing quality standards may decline after FDA or EU approval. This work aimed to demonstrate the accuracy and precision of two BGMS devices currently available on the market. A laboratory study was conducted from June to August 2021 using two BGMS devices. One hundred samples were collected and evaluated according to ISO 15197:2015 guidelines. Over 95% accuracy was achieved by both devices using stricter ISO criteria (at least 95% of values within ±10 mg/dL or ±10% of the results of the reference measurement procedure). Analysis of the error grid showed that 99.5% of the results were in zone A. Surveillance of the accuracy and precision of BGMS devices after FDA and EU approval is an essential procedure to help patients and physicians manage glycemia and determine an appropriate outcome and personalized approach to diabetes treatment.

1. Introduction

Blood Glucose Monitoring Systems (BGMS) are essential for controlling blood glucose (BG) in patients with diabetes because they help patients, their clinicians, and caregivers in the daily management of the disease to monitor diabetes therapy and maintain glycemic targets [1]. In addition, monitoring patients’ blood glucose helps clinicians prevent potential and dangerous metabolic conditions, such as hypoglycemia or hyperglycemia, facilitates their work, and promotes optimal insulin management, especially in patients with intensive insulin therapy or insulin pumps [2,3,4,5]. Therefore, the reliability and accuracy of BGMS are essential requirements for supplying adequate support to clinical decisions [6].

The International Organization for Standardization’s (ISO) standard 15197:2013, first published in 2003 and revised in 2013 with the introduction of more stringent criteria (harmonization EN ISO 15197:2015), defines the accuracy and precision requirements necessary for the BGMS [7]. Several studies have demonstrated that not all BGMS approved in Europe (UE) or the USA fulfilled accuracy standards after being approved and launched on the sale market. Keeping high accuracy and precision levels is crucial in reducing hypoglycemia and HbA_1c and assuring optimal therapy [8]. According to King et al., 46.4% of BGMS currently available on the market failed to pass ISO 15197:2013 after approval [9] because of a probable decline in quality standards in the manufacturing process [10]. Inaccurate BGMS measures could be potentially life-threatening and with relevant economic repercussions [9].

Two BGMS, Oh’CareLite (Device 1) and Oh’Care (Device 2) (OSANG HealthCare Co., Ltd., Anyangcheondong-ro, Dongan-gu, Anyang-si, Gyeonggi-do, Korea), currently available on the European and American markets, have been developed and monitored to maintain accuracy standards during the entire production cycle. This work aimed to demonstrate the accuracy and precision of these two BGMS devices as assessed by trained operators according to the criteria outlined in ISO 15197:2013.

2. Materials and Methods

The Institutional Review Board approved the study—Ethics Committee of the Regione Calabria Sezione Area Centro (protocol number 143, approval date 22 April 2021). The study was conducted from June to August 2021. Key features of the two BGMS are reported in

Table 1. Technical specification.

Features	Description
Blood sample	Capillary and/or venous whole blood
Sample volume	0.5 µL
Measuring range	20–600 mg/dL
Analysis time	5 s
Operating conditions	Temperature: 10–40 °C 20–25 °C (control measurement) Humidity: 10–90% (meter, test strips)
Storage conditions	Temperature: 2–30 °C (meter, test strips) 8–30 °C (control solution) Humidity: dry place
Hematocrit range	20–65%
Measurement technology	Glucose dehydrogenase
Comparison measurement method	Glucose oxidase
Calibration	Plasma
Coding	Auto-coding

.

The accuracy and precision of the two BGMS devices were determined by comparing the glycemic results of both devices with those obtained through the glucose-oxidase (GOD) method (SAM 300 MINDRAY BS200). The evaluation was conducted according to the real use-conditions of the two devices so that both systematic errors (bias) and random errors (non-precision) could have been included in the evaluation. Three lots of the reagent system and at least 20 reagent system packages for each lot were used. The choice of lots and reagent system packages was completely random. According to information provided by the manufacturer of the comparison measurement system, the SAM 300 MINDRAY BS200 glucose analyzer was checked for traceability requirements of EN ISO 17511:2003. In addition, the trueness and precision of the comparison measurement system were verified during the test procedures.

Participants were volunteers without any form of remuneration and were enrolled randomly. Inclusion criteria were: (1) Adults (≥18 years old); (2) Patients with type I diabetes and intensified insulin therapy or pump therapy; (3) Subjects with blood glucose values below 80 mg/dL or above 300 mg/dL after short-term alteration of insulin therapy; (4) Subjects legally capable of understanding and willing and able to understand the character, meaning and consequences of the study; (5) Subjects who gave informed consent in writing. Exclusion criteria were: (1) Patients with type I diabetes and simultaneously suffering from one of the following pathologies: coronary heart disease, myocardial infarction, brain events, occlusive peripheral arterial disease or in an unrecognized situation of hypoglycemia; (2) Pregnant or breastfeeding patients; (3) Patients with severe acute illness; (4) Patients with severe chronic illness; (5) Anemia; (6) Suspected lack of compliance with the inclusion criteria; (7) Relationship of dependence on the investigator or sponsor. Inclusion and exclusion criteria were defined according to ISO 15197:2013 and EN ISO 15197:2015.

The medical history and drug therapy of each subject involved in the study had been evaluated by an experienced physician. The possible presence of interfering substances (acetaminophen or paracetamol, salicylic acid, ascorbic acid, dopamine) was assessed. In addition, the hematocrit value of each subject was checked to be within the indicator range on the manufacturer’s label.

Capillary blood samples were used to evaluate the accuracy of the system. First, for each sample, a sufficient volume of blood was collected. Then, the glucose concentrations were distributed according to the classifications shown in the following

Table 2. Distribution of BG concentration of samples for BGMS as specified in ISO 15197:2015.

Concentration Category	Glucose Concentration (mg/dL (mmol/L))	Percentage of Samples (%)
1	≤50 (≤2.77)	5
2	50–80 (>2.77–4.44)	15
3	80–120 (>4.44–6.66)	20
4	120–200 (>6.66–11.10)	30
5	200–300 (>11.10–16.65)	15
6	300–400 (>16.65–22.20)	10
7	>400 (>22.20)	5

.

Each sample was assigned to the reference category according to the value reported by the average of two repeated measurements. For glucose concentrations above 400 mg/dL, a saline solution containing glucose was added. For glucose concentrations lower than or equal to 50 mg/dL, the adjustment was performed by incubation at a temperature of 34.3 °C and glycolysis.

For adjusted samples, PO₂ was checked using a blood gas analyzer immediately after the test procedure to ensure that PO₂ values were comparable with levels found in the native capillary blood samples [11]. The accuracy of the blood gas analyzer was verified during the experimental procedures.

Moreover, venous blood was taken from 5 participants to measure BGMS precision. The collected venous blood was placed in tubes containing anticoagulant EDTA.

2.1. Operating Procedures

Before carrying out each procedure, participants were be asked to wash and dry their hands. Next, we performed the blood glucose measurements maintaining controlled environmental temperature (from 18 °C up to 28 °C) and humidity (32.4% up to 50.9%) according to the manufacturer’s specifications and the ISO 15,197 standard. The procedures were performed by healthcare personnel trained to ensure the participants’ safety, the correct application of the protocol, limit possible bias due to instrument limitations, and avoid accidental use not following the manufacturer’s labelling. Finally, the devices for testing were cleaned and disinfected after each subject according to the recommendations on the manufacturer’s label.

2.2. Statistical Procedures

A total of 200 samples from at least 100 capillary blood samples/subjects were analyzed for each system. The standard deviation between the first and second measurements was determined. The deviation between the first and second measurements should be lower than or equal to 4 mg/dL for BG concentrations below 100 mg/dL or lower than or equal to ±4% for BG concentrations higher than or equal to 100 mg/dL. If the reference results exceeded these criteria, the measured values were not included in the analysis and all measurements were repeated on a different sample.

The accuracy of each system was evaluated by comparing the measurement results with the results of the measurement procedure obtained by GOD method. According to ISO 15197, accuracy was determined by adding the relative number of results within ±15 mg/dL (0.83 mmol/L) for glucose concentrations below 100 mg/dL (5.55 mmol/L) and the number of results within ±15% for glucose concentrations ≥100 mg/dL (5.55 mmol/L). More stringent criteria of ±10 mg/dL (0.56 mmol/L) and ±10% were also applied.

Consensus Error Grid (CEG) analysis of the three-reagent system lots was also investigated as defined by ISO 15197:2015. The number and the percentage of results within the clinically acceptable CEG zones A and B were evaluated for each lot and each BGMS. According to the Bland–Altman procedure, the relative bias was be calculated by reporting the relative confidence intervals [12]. The Passing–Bablok regression was also be presented graphically through scatter plots.

Data were collected in an Excel file and analyzed with STATA17 (StataCorp., College Station, TX, USA) and RStudio (RStudio Team 2020. Version 1.4.1717 for Mac RStudio: Integrated Development for R. RStudio, PBC, Boston, MA, USA, Version 1.4.1717).

3. Results

A total of 120 subjects were enrolled (68 females and 52 males, age: 60 ± 18 years), and only 100 were included in the study. Exclusion reasons are reported in

Table 3. Samples exclusion reasons.

Sample Taken from 120 Different Subjects	120
	Number of samples rejected
Based on ISO 15197:2013/EN ISO 15197:2015
The quality control measurement result (comparison method) was outside the acceptable limits o no valid measurement result was obtained	6
The concentration category was already filled	14

. Operating temperature and humidity conditions were 24 °C and 45%, respectively.

3.1. Accuracy of BGMS According to ISO 15197:2015 Guidelines

Minimum accuracy (within ±15 mg/dL or ±15%) requirements were fulfilled by both tested BGMS (Device 1: 98.5%; Device 2: 99.5%). More stringent criteria (at least 95% of values within ±10 mg/dL or ±10% of the reference measurement procedure’s results) was achieved by both tested BGMS (Device 1: 95%; Device 2: 96%). More stringent criteria (at least 95% of values within ±5 mg/dL or ±5% of the reference measurement procedure’s results) was not achieved by both tested BGMS: 67% of samples were within the reference range. BGMS achieved an accuracy level of 100% (

Table 4. BGMS accuracy.

(a) First Device
Glucose concentration < 5.55 mmol/L (< 100 mg/dL)
	Within ±0.28 mmol/L (within ±5 mg/dL)	Within ±0.56 mmol/L (within ±10 mg/dL)	Within ±0.83 mmol/L (within ±15 mg/dL)
Lot 1	55/69 (79.7%)	67/69 (97.1%)	69/69 (100%)
Lot 2	49/69 (71.0%)	67/69 (97.1%)	69/69 (100%)
Lot 3	48/69 (69.6%)	66/69 (95.7%)	69/69 (100%)
Glucose concentration ≥ 5.55 mmol/L (≥100 mg/dL)
	Within ±5%	Within ±10%	Within ±15%
Lot 1	85/131 (64.9%)	124/131 (94.7%)	128/131 (97.7%)
Lot 2	90/131 (68.7%)	125/131 (95.4%)	128/131 (97.7%)
Lot 3	80/131 (61.1%)	118/131 (90.1%)	128/131 (97.7%)
System accuracy results for glucose concentration between 37 mg/dL and 450 mg/dL
	Within ±0.28 mmol/L or ±5%	Within ±0.56 mmol/L or ±10%	Within ±0.83 mmol/L or ±15%
Lot 1	140/200 (70.0%)	191/200 (95.5%)	197/200 (98.5%)
Lot 2	139/200 (69.5%)	192/200 (96.0%)	197/200 (98.5%)
Lot 3	128/200 (64.0%)	184/200 (92.0%)	197/200 (98.5%)
(b) Second Device
Glucose concentration < 5.55 mmol/L (<100 mg/dL)
	Within ±0.28 mmol/L (within ±5 mg/dL)	Within ±0.56 mmol/L (within ±10 mg/dL)	Within ±0.83 mmol/L (within ±15 mg/dL)
Lot 1	51/72 (70.8%)	67/72 (93.1%)	71/72 (98.6%)
Lot 2	39/72 (54.2%)	69/72 (94.4%)	72/72 (100%)
Lot 3	51/72 (70.8%)	70/72 (97.2%)	72/72 (100%)
Glucose concentration ≥ 5.55 mmol/L (≥100 mg/dL)
	Within ±5%	Within ±10%	Within ±15%
Lot 1	88/128 (68.8%)	124/128 (96.9%)	127/128 (99.2%)
Lot 2	90/128 (70.3%)	123/128 (96.1%)	128/128 (100%)
Lot 3	81/128 (63.3%)	122/128 (95.3%)	128/128 (100%)
System accuracy results for glucose concentration between 37 mg/dL and 450 mg/dL
	Within ±0.28 mmol/L or ±5%	Within ±0.56 mmol/L or ±10%	Within ±0.83 mmol/L or ±15%
Lot 1	139/200 (69.5%)	191/200 (95.5%)	198/200 (99.0%)
Lot 2	129/200 (64.5%)	191/200 (95.5%)	200/200 (100%)
Lot 3	132/200 (66.0%)	192/200 (96.0%)	200/200 (100%)

).

3.2. The Consensus Error Grid (CEG) Analysis According to ISO 15197:2015

CEG analysis showed 100% of results within the clinically acceptable zones A and B (Supplementary Materials—Figures S4 and S8). Only one sample for Device 1 and two samples for Device 2 were within zone B; the remaining samples fell within zone A.

3.3. The Bias Analysis

The bias of the measurement results was calculated according to the Bland–Altman for each lot of a system. Bland–Altman (Tables S1 and S3; Figures S2 and S6) and Passing–Bablock regression (Tables S2 and S4; Figures S3 and S7) confirmed a high level of accuracy for both devices.

3.4. Precision Analysis

A high level of precision emerged for both devices (

Table 5. BGMS precision.

First Device
	1.7 to 2.8 mmol/L(30–50 mg/dL)	2.9 to 6.1 mmol/L(51–110 mg/dL)	6.2 to 8.3 mmol/L(111–150 mg/dL)	8.4 to 13.9 mmol/L(151–250 mg/dL)	14.0 to 22.2 mmol/L(251–400 mg/dL)
Mean	40.75	101.2	130.97	165.63	310.51
SD	3.16	4.018	4.51	3.99	5.98
CV	0.078	0.040	0.034	0.024	0.019
Second Device
Mean	39.10	73.80	140.42	194.16	279.92
SD	2.55	3.10	3.99	3.82	5.00
CV	0.07	0.04	0.03	0.02	0.02

).

4. Discussion

BGMS accuracy represents a crucial feature to reduce hypoglycemia and monitor glucose in diabetic patients. Monitoring BGMS accuracy after FDA and EU market approval and during the entire production cycle is a needed procedure to assure a high level of safety of the BGMS.

This work aimed to demonstrate the accuracy and precision of two BGMS devices already cleared by FDA and UE and currently available on the market. The study was performed in accordance with the clinical trial protocol as defined in the ISO 15197:2013 guideline.

Recently, two studies demonstrated a loss of accuracy in the marketed current-generation BGMS as specified by ISO standards [6,13]. In Europe, 14 out of 18 BGMS had ≥95% of results within ±15 mg/dl, or ±15%, 3 out of 18 had 95% of results within ±10 mg/dL or ±10%, and none had 95% of results within ±5 mg/dL or ±5% [6]

From our findings, a high level of accuracy was achieved by both BGMS devices according to ISO standards (within ±15 mg/dL or ±15%). Furthermore, this level of accuracy was kept also applying more stringent criteria (within ±10 mg/dL or ±10%), while it was not achieved reducing the error margin within ±5 mg/dL or ±5%. Only 66% of measures fell in this last stringent criterion.

More than 99% of results were within zone A, which is considered to have “little or no effect on clinical outcome”. This result is encouraging given that the system accuracy was evaluated in a laboratory setting following testing procedures (optimal conditions of temperature and humidity and by trained personnel) as described by ISO 15197:2015 and not tested in a real use setting as everyday patient use.

Some studies indicate that BGMS performance could result in lower accuracy if tested by diabetic patients instead of trained personnel [14,15,16]. Although in our work we did not investigate possible measurement accuracy differences between trained and not-trained users, we believe that proper compliance to the manufacturer’s instructions (e.g., wash hand before use [17] or wait at least 15 min for temperature equilibration in case of a shift in the ambient temperature immediately before measurement [18]) in the use of these devices may prevent altered results. Moreover, the tested devices appeared insensitive to operator errors as technically possible. Future studies will evaluate the accuracy of BGMS available on the sale market considering users, environmental temperature, and humidity to reproduce real-world context.

The appropriate monitoring and evaluation of BG in diabetic patients are fundamental for clinicians and patients. Making accurate treatment decisions about insulin dosing, taking clinical actions to manage blood glucose levels, calibrating glucose monitoring and controlling closed-loop insulin delivery systems play a crucial role in managing diabetic patients. After approval by FDA and UE, BGMS devices need to be subjected to stringent surveillance to allow clinicians and patients to rely on ever-more precise, safe and accurate treatments.