Next Article in Journal
Focal Uptake in the Sternum on 18F-FDG-PET/CT Caused by G-CSF Therapy after Chemotherapy Mimicking Bone Metastasis of Breast Cancer
Previous Article in Journal
De Novo Mosaic 6p23-p25.3 Tetrasomy Caused by a Small Supernumerary Marker Chromosome Presenting Trisomy Distal 6p Phenotype: A Case Report and Literature Review
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Diagnostics
Volume 12
Issue 10
10.3390/diagnostics12102307
diagnostics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Combined Locked Nucleic Acid Probes and High-Resolution Melting Curve Analysis for Detection of Rifampicin-Resistant Tuberculosis in Northern Thailand

by
Yee Mon Thant
1,
Sukanya Saikaew
1,2,
Chayada Sitthidet Tharinjaroen
1,3,
Ponrut Phunpae
1,3,
Rodjana Pongsararuk
4,
Kanya Preechasuth
1,3,
Bordin Butr-Indr
1,3,
Sorasak Intorasoot
1,3,
Khajornsak Tragoolpua
1,3,
Angkana Chaiprasert
5 and
Usanee Wattananandkul
1,3,6,*
1
Division of Clinical Microbiology, Department of Medical Technology, Faculty of Associated Medical Sciences, Chiang Mai University, Chiang Mai 50200, Thailand
2
Faculty of Public Health, Chiang Mai University, Muang District, Chiang Mai 50200, Thailand
3
Infectious Diseases Research Unit (IDRU), Faculty of Associated Medical Sciences, Chiang Mai University, Muang District, Chiang Mai 50200, Thailand
4
Office of Disease Prevention and Control Region, 1 Chiang Mai, Department of Disease Control, the Ministry of Public Health, Muang District, Chiang Mai 50000, Thailand
5
Office for Research and Development, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok 10700, Thailand
6
Epidemiology Research Group of Infectious Disease (ERGID), Faculty of Nursing, Chiang Mai University, Chiang Mai 50200, Thailand
*
Author to whom correspondence should be addressed.
Diagnostics 2022, 12(10), 2307; https://doi.org/10.3390/diagnostics12102307
Submission received: 9 September 2022 / Revised: 21 September 2022 / Accepted: 21 September 2022 / Published: 24 September 2022
(This article belongs to the Section Diagnostic Microbiology and Infectious Disease)
Browse Figures
Versions Notes

Abstract

:
Rifampicin-resistant tuberculosis (RR-TB) has become a major threat globally. This study aims to develop a new assay, RIF-RDp, to enhance the detection of RR-TB based on combined locked nucleic acid (LNA) probes with high-resolution melting curve analysis (HRM). Two new LNA probes were designed to target the class-III and IV mutations of rpoB, H526D, and D516V. LNA probes showed 100% specificity in the detection of mutant targets among characterized and blinded Mycobacterium tuberculosis (Mtb) isolates. The performance of RIF-RDp was evaluated using 110 blinded clinical Mtb isolates in northern Thailand against drug-susceptibility testing (DST), DNA sequencing, and a commercial real-time PCR kit. This assay showed sensitivity and specificity of 94.55% and 98.18% compared to DST, and 96.36% and 100% compared to DNA sequencing. The efficacy of RIF-RDp was comparable to the commercial kit and DNA sequencing. The Cohen’s Kappa statistic showed almost perfect agreement between RIF-RDp and the commercial kit (κ = 0.95), and RIF-RDp and DNA sequencing (κ = 0.96). Furthermore, this is the first report of the rare mutation profiles, S531W, and a triple codon deletion (510–512) in northern Thailand. According to high accuracy, the RIF-RDp assay may render an easy-to-use, low-cost, and promising diagnostics of RR-TB in the future.

1. Introduction

Tuberculosis (TB) is a serious threat to global health that remains the second deadliest infectious disease after COVID-19. According to the WHO report, in 2019, an estimation of TB and death cases was 10.0 million and 1.4 million, respectively [1]. The emergence of drug-resistant TB (DR-TB) is a significant problem for TB control programs worldwide. In 2019, there were 465,000 incident cases of MDR-TB, and a global proportion of 78% of rifampicin (RIF) resistance TB had multidrug-resistant TB (MDR-TB) [1]. MDR-TB is caused by Mycobacterium tuberculosis (Mtb) strains that are simultaneously resistant to RIF and isoniazid (INH). It is also one of the most urgent and difficult challenges facing the global epidemic [2]. As RIF resistance is frequently concomitant to INH resistance, it was proposed as a surrogate marker for MDR-TB. Moreover, in 2018, the WHO introduced a short-course, all-oral regimen for the treatment of RIF-resistant (RR-TB) [3]. Thus, early detection of RIF resistance is vital to improve the management of MDR/RR-TB and eradicate TB from the world. However, a simple and accurate method for the diagnosis of RR-TB is needed for the successful treatment and prevention of DR-TB transmission.
Rifampicin (RIF), also known as rifampin, was introduced for use as an anti-TB drug in 1972 [4]. It serves as one of the major bactericidal antibiotics used to treat several types of bacterial infections, including TB. Nearly 95% of RIF-resistant mutations are associated with the rpoB mutation hot spot region so-called “the rifampicin resistance determining region (RRDR)” [5,6]. This 81-bp region ranges from codons 507 to 533 (27 amino acids) of the rpoB gene [7]. Rifampicin resistance found in Mtb is usually associated with a single-base substitution mutation within the rpoB gene that causes the single-nucleotide polymorphisms (SNPs). Four classes of SNPs are classified based on nucleotide base exchanges [8,9]. Class-I SNP involves transition substitutions (A/G→G/A or T/C→C/T). Transversion mutations of class-II, -III, and -IV SNPs contain A/C→C/A or T/G→G/T, C/G→G/C, and A/T→T/A changes, respectively.
Delay diagnosis of DR-TB impedes TB treatment and control. Conventional TB culture and drug susceptibility testing (DST) may take a long time (approximately 8−14 weeks) to obtain the Mtb drug susceptibility profiles. Although DST remains the standard method for the detection of DR-TB, its complexity, biosafety concern, the lack of facilities, and technical expertise are major limitations in low and middle-income countries (LIMCs). Several genotypic methods have been developed to offer an advantage over time-consuming phenotypic tests. However, the WHO recommended the Xpert MTB/RIF and line probe assays (LPAs) for diagnosis of DR-TB. The target coverage limitation, false positive results, and high rate of uninterpretable results are remaining problems for these methods [10,11]. Next-generation DNA sequencing is a high-throughput technique for the prediction of drug susceptibility among Mtb strains by detecting known and unknown mutations of antimicrobial resistance genes. However, the application of sequencing is prohibited by the need for robust software, massive output, and complex data analysis [12,13].
Real-time PCR with high-resolution melting curve analysis (HRM) has been applied as molecular detection of DR-TB [14,15,16]. HRM analysis provides the accurate identification of single-point mutations based on the different melting curves of PCR amplicons. However, only HRM analysis cannot detect all possible SNPs, especially class-III and -IV transversion mutations [17,18,19]. There are no changes in the number of hydrogen bonds of these two SNPs classes, thus, very little or no difference in melting temperature (Tm) between wild type and mutants. Recently, Anukool et al. [20] showed that the assays based on HRM, “RIF-RD” and “INH-RD”, have an overall good performance for the rapid detection of RR-Mtb and INH-R Mtb. However, the RIF-RD assay cannot detect H526D and D516V mutations among Mtb isolates from northern Thailand, which accounted for 10% and 3.3%, respectively [20]. Locked nucleic acids (LNAs) probes have previously been used in either real-time PCR or real-time PCR combined with HRM analysis (real time/HRM) to identify DR-TB [21,22]. Ramirez et al., 2010, reported the use of newly designed LNA probes in real time/HRM for the detection of the most common transversion mutations in MDR-TB [20,23]. LNA probes consist of modified oligonucleotide bases, which can increase the thermal stability, thus rendering high sensitivity detection of SNPs or single-point mutation and allowing a preferable signal-to-noise ratio. Therefore, this study designed two new LNA probes for the detection of the two most common class-III and -IV SNPs mutation found in Thailand [20,23]. Combining these probes with the previously developed assay, the new RIF-RDp assay was then established to improve the sensitivity of RR-TB detection. The potential of RIF-RDp assay efficiency was evaluated using blinded clinical isolates against standard phenotypic and genotypic assays, including a commercial real-time PCR test kit.

2. Materials and Methods

2.1. Clinical Mycobacterium Tuberculosis Strains and DNA Samples

For the optimization and validation of the RIF-RDp assay, reference strains of Mycobacterium tuberculosis ATCC 27294 (Mtb H37Rv), 20 known RIF-susceptible (RS), and 14 RIF-resistant (RR) strains were used (Table 1). One hundred and ten clinical Mtb isolates were included for the assay evaluation. DNA samples of all Mtb isolates were obtained from the TB regional laboratory, the Office of Disease Prevention and Control 1 (ODPC 1) Chiang Mai, Department of Disease Control, the Ministry of Public Health, Thailand. They were isolated from pulmonary TB patients at ODPC 1, which covers 15 hospitals in 8 provinces in northern Thailand (Chiang Mai, Chiang Rai, Phayao, Nan, Phrae, Lampang, Lamphun, and Mae Hong Son) and Chiangrai Prachanukroa Hospital (CPH) during 2015 to 2017. The identification of M. tuberculosis using the standard mycobacterial culture method. The Mtb isolates were recovered in 2% Ogawa medium at 37 °C for 4–8 weeks. Then, the genomic DNA was extracted using the extraction reagents of AnyplexTM MTB/NTM Real-time Detection (Seegene, Republic of Korea) according to the manufacturer’s instructions. The DNA was stored at −20 °C before being used in the experiments.

2.2. The Design of H526D Probe and D516V Probe

Two new LNA probes were designed to detect specific SNP transversions, the H526D mutation (C→G, class-III SNP) and the D516V mutation (A→T, class-IV SNP) by modifying the target nucleotide specifically. Three LNAs were placed into the probe by using Integrated DNA Technologies (OligoAnalyzer™ Tool): one LNA at the specific target site as the center, and the remaining two are at every third base on each side. The probes were designed based on the introduction of SNPs in H37Rv Mtb (GenBank accession no. JX303332.1). The 16-bp H526D probe covers nucleotide positions 1346 to 1361 and is labeled with HEX at the 5’ end and a black hole quencher, BHQ1, at the 3’ end. The 18-bp D516V probe ranges from nucleotide position of 1316 to 1333 and is labeled with Cy5 at the 5’ end and BHQ2 at the 3’ end. These two probes were synthesized by Integrated DNA Technologies, Coralville, IA, USA.

2.3. The RIF-RDp Assay

The RIF-RDp assay was designed to identify RR-TB by detection of H526D and D516V mutations using LNA probes and other mutations on the rpoB gene using HRM analysis. The optimal conditions of the RIF-RDp assay, including annealing temperature, primer and probe concentrations were initially determined. Genomic DNA of Mtb H37Rv (wild type reference) and MD62 (RR-Mtb reference), and DNase free water (negative control) were used as control of the RIF-RDp assay. The 200-bp H526D and D516V synthetic DNA templates (Integrated DNA Technologies, Coralville, IA, USA) were also included as mutant controls in each run of the RIF-RDp. The assay was performed in a total of 25 µL per reaction: 0.8X LightCycler® 480 HRM master mix including ResoLight high-resolution melting dye (Roche Diagnostics, Mannheim, Germany), 3.5 mM MgCl2, 0.3 µM of rpoB primers, 0.75 µM of each probe, and 100 ng of DNA template. The cycling profile for the assay was performed on the LightCycler® 96 real-time PCR system (Roche Diagnostics, Basel, Switzerland) under the following conditions: 95 °C for 3 min, and 40 cycles of 95 °C for 10 s, 62 °C for 15 s, and 72 °C for 30 s. After amplification, the PCR products were analyzed by HRM analysis based on the following conditions: amplicons were heated at 95 °C and cooled down at 40 °C for 1 min each, and the temperature rose from 65 to 97 °C at a rate of 0.04 °C/s.

2.4. Limit of Detection (LOD) of the RIF-RDp Assay

The LOD of the RIF-RDp assay (25-µL reaction) was performed in three separate reactions in duplication. To determine the assay LOD for the detection of the rpoB gene, the genome copies (gc) of Mtb H37Rv were calculated based on the amount of genomic DNA and Mtb H37Rv genome size [24]. Then, the Mtb H37Rv genomic DNA was 10-fold serially diluted from 2 × 106 gc (10 ng) to 2 × 101 gc (100 fg). To obtain 1 × 102 gc (500 fg), 2 × 102 Mtb H37Rv gc was two-fold diluted. Then, 2 × 106, 2 × 105, 2 × 104, 2 × 103, 2 × 102, 1 × 102, 2 × 101 H37Rv gc were used as DNA template for RIF-RDp LOD testing. To determine the LOD for the detection of H526D and D516V mutations in HEX and Cy5 channels, the gc of H526D and D516V synthetic DNA were calculated and the dilutions of the DNA were prepared in the same manner from 2 × 106 to 2 × 101 gc.

2.5. Evaluation of the RIF-RDp Assay

The RIF-RDp assay was performed on a total of 110 blinded Mtb isolates. These isolates were also analyzed by phenotypic DST, DNA sequencing, and a commercial kit (Anyplex™ II MTB/MDR assay, Seegene, the Republic of Korea).

2.5.1. Drug Susceptibility Testing (DST)

For the standard DST, Mtb isolates were performed by the proportion method according to the Clinical and Laboratory Standards Institute (CLSI) document M24-A2 [25] using Middlebrook 7H10 agar and incubated at 37 °C, ambient air. Isolates were considered resistant when ≥1 percent of colony growth was observed on the agar medium containing the critical concentrations of RIF at 1.0 μg/mL. The Mtb H37Rv was used as a quality control strain.

2.5.2. DNA Sequencing

The 543-bp rpoB gene amplicons were amplified by PCR primers previously described by Ramirez et al., 2010 [22], covering the RRDR. Before sequencing, amplicons were purified with NucleoSpin® Gel and PCR Clean-up (MACHEREY-NAGEL GmbH & Co. KG, Düren, Germany). DNA sequencing was performed by Bioneer Co., Ltd., Daejeon, South Korea using the ABI 3730xl DNA analyzer (Thermo Fisher Scientific, CA, USA). The mutation profiles were examined by Bioedit version 7.2 [26].

2.5.3. Multiplex Real-Time PCR Commercial Kit

The AnyplexTM II MTB/MDR Detection assay (commercial kit) was performed using the CFX96 real-time PCR System (Bio-Rad, Hercules, CA, USA). The 20-µL reaction mixture in a multiplex real-time PCR commercial kit, including 15 µL of PCR master mix and 5 µL of DNA sample, was analyzed according to the manufacturer’s protocol. Results were automatically interpreted using Seegene Viewer software (Seegene Inc., Seoul, Republic of South Korea) following the presets outlined by the manufacturer.

2.6. Statistical Analysis

For HRM analysis, the melting temperature (Tm) of the rpoB amplicons was analyzed. The Tm difference (∆Tm) of each sample was then calculated by subtraction of its Tm from the Tm of Mtb H37Rv amplicons. Based on the analysis of ∆Tm of all samples, the Survival Analysis in XLSTAT (Addinsoft, New York, NY, USA) was used to generate the receiver operating characteristic (ROC) curve for establishing the ∆Tm cutoff value for accurate differentiation between RR-Mtb and RS-Mtb strains [27]. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were determined to evaluate the performance of the RIF-RDp assay by comparing the efficiency with phenotypic DST, DNA sequencing, and Anyplex™ II MTB/MDR Detection assay. Moreover, the Cohen’s kappa agreement test between the assays was calculated using SPSS software v. 22.0.

3. Results

3.1. Optimization and the Validation of the RIF-RDp Assay

The RIF-RDp assay was set up based on the HRM analysis combined with LNA probes to identify RR-Mtb strains. The results interpretation algorithm was shown in Figure 1. If either H526D or D516V mutation was detected in the HEX or Cy5 channels (positive result), the result can be interpreted as resistant (R). When none of H526D or D516V was detected, other rpoB mutations were determined based on the ∆Tm cutoff value to differentiate RR-Mtb from RS-Mtb strains.
The annealing temperature was determined using PCR gradient, by increasing 2 °C within a range of 56–66 °C and the result showed the optimal annealing temperature at 62 °C. For the validation of LNA probes, RIF-RDp correctly distinguished H526D and D516V mutant strains from all RS-Mtb strains and RR-Mtb strains containing mutation profiles other than H526D and D516V (Figure 2).

3.2. LOD of the RIF-RDp Assay

It was found that the LOD of the RIF-RDp assay for the detection of rpoB gene, H526D, and D516V mutations was 100 genome copies of Mtb H37Rv, which is equivalent to 100 bacilli of Mtb as shown in Figure 3.

3.3. Evaluation of the RIF-RDp Assay and ROC Curve Analysis

Using 110 blinded clinical Mtb isolates, the RIF-RDp assay was evaluated versus DST (55 RR- and 55 RS-Mtb isolates), DNA sequencing, and AnyplexTM II MTB/MDR Detection assay. Using the ∆Tm of all isolates, the receiver operating characteristic (ROC) curve established the ∆Tm cutoff value at 0.245 °C to differentiate between RR-TB and RS-TB. The ROC curve showed the area under the curve (AUC) at 0.9, which indicated that this assay was highly accurate for the detection of rpoB mutations (Figure 4). Based on this cutoff value, the RIF-RDp assay efficiently distinguished RR-Mtb from RS-Mtb isolates (Figure 5a,b). H526D and D516V mutations were detected in the HEX and Cy5 channels (Figure 5c,d). The sensitivity, specificity, PPV, and NPV of the RIF-RDp assay and commercial kit against DST and DNA sequencing were shown in Table 2. The RIF-RDp assay showed high sensitivity and accuracy, that was comparable to the commercial kit and DNA sequencing. The RIF-RDp assay showed sensitivity, specificity, PPV, and NPV at 96.36, 100, 100, and 96.49% while the commercial kit achieved 94.55, 100, 100, and 94.83% when compared to DNA sequencing. In addition, the RIF-RDp assay had almost perfect agreement by Cohen’s Kappa coefficients with the commercial kit (κ = 0.95, 95%CI, 0.89–1), DNA sequencing (κ = 0.96, 95%CI, 0.92–1), and DST (κ = 0.93, 95%CI, 0.86–1).
In addition, to evaluate its potential use in Thailand, the price per test of the new assay was determined and compared to the commercial molecular tests available in the country. The cost of the RIF-RDp assay was assessed based on the expense on DNA extraction and HRM real time PCR. The machine maintenance, electricity, and personnel expenses were not included. The DNA extraction of Mtb isolates using AnyplexTM MTB/NTM Real-time Detection (Seegene, Republic of Korea) cost 200 Thai bahts (THB) per sample. The cost of a real-time PCR reaction (the HRM reaction mixture, one pair of primers, and 2 LNA probes) was 65 Thai bahts (THB) per 25-µL reaction. The overall cost was 265 THB, which were approximately 7.18 US Dollars ($) (the foreign exchange rates at 36.968 THB/US Dollar, as of 20 September 2022, https://www.bot.or.th). Anyplex™ II MTB/MDR assay and DNA extraction cost 42,500 THB per 50 samples. Therefore, it cost 850 THB ($23.01) per test. The price of Gene Xpert MTB/RIF cartridges was 500 THB each ($13.54), and LPA test including DNA extraction by the same kit (200 THB)/sample) and the GenoType MTBDRplus (Hain Lifescience, Nehren, Germany) (1800 THB) cost 2000 THB ($54.16).

4. Discussion

In this study, we developed a new assay RIF-RDp that incorporated the fluorophores-labeled LNA probes with the real-time PCR and HRM analysis to detect RR-TB that is associated with rpoB mutations in clinical Mtb isolates. This new assay successfully detected the class-III and –IV SNPs; H526D (C→G), and D516V (A→T) commonly found in northern Thailand [28]. Due to a slight shift of Tm detected in these SNPs, they were undetectable by the HRM-based assays previously reported [17,18,19,20]. This newly developed RIF-RDp assay showed a higher sensitivity compared to the previous RIF-RD assay [20]. The RR- and RS-Mtb isolates can be differentiated based on the accurate ∆Tm cutoff value. The LOD of the RIF-RDp assay was equivalent to 100 genome copies of Mtb H37Rv, which is lower than that of the RIF-RD assay (210 genome copies) [20] but higher than that of the Anyplex™ II MTB/MDR (20 genome copies).
The performance of the developed RIF-RDp assay on 110 blinded clinical isolates showed 94.55% sensitivity and 98.18% specificity when compared to DST. Among the blinded samples, we found that four samples showed discordant results. Three RR-Mtb isolates were identified as susceptible (negative) by RIF-RDp. The DNA sequencing result showed that two of three isolates contained rpoB mutation, one possessed class-II SNP (H526P, A→C) and another class-III SNP (S531W, C→G) mutations. Both S531W and H526P mutant samples generated the ∆Tm below the cutoff value though H526P mutation is class-II SNP. This may be caused by a low level of amplification due to inadequate DNA quality. This isolate was also identified as RIF susceptible by the commercial kit because H526P is not included in the assay targets. The last isolate showed no mutation by DNA sequencing of the 543-bp (codons 422 to 603) of rpoB and was also identified as susceptible by the commercial kit. This conferred RIF resistance may be resulting from other mutations outside the rpoB-target region or alternative mechanisms such as efflux-related mechanisms [29,30] or drug modifying enzymes [31]. In one isolate with a false positive result, DNA sequencing confirmed that it contained nonsense mutation with class-I SNP (CCC→CCT) at codon 535.
In this study, the most predominant mutations at codons 531, 526, and 516 accounted up to 88.8% of all RR-Mtb strains. Fifteen mutation profiles were detected by DNA sequencing. Interestingly, an Mtb isolate with a rare mutation profile, S531W, was firstly found in northern Thailand. S531W is a TCG→TGG substitution mutation that was associated with an MDR-TB outbreak in London, England during 1995–2004. This mutant isolate was firstly found in an MDR-TB developed patient with poor compliance therapy and was then detected in a primary MDR-TB patient [32]. Nevertheless, the S531W mutation was also the second most prevalent in Santa Catarina state, southern Brazil at 20.8% (11/53) [33].
To the best of our knowledge, this is the first report for triple codon deletion at codons 510, 511, and 512 in Mtb confirmed by DNA sequencing. Since the deletion causes the loss of three amino acids (glutamine, leucine, and serine), thus, conferring RIF resistance that correlates with DST. In 2008, Prammananan T, et al. reported that two MDR-Mtb isolates in Thailand harbored double codon deletion: one at codons 513 and 514; and another at codons 518 and 519 [23]. A study reported the detection of two RR-Mtb isolates in India that contained deletion at codon 518, could not be detected by LPAs due to the overlapping between the WT4 and WT5 probes [34]. In Brazil, one MDR-Mtb isolate containing four codon deletions at 516-518 has been reported in 2000. This study found that some codon deletions were combined with other single-point mutations, for example, one MDR-Mtb isolate had triple codon deletion from codons 514 to 516 and a point mutation at codon 513, and one isolate had triple codon deletion at codon 524, 525, and 526 and one codon deletion at codon 527 [35].
The RIF-RDp assay and the commercial kit showed equal specificity (100%) when compared to DNA sequencing. Although the RIF-RDp assay and the commercial kit are based on real-time PCR and can be performed up to 96 reactions in a single run, there are advantages and disadvantages to each one. The RIF-RDp assay rendered less time for detection. It took approximately 2.5 h from the preparation of reaction mixture to the analysis and interpretation of the result while the Anyplex™ II MTB/MDR assay took up to 3.5 h. If it is performed routinely, the RIF-RDp may need a 1-day total turnaround time (TAT) to report the results, which is comparable to other available Nucleic Acid Amplification tests (NAATs). However, the result interpretation of AnyplexTM II MTB/MDR is more convenient due to an automatic data analysis. The RIF-RDp assay requires the calculation of ΔTm between test samples and wild type reference to distinguish between RR-TB and RS-TB. A faster and more automated analysis and interpretation by integrating the user-friendly software for RIF-RDp like a viewing program used for Anyplex™ II MTB/MDR assay shall be innovated in future. In addition, the cost of the RIF-RDp assay per test using a 25-µL reaction is only US$7.18, which is about 2–3 times cheaper compared to the commonly used molecular tests in Thailand, Anyplex™ II MTB/MDR assay ($23.01/850 THB), Gene Xpert MTB/RIF ($13.54/500 THB), and LPAs ($54.16/2000 THB). Due to the high-cost equipment and cartridge, the use of Xpert MTB/RIF is still limited. LPA provides profiles of rpoB mutations but it requires a complicated process with PCR and DNA hybridization. Therefore, it takes a longer TAT, approximately 2 days. As there is a need for a real-time PCR machine and well-trained staff in RIF-RDp assay, it may be used as a candidate RR-TB assay in the reference or regional laboratories.
Since the RIF-RDp was developed based on HRM analysis that requires a high quality of DNA samples. The low quality of DNA samples can cause a low amplification of target that affects the melting process and may cause false negative result. This new assay was only tested in M. tuberculosis isolates cultured from clinical specimens. Further study of the direct detection of RR-Mtb in patient specimens like sputum is still needed to speed up the diagnosis of RR-TB. Only 3 H526D and 1 D516V mutant isolates and 14 isolates with Class-I-IV SNP mutation were available for the validation of the designed LNA probes and the evaluation of RIF-RDp. Testing with more mutant isolates harboring H526D and D516V mutations, and also other mutations, may be necessary to assure the reliability of this assay. Even though, there are several rare transversion mutation class-III and -IV SNPs have been reported [28]; herein, the RIF-RDp assay was designed to detect the two most common class-III and IV mutations found in northern Thailand. Since all RIF resistance-related mutations that occur outside the RRDR are not covered by the RIF-RDp assay, thus it cannot replace the standard DST method. However, the RIF-RDp assay can detect the majority of known, unknown, and rare mutations within RRDR that are associated with rpoB mutation in RR-TB, making it superior to other available molecular assays in terms of assay spectrum.

5. Conclusions

We developed the RIF-RDp assay by combining LNA probes with HRM analysis to detect RR-TB with high sensitivity and specificity. The newly designed probes showed 100% specificity for detection of the H526D and D516V mutant strains, which were not identified by HRM analysis alone. This assay is easy to perform, cost-effective, and beneficial for the diagnosis of RR-TB. The application of this assay at the reference or the regional laboratories in Thailand is proposed to improve the capacity in preventing treatment failure and control the future spread of drug-resistant TB.

Author Contributions

Conceptualization, U.W.; methodology, Y.M.T., C.S.T. and U.W.; software, Y.M.T., U.W. and S.S.; validation, U.W. and S.S.; formal analysis, Y.M.T. and U.W.; investigation, Y.M.T. and S.S.; resources, U.W., R.P., P.P., B.B.-I., S.I., K.P. and K.T.; data curation, Y.M.T., S.S. and U.W.; writing—original draft preparation, Y.M.T.; writing—review and editing, U.W., C.S.T. and A.C.; visualization, Y.M.T. and U.W.; supervision, U.W. and C.S.T.; project administration, U.W.; funding acquisition, U.W. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Faculty of Associated Medical Sciences, Chiang Mai University (grant no. R4030371).

Institutional Review Board Statement

This study was classified as exempt research by the Faculty of Associated Medical Sciences Ethic Committee (AMSEC-64EM-021).

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

This research work was partially supported by Chiang Mai University (CMU). Y.M.T. studied in a Master’s degree Program in Medical Technology, the Faculty of Associated Medical Sciences, CMU, under the CMU Presidential Scholarship. We also would like to thank Angkana Thanraka and Naowarat Kunyanone, Department of Medical Technology, Chiangrai Prachanukroh Hospital, Chiang Rai for providing DNA samples used in the experiment.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. World Health Organization. Global Tuberculosis Report. 2020. Available online: https://www.who.int/publications/i/item/9789240013131 (accessed on 30 August 2022).
  2. Solo, E.S.; Suzuki, Y.; Kaile, T.; Bwalya, P.; Lungu, P.; Chizimu, J.Y.; Shah, Y.; Nakajima, C. Characterization of Mycobacterium tuberculosis genotypes and their correlation to multidrug resistance in Lusaka, Zambia. Int. J. Infect. Dis. 2021, 102, 489–496. [Google Scholar] [CrossRef] [PubMed]
  3. World Health Organization. Rapid Communication: Key Changes to Treatment of Multidrug-and Rifampicin-Resistant Tuberculosis (MDR/RR-TB); World Health Organization: Geneva, Switzerland, 2018. [Google Scholar]
  4. Woodley, C.L.; Kilburn, J.O.; David, H.L.; Silcox, V.A. Susceptibility of mycobacteria to rifampin. Antimicrob. Agents Chemother. 1972, 2, 245–249. [Google Scholar] [CrossRef] [PubMed]
  5. Blanchard, J.S. Molecular mechanisms of drug resistance in Mycobacterium tuberculosis. Annu. Rev. Biochem. 1996, 65, 215–239. [Google Scholar] [CrossRef] [PubMed]
  6. Zaw, M.T.; Emran, N.A.; Lin, Z. Mutations inside rifampicin-resistance determining region of rpoB gene associated with rifampicin-resistance in Mycobacterium tuberculosis. J. Infect. Public. Health 2018, 11, 605–610. [Google Scholar] [CrossRef] [PubMed]
  7. Hillemann, D.; Weizenegger, M.; Kubica, T.; Richter, E.; Niemann, S. Use of the genotype MTBDR assay for rapid detection of rifampin and isoniazid resistance in Mycobacterium tuberculosis complex isolates. J. Clin. Microbiol. 2005, 43, 3699–3703. [Google Scholar] [CrossRef] [PubMed]
  8. Liew, M.; Pryor, R.; Palais, R.; Meadows, C.; Erali, M.; Lyon, E.; Wittwer, C. Genotyping of single-nucleotide polymorphisms by high-resolution melting of small amplicons. Clin. Chem. 2004, 50, 1156–1164. [Google Scholar] [CrossRef]
  9. Vignal, A.; Milan, D.; SanCristobal, M.; Eggen, A. A review on SNP and other types of molecular markers and their use in animal genetics. Genet. Sel. Evol. 2002, 34, 275. [Google Scholar] [CrossRef]
  10. Tomasicchio, M.; Theron, G.; Pietersen, E.; Streicher, E.; Stanley-Josephs, D.; van Helden, P.; Warren, R.; Dheda, K. The diagnostic accuracy of the MTBDRplus and MTBDRsl assays for drug-resistant TB detection when performed on sputum and culture isolates. Sci. Rep. 2016, 6, 17850. [Google Scholar] [CrossRef]
  11. Bunsow, E.; Ruiz-Serrano, M.J.; López Roa, P.; Kestler, M.; Viedma, D.G.; Bouza, E. Evaluation of GeneXpert MTB/RIF for the detection of Mycobacterium tuberculosis and resistance to rifampin in clinical specimens. J. Infect. 2014, 68, 338–343. [Google Scholar] [CrossRef]
  12. World Health Organization. The Use of Next-Generation Sequencing Technologies for the Detection of Mutations Associated with Drug Resistance in Mycobacterium Tuberculosis Complex: Technical Guide; World Health Organization: Geneva, Switzerland, 2018. [Google Scholar]
  13. Phelan, J.; O’Sullivan, D.M.; Machado, D.; Ramos, J.; Whale, A.S.; O’Grady, J.; Dheda, K.; Campino, S.; McNerney, R.; Viveiros, M.; et al. The variability and reproducibility of whole genome sequencing technology for detecting resistance to anti-tuberculous drugs. Genome Med. 2016, 8, 132. [Google Scholar] [CrossRef] [Green Version]
  14. Pietzka, A.T.; Indra, A.; Stöger, A.; Zeinzinger, J.; Konrad, M.; Hasenberger, P.; Allerberger, F.; Ruppitsch, W. Rapid identification of multidrug-resistant Mycobacterium tuberculosis isolates by rpoB gene scanning using high-resolution melting curve PCR analysis. J. Antimicrob. Chemother. 2009, 63, 1121–1127. [Google Scholar] [CrossRef] [PubMed]
  15. Pang, Y.; Dong, H.; Tan, Y.; Deng, Y.; Cai, X.; Jing, H.; Xia, H.; Li, Q.; Ou, X.; Su, B.; et al. Rapid diagnosis of MDR and XDR tuberculosis with the MeltPro TB assay in China. Sci. Rep. 2016, 6, 25330. [Google Scholar] [CrossRef] [PubMed]
  16. Anthwal, D.; Gupta, R.K.; Bhalla, M.; Bhatnagar, S.; Tyagi, J.S.; Haldar, S. Direct detection of rifampin and isoniazid resistance in sputum samples from tuberculosis patients by high-resolution melt curve analysis. J. Clin. Microbiol. 2017, 55, 1755–1766. [Google Scholar] [CrossRef] [PubMed]
  17. Choi, G.E.; Lee, S.M.; Yi, J.; Hwang, S.H.; Kim, H.H.; Lee, E.Y.; Cho, E.H.; Kim, J.H.; Kim, H.J.; Chang, C.L. High-resolution melting curve analysis for rapid detection of rifampin and isoniazid resistance in Mycobacterium tuberculosis clinical isolates. J. Clin. Microbiol. 2010, 48, 3893–3898. [Google Scholar] [CrossRef] [PubMed]
  18. Yadav, R.; Sethi, S.; Mewara, A.; Dhatwalia, S.K.; Gupta, D.; Sharma, M. Rapid detection of rifampicin, isoniazid and streptomycin resistance in Mycobacterium tuberculosis clinical isolates by high-resolution melting curve analysis. J. Appl. Microbiol. 2012, 113, 856–862. [Google Scholar] [CrossRef]
  19. Malhotra, B.; Goyal, S.; Bhargava, S.; Reddy, P.V.; Chauhan, A.; Tiwari, J. Rapid detection of rifampicin resistance in Mycobacterium tuberculosis by high-resolution melting curve analysis. Int. J. Tuberc. Lung Dis. 2015, 19, 1536–1541. [Google Scholar] [CrossRef]
  20. Anukool, U.; Phunpae, P.; Tharinjaroen, C.S.; Butr-Indr, B.; Saikaew, S.; Netirat, N.; Intorasoot, S.; Suthachai, V.; Tragoolpua, K.; Chaiprasert, A. Genotypic Distribution and a potential diagnostic assay of multidrug-resistant tuberculosis in northern Thailand. Infect. Drug Resist. 2020, 13, 3375–3382. [Google Scholar] [CrossRef]
  21. Zhao, Y.; Li, G.; Sun, C.; Li, C.; Wang, X.; Liu, H.; Zhang, P.; Zhao, X.; Wang, X.; Jiang, Y.; et al. Locked nucleic acid probe-based real-time PCR assay for the rapid detection of rifampin-resistant Mycobacterium tuberculosis. PLoS ONE 2015, 10, e0143444. [Google Scholar] [CrossRef]
  22. Ramirez, M.V.; Cowart, K.C.; Campbell, P.J.; Morlock, G.P.; Sikes, D.; Winchell, J.M.; Posey, J.E. Rapid detection of multidrug-resistant Mycobacterium tuberculosis by use of real-time PCR and high-resolution melt analysis. J. Clin. Microbiol. 2010, 48, 4003–4009. [Google Scholar] [CrossRef]
  23. Prammananan, T.; Cheunoy, W.; Taechamahapun, D.; Yorsangsukkamol, J.; Phunpruch, S.; Phdarat, P.; Leechawengwong, M.; Chaiprasert, A. Distribution of rpoB mutations among multidrug-resistant Mycobacterium tuberculosis (MDRTB) strains from Thailand and development of a rapid method for mutation detection. Clin. Microbiol. Infect. 2008, 14, 446–453. [Google Scholar] [CrossRef] [Green Version]
  24. Staroscik, A. Calculator for determining the number of copies of a template. URI Genom. Seq. Cent. 2004, 19, 2012. [Google Scholar]
  25. Woods, G.L.; Brown-Elliott, B.A.; Conville, P.S.; Desmond, E.P.; Hall, G.S.; Lin, G.; Pfyffer, G.E.; Ridderhof, J.C.; Siddiqi, S.H.; Wallace, R.J., Jr.; et al. Susceptibility Testing of Mycobacteria, Nocardiae, and Other Aerobic Actinomycetes; Clinical and Laboratory Standards Institute: Wayne, PA, USA, 2011. [Google Scholar]
  26. Hall, T.A. BioEdit: A user-friendly biological sequence alignment editor and analysis program for Windows 95/98/NT. Nucleic Acids Symp. Ser. 1999, 41, 95–98. [Google Scholar]
  27. XLSTAT. Data Analysis and Statistical Solution for Microsoft Excel; Addinsof Inc.: New York, NY, USA, 2017. [Google Scholar]
  28. O’Sullivan, D.M.; McHugh, T.D.; Gillespie, S.H. Analysis of rpoB and pncA mutations in the published literature: An insight into the role of oxidative stress in Mycobacterium tuberculosis evolution? J. Antimicrob. Chemother. 2005, 55, 674–679. [Google Scholar] [CrossRef] [PubMed]
  29. Almeida da Silva, P.E.; Palomino, J.C. Molecular basis and mechanisms of drug resistance in Mycobacterium tuberculosis: Classical and new drugs. J. Antimicrob. Chemother. 2011, 66, 1417–1430. [Google Scholar] [CrossRef] [PubMed]
  30. Louw, G.E.; Warren, R.M.; Gey van Pittius, N.C.; McEvoy, C.R.; Van Helden, P.D.; Victor, T.C. A balancing act: Efflux/influx in mycobacterial drug resistance. Antimicrob. Agents Chemother. 2009, 53, 3181–3189. [Google Scholar] [CrossRef]
  31. Draker, K.A.; Boehr, D.D.; Elowe, N.H.; Noga, T.J.; Wright, G.D. Functional annotation of putative aminoglycoside antibiotic modifying proteins in Mycobacterium tuberculosis H37Rv. J. Antibiot. 2003, 56, 135–142. [Google Scholar] [CrossRef]
  32. Jenkins, C.; Claxton, A.P.; Shorten, R.J.; McHugh, T.D.; Gillespie, S.H. Rifampicin resistance in tuberculosis outbreak, London, England. Emerg. Infect. Dis. 2005, 11, 931. [Google Scholar] [CrossRef]
  33. Prim, R.I.; Schörner, M.A.; Senna, S.G.; Nogueira, C.L.; Figueiredo, A.C.; Oliveira, J.G.; Rovaris, D.B.; Bazzo, M.L. Molecular profiling of drug resistant isolates of Mycobacterium tuberculosis in the state of Santa Catarina, southern Brazil. Mem. Inst. Oswaldo Cruz 2015, 110, 618–623. [Google Scholar] [CrossRef]
  34. Sinha, P.; Srivastava, G.N.; Tripathi, R.; Mishra, M.N.; Anupurba, S. Detection of mutations in the rpoB gene of rifampicin-resistant Mycobacterium tuberculosis strains inhibiting wild type probe hybridization in the MTBDR plus assay by DNA sequencing directly from clinical specimens. BMC Microbiol. 2020, 20, 284. [Google Scholar] [CrossRef]
  35. Valim, A.R.; Rossetti, M.L.; Ribeiro, M.O.; Zaha, A. Mutations in the rpoB gene of multidrug-resistant Mycobacterium tuberculosis isolates from Brazil. J. Clin. Microbiol. 2000, 38, 3119–3122. [Google Scholar] [CrossRef] [Green Version]
Diagnostics 12 02307 g001 550
Figure 1. The algorithm for result interpretation of the RIF-RDp assay. H526D and D516V mutations can be detected in the HEX channel and Cy5 channel, respectively, and identified as RR-TB. Other mutations were differentiated based on the ∆Tm cutoff value. If the ∆Tm cutoff value is ≥0.245 °C, it was interpreted as rifampicin-resistant tuberculosis (RR-TB) and if it is <0.245 °C, it was identified as rifampicin-susceptible tuberculosis (RS-TB).
Figure 1. The algorithm for result interpretation of the RIF-RDp assay. H526D and D516V mutations can be detected in the HEX channel and Cy5 channel, respectively, and identified as RR-TB. Other mutations were differentiated based on the ∆Tm cutoff value. If the ∆Tm cutoff value is ≥0.245 °C, it was interpreted as rifampicin-resistant tuberculosis (RR-TB) and if it is <0.245 °C, it was identified as rifampicin-susceptible tuberculosis (RS-TB).
Diagnostics 12 02307 g001
Diagnostics 12 02307 g002 550
Figure 2. Validation of the RIF-RDp assay for detection of H526D and D516V mutant Mtb strains using 14 rifampicin-resistant M. tuberculosis (RR-Mtb) strains (7, 3, 3, and 1 strain of Class-I, Class-II, Class-III (H526D), and Class-IV SNP (D516V)), 20 rifampicin-susceptible M. tuberculosis (RS-Mtb) strains, H526D and D516V mutant controls, and M. tuberculosis H37Rv reference strain. The fluorescence signal detected in the HEX channel (a,b), or Cy5 channel (c,d) illustrates the detection of H526D or D516V mutant Mtb strains, respectively.
Figure 2. Validation of the RIF-RDp assay for detection of H526D and D516V mutant Mtb strains using 14 rifampicin-resistant M. tuberculosis (RR-Mtb) strains (7, 3, 3, and 1 strain of Class-I, Class-II, Class-III (H526D), and Class-IV SNP (D516V)), 20 rifampicin-susceptible M. tuberculosis (RS-Mtb) strains, H526D and D516V mutant controls, and M. tuberculosis H37Rv reference strain. The fluorescence signal detected in the HEX channel (a,b), or Cy5 channel (c,d) illustrates the detection of H526D or D516V mutant Mtb strains, respectively.
Diagnostics 12 02307 g002
Diagnostics 12 02307 g003 550
Figure 3. LOD determination of the RIF-RDp assay for detection of (a) rpoB gene, (b) H526D mutation, and (c) D516V mutation. The duplicate DNA samples of M. tuberculosis H37Rv (H37Rv), and the H526D and D516V mutant controls using 2 × 106 to 2 × 101 genome copies (gc) of Mtb H37Rv and negative control (sterile distilled water).
Figure 3. LOD determination of the RIF-RDp assay for detection of (a) rpoB gene, (b) H526D mutation, and (c) D516V mutation. The duplicate DNA samples of M. tuberculosis H37Rv (H37Rv), and the H526D and D516V mutant controls using 2 × 106 to 2 × 101 genome copies (gc) of Mtb H37Rv and negative control (sterile distilled water).
Diagnostics 12 02307 g003
Diagnostics 12 02307 g004 550
Figure 4. ROC curve for diagnosis of RR-TB based on HRM analysis. The ∆Tm cutoff value at 0.245°C was established to differentiate between rifampicin-resistant M. tuberculosis (RR-Mtb) and rifampicin-susceptible M. tuberculosis (RS-Mtb).
Figure 4. ROC curve for diagnosis of RR-TB based on HRM analysis. The ∆Tm cutoff value at 0.245°C was established to differentiate between rifampicin-resistant M. tuberculosis (RR-Mtb) and rifampicin-susceptible M. tuberculosis (RS-Mtb).
Diagnostics 12 02307 g004
Diagnostics 12 02307 g005 550
Figure 5. The performance of the RIF-RDp assay in the detection of rifampicin-resistant M. tuberculosis (RR-Mtb). The normalized melting peaks (a), and normalized melting curves (b) clearly distinguished RR-Mtb from rifampicin-susceptible M. tuberculosis (RS-Mtb) isolates based on HRM analysis. RR-Mtb strains harboring H526D or D516V mutation were detected in the HEX (c) or Cy5 channels (d), respectively.
Figure 5. The performance of the RIF-RDp assay in the detection of rifampicin-resistant M. tuberculosis (RR-Mtb). The normalized melting peaks (a), and normalized melting curves (b) clearly distinguished RR-Mtb from rifampicin-susceptible M. tuberculosis (RS-Mtb) isolates based on HRM analysis. RR-Mtb strains harboring H526D or D516V mutation were detected in the HEX (c) or Cy5 channels (d), respectively.
Diagnostics 12 02307 g005
Table
Table 1. Mycobacterium tuberculosis isolates used in the validation process of the RIF-RDp assay.
Table 1. Mycobacterium tuberculosis isolates used in the validation process of the RIF-RDp assay.
Sample CodeMutation ProfileSNP TypeSource
Rifampicin-resistant M. tuberculosis (n = 14)
CM1S531L (TCG→TTG)Class-I (n = 7)ODPC 1 *
CM2S531L (TCG→TTG)ODPC 1
CM3S531L (TCG→TTG)ODPC 1
CM4S531L (TCG→TTG)ODPC 1
CM5H526Y (CAC→TAC)ODPC 1
CR1H526C (CAC→TGC)CPH **
CM6S522L (TCG→TTG)ODPC 1
CM7L511P (CTG→CGG)Class-II (n = 3)ODPC 1
CR2Q513P (CAA→CCA)CPH
CR3H526P (CAC→CCC)CPH
CM8H526D (CAC→GAC)Class-III (n = 3)ODPC 1
CR4H526D (CAC→GAC)CPH
CR5H526D (CAC→GAC)CPH
CR6D516V (GAC→GTC)Class-IV (n = 1)CPH
Rifampicin-susceptible M. tuberculosis(n = 20)
CM9-CM28No mutationODPC 1
* ODPC 1: Office of Disease Prevention and Control ** CPH: Chiangrai Prachanukroa Hospital.
Table
Table 2. Comparison of the performance between the RIF-RDp assay and Anyplex™ II MTB/MDR in the detection of rifampicin-resistant tuberculosis against drug susceptibility testing (DST) and DNA sequencing method.
Table 2. Comparison of the performance between the RIF-RDp assay and Anyplex™ II MTB/MDR in the detection of rifampicin-resistant tuberculosis against drug susceptibility testing (DST) and DNA sequencing method.
AssaysRIF-RDpPPV, NPV * (95% CI)AnyplexTM II MTB/MDRPPV, NPV * (95% CI)
RSRS
DSTR52398.11%
(88.16–99.73)		523100%
S15494.74%
(85.69–98.19)		05594.83%
(85.92–98.22)
Sensitivity, Specificity
(95% CI)		94.55%
(84.88–98.86)		98.18%
(90.28–99.95)		 94.55%
(84.88–98.86)		100%
(93.51–100)
DNA
sequencing		R532100%523100%
S05596.49%
(87.58–99.08)		05594.83%
(85.92–98.22)
Sensitivity, Specificity
(95% CI)		96.36%
(87.47–99.56)		100%
(93.51–100)		 94.55%
(84.88–98.86)		100%
(93.51–100)
* PPV: positive predictive value, NPV: negative predictive value, 95% CI: confidence interval.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Thant, Y.M.; Saikaew, S.; Tharinjaroen, C.S.; Phunpae, P.; Pongsararuk, R.; Preechasuth, K.; Butr-Indr, B.; Intorasoot, S.; Tragoolpua, K.; Chaiprasert, A.; et al. Combined Locked Nucleic Acid Probes and High-Resolution Melting Curve Analysis for Detection of Rifampicin-Resistant Tuberculosis in Northern Thailand. Diagnostics 2022, 12, 2307. https://doi.org/10.3390/diagnostics12102307

AMA Style

Thant YM, Saikaew S, Tharinjaroen CS, Phunpae P, Pongsararuk R, Preechasuth K, Butr-Indr B, Intorasoot S, Tragoolpua K, Chaiprasert A, et al. Combined Locked Nucleic Acid Probes and High-Resolution Melting Curve Analysis for Detection of Rifampicin-Resistant Tuberculosis in Northern Thailand. Diagnostics. 2022; 12(10):2307. https://doi.org/10.3390/diagnostics12102307

Chicago/Turabian Style

Thant, Yee Mon, Sukanya Saikaew, Chayada Sitthidet Tharinjaroen, Ponrut Phunpae, Rodjana Pongsararuk, Kanya Preechasuth, Bordin Butr-Indr, Sorasak Intorasoot, Khajornsak Tragoolpua, Angkana Chaiprasert, and et al. 2022. "Combined Locked Nucleic Acid Probes and High-Resolution Melting Curve Analysis for Detection of Rifampicin-Resistant Tuberculosis in Northern Thailand" Diagnostics 12, no. 10: 2307. https://doi.org/10.3390/diagnostics12102307

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop