Comparative Analysis of Ion Mobility Spectrometry-Based Explosive Trace Detectors

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This manuscript presents a comparative analysis of two commercially available ion mobility spectrometry (IMS)-based explosive trace detectors (ETDs) focusing on measurement uncertainty and operational stability. The authors conducted systematic testing using a 5 ng solution of TNT (in acetone) across different consecutive operation intervals and analysed the results using statistical methods and data visualization techniques.

The paper is suitable for publication with major revisions, as it makes a solid contribution to the field despite some limitations such as number of devices (and technical details about the differences between them) and explosive compound used in this study. The methodology could be valuable for future comparative studies, and the findings about device-specific performance variations are important for practitioners in aviation security.

 

Comments

• Product identification:

Major question. Is it not possible for the authors to give the details of the two products? To enable other to repeat this study such information is critical. They mention a previous study which utilised the Smiths ionscan 6000, so why can they not provide the make/model info here?

If not, can the authors at least divulge the product information to the reviewers and editor so that we can make a judgment call as to whether this is actually a fair comparison.

• Ionization

Moreover, even before I finished reading the manuscript, when I read Table 1, I expected that product B might have some issues – as corona discharge ionisation, whilst very effective, self-perpetuates variation because the corona itself can damage the sharp needle tip that produces it (as well as being sensitive to environmental conditions). Nevertheless, corona discharge is often used to avoid radioactive sources (eg., see: https://doi.org/10.1007/s13361-018-1970-6) even directly from surfaces (https://doi.org/10.1021/acs.analchem.0c01357).

The differences between the ionization methods should be discussed in more detail. Moreover, is it not possible for the authors to safely gain access to product A’s ionisation region to try to decipher with more confidence the mode of operation?

• Further product details

Can the authors give us an indication as to the form factor and whether or not the systems are handheld or benchtop, and their weight? Further, can they give us an indication as to whether they are in the same price range?

For instance, if product A is benchtop (and B is handheld), and A usually retails at  > $50,000 and B is priced around $20,000 then we might have better rationale for the differences in performance.

• Calibration

Can the authors clarify that the calibration procedure and parameters used for both devices, as this information is important for understanding measurement stability.

• Machine data output

Usually these products give quite a restricted output which has been highly processed and often convoluted. Can the authors clarify how the data was extracted and normalized to ensure fair comparison? If they could include an example describing and possibly illustrating clearly how the data was extracted and normalised it would be helpful.

• Sample interfacing

In my experience, the most variability usually happens at the sample introduction, and progressively reduces towards the detector. So can the authors please give more detail as to how the user interface the swab to these machines. Is the swab placed in a thermal desorption chamber? How long is the swab in the device? Is this all manual or is there semi-automation? etc

Other minor points:

Figure 2: I don’t understand it, there isn’t enough information. How big is it? What is the circle in the middle? What is the red box? What is it made out of? Etc.

Statistics: can the authors provide supporting information which gives more basic details about the statistical tests used, and an example of how the p values were calculated. This would be most useful as many readers will not be overly familiar with, ofr instance, the Shapiro-Wilk and Anderson-Darling tests.

Figure 4 discussion: At first glance figure 4 seems to show that product B looks to be better besides the ‘blip’ at index 240. What could be the reason for this blip? Otherwise, many of the data points from the normalized plot are grouped tightly.

Cleaning procedures: Clearly describe the cleaning procedures performed between measurements, as this affects measurement consistency and device comparison validity.

Figure 5 analysis: The plots in Figure 5 show interesting patterns but lack sufficient discussion. Additionally, briefly discuss the relationship between measurement uncertainty and alarm threshold performance.

Author Response

Comments 1:
Product identification: Major question. Is it not possible for the authors to give the details of the two products? To enable other to repeat this study such information is critical. They mention a previous study which utilised the Smiths ionscan 6000, so why can they not provide the make/model info here?
If not, can the authors at least divulge the product information to the reviewers and editor so that we can make a judgment call as to whether this is actually a fair comparison.
Response 1: There is a concern that readers may form a biased impression based on a single study. However, this can be disclosed to reviewers and editors.
Product A: Smith Detection IONSCAN 600 Product B: NEWONE S&T IONAB

Comments 2: Ionization
Moreover, even before I finished reading the manuscript, when I read Table 1, I expected that product B might have some issues – as corona discharge ionisation, whilst very effective, self-perpetuates variation because the corona itself can damage the sharp needle tip that produces it (as well as being sensitive to environmental conditions). Nevertheless, corona discharge is often used to avoid radioactive sources (eg., see: https://doi.org/10.1007/s13361-018-1970-6) even directly from surfaces (https://doi.org/10.1021/acs.analchem.0c01357).
The differences between the ionization methods should be discussed in more detail. Moreover, is it not possible for the authors to safely gain access to product A’s ionisation region to try to decipher with more confidence the mode of operation?
Response 2: This study focuses on analyzing the variation in measurement values under different operational conditions from a statistical perspective. Since it is not possible to determine how each manufacturer has customized the ionization technology, a detailed physical interpretation was intentionally avoided to prevent potential misinterpretation. However, in response to the reviewer’s suggestion, the basic principles and general advantages and disadvantages of the ionization methods used by each manufacturer have been added below Table 1, with appropriate references.

 

Comments 3:Further product details
Can the authors give us an indication as to the form factor and whether or not the systems are handheld or benchtop, and their weight? Further, can they give us an indication as to whether they are in the same price range?
For instance, if product A is benchtop (and B is handheld), and A usually retails at > $50,000 and B is priced around $20,000 then we might have better rationale for the differences in performance.
Response 3: Both products are similarly priced. Product A is a benchtop device with some portability, whereas Product B is handheld.

 

Comments 4: Calibration
Can the authors clarify that the calibration procedure and parameters used for both devices, as this information is important for understanding measurement stability.
Response 4: In response to the reviewer’s comment, we added the following sentence below Figure 2: “For intervals exceeding 8 hours between experiments, the instrument was rebooted and calibrated using the calibration pen provided with the device—applied to a swab—prior to resuming testing.” And no parameters were configured separately.

 

Comments 5: Machine data output
Usually these products give quite a restricted output which has been highly processed and often convoluted. Can the authors clarify how the data was extracted and normalized to ensure fair comparison? If they could include an example describing and possibly illustrating clearly how the data was extracted and normalised it would be helpful.
Response 5: The measurement values were transcribed exactly as they appeared on each instrument display; therefore, a separate description of the data-extraction procedure is not provided in the manuscript. Because the two instruments operate on different measurement scales, all recorded values were normalized with the min–max algorithm. A brief explanation of this normalization has been inserted into the paragraph immediately preceding Figure 3.

Comments 6: Sample interfacing
In my experience, the most variability usually happens at the sample introduction, and progressively reduces towards the detector. So can the authors please give more detail as to how the user interface the swab to these machines. Is the swab placed in a thermal desorption chamber? How long is the swab in the device? Is this all manual or is there semi-automation? etc
Response 6: Both instruments are equipped with a thermal desorption chamber into which a swab coated with the test solution is inserted. The swab was manually placed and remained inside the chamber only until detection was confirmed. This procedure has been described above Figure 2.

 

Comments 7:
Figure 2: I don’t understand it, there isn’t enough information. How big is it? What is the circle in the middle? What is the red box? What is it made out of? Etc.
Response 7: Figure 2 presents a schematic illustration of a swab with TNT applied. A photograph was not used to avoid the possibility of identifying the specific instrument. The central circle represents a typical hole found in commercial swabs. Additional explanation has been provided in the paragraph above Figure 2.

Comments 8: Statistics: can the authors provide supporting information which gives more basic details about the statistical tests used, and an example of how the p values were calculated. This would be most useful as many readers will not be overly familiar with, for instance, the Shapiro-Wilk and Anderson-Darling tests.
Response 8: The text between Tables 4 and 5 was revised to reflect the reviewer’s comments.

Comments 9: Figure 4 discussion: At first glance figure 4 seems to show that product B looks to be better besides the ‘blip’ at index 240. What could be the reason for this blip? Otherwise, many of the data points from the normalized plot are grouped tightly.
Response 9: The issue raised cannot be determined within the scope of this study and would require a different methodological approach. While Figure 4 clearly shows that Product B exhibits lower dispersion, it also records lower measurement values despite the same TNT amount. This suggests that Product A may respond more sensitively to small quantities of explosives. Therefore, determining which product is superior depends on the analytical perspective, making a definitive judgment difficult.

Comments 10: Cleaning procedures: Clearly describe the cleaning procedures performed between measurements, as this affects measurement consistency and device comparison validity.
Response 10: In response to the reviewer’s comments, we added the cleaning procedure
and, additionally, included details on the observation procedure.

 

Comments 11: Figure 5 analysis: The plots in Figure 5 show interesting patterns but lack sufficient discussion. Additionally, briefly discuss the relationship between measurement uncertainty and alarm threshold performance.
Response 11: Beyond the described findings, no additional discernible patterns were observed, making further discussion difficult. Furthermore, the present experimental design does not permit an interpretation of the relationship between measurement uncertainty and alarm threshold performance.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript entitled "Comparative Analysis of Ion Mobility Spectrometry-based Explosive Trace Detectors" by Hyun Su Sim, Jaeseong Lee, Chanhwi Kim, and Wonjoo Lee addresses a highly relevant, practical, and timely topic.

 

The study holds significant value for the field, and I recommend the manuscript for publication following minor revisions as outlined below.

 

I kindly ask the authors to provide a more detailed explanation of the experimental procedure involving instruments A and B. Specifically, it would be helpful to clarify whether all measurements were performed using a single surface or multiple surfaces containing the sample, and whether a fresh sample surface was used for each set of replicates.

 

I would ask the authors to point out if a similar study has already been done in the field of analytics with a similar approach to this paper. What makes their approach unique?

 

It would also be useful for readers if the authors would indicate (if possible) what specific improvements could be made to the framework of this statistical approach.

Author Response

Comments1: I kindly ask the authors to provide a more detailed explanation of the experimental procedure involving instruments A and B. Specifically, it would be helpful to clarify whether all measurements were performed using a single surface or multiple surfaces containing the sample, and whether a fresh sample surface was used for each set of replicates.
Response 1: Additional information addressing the reviewer’s comment has been incorporated into the paragraph above Figure 2.

Comments 2: I would ask the authors to point out if a similar study has already been done in the field of analytics with a similar approach to this paper. What makes their approach unique?
Response 2: The content has been incorporated into the Introduction section, specifically in the paragraph beginning with “Despite these efforts,~ certification processes.”

Comments 3: It would also be useful for readers if the authors would indicate (if possible) what specific improvements could be made to the framework of this statistical approach.
Response 3: Although various statistical techniques are available, implementing new methods would require a revised experimental design. However, suggesting an alternative design may broaden the scope beyond the intended focus of this study. Therefore, such discussion is deemed beyond the scope of this work and is not addressed in the current manuscript.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

This manuscript presents a precise and well-executed comparative analysis of the measurement uncertainty and performance stability of two IMS-based explosives trace detectors.
The paper makes an important contribution to the knowledge of IMS detectors. The manuscript is written clearly and presented in a well-structured manner. The conclusions are consistent with the evidence and arguments presented. The study is well-designed and technically sound. The sections are well-developed.
I recommend that the manuscript be accepted in present form.  
I have only questions for the authors. Were the detectors new?

Author Response

Comments 1: I have only questions for the authors. Were the detectors new?
Response 1: Although the instruments were not newly manufactured, both were purchased around the same time and have been consistently maintained. This information has been added to the paragraph above Figure 2.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

In this paper, the authors evaluated two commercially available IMS-based ETDs using statistical analysis and data visualization. Repeated TNT (2,4,6-trinitrotoluene) detection tests were conducted to assess performance over consecutive operations. The results revealed significant differences in measurement uncertainty between the two devices.

 

1. In "Introduction", the authors described more about TNT detection methods, but few was related to the full paper. It is suggested to simplify this section.
2. To evaluate the different devices, the authors should detect more samples to demonstrate the results.
3. The authors should detect the samples in complicated systems to evaluate the performance of different devices.
4. The authors should give the full name of the abbreviation in Table 1.
5. More related literatures on MS (Small Methods, 2024, 8, 2301192; VIEW. 2024, 5, 20240015) should be included to enhance the scope of this work.
6. In this paper, the authors compared different devices, what is the meaning of the finding? Whether the results could indicate the future application?

Author Response

Comments 1: In "Introduction", the authors described more about TNT detection methods, but few was related to the full paper. It is suggested to simplify this section.
Response 1: The Introduction has been revised in accordance with the reviewer’s suggestion.

Comments 2: To evaluate the different devices, the authors should detect more samples to demonstrate the results.
Response 2: In real operational environments, 20 consecutive detections constitute a substantial workload; therefore, it is deemed appropriate to design experiments using alternative methods in future studies.

 

Comments 3: The authors should detect the samples in complicated systems to evaluate the performance of different devices.
Response 3: In complex systems, sample detection can introduce interpretational challenges, thereby complicating performance evaluation.

Comments 4: The authors should give the full name of the abbreviation in Table 1.
Response 4: Table 1 has been updated in accordance with the author’s suggestion.

Comments 5: More related literatures on MS (Small Methods, 2024, 8, 2301192; VIEW. 2024, 5, 20240015) should be included to enhance the scope of this work.

Response 5: We appreciate this valuable suggestion. We agree that including recent literature on MS would enhance the manuscript. Accordingly, we have revised the Introduction section to briefly mention the broader applications of MS technology, and have cited the recommended literature to support this statement (Ref. [7]).
While a direct experimental comparison with MS-based systems is beyond the scope of this current study, we acknowledge this is an important point. We plan to consider expanding our research scope to include such a comparison in our future work.

Comments 6: In this paper, the authors compared different devices, what is the meaning of the finding? Whether the results could indicate the future application?
Response 6: In response to the reviewer’s comments, we have expanded the Conclusions to clarify the significance of our findings and to state that the results can guide improvements to ETD performance-certification test methods.

Author Response File: Author Response.pdf

Reviewer 5 Report

Comments and Suggestions for Authors

Presented paper is related to the comparative analysis of two commercial IMS detectors, tested in more than 240 repeated experiments detecting presence of 5 ng sample of TNT. Topic is not interesting for majority of researchers involved in Mass Spectroscopy but is crucial for people responsible for choosing, buying, using Explosive Trace Detectors or Drugs Detectors. The Authors showed way for … a methodological framework for ETD comparison and provides insights to support more rigorous evaluation and certification practices in aviation security… and in my opinion this topic is not related only to the aviation security but all other mass gathering where detection of drugs is also very important.

The Authors made a comparative analysis with ….Four consecutive operation intervals (20, 40, 60, and 80 measurements)….; calculated standard uncertainity (Table 4); p value form Normality Testes (Table 5) and next made an analysis with several statistical analysis tests (Levene’s; comparison of patterns using visualization; role of Temperature and RH changes on obtained results). All of these results are clearly presented but I would like to ask the Authors to be a more clear in explanations of obtained results, what is their meaning, especially for readers not so familiar with the statistical methods and tools. Simply speaking I would like to ask the Authors to make a statements ..this result is fine, this result is too high, this is acceptable level, this must be improved.

Author Response

Comments 1: The Authors made a comparative analysis with ….Four consecutive operation intervals (20, 40, 60, and 80 measurements)….; calculated standard uncertainity (Table 4); p value form Normality Testes (Table 5) and next made an analysis with several statistical analysis tests (Levene’s; comparison of patterns using visualization; role of Temperature and RH changes on obtained results). All of these results are clearly presented but I would like to ask the Authors to be a more clear in explanations of obtained results, what is their meaning, especially for readers not so familiar with the statistical methods and tools. Simply speaking I would like to ask the Authors to make a statements ..this result is fine, this result is too high, this is acceptable level, this must be improved.
Response 1: In accordance with the reviewer’s suggestion, we have clarified the findings below Tables 7 and 8 to improve reader comprehension by explicitly stating the key results.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Overall, the manuscript is in better shape. 

There is 1 comment that was not adequately addressed ("Nevertheless, corona discharge is often used to avoid radioactive sources (eg., see: https://doi.org/10.1007/s13361-018-1970-6) even directly from surfaces (https://doi.org/10.1021/acs.analchem.0c01357)." The authors should give a more balanced discussion of Corona Discharge ionization in the introduction. 

Author Response

Comments 1:
There is 1 comment that was not adequately addressed ("Nevertheless, corona discharge is often used to avoid radioactive sources (eg., see: https://doi.org/10.1007/s13361-018-1970-6) even directly from surfaces (https://doi.org/10.1021/acs.analchem.0c01357)." The authors should give a more balanced discussion of Corona Discharge ionization in the introduction.
Response 1: 
Reflecting the reviewer’s comment, we have revised the text below Table 1 to include reference [15].