ODIN IVR-Interactive Solution for Emergency Calls Handling

Round 1

Reviewer 1 Report

The authors do an excellent work developing an “Interactive solution for emergency calls handling”. However, the article would be improved by including more comparisons to existing scientific research. The paper under review focuses too much on engineering and not enough on the scientific aspects of the research. This is likely due to the authors' backgrounds in engineering. The authors do mention some previous studies, but a more in-depth analysis would help to put their work in context. Additionally, including more data from other scientific research would help to support the authors' solution.

Some proofreading for the paper is needed to improve the paper’s quality, for example in Table 1 Hardware resources - CPU <70% is not understood by the readers; 70% on what? And what do the authors imply by “Coverage”, and at what level?

Author Response

The authors do an excellent work developing an “Interactive solution for emergency calls handling”. However, the article would be improved by including more comparisons to existing scientific research. The paper under review focuses too much on engineering and not enough on the scientific aspects of the research. This is likely due to the authors' backgrounds in engineering. The authors do mention some previous studies, but a more in-depth analysis would help to put their work in context. Additionally, including more data from other scientific research would help to support the authors' solution.

Answer: First, we would like to thank you for your review and also to assure you that we appreciate and take into consideration your comments and observations. 

Secondly, regarding the tendency of the paper on engineering, we would like to elaborate on the fact that the scientific contributions of the paper consist of the design, building, and evaluation of the decision tree for emergency situations. The key element of our paper is the impact of the use case and the fact that this solution is the first of its kind in Romania. Furthermore, we elaborate on the emergency use case and compare our solution with a similar system used in Belgium in the national emergency system.

The aspects mentioned in this paragraph are reflected in the manuscript in Section 1. Introduction, lines: 104-107. 

Thirdly, we elaborate on the in-depth analysis of similar solutions in other countries and we identified a similar system called Options Menu (https://www.sos112.be/en/menu/) which is used since 2020 in Belgium for the 112Sos emergency system. In comparison with our solution, the design of the Belgium system does not include an advanced decision tree. In contrast, their approach has only two options: fire brigade and ambulance or police. In terms of evaluation, the Belgium system was first tested as a pilot project between 1st October 2018 and 17 January 2019 in Namur and West Flanders provinces. The population of these two provinces used the new system without any problem and the FPS Home Affairs did not receive any complaints, which led to expanding the project nationwide.

The Belgium system proved to have a positive effect both on reducing the number of unnecessary calls (such as pocket calls or malicious calls) and on the number of calls that need to be transferred from 112 to 101 and vice versa.

The aspects mentioned in this paragraph are reflected in the manuscript in Section 1. Introduction, lines 44-46, 56-66, and Section 5. Conclusions, on lines: 468 - 465. 

Some proofreading for the paper is needed to improve the paper’s quality, for example in Table 1 Hardware resources - CPU <70% is not understood by the readers; 70% on what? And what do the authors imply by “Coverage”, and at what level?

Answer: We would like to thank you for your valuable observation. We added additional information regarding the requirements of the chatbot/IVR solution presented in  Table 1. We also detailed the meaning of the “Coverage” requirement.

Reviewer 2 Report

This work presents a solution that is novel in Romania, related to emergency calls assistance. In this paper, authors identify constraints in the national emergency call system (112) and propose interactive systems to overcome these difficulties. The methods are well explained and include relevant requirements, use cases and examples. Although, I present below my suggestions to improve the paper quality and make it more interesting for the audience.

1. Authors have many English and sentence errors. Therefore, authors must review the English writing carefully to correct them. Here are part of them that were identified:

Line 86: "specifications<"

Lines 112-113: "the Chatbot ... prize AN annual Turing test"

Lines 117-118: "These type ... does not have"

Figures 1 and 2 also have a few English errors that should be corrected.

Line 332: "Second, we introduce a second"

2. Authors should review the abbreviations. In abstract, QoS and PoC are not defined and require a search on the entire document for their meaning. Furthermore, DTMF is not defined in the entire document.

3. Authors state that this work is not a major breakthrough in science, but the first implementation of such system in Romania. This work is focused on emergency assistance using "new" techniques such as chatbots. Although, authors do not present a research on emergency-tailored systems besides the description of 112. I suggest authors to include a research on related works and include a comparison of them with the proposed solution.

4. Table 2 and Figure 2 should be better placed. I suggest Table 2 to appear right after the paragraph where it is referenced, and insert the Figure 2 right after the paragraph where it is referred.

5. Section 4.2.3. represents the human evaluation of the solution. Authors made a notable work in evaluating the systems' performance but, as it is commented in conclusions, lack end-user experiments and evaluation. For the scope of this work, I consider that users' tests, results and discussion is relevant to prove that the solution is ready and robust to be used in such critical scenarios as emergencies are. 

Author Response

This work presents a solution that is novel in Romania, related to emergency calls assistance. In this paper, authors identify constraints in the national emergency call system (112) and propose interactive systems to overcome these difficulties. The methods are well explained and include relevant requirements, use cases and examples. Although, I present below my suggestions to improve the paper quality and make it more interesting for the audience.

Answer: Thank you for your valuable observations and suggestions. We addressed them with great interest. Concerning the fact that the manuscript contains multiple English and sentence errors, we apologize for them and conducted additional English proofing.

1. Authors have many English and sentence errors. Therefore, authors must review the English writing carefully to correct them. Here are part of them that were identified:
 
Line 86: "specifications<"

Answer: The error was corrected. Line 96.

Lines 112-113: "the Chatbot ... prize AN annual Turing test"

Answer: A better explanation was provided. Lines 126-128.

Lines 117-118: "These type ... does not have"

Answer: The error was corrected. Line 133.

Figures 1 and 2 also have a few English errors that should be corrected.

Answer: Figures 1 and 2 were corrected. Page 4 and page 8.

Line 332: "Second, we introduce a second"

Answer: The issue was corrected. Line 354.

2. Authors should review the abbreviations. In abstract, QoS and PoC are not defined and require a search on the entire document for their meaning. Furthermore, DTMF is not defined in the entire document.

Answer: Thank you for your observation and please accept our apologies regarding the abbreviations. We have double-checked the abbreviation lists and corrected all the problems. 
The observations were implemented in the manuscript on Lines 9-10 and 31.

3. Authors state that this work is not a major breakthrough in science, but the first implementation of such system in Romania. This work is focused on emergency assistance using "new" techniques such as chatbots. Although, authors do not present a research on emergency-tailored systems besides the description of 112. I suggest authors to include a research on related works and include a comparison of them with the proposed solution.

Answer: Thank you for your insightful comment. We addressed this issue by comparing the proposed solution with the Belgium IVR system called the Options Menu in  Section 1. Introduction, lines 44-46, 56-66, and Section 5. Conclusions, on lines: 468 - 465. 
 
4. Table 2 and Figure 2 should be better placed. I suggest Table 2 to appear right after the paragraph where it is referenced, and insert the Figure 2 right after the paragraph where it is referred.

Answer: Thank you for your suggestions. We have placed Table 2 and Figure 2 as suggested. 

5. Section 4.2.3. represents the human evaluation of the solution. Authors made a notable work in evaluating the systems' performance but, as it is commented in conclusions, lack end-user experiments and evaluation. For the scope of this work, I consider that users' tests, results and discussion is relevant to prove that the solution is ready and robust to be used in such critical scenarios as emergencies are. 

Answer: Thank you for your valuable observations. In response to your observations, we present an empirical evaluation of the decision tree of the IVR solution. We conduct several laboratory experimental calls related to the four different situations that the IVR system can handle. We compared the obtained results with the expected values for the emergency type classification. The obtained results indicate good performance of the decision tree classification.   
The aspects mentioned in this paragraph are reflected in the manuscript in Section 4.2.2. Effectiveness evaluation, lines: 424-432.

Round 2

Reviewer 1 Report

The author has made a concerted effort to correct the previous remarques and has succeeded in presenting a well-argued and well-supported scientific paper.

Author Response

We would like to thank you and show our full appreciation for all the valuable observations and recommendations.

Reviewer 2 Report

The paper quality surely improved and is more robust now.

I still have some concerns related to presentation:

[1] Lines 64-66. "The British IVR". It is not clear if this IVR is the same implemented by Devon and Cornwall Police Departments. Please clarify this.

Moreover, Devon and Cornwall Police Department have implemented an interesting 64

IVR 2 system that helps callers to reach each person or department that is best placed to resolve their inquiries. The British IVR system uses a two-level decision tree.

[2] Figures still have many mistakes

Figure 5. has English mistakes. (Performnace, consummed) 

Figura 7. also does. (Current calls count - should not be concurrent?)

Figure 9.: Pachet instead of Packet

[3] Tests and conclusions were improved. Although, I have some concerns about the results presented in Figure 11.

This figure shows that 8,94% of the medical emergencies, 15,85% of the police emergencies, and 17,17% of the fire emergencies are classified as Non-urgent. What is the relevance of this values, what they mean, what is this classification's impact in the emergency response?

For example, if a fire emergency call is classified as non-urgent.

I believe that this discussion should be present in the text, and with strategies and measures to improve the solution's performance.

Author Response

[1] Lines 64-66. "The British IVR". It is unclear if this IVR is the same one implemented by Devon and Cornwall Police Departments. Please clarify this.
Moreover, Devon and Cornwall Police Department have implemented an interesting 64
IVR 2 system helps callers to reach each person or department that is best placed to resolve their inquiries. The British IVR system uses a two-level decision tree.
Thank you for the valuable observation. We modified the paragraph in order to be more understandable.
[2] Figures still have many mistakes
Figure 5. has English mistakes. (Performnace, consummed) 
Figure 7. also does. (Current calls count - should not be concurrent?)
Figure 9.: Pachet instead of Packet
 We apologize for the mistakes. We corrected them as kindly suggested.
[3] Tests and conclusions were improved. Although, I have some concerns about the results presented in Figure 11.
This figure shows that 8,94% of medical emergencies, 15,85% of police emergencies, and 17,17% of fire emergencies are classified as Non-urgent. What is the relevance of these values, what do they mean, and what is this classification's impact on the emergency response?
For example, if a fire emergency call is classified as non-urgent.
I believe that this discussion should be present in the text, with strategies and measures to improve the solution's performance.

Thank you for your insightful observation. We have provided a more comprehensive explanation of the decision tree evaluation. In the first place, we mention the fact that our solution determines booth non-urgent situations in case of non-urgent selection or depending on the answers provided by the user in the emergency flows. Therefore, it is possible that our solution to classify an emergency as non-urgent even though it came from an urgent branch of the decision tree (eg. Other types of medical emergency that do not put life in imminent danger). Secondly, our evaluation shows that the decision tree is the most suitable because it classifies emergencies according to the user's input (when a user selects a medical issue the IVR will never indicate a fire or police emergency.).
We have better explained our approach in order to understand better the functionality of the decision tree approach. Lines 442-448.

We would like to thank you and show our full appreciation for all the valuable observations and recommendations.