Managing the Performance of Asset Acquisition and Operation with Decision Support Tools

Round 1

Reviewer 1 Report

Editorial

The ISO 5500x standard are cited either as “IS0 5500x:2014” or “ISO 5500x”, please use the same notation all over the paper

Please check for typos. “However, an asset is anything which which has potential”; “the scope of AM systems increase to to include natural assets”; NGET singular or plural? line 121 “NGET plays”, line 130 “NGET make”, …

Comments to the authors

A much deeper state of the art on DST is needed. References on this topic are way too few. Even for the best practices a single and old (2015) source is cited 

It may help the readers to have the description of the NGET (paragraph 3) in the case study part, where it belongs, and not before the description of the research approach. 

The “research approach” section doesn’t clarify anything about the research method. It seems a summary of the research. Moreover, it describes also parts that have been already presented as the literature review. A proper “research method” paragraph may help the reader to better understand the work done. Now the full research method is summarized in a single sentence: “stage 3: Within this stage understanding generated from the previous two stages was applied to inform the creation of the conceptual approach”. How the “understanding” was “applied to inform the creation”? 

Figure 4 is redundant, it is only an enlargement of part of Figure 3 and does not provide any new information.

From line 214: “Figure 4 presents the DST Performance Management Process in detail. This shows that the elements common to both processes are:…”. It is not clear which are the two processes. If they are Risk management and DST performance management then the figure that shows what they have in common is Figure 3.

From line 263: A better description of the focus group is necessary. How “job functions” and “skill sets” are distributed across the five groups? Why “The participants were nominated by their NGET managers” instead of being selected in a process guided by the researchers? What are the expertises of the NGET managers in forming focus groups for research purpose?

Conclusions are poor: “A key finding of the research was that having ‘fit for purpose’ DSTs will require the involvement of a range of departments.”, “The introduction of a DST Performance Management Approach works towards organisational unification”. The article may be improved adding some discussions to the conclusions.

“Future works” is generally intended as the place to recommend areas to be researched in the future. Here it is used to tell us that NGET “have declared an intention to implement the process within their organisation” and that “This implementation will form the next stage of the research”. This section should be developed.

Author Response

We want to thank the reviewers for their careful reading, insightful comments and suggestions.  In revising the manuscript, we have addressed all the comments and suggestions.  In doing so we rewrote and reorganized parts of the paper and believe that the paper has improved significantly as a result. A separate response is given to each reviewer comment, to include the original comment (black) and our response (red).  For ease, line numbers where changes have been made are provided.

R1.1.  The ISO 5500x standard are cited either as “IS0 5500x:2014” or “ISO 5500x”, please use the same notation all over the paper.

Response:  All references to the Standard have been amended to the standard notation “ISO 5500x:2014”.

R1.2.  Please check for typos. “However, an asset is anything which which has potential”; “the scope of AM systems increase to to include natural assets”; NGET singular or plural? line 121 “NGET plays”, line 130 “NGET make”, …

Response:  The three typos identified have been specifically addressed.  Following the revisions, the paper has also been independently proof checked.

R1.3.  A much deeper state of the art on DST is needed. References on this topic are way too few. Even for the best practices a single and old (2015) source is cited.

Response:  The reviewer raises a good point.  The paper would be enhanced by a deeper state of the art.  Accordingly, the Background (Section 2) literature has been revisited to include a wider number of papers (Lines 89-90). 

Specific to the 2015 source which refers to a Subject Specific Guidelines (SSG).  SSGs are a core document within the Institute of Asset Management’s body of knowledge.  This 2015 source contains the IAMs most recent consolidation of DST “good practice”.  On reflection the importance of this as a source document was not made clear. The paper has been revised accordingly (lines 95-98).

R1.4.  It may help the readers to have the description of the NGET (paragraph 3) in the case study part, where it belongs, and not before the description of the research approach. 

Response:  This item is addressed within the response to comment R1.5.

R1.5.  The “research approach” section doesn’t clarify anything about the research method. It seems a summary of the research. Moreover, it describes also parts that have been already presented as the literature review. A proper “research method” paragraph may help the reader to better understand the work done. Now the full research method is summarized in a single sentence: “stage 3: Within this stage understanding generated from the previous two stages was applied to inform the creation of the conceptual approach”. How the “understanding” was “applied to inform the creation”? 

Response:  In responding to items R1.4 & R1.5 the Research Approach (Section 3) has been revised and restructured.  Within the original paper this section was limited to an explanation of the high-level methodology used to structure the project.  On reflection it failed to give the specifics of the research method.  As suggested, the detail of NGET has been expanded and presented as a case study within the Research Approach (Section 3.1). This is presented together with a detailed description of the methods used in defining the DST performance management approach requirements (Section 3.2).

R1.6.  Figure 4 is redundant, it is only an enlargement of part of Figure 3 and does not provide any new information.

Response:  See response to item R1.7.

R1.7.  From line 214: “Figure 4 presents the DST Performance Management Process in detail. This shows that the elements common to both processes are:…”. It is not clear which are the two processes. If they are Risk management and DST performance management then the figure that shows what they have in common is Figure 3.

Response (R1.6 & R1.7 & R3.10):  The overlap between Figures 3 & 4 was raised by both Reviewer 1 (R1.6) and Reviewer 3 (R3.10).  The Editor suggests a possible way forward with the replacement of Figure 3 with a table (see Table 4 line 263).  This was an excellent idea as it facilitated comparison of the two processes to include the new ‘recording and reporting’ element (lines 298-304).

R1.8.  From line 263: A better description of the focus group is necessary. How “job functions” and “skill sets” are distributed across the five groups? Why “The participants were nominated by their NGET managers” instead of being selected in a process guided by the researchers? What are the expertises of the NGET managers in forming focus groups for research purpose?

Response:  The reviewer makes a good suggestion.  On reflection, expansion of this section would significantly improve the paper.  A table has been added to provide details of the participants: department, job title, and responsibilities in regards to DSTs (Table 5 line 339) and the text has been revised to reflect how the selection of the participants was guided by the theory and undertaken in collaboration with the NGET senior management (lines 330-337)

R1.9.  Conclusions are poor: “A key finding of the research was that having ‘fit for purpose’ DSTs will require the involvement of a range of departments.”, “The introduction of a DST Performance Management Approach works towards organisational unification”. The article may be improved adding some discussions to the conclusions.

Response:  The process of responding to reviewer’s comments caused the paper to expand.  This created a clearer ‘pull through’ of findings.  In reflecting this the Conclusions (Section 6) have been rewritten.

R1.10.  “Future works” is generally intended as the place to recommend areas to be researched in the future. Here it is used to tell us that NGET “have declared an intention to implement the process within their organisation” and that “This implementation will form the next stage of the research”. This section should be developed.

Response: We agreed with the reviewer.  This section would benefit from being developed.  In response to the reviewer comments, Future Works (Section 7) has been enhanced identifying three clear areas for focus: Field tests, transferability, and methods and studies to assess life cycle cost versus benefit of DST performance management.

Reviewer 2 Report

This paper aims to create a conceptual approach for managing the performance of DST within Asset Management organisation. Overall the paper is clear and relevant. The background and literature review is clear and research gap clearly articulated. The research method is clear. However, the description of the Performance Management Process requires further clarifications. Particularly, on the aspect of the adoption of Risk Management Process and its alignment to the DST Performance Management Process. Why use the Risk Management Process? Is it because this has been used before somewhere else and proven to be useful and successful? In any case, more justification is necessary to strengthen the adoption and use in current paper. 

Author Response

We want to thank the reviewers for their careful reading, insightful comments and suggestions.  In revising the manuscript, we have addressed all the comments and suggestions.  In doing so we rewrote and reorganized parts of the paper and believe that the paper has improved significantly as a result. A separate response is given to each reviewer comment, to include the original comment (black) and our response (red).  For ease, line numbers where changes have been made are provided.

R2.1.  The description of the Performance Management Process requires further clarifications. Particularly, on the aspect of the adoption of Risk Management Process and its alignment to the DST Performance Management Process. Why use the Risk Management Process? Is it because this has been used before somewhere else and proven to be useful and successful? In any case, more justification is necessary to strengthen the adoption and use in current paper. 

Response:  Thank you for your comments.  On reflection providing further clarification for the use of Risk Management process would improve the paper.  The paper has been revised to clearly identify the four main reasons which supported the adoption of the Risk Management Process (see lines 243-252).

Reviewer 3 Report

The authors presented a very interesting topic in this paper - they proposed a conceptual approach for managing DSTs performance after implementation. The authors made great effort in this manuscript. However, I find several critical comments and questions for the authors:

1. The English writing needs to be improved very much. There are too many places that a comma "," needs to be added. For example, Page 1 Line 34. a comma should be added between the word "savings" and "the infrastructure". Another example would be Page 2 Line 68, a comma should be added between the word "history" and "the discipline". Sometimes a missing comma makes it very hard to understand the sentence. I highly recommend that the authors perform professional English proof reading before submitting.
2. The title is not precise enough. The approach that manages the DST performance contains both DST performance management process and DST performance assessment techniques. This paper only discusses DST performance management process, not the whole approach. Therefore, I think the title is larger than the scope of this paper and should be revised
3. Page 2 Line 59 to Line 66. When describing the paper structure, I recommend the authors use Section XX instead of (XX). For example, Line 63 says that "this research approach is described (4)". This is not a very good way to express Section 4.
4. Page 3 Line 83. There are two "which". please delete one. 
5. Page 3 Line 85. There are two "to". Please delete one.
6. Page 4 Line 143. The author says "Through the case study, there was...". What kind of case study it is? The authors did not explain with details, nor did they provide the citation. Please add more explanation
7. Technical comments. Page 5. In Step 4, the authors mentioned that there are 5 evaluations undertaken in total. "due to the limitations imposed within a paper, only the final evaluation of the ....". What are the limitation? I did not see such limitations and I think that it is necessary to show the results of all 5 evaluations because they evaluated different things - 3 of them during the process of conducting the research and 2 following the creation of the research. I think all of them are valuable and should be presented. It is not valid to present only final evaluation results
8. Page 5 Line 177. "After, the specifics of the DST Performance Management Process." This is an incomplete sentence. Please revise.
9. Page 7 Line 214 to Line 216. The authors said that "This shows that the elements comment to both processes are: ...". This is not obtained from Figure 4. This is observed from Figure 3 where both processes are presented.
10. Page 8. I think Figure 4 is not necessary. It is only the right part of Figure 3. Figure 4 does not show any new information. Please consider delete.
11. Page 8 Line 225. As a component of the process, "Monitoring and review" should be a new paragraph itself
12. Technical comment. Page 9 Section 6. In conducting this study, only 5 subject matter experts (SMEs) are involved. And they come from three different job functions: DST user, modelers, and information quality managers. At least one of the job functions have only one representative. I doubt that the sample size is too small. Many times, the opinions from a SME is subjective. Therefore, I do not think it is appropriate to include only 5 SMEs in this study. In other words, the size is too small and I would doubt the validation of the results.
13. Technical comment. Page 9 Section 6. The authors only described the types of questions to ask the SMEs. However, I think this is not enough. In case study session, the authors should provide the details of how they conducted the research. In other words, the authors should provide the detailed questions that the SMEs were asked so that the readers can better understand. If no questions are provided, I would be doubtful. 
14. Technical comment. The case study section is not persuasive at all to me. Based on the level of details provided by the authors, I do not feel comfortable to draw any conclusions from the case study.
15. Please consider to provide more quantitative analyses instead of qualitative descriptions in case study section.
16. Technical comment. I do not think this paper is innovative and contribute that much to the current practice. The models and process used in this paper have already been developed by other researchers or papers. The only thing the authors did is to replace risk assessment part with DST performance assessment. This makes the paper little value in my opinion.
17. Page 10. The first paragraph of Section 7 Conclusion are repetitive with the introduction part. Some sentences are exactly the same. I do not think this is appropriate.

Hope my comments can help you further improve the paper.

Author Response

We want to thank the reviewers for their careful reading, insightful comments and suggestions.  In revising the manuscript, we have addressed all the comments and suggestions.  In doing so we rewrote and reorganized parts of the paper and believe that the paper has improved significantly as a result. A separate response is given to each reviewer comment, to include the original comment (black) and our response (red).  For ease, line numbers where changes have been made are provided.

R3.1. The English writing needs to be improved very much. There are too many places that a comma "," needs to be added. For example, Page 1 Line 34. a comma should be added between the word "savings" and "the infrastructure". Another example would be Page 2 Line 68, a comma should be added between the word "history" and "the discipline". Sometimes a missing comma makes it very hard to understand the sentence. I highly recommend that the authors perform professional English proof reading before submitting.

Response:  Following the paper revisions the two examples of grammatical errors are eliminated. As suggested, the revised paper has been independently proof checked.

R3.2. The title is not precise enough. The approach that manages the DST performance contains both DST performance management process and DST performance assessment techniques. This paper only discusses DST performance management process, not the whole approach. Therefore, I think the title is larger than the scope of this paper and should be revised.

Response:  We agree.  The title has been revised to “Managing the Performance of Asset Management Decision Support Tools” which we believe better reflects the scope and focus of the paper.

R3.3.  Page 2 Line 59 to Line 66. When describing the paper structure, I recommend the authors use Section XX instead of (XX). For example, Line 63 says that "this research approach is described (4)". This is not a very good way to express Section 4.

Response:  Agreed.  The paper has been updated accordingly (lines 51-58).

R3.4. Page 3 Line 83. There are two "which". please delete one. 

Response:  Following the paper revisions this comment is no longer relevant.

R3.5.  Page 3 Line 85. There are two "to". Please delete one.

Response:  This has been corrected.

R3.6.  Page 4 Line 143. The author says "Through the case study, there was...". What kind of case study it is? The authors did not explain with details, nor did they provide the citation. Please add more explanation.

Response:  In addressing the comments of Reviewer 1 the detail of NGET has been enhanced with the case study now incorporated within the Research Approach (Section 3.1 lines). 

R3.7.  Technical comments. Page 5. In Step 4, the authors mentioned that there are 5 evaluations undertaken in total. "due to the limitations imposed within a paper, only the final evaluation of the ....". What are the limitation? I did not see such limitations and I think that it is necessary to show the results of all 5 evaluations because they evaluated different things - 3 of them during the process of conducting the research and 2 following the creation of the research. I think all of them are valuable and should be presented. It is not valid to present only final evaluation results

Response:  The “5 evaluations” were within a section which described the structure of the overall project. These covered all aspects of the research undertaken to create the approach.  Of these, two evaluations were directly relevant to the creation of the process:  Evaluation of the outputs of the RE exercise (during which the approach requirements were created), and final evaluation of the designed process.

The “limit of paper” referred to the paper focus on the process and not the whole approach.  In reflection this was unclear.

In revising the paper, the structure of the overall project has been removed.  The expanded Research Approach (Section 3) incorporates detail on how the approach requirements were defined (Section 3.2). Within this, the validation activity is explained in full lines (194-216). 

As previously, the final evaluation is detailed within Section 5.

R3.8.  Page 5 Line 177. "After, the specifics of the DST Performance Management Process." This is an incomplete sentence. Please revise.

Response:  Following the paper revisions this comment is no longer relevant.

R3.9.  Page 7 Line 214 to Line 216. The authors said that "This shows that the elements comment to both processes are: ...". This is not obtained from Figure 4. This is observed from Figure 3 where both processes are presented.

Response:  In responding to the comments of the Reviewer 1 and the suggestions of the Editor, Figure 3 is replaced by a table (Table 4).  We believe this has greatly improved the ability to compare the two processes. 

R3.10.  Page 8. I think Figure 4 is not necessary. It is only the right part of Figure 3. Figure 4 does not show any new information. Please consider delete.

Response (R1.6 & R1.7 & R3.10):  The overlap between Figures 3 & 4 was raised by both Reviewer 1 (R1.6) and Reviewer 3 (R3.10).  The Editor suggests a possible way forward with the replacement of Figure 3 with a table (see Table 4).  This was an excellent idea as it facilitated comparison of the two process to include the new ‘recording and reporting’ element (lines 298-304).

R3.11.  Page 8 Line 225. As a component of the process, "Monitoring and review" should be a new paragraph itself.

Response: The paper has been revised accordingly. 

R3.12.  Page 9 Section 6. In conducting this study, only 5 subject matter experts (SMEs) are involved. And they come from three different job functions: DST user, modelers, and information quality managers. At least one of the job functions have only one representative. I doubt that the sample size is too small. Many times, the opinions from a SME is subjective. Therefore, I do not think it is appropriate to include only 5 SMEs in this study. In other words, the size is too small and I would doubt the validation of the results.

Response:  I appreciate the reviewer raising this point as it allows me to clarify the stance taken.  The evaluation was undertaken within a focus group.  A focus group is a commonly used research technique in which participants are brought together to explore an idea.  Although there is no ideal focus group size, traditionally there would be between six to twelve participants (Bell 2010, Denscombe 2010, Gray 2014).  Recently however, the use of mini focus groups of between two and five people have emerged.  Mini focus groups have been found to be particularly useful in situations where recruitment is difficult, or where the intensity of the discussions means that having less participants would be preferential (Feltwell and Rees 2004, Taylor, Hare et al. 2016, Githaiga 2017). 

Within this research the availability of participants was subject to certain constraints this included the number of people within the organisation who can be considered experts, availability of those experts at a given time, and the motivation of the experts to take part.  

This research aims to create a conceptual approach for managing DST performance.  The philosophical position was constructionist, and interpretivism.  That is, the social world is a creation of the human mind which is constructed and then reinforced by social interactions (there are multiple social realities) and that knowledge is gained by way of human interpretation (Denscombe 2010). In adopting this stance it is accepted that the approach requirements, the approach which is created, and the evaluation of the approach reflect the reality of those involved at a certain point in time.  The process of progressing the approach from conceptual design through to industry adoption will change perceptions of what is possible, wanted, and needed. 

On reflection, the evolutionary nature of research of this type was not made explicit within the paper.  This has been addressed (lines 383-390).

R3.13.  Page 9 Section 6. The authors only described the types of questions to ask the SMEs. However, I think this is not enough. In case study session, the authors should provide the details of how they conducted the research. In other words, the authors should provide the detailed questions that the SMEs were asked so that the readers can better understand. If no questions are provided, I would be doubtful. 

Response:  In responding we assume that the reviewer is referring to the evaluation focus group rather than case study.  The focus group looked to evaluate the approach. In total 18 specific questions were explored, and a SWOT analysis conducted. Of these, eight were directly relevant to the evaluation of the Process.  The evaluation (Section 6) has been expanded to include the detail of these questions (lines 313-322).

R3.14.  The case study section is not persuasive at all to me. Based on the level of details provided by the authors, I do not feel comfortable to draw any conclusions from the case study.

Response:  In responding to the comments of Reviewer 1 the case study has been restructured and expanded. It now provides greater detail to include the inputs to the case study (Figure 1).

R3.15.  Please consider to provide more quantitative analyses instead of qualitative descriptions in case study section.

Response:  Thank you for raising this point.  As detailed in my response to item R3.14, the case study (Section 3.1) has been restructured and expanded to include the details of the inputs (Figure 1).  As seen in my response to item R3.13 the philosophical stance taken in conducting this research was constructionist, and interpretivism. In supporting this stance mixed qualitative methods are used.

R3.16. I do not think this paper is innovative and contribute that much to the current practice. The models and process used in this paper have already been developed by other researchers or papers. The only thing the authors did is to replace risk assessment part with DST performance assessment. This makes the paper little value in my opinion.

Response:  Thank you for your comment. 

In creating this approach the Design Research Methodology was followed (Blessing and Chakrabarti 2009).  A research gap was identified. Data was gathered which informed the approach.  A solution was created and then evaluated. 

As expected, the approach builds on the research which goes before it and in doing so provides a solution to a challenge which did previously exist. 

The approach will be implemented by NGET and has the potential to create real societal value. The value of this is not limited to NGET.  Interest from the practitioner community is high and we have been asked to present the approach at the IAM conference, and for a special interest group of the British Computer Society.  Additionally, we have published pieces within the member’s magazines of both the IAM and Chartered Quality Institute.

R3.17.  The first paragraph of Section 7 Conclusion are repetitive with the introduction part. Some sentences are exactly the same. I do not think this is appropriate.

Response:  In response to the comments by Reviewer 1 the conclusions have been revised. 

Reviewer 4 Report

The paper is of interest and could trigger further research work in this domain. A statement on uncertainties, which obviously will arise in this process, would be nice to see. In general, the rules mentioned could be a little more elaborated. This could be mentioned in the last paragraph on future work to be performed.

There are a few small typos which could be addressed:

Line 48: There is one zero too much in ISO 55000

Line 82: The definition of AM is included in ISO 55000 and should be quoted.

Line 83: The word “which” is used twice.

Line 85: The word “to” is used twice

Line 140: Spelling of “demonstrating”

Line 200: Separate the sentence into two

Line 301 – 302: There should be a comma after “significant” and after “sustain”

Author Response

We want to thank the reviewers for their careful reading, insightful comments and suggestions.  In revising the manuscript, we have addressed all the comments and suggestions.  In doing so we rewrote and reorganized parts of the paper and believe that the paper has improved significantly as a result. A separate response is given to each reviewer comment, to include the original comment (black) and our response (red).  For ease, line numbers where changes have been made are provided.

R4.1.  A statement on uncertainties, which obviously will arise in this process, would be nice to see. In general, the rules mentioned could be a little more elaborated. This could be mentioned in the last paragraph on future work to be performed.

Response:  That you for this comment.  On reflection, including a statement of uncertainties would significantly improve the paper.  The Future Work section has been updated accordingly (lines 383-390).

R4.2.  Line 48: There is one zero too much in ISO 55000

Response:  This has been corrected.

R4.3.  Line 82: The definition of AM is included in ISO 55000 and should be quoted.

Response:  The citation has now been included (line 85).

R4.4.  Line 83: The word “which” is used twice.

Response:  Following the paper revisions this comment is no longer relevant.

R4.5.  Line 85: The word “to” is used twice

Response:  This has been corrected.

R4.6.  Line 140: Spelling of “demonstrating”

Response:  This has been corrected.

R4.7.  Line 200: Separate the sentence into two

Response:  Following the paper revisions this comment is no longer relevant.

R4.8.  Line 301 – 302: There should be a comma after “significant” and after “sustain”

Response:  Following the paper revisions this comment is no longer relevant.

Round 2

Reviewer 3 Report

The authors did a great job in addressing my comments. I appreciate the effort and am satisfied with the response. I think the manuscript is much better than first version.

One more comment from me:

When responding my comment on the contribution of the paper, the authors replied

"Thank you for your comment.

In creating this approach the Design Research Methodology was followed (Blessing and Chakrabarti 2009). A research gap was identified. Data was gathered which informed the approach. A solution was created and then evaluated.

As expected, the approach builds on the research which goes before it and in doing so provides a solution to a challenge which did previously exist.

The approach will be implemented by NGET and has the potential to create real societal value. The value of this is not limited to NGET. Interest from the practitioner community is high and we have been asked to present the approach at the IAM conference, and for a special interest group of the British Computer Society. Additionally, we have published pieces within the member’s magazines of both the IAM and Chartered Quality Institute."

Please consider to add these motivation, research gaps, and contributions to the paper where applicable (especially abstract and conclusion). This will improve the value of the paper.

Author Response

Once again I would like to thank the reviewer for their comments.  Being so close to a subject often makes it difficult to see what is necessary to effectively communicate the work.  I appreciate their input which has undoubtedly improved the paper.

R3.1.

Please consider to add these motivation, research gaps, and contributions to the paper where applicable (especially abstract and conclusion). This will improve the value of the paper.

Response:  I agree with the reviewer's comments.  As suggested I have made revisions to both the abstract and the conclusions sections:

The abstract has been revised as follows: 

"Decision support tools (DSTs) are increasingly being used to assist with asset acquisition and management decisions.  Whether these tools are “fit for purpose” will have both economic and non-economic implications.  Despite this, the on-going governance of DST performance receives only limited attention within both the academic and industry literature. This work addresses that research gap."

An additional paragraph has been added to the conclusions:

"Indications are that this research has the potential to create impact outside of academia.  NGET have indicated their intention to implement the approach within their business [69]. Interest from the practitioner community is high with requests made to present the approach at the IAM Conference (Nov 2017), and for a special interest group of the British Computer Society (April 2018). Additionally, an overview of the approach has been published within the member’s magazines of both the IAM [70] and Chartered Quality Institute [71]."