Expert Opinion Valuation Method to Quantify Digital Water Metering Benefits

Business cases promoting the introduction of digital water metering (DWM) have, to date, focused on a limited number of benefits, especially water savings, metering costs, occupational health and safety (OHS), and deferral of capital works. An earlier study by the authors catalogued 75 possible benefits and developed a taxonomy based on a literature review, interviews and water industry reports. The objective of the present study was to elicit the opinions of Australian water industry experts on the benefits, then use the opinions to form probability distributions which, in future work, could be used to model the value of DWM benefits. The study findings have implications for researchers and practitioners seeking to accurately and stochastically model the benefits of DWM transformation programmes. Thematic analyses on the open ended responses scaled likelihood and estimated value of benefits into comparable units. We found 82% support for the benefits of DWM with only 6% disagreement and 12% non-commital; the savings value of cost of water benefits were predominately expected to range between 5% and 10% and much higher in some individual situations, while charges/operational costs benefits were predominately expected to range between 45% and 100%; and, moreover, we indicated how a risk-based range of project benefit could potentially be calculated. Opportunities for further investigations were identified.


Introduction
Many benefits of digital water metering (DWM) have been reported since one of the first reports on the "new automated meter reading" (AMR) systems emanated from the US in 1994 [1]. Use of AMR was being promoted as both "strategic" and "best practice" just three years later [2], with further deployments being reported in US cities from 1997-1999 [3,4]. Deployments have continued in the US and other countries since, and documented in case studies and reviews [5][6][7][8][9][10][11][12].
Research into specific issues has been conducted, increasing both awareness and knowledge of the potential benefits. Australian researchers have canvassed new opportunities to exploit the "big data" from DWM for water end-use analytics, leak identification, infrastructure planning, and sustainability issues [11,[13][14][15][16]. International researchers have reported the benefits to tourist and holiday precincts, vacant properties, and water conservation practices from trials and case studies of DWM implementations [5][6][7][8][9][10][11][12].

Research Method
The use of structured interviews, and qualitative and quantitative analysis, was designed to address the lack of empirical evidence covering all benefits from previous studies.

Structured Interview Development
Structured interviews were used to elicit the opinions of experts on a very high number of issues (71) across many business areas where they might or might not have expertise. Group sessions, such as workshops, offered the opportunity to explore issues and build consensus. However, logistical issues such as gaining the attendance of all the necessary participants at the same time, the duration required to cover the issues and maintain participant interest, made in-person workshops unworkable. One-on-one interviews were ruled out for similar reasons, as well as the impact on the interviewer's time and travel requirements. The use of an online survey offered many advantages including providing participants with the option of suspending and resuming the survey at a later time to enable work commitments to be met. A disadvantage of online surveys is the lack of opportunity to clarify questions and avoid misunderstandings. To counter this issue, and achieve a common understanding and a consistency of input, the questions were supplemented with some background information [61].
The question for the benefit of potentially reducing the number of special readings, D12, illustrates the typical question structure used in the survey: "D12. From your experience/knowledge will water businesses benefit from a reduction in special meter reads and to what extent? BACKGROUND: When customers leave properties the water meter needs to be read to enable the leaving customer to be billed and the arriving customer to start from zero. By taking the electronic reading of the meter at a nominated date and time can eliminate meter readers having to visit the property." Qualitative analysis was used to encode the expert assessments of likelihood and benefit value which could then be examined via quantitative analysis. Firstly a qualitative approach was used to elicit which benefits were supported as being achievable, and to what degree. Open-ended questions were chosen to allow the experts to quantify their knowledge using the units most familiar to them and ensure elicitation could be more accurate [60]. Secondly, quantitative analysis enabled probability distributions to be developed from the expert assessments. No attempt was made to establish a consensus among the expert opinions, so that the various 'raw' views have been used to establish the range and frequency of possible outcomes [62]. Thirdly, modelling of the benefit value demonstrated that cost benefits could be presented as statistical distributions [63].
In addition to the collection of opinions on the benefits, the survey instrument collected profiling data for each participant to enable any differences between expert cohorts to be identified.

Interview Design
A long survey raises issues around participant fatigue and maintaining interest [64,65]. The DWM benefits were grouped into blocks based on the business expertise relevant to the benefit. This 'blocking' filtered questions so that experts could avoid questions they would not be able to answer and, instead, concentrate their attention in areas in which they had the knowledge and/or expertise. Experts were able to select which blocks they would answer with the exception of the block of questions on customer goodwill, which appeared as the last block. While the Goodwill question-block was not mandatory, the experts were asked to answer as a customer as well as an expert, if they had expertise in the area.
For each benefit, experts were asked their views on the likelihood of the benefit being achieved and the extent of the benefit on savings or improvements. They were asked to describe any experience or evidence of the benefit that they were aware of. Each question was supported by some neutrally worded background information to assist the participant in understanding the context of the benefit [61]. The online survey included all benefits that had been identified at the time. Four additional benefits were identified during the course of the structured interviews. The number of questions in the different sections of the structured interviews comprised: • Validity checks were conducted on the interview questions using an academic in the field and piloted the survey using a water industry professional, which refined question flow and wording.
The target audience were staff within water utilities and supporting organisations who had an involvement in key business areas. These areas include digital metering, water metering, operational planning, non-revenue water, customers, complaint handling or business support functions. We also recruited independent experts who had industry experience. Invitations were sent to liaising staff at water businesses who identified subject matter experts within the business and passed the invitation to them. Invitations were sent directly by the researchers to the independent experts.

Data Cleansing and Analysis Procedure
The experts' responses were downloaded from the survey system and imported into a SQL database [66] for analysis. Limited data cleansing was applied to provide clarity in a number of ways: spelling and other obvious typographical errors, answers of "as above" and similar derivatives were modified to include the previous answer being referred to, where identifiable, and names of water utilities were removed to ensure their anonymity.
A sample of the responses were manually reviewed to identify keywords describing the experts' assessments of the likelihood that the benefit could be achieved, and the extent of that benefit. An automated thematic analysis system was then iteratively developed [67][68][69][70] using SQL-queries and online dictionaries to group keywords and phrases into common levels [71,72].

Quantitative Analysis
We indicate how expert opinions could eventually be utilized, in a benefit value model for DWM. A model could be built using the @Risk [73] add-on to MS Excel [74] to demonstrate the use of distributions developed from expert assessments, and the Monte Carlo simulation of benefit value for a DWM project. The aim would be to provide a probablistically-weighted range of outcomes that could be a plausible input to business cases [63]. A set of hypothetical annual operating costs was developed by the researchers from published reports of water utilities and regulators, then used to test such a model. Here we merely illustrate the type of outputs available from such a model.

Descriptive Statistics of Participant Profiles
Experts participating in the survey (n = 52) came from a cross-section of water utility types in Australia (large metropolitan authorities, regional urban authorities, water authorities within Local Government bodies) and related organisations (regulators, advocates, service industries). Their range of expertise and experience represented by their years in the industry, business areas and previous work with DWM are recorded in the following tables. Twenty-nine (56%) came from experts in Metropolitan water utilities, sixteen (31%) from regional water utilities and seven (13%) were former employees of water utilities or who work for organisations servicing the industry. Table 1 shows a breakdown of the number of years of experience of the participants for each of the service organization types. Participants were asked to identify the business areas in which they had experience and/or expertise. Table 2 shows the break-down for major business groups characterized as customer, technical and support. The survey asked for their level of specific experience and knowledge of DWM. Table 3 summarises the participants' level of experience within the organization types. Table 1. Years of experience in the water industry by water business service organisation type.

Agreement Levels for DWM Benefits
The first thematic analysis identified codes describing the experts' assessments for the first question of "agreement level", and six themes were formed: strongly agree, agree, somewhat agree, low-agreement, do not agree, do not know (see Table 4). Codes were categorized into themes [71,72] with a Likert style scoring. The thematic analysis and Likert style scoring allowed responses to be scored and frequency distributions built for the agreement level of each benefit. While the level of agreement varied between individual benefits, overall 82% agreed that digital water meters would deliver benefits with only 6% disagreeing and 12% not offering an opinion ('do not know'). The benefits for water businesses scored higher levels of acceptance (83% agree, 7% disagree) than those benefits for customers (76% agree, 7% disagree) (see Figure 1). The level of 'do not know' responses jumped from 10% for the business benefits to 17% for the customer benefits. The number of 'do not know' responses was unexpected as questions had been blocked into 13 areas of expertise and participants were asked to select the block or blocks that they felt they could answer, but it may indicate that experts had not considered the benefit previously and were unwilling to make an assessment at the time. The level of agreement scores by experience with digital metering again showed the discrepancy between business benefits and customer benefits, but also showed that inexperienced staff were more inclined to answer "do not know" (see Figure 2). After ignoring the 'do not know' responses the levels of agreement were consistent, ranging between 7.3% (experienced staff, business benefits) to 9.3% (experienced staff, customer benefits) with inexperienced staff in the middle (7.8%) for both  The level of agreement scores by experience with digital metering again showed the discrepancy between business benefits and customer benefits, but also showed that inexperienced staff were more inclined to answer "do not know" (see Figure 2). After ignoring the 'do not know' responses the levels of agreement were consistent, ranging between 7.3% (experienced staff, business benefits) to 9.3% (experienced staff, customer benefits) with inexperienced staff in the middle (7.8%) for both beneficiary groups. The level of agreement scores by experience with digital metering again showed the discrepancy between business benefits and customer benefits, but also showed that inexperienced staff were more inclined to answer "do not know" (see Figure 2). After ignoring the 'do not know' responses the levels of agreement were consistent, ranging between 7.3% (experienced staff, business benefits) to 9.3% (experienced staff, customer benefits) with inexperienced staff in the middle (7.8%) for both beneficiary groups. Experts with more years of experience were more likely to provide an opinion than inexperienced participants. Also, customer benefits attracted more 'do not know' responses across all years in the industry (see Figure 3). Experts with more years of experience were more likely to provide an opinion than inexperienced participants. Also, customer benefits attracted more 'do not know' responses across all years in the industry (see Figure 3). The level of agreement scores by the job role of staff is shown in Figure 4. Technical staff were more likely to provide opinions and, unexpectedly, 15% of the time, experts in customer service job roles did not provide an opinion for customer benefits. However, after excluding the 'do not know' responses, the levels of agreement were consistent across benefits regardless of the job role. The level of agreement scores by the job role of staff is shown in Figure 4. Technical staff were more likely to provide opinions and, unexpectedly, 15% of the time, experts in customer service job roles did not provide an opinion for customer benefits. However, after excluding the 'do not know' responses, the levels of agreement were consistent across benefits regardless of the job role. The level of agreement scores by the job role of staff is shown in Figure 4. Technical staff were more likely to provide opinions and, unexpectedly, 15% of the time, experts in customer service job roles did not provide an opinion for customer benefits. However, after excluding the 'do not know' responses, the levels of agreement were consistent across benefits regardless of the job role. A general observation from the preceding graphs is that the level of agreement is increased with previous digital metering experience or years in the industry. The graph for job role shows that technical staff appear more aware of the benefits of DWM than customer service staff, whereas staff in other areas are least aware. It is notable that the business benefits consistently achieve a higher level of acceptance than customer benefits.
A summary of responses for each benefit, including reasons given for disagreeing with the benefits, and the frequency of scores for each of the benefits for the thematic analysis are listed in Section 2 and Section 3 of Supplementary Material A respectively. A general observation from the preceding graphs is that the level of agreement is increased with previous digital metering experience or years in the industry. The graph for job role shows that technical staff appear more aware of the benefits of DWM than customer service staff, whereas staff in other areas are least aware. It is notable that the business benefits consistently achieve a higher level of acceptance than customer benefits.
A summary of responses for each benefit, including reasons given for disagreeing with the benefits, and the frequency of scores for each of the benefits for the thematic analysis are listed in Sections 2 and 3 of Supplementary Material A respectively.

Benefit Quantification and Model Development
Having qualified the level of agreement among the experts, the study proceeded to quantify the extent of the value of each benefit to operationalise those qualitative findings into a quantitative model.

Initial Analysis
The second thematic analysis of responses looked to identify the extent of the benefit "value". This analysis was limited to those responses with a positive level of agreement (ie. levels other than "do not know" or "disagree"). The extent of benefit was indicated by numbers, or by words and phrases such "substantial", "significant", "small", "marginal", and "negligible". Forty-six (6%) of the 806 responses were quantified by a number, over half, 461 (57%), were quantified by the phrase, but 299 (37%) were neither quantified by number or phrase. Codes for the experts' assessments of the extent of the benefit "value" grouped into five themes: substantial, significant, reduce (or improve), marginal and negligible. Table 5 lists a sample of codes categorized into themes [71,72] and a Likert style scoring that was applied to these themes. Where a descriptive word/phrase was not provided but a numeric benefit value was, the descriptive word was taken from the question (eg. reduce, improve) and this was then qualified by the benefit value, e.g., Q:" . . . reduce . . . ?" A: "Yes-60%" Interpreted as "Reduce 60%". Benefit values ranged from near 0% to near 100%. The wide range of values indicated in the expert assessments (see Figures 5 and 6) lead to an examination of the business context of each benefit.    In Figure 5 and Figure 6, trend lines shown on the graph are illustrative only. The x-axis labels show the extent of benefit value nominated in the experts' assessment to benefit questions. The y-axis labels show the benefit value phrase used in the experts' assessment to the assessed question (indicated by the question code in brackets from the survey questions in Section 3 of Supplementary Material A). Many experts quantified their assessments of benefits as ranges, so all quantified values were interpreted as both indicative, rather than absolute, and as ranges, rather than single values. The rules for this interpretation were: • A single number was converted to a range by adding and subtracting 5% from the nominated value (less was added or subtracted when the number was near the extremes-100% and 0%).
(e.g., (L15) reduce 80% interpreted to 75% to 85%) • Where a range was indicated by phrase, but not specifically provided, 10% was added or taken from the limit value provided. For example, "25% or more" was interpreted as 25-35%, "up to 100%" was interpreted as 90-100%. When the nominated number was 10% or under, or 90% or over half the value to the limit was added or subtracted (e.g., (D15) reduce at least 95% interpreted as 95% to 97.5%) Figure 6. Charges and operational costs benefits: the phrase used to describe %-benefit value (y-axis) in expert assessments, shown as high-low ranges (rectangles), against the expected %-benefit value i.e., savings (x-axis). Same annotations as in Figure 5.

Examination of Benefit Value Assessments
The forty-six responses that included both descriptive word/phrase and value, provided the bridge between qualitative to quantitative analysis, and were the foundation for quantifying the ranges of benefit value. The encoded values of benefits appeared to have two distinctly different patterns-low rate (less than 5% and up to 25%) and a high rate (between 25% and 100%) (see Figures 5  and 6). The benefits in each pattern were identified as being for two distinct contexts. The "low" rate was common for the 15 benefits related to the cost of water to the business or customer. The "high" rate was common for the 31 benefits related to charges and operational costs. Three high water saving estimates among the 46 expert assessments were noted. These were value estimates provided by experts of a regional water utility with low direct experience of digital metering. Such estimates are well above reported ranges of 3-8% (percentile 10-90%) [40]. High expectations may be valid for particular situations (e.g., an existing high non-revenue water (NRW) rate due to ageing and leaky network infrastructure and/or water theft). Furthermore, if used in modelling, this information would need to be tailored to local conditions. In Figures 5 and 6, trend lines shown on the graph are illustrative only. The x-axis labels show the extent of benefit value nominated in the experts' assessment to benefit questions. The y-axis labels show the benefit value phrase used in the experts' assessment to the assessed question (indicated by the question code in brackets from the survey questions in Section 3 of Supplementary Material A). Many experts quantified their assessments of benefits as ranges, so all quantified values were interpreted as both indicative, rather than absolute, and as ranges, rather than single values. The rules for this interpretation were: • A single number was converted to a range by adding and subtracting 5% from the nominated value (less was added or subtracted when the number was near the extremes-100% and 0%). (e.g., (L15) reduce 80% interpreted to 75% to 85%) • Where a range was indicated by phrase, but not specifically provided, 10% was added or taken from the limit value provided. For example, "25% or more" was interpreted as 25-35%, "up to 100%" was interpreted as 90-100%. When the nominated number was 10% or under, or 90% or over half the value to the limit was added or subtracted (e.g., (D15) reduce at least 95% interpreted as 95% to 97.5%) The frequency of scores for each of the benefits for the second thematic analysis is listed in Section 3 of Supplementary Material A.

Quantification of Benefit Value
Ranges of benefit value for percentage savings were assigned to the five themes using the flex points on the trend lines in Figures 5 and 6 as the bounds of each range (see Table 6). While this was done for both contexts it should be noted that there was scant data for the low ends of both contexts, so assumptions have been made by the researchers for the ranges of Negligible and Marginal benefits. A sixth benefit value theme, disagree, has been added to cover the possibility of no benefit, i.e., a benefit of 0% saving. Together, the six themes can be considered to be the benefit-modelling themes. The sixth theme comes directly from the disagree theme for likelihood. As such, binary pair 'disagree' and 'not disagree', where 'not disagree' covers the positive likelihood themes listed in Table 4, creates a probability condition that can be used when modelling the benefit value.

Probability Distributions of Benefit Value
Frequency distributions for each of the six benefit modelling themes were created and used to build the stochastic model of benefit value [75] for each of the two contexts. Those expert assessments with a quantified benefit value or phrase were assigned to themes directly.
The other 299 expert assessments had a positive agreement level but no benefit value, for example, the simple response "yes". These were considered as being 'missing at random' and three methods for imputing their value [70,[76][77][78] were considered-use of the mode, proportionate, and surrogate.
The surrogate method was adopted as use of mode biased the frequencies due to the number of responses involved, and the proportionate method had no impact on the relative frequencies. The use of the Agreement Level score was considered a suitable surrogate as it included a level of enthusiasm as part of the agreement. That is, weak agreement (1) could be equated to negligible benefit value (1), etc. Assessments with the fourth agreement level group, strong agreement, were distributed to benefit levels 4 and 5 proportionately based on the overall levels of benefit levels 4 and 5. The confusion matrix (see Table 7) shows the number of responses by agreement level for each of the benefit value levels with benefit levels 4 and 5 grouped to enable comparison to the highest agreement level, 4. Empirical distributions of the scores were transformed into relative frequency probability distributions [79,80] for the benefit modelling themes and combined with the benefit value ranges for use in stochastic modelling. Table 8 provides a transformation of the frequency distributions to the benefit value ranges, for each theme, for the benefits of the cost of water context. The ranges are based on the flex points on the trend lines. Column 4 provides the raw sfrequencies for the themes. Column 5 includes both raw and imputed values for each theme. Column 6 is the cumulative frequencies and column 7 is the relative cumutaive frequncy (%). Table 9 provides the transformation, for each theme, for the benefits of the charges/operational costs context. The column definitions in Table 9 match the columns definitions in Table 8. 1 25% is the nominal maximum benefit of the cost of water savings. Higher-benefit may be possible in particularly leaky networks or wasteful situations. 2 The 299 answers that were not quantified by number or phrase have been given a default Benefit Value score of 3 based on the discussion above. 3 . The number of participants that disagreed with benefits within the Context is included as they represent views of achieving no benefit (i.e., 0%).
Based on the quantification table (see Table 8) a transformation from the cumulative frequency (for all responses) to the expected %-benefit value for the cost of water context has been developed (see Figure 7). The cumulative frequency chart is shown (see Figure 8). Based on the cumulative frequency, 5% of experts would score a benefit of 0%, another 5% (>5, <=10%) of experts would score a negligible benefit of between 0% and < 1%, etc. 1 Nominal ranges for marginal and negligible %-changes to charges and operational costs are included without any quantified values being provided in the answers of the benefits for these phrases. 2 The 299 answers that were not quantified by number or phrase have been given a default Benefit Value score equivalent to the Agreement level score (see discussion above). 3. The number of participants that disagreed with benefits within the Context is included as they represent views of achieving no benefit (i.e., 0%).     Based on the quantification table (see Table 9), a transformation from the cumulative frequency of expected %-benefit value for the charges/operational costs context has been developed (see Figure 9). The cumulative frequency chart is shown (see Figure 10). Based on the cumulative frequency, 10% of experts would score a benefit of 0%, another 2% (>10, <=12%) of experts would score a negligible benefit of between 0% and < 25%, etc.
without any quantified values being provided in the answers of the benefits for these phrases. 2 The 299 answers that were not quantified by number or phrase have been given a default Benefit Value score equivalent to the Agreement level score (see discussion above). 3. The number of participants that disagreed with benefits within the Context is included as they represent views of achieving no benefit (i.e., 0%).
Based on the quantification table (see Table 9), a transformation from the cumulative frequency of expected %-benefit value for the charges/operational costs context has been developed (see Figure  9). The cumulative frequency chart is shown (see Figure 10). Based on the cumulative frequency, 10% of experts would score a benefit of 0%, another 2% (>10, <=12%) of experts would score a negligible benefit of between 0% and < 25%, etc.   Figure 9 to enable easy comparison.

Benefit Value Modelling
A benefit value model could be developed using the frequency distributions for each benefit obtained here from the expert opinion survey. Outcomes of such models could be used in business cases to qualify the advice to management. Switches in the model could be used to exclude benefits or turn off business changes for scenario analysis to further inform business decisions. A straightforward methodology would estimate annual benefits via a Monte Carlo simulation. Frequency distributions could be used as a basis for simulating plausible benefits.
Transformations of expert opinions to benefit values (see Figure 8 and Figure 10) could provide the benefit value for each simulation. To demonstrate the approach, the @Risk [73] add-on to MS Excel [74], was used on a set of hypothetical costs for a mid-sized water utility (500,000 customers) created by the researchers from published reports of water utilities, regulators and advocates. The type of output from such a model, in the form of a range of possible total benefit values that could be estimated for a single year for 50,000 simulations, is illustrated in Figure 11. Excel formulas could be used to aggregate the individual outcomes of each benefit to determine the total cost savings.  Figure 9 to enable easy comparison.

Benefit Value Modelling
A benefit value model could be developed using the frequency distributions for each benefit obtained here from the expert opinion survey. Outcomes of such models could be used in business cases to qualify the advice to management. Switches in the model could be used to exclude benefits or turn off business changes for scenario analysis to further inform business decisions. A straightforward methodology would estimate annual benefits via a Monte Carlo simulation. Frequency distributions could be used as a basis for simulating plausible benefits.
Transformations of expert opinions to benefit values (see Figures 8 and 10) could provide the benefit value for each simulation. To demonstrate the approach, the @Risk [73] add-on to MS Excel [74], was used on a set of hypothetical costs for a mid-sized water utility (500,000 customers) created by the researchers from published reports of water utilities, regulators and advocates. The type of output from such a model, in the form of a range of possible total benefit values that could be estimated for a single year for 50,000 simulations, is illustrated in Figure 11. Excel formulas could be used to aggregate the individual outcomes of each benefit to determine the total cost savings.

Limitations
While the experts' structured interviews had a relatively low number of participants (N=52) they were drawn from water utilities that covered city, urban and regional areas of Australia plus other industry experts, and were representative of business functions of water businesses.
Recruitment of experts within water authorities was undertaken by the liaison officers within the water authority. While this appears to have been generally successful, it is not known if other expert staff might also have been available to complete the survey.
The use of an online survey to collect the opinions meant that participants needed a common level of understanding of what they were being asked to pass an opinion, since there was no interaction with the interviewer. The 'background' notes that accompanied each benefit question were intended to provide sufficient detail to establish a common understanding. This may not have always been achieved as five answers to the question of reduced claims by customers for water damage from concealed leaks and theft from behind the meter (question number 62 (L18)) referred to claims against the water utility (question number 11 (E19)).
As the benefit questions invited open commentary through free text responses, the answers required interpretation to synthesise into consistent outcomes. Where answers included terms like "as above", care has been taken to identify the answer being referred to. Additionally, the answers reflect each individual's use of language and expressiveness.
Forty of the benefits were identified as impacting customers (some shared with the water business). There has not been any attempt in the structured interviews to collect customer opinions on the benefits and the value of these benefits to them. At this stage, the only value of these benefits is the view of the expert participants. The demonstration benefit modelling tool has not attempted to model any of the customer only benefits. Results shown are based on the hypothetical set of annual operating costs for a water utility's business and shared benefits only. Using such an approach (e.g., Figure 11), a model running against a set of hypothetical annual operating costs returned a near bell-like distribution of possible outcomes with a median (50% quantile) value of $13.3 M savings per annum. There could be a 95% probability of achieving at least $8.1 M pa and a 5% probability of achieving in excess of $17.4 M pa. Further development of such a model will be detailed in a future paper which will specify all the inputs to the model and include: annual costs for every kind of benefit, use of the distributions and the calculation formulas.

Limitations
While the experts' structured interviews had a relatively low number of participants (N = 52) they were drawn from water utilities that covered city, urban and regional areas of Australia plus other industry experts, and were representative of business functions of water businesses.
Recruitment of experts within water authorities was undertaken by the liaison officers within the water authority. While this appears to have been generally successful, it is not known if other expert staff might also have been available to complete the survey.
The use of an online survey to collect the opinions meant that participants needed a common level of understanding of what they were being asked to pass an opinion, since there was no interaction with the interviewer. The 'background' notes that accompanied each benefit question were intended to provide sufficient detail to establish a common understanding. This may not have always been achieved as five answers to the question of reduced claims by customers for water damage from concealed leaks and theft from behind the meter (question number 62 (L18)) referred to claims against the water utility (question number 11 (E19)).
As the benefit questions invited open commentary through free text responses, the answers required interpretation to synthesise into consistent outcomes. Where answers included terms like "as above", care has been taken to identify the answer being referred to. Additionally, the answers reflect each individual's use of language and expressiveness.
Forty of the benefits were identified as impacting customers (some shared with the water business). There has not been any attempt in the structured interviews to collect customer opinions on the benefits and the value of these benefits to them. At this stage, the only value of these benefits is the view of the expert participants. The demonstration benefit modelling tool has not attempted to model any of the customer only benefits. Results shown are based on the hypothetical set of annual operating costs for a water utility's business and shared benefits only.

Discussion
The benefits of DWM put forward in the structured interviews were predominantly agreed with or strongly agreed with by the participants. Overall, 67.5% agreed or strongly agreed with the benefits proposed, another 14.5% offered muted agreement while only 6% disagreed and 12% offered no opinion. The authors' previous literature review [41] identified sources of benefits of DWM, including the outcomes of academic research and reviews, which collectively reflect this strong level of support. Water savings and operational savings expressed by the experts were consistent with findings by researchers and practitioners in Australia [5,40] and overseas [6].
The benefits to water utilities were better understood than benefits to customers. This suggests that water utilities should put more focus on the customer and how they might benefit, that is, become more customer-centric [81]. This might be achieved by inviting greater involvement of customer-facing and back-office staff in discussions and workshops on the impact of digital water meters. Similarly, and while the participants were considered experts, the number of "don't know" and "don't agree" comments suggests that more work needs to be done internally to bring all staff into discussions and workshops to canvass the opportunities and to reach consensus. Academics and vendors of DWM solutions have a role to play in this by promoting their studies and success stories [82] to water utilities while also documenting any issues encountered, less favourable outcomes and mitigating actions.
In a business environment, at least in Australia, where the negative sentiment towards electricity smart meters dominates customer perceptions, the ability to promote the innovation would be enhanced if the customer benefits were well understood and articulated. Victoria's water Ombudsman is of the opinion that water companies need to take the initiative to collect and promote the many success stories coming out of DWM trials [29].
Three benefits stood out for the mixed or qualified responses that they received.
• Call Centre impact-rather than an expected decrease in calls as suggested by the benefit questions, many participants expected calls to actually increase, due to increased information and questions raised in the minds of customers as to what the information meant.

•
Meter reading costs-many participants were skeptical that the meter reading cost-savings could be achieved and mention the new costs of data communication, collection and storage.

•
Failing meter detection-the operating life of batteries was often noted as a limiting factor when considering the extended life of meters and meter failure detection.
The issues raised for these three benefits might be the subject of short studies to determine the actual situation or, for battery life, mitigating strategies.
The model developed in this study has been based around the digital metering taxonomy. The taxonomy provides generalised categories of benefits and is sufficiently flexible in design to allow for benefit items to be included or removed depending on the business operations of the water utility and their corporate goals. Such a modelling tool has the potential to be applied to metering within other industries such as energy [83].

Conclusions
This study has focused on the benefit likelihood and value of DWM. It has not considered the cost of implementing changes to achieve the benefits as these are considered subject to advances in AMI technologies, implementation strategies, contract negotiations and purchasing rules within individual water utilities.
Expert opinions have been collected to verify and quantify the benefits identified in the preceding literature review of DWM. In doing so the study identified two distinct probability distributions covering the expected value of benefits in two contexts: cost of water, charges/operational costs. Empirical distributions formed through thematic analysis and quantification of expert opinions have been transformed into probability distributions. A benefit value modeling tool could be developed in order to make use of these distributions. Here we provide an indication of the kind of results that could be obtained. If such a tool were used to run models, using a hypothetical set of annual operating costs. to simulate DWM benefits and it could provide stochastic outcomes for possible use in business cases.
Further studies seeking empirical evidence of benefits considered intangible is warranted to add further to business cases. Forty of the benefits (over half) relate to customer satisfaction and, while the experts generally agreed with the benefits, the opinions of customers themselves should be considered. Future work by the researchers will present the results of a study of the possible impact on customer satisfaction resulting from the change to DWM [84].
Further development of a modelling tool such as is described here, and used to illustrate how cost savings analysis can be conducted from expert opinion, will be presented in a future paper.