With improved connectivity and dramatically increased access to low-cost computational power, many industries are currently on the verge of a second digital revolution known as ‘Industry 4.0′ [1
]. Of the many advantages that can potentially be leveraged by Industry 4.0, the promise of increased integration of the real-time control, monitoring, and operational aspects of equipment in the process industries is one of the most attractive [1
]. It has previously been suggested that one of the so-called grand challenges which can be addressed by digitization is the notion of the ‘sustainable, 100% available plant’ [1
]. The 100% availability notion captures the vision that in the future there will be only ever be planned maintenance stops; while clearly never achievable in practice, the right balance between maintenance cost and risk must clearly be aimed for. At the heart of this aspect of industrial digitalization the topics of real-time Condition Monitoring (CM) and Predictive Maintenance (PM) can be found [1
]. As with other Internet-of-Things and Industry 4.0 applications, CM/PM applications can potentially generate large volumes of high-velocity real-time data; therefore, it is beneficial to run local processing of this data at the edge of the network (on an ‘edge device’), to reduce the frequency and real-time requirements of communication to core devices [3
]. In addition, since it is essential to have adequate fault indices that could serve as indicators in order to predict early failures and create an effective PM schedule [2
], in an Industry 4.0 context an appropriate approach would seem to be to distribute the tasks of CM/indices generation and PM analytics between the network edges and Core.
To this end, this paper is concerned with the development, validation and field-testing of a real-time CM/PM system for deployment upon large-scale pumping equipment in the water industry in an Industry 4.0 context. The CM and fault detection is achieved with an edge device implementing sliding-mode observers as soft sensors, which are designed to monitor equipment locally in a water pumping station. The role of the observers is to detect equipment wear - from mechanical stresses, or the ingress of coolant or other particulates into coupling bearing support housing - before further damage occurs to the equipment. It achieves this by monitoring several low-cost transducer signals from the equipment, and building a digital twin of the equipment being monitored; including the estimation of two critical internal parameters of the rotating equipment (bearing friction factor and coolant flow) which are impossible or difficult/expensive to directly measure. The estimated values (in conjunction with other instrumented measurements) are used as fault diagnosis indices. The edge device also performs data sampling and filtering and provides outputs for fault detection in the form of local status indicators and alarms. A Modbus RTU server is implemented on the device, and wider connectivity to the network core is provided through an additional Serial to Mobile Data (3G/4G) gateway. The core is able to pull preprocessed and summarized digital data related to local real-time parameter measurements and estimations at a lower velocity through this gateway, where further analytics related to PM are applied; this provides an architecture in keeping with the decentralized, hierarchical structure of the digitalization revolution and IoT [1
]. The focus of this paper is principally upon the implementation and testing of the edge device and soft sensors for CM.
Given the relative importance of the ‘100% available plant’ aspirational target, the CM/PM application in question can be considered to be mission-critical as any ‘false negatives’ indicated by the system may result in avoidable permanent damage to equipment, and ‘false positives’ may cause unnecessary system downtime. Both unwanted outcomes may lead to large and unnecessary economic costs and can potentially interrupt a region’s critical water supply infrastructure. It is therefore of vital importance to ensure that adequate Verification and Validation (V&V) exercises and prototype field trials are carried out on any implementation of this edge device. This paper describes such a set of exercises, carried out on a prototype Commercial Off The Shelf (COTS) microcontroller-based embedded solution, which is to be deployed locally in a pump house; a potentially harsh industrial environment.
Much previous work has been invested into developing accurate condition monitoring of rotating equipment in industrial applications; many focus on the use of vibration or acoustic analysis using real-time data processing techniques such as Wavelet transforms, Machine Learning approaches and Parameter Optimization (see, e.g., [2
]). However, the specific focus of this paper is on sliding mode observer techniques, which have also been widely used for condition monitoring and fault diagnosis in recent years (see e.g., [9
]). Two of their main advantages are that they exhibit fundamental robustness against certain kinds of parameter variations and disturbances, while also avoiding the need for high-speed signal processing techniques required for acoustic or vibration data; a specialized Sliding Mode Observer (SMO) for fault detection and isolation was developed by Edwards et al. to exploit these features [10
]. Additional key benefits of this observer approach include good accuracy of operation, combined with comparatively lower-overheads than related techniques [9
], making them a good candidate for real-time implementation in an Industry 4.0 context. Therefore, the principal goal of this paper is to help bridge the theory and practice gap by providing a detailed description and credible demonstration of an industrial implementation of a simple but effective edge device implementing this observer-based technique for fault detection of pumping equipment. Both the observer itself, and the techniques applied to implement the observer in real-time on a small, low-cost portable embedded system, have a fundamental theoretical grounding. Through laboratory-based validation using fault injection, combined with online field trials, it is demonstrated that the developed system seems a good candidate for wider deployment in industry to help achieve the goals of the second digital revolution.
The remainder of this paper is organized as follows. Section 2
describes the current application area in more depth, along with the motivation for using observers as soft sensors and their mode of operation. Section 3
describes the prototype hardware and software implementation of the edge device, along with a discussion of communication network integration. Section 4
discusses off-line validation testing of the device, while Section 5
is concerned with on-line testing (field trials). Section 6
concludes the paper.
4. Offline Validation Testing
Hardware-In-The-Loop (HIL) testing as a means for both early and late-stage system validation in real-time and embedded systems is now a well-established concept [12
]. The principle of HIL simulation of an embedded system is illustrated in Figure 7
; the unit under test (UUT) executes on representative hardware, and its outputs are fed directly to the simulator. The UUT outputs are sampled and used as input variables to a host dynamic simulation model, which is evaluated in real-time. The simulation outputs, which are synthesized representations of signals internal to the dynamic model, are then fed back to the UUT thereby closing the control loop. Since the UUT is representative of the final system implementation, many types of specification defect, omissions or otherwise unexpected interactions that may result in failures or otherwise unacceptable behaviors may be removed before the system enters operational service. However, the simulation must be designed and implemented correctly in order for results to be reliable. If this can be achieved, then HIL testing can be regarded as a representative virtualization technique which allows a potentially large design space to be explored and tested, with few (if any) consequences should the UUT malfunction or behave inappropriately at run-time.
As mentioned, during development of the observer equations, detailed models of the pumping process were created for validation purposes. The popular design and analysis package Simulink© was employed for their implementation. In order to allow HIL testing of the edge device to take place, the Simulink models of the process were employed along with real-time model simulation and interface devices provided by dSpace©. This allowed three scenario’s (baseline, coolant flow fault and bearing friction fault) to be validated on the embedded implementation. A screenshot of the HIL user interface and testing procedure is as shown in Figure 8
. In addition to simulation of the bearing unit itself, type k
thermocouples along with signal transmitters and signal isolators of the same characteristics as employed in the field trials (to be described in the following Section) were simulated as a part of the overall model.
shows data captured comparing the Simulink-based observer implementation with the edge device observer implementation during the baseline scenario. Figure 10
shows data captured comparing the Simulink-based observer implementation with the embedded observer implementation during the coolant flow fault scenario. In this fault scenario, the coolant flow rate is dropped significantly 2150 s into the simulation run, and remains as such for a duration of 600 s. As can be seen in the captured results, although the coolant flow is not a directly instrumented parameter, the observer implementation accurately tracks the coolant temperature and estimates the reduced coolant flow condition. Figure 11
shows data captured comparing the Simulink-based observer implementation with the embedded observer implementation during the coolant flow fault scenario. In this fault scenario, the bearing friction factor μ
is allowed to drift upwards starting 1000 s into the simulation run, and increases slowly over time thereafter. As can be seen in the captured results, although the bearing friction factor μ
is a non–measurable model parameter, the observer implementation detects and starts to track the gradual increase in the parameter, albeit with a small offset.
In all tested scenarios, the results obtained indicated that the observer implementations were accurate enough for the intended purpose, giving results almost indistinguishable from the (known) internal variables in the Simulink model–with less than 5% maximum and 1% average (mean) errors recorded for the estimated quantities. As such, testing progressed to field trials.
5. Online Validation Testing
Following the successful off-line validation testing, implementation tests of the edge device were then performed on Sulzer© Water Pumps at Lobwood pumping station, West Yorkshire, UK. As shown in Figure 12
, the pump house contains three pumps; two of which are active at any point in time, with the other being ready as standby in case failure. The active pumps are rotated on a scheduled basis, allowing routine checks and maintenance to be carried out on the third. The use of the observer implementation to detect bearing wear and coolant flow failure will clearly provide guidance allowing optimization of the maintenance schedule, helping to prevent both active pump failures and unnecessary changeover and maintenance. During online testing, the observer implementation was deployed on pumps two and three as indicated in Figure 12
The Input/Output schedule for the observer implementation was as shown in Figure 13
. Temperatures were measured via externally mounted, low-cost type-k
thermocouples. ALM-46 low-cost head-mount transmitters were employed to convert effective signal ranges to current transmission, and COM-3B signal isolators were employed to convert to current to voltage prior to conversion by the ADCs. The calibration of the combined sensor, transmitter, isolator and ADC for each temperature measurement channel was as follows: [0 °C:100 °C] → [4 mA:20 mA] → [0 V DC:5 V DC] → [0000h
]. Stated combined accuracies of the transmitter and isolator (due to linearization) were ±0.1% and ±0.15% of full-scale max, and the combined accuracy of the ADC converter was ±0.0015% max. Although a full meteorological analysis is deferred to future work, the accuracy of these measurement chains seems adequate to reproduce results of similar accuracy in the prototype field trials as in the laboratory-based HIL test results, as models of the same thermocouples, signal transmitter and signal isolator were employed in the laboratory simulations. During testing, a portable ultrasonic flow meter was used to measure coolant mass flow rates and hence help to calibrate the observer. Where appropriate, information on pump speed and power was obtained directly from the electronic switchgear. The observer device was mounted in an IP67 enclosure with input and output connections. The observer was installed on-site for long-term trials with data recorded to a standard desktop via the TIA-232 point-to-point connection, as shown in Figure 14
. In total, neglecting wiring costs, component costs for the encapsulated observer implementation were ≈ £200.
As shown in Figure 15
, the online site trial for Pump 3 (fixed speed) took place over a period of 7 days in which the CM system ran uninterrupted. Tracking of the observer was highly effective, with the estimated coolant flow remaining constant except during shut down periods (day 2 and day 4/5), when the flow rate fell to zero following the bearing temperature cooling (as expected). The bearing friction factor remained constant at the expected level, again indicating fault free operation, aside from during the shutdown periods when it fell to zero (as expected).
shows the online site trial for Pump 2 (variable speed) which again took place over a period of 7 days in which the CM system ran uninterrupted. Tracking of the observer was again just as effective, with the estimated coolant flow remaining piecewise constant, tracking small speed changes at several points during the trial. During the course of the experiment, pump trips occurred eight times; during the longer of these trip events (occurring 8000 s into the trial) the estimated flow rate fell to zero following the bearing temperature cooling (as expected). The bearing friction factor remained constant, again indicating fault free operation independent from pump load, aside from during the longer trip event when it fell considerably (as expected). In summary, both the on-line and off-line simulations and tests demonstrated that the observer-based edge device functioned as expected, and that parameter estimations were within acceptable limits for the proposed application. During the course of the field trials, however, it was noted that some simple improvements to fault detection and annunciation logic will be required for some specific situations when progressing from trials to continuous operation; most notably, supressing fault detection and annunciation during shut-down and trip conditions. These, and other improvements, are currently being implemented, and a full metrological analysis of the modified system is underway. Overall, however, as the device is a relatively low-cost solution (as mentioned approximately £200, excluding cabling, gateway, and network access costs) in comparison with the value of the plant, this gives a very promising indication of its overall suitability.