The MACRO system is evaluated in the scope of a printing unit demonstrator application for machine condition under laboratory conditions. Its general performance in a real-world scenario with the availability of only a small set of training examples lacking negative examples is shown. In addition, the evaluation illustrates MACRO’s conflict-solving capability. The background of the application is described in the following example.
The printing unit demonstrator is utilised for two experiments. In the first experiment, the demonstrator is operated without any changes, whereas the demonstrator as well as one of the involved sensors is manipulated in the second experiment to enforce a conflict between the sensor signals. The data acquired during the first experiment is assigned to the PU data set and that of the second experiment to the PU data set. Both sets were evaluated using MACRO, the naïve Bayes, and the Support Vector Machine algorithms in order to deduce the current condition of the demonstrator.
The evaluation setup is detailed in the next section.
4.1. Evaluation Setup
The instances in both printing unit demonstrator data sets are not labelled. Nevertheless, the printing unit demonstrator is not manipulated and is considered to be operating in normal condition
at least at the beginning of both experiments: Considering the PU
data set, all instances contained in this set represent
, whereas in the PU
data set the first manipulation of the printing unit demonstrator begins at
(cf. Appendix A
). Therefore, the first 100 instances of each data set are utilised for training. The whole data set is normalised before further processing based on normalisation parameters determined from the training instances. Afterwards, these instances are utilised to train the fuzzy membership functions applied for μ
BalTLCS fusion in MACRO’s attributes.
It is the task of the evaluated algorithms to assess the instances with respect to its compatibility to the normal condition of the printing unit demonstrator. Changes in the operation condition affecting the actual condition are to be detected.
The MACRO experiment results presented in Section 4.2
and Section 4.3
are obtained with parameter
set identically for each membership function to
. This setting intentionally allows variations of the sensor signals during the demonstrator operation additional to the variations covered during the training phase. The system layer fusion’s andness degree is set in all cases to
. Based on the distribution of each feature’s values, the membership function parameters
are set empirically to the values shown in Table 3
MACRO involves three attributes in order to assess the normal condition
of the printing unit demonstrator. Each attribute is composed of features representing a physical property of the printing unit demonstrator:
Attribute 1 (Motors):
attribute involves the features of the motors’ electrical currents as well as the index of the solid-borne sound frequency with the highest amplitude. It facilitates assessment of the operation of the motors and its attached mechanical parts. Deteriorations or defects of these parts likely lead to changes in the electric currents and/or vibrations emitted by the parts (cf. [46
Attribute 2 (Contact Pressure):
This attribute subsumes all features containing information about the pressure between the wiping and plate cylinder. It contains features of the solid-borne sound intensity, the wiping cylinder motor’s electric current, and the contact force sensor.
Attribute 3 (Motor Currents):
Here, the features of the motors’ electric currents are evaluated to assess the energy consumption of the system.
These attributes are applied identically in the evaluations of both the PU
data sets. Table 4
summarises the attributes’ compositions.
The MACRO system has been implemented in MATLAB
according to the formal definitions given in Section 3.5
and Section 3.6
. The evaluation has been carried out with MATLAB/Simulink 2016a (184.108.40.2061360) 64-bit for Microsoft Windows from The MathWorks, Inc. (The MathWorks, Inc., Natick, MA, USA) [96
The following fusion algorithms, which originate from machine learning and classification, are utilised as benchmark algorithms. All benchmark evaluations have been carried out within Waikato Environment for Knowledge Analysis
] in order to use established implementations of the benchmark fusion algorithms to which this article’s contributions are compared. It offers a number of algorithms for machine learning disciplines like classification, clustering, or feature selection. WEKA 3.8.0 (Machine Learning Group at the University of Waikato, Waikato, New Zealand) was utilised to generate the benchmark results of the condition monitoring experiment. As only training data for the normal condition
is available in the data sets, all benchmark algorithms are evaluated as one-class classifiers:
This is a fusion algorithm originating from probability theory. It determines a conditional probability following Bayes’ theorem. In the context of the condition monitoring experiments evaluated in this section, the conditional probability
is determined. That is, the probability is computed that the feature vector
represents the normal condition
of the printing unit demonstrator. Two variants of the naïve Bayes algorithm are evaluated, which differ in the form of the applied prior distribution:
nB: The nB variant models the prior distribution as normal distribution. It adjusts the distribution’s mean and standard deviation based on the training data.
nB: No certain probability distribution is assumed for the prior distribution. It is instead estimated based on the training data by kernel density estimation applying Gaussian kernels.
WEKA implements both variants of naïve Bayes in its NaiveBayes classifier. Details on the background of the implementation are found in [99
This classifier (and also all other naïve Bayes implementations found by the authors) is only capable to be applied if data for more than one class is available in the training data. The printing unit demonstrator experiments deliver only data about the demonstrator’s condition, which is per se
assumed to represent its normal condition. Thus, the naïve Bayes implementation is applied in combination with the WEKA package OneClassClassifier (WEKA packages are conveniently installed by utilisation of its integrated package manager). This is a meta-classifier, which allows to apply any classifier on one-class problems like the printing unit demonstrator condition monitoring experiments: Based on the training data, artificial data representing its counter-class is generated, facilitating to handle the original one-class problem as two-class problem. The result is obtained by the combination of the prior information derived from the training data with the employed classifier’s output. It utilises Bayes’ theorem for this task. For details on the background of OneClassClassifier see [100
Support Vector Machine:
The Support Vector Machine
(SVM) is a classification concept, which linearly separates the data in an n
-dimensional hyperspace. Its binary output
describes whether the feature vector
represents the normal condition
of the printing unit demonstrator. The linear hyperplane is determined based on the training data and encoded in the SVM’s support vectors. It involves kernel functions, which transform the input data into a higher-dimensional space, in which linear separation is possible. In the scope of the printing unit demonstrator experiments, it is applied with a Gaussian radial basis function
(RBF). This is a parameterisable kernel, whose parameter
adjusts the kernel’s variance. Details on SVMs is found in [101
]. For the printing unit demonstrator condition monitoring experiment, WEKA’s LibSVM package is utilised. It is a wrapper classifier providing access to the libSVM implementation, a free SVM library, which contains a one-class SVM
implementation, by Chang and Lin [102
]. Thus, it applicable to the printing unit demonstrator experiments without further adjustments.
All instances of the printing unit demonstrator data sets are evaluated in the following, i.e., the training data is also evaluated. The next section presents the evaluation results on the PU
data set, both obtained by applying MACRO fusion and the benchmark fusion algorithms. Afterwards, the results on the PU
data set are presented in Section 4.3
4.2. PUstatic Data Set Results
The inputs of the μ
BalTLCS fusion on MACRO’s attribute layer are the fuzzy memberships of the features, which are extracted from the sensor signals. Plots of the attribute healths are depicted in Figure B1
. The three attributes defined in Table 4
are fused by the IIWOWA operator on system layer to obtain the system health
of the printing unit demonstrator. Then, all information to assess the current state of the printing unit demonstrator is available from the continuously evolving system health. In order to obtain a crisp decision about the system condition, the system health function
is evaluated with respect to the following thresholds
: In this range, the system operation is considered to be normal. Deviations from % are intentionally allowed as the behaviour of physical systems is usually not constant (e.g., due to variations in the system’s environment).
: If the system health determined during operation is in this range, it is neither considered normal nor in an emergency condition. Instead, it is in a warning condition. This state may, for example, be utilised to increase attention of maintenance personnel. This range is considered as a transient area, in which it is likely that a system defect will follow in the future.
: In this case, the system is considered to be in an emergency condition. It might already bear a defect and appropriate measures, like an emergency stop of the system, have to be taken.
The thresholds are set dependent on the respective application. In the scope of the printing unit demonstrator, these are configured as
. The results of the system health including the warning and emergency areas are depicted in Figure 5
The system health values
are greater than
for all t
. Hence, the system state is correctly classified as normal for all data set instances, despite of feature variations due to the demonstrator being in its start-up phase: It is visible that both the outliers of attribute 1 and the devaluation of attribute 2, which does not correspond to the real demonstrator condition (cf. Figure B1
), have a decreased influence on the result on system layer due to their decreased importance. A slight decrease in system health caused by these effects is nevertheless perceptible, but not to the same amount as they influenced the attribute healths.
The features of the PU
data set are also evaluated by the probabilistic naïve Bayes algorithm and the SVM. Results obtained by naïve Bayes are depicted in Figure 6
The variants nB and nB perform similar. With respect to the training phase (up to 02:59 (min:s)), both variants assess the printing unit demonstrator to be mostly in warning or emergency condition (the same thresholds as in the MACRO evaluation are utilised for the naïve Bayes evaluation). Only 3 of the 100 training instances by nB and 2 by nB are assessed as normal system operation. After training finished, the naïve Bayes classifiers assess all following instances to be no normal operation. The system health determined by nB decays in average up to 16:31 min, when it reaches for the remainder of the experiment. The nB variant is more optimistic and yields non-zero probabilities up to 20:05 (min:s).
Altogether, both naïve Bayes approaches do not represent the actual normal operation condition of the printing unit demonstrator. However, they indicate a continuous drift in the system behaviour, which is plausible as the demonstrator is in its start-up phase (cf. Appendix A
The same is concluded for the SVM. Its Gaussian radial basis function kernel is parameterised with
. With this parameterisation, the SVM achieved minimal classification error for 10-fold cross-validation of the training data. Its evaluation results are depicted in Figure 7
During the training phase, 26 of the 100 instances are classified as abnormal condition. It further fluctuates between normal and abnormal condition, also after the training phase, without a physical cause. Stable classification of the printing unit demonstrator condition is hence not possible.
The next section evaluates the data collected during the manipulated printing unit demonstrator operation.
4.3. PUmanip Data Set Results
In contrast with the PU
data set, PU
contains data acquired during printing unit demonstrator operation, which also includes a number of intended and unintended influences, which the demonstrator is exposed to (cf. Table A3
). In addition, the printing unit demonstrator had already been running for around 23:00 (min:s) before data acquisition started. Thus, the data is assumed to be acquired during operation in a stable operation point of the printing unit demonstrator. With respect to the actual condition of the printing unit demonstrator, this situation is prevailing during the first seven hours and eight minutes of the experiment: During this time no changes in the system’s operation occurred or were induced.
However, filtering of the solid-borne sound sensor started at 03:45 (min:s) by activating an analogue low-pass filter to simulate a sensor defect. Between 03:55 (min:s) and 04:33 (min:s) the cutoff frequency of the filter is continuously decreased until the smallest possible is set. The filter setting is kept until 06:36 (min:s), when the filter is deactivated again. Consequently, the printing unit demonstrator was operated from 06:36 (min:s) on under the same constraints as at the beginning of the experiment. It changed from 07:08 (min:s) on, when uneven rotations of the cylinders were perceptible. This change in the behaviour of the system was not intended and is interpreted as a temporary defect of the demonstrator. The experiment ends by intentionally lifting the wiping cylinder at 08:33 (min:s) in order to cancel contact pressure between the cylinders. This represents a new operation point and not a malfunction, as no defect resulted in the decreased contact pressure.
It is shown in the following that the MACRO system is capable to represent the actual situation of the printing unit demonstrator in its outputs. The MACRO attribute healths of this case are depicted in Figure B2
. The resulting system health is depicted in Figure 8
. Here, the same warning and emergency thresholds are applied as for PU
The system health is constantly from the beginning of the experiment until the activation of the low-pass filter. Then starts to decrease, but remains above the warning level until the minimal cutoff frequency is set at 04:33 (min:s). Hence, the evolving defect of one sensor is compensated by MACRO and does not result in assessing the printing unit demonstrator to be in emergency condition.
In the following period up to 06:36 (min:s) the system health falls into the warning area twice before the system health temporarily also falls in the emergency area. The simulated sensor defect is not the main reason for the decrease of . Instead, actual imperceptible variations in the demonstrator’s behaviour affect the motor currents. The variations continue on the one hand between 06:36 (min:s) and 07:08 (min:s), when the printing unit demonstrator’s operation setup is reset by “repairing” the solid-borne sound sensor. On the other hand the average system health value increases again, reflecting the improvement in the sensor equipment.
The uneven rotations of the cylinders between 07:08 (min:s) and 08:33 (min:s) result in low system health values. Its magnitude is limited by attributes 1 and 3, which both are zeroed, whereas the course of
follows attribute 2 (cf. Figure B2
). Nevertheless, the latter attribute affects the system health only to a small extent due to its decreased importance during this period.
During the remaining time of the experiment, the system health is zeroed, according to the attributes and the printing unit demonstrator’s actual condition: It is completely different from the condition during the training phase due to lifting the wiping cylinder.
The results of the naïve Bayes classification algorithms with respect to the features contained in the PU
data set are visualised in Figure 9
The printing unit demonstrator is physically in normal condition up to 07:08 (min:s). This is only partly represented in the naïve Bayes classifications before sound filter activation at 03:45 (min:s). During this time, the probabilities of both variants nB and nB vary and determine the demonstrator’s condition to be mostly in a warn or emergency state. Only 23 (nB) and 25 (nB) data set instances are assigned normal condition.
Along with the activation of the sound filter, the probabilities decrease further. The gradually increasing attenuation of the solid-borne sound signal is also represented in the outputs of the naïve Bayes algorithms: Their probabilities approach in the respective time frame between 03:55 (min:s) and 04:33 (min:s). The probabilities of nB remain on this level until the end of the experiment, whereas nB classifies two instances representing normal condition ( (min:s)). These are assumed to be numerical errors caused by the classifier’s implementation rather than caused by the demonstrator: the plate cylinder turns unevenly during this time and hence the printing unit demonstrator is not in normal operation condition. However, it was not possible to verify this assumption.
Altogether, both naïve Bayes approaches do not represent the actual normal operation condition of the printing unit demonstrator. They are also misled by the simulated solid-borne sound sensor defect, which does not affect the true physical condition of the printing unit demonstrator.
Similar results are obtained during SVM evaluation. The SVM’s Gaussian radial basis function kernel is parameterised with
. With this parameterisation, the SVM achieved minimal classification error for 10-fold cross-validation of the training data. Its evaluation results are depicted in Figure 10
The SVM constantly classifies the instances in the training data to represent normal condition up to 01:31 (min:s). Then, the SVM begins to vary in its decision, until its decision is from 03:45 (min:s) on. It is thus correct in its decisions from 07:08 (min:s) onwards, when the printing unit demonstrator is actually no more in the condition in which it was during the training phase. Nevertheless, the SVM approach is unstable shortly after the beginning of the experiment in its decisions. It is also misled by the solid-borne sound sensor defect.
The results obtained during the printing unit demonstrator condition monitoring experiments are discussed in the following section.