Evaluating Interactive Visualization of Multidimensional Data Projection with Feature Transformation

There has been extensive research on dimensionality reduction techniques. While these make it possible to present visually the high-dimensional data in 2D or 3D, it remains a challenge for users to make sense of such projected data. Recently, interactive techniques, such as Feature Transformation, have been introduced to address this. This paper describes a user study that was designed to understand how the feature transformation techniques affect user’s understanding of multi-dimensional data visualisation. It was compared with the traditional dimension reduction techniques, both unsupervised (PCA) and supervised (MCML). Thirty-one participants were recruited to detect visual clusters and outliers using visualisations produced by these techniques. Six different datasets with a range of dimensionality and data size were used in the experiment. Five of these are benchmark datasets, which makes it possible to compare with other studies using the same datasets. Both task accuracy and completion time were recorded for comparison. The results show that there is a strong case for the feature transformation technique. Participants performed best with the visualisations produced with high-level feature transformation, in terms of both accuracy and completion time. The improvements over other techniques are substantial, particularly in the case of the accuracy of the clustering task. However, visualising data with very high dimensionality (i.e., greater than 100 dimensions) remains a challenge.


Introduction
With the explosive growth in the size of available data (Big Data), there is an increasing demand to help users better understand the Big Data they have.A large portion of the Big Data is high dimensional and is notoriously difficult for humans to comprehend because of the lack of physical analogy of data with more than three dimensions.Various dimension reduction techniques have been developed to reduce the data dimensions, so they can be visually displayed [1,2].Dimensionality Reduction (DR) techniques such as Principal Component Analysis (PCA) and Multidimensional Scaling (MDS) allow analysts to project multidimensional data to a lower dimensional (2D or 3D) visual display as scatterplot diagrams where patterns such as groups and outliers can be easily identified.The approach is widely used for explorative analysis of large information spaces.developed, for example Sammon proposed a version of the MDS algorithm [16] to compute a projection that is able to represent non-linear structures in the data.In the beginning of the 21st century, newer non-linear techniques, based on neighbour embedding, were proposed.These algorithms compute a manifold in a low-dimensional space from high dimensional data with an underlying structure.Some of the best known examples are isometric embedding mapping or Isomap [17], Laplacian Eigenmaps (LE) [18], locally linear embedding (LLE) [19], local tangent subspace alignment (LTSA) [20] and t-Distributed Stochastic Neighbour Embedding (t-SNE) [21].
Moreover there are methods that use class information to guide the computation of the projection, that is, supervised dimensionality reduction.Available supervised methods include the Linear Discriminative Analysis (LDA) [22] that extracts the discriminative features to the class labels and uses them to generate embedding, the Neighborhood Components Analysis (NCA) [23] that learns a distance metric by finding a linear transformation of input data such that the average classification performance is maximized in the projection space, and the Maximally Collapsing Metric Learning (MCML) [24] that aims at learning a distance metric that tries to collapse all objects in the same class to a single point and push objects in other classes far away.
DR techniques estimate the underlying structure and reveal relationships in multidimensional data.However, due to noise and irrelevant attributes, a satisfactory projection is not always obtained.Feature selection and transformations have been developed to improve performance of many applications in several research fields [25,26].A recent approach [8] transforms the feature space by extending specific features of selected dimensions.The result can be applied to improve group separation and reduce visual cluttering in the final embedding.
Furthermore, with the increasing size and complexity of data, it becomes more difficult to generate meaningful projections in a fully automatic way.This leads to the development of interactive multidimensional data projection techniques that facilitate interactive analysis by integrating the analyst's knowledge about the data with the knowledge gained during the learning process.Examples include the iPCA approach [6] that provides coordinated views for interactive analysis of projections computed by PCA method and the iVisClassifier system [7] which improves data exploration based on a supervised DR technique (LDA).Moreover, the DimStiller framework [27] analyzes dimension reduction techniques with interactive controls that guide the user during the analysis process and Dis-Function [28] provides an interactive visualisation to define a distance function.Similarly, AxiSketcher [10] allows user to change the projection dimensions interactively.Perez et al. [9] proposed an interactive framework for feature space extension that allows the user to incorporate class labels into the projection gradually.A hierarchical interpretation can be done using the clusters of the initial projection and the class labels that are revealed by the method.More details of this technique can be seen in Section 3.
The previously mentioned techniques are only part of a rich body of research that exists on multidimensional data visualisation.Integrating human knowledge into the analysis loop requires understanding of the usability of the techniques mentioned.There are metrics for comparing the quality of visualisation layouts, but they do not consider human perception.Examples include the rank-based criteria framework by Lee and Verleysen [29] that is scale independent and many high-dimensional data visualisation quality metrics discussed in the survey by Bertini et al. [30].
There are a number of experiments studying the effectiveness of the projections from a user's perspective.Different quality measures were proposed to evaluate scatterplots based on visual perception, for example in terms of correlation [31], cluster separation [32], or both [33].Lewis et al. [34] investigated whether human evaluations of the projections are reliable, showing that user experts are reasonably consistent about layout quality, but novices disagree on the quality.Recently, a controlled user experiment [4] was performed to evaluate the human performance on multiple tasks with different projection techniques.The results demonstrated that performance of projection techniques varies with cognition task and is also data dependent.As far as we know, there has been no user evaluation on the effectiveness of interactive visualisation techniques for DR, which this work aims to address.

Feature Transformation
The main idea of the interactive feature transformations proposed in [9] is to extend the attributes based on prior knowledge such as class labels.Assuming a data matrix X where rows correspond to objects, columns are features, and the labels y describe the categorical class of each object: Being i = 1, . . ., n and j = 1, . . ., d, where n is the number of points and d the number of dimensions.Then a new data matrix X is defined using the original data matrix X and a new extended part X as follows: This extended part corresponds to the statistical value based on the class labels.Here we use the mean values of each class member.Using the extension of the full feature space, then this part X corresponds to the centroids of each class member.
where C y i is the set of objects belonging to class y i .
A real parameter λ ∈ [0, 1] allows the transition between original data (X) and the extended part ( X) by applying simple changes in the metrics of the feature space using the matrix W λ ∈ R 2d×2d .This matrix allows a weighted feature extension of the both parts of the matrix: where the matrix W λ is defined as follows: The parameter λ controls the changes between the original data structure and the centroids of the introduced classes.Theses changes are independent of the technique used for computing the projection.They produce a better separation of the introduced groups in the projections.Therefore a visual improvement is achieved by means of a controlled modification of the original structure, essentially a trade-off between visual clarification and structural preservation.
Below is an example using the iris flower data [35] that contains three species of iris: setosa, virginica and versicolor.Each species has four features: the length and width of the sepals and petals, measured in centimetres.This data set has been used in data analysis, as an example by many classification techniques in machine learning.Below is part of this data set represented as a matrix as described in Equation (1): The new data matrix X (as in Equation ( 2)) is composed by the original data and the extended part using the class information from the species of iris.This extension is built using the mean feature vector for each class.For instance, if the mean feature vector for setosa is m setosa = (5.01,3.43, 1.46, 0.25) and for virginica m virginica = (5.93,2.77, 4.26, 1.32), then the new data matrix is as follows: The two parts of this new matrix are then weighted using the λ parameter defined in Equation ( 5), where λ = 0 corresponds to the original matrix and λ = 1 leaves the extended part only.Finally, embeddings can be computed with a DR technique.Figure 1 shows the resulting projections with a series of λ values using a supervised DR technique MCML (as discussed in Section 2).
Projections of the Iris dataset with λ value from 0 to 1.The colour is used to help illustrate different clusters here, and was not used in the actual experiment.

Experiment
A controlled experiment was conducted to evaluate the effectiveness of the interactive feature transformation technique.The goal is to understand its impact on high dimensional data visualisation, and consequently the user's ability to gain insight from the data.The experiment followed a within-subject design, and task accuracy and completion time were collected for comparison.

Pilot
A pilot study was conducted with three participants using the three conditions: 1.
Visualisation generated by PCA.This is the same as the first condition in the final experiment (as described in Section 4.2).

2.
Static Feature Transformation.The visualisation in this condition included the distortion introduced by the feature transformation.However, the user was not allowed to change the level of distortion, so the visualisation was static.

3.
Interactive Feature Transformation.This is similar to the previous condition, but users could interactively change the level of distortion introduced by feature transform.This is achieved through a slider that changes the λ value.
Two issues were identified after analysing the results from the pilot study: • Both Feature Transformation conditions performed better than the PCA condition.However, this is partly due to the fact that they utilise the clustering information, whereas PCA does not.We believe that this gave the two Feature Transformation conditions unfair advantage.As a result, we decided to introduce a new DR technique that also uses the clustering information.
• There was large variation in the performance of the interaction feature transformation condition.One participant always set the λ to the maximum value.As a result, each cluster transformed into a single point and the tasks became trivial.To avoid this scenario, we removed the interactive feature transformation condition, and replaced it with two static feature transformation conditions that have low and high level of distortion respectively.

Conditions
Four revised conditions were included in the main experiment: 1.
Visualisation generated by PCA.The PCA is used as an example of DR technique that does not utilize clustering information.While it is possible to include additional DR method such as MDS, it will make the experiment overly long (it is close to one hour already with the four conditions) and it is not the focus of this study to compare DR techniques that do and do not use clustering information.

2.
Visualisation generated by MCML.This represents supervised techniques that take into account the class labels information during dimension reduction, since feature transformation also requires class information.This should produce visually more separated results than PCA because of the additional class labels information.Because feature transformation is independent of the DR technique used, any technique that uses class label can be used, so long as it is also used in the two feature transformation conditions.

3.
Visualisation generated by low-level feature transformation distortion (FT-low), based on the results of MCML.The visualisation in this condition includes low level distortion introduced by the feature transform, and the user was not allowed to change the level of distortion.A small λ value was selected manually to ensure considerable visual difference from the MCML condition.This is to emulate the scenario when a low level of distortion is introduced through interactive feature transformation.

4.
Visualisation generated by high-level feature transformation distortion (FT-high), based on the results of MCML.This is similar to the last condition except that the distortion level was higher.
A larger λ value was selected manually to (1) ensure considerable visual difference from the FT-low condition; and (2) avoid reducing the question to a trivial task, e.g., every cluster is reduced to a single point.This is to emulate the scenario when a high level of distortion is introduced through interactive feature transformation.
We selected λ = 0.1 and λ = 0.3 for the FT-low and FT-high condition respectively after considering different λ levels for all the datasets used.This ensures for all datasets enough visual difference between these two conditions and from the MCML only condition (Condition 2), without reducing the question to a trivial task.For example, Figure 1 shows the distorted projections of the iris dataset with different λ values.Please note that the colour here is to help demonstrate the effect of feature transformation.All the data points appear black in the experiment; no clustering information was provided through colour.

Tasks
The participants were asked to complete two types of tasks during the experiment: identifying clustering and outlier.They are common in high-dimensional data analysis, and usually form the basis of more complex analysis tasks.

•
Clustering: The participants were asked to identify visually the number of clusters in the display.This is to test how well the resulting visualisation reveals the clustering structure within the original high-dimensional dataset.

•
Outlier: Similarly, this task requires participants to identify visually an outlier within the original dataset, which is another important property of high-dimensional data.To simplify the accuracy measurement, each dataset has exactly one outlier, so the answer can be either correct or incorrect.This avoids the case of 'partially correct' answers when there are two or more outliers.
We deliberately did not give formal definition of 'clustering' and 'outlier' during the training stage of the experiment.We wanted to see the participants' intuition about these concepts, and its impact on task performance.As it turned out, all participants were able to grasp these concepts easily with the examples given during the training stage, and apply them successfully in the following tasks.

Datasets
We used a number of benchmark and synthetic datasets in the experiment.The goal was to cover a wide range of data size, dimensionality, and number of clusters in the dataset.The benchmark datasets are widely used by machine learning and visualisation communities, and their details are in Table 1.The projections of all four conditions were checked before the experiment to ensure that the datasets do not favour any particular condition.We manually checked all the projections to make sure there were no trivial cases where clusters collapse into points.For each dataset, a new point was added as the outlier.For half of the datasets, we added an outlier with extremely large value, using the formula below: For the rest of the datasets, we added an outlier with extremely small value: where Q 1 is the lower quartile (or the 25th percentile), Q 3 is the upper quartile (or the 75th percentile), and IQR the inter-quartile range (Q 3 − Q 1 ).This computation was applied to all dimensions in the corresponding dataset.

Participants and Procedure
We recruited 41 participants, with valid data collected from 31 of them.In several cases, the participant did not complete the experiment (participant can quit the experiment at any time without giving a reason) or there was a software error, so their data were not included for analysis.The participants were of mixed age range and technical background, including university students, administration staffs, and family and friends.It is voluntary to provide demographic information.In total, 11 participants chose to provide information about their age group (one under 19, six 19-25, and four 26-39) and gender (ten males, one female).
The study lasted approximately 45 min and consisted of three sections: training, experiment, and feedback.The training section started with the consent and demographic information form.After that, the two experiment tasks were explained using one example each.This part also showed the participants how to answer questions using the experiment software interface.The last part of training was practice, during which participants needed to complete one question for each task type.During practice, feedback was given if the participant did not answer correctly.Figure 2   The second section was the main experiment.The interface was the same as the training stage, except without feedback.As a within-subject design, each participant completed the two tasks on all six datasets under all four conditions.This led to in total 2 × 6 × 4 = 48 questions.The order of the questions were counter balanced using Incomplete Block Design to avoid learning effect.Also, the same dataset appears quite differently under the four conditions, so it is unlikely that participants can recognise them under different conditions.Figure 3 shows the four conditions of the HIV dataset.Please note the data point colour and shape are for illustration only and they are not used in the actual experiment.It is not easy to recognize that these four projections are the same dataset, even when placing them next to each other with the colour and shape.The chance is very small that a participant can recognize so during the experiment when they appear randomly and without colour or shape.The task accuracy and completion time were recorded for further analysis.
The last section is feedback, during which the participants were asked to provide subjective comments about the tasks and visualisation.Because the participants are not aware of the four conditions (the information is not provided in the experiment), the feedback was not specific to experiment conditions.Figure 3.The four conditions of the HIV dataset.Please note the data point colour and shape are for illustration only and they are not used in the actual experiment.It is not easy to recognize that these four projections are the same dataset, even when placing them next to each other with the colour and shape.So when they appear randomly and without colour or shape, the chance that a participant could recognize them during the experiment was very small.

Hypotheses
Hypotheses 1.We hypothesise that participants will perform significantly better, in terms of both accuracy and completion time, with MCML than with PCA, because MCML takes advantage of additional clustering information.We hypothesise that this will be the case for both the clustering and outlier tasks, because the two require similar visual information, i.e., it is easier to identify outliers if the clustering is visually clear.
Hypotheses 2. Similarly, we hypothesise that participants will perform significantly better with FT-low than MCML, in terms of both accuracy and completion time.The only difference between the two is the distortion introduced by the feature transformation, which makes the clustering/outlier structure visually more obvious.Hypotheses 3. Finally, We hypothesise that participants will perform significantly better with FT-high than FT-low, but only in accuracy.The higher level of distortion in FT-high will usually result in even clearer clustering/outlier structure, thus better accuracy.While it is likely the completion time will be shorter with FT-high, it can be already quite short with FT-low.As a result, the difference may not be significant.

Results
We used a repeated-measure analysis of variance (RM-ANOVA) to analyse the task accuracy and completion time of 31 participants with valid collected data.Accuracy was measured as the percentage of correct answers.Completion time was measured in seconds; however, it was not normally distributed as shown by the result of a Shapiro--Wilk test.We used the logarithm of completion time to normalize the skewed distribution.

Methods
Overall, FT-high performed the best: it is significantly more accurate than the three other conditions (Figure 5a) and took significantly less time than PCA and FT-low (Figure 6a).This supports our Hypothesis 3 and demonstrated that feature transformation can help users better understand multi-dimensional data.The improvement is more obvious in term of accuracy (Figure 4a) and less so for completion time (Figure 4b).
FT-low did not perform as well as we expected.It is significantly more accurate than MCML (Figure 5a), as in Hypothesis 2, but it required longer completion time than MCML (Figure 6a), which is different from what we hypothesised.Figure 7a,b shows the detailed completion time of clustering and outlier task respectively, ordered by dataset size.Figure 7a shows that the completion time under the FT-low is comparable to other conditions for the clustering task.However, its time is much longer than the rest for the outlier task (Figure 7b), especially the HIV dataset.As in Table 1, the HIV data has the highest dimensionality (159) among all the data sets, which can be the cause of the poor completion time of the outlier task under FT-low.The performance of the MCML condition is one of the surprises in the experiment results.It has the lowest task accuracy (Figure 5a), and this is the case for both the clustering (Figure 5b) and outlier task (Figure 5c).It was expected to out-perform PCA (Hypothesis 1), given that it takes advantage of the clustering information, i.e., which data point belongs to which cluster. Figure 7c,d show the accuracy of the clustering and outlier task respectively.For the clustering task, the accuracy of MCML is particularly poor for the HIV dataset.The results of the same dataset are even more extreme for the the outlier task (Figure 7d): except for PCA, the accuracy of the other three methods are all 0%.The high dimensionality of the HIV dataset may be the cause here, particularly for the outlier task; it also led to long completion times for the outlier task for FT-low (Figure 7b) as discussed earlier.
Figure 3 shows the four conditions of the HIV dataset with the outlier inserted.The outlier is marked as class 6 (the red triangle).For clustering, it is obvious that the clusters are not well separated in all cases, particularly for MCML, which may explain the results in Figure 7c.Similarly, it is easy to see that the outlier is not well separated from other data points in MCML and FT-low, which makes it difficult to spot when the colouring is removed (no colouring was used in the experiment.)While the outlier is better separated in FT-high, the two data points in the top-right corner may make it difficult to select the true outlier.This can be the reason for the poor performance of these three conditions, as shown in Figure 7d.
The completion time of MCML is surprisingly fast.Overall there is no significant difference between MCML and FT-high, which was expected to have the fastest completion time (Figure 6a).However, the detailed results in Figure 7a,b show that the absolute difference is not that substantial, even if it is statistically significant.
Finally, PCA performed better than expected in the experiment.It was expected to be the least accurate method overall (Hypothesis 1), but this is not the case (Figure 5a).The poor performance of other methods on the HIV dataset, particularly the outlier task (Figure 7d), can be a contributing factor.Also, it is interesting that its accuracy varied dramatically for the outlier task among the datasets (Figure 7d): while it performed extremely well for the HIV dataset, the accuracy dropped to 0% for the Tse300 and Yeast dataset.Time-wise, PCA is comparable to other methods, except for the clustering task (Figure 4b).The detailed results in Figure 7a show that this may be the result of the large difference with the Bbdm13 dataset.However, further investigation into the individual completion time did not reveal any anomaly.Overall, being one of the classic DR methods, PCA does a reasonably good job to support user understanding even though it was not designed for this purpose.

Data Size and Dimensionality
It is important to understand how the performance of different methods scale with data.This is particularly relevant if these approaches are to be applied to Big Data.There are two possible scaling: data size, i.e., number of data points, and data dimensionality.The data sets in Figure 7a-d are ordered by their sizes, i.e., increasing from left to right.Figure 7a,b show that the completion time does not increase with data size.In fact, it took longer with the HIV dataset, which has the smallest number of data points (78), than the Yeast dataset, which has the largest number of data points (1452).This is the result of pre-attentative visual processing [39]: users use the data point location, which is one of pre-attentative visual features, to decide clustering structure, and the processing of such visual feature takes constant time, regardless the number of points.This is one of the main advantages of data visualisation: information represented with pre-attentative visual features can be processed very quickly irrespective of the data size.There is no obvious trend in the task accuracy (Figure 7c,d), either.Other factors, such as the complexity of the clustering structure and appropriateness of the visualisation method, may have more of an impact on the task performance than the data size does.
Figure 8 shows the same results as in Figure 7a-d, but ordered by the data set dimensionality, increasing from left to right.There is a weak trend of increasing completion time with the data dimensionality (Figure 8a,b), which is an indicator of the data set complexity.The trend is less clear for the accuracy results (Figure 8c,d), possibly because the suitability of the visualisation method is the main factor.For example, PCA led to low accuracy with the Yeast and Tse300 dataset, and performed very well with the result of data sets (Figure 8d).

Tasks and Participants
While not the main goal of this study, we also examined the performance difference between the two tasks used in the study.The results show that in general the clustering task is more difficult than the outlier task, which is supported by both the performance metrics and user preference.The clustering task has significantly lower accuracy than the outlier task (t-test, p < 10 −5 ), and the difference is obvious as shown in Figure 9a.Similarly, the clustering task took significantly longer to complete than the outlier task (t-test, p < 10 −6 ), and the difference is sizeable as shown in Figure 9b.User preference data (Figure 9c) showed a similar pattern, with the clustering task being perceived as significantly more difficult than the outlier task (Fisher's exact test, p < 10 −6 ).This strengthens the argument for applying a Feature Transformation type of approach when visualising high dimensional data: FT-high (high-level of feature transformation) was the only condition with more than 50% percent accuracy for the clustering task and beat the second best option FT-low by a healthy 30% margin (Figure 4a).There is a weak correlation between user preference and performance.For the clustering task, the Spearman's correlation coefficient is 0.0692892 (almost no relation) between rating and accuracy, and 0.3012618 (a weak positive-more difficult, more time spent) between rating and completion time.Similarly, for the outlier task, the Spearman's correlation coefficient is -0.2622217 (a weak negative-more difficult, less accurate) between rating and accuracy and -0.1281551 (a weak positive) between rating and completion time.
We analysed the relationship between participants' performance and their demographic information such as age group.Both completion time and accuracy of the three age groups are shown in Figure 10, and they appear to be similar across the groups.The small number of participants (11) who provided their information does not allow any meaningful significance tests.Finally, we checked the performance variations among the individuals participated the study.Figure 11 shows the average completion time (Figure 11a) and accuracy (Figure 11b) of each participant across all tasks.There appears to be larger variation among the performance of the completion time than that of the accuracy, and this is the confirmed by their coefficient of variation: 0.4198652 for time and 0.129759 for accuracy.We further investigated participant 14 who had the longest completion time.For the clustering task, his completion time (Figure 12a) appears to be similar to the average time (Figure 7a) except for a few questions, such as Bbdm13-PCA and HIV-FT-high.We speculate that he struggled with these questions and spent long time to find the right answers: he correctly answered four out of five questions that he spent most time on (>40 s).This is much higher than the average accuracy.Similarly for outlier task, his completion time is also close to the average except for one question (Iris-MCML), which he answered correctly.

Limitations
As with any user study, this experiment is not without its limitations.For example, the tasks were simplified to make the experiment manageable, and thus less representative of the real-world scenario: users were not able to interactively choose the λ value for the feature transformation and there is always one outlier in the outlier-detecting task.We were aware of these limitations, and consulted the end users during the experiment design stage.While not fully realistic, they thought the simplified tasks were good enough approximation of the real-world analysis as the first step to explore the performance difference among these techniques.More realistic set-up will be explored in the further studies.

Conclusions
This paper described a user study that was designed to understand how feature transformation technique affects the user's understanding of multi-dimensional data visualisation.Four different conditions were included: PCA, MCML, low-level feature transformation (FT-low), and high-level feature transformation (FT-high).Thirty-one participants were recruited to detect clusters and outliers using visualisation of six different datasets.Both task accuracy and completion time were recorded for comparison.

•
There is a strong case for the feature transformation technique.Participants performed best with the visualisation produced with high-level feature transform (FT-high), in term of both accuracy and completion time.The improvements over other techniques were substantial, particularly in the case of the accuracy of the clustering task.

•
Low-level feature transformation has a lesser impact on visualisation readability, and as a result does not have a clear advantage over existing techniques, represented by MCML (supervised DR) and PCA (un-supervised DR).

•
Very high dimensional data seems to be a challenge for all the techniques, but particularly MCML and to certain extend FT-low.MCML performed poorly with the HIV dataset, which has a much higher dimensionality (139) than the rest of the data sets.

•
The results of PCA were better than expected; its performance was close to that of the FT-low and MCML.Also, it performed surprisingly well on the very high-dimensional HIV dataset, matching the results of FT-high.

Scalability
• All the visualisation methods scaled well with data size, particularly with completion time.
There is no apparent increase in completion time as the number of data points grow (20 fold difference between the size of the smallest and largest dataset).This is the result of human pre-attentative visual processing, which requires constant time regardless of data size.This makes visualisation an effective tool for understanding large data.

•
The data dimensionality appears to have a larger impact on the user performance than the data size.It leads to an increase in completion time as the data dimensionality grows.The effect on the accuracy is less clear, with the performance of a certain method changes dramatically between data sets.This indicates that the suitability of a visualisation method to a particular data set can be the dominant factor for task accuracy.

•
Clustering is a more difficult task than outlier identification.Its accuracy is significantly lower and took significantly longer to complete.Except for FT-high, all techniques led to accuracy of only around 25%.This demonstrates that it is almost impossible to perform visual clustering analysis without feature transformation.
• Outlier detection is the relatively easier task, with faster completion time and higher accuracy.However, its accuracy varies dramatically between data sets and techniques.One technique can have close to 100% accuracy on one dataset, but 0% on another data set with similar size and dimensionality.Therefore, selecting an effective visualisation method is important for a successful analysis.

•
Participants perceived clustering as the significantly more difficult task, but there was only a weak correlation between user preference and actual performance.There is a larger variation among the individual completion time than that of the task accuracy.
In summary, the experiment results showed that visualisation is an effective approach for high dimensional data analysis, because it does not require additional time as the data size grows.The feature transformation technique can significantly improve user's understanding, increasing task accuracy and reducing completion time simultaneously.It is almost impossible to obtain meaningful results from visual clustering analysis without feature transformation.Visualising data with very high dimensionality (i.e., greater than 100 dimensions) remains a challenge.It will be an interesting future work to evaluate further the effectiveness of the feature transformation with more realistic task settings and when in combination with more advanced approaches such as t-SNE.
is a screen-shot of the training interface.

Figure 2 .
Figure 2. The training interface for the outlier task that includes the instructions (bottom right corner) and feedback ('Well Done!' for a correct answer).

Figure 4 .Figure 5 .
Figure 4. Mean accuracy and completion time overall and for each task.(a) Mean accuracy in percentage (the higher is better); (b) Mean completion time in seconds (the lower is better).

Figure 6 .
Figure 6.Significant results of paired t-tests for completion time.An arrow from condition A to condition B indicates that participants completed the tasks much faster under A than under B. (a) All Tasks; (b) Clustering; (c) Outlier.

Figure 7 .
Figure 7.The results of the clustering and outlier task, ordered by data size.(a) Completion time of the clustering task; (b) Completion time of the outlier task; (c) Accuracy of the clustering task; (d) Accuracy of the outlier task.

Figure 8 .
Figure 8.The results of the clustering and outlier task, ordered by data set dimensionality.(a) Completion time of the clustering task; (b) Completion time of the outlier task; (c) Accuracy of the clustering task; (d) Accuracy of the outlier task.

Figure 10 .
Figure 10.Performance in different age groups and tasks.(a) Completion time; (b) Accuracy.

Figure 12 .
Figure 12.Time completion of participant 14 broken down by condition and dataset.(a) Cluster task; (b) Outlier task.