# A Bounded Measure for Estimating the Benefit of Visualization (Part II): Case Studies and Empirical Evaluation

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

- Reviewed the related work that prepared for this cost–benefit measure, provided the measure with empirical evidence, and featured the application of the measure.
- Identified a shortcoming of using the Kullback–Leibler divergence (KL-divergence) in the cost–benefit measure and demonstrated the shortcoming using practical examples.
- Presented a theoretical discourse to justify the use of a bounded measure for finite alphabets.
- Proposed a new bounded divergence measure, while studying existing bounded divergence measures.
- Analyzed nine candidate measures using seven criteria reflecting desirable conceptual or mathematical properties, and narrowed the nine candidate measures to six measures.

- We report several case studies for collecting practical instances to evaluate the remaining candidate measures.
- We demonstrate the uses of the cost–benefit measurement to estimate the benefit of visualization in practical scenarios and the human knowledge used in the visualization processes.
- We report the discovery of a new conceptual criterion that a divergence measure is a summation of the entropic values of its components, which is useful in analyzing and visualizing empirical data.
- Finally, we bring the multi-criteria decision analysis (MCDA) in Parts I and II together and offer a recommendation to revise the information-theoretic measures proposed by Chen and Golan [3].

## 2. Related Work

#### 2.1. Measurement Science

#### 2.2. Metrics Development in Visualization

#### 2.3. Measurement in Empirica Experiments

## 3. Overview, Notations, and Problem Statement

#### 3.1. Brief Overview

#### 3.2. Mathematical Notations

- The positions of the two stations are fixed on each grid and there is only one path between the red station and the blue station.
- As shown on the top-right of Figure 3, only horizontal, and diagonal path-lines are allowed.
- When one path-line joins another, it can rotate by up to $\pm {45}^{\circ}$.
- All joints of path-lines can only be placed on grid points.

#### 3.3. Problem Statement

## 4. Evaluation Methodology and Criteria

- (a)
- P is close to a uniform PMF${P}_{\mathrm{uniform}}$, while the ground truth Q is dissimilar to a uniform PMF—This suggests that the users may not have adequate knowledge and may have been making random guesses. In such a case, their task performance would lead to a PMF similar to ${P}_{\mathrm{uniform}}$.
- (b)
- P is close to a PMF${P}_{\mathrm{visinfo}}$that characterizes the available visual information while the ground truth Q differs from${P}_{\mathrm{visinfo}}$noticeably—This suggests that the users may not have adequate knowledge and may have been reasoning about the options in ${\mathbb{Z}}_{t}$ entirely based on what is depicted visually. In such a case, their performance would result in a PMF similar to ${P}_{\mathrm{visinfo}}$.
- (c)
- P is close to the ground truth Q, while Q differs from${P}_{\mathrm{uniform}}$and${P}_{\mathrm{visinfo}}$noticeably—This suggests that the users may have been able to make the perfect combination of the available visual information and their knowledge. In such a case, their task performance could lead to a PMF similar to the ideal PMF Q.

## 5. Synthetic Case Studies

#### 5.1. Synthetic Case S_{1}

- LD—The user has a little doubt about the output of the process, and decides the letter of bad 90% of the time, and the letter of good 10% of the time, i.e., with PMF ${P}_{\mathrm{LD}}=\{0.1,0.9\}$.
- FD—The user has a fair amount of doubt, with ${P}_{\mathrm{FD}}=\{0.3,0.7\}$.
- RG—The user makes a random guess, with ${P}_{\mathrm{RG}}=\{0.5,0.5\}$.
- UC—The user has adequate knowledge about ${\mathbb{Z}}_{w}$ but under-compensates it slightly, with ${P}_{\mathrm{UC}}=\{0.7,0.3\}$.
- OC—The user has adequate knowledge about ${\mathbb{Z}}_{w}$ but over-compensates it slightly, with ${P}_{\mathrm{OC}}=\{0.9,0.1\}$.

**ordering**, we consider ${\mathcal{D}}_{\mathrm{new}}^{k}$ “excellent”, ${\mathcal{D}}_{\mathrm{ncm}}^{k}$ “good” due to asymmetry, and ${\mathcal{D}}_{\mathrm{JS}}$ and $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$ “adequate” as the non-equal UC and OU measures are not so intuitive.

**benefit quantification**, we consider ${\mathcal{D}}_{\mathrm{ncm}}^{k=1}$ and ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ are “excellent”, $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$, ${\mathcal{D}}_{\mathrm{ncm}}^{k=2}$, and ${\mathcal{D}}_{\mathrm{new}}^{k=2}$ “good”; and ${\mathcal{D}}_{\mathrm{JS}}$ “adequate”.

**knowledge impact**, we consider all “excellent”.

#### 5.2. Synthetic Case S${}_{2}$

- CG makes random guess, ${P}_{\mathrm{CG}}=\{0.25,0.25,0.25,0.25\}$.
- CU has useful knowledge, ${P}_{\mathrm{CU}}=\{0.1,0.4,0.1,0.4\}$.
- CB is highly biased, ${P}_{\mathrm{CB}}=\{0.4,0.1,0.4,0.1\}$.
- BG makes guess based on ${R}_{\mathrm{biased}}$, ${P}_{\mathrm{BG}}=\{0.0,0.0,0.5,0.5\}$.
- BS makes a small adjustment, ${P}_{\mathrm{BS}}=\{0.1,0.1,0.4,0.4\}$.
- BM makes a major adjustment, ${P}_{\mathrm{BM}}=\{0.2,0.2,0.3,0.3\}$.

**order of divergence**can be observed in the bar chart as the first PCP where the divergence values are scaled by ${\mathcal{H}}_{max}=2$ bits. Using the collective votes as the benchmark, we consider ${\mathcal{D}}_{\mathrm{JS}}$, $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$, and ${\mathcal{D}}_{\mathrm{new}}^{k=2}$ “excellent”, ${\mathcal{D}}_{\mathrm{ncm}}^{k=2}$ and ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ “good”, and ${\mathcal{D}}_{\mathrm{ncm}}^{k=1}$ “adequate”.

**benefit quantification**, we consider $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$ to be “excellent”, ${\mathcal{D}}_{\mathrm{ncm}}^{k=1}$ and ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ “good”, and the others “adequate”.

**knowledge**can be gained from other visualization. For example, we can postulate that the reason CU, BS, and BM can make adjustments against what the clustering algorithm says is because they have seen some visualizations of the raw data without clustering at an early stage of a workflow. In general, we cannot find any major issues with the PCPs for ${K}_{\upsilon}$ and ${K}_{\psi}$. We thus rate all candidate measures as “excellent”.

#### 5.3. An Extra Conceptual Criterion

## 6. Experimental Case Studies

#### 6.1. Volume Visualization (Criterion R${}_{1}$)

**divergence ordering**, we notice a major anomaly that ${\mathcal{D}}_{\mathrm{ncm}}^{k=1}$ returns divergence values indicating the “experts” group has the most divergence, followed by “all” and then “rest”. Looking at some marginal difference in detail, ${\mathcal{D}}_{\mathrm{ncm}}^{k=2}$ indicates that “all” has the highest divergence, followed by “rest” and then “experts”. ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ indicates the group giving answer D has marginally more divergence than that giving answer C. These ordering conclusions are not intuitive. ${\mathcal{D}}_{\mathrm{JS}}$, $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$, and ${\mathcal{D}}_{\mathrm{new}}^{k=2}$ returned the expected ordering, i.e., “rest” > “all” > “experts”, and C > D > A > B.

**benefit quantification**, ${\mathcal{D}}_{\mathrm{JS}}$ and $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$ suggest that “expert” is similar to making random guesses and “rest” is similar to the A group. ${\mathcal{D}}_{\mathrm{ncm}}^{k}$ and ${\mathcal{D}}_{\mathrm{new}}^{k}$ all consider that making random guesses is more beneficial than “expert”. This becomes a question about how to interpret the difference between ${Q}_{1}$ and $\{0.25,0.25,0.25,0.25\}$, and that between ${Q}_{1}$ and $\{0.75,0.25,0,0\}$, i.e., which is the more meaningful difference?

**divergence order**, ${\mathcal{D}}_{\mathrm{ncm}}^{k=1}$ shows an outlier, indicating the A group has more divergence than random guesses. With the observation of two PCPs in the first column of Figure 7, we consider ${\mathcal{D}}_{\mathrm{JS}}$, $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$, and ${\mathcal{D}}_{\mathrm{new}}^{k=2}$ “excellent”, ${\mathcal{D}}_{\mathrm{ncm}}^{k=2}$ and ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ “good”, and ${\mathcal{D}}_{\mathrm{ncm}}^{k=1}$ “inadequate”.

**benefit quantification**, we cannot observe any major issues in the second column of Figure 7. We thus rate all candidate measures “excellent”.

**impact of knowledge**. This is understandable, as ${Q}_{1}$ deviates more from the survey results. If we assume ${Q}_{1}$ is correct, then participants clearly do not have the necessary knowledge to answer the question in Figure 6 with the misleading MIP visualization. If ${Q}_{2}$ is correct, not only do the “experts” have the knowledge, but the “rest” group also seems to have some useful knowledge. In the ${Q}_{1}$-${K}_{\psi}$ PCP, only ${\mathcal{D}}_{\mathrm{JS}}$ and $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$ indicate a positive knowledge impact for the “experts”. This is intuitive. In the ${Q}_{2}$-${K}_{\psi}$ PCP, only ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ indicates a negative knowledge impact for the A group.

#### 6.2. London Underground Map (Criterion R${}_{2}$)

Benefit for: | ${\mathcal{D}}_{\mathrm{JS}}$ | $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$ | ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ | ${\mathcal{D}}_{\mathrm{new}}^{k=2}$ | ${\mathcal{D}}_{\mathrm{ncm}}^{k=1}$ | ${\mathcal{D}}_{\mathrm{ncm}}^{k=2}$ |

spot on | $-1.765$ | $-2.777$ | $-0.418$ | 0.287 | $-3.252$ | $-2.585$ |

close | $-3.266$ | $-3.608$ | $-0.439$ | 0.033 | $-3.815$ | $-3.666$ |

wild guess | $-3.963$ | $-3.965$ | $-0.416$ | $-0.017$ | $-3.966$ | $-3.965$ |

**benefit quantification**, we consider thus ${\mathcal{D}}_{\mathrm{new}}^{k=2}$ “excellent” and the other five measures “adequate”.

**divergence order**, we consider ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ “adequate” and the other five measures “excellent”. We have not detected any major issues with the values for ${K}_{\upsilon}$ and ${K}_{\psi}$. For the

**impact of knowledge**, we thus rate all candidate measures “excellent”.

## 7. Conclusions

## Supplementary Materials

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## Abbreviations

BG | Biased process in conjunction with faithful Guess |

BM | Biased process in conjunction with a Major adjustment in decision |

BS | Biased process in conjunction with a Small adjustment in decision |

CB | Correct process in conjunction with Biased reasoning |

CG | Correct process in conjunction with Random guess |

CU | Correct process in conjunction with Useful knowledge |

FD | a Fair amount of Doubt |

KL | Kullback–Leibler |

LD | Little Doubt |

MCDA | Multi-Criteria Decision Analysis |

ML | Machine Learning |

PCP | Parallel Coordinates Plot |

PMF | Probability Mass Function |

OC | Over-Compensate |

RG | Random Guess |

UC | Under-Compensate |

## Appendix A. Explanation of the Original Cost-Benefit Measure

#### Appendix A.1. An Information-Theoretic Measure for Cost-Benefit Analysis

- Alphabet Compression (AC) measures the amount of entropy reduction (or information loss) achieved by a process. As it was noticed in [3], most visual analytics processes (e.g., statistical aggregation, sorting, clustering, visual mapping, and interaction) feature many-to-one mappings from input to output, hence losing information. Although information loss is commonly regarded as harmful, it cannot be all bad if it is a general trend of VA workflows. Thus, the cost–benefit ratio makes AC a positive component.
- Potential Distortion (PD) balances the positive nature of AC by measuring the errors typically due to information loss. Instead of measuring mapping errors using some third party metrics, PD measures the potential distortion when one reconstructs inputs from outputs. The measurement takes into account humans’ knowledge that can be used to improve the reconstruction processes. For example, given an average mark of 62%, the teacher who taught the class can normally guess the distribution of the marks among the students better than an arbitrary person.
- Cost (Ct) of the forward transformation from input to output and the inverse transformation of reconstruction provides a further balancing factor in the cost–benefit ratio in addition to the trade-off between AC and PD. In practice, one may measure the cost using time or a monetary measurement.

#### Appendix A.2. An Information-Theoretic Reasoning about Why Visualization Is Useful

## Appendix B. How Tasks and Users Are Featured in the Cost-Benefit Ratio?

- The appropriateness depends on many attributes of a task, such as the selection of variables in the data and their encoded visual resolution required to complete a task satisfactorily, and the time allowed to complete a task.
- The appropriateness depends also on other factors in a visualization process, such as the original data resolution, the viewer’s familiarity of the data, the extra information that is not in the data but the viewer knows, and the available visualization resources.
- The phrase creates a gray area as to whether information loss is allowed or not, and when or where one could violate some principles such as those principles in [64].

**Figure A1.**A visual analytics workflow features a general trend of alphabet compression from left (World) to right (Tasks). The potential distortion compares at an information space reconstructed based on the output with the original input information space. When we place different processes (i.e., (

**a**,

_{1}**a**,

_{2}**b**–

**d**)), in the workflow, we can appreciate that statistics, algorithms, visualization, and interaction have different levels of alphabet compression, potential distortion, and cost.

**A**that has a fair amount of prior knowledge about the London underground system and another user

**B**that has little. If both are shown some statistics about the system (e.g., the total number of stations of each line),

**A**can redraw the deformed map more accurately than

**B**and more accurately than without the statistics, even though the statistical information is not meant to support the users’ this task. Hence, to

**A**, having a deformed map to help appreciate the statistics may not be necessary, while to

**B**, viewing both statistics and the deformed map may help reduced the PD but may also incur more cost in terms of effort. Hence, visualization is more useful to

**B**.

**Table A1.**The answers by ten surveyees to the questions in the volume visualization survey. The surveyees are ordered from left to right according to their self-ranking about the knowledge of volume visualization. In rows 1–8, the dataset used in each question is indicated in square brackets. Correct answers are indicated by letters in round brackets. The upper case letters are the most appropriate answers, while the lower case letters with brackets are acceptable answers as they are correct in some circumstances. The lower case letters without brackets are incorrect answers. In rows 9 and 10, the self-assessment scores are in the range of [1 lowest, 5 highest].

Surveyee’s ID | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|

Questions with (Correct Answers) and [Database] in Brackets | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | P9 | P10 |

1. Use of different transfer functions (D), [Carp] | (D) | (D) | (D) | (D) | (D) | c | b | (D) | a | c |

2. Use of translucency in volume rendering (C), [Engine Block] | (C) | (C) | (C) | (C) | (C) | (C) | (C) | (C) | d | (C) |

3. Omission of voxels of soft tissue and muscle (D), [CT head] | (D) | (D) | (D) | (D) | b | b | a | (D) | a | (D) |

4. sharp objects in volume-rendered CT data (C), [CT head] | (C) | (C) | a | (C) | a | b | d | b | b | b |

5. Loss of 3D information with MIP (B, a), [Aneurysm] | (a) | (B) | (a) | (a) | (a) | (a) | D | (a) | (a) | (a) |

6. Use of volume deformation (A), [CT head] | (A) | (A) | b | (A) | (A) | b | b | (A) | b | b |

7. Toenails in non-photo-realistic volume rendering (B, c), [Foot] | (c) | (c) | (c) | (B) | (c) | (B) | (B) | (B) | (B) | (c) |

8. Noise in non-photo-realistic volume rendering (B), [Foot] | (B) | (B) | (B) | (B) | (B) | (B) | a | (B) | c | (B) |

9. Knowledge about 3D medical imaging technology | 4 | 3 | 4 | 5 | 3 | 3 | 3 | 3 | 2 | 1 |

10. Knowledge about volume rendering techniques | 5 | 5 | 4–5 | 4 | 4 | 3 | 3 | 3 | 2 | 1 |

**Table A2.**Summary statistics of the survey results in Table A1, where we classified experts simply based on their self-assessment with an average rate ($\ge 4$) in answering Q9 and Q10. They are S1, S2, S3, and S4.

All Participants | Experts | The Rest | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|

Question | A | B | C | D | A | B | C | D | A | B | C | D | |

1. (Carp) | numbers: | 1 | 1 | 2 | 6 | 0 | 0 | 0 | 4 | 1 | 1 | 2 | 2 |

probability: | 0.10 | 0.10 | 0.20 | 0.60 | 0.00 | 0.00 | 0.00 | 1.00 | 0.17 | 0.17 | 0.33 | 0.33 | |

2. (Engine Block): | numbers: | 0 | 0 | 1 | 9 | 0 | 0 | 0 | 4 | 0 | 0 | 1 | 5 |

probability: | 0.00 | 0.00 | 0.10 | 0.90 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.17 | 0.83 | |

3. (CT head) | numbers: | 2 | 2 | 0 | 6 | 0 | 0 | 0 | 4 | 2 | 2 | 0 | 2 |

probability: | 0.20 | 0.20 | 0.00 | 0.60 | 0.0 | 0.0 | 0.0 | 1.00 | 0.33 | 0.33 | 0.00 | 0.33 | |

4. (CT head) | numbers: | 2 | 4 | 3 | 1 | 1 | 0 | 0 3 | 0 | 1 | 4 | 0 | 1 |

probability: | 0.20 | 0.40 | 0.30 | 0.10 | 0.25 | 0.00 | 0.75 | 0.00 | 0.17 | 0.67 | 0.00 | 0.17 | |

5. (Aneurism) | numbers: | 8 | 1 | 0 | 1 | 3 | 1 | 0 | 0 | 5 | 0 | 0 | 1 |

probability: | 0.80 | 0.10 | 0.00 | 0.10 | 0.75 | 0.25 | 0.00 | 0.00 | 0.83 | 0.00 | 0.00 | 0.17 | |

6. (CT head) | numbers: | 5 | 5 | 0 | 0 | 3 | 1 | 0 | 0 | 2 | 4 | 0 | 0 |

probability: | 0.50 | 0.50 | 0.00 | 0.00 | 0.75 | 0.25 | 0.00 | 0.00 | 0.33 | 0.67 | 0.00 | 0.00 | |

7. (Foot) | numbers: | 0 | 5 | 5 | 0 | 0 | 1 | 3 | 0 | 0 | 4 | 2 | 0 |

probability: | 0.00 | 0.50 | 0.50 | 0.00 | 0.00 | 0.25 | 0.75 | 0.00 | 0.00 | 0.67 | 0.33 | 0.00 | |

8. (Foot) | numbers: | 1 | 8 | 1 | 0 | 0 | 4 | 0 | 0 | 1 | 4 | 1 | 0 |

probability: | 0.10 | 0.80 | 0.10 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.17 | 0.67 | 0.17 | 0.00 |

## Appendix C. Survey Results of Useful Knowledge in Volume Visualization

**benefit of visualization**and

**knowledge impact**based on the survey result of Question 5. We first define the following:

- Ground truth PMF $Q=\{0.1,0.878,0.002,0.02\}$.
- If one always answers A: ${P}_{a}=\{1,0,0,0\}$.
- If one always answers B: ${P}_{b}=\{0,1,0,0\}$.
- If one always answers C: ${P}_{c}=\{0,0,1,0\}$.
- If one always answers D: ${P}_{d}=\{0,0,0,1\}$.
- Survey results (all): ${P}_{\mathrm{all}}=\{0.8,0.1,0,0.1\}$.
- Survey results (expert): ${P}_{\mathrm{expert}}=\{0.75,0.25,0,0\}$.
- Survey results (rest): ${P}_{\mathrm{rest}}=\{0.83,0,0,0.17\}$.

- Survey results (all): ${P}_{\mathrm{all}}=\{0.8,0.1,0,0.1\}$.
- Survey results (expert): ${P}_{\mathrm{expert}}=\{0.75,0.25,0,0\}$.
- Survey results (rest): ${P}_{\mathrm{rest}}=\{0.83,0,0,0.17\}$.

**Figure A2.**Estimating the benefit of visualization and knowledge impact in relation to the survey result of Question 5 (Figure 6).

## Appendix D. Survey Results of Useful Knowledge in Viewing London Underground Maps

**Figure A3.**A survey for collecting data that reflects the use of some knowledge in viewing two types of London underground maps.

**Table A3.**The answers by twelve surveyees at King’s College London to the questions in the London underground survey.

Surveyee’s ID | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

Questions | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | P10 | P11 | P12 | Mean | |

Q1: | answer (min.) | 8 | 30 | 12 | 16 | 20 | 15 | 10 | 30 | 20 | 20 | 20 | 30 | 19.25 |

time (sec.) | 06.22 | 07.66 | 09.78 | 11.66 | 03.72 | 04.85 | 08.85 | 21.12 | 12.72 | 11.22 | 03.38 | 10.06 | 09.27 | |

Q2: | answer (min.) | 15 | 30 | 5 | 22 | 15 | 14 | 20 | 20 | 25 | 25 | 25 | 20 | 19.67 |

time (sec.) | 10.25 | 09.78 | 06.44 | 09.29 | 12.12 | 06.09 | 17.28 | 06.75 | 12.31 | 06.85 | 06.03 | 10.56 | 09.48 | |

Q3: | answer (min.) | 20 | 45 | 10 | 70 | 20 | 20 | 20 | 35 | 25 | 30 | 20 | 240 | 46.25 |

time (sec.) | 19.43 | 13.37 | 10.06 | 09.25 | 14.06 | 10.84 | 12.46 | 19.03 | 11.50 | 16.09 | 11.28 | 28.41 | 14.65 | |

Q4: | answer (min.) | 60 | 60 | 35 | 100 | 30 | 20 | 45 | 35 | 45 | 120 | 40 | 120 | 59.17 |

time (sec.) | 11.31 | 10.62 | 10.56 | 12.47 | 08.21 | 07.15 | 18.72 | 08.91 | 08.06 | 12.62 | 03.88 | 24.19 | 11.39 | |

Q5: | time 1 (sec.) | 22.15 | 01.75 | 07.25 | 03.78 | 14.25 | 37.68 | 06.63 | 13.75 | 19.41 | 06.47 | 03.41 | 34.97 | 14.29 |

time 2 (sec.) | 24.22 | 08.28 | 17.94 | 05.60 | 17.94 | 57.99 | 21.76 | 20.50 | 27.16 | 13.24 | 22.66 | 40.88 | 23.18 | |

answer (10) | 10 | 10 | 10 | 9 | 10 | 10 | 10 | 10 | 9 | 10 | 10 | 10 | ||

time (sec.) | 06.13 | 28.81 | 08.35 | 06.22 | 09.06 | 06.35 | 09.93 | 12.69 | 10.47 | 05.54 | 08.66 | 27.75 | 11.66 | |

Q6: | time 1 (sec.) | 02.43 | 08.28 | 01.97 | 08.87 | 05.06 | 02.84 | 06.97 | 10.15 | 18.10 | 21.53 | 03.00 | 07.40 | 08.05 |

time 2 (sec.) | 12.99 | 27.69 | 04.81 | 10.31 | 15.97 | 04.65 | 17.56 | 16.31 | 20.25 | 24.69 | 15.34 | 20.68 | 15.94 | |

answer (9) | 9 | 10 | 9 | 9 | 4 | 9 | 9 | 9 | 8 | 9 | 9 | 9 | ||

time (sec.) | 07.50 | 06.53 | 04.44 | 16.53 | 19.41 | 05.06 | 13.47 | 07.03 | 12.44 | 04.78 | 07.91 | 16.34 | 10.12 | |

Q7: | time 1 (sec.) | 17.37 | 08.56 | 01.34 | 03.16 | 08.12 | 01.25 | 21.75 | 15.56 | 02.81 | 07.84 | 02.22 | 46.72 | 11.39 |

time 2 (sec.) | 17.38 | 13.15 | 02.34 | 03.70 | 08.81 | 02.25 | 22.75 | 26.00 | 17.97 | 10.37 | 03.18 | 47.75 | 14.64 | |

answer (7) | 7 | 7 | 7 | 7 | 6 | 7 | 7 | 7 | 6 | 7 | 7 | 7 | ||

time (sec.) | 07.53 | 06.34 | 03.47 | 03.87 | 02.75 | 04.09 | 02.16 | 04.94 | 26.88 | 05.31 | 06.63 | 12.84 | 07.23 | |

Q8: | time 1 (sec.) | 12.00 | 08.50 | 06.09 | 02.88 | 08.62 | 14.78 | 19.12 | 08.53 | 12.50 | 10.22 | 12.50 | 20.00 | 11.31 |

time 2 (sec.) | 13.44 | 10.78 | 23.37 | 09.29 | 13.03 | 36.34 | 23.55 | 09.50 | 13.53 | 10.23 | 32.44 | 22.60 | 18.18 | |

answer (6) | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | ||

time (sec.) | 02.62 | 05.94 | 02.15 | 04.09 | 04.94 | 07.06 | 07.50 | 04.90 | 04.37 | 04.53 | 05.47 | 09.43 | 05.25 | |

Q9: | answer (P) | P | P | P | P | P | P | P | P | P | P | P | P | |

time (sec.) | 35.78 | 02.87 | 07.40 | 13.03 | 06.97 | 52.15 | 13.56 | 02.16 | 08.13 | 09.06 | 01.93 | 08.44 | 13.46 | |

Q10: | answer (LB) | LB | LB | LB | LB | LB | LB | LB | LB | LB | LB | LB | LB | |

time (sec.) | 05.50 | 03.13 | 12.04 | 14.97 | 07.00 | 26.38 | 11.31 | 03.38 | 06.75 | 07.47 | 06.50 | 09.82 | 09.52 | |

Q11: | answer (WP) | WP | WP | WP | WP | WP | WP | WP | WP | WP | WP | WP | WP | |

time (sec.) | 06.07 | 05.35 | 07.72 | 05.00 | 04.32 | 23.72 | 05.25 | 03.07 | 10.66 | 05.37 | 02.94 | 17.37 | 08.07 | |

Q12: | answer (FP) | FP | FP | FP | FP | FP | FP | FP | FP | FP | FP | FP | FP | |

time (sec.) | 05.16 | 02.56 | 11.78 | 08.62 | 03.60 | 19.72 | 11.28 | 03.94 | 20.72 | 01.56 | 02.50 | 06.84 | 08.19 | |

live in metro city | >5 yr | >5 yr | mths | 1–5 yr | >5 yr | 1–5 yr | weeks | >5 yr | 1–5 yr | >5 yr | mths | mths | ||

live in London | >5 yr | >5 yr | mths | 1–5 yr | 1–5 yr | mths | mths | mths | mths | mths | mths | mths |

**Table A4.**The answers by four surveyees at the University of Oxford to the questions in the London underground survey.

Surveyee’s ID | ||||||
---|---|---|---|---|---|---|

Questions | P13 | P14 | P15 | P16 | Mean | |

Q1: | answer (min.) | 15 | 20 | 15 | 15 | 16.25 |

time (sec.) | 11.81 | 18.52 | 08.18 | 07.63 | 11.52 | |

Q2: | answer (min.) | 5 | 5 | 15 | 15 | 10.00 |

time (sec.) | 11.10 | 02.46 | 13.77 | 10.94 | 09.57 | |

Q3: | answer (min.) | 35 | 60 | 30 | 25 | 37.50 |

time (sec.) | 21.91 | 16.11 | 10.08 | 22.53 | 17.66 | |

Q4: | answer (min.) | 20 | 30 | 60 | 25 | 33.75 |

time (sec.) | 13.28 | 16.21 | 08.71 | 18.87 | 14.27 | |

Q5: | time 1 (sec.) | 17.72 | 07.35 | 17.22 | 09.25 | 12.89 |

time 2 (sec.) | 21.06 | 17.00 | 19.04 | 12.37 | 17.37 | |

answer (10) | 10 | 8 | 10 | 10 | ||

time (sec.) | 04.82 | 02.45 | 02.96 | 15.57 | 06.45 | |

Q6: | time 1 (sec.) | 35.04 | 38.12 | 11.29 | 07.55 | 23.00 |

time 2 (sec.) | 45.60 | 41.32 | 20.23 | 40.12 | 36.82 | |

answer (9) | 9 | 10 | 9 | 8 | ||

time (sec.) | 03.82 | 13.57 | 08.15 | 34.32 | 14.97 | |

Q7: | time 1 (sec.) | 01.05 | 02.39 | 09.55 | 11.19 | 06.05 |

time 2 (sec.) | 02.15 | 05.45 | 09.58 | 13.47 | 07.66 | |

answer (7) | 10 | 6 | 7 | 7 | ||

time (sec.) | 01.06 | 01.60 | 02.51 | 14.06 | 04.81 | |

Q8: | time 1 (sec.) | 08.74 | 26.14 | 20.37 | 15.01 | 17.57 |

time 2 (sec.) | 16.50 | 30.55 | 27.01 | 17.91 | 22.99 | |

answer (6) | 6 | 6 | 6 | 6 | ||

time (sec.) | 09.30 | 03.00 | 02.11 | 04.94 | 04.48 | |

Q9: | answer (P) | P | P | P | P | |

time (sec.) | 05.96 | 09.38 | 04.56 | 05.16 | 06.27 | |

Q10: | answer (LB) | LB | LB | LB | LB | |

time (sec.) | 12.74 | 07.77 | 01.30 | 09.94 | 07.94 | |

Q11: | answer (WP) | WP | WP | WP | WP | |

time (sec.) | 09.84 | 04.43 | 03.39 | 07.18 | 06.21 | |

Q12: | answer (FP) | FP | FP | FP | FP | |

time (sec.) | 06.22 | 10.46 | 06.78 | 05.10 | 07.14 | |

live in metro city | never | days | days | days | ||

live in London | never | days | days | days |

**Figure A7.**The average time used by surveyees for answering each of the 12 questions. The data does not indicate any significant advantage of using the geographically-deformed map.

**Figure A9.**The PCPs of the data in Figure A8.

## References

- Klein, H.A. The Science of Measurement: A Historical Survey; Dover Publications: Mineola, NY, USA, 2012. [Google Scholar]
- Stevens, S.S. On the Theory of Scales of Measurement. Science
**1946**, 103, 677–680. [Google Scholar] [CrossRef] [PubMed] - Chen, M.; Golan, A. What May Visualization Processes Optimize? IEEE Trans. Vis. Comput. Graph.
**2016**, 22, 2619–2632. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Chen, M.; Sbert, M. A Bounded Measure for Estimating the Benefit of Visualization (Part I): Theoretical Discourse and Conceptual Evaluation. Entropy
**2022**, 24, 228. [Google Scholar] [CrossRef] - Chen, M. A Short Introduction to Information-Theoretic Cost-Benefit Analysis. arXiv
**2021**, arXiv:2103.15113. [Google Scholar] - Boslaugh, S.; Watters, P.A. Statistics in a Nutshell: A Desktop Quick Reference Paperback; O’Reilly: Akron, OH, USA, 2008. [Google Scholar]
- Pedhazur, E.J.; Schmelkin, L.P. Measurement, Design, and Analysis: An Integrated Approach; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 1991. [Google Scholar]
- Tal, E. Measurement in Science. In The Stanford Encyclopedia of Philosophy; Zalta, E.N., Ed.; Stanford University: Palo Alto, CA, USA, 2020; ISSN 1095-5054. [Google Scholar]
- Schlaudt, O. Measurement. Online Encycl. Philos. Nat.
**2020**, 5, 11. [Google Scholar] [CrossRef] - Newton, I. Scala graduum caloris. Philos. Trans.
**1701**, 22, 824–829. [Google Scholar] - Grigull, U. Newton’s temperature scale and the law of cooling. Wärme Stoffuübertrag
**1984**, 18, 195–199. [Google Scholar] [CrossRef] - Wikipedia. Newton Scale. Available online: https://en.wikipedia.org/wiki/Newton_scale (accessed on 20 November 2021).
- Behrisch, M.; Blumenschein, M.; Kim, N.W.; Shao, L.; El-Assady, M.; Fuchs, J.; Seebacher, D.; Diehl, A.; Brandes, U.; Pfister, H.; et al. Quality Metrics for Information Visualization. Comput. Graph. Forum
**2018**, 37, 625–662. [Google Scholar] [CrossRef] - Bertini, E.; Tatu, A.; Keim, D. Quality Metrics in High-Dimensional Data Visualization: An Overview and Systematization. IEEE Trans. Vis. Comput. Graph.
**2011**, 17, 2203–2212. [Google Scholar] [CrossRef] - Tufte, E.R. The Visual Display of Quantitative Information; Graphics Press: Cheshire, CT, USA, 1986. [Google Scholar]
- Cui, Q.; Yang, J.; Ward, M.; Rundensteiner, E. Measuring Data Abstraction Quality in Multiresolution Visualizations. IEEE Trans. Vis. Comput. Graph.
**2006**, 12, 709–716. [Google Scholar] [CrossRef] - Johansson, J.; Cooper, M. A Screen Space Quality Method for Data Abstraction. Comput. Graph. Forum
**2008**, 27, 1039–1046. [Google Scholar] [CrossRef] - Friedman, J.H.; Tukey, J.W. A Projection Pursuit Algorithm for Exploratory Data Analysis. IEEE Trans. Comput.
**1974**, 23, 881–890. [Google Scholar] [CrossRef] - Tukey, J.W.; Tukey, P. Computer Graphics and Exploratory Data Analysis: An Introduction. In Proceedings of the 6th Annual Conference and Exposition: Computer Graphics, Dallas, TX, USA, 14–18 April 1985; pp. 773–785. [Google Scholar]
- Bertini, E.; Santucci, G. Quality Metrics for 2D Scatterplot Graphics: Automatically Reducing Visual Clutter. In International Symposium on Smart Graphics; Butz, A., Krüger, A., Olivier, P., Eds.; Springer: Berlin/Heidelberg, Germany, 2004; Volume 3031, pp. 77–89. [Google Scholar]
- Wilkinson, L.; Anand, A.; Grossman, R. Graph-theoretic Scagnostics. In Information Visualization, IEEE Symposium; IEEE Computer Society: Washington, DC, USA, 2005; pp. 157–164. [Google Scholar]
- Sips, M.; Neubert, B.; Lewis, J.P.; Hanrahan, P. Selecting Good Views of High-dimensional Data using Class Consistency. Comput. Graph. Forum
**2009**, 28, 831–838. [Google Scholar] [CrossRef] - Tatu, A.; Albuquerque, G.; Eisemann, M.; Schneidewind, J.; Theisel, H.; Magnor, M.; Keim, D. Combining Automated Analysis and Visualization Techniques for Effective Exploration of High-dimensional Data. In Proceedings of the IEEE Symposium on Visual Analytics Science and Technology, Atlantic City, NJ, USA, 2–13 October 2009; pp. 59–66. [Google Scholar]
- Tatu, A.; Bak, P.; Bertini, E.; Keim, D.; Schneidewind, J. Visual Quality Metrics and Human Perception: An Initial Study on 2D Projections of Large Multidimensional Data. In Proceedings of the International Conference on Advanced Visual Interfaces, Bari, Italy, 7–10 June 2010; pp. 49–56. [Google Scholar]
- Dasgupta, A.; Kosara, R. Pargnostics: Screen-Space Metrics for Parallel Coordinates. IEEE Trans. Vis. Comput. Graph.
**2010**, 16, 1017–1026. [Google Scholar] [CrossRef] - Peng, W.; Ward, M.O.; Rundensteiner, E. Clutter Reduction in Multi-dimensional Data Visualization using Dimension Reordering. In Proceedings of the IEEE Symposium on Information Visualization, Austin, TX, USA, 10 October 2004; pp. 89–96. [Google Scholar]
- Rosenholtz, R.; Li, Y.; Nakano, L. Measuring Visual Clutter. J. Vis.
**2007**, 7, 1–22. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Yu, C.P.; Samaras, D.; Zelinsky, G.J. Modelling visual clutter perception using proto-object segmentation. J. Vis.
**2014**, 14, 4. [Google Scholar] [CrossRef] - Filonik, D.; Baur, D. Measuring Aesthetics for Information Visualization. In Proceedings of the IEEE Information Visualization, Atlantic City, NJ, USA, 11–16 October 2009; pp. 579–584. [Google Scholar]
- Jänicke, H.; Chen, M. A Salience-based Quality Metric for Visualization. Comput. Graph. Forum
**2010**, 29, 1183–1192. [Google Scholar] [CrossRef] - Bernard, J.; Steiger, M.; Mittelstädt, S.; Thum, S.; Keim, D.A.; Kohlhammer, J. A survey and task-based quality assessment of static 2D color maps. In Visualization and Data Analysis; International Society for Optics and Photonics: Bellingham, WA, USA, 2015. [Google Scholar]
- Mittelstädt, S.; Jäckle, D.; Stoffel, F.; Keim, D.A. ColorCAT: Guided Design of Colormaps for Combined Analysis Tasks. In Proceedings of the Conference on Visualization (EuroVis), Sardinia, Italy, 25–29 May 2015; pp. 115–119. [Google Scholar]
- Mittelstädt, S.; Keim, D.A. Efficient Contrast Effect Compensation with Personalized Perception Models. Comput. Graph. Forum
**2015**, 34, 211–220. [Google Scholar] [CrossRef] [Green Version] - Gramazio, C.C.; Laidlaw, D.H.; Schloss, K.B. Colorgorical: Creating discriminable and preferable color palettes for information visualization. IEEE Trans. Vis. Comput. Graph.
**2017**, 23, 521–530. [Google Scholar] [CrossRef] - Cleveland, W.S.; McGill, R. Graphical perception: Theory, experimentation, and application to the development of graphical methods. J. Am. Stat. Assoc.
**1984**, 79, 531–554. [Google Scholar] [CrossRef] - Isenberg, T. Evaluating and Validating Non-photorealistic and Illustrative Rendering. In Image and Video-Based Artistic Stylisation; Rosin, P., Collomosse, J., Eds.; Springer: London, UK, 2013; pp. 311–331. [Google Scholar]
- Isenberg, T.; Neumann, P.; Carpendale, S.; Sousa, M.C.; Jorge, J.A. Non-Photorealistic Rendering in Context: An Observational Study. In Proceedings of the 4th International Symposium on Non-Photorealistic Animation and Rendering, Annecy, France, 5–7 June 2006; pp. 115–126. [Google Scholar]
- Cole, F.; Sanik, K.; DeCarlo, D.; Finkelstein, A.; Funkhouser, T.; Rusinkiewicz, S.; Singh, M. How Well Do Line Drawings Depict Shape? In Proceedings of the ACM SIGGRAPH Conference, New Orleans, LA, USA, 3–7 August 2009; p. 28. [Google Scholar]
- Mandryk, R.L.; Mould, D.; Li, H. Evaluation of Emotional Response to Non-Photorealistic Images. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Non-Photorealistic Animation and Rendering, Vancouver, BC, Canada, 5–7 August 2011; pp. 7–16. [Google Scholar]
- Liu, Z.; Li, Z. Impact of schematic designs on the cognition of underground tube maps. ISPRS—Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2016**, 49, 421–423. [Google Scholar] [CrossRef] [Green Version] - Hong, S.; Yoo, M.J.; Chinh, B.; Han, A.; Battersby, S.; Kim, J. To Distort or Not to Distort: Distance Cartograms in the Wild. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21–26 April 2018; pp. 1–12. [Google Scholar]
- Tam, G.K.L.; Kothari, V.; Chen, M. An analysis of machine- and human-analytics in classification. IEEE Trans. Vis. Comput. Graph.
**2017**, 23, 71–80. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Kijmongkolchai, N.; Abdul-Rahman, A.; Chen, M. Empirically measuring soft knowledge in visualization. Comput. Graph. Forum
**2017**, 36, 73–85. [Google Scholar] [CrossRef] - Wall, E.; Agnihotri, M.; Matzen, L.; Divis, K.; Haass, M.; Endert, A.; Stasko, J. A heuristic approach to value-driven evaluation of visualizations. IEEE Trans. Vis. Comput. Graph.
**2018**, 25, 491–500. [Google Scholar] [CrossRef] [PubMed] - Saket, B.; Srinivasan, A.; Ragan, E.D.; Endert, A. Evaluating interactive graphical encodings for data visualization. IEEE Trans. Vis. Comput. Graph.
**2017**, 24, 1316–1330. [Google Scholar] [CrossRef] - Sacha, D.; Stoffel, A.; Stoffel, F.; Kwon, B.C.; Ellis, G.; Keim, D.A. Knowledge generation model for visual analytics. IEEE Trans. Vis. Comput. Graph.
**2014**, 20, 1604–1613. [Google Scholar] [CrossRef] - Chen, M.; Ebert, D.S. An ontological framework for supporting the design and evaluation of visual analytics systems. Comput. Graph. Forum
**2019**, 38, 131–144. [Google Scholar] [CrossRef] - Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J.
**1948**, 27, 379–423. [Google Scholar] [CrossRef] [Green Version] - Cover, T.M.; Thomas, J.A. Elements of Information Theory; John Wiley & Sons: Hoboken, NJ, USA, 2006. [Google Scholar]
- Chen, M.; Gaither, K.; John, N.W.; McCann, B. Cost–benefit analysis of visualization in virtual environments. IEEE Trans. Vis. Comput. Graph.
**2019**, 25, 32–42. [Google Scholar] [CrossRef] [Green Version] - Ishizaka, A.; Nemery, P. Multi-Criteria Decision Analysis: Methods and Software; John Wiley & Sons: Hoboken, NJ, USA, 2013. [Google Scholar]
- Lin, J. Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theory
**1991**, 37, 145–151. [Google Scholar] [CrossRef] [Green Version] - Roettger, S. The Volume Library. Available online: http://schorsch.efi.fh-nuernberg.de/data/volume/ (accessed on 20 November 2021).
- Nagy, Z.; Schneide, J.; Westerman, R. Interactive Volume Illustration. In Proceedings of the Vision, Modeling and Visualization, Erlangen, Germany, 20–22 November 2002. [Google Scholar]
- Correa, C.; Silver, D.; Chen, M. Feature aligned volume manipulation for illustration and visualization. IEEE Trans. Vis. Comput. Graph.
**2006**, 12, 1069–1076. [Google Scholar] [CrossRef] - Wu, Y.; Qu, H. Interactive Transfer Function Design Based on Editing Direct Volume Rendered Images. IEEE Trans. Vis. Comput. Graph.
**2007**, 13, 1027–1040. [Google Scholar] [CrossRef] [PubMed] - Jung, Y. Instantreality 1.0. Available online: https://doc.instantreality.org/tutorial/volume-rendering/ (accessed on 20 November 2021).
- Max, N.; Chen, M. Local and global illumination in the volume rendering integral. In Scientific Visualization: Advanced Concepts; Hagen, H., Ed.; Schloss Dagstuhl: Wadern, Germany, 2010. [Google Scholar]
- Chen, M. Cost-Benefit Analysis of Data Intelligence—Its Broader Interpretations. In Advances in Info-Metrics: Information and Information Processing across Disciplines; Chen, M., Dunn, J.M., Golan, A., Ullah, A., Eds.; Oxford University Press: Oxford, UK, 2020. [Google Scholar]
- Streeb, D.; El-Assady, M.; Keim, D.; Chen, M. Why visualize? Untangling a large network of arguments. IEEE Trans. Vis. Comput. Graph.
**2019**, 27, 2220–2236. [Google Scholar] [CrossRef] [PubMed] - Chen, M.; Edwards, D.J. ‘Isms’ in Visualization. In Foundations of Data Visualization; Chen, M., Hauser, H., Rheingans, P., Scheuermann, G., Eds.; Springer: Berlin/Heidelberg, Germany, 2020. [Google Scholar]
- Dasgupta, A.; Chen, M.; Kosara, R. Conceptualizing visual uncertainty in parallel coordinates. Comput. Graph. Forum
**2012**, 31, 1015–1024. [Google Scholar] [CrossRef] [Green Version] - Viola, I.; Chen, M.; Isenberg, T. Visual Abstraction. In Foundations of Data Visualization; Chen, M., Hauser, H., Rheingans, P., Scheuermann, G., Eds.; Springer: Berlin/Heidelberg, Germany, 2020. [Google Scholar]
- Kindlmann, G.; Scheidegger, C. An Algebraic Process for Visualization Design. IEEE Trans. Vis. Comput. Graph.
**2014**, 20, 2181–2190. [Google Scholar] [CrossRef] [PubMed] - Chen, M. The value of interaction in data intelligence. arXiv
**2018**, arXiv:1812.06051. [Google Scholar] - Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat.
**1951**, 22, 79–86. [Google Scholar] [CrossRef] - Chen, M.; Trefethen, A.; Banares-Alcantara, R.; Jirotka, M.; Coecke, B.; Ertl, T.; Schmidt, A. From data analysis and visualization to causality discovery. IEEE Comput.
**2011**, 44, 84–87. [Google Scholar] [CrossRef]

**Figure 1.**The London underground map (

**right**) is a deformed map. In comparison with a relatively more faithful map

**(left**), there is a significant amount of information loss due to many-to-one mappings in the deformed map, which omits some detailed variations among different connection routes between pairs of stations (e.g., distance and geometry). One common rationale is that the deformed map was designed for certain visualization tasks, which likely excluded the task for estimating the walking time between a pair of stations indicated by a pair of red or blue arrows. In one of our experiments, when asked to perform such tasks using the deformed map, some participants with little knowledge about London or London Underground performed these tasks well. Can information theory explain this phenomenon? Can we quantitatively measure relevant factors in this visualization process?

**Figure 2.**Major temperate scales proposed in history. Different lines show instances used as observation points, some of which became major reference points. Note: “Celsius* 1742” indicates the original scale proposed by Anders Celsius, while “Celsius 1743” indicates the revised Celsius scale used today that was proposed by Jean-Pierre Christin. The Newton scale is not linearly related to the others (shown as dash lines).

**Figure 3.**Three alphabets illustrate possible metro maps (letters) in different grid resolutions. Increasing the resolution enables the depiction of more reality, while reducing the resolution compels more abstraction.

**Figure 4.**An example scenario with two states good and bad has a ground truth PMF $Q=\{0.8,0.2\}$. From the output of a biased process that always informs users that the situation is bad. Five users, LD, DF, RG, UC, and OC, have different knowledge and thus different divergence. The five candidate measures return different values of divergence. We would like to see which sets of values are more intuitive. The illustration on the top-right shows two transformations of the alphabets and their PMFs, one by the misleading communication and the other by the reconstruction. The bar chart shows the divergence values calculated by each candidate measure, while the four parallel coordinate plots (PCPs) show the values of ${\mathcal{H}}_{max}\mathcal{D}$ (divergence scaled by $S{E}_{max}$), benefit, ${K}_{\upsilon}$ (impact of knowledge against relying solely on visual information), and ${K}_{\psi}$ (against random guess).

**Figure 5.**An example scenario with four data values: A, B, C, and D. Two processes (one correct and one biased) aggregated them to two values AB and CD. Users CG, CU, CB attempt to reconstruct [A, B, C, D] from the output [AB, CD] of the correct process, while BG, BS, and BM attempt to do so with the output from the biased processes. The bar chart shows the divergence values of the six users computed using the five candidate measures. The illustration on the right shows two transformations of the alphabets and their PMFs, one by the correct or biased process (pr.) and the other by the reconstruction. The bar chart shows the divergence values calculated by each candidate measure, while the four PCPs show the values of ${\mathcal{H}}_{max}\mathcal{D}$ (i.e., divergence scaled by $S{E}_{max}$), benefit, ${K}_{\upsilon}$ and ${K}_{\psi}$. The values for Cvi and Bvi correspond to ${R}_{\mathrm{correct}}$ and ${R}_{\mathrm{biased}}$, respectively.

**Figure 6.**A volume dataset was rendered using the maximum intensity projection (MIP) method, which causes curved surfaces of arteries to appear rather flat. Posing a question about a “flat area” in the image can be used to tease out a viewer’s knowledge that is useful in a visualization process. This example was first described in Part I of this two-part paper [4] for demonstrating the role of human knowledge in dealing with information loss due to many-to-one mappings in such a visualization image. Similar to Figure 3 (Section 3) in this part, the example was used in Part I to illustrate the difficulty to interpret the unboundedness of the KL-divergence when considering a binary alphabet $\mathbb{A}=\{\mathit{curved},\mathit{flat}\}$ with maximum entropy of 1 bit.

**Figure 7.**For the survey question shown in Figure 6, our survey of 10 participants returned 8 answers for A, 1 for B, 0 for C, and 1 for D. Among them, more knowledgeable participants (referred to as experts) returned 3 answers for A and 1 for B, and none for C or D. We consider two possible ground truth PMFs. ${Q}_{1}=\{0.1,0.878,0.002,0.02\}$ is based on our observations of photographs of arteries, and ${Q}_{2}=\{0.75,0.25,0.0,0.0\}$ is based on the experts’ survey results. The top four PCPs show the values of ${\mathcal{H}}_{max}\mathcal{D}$, benefit, ${K}_{\upsilon}$, and ${K}_{\psi}$ calculated based on ${Q}_{1}$, while the bottom four PCPs are measured based on ${Q}_{2}$. In addition, we also consider five other groups that make a random guess or always answer A, B, C, or D.

**Table 1.**A summary of the multi-criteria decision analysis (MCDA). Each measure is scored against a criterion using an integer in [0, 5] with 5 being the best. Scores are calculated as: starting with a full score of 5. For each “good” deduct 1, each “adequate” deduct 2, and each “inadequate” deduct 3. The top table summarize the empirical scores obtained from the two synthetic case studies (${\mathbf{S}}_{\mathbf{1}}$ and ${\mathbf{S}}_{\mathbf{2}}$) in Section 5 and two experimental case studies (${\mathbf{R}}_{\mathbf{1}}$ and ${\mathbf{R}}_{\mathbf{2}}$) in Section 6. The bottom table presents the final results of MCDA by combining the subtotals of the seven conceptual criteria in the first part of the paper, the subtotals of the empirical criteria in this second part of the paper, and the scores of the extra conceptual criterion discussed in Section 5.3.

A Summary of the Empirical Scores Obtained of the Four Case Studies | |||||||
---|---|---|---|---|---|---|---|

Criteria | ${\mathcal{D}}_{\mathrm{JS}}$ | $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$ | ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ | ${\mathcal{D}}_{\mathrm{new}}^{k=2}$ | ${\mathcal{D}}_{\mathrm{ncm}}^{k=1}$ | ${\mathcal{D}}_{\mathrm{ncm}}^{k=2}$ | |

S_{1}: | order | adequate | adequate | excellent | excellent | good | good |

benefit | adequate | good | excellent | good | excellent | good | |

knowledge | excellent | excellent | excellent | excellent | excellent | excellent | |

score | 1 | 2 | 5 | 4 | 4 | 3 | |

${\mathbf{S}}_{\mathbf{2}}:$ | order | excellent | excellent | good | excellent | adequate | good |

benefit | adequate | excellent | good | adequate | good | adequate | |

knowledge | excellent | excellent | excellent | excellent | excellent | excellent | |

score | 3 | 5 | 3 | 3 | 2 | 2 | |

${\mathbf{R}}_{\mathbf{1}}:$ | order | excellent | excellent | good | excellent | inadequate | good |

benefit | excellent | excellent | excellent | excellent | excellent | excellent | |

knowledge | excellent | excellent | good | good | adequate | good | |

score | 5 | 5 | 3 | 4 | 0 | 3 | |

${\mathbf{R}}_{\mathbf{2}}:$ | order | excellent | excellent | excellent | excellent | adequate | excellent |

benefit | adequate | adequate | adequate | excellent | adequate | adequate | |

knowledge | excellent | excellent | excellent | excellent | excellent | excellent | |

score | 3 | 3 | 3 | 5 | 1 | 3 | |

Empirical Subtotal: | 12 | 15 | 14 | 16 | 7 | 11 | |

Combining All Scores Obtained from the Conceptual and Empirical Evaluation | |||||||

Criteria | ${\mathcal{D}}_{\mathrm{JS}}$ | $\sqrt{{\mathcal{D}}_{\mathrm{JS}}}$ | ${\mathcal{D}}_{\mathrm{new}}^{k=1}$ | ${\mathcal{D}}_{\mathrm{new}}^{k=2}$ | ${\mathcal{D}}_{\mathrm{ncm}}^{k=1}$ | ${\mathcal{D}}_{\mathrm{ncm}}^{k=2}$ | |

Conceptual Subtotal [4]: | 30 | 30 | 28 | 30 | 26 | 29 | |

Empirical Subtotal: | 12 | 15 | 14 | 16 | 7 | 11 | |

Componentization (extra criterion): | 5 | 1 | 5 | 5 | 5 | 5 | |

Total without the extra criterion: | 42 | 45 | 42 | 46 | 33 | 40 | |

Total with the extra criterion: | 47 | 46 | 47 | 51 | 38 | 45 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Chen, M.; Abdul-Rahman, A.; Silver, D.; Sbert, M.
A Bounded Measure for Estimating the Benefit of Visualization (Part II): Case Studies and Empirical Evaluation. *Entropy* **2022**, *24*, 282.
https://doi.org/10.3390/e24020282

**AMA Style**

Chen M, Abdul-Rahman A, Silver D, Sbert M.
A Bounded Measure for Estimating the Benefit of Visualization (Part II): Case Studies and Empirical Evaluation. *Entropy*. 2022; 24(2):282.
https://doi.org/10.3390/e24020282

**Chicago/Turabian Style**

Chen, Min, Alfie Abdul-Rahman, Deborah Silver, and Mateu Sbert.
2022. "A Bounded Measure for Estimating the Benefit of Visualization (Part II): Case Studies and Empirical Evaluation" *Entropy* 24, no. 2: 282.
https://doi.org/10.3390/e24020282