# Evaluating Museum Virtual Tours: The Case Study of Italy

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

- Natural engagement: how close the interaction is to the real world (Ne).
- Compatibility with the user’s task and the domain: how close the behavior of objects is to the real world and affordance for task action (C).
- The natural expression of action: does the system allow the user to act naturally? (Na).
- Close coordination of action and representation: quality of the response between user movement and Virtual Environment (Cr).
- Realistic feedback: visibility of the effect of users’ actions and conformity to the laws of physics (Rf).
- Faithful viewpoints: the naturalness of change between viewpoints (Fv).
- Navigation and orientation support: naturalness in orientation and navigation. Is it clear where they are and how they return? (No).
- Clear entry and exit points: clearness of entry and exit points (Ce).
- Consistent departures: consistency of departure actions (Cd).
- Support for learning: promotion of learning (L).
- Clear turn-taking: clearness of who has the initiative (Tt).
- Sense of presence: the naturalness of the user’s perception of engagement in the system and being in a ‘real’ world (P).

#### 2.1. AHP

- Form the set of evaluators: for the estimation of the weights of the heuristics, it is important to have the view of experts on the field. Therefore, the set of evaluators is only composed of human experts. The selection of expert-based evaluations has many advantages [5] and the correct choice of experts gives more reliable and valid results.
- Setting up a pairwise comparison matrix of heuristics: in this step, a comparison matrix is formed so that the heuristics are pairwise compared.
- Calculating the weights of criteria: after making pairwise comparisons, estimations are made that result in the final set of weights of the criteria.

#### 2.2. Fuzzy TOPSIS

- Forming a new set of evaluators: in this phase of the evaluation experiment, the set of evaluators was formed, following the taxonomy of types of users of cultural websites proposed by Sweetnam et al. [23]. This group may be the same as the one formed in step 1 of the application of AHP or may be different.
- Assigning values to the criteria: in order to make this process easier for the user, especially for those that do not have experience in multi-criteria analysis, the users could use linguistic terms for characterizing the twelve heuristics presented above. The linguistic terms are presented in Table 1.
- Construction of the Multi-Criteria Decision Making (MCDM) matrix: a fuzzy multi-criteria group decision-making problem can be expressed in matrix format. Each element of the matrix is a fuzzy number. However, in order to aggregate all the values of the decision-makers in one single value, the geometric mean is used. The geometric mean of two fuzzy numbers $\tilde{a}=({a}_{1},{a}_{2},{a}_{3})$ and $\tilde{b}=({b}_{1},{b}_{2},{b}_{3})$ is calculated as follows:$$\tilde{c}=(\sqrt{{a}_{1}{b}_{1}},\sqrt{{a}_{2}{b}_{2}},\sqrt{{a}_{3}{b}_{3}})$$$$\tilde{D}=\begin{array}{cc}& \begin{array}{cccc}{C}_{1}& {C}_{2}& & {C}_{n}\end{array}\\ \begin{array}{c}{A}_{1}\\ {A}_{2}\\ \\ {A}_{m}\end{array}& \left(\begin{array}{cccc}{\tilde{x}}_{11}& {\tilde{x}}_{12}& & {\tilde{x}}_{1n}\\ {\tilde{x}}_{21}& {\tilde{x}}_{22}& & {\tilde{x}}_{2n}\\ & & {\tilde{x}}_{ij}& \\ {\tilde{x}}_{m1}& {\tilde{x}}_{m2}& & {\tilde{x}}_{mn}\end{array}\right)\end{array},\text{}i=1,2,\dots ,m;j=1,2,\dots ,n,\text{}{\tilde{x}}_{ij}=({a}_{ij},{b}_{ij},{c}_{ij})$$
- The normalization of fuzzy numbers: to avoid the complicated normalization formula used in classical TOPSIS, the authors in [16] propose a linear scale transformation in order to transform the various criteria scales into a comparable scale. This particular normalization method aims at preserving the property that the ranges of normalized triangular fuzzy numbers belong to [0,1]. The normalization of a fuzzy number ${\tilde{x}}_{ij}=({a}_{ij},{b}_{ij},{c}_{ij})$ is given by the formula:$${\tilde{r}}_{ij}=(\frac{{a}_{ij}}{{c}_{j}^{*}},\frac{{b}_{ij}}{{c}_{j}^{*}},\frac{{c}_{ij}}{{c}_{j}^{*}}),\text{}\mathrm{where}\text{}{c}_{j}^{*}=\underset{i}{\mathrm{max}}{c}_{ij}^{}$$
- Calculating the weighted normalized fuzzy numbers of the MCDM matrix: considering the different importance of each criterion, which is imprinted in the weights of the criteria, the weighted normalized fuzzy numbers are calculated: ${\tilde{u}}_{ij}={\tilde{r}}_{ij}\left(\xb7\right){\tilde{w}}_{j}$ and these values are used to construct the weighted normalized fuzzy MCDM matrix $\tilde{V}={[{\tilde{u}}_{ij}]}_{M\times N},i=1,2,\dots ,m;j=1,2,\dots ,n$.
- Determination of the fuzzy positive-ideal solution (FPIS) and the fuzzy negative-ideal solution (FNIS): the FPIS and the FNIS are calculated as follows.$$\mathrm{FPIS}:\text{}{A}^{*}=\{{\tilde{u}}_{1}^{*},{\tilde{u}}_{2}^{*},\dots ,{\tilde{u}}_{i}^{*},\dots ,{\tilde{u}}_{n}^{*}\},\text{}{\tilde{u}}_{j}^{*}=(1,1,1)$$$$\mathrm{FNIS}:\text{}{A}^{-}=\{{\tilde{u}}_{1}^{-},{\tilde{u}}_{2}^{-},\dots ,{\tilde{u}}_{i}^{-},\dots ,{\tilde{u}}_{n}^{-}\},\text{}{\tilde{u}}_{j}^{-}=(0,0,0)$$
- Calculation of the distance of each alternative from FPIS and FNIS: the distances (${d}_{i}^{*}$ and ${d}_{i}^{-}$) of each weighted alternative (${d}_{i}^{*}$ and ${d}_{i}^{-}$) from FPIS and FNIS are calculated as follows.$${d}_{i}^{*}={\displaystyle \sum _{j=1}^{n}{d}_{u}({\tilde{u}}_{ij},{\tilde{u}}_{j}^{*}}),\text{}i=1,2,\dots ,m$$$${d}_{i}^{-}={\displaystyle \sum _{j=1}^{n}{d}_{u}({\tilde{u}}_{ij},{\tilde{u}}_{j}^{-}}),\text{}i=1,2,\dots ,m$$$$d(\tilde{\alpha},\tilde{b})=\sqrt{\frac{1}{3}[{({\alpha}_{1}-{b}_{1})}^{2}+{({a}_{2}-{b}_{2})}^{2}+{({a}_{3}-{b}_{3})}^{2}]}$$
- Calculation of the closeness coefficient of each alternative: the closeness coefficient of each alternative j is given by the formula $C{C}_{i}=\frac{{d}_{i}^{-}}{{d}_{i}^{*}+{d}_{i}^{-}}$, $0\le C{C}_{i}^{}\le 1$. According to the values of the closeness coefficient, the ranking order of all the alternatives is determined. The alternative that is closer to FPIS and further from FNIS as $C{C}_{i}^{}$ approaches 1. The values of the closeness coefficient of each alternative and the final ranking of the evaluated websites are presented in Table 2.

## 3. Implementation of the Experiment

- MAO Museum of Oriental Art (Maotorino)
- Vatican Museums (Museivaticani)
- Jewish Museum in Bologna (Ebraico)
- AraPacis Museum (dell’AraPacis)
- Napoleonic Museum (Napoleonico)
- Capitoline Museums (Capitolini)
- Imperial Forum Museum (Mercati di Traiano)
- Villa Torlonia Museums (Villatorlonia)
- Chimney Sweep Museum (Spazzacamino)
- Zoology Museum “P. Doderlein” (Zoologia)
- Geological MuseumGemmellaro (Geologia)
- Historical Museum of Engines and Mechanisms (Motori)
- Toy Horse Museum (Cavallogiocattolo)
- Ca’Rezzonico—Eighteenth-Century Venetian Museum (Carezzonico)
- Langhe Museum(Castellogrinzane)
- Etna Museum (Museodelletna)

## 4. Results

## 5. Analysis of the Results and Discussion

## 6. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Walczak, K.; Cellary, W.; White, M. Virtual museum exhibitions. Computers
**2006**, 39, 93–95. [Google Scholar] [CrossRef] - Barbieri, L.; Bruno, F.; Muzzupappa, M. Virtual museum systems evaluation through user studies. J. Cult. Herit.
**2017**, 26, 101–108. [Google Scholar] [CrossRef] - Latos, A.; Komianos, V.; Oikonomou, K. Interaction and Information Communication in Virtual Museums. In IOP Conference Series: Materials Science and Engineering; IOP Publishing: Bristol, UK, 2018; Volume 364, p. 012038. [Google Scholar]
- Komianos, V.; Kavvadia, E.; Oikonomou, K. Cultural heritage recommendations and user navigation in large scale virtual environments. IJCIStudies
**2015**, 4, 151–172. [Google Scholar] [CrossRef] - Karoulis, A.; Sylaiou, S.; White, M. Usability Evaluation of a Virtual Museum Interface. Informatica
**2006**, 17, 363–380. [Google Scholar] - Othman, M.K.; Petrie, H.; Power, C. Measuring the usability of a smartphone delivered museum guide. The 9th International Conference on Cognitive Science. Procedia Soc. Behav. Sci.
**2013**, 97, 629–637. [Google Scholar] [CrossRef] - Hix, D.; Gabbard, J.L. Usability engineering of virtual environments. In Handbook of Virtual Environments: Design, Implementation and Applications; Stanney, K., Ed.; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 2002; pp. 681–699. [Google Scholar]
- Stanney, K.M.; Mollaghasemi, M.; Reeves, L.; Breaux, R.; Graeber, D.A. Usability engineering of virtual environments (VEs): Identifying multiple criteria that drive effective VE system design. Intell. J. Hum.-Comput. Stud.
**2003**, 58, 447–481. [Google Scholar] [CrossRef] - Bazargan, K.; Falquet, G.; Metral, C. Testing the usability of information visualization techniques in interactive 3D virtual environments: A toolkit. In Proceedings of the IADIS International Conference Interfaces and Human Computer Interaction, Freiburg, Germany, 28–30 July 2010. [Google Scholar]
- Nielsen, J. Usability Engineering; Academic Press: New York, NY, USA, 1993. [Google Scholar]
- Sutcliffe, A.; Gault, B. Heuristic evaluation of virtual reality applications. Interact. Comput.
**2004**, 16, 831–849. [Google Scholar] [CrossRef] - Sylaiou, S.; Killintzis, V.; Paliokas, I.; Mania, K.; Patias, P. Usability evaluation of virtual museums’ interfaces visualization technologies. In Proceedings of the International Conference on Virtual, Augmented and Mixed Reality, Crete, Greece, 22–27 June 2014; pp. 124–133. [Google Scholar]
- Kabassi, K.; Maravelakis, E.; Konstantaras, A. Heuristics and Fuzzy Multi-criteria Decision Making for Evaluating Museum Virtual Tours. Int. J. Incl. Mus.
**2018**, 11, 1–21. [Google Scholar] [CrossRef] - Chou, S.-Y.; Chang, Y.-H.; Shen, C.-Y. A fuzzy simple additive weighting system under group decision-making for facility location selection with objective/subjective attributes. Eur. J. Oper. Res.
**2008**, 189, 132–145. [Google Scholar] [CrossRef] - Saaty, T. The Analytic Hierarchy Process; McGraw-Hill: New York, NY, USA, 1980. [Google Scholar]
- Chen, C.T. Extensions of the TOPSIS for group decision-making under fuzzy environment. Fuzzy Sets Syst.
**2000**, 114, 1–9. [Google Scholar] [CrossRef] - Mulubrhan, F.; Akmar Mokhtar, A.; Muhammad, M. Comparative Analysis between Fuzzy and Traditional Analytical Hierarchy Process. MATEC Web Conf.
**2014**, 13, 01006. [Google Scholar] [CrossRef] - Hwang, C.L.; Yoon, K. Multiple Attribute Decision Making: Methods and Applications; Springer: New York, NY, USA, 1981. [Google Scholar]
- Soleymaninejad, M.; Shadifar, M.; Karimi, A. Evaluation of Two Major Online Travel Agencies of US Using TOPSIS Method. Digit. Technol.
**2016**, 2, 1–8. [Google Scholar] - Zhang, W. Group-Buying Websites Evaluation Model Based on AHP-TOPSIS under the Environment of Multi-Attribute Decision-Making. Int. J. Multimed. Ubiquitous Eng.
**2015**, 10, 31–40. [Google Scholar] [CrossRef] - Nagpal, R.; Mehrotra, D.; Kumar Bhatia, P.; Sharma, A. Rank University Websites Using Fuzzy AHP and Fuzzy TOPSIS Approach on Usability. Int. J. Inf. Eng. Electron. Bus.
**2015**, 1, 29–36. [Google Scholar] [CrossRef] - Büyüközkan, D.; Ruan, D. Evaluating government websites based on a fuzzy multiple criteria decision-making approach. Int. J. Uncertain. Fuzziness Knowl.-Based Syst.
**2007**, 15, 321–343. [Google Scholar] [CrossRef] - Sweetnam, M.S.; Agosti, M.; Orio, N.; Ponchia, C.; Steiner, C.M.; Hillemann, E.-C.; Siochrú, M.; Lawless, S. User needs for enhanced engagement with cultural heritage collections. Proceedings of Second International Conference TPDL, Paphos, Cyprus, 23–27 September 2012; pp. 64–75. [Google Scholar]
- Sirah, S.; Mikhailov, L.; Keane, J.A. PriEsT: An interactive decision support tool to estimate priorities from pair-wise comparison judgments. Int. Trans. Oper. Res.
**2015**, 22, 203–382. [Google Scholar] - Sylaiou, S.; Liarokapis, F.; Kotsakis, K.; Patias, P. Virtual museums, a survey and some issues for consideration. J. Cult. Herit.
**2009**, 10, 520–528. [Google Scholar]

Linguistic Term | Fuzzy Number |
---|---|

Very Poor | (0,0,1) |

Poor | (0,1,3) |

Fair | (3,5,7) |

Good | (7,9,10) |

Very Good | (9,10,10) |

**Table 2.**${d}_{i}^{*}$, ${d}_{i}^{-}$ the closeness coefficient (CC) and the ranking of each alternative virtual museum.

d* | d^{−} | CC | Ranking | |
---|---|---|---|---|

Maotorino | 11.78306 | 0.288359 | 0.023888 | 15 |

Museivaticani | 11.61684 | 0.457118 | 0.03786 | 5 |

Ebraico | 11.479 | 0.583454 | 0.048369 | 3 |

dell’AraPacis | 11.7397 | 0.328556 | 0.027225 | 14 |

Napoleonico | 11.72471 | 0.344694 | 0.028559 | 13 |

Capitolini | 11.70928 | 0.366162 | 0.030323 | 11 |

MercatidiTraiano | 11.67833 | 0.399702 | 0.033093 | 7 |

Villatorlonia | 11.70072 | 0.378629 | 0.031345 | 9 |

Spazzacamino | 11.71159 | 0.363352 | 0.030091 | 12 |

Zoologia | 11.87914 | 0.177064 | 0.014687 | 16 |

Geologia | 11.70604 | 0.375118 | 0.03105 | 10 |

Motori | 11.54909 | 0.52448 | 0.04344 | 4 |

Cavallogiocattolo | 11.45735 | 0.594157 | 0.049301 | 2 |

Carezzonico | 11.30168 | 0.735233 | 0.061082 | 1 |

Castellogrinzane | 11.69436 | 0.37978 | 0.031454 | 8 |

Museodelletna | 11.62822 | 0.43481 | 0.036045 | 6 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kabassi, K.; Amelio, A.; Komianos, V.; Oikonomou, K. Evaluating Museum Virtual Tours: The Case Study of Italy. *Information* **2019**, *10*, 351.
https://doi.org/10.3390/info10110351

**AMA Style**

Kabassi K, Amelio A, Komianos V, Oikonomou K. Evaluating Museum Virtual Tours: The Case Study of Italy. *Information*. 2019; 10(11):351.
https://doi.org/10.3390/info10110351

**Chicago/Turabian Style**

Kabassi, Katerina, Alessia Amelio, Vasileios Komianos, and Konstantinos Oikonomou. 2019. "Evaluating Museum Virtual Tours: The Case Study of Italy" *Information* 10, no. 11: 351.
https://doi.org/10.3390/info10110351