Next Article in Journal
Scenario-Based Risk Assessment of Earthquake Disaster Using Slope Displacement, PGA, and Population Density in the Guyuan Region, China
Next Article in Special Issue
GeoAnnotator: A Collaborative Semi-Automatic Platform for Constructing Geo-Annotated Text Corpora
Previous Article in Journal
Accuracy Analysis of a 3D Model of Excavation, Created from Images Acquired with an Action Camera from Low Altitudes
Previous Article in Special Issue
3D Landform Modeling to Enhance Geospatial Thinking
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Studying Social Uses of 3D Geovisualizations: Lessons Learned from Action-Research Projects in the Field of Flood Mitigation Planning

University Lyon, UJM Saint-Etienne, CNRS UMR 5600 Environnement-Ville-Société, 6 rue basse des rives, 42 023 Saint-Etienne, France
University Lyon, Université Lumière Lyon 2, ELICO EA 4147, 5 av. Pierre Mendès France, 69500 Bron, France
Author to whom correspondence should be addressed.
ISPRS Int. J. Geo-Inf. 2019, 8(2), 84;
Received: 15 November 2018 / Revised: 21 January 2019 / Accepted: 11 February 2019 / Published: 14 February 2019
(This article belongs to the Special Issue Human-Centered Geovisual Analytics and Visuospatial Display Design)


Risk management seeks more and more the mobilization of all citizens, including elected representatives and inhabitants. Three-dimensional (3D) geovisualizations have been used between 2009 and 2017 in order to associate citizens to flood mitigation policies along the river Rhône. We focused our studies on the effects 3D geovisualizations can have on the communication and understanding of information and their ability to foster exchanges between heterogeneous actors as well as participation of the grand public to planning processes. Facing both discrepancies in scientific studies of the uses of 3D geovisualizations and a lack of validated theoretical elements, we resorted to an exploratory method based on grounded theory and ethnographic observation in order to produce empirical knowledge on the uses of 3D geovisualizations in collective settings, including heterogeneous actors (risk managers, elected representatives, citizens). Observation showed that 3D geovisualizations can be useful for the dissemination of information about flood risk. Many observed effects were not anticipated during the production of 3D geovisualizations. Qualitative analysis of empirical data through actor–network theory and from a communication studies perspective shed light on some factors influencing the roles of 3D geovisualizations and help put into perspective existing and sometimes contradictory scientific works on 3D geovisualizations’ uses.

1. Introduction

In this paper, we propose an analysis and a discussion of the observed roles played by three-dimensional (3D) geovisualizations in the context of flood mitigation planning. More specifically, we analyze results from a study focused on the possibilities offered by 3D geovisualizations to risk managers in order to communicate information about risk to elected representatives and citizens and to foster exchanges between heterogeneous actors in collective and collaborative settings. Analysis of the feedbacks from eight years of action-research projects sheds light on methodological approaches that are useful in order to understand social uses of 3D geovisualizations in urban planning. Our multidisciplinary approach, combining geographical information science, geography, sociology, and communication studies, allows us to provide original results, shedding light on some shortcomings of previous studies on the uses of 3D geovisualizations.
Flood mitigation planning offers a particularly interesting setting to study uses of 3D geovisualizations as a mean to communicate technical (in this case hydraulic) data to a large and heterogeneous audience and to support understanding of those data and collective discussions about a common topic. Dissemination of information about risk is key to the success of flood mitigation policies. Since floods cannot be prevented from happening, and given the damages they inflict on populations and territories, it is now widely considered that flood mitigation policies are all the more efficient when they are collectively conceived and implemented, that is to say, together with elected representatives, citizens, and any other interested parties. Both European and French legislations insist on this participatory aspect, in the hope that flood risk will be better taken into account by planners as well as citizens and organizations in their spatial and social practice. This evolution in the design of risk mitigation policies affects the status of information about flood risk and its diffusion among communities. Risk managers are thus led to look for efficient means to communicate technical information about risk in order to raise public awareness of the issue and to get elected representatives, firms, associations, and citizens to participate in flood mitigation planning processes. The extent of the understanding of flood risk by all stakeholders is thus considered as one of the factors in ensuring the efficiency of flood mitigation policies.
In this context, we worked over eight years (2009–2017) together with risk managers from local government agencies in order to co-produce 3D geovisualizations and evaluate their uses and benefits in practice as far as the understanding of data about risk is concerned. Scientific literature is unfortunately not offering a satisfactory theoretical framework to understand and analyze the various uses of geovisualizations, and this is still a major issue according to many researchers [1].
Indeed, from a scientific perspective, the effects that 3D geovisualizations have on the communication and understanding of information and their ability to foster exchanges between heterogeneous actors and participation of the grand public to planning processes have not yet been established clearly. Although 3D geovisualizations are more and more resorted to in urban planning, the evaluation of their effects remains debated among researchers and practitioners. Many researchers in geography and geographical information science have underlined the need for specific studies on the use of 3D spatial representations [2,3,4,5,6,7,8,9]. Several researchers, after careful literature review, have stated that the evaluation of the benefits of 3D geovisualizations in urban planning cannot be confirmed due to the limitations of feedback from individual case studies and the scarcity of theoretical elements, making it insufficient to provide a framework for further studies [5,6,8]. Bishop and Lange [5] further noted that studies trying to determine which level of detail is suitable for a given purpose led to contradictory results and that they have no rational means to decide between them. Other researchers make contradictory assertions, some stating that 3D geovisualizations being “realistic” images are mostly counter-productive, if not harmful for citizen implication in planning. They indeed consider those images as a mere political mean of communicating serving elected representatives’ own interests [10], or, worse, as producing a visual illusion that can dupe citizens and prevent them to think about what is presented to them [11,12]. Others assert that 3D geovisualizations, being “realistic”, are simply used for the communication of information and seem to consider this information is received straightforwardly by their audience, or at least, more straightforwardly than 2D representations [13,14].
In the field of flood mitigation, the various uses and effects of 3D geovisualizations have not been treated as a specific field of scientific investigation. As far as geographical information systems (GIS) and 3D GIS are concerned, scientific literature is mostly dedicated to the use of georeferenced data and tools to produce data about risk in relation to hydraulic modelling, to produce analysis of flood risk and to build cartographies of flood risk and their impact on a given territory (see for instance [15,16,17,18]). Three-dimensional geovisualizations tools are also mobilized in crisis management contexts as tools to visually analyze a given phenomenon [19] or to deliver in situ indications to endangered populations (for instance evacuation itineraries) during a crisis [20]. More broadly, 3D geovisualizations have been used to produce visualizations of risk as part of decision support systems, oriented toward experts [21].
Confronted with discrepancies in the scientific analysis of the benefits of 3D geovisualizations in urban planning contexts, especially used with elected representatives and citizens as well to contradictory results from empirical studies on which type of 3D geovisualizations should be used in a given situation, we opted for an exploratory method. Our method is based on the observation of the production and uses of 3D geovisualizations, which we were involved in, through action-research. It aimed at producing empirical results about 3D geovisualizations’ uses. Ethnographic observation of several cases over the years and analysis of our case studies enabled us to make sense of contradictory results from previous scientific studies, shedding light on some theoretical shortcomings in the way to apprehend 3D geovisulisations and their uses in urban planning. Researchers from different disciplines resorted to their own theoretical framework for the analysis of gathered data. From a geographical information science’s perspective, with the help of sociological notions and theoretical elements, 3D geovisualizations have been treated as technical objects used in collective settings to help actors with heterogeneous backgrounds to understand how flood can affect their territory and to foster exchanges between those actors about how to mitigate this risk. From a communication studies’ perspective, 3D geovisualizations were defined as techno-semiotic devices aimed at semiotic and cognitive performance. Both approaches produced complementary results on the effects of 3D geovisualizations in the context of flood mitigation. Taking into account results from both approaches allows to put commonly used notions and assumptions in each discipline into perspective. Our empirical and exploratory approach do not allow us to generalize from our results, but does provide useful insights on how to analyze and understand 3D geovisualization uses.

2. Materials and Methods

Our approach was based on observation of the use of 3D geovisualizations in similar situations in various districts along the river Rhône. It aimed at gathering evidence about their contribution to the comprehension of technical data about risk as well as to collective thinking and exchange of ideas among heterogeneous actors. Heterogeneity of actors is here defined by the diversity of their implications in flood mitigation policies (professionals/non-professionals), their various professional backgrounds, and knowledge about risk. The ability to participate in flood mitigation planning policy in vivo offered us a great opportunity to try to develop a better understanding of 3D geovisualizations’ uses, since scientific studies were imprecise on the matter. Nevertheless, participating in real life situations by producing 3D geovisualizations for practitioners, we had to adapt to practitioners needs and demands and could not impose, and thus not experiment, some design options as can be done in other experimental settings.
Concretely, we developed four action-research projects over eight years (2009–2017). The 3D geovisualizations have been produced for eleven different flood mitigation processes, scattered throughout six different regions and covering about fifty districts, and have been used in seventeen meetings. While elected representatives were present in every seventeen of those meetings (3 to 10 elected representatives, depending on the meeting), associations and residents were included in only ten of them (4 to 70 of them, depending on the meeting). This lower number of observed meetings including citizens is linked to the length of the process of elaboration of flood mitigation plans. Thus, some consultation meetings took place after the end of our projects and we could not observe them. On the whole, between 2009 and 2015, more than 150 persons were present in the observed meetings and about a hundred were observed actively interacting with 3D geovisualizations (at least commenting on them orally if not manipulating them in some way). Our last project (2015–2017) was focused on public events aiming at raising citizens’ awareness about flood risk, so we were mostly confronted with inhabitants from areas at risk (around 100 persons over two days of experiment, using different 3D geovisualizations that were prepared for them).

2.1. First Set of Experiments (2009–2014)

2.1.1. Produced and Used 3D Geovisualizations

During the first three action-research projects, produced 3D geovisualizations were schematic 3D georeferenced models produced from automated modelling of geographical information provided by the French mapping agency (Institut national de l’information géographique et forestière—IGN) with the Spaceyes 3D software (versions 4, 5, and 6). Since we produced 3D geovisualizations over many years, we used several versions of the software. However, we did use the same tools and operations over the years, notwithstanding the version of the software. Great improvements were made in version 6 to reduce the time needed to export pictures and films.
Representational choices were made according to risk managers’ needs and demands in an iterative process during various meetings, so as to get as close as possible to what was expected by practitioners. As we were part of a larger set of actions led by practitioners regarding flood mitigation planning, we could not propose any design we wanted for the produced 3D geovisualizations. In particular, practitioners were looking for efficient 3D geovisualizations that were as cheap and easy to produce as possible, so that our experiments could help other territories to resort to 3D geovisualizations with the same methods. This is why we could not test more interactive design; interactivity with our 3D scene was limited to navigation in space and time (during the flood). On the other hand, it allowed us to observe the efficiency of basic schematic models in many situations.
Schematic 3D models were produced from a digital elevation model (2 meters resolution), an orthoimagery (50 centimeters resolution), extrusion of buildings from the IGN BD TOPO® (2.5D modelling), and sometimes a schematic representation of a selection of the site’s vegetation (mostly a few individual trees, well known from the inhabitants and textured polygons from large areas of vegetation and typical hedges). Then, hydraulic data was added through water heights map represented in 2D colored polygons and through a transparent 3D volume. A few landmarks, considered by local risk managers as useful points of references for elected representatives and inhabitants to orient themselves in the 3D geovisualizations, were added through 3D objects (for instance, a particular dam, windmills, quarries, etc.). Figure 1 shows a typical result (aerial view). Figure 2 shows two examples of views showing the 3D representation of water flows through a 3D transparent volume.
For all meetings, still images and movies following prepared trajectories aimed at explaining how a flood could invade the given territory were produced from 3D scenes in which real-time navigation was possible. Risk managers wanted to be able to include them in their presentations and send them easily to their colleagues as well as to stakeholders (elected representatives and associations in particular). Real-time navigation was also possible on demand since the researcher who had produced them was present during each meeting and had the necessary equipment (laptop and software). Real-time navigation was scarcely used to answer some specific question and by practitioners themselves to work on their own data and models.

2.1.2. Methodology for Data Gathering and Analysis

Empirical data was gathered through participant observation [22,23,24,25,26]. A researcher in geography and urban planning was responsible for the production of 3D geovisualizations and was directly observing the production and use of those 3D geovisualizations throughout preparation meetings with risk managers as well as during meetings with stakeholders. This specific implication allowed the researcher to participate in and observe meetings to which access is restricted to involved parties. This includes meetings between government agencies and elected representatives, where negotiations about flood mitigation planning are conducted. It also includes informal exchanges between actors before and after official meetings. This allowed the gathering of numerous data (field diary, observation transcripts, interviews, case studies’ documents produced by observed actors such as synthesis, e-mails, etc.).
Ethnographic methods were used to collect data [27,28,29] and qualitative methods based on grounded theory and inductive approaches were needed to compare the various case studies [30,31] and to exploit the gathered empirical data. Data were tagged and analyzed during the whole duration of the project so as to find results that matched all gathered data, which is quite time consuming, but allows to produce valid statements when existing scientific studies do not provide solid theoretical elements. When data was in contradiction with intermediate results, all data was analyzed again so as to refine the analysis and propose adequate results. As one researcher was working full time with civil servants in charge of flood mitigation policy over the whole period, data about their uses of 3D geovisualizations and evaluation of the advantages and shortcomings of those spatial representation was gathered through regular discussions with them and registered in a field diary together with the documents related to the case studies. This proximity allowed the researcher to ask any question he needed to ask, sometimes repeatedly and over many years, to obtain precise answers, and to refine his analysis through numerous discussions, so that fewer formal interviews were conducted (only with stakeholders that were only seen once, that is to say mostly riverside residents). Ethnographic observation was specifically used to describe the performance of 3D geovisualizations during meetings. It was also used to describe the extent of the understanding of technical data about flood by elected representatives and riverside residents thanks to 3D geovisualizations. Their understanding was evaluated through oral comments, that is to say that it was considered that they had understood what was presented to them only when they had explicitly reformulated it. Discussions between elected representatives and the researcher after the meetings provided data on their affinity (or absence of) with 2D maps and on the extent of their knowledge and previous experience of floods. This could not be done with every riverside residents, although many of them were quickly interviewed after meetings when possible. This lack of data about the experience, strategies, and opinions of citizens led us to ask researchers in communication sciences to help us with the observation in our fourth project (see Section 2.2).
Finally, sociological theories were mobilized in order to analyze the various uses and effects of 3D geovisualizations in the observed courses of actions relying on sociological analysis of other types of technical objects and representations in use in collective settings [32,33,34]. Theoretical elements were imported from sociology, more specifically from actor–network theory [35,36,37,38,39,40] and the conceptualization of “intermediary objects” [27,41,42,43]. Interestingly, this notion of intermediary objects has been used for many years by sociologists to study the use of various technical objects. As far as spatial representations used in urban planning are concerned, this notion was partly used in a few scientific studies [14,44] in order to characterize the uses of given spatial representations. Nevertheless, its use was always limited in the sense that only one use was considered for a given spatial representation in a given situation, setting aside the potentially transitory aspect of the notion, which allows for a technical object to have successive roles during a single course of action [45,46]. We quickly noticed through ethnographic observation that 3D geovisualizations indeed played many different roles in a single course of action, so careful observation was crucial to the understanding of their effects and uses.

2.2. Second Set of Experiments (2015–2017)

2.2.1. Produced and Used 3D Geovisualizations

Similar methods as those used in previous projects, and described above (see Section 2.1.1), were mobilized in order to create a schematic 3D geovisualizations of the territory selected for the experiments. A 3D scene containing all data was produced from which still images and movies could be exported. Temporal hydraulic data were used in order to create several visuals, each corresponding to a different moment during a simulated flood. We mostly had water height data every 8 h into the flood, which lasted over approximately 8 days. Sometimes water height data were available every 4 h to illustrate a particular dispersion pattern in the water flows. Those 3D geovisualizations of several moments during the flood were then exported as images and movies from selected points of view. Points of view were selected by risk managers so as to be used as visual aid when explaining the progress of the simulated flood (see Figure 3).
Movies could be played on any devices used during public events (tablets and smartphones). Still images were imported into an interactive interface so that inhabitants could browse to any phase of the simulated flood and observe what would happen for a given water flow (see Figure 4). Those were consulted through interactive tablets by mediators and citizens during the event. Lastly, 3D scenes from symbolic public spaces were realized with SketchUpMake 2016, from the same geographical data and pictures taken on site, so as to provide 360° visuals of the evolution of water heights from a pedestrian’s point of view (see Figure 5). Those visuals were also inserted into an interactive interface allowing to navigate between them to visualize any moment during the simulated flood. They were consulted through virtual reality headsets (Homido®) in which a smartphone had been inserted (either iPhone 6 or Samsung Galaxy S5).
All those visuals were imported on tablets to be used by risk managers to explain the flooding phenomenon on the area and to be consulted by inhabitants. The 3D scenes (cf. Figure 5) were imported on smartphones to be visualized with virtual reality headsets on site (i.e., on the site corresponding to the 3D scene). Figure 6 shows some pictures that were taken during the events.

2.2.2. Methodology for Data Gathering and Analysis

During our last action-research project (2015–2017), researchers in communication studies led an evaluative task throughout the whole duration of the project, from the production to the use of 3D geovisualizations. From a communication studies perspective, 3D geovisualizations are considered as techno-semiotic devices, that is to say an arrangement of objective elements aimed at semiotic and cognitive performance [47,48,49]. As shown in Figure 4, the produced 3D geovisualizations combine iconic, figurative, plastic, linguistic and diagram elements, as well as exogenous documents, such as photographs from previous floods (archives). Figure 6a also highlights the suggested semiotic relationship between the view presented on the tablet screen and the panoramic view from the hill (from where tablets with 3D geovisualizations are consulted). With regard to the virtual reality device, the physical place where the images are viewed (the place in the village) is the same as the one represented virtually: in other words, individuals located in a place see this place transformed by the flood when using the virtual reality headset (Figure 5 and Figure 6b). This second device requires interaction with the interface, but also involves a sensitive dimension in the user experience.
Once these characteristics have been highlighted, it is important to understand the mediating role of these mechanisms in social situations, within the institutional framework of risk awareness. Social uses and meaningfulness of those techno-semiotic devices are the subject of many works, for example in museographic and scientific mediation contexts [50,51,52,53]. Scientific works also analyze them as means to “govern the social”, structuring contemporary democratic societies, especially in relation to the management and prevention of risks by State based on the instrumentalization of expert knowledge and their representation [54,55]. Flood mitigation policy combines those two aspects. Risk managers, as they explicitly stated, rely both on the authority conveyed by experts’ data portrayed in 3D geovisualizations and on their staging in several playful activities (on-site interactive visualization with tablets and on-site immersive visualization with VR headsets) during a public event in order to raise citizens’ awareness about flood risk.
The methodology of evaluation is based on 3D geovisualizations as devices conveying meaning and subjects to interpretation. As a medium, 3D geovisualizations fit into a framework that provides a context for their uses. This framework is spatial, temporal, symbolic, cognitive, and social, and needs to be studied as part of the evaluation of 3D geovisualizations as techno-semiotic device. As symbolic representations, 3D geovisualizations, like other visual media, produce a discourse [56,57] and prefigures interaction between the user and the contents. We thus analyze 3D geovisualizations as a discourse about flood risk and the territory. Finally, 3D geovisualizations are part of a mediation, as they are building and organizing relationships between people, objects, knowledge, institutions, and territories.
The semiotic characterization of the “meaning effects” of 3D geovisualizations and the construction of knowledge that they allow are based on two steps: first, the observation of their conception, and then the observation of their uses with citizens (inhabitants and others) during the events.
The first stage of the evaluation was the realization of an ethno-semiotic study of the devices in the process of conception. Technical and symbolic choices in terms of representation that were made by risk managers and involved researchers were observed. During the production phase, researchers in communication studies observed and participated in the preparation and meetings with risk managers and helped with the implementation of 3D geovisualizations in interactive devices (tablets and smartphones). They thus exchanged interactively through a continuous dialogue with all the actors of the institutions involved in the project: scientific (computer science and geography), political, and public.
The evaluation had several objectives. The first goal was to accompany the production process and help risk managers, computer scientists, and researchers in geographical information to have a reflexivity about the representations which they produced. Semiotical analysis of geovisualization was a good way to shed light on some issues related to the visual transcription of data. The second goal aimed at validating the relationships between potential cognitive issues and visualization devices as they were produced. For example, which preliminary knowledge was necessary to interpret the representation of the territory? Which digital skills? The last of our main goals was to question the ability of devices and contents to be appropriate in the context of experimentation, as planned by risk managers and involved researchers.
The second stage of the evaluation was the realization of an ethnographic study of the two public experiments through direct observations completed by semi-directive interviews with the citizens who had tested the devices. During the first event, 20 users were interviewed (among 50 visitors), 4 groups of users were followed and observed from their arrival to their departure. During the second event, 10 interviews were conducted (for 50 visitors) and 10 groups of users were continuously observed. A specific interview guide was produced in order to complete the evaluation.
The multiple objectives of this evaluation are detailed in [58]. Basically, we wanted to confirm that citizens had a relevant use of media and visualizations (i.e., that they gathered some knowledge from it). We wished to identify the encyclopedia, literacy, skills, and ordinary knowledge mobilized by the interviewed citizens during their test of the devices (for instance digital and cartographic literacy, spatial knowledge, risk management skills, etc.). We linked individuals’ skills to in situ observations of their interaction with 3D geovisualizations and noted how interviewed individuals referred to their individual memory, their knowledge and previous experience of the modeled territory, and how they justified their presence at the event and their potential motivation.

3. Results

3.1. Results from the Observation of the Elaboration of Flood Mitigation Plans (2009–2014)

As stated above in the methodology section (see Section 2.1), we present in this section results from our first five years of observation of the elaboration of eleven flood mitigation plans and of the use of 3D geovisualizations during seventeen meetings with risk managers and stakeholders. Our experimental results are derived from participant and in vivo observation. Therefore, we could neither control nor quantify precisely many parameters as done in experiments conducted in research laboratories. Nevertheless, qualitative analysis led according to the Grounded Theory and inductive approaches principles [30,31] allowed us to produce statements validated by all gathered data (field diary, observation transcripts, interviews, case studies’ documents produced by observed actors such as synthesis, e-mails, etc.). Those statements, although being quite generic, allowed us to explain why previous studies had led to discrepancies and contradictory results [46]. In this respect, those statements are also compatible with results from previous studies. Since we could observe factors influencing the effects of 3D geovisualizations in context, our qualitative results provide insights on how to design further studies on the uses of 3D geovisualizations, but they do not provide an exhaustive and precise list of all factors influencing 3D geovisualizations effects and uses. Those results are complementary to quantitative studies conducted in more controlled environments and allow to broaden theoretical perspectives toward a more comprehensive approach of 3D geovisualizations’ uses.
The most striking result is the variety of roles a given 3D geovisualization plays in a single situation, i.e., per observed meeting, which leads us to go beyond the association of a single use for a given 3D geovisualization. Careful ethnographic observation of discourse, gestures, and oral exchanges between actors reveals that 3D geovisualizations are useful in order to communicate information about risk to heterogeneous actors, provided its content has been first described and understood by all actors present. In other words, elected representatives and citizens gained understanding about hydraulic data (i.e., how a flood is likely to invade their territory and to which extent) by looking at and interacting with 3D geovisualizations, as proved by the discussions among them and with risk managers during meetings. Interestingly, 3D geovisualizations were also used in several meetings (three-quarters of them) as a visual aid for discussion, negotiations, and decision making, although they were only supposed to be used in the first part of the meetings as a tool to present hydraulic data and help elected representatives and citizens to understand the extent of a potential flood. In fact, once actors have grasped what 3D geovisualizations represent and have successfully found enough landmarks to orient themselves in the 3D scene, they make use of 3D visuals not only to increase their knowledge about flood risk, but also to support their assertions, think about a spatial configuration, or discuss together with other actors, according to the context, their own strategy, and their reaction to the action of other stakeholders.
Another striking result is that several of these uses were not anticipated, including uses of 3D geovisualizations by risk managers who ordered their production, which advocates for more in vivo studies of 3D geovisualizations’ uses. Risk managers had first asserted that 3D geovisualizations were only useful to communicate technical (hydraulic) data to elected representatives and citizens, since risk managers themselves produced the hydraulic data, knew it well, and visualized it in other specific software with 2D interfaces. Interestingly, during internal meetings organized to validate the produced 3D geovisualizations before public use, risk managers did use 3D geovisualizations of their own hydraulic data as a collective visual aid to discuss among themselves about the quality of the data and the appropriate measures to propose to elected representatives. When asked individually about this use after the meeting, they first reacted with surprise, re-asserting that 3D geovisualizations were merely communication tools aimed at elected representatives and citizens before recollecting what had happened and acknowledging it had actually been a useful visual aid for them as well.
Nevertheless, empirical data shed light on the necessary appropriation of 3D geovisualizations by their public, which weakens the claim that 3D geovisualizations are “realistic” and, as a consequence, straightforwardly deciphered. This appropriation seems to be a prerequisite to almost any use of 3D geovisualizations. The ability to grasp the content of a given 3D geovisualization and to use it in order to explore spatial data depends not only on the way elements of a territory or landscape are figured, but also on the way 3D geovisualizations are presented to an audience and how this audience can handle it, whether through sight or real-time navigation in the 3D scene. Empirical data further show that an explanation of how objects are represented and modeled is crucial to the intelligibility of 3D geovisualizations when they are first showed and has a decisive influence on the way actors will be able to use them during collective tasks. Although 3D geovisualizations are clearly easier to grasp than 2D maps of areas at risk according to many of the observed actors, individuals cannot always decipher them without time and a little help. Some of them need time to get used to this way of representing the world.
Comments during the presentation of 3D geovisualizations to elected representatives and citizens illustrate their need to find correspondences between the territory they live in or work in and its schematic 3D model. Once some objects (a given building, a known road, etc.) have been recognized, elected representatives as well as citizens are able to link their own knowledge of the territory with data displayed in the 3D model. The ability to compare abstract data with individual experiences is a key element in the process of understanding technical data about risk. As far as flood risk is concerned, water heights data has proven to be well understood when it was compared with individual experience of past floods (in the case of the river Rhône, recent floods are still remembered by many inhabitants). Although we are aware that we used schematic 3D geovisualizations, this does not mean that a greater level of detail would be straightforwardly deciphered (too many information can also lessen 3D geovisualizations’ effects). In other words, notwithstanding the level of detail, the need to link the visualizations to known aspects of a known territory pertains in similar settings where inhabitants are shown 3D geovisualizations of their own territory. This need can be solved, but will not automatically be solved in every situation, by a greater level of detail.
Surprisingly, the technical functionalities allowing interactive navigation in real time in a 3D geovisualization do not seem to be decisive in the observed collaborative uses and active mobilization of 3D geovisualizations by any actor, whether he is presenting the 3D geovisualizations or is standing in the audience. In other words, interactivity with 3D geovisualizations is not per se the only factor to define or restrict their uses, and other factors linked to the situation (mediation, oral explanation, tasks, and actor background and strategies) can either deepen or restrict 3D geovisualizations’ effects.
In fact, still images have proven to be quite useful in collaborative thinking about perimeters and limits to draw for specific mitigation action considering hydraulic and topographic parameters, at least in settings where the public is not far from the projected 3D geovisualizations. When they cannot interact directly with the 3D geovisualizations, actors simply point their fingers to features on the screen (or draw on the board on which the 3D geovisualization is projected). In other words, visually representing some aspects of the territory can be used to mentally and visually validate or invalidate a hypothesis or a proposition. Thus, in some meetings, still images had been brought as 3D geovisualizations and were only expected to show some basic information about the extent of flooded areas, and those still images were used in collective discussions just as real-time navigation models were used for the same purpose in other meetings. As a consequence, the active use of 3D geovisualizations probably depended less on their technical level of interactivity than on how their audience is given resources (time, explanation, etc.) to be able to comprehend their content and the goals and strategies of the audience.
Those empirical results invite us to broaden our theoretical perspective on the uses of 3D geovisualizations by taking into account their multiple contributions to interactions between actors. Our first years of experiments allowed us to gather data on the benefits of resorting to 3D geovisualizations and factors influencing those benefits from the point of view of risk managers (especially time and explanations needed for appropriation). It also shed light on factors influencing the uses of 3D geovisualizations in practice that are related to the public’s experience during the time they are exposed to 3D geovisualizations (presentation by risk managers, collective discussions, and decision making).
Unfortunately, we only gathered scarce evidence directly from citizens other than what they had explicitly expressed during meetings and were left with many questions. First, how does the staging of the presentation of 3D geovisualizations influence the perception and understanding of its content? For instance, when a film derived from 3D geovisualizations was once shown to inhabitants in the local cinema, putting them more in a spectators’ position, were they impressed or could they also gain knowledge about flood risk? Second, how are the 3D geovisualizations perceived by inhabitants when it is shown to them by risk managers or by their elected representatives? Does that make any difference? Does the trust in the 3D visuals rely solely on the 3D geovisualizations themselves (i.e., because they are derived from geographic databases produced by experts)? More broadly, considering the importance of interactions between actors in collective settings, how can we understand the public’s experience and the factors influencing the uses they can make of the 3D geovisualizations they are presented with and the knowledge they can derive from those uses?
This is why, for our fourth project, focused on public events for raising citizens’ awareness about flood risk, we worked together with researchers in communication sciences, who brought their own theoretical apparatus and opened up a new perspective on our research questions.

3.2. Results from the Observation of Public Events Aimed at Raising Citizens’ Awareness about Flood Risk

As a response to initial questions from researchers in geographical information science, researchers in communication studies developed a different point of view on our research question. Using their own theoretical framework and observation methods, they shed light on some of the limits in the perception and evaluation of 3D geovisualizations and their uses by researchers in geographical information science.

3.2.1. Factors Influencing the Production Process

Observation of the meetings and exchanges between the institutional and research actors of the project showed that the technical decisions regarding the production of the 3D geovisualizations is closely associated with the definition of the aims of mediation, entailing various aspects. For the designer of 3D geovisualizations aimed at a given audience, representational choices rely on various and sometimes competing aspects, way beyond technical considerations.
First, representational choices are discussed in terms of the characterization of the relevant geographic scales or landmarks to model (for instance well-known buildings, bridges, or dams). Two different aspects are taken into account when making those choices. On the one hand, the necessary link with the territory is considered as essential, so that the public of the geovisualizations will actually recognize what is modelled and presented to them. This led producers of the 3D geovisualizations to add landmarks and historical pictures to their visualizations. On the other hand, the way the reception of 3D geovisualizations by non-experts is anticipated is also considered by risk managers and involved researchers in the design of the geovisualization. This led to choices for the representation of vegetation for instance, in relation with what the public supposedly needed to see (some vegetation as a point of reference to orient itself but not too much vegetation hiding the representation of flood risk) in order to ensure the message about flood risk would be understood.
Secondly, designers feel the need to control the restitution of the data in relation to their verisimilitude, so as to be sure that the geovisualizations will not be perceived as a mere video game but as a truthful representation of an existing territory. Moreover, designers want to provide their targeted public (non-expert audience) with the necessary modalities to engage with the representation so as to generate recognition and involvement. In terms of flood mitigation planning, reaching out to citizens is a key issue, and risk managers work hard on producing attractive activities. This is done through figurative simulation and a semiotic organization allowing the consultation and interpretation of contents, that is to say a computerized medium including signs allowing navigation (buttons, timeline, etc.) [57].
These aspects are delicate to articulate and completely linked in the process of editorialization of geographic information. In a non-numerical framework, Daniel Jacobi [59] underlines it with regard to 3D physical model used in museography. The physical 3D model is then both a relevant device to represent a dimension of the scientific discourse (a “reconstitution”) and a spectacular device that fosters a posture of playful recognition (spontaneously decipherable, acting as a scientific theater and according to a narrative logic).

3.2.2. Meaning’s Effects in the Visualization Project

One striking result of our evaluation is the complexity of the mediatic dimensions of a supposedly straightforward 3D geovisualization representing an existing territory and a simulated flood on this territory. This complexity has several dimensions. The specificity of the conversion of data sets into figurative representations, but also the multimedia and intermediate nature of the devices placed in the situation. Indeed, 3D geovisualizations coexist with other images and signs as we have already mentioned: pictures from past floods and diagrams integrated into the representations themselves (which leads to associate very different types of images from the point of view of their referent and production context). Moreover, during the events, maps and printed photographs were presented to attract passers-by but also to serve as a support for information and explanation.
It invites us to be careful when affecting a message, a piece of information, or knowledge to a 3D geovisualization that it is supposed to convey. The 3D geovisualizations containing multiple data can be interpreted in various ways and be used to explore some of the featured data, even when this was not anticipated by the designers of the geovisualization, as stated above (see Section 3.1).
Any image seen when viewing a 3D geovisualization generates a set of issues and require a sharp expertise on media dimensions it engages. The visual representation in a single model tend to make the various data sets and types of knowledge mobilized to produce the 3D geovisualization become indistinguishable. Indeed, the uniqueness of the framework and the representation (the simulation of a coherent environment from diverse data) semiotically integrates the objects, then exposed to the risk of “freezing the knowledge in a single model” [60]. The risk then is that the public sees and interprets those images without the necessary cognitive skills to qualify and understand them. For instance, the statistical and hypothetical dimension of the simulation of the flood must be perceived by the public.
In the case of a simulated flood, visualizing a single coherent result is almost contradictory to the complexity and the inherent uncertainty in the construction of knowledge, for example scientific, mobilized to model a flood. A simulated flood is indeed a “virtual” flood that could take place but has never happened and could very well never happen. In other words, a 3D geovisualization of simulated floods, more than a visualization of geographical information, is a “mediatic enunciation”. The temporal 3D geovisualizations of the simulated flood thus constitute a supporting narrative aimed at raising awareness through the explanation of the behavior of the river much more than a diffusion to the public of scientific information visually represented.
The design of visualization devices has to integrate these issues, especially in the adjustments between schematization (in relation to the used scientific models) and narration (storytelling prepared for the public events). As stated above, geovisualizations of data are necessarily vacillating between two logics: instrumentation of information (organization of geographic information, data sets) and “becoming-media” (producing a medium able to convey some information about flood risk to the public in the future event). In such a setting, questions about how to represent a given data rarely have unique and well-established answers and designers have to cope with many uncertainties.
For 3D geovisualizations to operate as authentic and credible simulations (and not as playful and spectacular simulations), two modes of territorialization are resorted to by designers. First, 3D geovisualizations have to evoke a “world”: a reference territory (circumscribed, physical) with the paradox of representing at the same time its present and potential future (statistical projection of the flood). Second, 3D geovisualizations are calibrated so as to participate in a given in situ event, located in space (site of the public event and prepared tour itinerary) and time (date and timing of the event).
In addition to the design of 3D geovisualizations and their inclusion in a planned event, local authorities involved in the project, as organizers of the events, offer a framework of warranties that institutes the simulation as credible and attributes power and value to the geographic information represented. The individuals who took part in the experiment aimed at testing 3D geovisualizations, and considered as the “public”, participated with good will in the “test” because they had confidence in the proposed framework [61].
In fact, their knowledge about risk or their subjectivation by the visualization depends on this framework. Individuals may allow themselves explorations and reformulations of information and knowledge in a communicative framework only if it gives them those guarantees (authenticity of views and quality of data). In this respect, signs of authority (logos for example) can act as guarantees, as well as the presence of local authorities or risk experts on site during the events.

3.2.3. Cognitive and Symbolic Dimensions of the Reception and the Interpretation of 3D Geovisualizations by Citizens

Another aim of the evaluation was to qualify the level of engagement of the public during the event, since this engagement is considered by risk managers as key to the success of their event. Indeed, if citizens play the game and gain knowledge about flood risk, they will be more likely to make decisions limiting the impact of a flood on their territory.
As demonstrated about the reception of immersive exhibitions in the museographic context, the appropriation of knowledge depends on a tension between immersion and distancing [60] (p. 101), which supposes that the visitor qualifies the activity in which he is engaged. Also, we wanted to understand the links between the devices (3D geovisualization) and the activities proposed to citizens, i.e., the context of the mediation.
From the verbalization by visitors of their experience during immersive visits, Belaën has identified five types of involvement by visitors. These five types establish a fine gradation, which we have observed in our case study. In other words, various levels of engagement were observed, confirming, if needed, that 3D geovisualizations have various effects, especially depending on the individuals and the way they engage with a given medium. In this respect, this level of involvement is one of the factors that can greatly influence the potential knowledge gain from the interaction with a 3D geovisualization.
We illustrate those five types of involvements below in the case of flood mitigation with extracts from our semi-directive interviews. We indicate after each quote the gender and age of the person interviewed (the proportion of men and women who tested the devices is very similar), but, in our exploratory process, this data has not been used. It is important to add that, with a few exceptions, all participants tested the devices collectively: either with their family, friends, as a couple, or simply by their co-presence during the event. This is linked to the small size of the events, their location in a rural area, and the fact that they were held on Sundays.
Resonance describes the acceptance of staging effects as proposed by the organizers of the event as well as the ability to discuss objectives and institutional discourse:
“I must confess I know the risks linked to floods here. But I was really surprised that the water goes up so high, I never thought that could happen. And what surprised me too is the slow velocity of the flood. It is as they explained for Paris on TV these days. I thought it was faster, that the Rhone had more temperament [laughs].” (Man, 53)
Submersion implies a visitation activity without distancing, where emotion is dominant:
“The glasses are great, we really have water that rises to our knees! And the tablet is the same, I loved it! But we did not have enough time, I would have liked to continue, there are lots of things I have not seen. But the glasses … it’s still what tells the most.” (Woman, 57)
Critical distance is the mobilization of a reflexivity, for example by the search for answers and traps or errors. Many of the interviews concern this critical distance, either globally or partially, which is not surprising given the “test” situation in which the interviews were conducted. Individuals adjust their response to what they think is the response expected by the investigator, especially in an institutional setting.
“It’s good. We believe in what we see. We say that it can go up as much.” (Man, 38). Or, about the media: “These are not Google photos! That’s good, we have to give them a little competition! “That’s the thing. But you need more power, put a computer next to it, the tablet is overloaded.” (Man, 50)
“What did I see… it was the stages of the flood, there was more and more water and more and more places were flooded. Yes, it was understandable, but there was too much sun. I couldn’t see much, but when there was a cloud… I saw a lot of things, it was very clear, yes." (Woman, 52)
“Yes, after that I also saw the overall view of the territory. It was good to see the stages of the flood, I think it’s good to explain. But to be interesting, you have to use glasses instead. » (Woman, 44)
Banalization refers to regret expressed in the face of what is considered a disproportion between the techno-semiotic device and the low content or a message that is too altered:
“Ah, were these steps? No, well - I did not understand everything. You show the course of a flood in time? No, I did not see that.” (Man, 51)”
“It’s not accurate enough for me. But otherwise it’s really successful, also with all the functions and especially with the photos. But I feel like it takes an hour to load something, it’s annoying. » (Man, 53)
Finally, reject evokes an implication in the visit which fails, insisting on the gap between the expectations and the proposal, the visitor remaining foreign to the subject and the techno-semiotic device.
“Your tablet there … yes, I saw it. But I did not really understand how we zoom, she showed me stuff with her fingers there … It’s more for young people. You have to be young for that. But we spoke with the flood expert, he explained everything to us. And it was very good, really. Pedagogical and understandable.” (Woman, 65)
These typical responses we observed among the interviewees are very valuable in highlighting the variability in the degrees of involvement we have encountered in our observations. The analysis of these contrasting reactions to similar devices underlines again how much the observation and analysis of the (multiple) uses of 3D geovisualizations (or any other cartographic device, regardless if it is digital and interactive) must rest partly on a conceptual framework of how human actors react to these medium, understand them or interpret them, individually or collectively and in relation to both 3D geovisualizations and human mediators. In fact, these different postures reflect two transversal dimensions: on one hand, the importance of the referential dimension of visual representations; on the other hand, the interpretive activity of the individuals, articulating the techno-semiotic devices to the whole event.
In our case study, the role of human mediators has been a dominant finding of evaluation. This confirms the necessary appropriation of the devices already spotted by researchers in geographical information science. This came out of interviews, as here in this caricatural example:
“What I liked most? It was Monsieur with his explanations, at the top. He looked very competent.” (Man, 53)
Observation of the two public events shows that the idea of autonomous mediation using techno-semiotic devices is an aporia in this context. Finally, our evaluation aimed at establishing the level of confidence required for any appropriation of the devices in the context of the proposed experiment. This can be linked to the complexity of the mediatic dimensions of 3D geovisualizations described above and to difficulties in the manipulation of the proposed devices (tablets and smartphones). Individuals intuitively understand how to manipulate a tablet when the mediator manipulates it in front of them, and the mediator directs the content of the mediation. Without human mediation, there would be no structured content delivered. There are of course other design options that can embed some of the necessary mediation into a 3D geovisualization framework and we are aware that, not being able to choose the design of the 3D geovisualizations ourselves, other design options could have been explored. Nevertheless, the produced 3D geovisualizations were similar to what practitioners were able to produce easily in France at the time, thanks to IGN data and not too complicated tools, so that our results provide useful insights for many actual uses of 3D geovisualizations in urban planning. Moreover, our results can also provide useful insights about dimensions that should be taken into account when designing more elaborate 3D geovisualizations, although it does not tackle the issue of how to technically integrate them.

4. Discussion

Empirical data collected during this research project and their analysis thanks to sociological theories and from a communication studies’ perspective provide original insights on the uses and effects of 3D geovisualizations in collective settings in the field of flood mitigation planning. Importing theoretical elements from sociology (developed for other types of technical objects and representation) and from communication studies (designed to study mediatic dimensions of technical devices) helps to fill the gaps in existing theoretical approaches of 3D geovisualizations and allow to study their effects beyond semiotic aspects linked to the deciphering of a graphic representation in an individual setting (as proposed by Alan MacEachren [62] in his seminal book, “How Maps Work”). Experiments confirmed that 3D geovisualizations can help elected representatives and citizens to deepen their understanding of the results of hydraulic modelling, giving them access to a strengthen knowledge about flood risk. Case studies’ analysis also confirmed that 3D geovisualizations can be used for supporting exchanges of ideas and collaborative thinking and that they can foster shared decision, acting as a medium that facilitates the making of a shared diagnosis and helps reaching agreements over measures to be taken as far as flood mitigation policy is concerned. Those results confirm the importance of visual analysis that can be achieved by looking at and manipulating 3D geovisualizations, beyond the ability to convey pieces of information to a given public. Since we only used schematic 3D geovisualizations with little interactivity, our results underline the great visual potential of schematic 3D geovisualizations, but we cannot offer insights on the effects of more interactive devices.
Our methodological approach, developed over the years, is explained and discussed in detail in [46]. Resorting to an exploratory method and multidisciplinary theoretical framework allowed us to shed light on aspects that we would not have considered otherwise. Those results could not have been reached without the proper settings, i.e., the action-research projects in which researchers had continuous access to the field (actors and courses of action) and were participating in the process. Observation and analysis of our experiments over eight years in the same professional field allowed us to produce numerous case descriptions and to pinpoint several factors that can potentially influence 3D geovisualizations’ effects in collective settings. Working from empirical data gathered on several use case involving the same actors over several years, we were able to produce coherent results, i.e., that were validated by all the gathered data (i.e., all the case studies) [30,31].
The exploratory method was particularly useful since we had not found in the scientific literature strong hypothesis that could be tested. The process of analysis is, however, very time consuming since data need to be analyzed regularly in order to produce hypothesis that can be tested through further interviews and during meetings. It thus takes time to formulate relevant hypothesis. On the other hand, the effort made to confront any new hypothesis to all gathered data allow for the production of results validated by numerous case studies. This was meant to produce strong empirical evidence, which were lacking when we first started our projects, so as to be able to propose new hypothesis to be tested. Nevertheless, we could not anticipate on our findings and could not design an evaluative framework that would allow us to produce more quantitative results. Our exploratory method is a first step toward the identification and study of the factors that influence 3D geovisualizations’ effects in practice. Working more deeply on those factors should help with the study of 3D geovisualizations’ uses.
The fact that our action-research projects were led over many years was also key to be able to have several case studies to observe. We have used similar 3D geovisualizations in all our case studies, since one operational goal of the project was to produce an accessible and easily reproducible methodology that could be used by any actor along the Rhône river basin. We also use them in similar situations (elaboration of a flood mitigation plan) with similar actors. We would thus try to use each case to validate hypothesis formulated through our previous analysis. As a consequence, the ability to participate in successive experiments allowed us to strengthen our findings by trying to test them in several meetings. Evidently, experiments in real life cannot be as controlled as those in laboratories: it is never the same meeting, involved actors have diverse backgrounds and strategies, and each district has its specificities as far as flood risk is concerned. On the other hand, as said, being confronted by many case studies and observations, the results were validated by empirical data and can provide a strong basis for further studies, although they only apply to one type of 3D geovisualization.
Another key aspect of our approach, which accounts for our results, is its multidisciplinary quality. Ethnographic observation was used by researchers in geographical information, particularly observing oral comments, gestures, and postures of users of the 3D geovisualizations, in relation to actors’ strategies in the context of flood mitigation planning [63]. Ethnographic observation was also used by researchers in communication studies, but with different objectives (see Section 2.2.2), based on a totally different theoretical framework and, as a consequence, shed light on other factors influencing the roles of 3D geovisualizations in practice. Those factors were not taken into account by researchers in geographical information in the first years of the project, because they were lacking the appropriate theoretical apparatus. This is why we firmly believe in the need to produce further evaluation together with scientists from other disciplines, so as to gain a better understanding of some factors influencing the performativity of 3D geovisualizations in context.
Since we do not have enough data to have a representative sample, we cannot offer direct generalization from our findings. Nonetheless, we can compare our study with other studies. We found such comparisons particularly useful in order to open new research perspectives, especially regarding the use and definition of several notions that are commonly used in order to analyze 3D geovisualizations’ uses.
Firstly, ethnographic observations put forward the necessity to take into account the multiplicity of uses of a single 3D geovisualization. These uses can be defined in relation to their context, in a given time and place, and can very well be transitory, that is to say temporary or (very) brief (hence the need for careful ethnographic observation). The 3D geovisualizations can thus be used for diverse and sometimes antagonistic objectives. In the context of urban planning, the same 3D geovisualization can very well be used among several actors to discuss a project and then with other actors to impose on them a specific vision of a project, depending especially on how the 3D geovisualization is presented (and if its appropriation by the public is facilitated) [63]. This explains why empirical studies that try to link one use or another to one or several characteristics of a 3D geovisualization (usually its level of detail, its level of “realism”, and its level of interactivity) can easily produce contradictory results, depending on the considered or observed context of use (actors’ strategies, physical setting of the meeting, offered framework allowing the user to trust, understand, or even manipulate the geovisualization, etc.).
Thus, trying to link one use to one technical characteristic does not seem a very fruitful way for producing knowledge about 3D geovisualizations’ uses. This is all the more true since even our schematic 3D geovisualizations are complicated techno-semiotic devices, mixing diverse data, which require several cognitive skills to be understood, let alone explored. In our case, we would not had been able to define a single level of detail for our geovisualization since we used different level of details for each data set, according to its anticipated role in the comprehension of a flood and dispersion of water flows on the considered territory. The great variety of data and available tools and modelling techniques would require a huge number of studies to try to explore any possibility.
On the other hand, using only simplified criteria or generic notions to define types of 3D geovisualizations must be done with care. In the case of 3D geovisualizations in urban planning, as stated above, many scientists have stated that 3D geovisualizations, being more “realistic” than 2D geovisualizations, were more easily understood by anyone. Aside from the fact that the very notion of “realism” is difficult, if not impossible, to define clearly [46], our empirical data suggest that apprehending a 3D geovisualization is not always easy and that some individuals can need time and help to understand it. Nevertheless, observations and interviews throughout our project also acknowledged that 3D geovisualizations were preferred by many, and that they stated they understood hydraulic data better thanks to 3D geovisualization. This can be easily explained by analyzing more closely the differences between 2D and 3D representations and their perception. Indeed, there is no contradiction between those two facts. In the specific case of flood mitigation plans, in the French context, water height maps are often drawn on a 2D representation of the cadastre, so as to be able to see precisely which parcel can be potentially flooded and will be submitted to specific regulations. Since parcels’ limits are administrative boundaries, they do not always correspond to physical elements of the territory and many individuals are not used to visualize cadastre or using graphical representation of cadastre in their daily life. In addition, those technical maps do not contain many landmarks to help people find their way into the represented data. Those maps do not contain an explicit representation of the topography of the territory and its built structures (buildings and infrastructure) so that the understanding of how a flood invades a territory is not easy to derive from this kind of maps (and it is not what they are aimed at). In this context, 3D geovisualizations made specifically to help citizens understand a flood and its water flows on their territory simply offers the necessary point of reference. 3D modelling of the territory offering a more straightforward representation of topography is also particularly useful in the case of flood risk, since dispersion of water flows depends on topography. Nevertheless, this does not validate the hypothesis that only the visualization of geographical information in 3D is sufficient for anyone to better understand a water height map. Trying to understand the various factors that influence 3D geovisualizations’ effects is thus needed to be able to propose valid statements and hypothesis and to prevent researchers from producing incorrect generalization.
Added to the great variety of produced and potential geovisualizations, the very notion of use is to be questioned, as already advised by sociologists working on digital visualization interfaces. As Pascal Lécaille and Dominique Vinck [34] stated in their study of the roles of 3D digital visualization for car engineering, we have to go beyond a simplistic vision that would consider a technical device as an ensemble that cannot be decomposed and that is aimed at a unique use. On the contrary, digital tools require a pragmatic and situated approach:
“Tools, in our case digital, allow, on the contrary, uses that are numerous and continually deferred so that it is impossible to apprehend their use on a globally (…) The very notion of situation explodes into many places, and all the more since digital tools are destined to be multisite (…) It is thus necessary to set aside the simplistic vision of context of use, and, on the contrary, to apprehend all the activities, tools and human actors as well as processes that articulate them.” [34] (p. 12)
As a consequence, the study of 3D geovisualizations’ uses is to be fed by the analysis of the technical device itself as by the analysis of its performance in a course of action. Analysis of the performance in practice is key to understand their effects and to design relevant geovisualizations. Indeed, as shown by the analysis of level of involvements of citizens from a communication studies perspective (see Section 3.2.2), some factors (necessary guarantees so that the public trusts the visualization, design of an interactive framework, etc.) that can be included in the visualization framework (i.e., during its design) can be key to their appropriation by their audience. The necessity of in vivo studies seems particularly needed in this respect as already stated by Lloyd [64] in the case of geovisualizations used in urban planning. In the field of cartography, similar ways of studying maps have been proposed by some authors in critical geography [65,66]. Having pinpointed the various limits of the study of map relying, on one hand, on an analysis of its content and, on the other hand, on how a single individual can decipher, they propose a study of practices developed by users of maps.
Another striking result from the confrontation with communication studies is the necessity to redefine the notion of “communication”, often used by researcher in geographical information in a narrow way. Although the limits of the communication paradigm based on information theory used to study uses of geographical information have already been described [67], many studies define communication as the transmission of a piece of information from a transmitter (the visualization) to a receiver (the user) [13]. Results from the evaluation by researchers in communication studies opens new perspective to define what is entailed in a communicative framework, its different dimension and factors influencing the roles geovisualizations can play and can be useful for the design or evaluation of geovisualizations.

5. Conclusions

As a conclusion, we would like to put the stress on the importance of pluridisciplinary studies, since one discipline simply cannot provide all the theoretical elements necessary in order to understand the performance of 3D geovisualizations in a given context. In vivo studies allow researchers to shed light on factors that can be useful for the design of geovisualizations, especially in terms of communication framework. Although they cannot be as controlled as studies conducted in laboratories, in vivo studies can put forward some factors that would not be observable in a controlled setting. This is why we feel those approaches are complementary and in vivo studies particularly useful in the field of planning where tasks performed with the help of 3D geovisualizations can be numerous and complex as many dimensions of a territory or phenomenon are considered by heterogeneous actors with various objectives and strategies.
In our studies, we could not link our results to one specific technical parameter of our 3D geovisualization. Nevertheless, the presented communicational approach allows to articulate different uses of 3D geovisualizations that were sometimes perceived as contradictory, when observed effects were only linked to 3D geovisualizations’ technical characteristics. As a consequence, once the analysis from a communication studies perspective is performed, during an experiment, then some recommendations can be suggested to improve geovisualizations and the communicative framework they offer to their public. This is why we feel our results can be useful for the design of a framework to evaluate 3D geovisualizations’ uses in vivo.
Issues raised by observing uses with citizens (truthfulness, necessary cognitive skills, information needed to understand the model, etc.) can then be translated into characteristics of a geovisualization or into methodology of presentation and use when human mediation is possible. As far as online geovisualizations without mediation are concerned, tackling those issues remains crucial. In this respect, benefiting from theoretical framework and evaluation from other disciplines can offer the possibility to integrate into the design or evaluation of geovisualizations factors which would otherwise have been thought of as random, uncontrollable, or depending on the individuals using the devices. Further studies including the analysis of those factors in their evaluative framework are needed to better understand their influence and to provide more quantitative data about them. In this respect, our exploratory approach is only one step toward a more comprehensive approach of 3D geovisualizations uses. In terms of virtual reality and augmented reality, further studies are also needed on the role of users’ body and its influence on how a user engages with a device. From a communication studies perspective, analysis of public events involving in situ visualization of 3D temporal visualizations and immersive visualizations through virtual reality headset shed light on the role of the users’ body in their appropriation of the devices. Since we had only two experiments, we lack empirical data to strengthen our first findings so that further studies are needed.

Author Contributions

Conceptualization, Florence Jacquinod and Julia Bonaccorsi; Methodology, Florence~Jacquinod and Julia Bonaccorsi; Formal analysis, Florence~Jacquinod and Julia Bonaccorsi; Investigation, Florence Jacquinod and Julia Bonaccorsi; Data curation, Florence Jacquinod and Julia Bonaccorsi; Writing—original draft preparation, Florence Jacquinod and Julia Bonaccorsi; Writing—review and editing, Florence Jacquinod and Julia Bonaccorsi; Project administration, Florence Jacquinod; Funding acquisition, Florence Jacquinod.




We are thankful to our many collaborators on those projects.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the scientific design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.


  1. Çöltekin, A.; Bleisch, S.; Andrienko, G.; Dykes, J. Persistent challenges in geovisualization—A community perspective. Int. J. Cartogr. 2017, 3, 115–139. [Google Scholar] [CrossRef]
  2. Ervin, S.M. Digital Landscape Modeling and Visualization: A Research Agenda. Landsc. Urban Plan. 2001, 54, 49–62. [Google Scholar] [CrossRef]
  3. Joliveau, T. Mémoire pour l’Habilitation à Soutenir des Recherches. Parcours 4. Planification Participative et Géomatique. Habilitation à diriger des recherches (HDR). Available online: (accessed on 3 February 2019).
  4. Joliveau, T. Voir le monde comme il paraît ou les enjeux des visualisations réalistes dans la gestion territoriale. Available online: (accessed on 3 February 2019).
  5. Bishop, I.D.; Lange, E. Visualization in Landscape and Environmental Planning: Technology and Applications; Taylor & Francis: London, UK, 2005; ISBN 0-415-30510-1. [Google Scholar]
  6. Sheppard, S.R.J. Landscape visualisation and climate change: The potential for influencing perceptions and behaviour. Environ. Sci. Policy 2005, 8, 637–654. [Google Scholar] [CrossRef]
  7. Dykes, J.; MacEachren, A.M.; Kraak, M.-J. Exploring Geovisualization; Elsevier: Amsterdam, The Netherlands, 2005; ISBN 0-08-044531-4. [Google Scholar]
  8. Riedjik, A.; van de Velde, R.J.; Pleizier, I.D.; Hoogerwerf, T.C.; van Lammeren, R.J.A.; Baltussen, W.H.M.; Jansen, J.; Wynia, P.; van Uum, J.H.; van Wilgenburg, R. Geo-Visualizations for Interactive Spatial Planning and Decision Making: From Wow to Impact; Vrije Universiteit: Amsterdam, The Netherlands, 2006. [Google Scholar]
  9. Nielsen, A. A Qualification of 3D Geovisualisation. Ph.D. Thesis, University of Aalborg, Aalborg, Denmark, 2007. [Google Scholar]
  10. Bord, J.-P. L’univers des Cartes: La Carte et le Cartographe; Mappemonde: Paris, France, 2012; ISBN 978-2-7011-5781-8. [Google Scholar]
  11. Vodoz, L. NTIC et Territoires: Enjeux Territoriaux des Nouvelles Technologies de l’information et de la Communication; Presses Polytechniques et Universitaires Romandes: Lausanne, Switzerland, 2001; ISBN 2-88074-484-9. [Google Scholar]
  12. Bailleul, H. Les nouvelles formes de la communication autour des projets urbains: Modalités, impacts, enjeux pour un débat participatif. Analyse du rôle des images dans le débat participatif autour de deux projets urbains en France. Metropoles 2008, 3, 99–139. [Google Scholar]
  13. Cauvin, C.; Escobar, F.; Serradj, A. Cartographie Thématique 5. Des Voies Nouvelles à Explorer; Hermès Science, Lavoisier: Paris, France, 2008. [Google Scholar]
  14. Lardon, S.; Maurel, P.; Piveteau, V. Représentations Spatiales et Développement Territorial; Hermes Science: Paris, France, 2001. [Google Scholar]
  15. Cabral, S.; Campos, J.N.B.; Silveira, C. Hydrologic and Hydraulic Modelling Integrated with Gis: A Study of the Acaraú River Basin–CE. J. Urban Environ. Eng. (JUEE) 2014, 8, 167–174. [Google Scholar] [CrossRef]
  16. Aiello, A.; Adamo, M.; Canora, F. Remote sensing and GIS to assess soil erosion with RUSLE3D and USPED at river basin scale in southern Italy. CATENA 2015, 131, 174–185. [Google Scholar] [CrossRef]
  17. Curebal, I.; Efe, R.; Ozdemir, H.; Soykan, A.; Sönmez, S. GIS-based approach for flood analysis: Case study of Keçidere flash flood event (Turkey). Geocarto Int. 2016, 31, 355–366. [Google Scholar] [CrossRef]
  18. Demir, V.; Kisi, O. Flood Hazard Mapping by Using Geographic Information System and Hydraulic Model: Mert River, Samsun, Turkey. Adv. Meteorol. 2016, 2016, 1–9. [Google Scholar] [CrossRef]
  19. Nishio, M.; Mori, M. Hydrologic analysis of a flood based on a new Digital Elevation Model. ISPRS Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2015, XL-7/W4, 127–134. [Google Scholar] [CrossRef]
  20. Lonergan, C.; Hedley, N. Navigating the future of tsunami risk communication: Using dimensionality, interactivity and situatedness to interface with society. Nat. Hazards 2015, 78, 179–201. [Google Scholar] [CrossRef]
  21. Saher, F.N.; Ali, N.B.M.; Bin Wan Ishak, W.M.F.; Bin Bidin, M.A.N. Development of an Intelligent Decision Support System for Flood Mitigation in the Pahang River. Int. J. Sustain. Land Use Urban Plan. (IJSLUP) 2015, 2. [Google Scholar] [CrossRef]
  22. Barbier, R. La Recherche Action; Economica: Paris, France, 1996; ISBN 2-7178-3062-6. [Google Scholar]
  23. Mucchielli, A. Méthodologie d’une recherche qualitative. In Dictionnaire des Méthodes Qualitatives en Sciences Humaines et Sociales; Armand Colin: Paris, France, 2004. [Google Scholar]
  24. Findeli, A. La Recherche-Projet: Une Méthode Pour la Recherche en Design; Swiss Design Network: Bern, Switzerland, 2005; pp. 40–51. [Google Scholar]
  25. Findeli, A.; Coste, A. De la Recherche-Création à la Recherche-Projet: Un Cadre Théorique et Méthodologique pour la Recherche Architecturale. Available online: (accessed on 3 February 2019).
  26. Soulé, B. Observation participante ou participation observante? Usages et justifications de la notion de participation observante en sciences sociales. Rech. Qual. 2007, 27, 127–140. [Google Scholar]
  27. Vinck, D. Ingénieurs au Quotidien—Etnographie de L’activité de Conception et D’innovation; Presses Universitaires de Grenoble: Grenoble, France, 1999; ISBN 2-7061-0876-2. [Google Scholar]
  28. Cefaï, D. L’enquête de Terrain; Recherches. Série Bibliothèque du MAUSS; Éditions la Découverte: Paris, France, 2003; ISBN 2-7071-4072-4. [Google Scholar]
  29. Cefaï, D. L’engagement Ethnographique; En temps & lieux; 16; Éd. de l’École des hautes études en sciences sociales: Paris, France, 2010. [Google Scholar]
  30. Laperrière, A. La théorisation ancrée (grounded theory): Démarche analytique et comparaison avec d’autres approches apparentées. In La Recherche Qualitative—Enjeux Épistémologiques et Méthodologiques; Poupart, J., Deslauriers, J.-P., Groulx, L.H., Lapperrière, A., Mayer, P., Pires, A.P., Eds.; Gaetan Morin: Montréal, QC, Canada, 1997. [Google Scholar]
  31. Laperrière, A. Les critères de scientificité des méthodes qualitatives. In La Recherche Qualitative—Enjeux Épistémologiques et Méthodologiques; Poupart, J., Deslauriers, J.-P., Groulx, L.-H., Laperrière, A., Mayer, R., Pires, A.P., Eds.; Gaëtan morin éditeur: Montreal, QC, Canada, 1997; pp. 365–389. ISBN 2-89105-661-2. [Google Scholar]
  32. Laureillard, P.; Vinck, D. Les représentations graphiques. In Ingénieurs au Quotidien; Presses Universitaires de Grenoble: Grenoble, France, 1999; ISBN 2-7061-0876-2. [Google Scholar]
  33. Vinck, D.; Blanco, E.; Garrigou, A.; Jeantet, A.; Lavoisy, O.; Lécaille, P. Digital Mock-Up Visualisation in Product Conception and Downstream Processes. Task 1.5 Virtual World Sociological and Ergonomical Aspects (SEA); European Commission: Brussels, Belgium, 1999; p. 49. [Google Scholar]
  34. Lécaille, P.; Vinck, D. Digital Mock-Up Visualisation in Product Conception and Downstream Processes. Task 1.5 Aspects Sociologiques du Monde Virtuel; European Commission: Brussels, Belgium, 2000; p. 59. [Google Scholar]
  35. Callon, M. Eléments pour une sociologie de la traduction. La domestication des coquilles Saint-Jacques et des marins-pêcheurs dans la Baie de Saint-Brieuc. Ann. Sociol. 1986, XXXVI, 169–208. [Google Scholar]
  36. Akrich, M. Comment décrire les objets techniques? Tech. Cult. 1987, 49–64. [Google Scholar] [CrossRef]
  37. Akrich, M. Les formes de la médiation technique. Réseaux 1993, 60, 87–98. [Google Scholar]
  38. Akrich, M.; Callon, M.; Latour, B. Sociologie de la Traduction—Textes Fondateurs; Presses de l’École des mines de Paris: Paris, France, 2006; ISBN 978-2-911762-75-8. [Google Scholar]
  39. Latour, B.; Woolgar, S. La vie de Laboratoire. La Production des Faits Scientifiques; Editions La Découverte: Paris, France, 2006. [Google Scholar]
  40. Latour, B. Changer de Société, Refaire de la Sociologie; Editions La Découverte: Paris, France, 2007. [Google Scholar]
  41. Vinck, D. Approches Sociologiques de la Cognition et Prise en Compte des Objets Intermédiaires. Available online: (accessed on 7 December 2010).
  42. Vinck, D.; Penz, B. L’équipement de l’organisation Industrielle. Les ERP à L’usage; Hermès science publication, Lavoisier: Paris, France, 2008. [Google Scholar]
  43. Vinck, D. De l’objet intermédiaire à l’objet-frontière—Vers la prise en compte du travail d’équipement. Revue D’anthropologie des Connaissances 2009, 3, 51–72. [Google Scholar] [CrossRef]
  44. Moquay, P.; Lardon, S.; Marcelpoil, E.; Piveteau, V. Représentations spatiales et proximité institutionnelle dans les processus de développement territorial. In Proximités et Changements Socio-Économiques dans les Mondes Ruraux; Torre, A., Filippi, M., Eds.; Un Point sur; Institut National de la Recherche Agronomique: Paris, France, 2005; pp. 201–214. ISBN 2-7380-1183-7. [Google Scholar]
  45. Jacquinod, F.; Joliveau, T. The Adding Value of the Concept of Intermediary Object to Describe the Use of 3D Geovisualizations in Urban Planning. In Proceedings of the International Cartographic Conference, Paris, France, 3–8 July 2011. [Google Scholar]
  46. Jacquinod, F. Production, Pratique et Usages des Géovisualisations 3D dans l’aménagement du Territoire. Ph.D. Thesis, Université de Saint-Etienne, Saint-Etienne, France, 2014. [Google Scholar]
  47. Verhaegen, P. Les dispositifs techno-sémiotiques. Signes ou objets? Hermès 1999, 25, 111–121. [Google Scholar] [CrossRef]
  48. Bonfils, P. Immersion et environnements numériques: Une approche méthodologique par l’expérience vécue. Quest. Commun. 2015, 27, 261–277. [Google Scholar] [CrossRef]
  49. Gentès, A.; Jutant, C. Expérimentation technique et création: L’implication des utilisateurs dans l’invention des médias. Commun. Lang. 2011, 97–111. [Google Scholar] [CrossRef]
  50. Jacobi, D. Du discours scientifique, de sa reformulation et de quelques usages sociaux de la science. Langue Française 1984, 1, 38–52. [Google Scholar] [CrossRef]
  51. Jacobi, D. Figures et figurabilité de la science dans des revues de vulgarisation. Langages 1984, 75, 23–42. [Google Scholar] [CrossRef]
  52. de Bideran, J.; Fraysse, P. Guide numérique et mise en scène du territoire, entre médiation patrimoniale et stratégie de communication touristique. Études de Commun. Lang. Inf. Médiat. 2015, 77–96. [Google Scholar] [CrossRef][Green Version]
  53. Jutant, C. S’ajuster, Interpréter et Qualifier une Pratique Culturelle: Approche Communicationnelle de la Visite Muséale. Ph.D. Thesis, Université d’Avignon, Avignon, France, 2011. [Google Scholar]
  54. Ollivier-Yaniv, C.; Rinn, M. (Director) Communication de l’Etat et Gouvernement du Social. Pour une Société Parfaite? Presses Universitaires de Grenoble: Grenoble, France, 2005; ISBN 2706115467.
  55. Pailliart, I. La publicisation de la Science: Exposer, Communiquer, Débattre, Publier, Vulgariser: Hommage à Jean Caune; Presses universitaires de Grenoble: Grenoble, France, 2005; ISBN 978-2-7061-1283-6. [Google Scholar]
  56. Dondero, M.G.; Fontanille, J.; Tabler, J. The Semiotic Challenge of Scientific Images. A Test Case for Visual Meaning; Legas Publishing: Ottawa, ON, Canada, 2014; ISBN 978-1-897493-48-9. [Google Scholar]
  57. Bonaccorsi, J. Approches sémiologiques du web. In Analyser le Web en Sciences Humaines et Sociales; Armand Colin: Paris, France, 2016; pp. 132–150. [Google Scholar]
  58. Bonaccorsi, J.; Jacquinod, F.; Vogt, M. Regard pluridisciplinaire sur les usages sociaux de géovisualisations 3D pour la sensibilisation au risque d’inondation: Un exemple rhodanien. Revue Internationale de Géomatique 2018, 28, 69–94. [Google Scholar] [CrossRef]
  59. Jacobi, D. La maquette entre reconstitution savante et récit imaginaire dans les expositions archéologiques. La Lettre de l’OCIM 2009, 123, 15–23. [Google Scholar] [CrossRef]
  60. Belaën, F. L’immersion dans les musées de science: Médiation ou séduction? Cult. Musées 2005, 5, 91–110. [Google Scholar] [CrossRef]
  61. Le Marec, J. Publics et Musées: La Confiance Éprouvée; L’Harmattan: Paris, France, 2007; ISBN 978-2-296-04338-1. [Google Scholar]
  62. MacEachren, A.M. How Maps Work: Representation, Visualization, and Design; Guilford Press: New York, NY, USA, 1995. [Google Scholar]
  63. Jacquinod, F.; Langumier, J. Géovisualisations 3D en action dans l’aménagement du territoire. Stratégies et usages de l’outil à l’occasion de l’étude d’un Plan de Prévention des Risques Inondations. Géocarrefour 2010, 85, 303–311. [Google Scholar] [CrossRef]
  64. Lloyd, D. Evaluating Human-Centered Approaches for Geovisualization; City University London: London, UK, 2009. [Google Scholar]
  65. Del Casino, V.J., Jr.; Del Casino, J., Jr. Beyond The ‘Binaries’: A Methodological Intervention for Interrogating Maps as Representational Practices. ACME 2005, 4, 34–56. [Google Scholar]
  66. Crampton, J.W. Mapping: A Critical Introduction to Cartography and GIS; Blackwell Critical Introductions to Geography; Royaume-Uni: Chichester, UK, 2010; ISBN 978-1-4051-2172-9. [Google Scholar]
  67. Poore, B.S.; Chrisman, N.R. Order from Noise: Toward a social Theory of Geographic Information. Ann. Assoc. Am. Geogr. 2006, 96, 508–523. [Google Scholar] [CrossRef]
Figure 1. Aerial view from a 3D geovisualization of a district impacted by a simulated flood. Image prepared by F. Jacquinod from data from IGN and CNR.
Figure 1. Aerial view from a 3D geovisualization of a district impacted by a simulated flood. Image prepared by F. Jacquinod from data from IGN and CNR.
Ijgi 08 00084 g001
Figure 2. Aerial view from a 3D geovisualization of a district impacted by a simulated flood. (a) and (b) show how water heights were represented, through a 3D transparent volume. Images prepared by F. Jacquinod from data from IGN and CNR.
Figure 2. Aerial view from a 3D geovisualization of a district impacted by a simulated flood. (a) and (b) show how water heights were represented, through a 3D transparent volume. Images prepared by F. Jacquinod from data from IGN and CNR.
Ijgi 08 00084 g002
Figure 3. Two images from two different moments in the simulated flood, from a high vantage point: (a) shows water height at day three (three days after the beginning of the simulated flood); (b) shows water heights at day five.
Figure 3. Two images from two different moments in the simulated flood, from a high vantage point: (a) shows water height at day three (three days after the beginning of the simulated flood); (b) shows water heights at day five.
Ijgi 08 00084 g003
Figure 4. Interface proposed to the public on tablets for the visualization of the simulated flood from a chosen point of view. At the top is a temporal navigation panel to choose a point in the timeline of the simulated flood. On the right is a panel that can be hidden or displayed and that provide a link between water flow (m3 per second) and historical floods that have occurred on the territory. It also enables the user to access historical pictures for some historical floods. At the bottom is a spatial navigation panel to go to the left or right, look up or down and zoom in or out. The information tool gives more details on how the flood was simulated.
Figure 4. Interface proposed to the public on tablets for the visualization of the simulated flood from a chosen point of view. At the top is a temporal navigation panel to choose a point in the timeline of the simulated flood. On the right is a panel that can be hidden or displayed and that provide a link between water flow (m3 per second) and historical floods that have occurred on the territory. It also enables the user to access historical pictures for some historical floods. At the bottom is a spatial navigation panel to go to the left or right, look up or down and zoom in or out. The information tool gives more details on how the flood was simulated.
Ijgi 08 00084 g004
Figure 5. Example of the 360° images produced for immersive visualization (with a smartphone inserted into a virtual reality (VR) headset). Purple and green panels can be used to navigate between different moments in time (during the simulated flood). Grey panel gives information about water flows (m3 per second), point in time (number of days since the beginning of the flood), and the corresponding level of vigilance (here red).
Figure 5. Example of the 360° images produced for immersive visualization (with a smartphone inserted into a virtual reality (VR) headset). Purple and green panels can be used to navigate between different moments in time (during the simulated flood). Grey panel gives information about water flows (m3 per second), point in time (number of days since the beginning of the flood), and the corresponding level of vigilance (here red).
Ijgi 08 00084 g005
Figure 6. Pictures taken during the events: (a) shows inhabitants consulting the 3D geovisualization with a tablet from a high vantage point (same viewpoint as the one selected for the images of the simulated flood they are visualizing); (b) shows inhabitants using the VR headset (background), while another is being shown the tablet by a mediator (foreground).
Figure 6. Pictures taken during the events: (a) shows inhabitants consulting the 3D geovisualization with a tablet from a high vantage point (same viewpoint as the one selected for the images of the simulated flood they are visualizing); (b) shows inhabitants using the VR headset (background), while another is being shown the tablet by a mediator (foreground).
Ijgi 08 00084 g006

Share and Cite

MDPI and ACS Style

Jacquinod, F.; Bonaccorsi, J. Studying Social Uses of 3D Geovisualizations: Lessons Learned from Action-Research Projects in the Field of Flood Mitigation Planning. ISPRS Int. J. Geo-Inf. 2019, 8, 84.

AMA Style

Jacquinod F, Bonaccorsi J. Studying Social Uses of 3D Geovisualizations: Lessons Learned from Action-Research Projects in the Field of Flood Mitigation Planning. ISPRS International Journal of Geo-Information. 2019; 8(2):84.

Chicago/Turabian Style

Jacquinod, Florence, and Julia Bonaccorsi. 2019. "Studying Social Uses of 3D Geovisualizations: Lessons Learned from Action-Research Projects in the Field of Flood Mitigation Planning" ISPRS International Journal of Geo-Information 8, no. 2: 84.

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop