3.1. 5W1H Model-Based Metadata for Context-Aware VR/AR Services
The 5W1H model-based metadata has the extensibility to be used in tourism, education, and heritage management applications, especially in heritage sites. It also has interoperability that can be utilized in various software and applications. This research attempts to utilize the proposed metadata for remote diagnosis in VR. The metadata proposed in this study is used for retrieval and visualization of information according to the context in VR that reflects realistic coordinates. Therefore, it can also be used for AR-based on-site diagnosis.
As the VR/AR service is widely used, research on the metadata standard that can increase the reusability and interoperability of VR/AR content is actively being performed. This study proposes a metadata structure to improve the reusability and interoperability of context-aware VR/AR content metadata. The metadata structure consists of POIs, content, and xAnchor elements. It is designed to solve the problem of existing metadata that can only be used in a specific domain and to enhance scalability. We also propose the POI linker and the content linker that represent the relation among POIs and among contents and explain the description of relation in detail. The proposed metadata structure can contribute to providing a context-aware VR/AR service by defining the elements, property, and relations necessary for context-aware VR/AR service.
The metadata proposed in the previous study has four problems in implementing context-aware VR/AR services. First, there is difficulty in authoring related services and contents because the metadata considering context-awareness is not defined. Second, it can only be used for a specific domain, which has a problem of lack of flexibility. Third, there is no element to define the relation between VR/AR content and it is difficult to provide related content. Finally, sensor information required for a context-aware VR/AR service is not defined in detail. This study proposes the metadata structure to overcome the above problems and proves the metadata structure through a prototype.
We define the content to be used in the augmented reality service as VR/AR scenes, which consists of POI, xAnchor, Content, linker, and linked content. The POI element consists of who (creator), where (location), what (ID, location), and how (volume, accessibility) to comprise the types of properties that can identify the VR/AR content’s location and status. The content element consists of who (author and uploader), when (created time and modified time), what (URI-Uniform Resource Identifier, metadata type, and MIME type-internet media type), and how (accessibility and commercial) to find and generate the content that the VR content creator wants (Figure 1
The xAnchor element consists of who (creator), when (created time and modified time), where (location), what (ID, POIID, content URI, visible distance, title, description, and coordinate system), and how (animation type, animation period, scale, and rotation) and manages to render methods. Commercial companies, such as Microsoft and Google, defined the anchor as an element that contains only information about location and direction. Microsoft defined the anchor as a common reference frame that allows multiple users to place digital contents in the same physical location [25
]. Google explains that an anchor keeps an object in the same location and orientation in space and can maintain the illusion of a virtual object placed in the real world [26
]. On the other hand, our proposed xAnchor contains extended information for interactions and content filtering, such as tags.
The proposed metadata contain media representation information in VR. This allows the author of the content to define the content to be experienced by the user specifically. In addition, this study includes a tag for searching POI-xAnchor-content so that users can search for desired information by filtering. In addition, the accessibility of content is added to distinguish it as public, friend, and private, as well as to distinguish commercial versus noncommercial content. The xAnchor elements are useful for expressing, storing, and managing content filtering.
The metadata structure proposed in this study has extensibility to be used in tourism, education, and risk management applications, especially in heritage sites. It also has interoperability that can be utilized in various software and applications. This study attempts to utilize 5W1H model-based metadata for retrieval and visualization of information according to context. The proposed metadata structure contributes to risk management of architectural heritage by being used in VR/AR.
3.2. Linker Metadata for Reflecting Domain Knowledge and Retrieving Related Information
We propose the domain ontology that expresses relations among information and relations among POI. We reflect them in the linker metadata. For these metadata, we conceptualized a domain ontology that represents domain knowledge of Korean architectural heritage. This ontology expresses the composition of building components and the cause–effect relationship among components and environmental factors (Figure 2
We divided the linker, which plays a role in linking the elements constituting VR/AR, into POI linker, xAnchor linker, and content linker. The POI linker is used to define relationships among real-world coordinates. The xAnchor linker represents the relationship between VR/AR scenes. The xAnchor linker establishes a connection among VR/AR scenes. The content linker is responsible for connecting external content. The content linker metadata, which reflects the risk ontology, defines the contents required for the damage type. The application retrieves information from the external database based on this metadata.
We want to support on-site risk management by linking VR/AR scenes using the xAnchor linker metadata. We designed the ontology for the cause–effect relation of risk and constructed an inference module that applies ontology-based rules. When a client enters a domain ontology-based rule, the visualization of VR/AR scenes’ relations can be performed. The xAnchor linker consists of what (ID, FromID, ToID), who (creator), and how (relation) and represent the relation among VR/AR scenes, for example, in the case of column corrosion, connect rafters, foundation, and floor (Figure 3
We added the content linker metadata that retrieves related information by using domain knowledge-based ontology. Specifically, we defined the content linker metadata as what (ID, FromURL, ToURL, xAnchorID), who (Creator), and how (Relation). We implemented rules to provide and update the data needed to analyze specific damage based on domain ontology. For example, if the column component has corrosion damage, the weather information of the meteorological agency can be retrieved as related information and visualized (Figure 4
3.3. Heritage Building Information Metadata for Risk Management
The heritage building information metadata is divided into building information, component information, and management information classes. The heritage building information package is defined by the 5W1H model showing basic descriptions and properties for risk management. For the definition of the heritage building information content, the building and component class of Korean wooden architectural heritage are defined to represent the structure of architectural heritage and relationships among components and risks (Figure 5
To define the component taxonomy of Korean wooden architectural heritage which is devoted to description and semantic structuring, we referred to architectural books, repair reports, and expert interviews and we applied the CIDOC Conceptual Reference Model (CRM) standards and vocabularies which was designed for cultural heritage information and the metadata of a portal system of the Korean Cultural Heritage Administration. For the ontology model, we applied the Monument Damage Information System (MONDIS) to represent the cause–effect relation of damages.
We define building and component class properties of Korean wooden architectural heritage for 4-D description and semantic structuring of architectural heritage. We refer to the 4-D description expression of CIDOC-CRM and the metadata of a portal system of the Korean Cultural Heritage Administration. We referred to the Monument Damage Information System (MONDIS) to represent risk and relation among risks [6
This metadata describes the detailed risk management information rather than the existing heritage building information metadata. It is advantageous for risk management to check past risk management information and decision making of risk responses. For example, the management information class includes the cause of the damage so that heritage manager can refer to an existing case in the future. This study suggests specialized metadata for risk management of architectural heritage.
Moreover, the heritage building information metadata proposed in this study represent relations between components or information. In particular, Category1 and Category2 are distinguished to express the hierarchy of wooden architectural heritage in East Asia. Category2 means the component group and Category1 means the component type. For example, different components of category 2 often has a cause-effect relation. Also, in the case of management information with the same component type, it is a notable example to understand the cause of damage. The metadata proposed in this study provides heritage information that can be used for risk management in conservation/restoration and relation rather than existing metadata.
3.4. Framework of Remote Diagnosis in Virtual Reality
The site manager and nonexpert use the mobile Geographic Information System (GIS)-based geo-tagged content tool in the field to upload pictures of a heritage building’s damage, descriptions, and tags. Specifically, pictures contain tags that include the uploader’s expertise, kind of damage, and severity that are attributes of the heritage building information metadata. This information is stored in a geo-tagged content database by a geospatial server. The geo-tagged database stores and manages on-site information that is accumulated, such as images and texts.
The expert on the remote site uses the Head Mounted Display (HMD) and connects to the remote VR application to enter the domain expertise and tasks. The server receives the user information (domain expertise and task) and retrieves content from the geo-tagged content database that contains the tag appropriate for the user information. Content includes historical and on-site information that is stored in the HBIM database.
The filtered content is provided to the expert according to selected expertise and task. If the expert selects his or her task and domain expertise, the application provides the risk management information according to the selected task and domain expertise. Experts can see all of the information needed for monitoring in a virtual environment, but too much information can overload to experts.
Among the filtered contents, the POI including the management information is arranged by color (green, yellow, and red) according to severity. The expert can determine the order of components by visualizing the condition color and the connection relationship of the component that is related to the damage of the component. The expert can also check related information with the linker content metadata according to the tags of uploaded information. Finally, the expert checks the diagnosis results in the checklist and stores them in the geo-tagged content database (Figure 6