Symmetric Graph-Based Visual Question Answering Using Neuro-Symbolic Approach

Abstract

As the applications of robots expand across a wide variety of areas, high-level task planning considering human–robot interactions is emerging as a critical issue. Various elements that facilitate flexible responses to humans in an ever-changing environment, such as scene understanding, natural language processing, and task planning, are thus being researched extensively. In this study, a visual question answering (VQA) task was examined in detail from among an array of technologies. By further developing conventional neuro-symbolic approaches, environmental information is stored and utilized in a symmetric graph format, which enables more flexible and complex high-level task planning. We construct a symmetric graph composed of information such as color, size, and position for the objects constituting the environmental scene. VQA, using graphs, largely consists of a part expressing a scene as a graph, a part converting a question into SPARQL, and a part reasoning the answer. The proposed method was verified using a public dataset, CLEVR, with which it successfully performed VQA. We were able to directly confirm the process of inferring answers using SPARQL queries converted from the original queries and environmental symmetric graph information, which is distinct from existing methods that make it difficult to trace the path to finding answers.

1. Introduction

Robots are widely utilized across various services, including education, medical, and logistics, mostly owing to the advancement of technology [1]. Educational robots are commonly used to trigger interest in students and heighten their learning motivation, while chatbots contribute to an improved learning efficiency by interacting with students [2]. In the medical field, robots demonstrate high precision and stability, thus providing various services, including surgical assistance for surgeons, rehabilitation therapy, and data analysis [3]. In logistics centers, robots are applied in sorting and transporting products, thus optimizing the overall logistics process [4]. Accordingly, service robots interact with humans in diverse settings and are applied in a wide range of fields. Numerous researchers are conducting studies on simultaneous localization and mapping (SLAM), robot vision, task planning [5], and human–robot interaction (HRI) for the development of service robots. However, relatively fewer number of studies are being conducted on task planning compared with other fields. In particular, little attention is being paid to research HRI-based task planning. Robots must be capable of executing task planning to perform high-level missions in real life, rather than performing just simple repetitive work. Furthermore, HRI-based task planning is especially important for successfully achieving the goal of a mission in an eco-system in which humans, robots, and the environment interact with each other [6].

Nonetheless, research has been conducted on task planning using robots in extreme environments, such as deep sea or space, and in varied environments including markets and factories. Cashmore et al. [7] introduced the ROSPlan framework, which is a system for planning tasks in water using Girona 500 AUV. Crosby et al. [8] proposed a robot with the ability of transporting boxes in a factory through six-step task planning. Galindo et al. [9] suggested a method for task planning using semantic maps consisting of hierarchical spatial information and semantic knowledge in a home environment. Similarly, the majority of task planning algorithms do not take the human factor into consideration. However, human intervention is necessary for task planning when performing high-level missions [10]. Regarding HRI-based task planning, Crosby et al. [10] developed a human-aware task planning system. In this particular system, human constraints were taken into account in planning to enable interactions between humans and the robot. Alami et al. [11] proposed a robot control architecture in which humans and robots share the burden of work to perform certain tasks. Especially, HRI-oriented task planning becomes a serious issue when unexpected external events occur in extreme situations in which direct interaction with humans is challenging.

To handle unexpected external events in extreme situations, robots must be able to obtain and understand information about their surrounding environments, as humans cannot always directly acquire such information. Furthermore, robots must be able to communicate with humans through scene understanding. Thus, visual question answering (VQA), i.e., the capability of question answering in the form of a natural language based on image information as input, is a significant problem in HRI-based task planning. VQA is largely studied from a connectionist approach [12], symbolic approach, and neuro-symbolic approach. Yang et al. [13] proposed stacked attention networks (SANs), in which natural language questions are answered based on images. Marino et al. [14] introduced Knowledge Reasoning with Implicit and Symbolic representations (KRISP), which is capable of executing VQA in open-domain. Mao et al. [15] designed a neuro-symbolic concept learner (NS-CL) capable of executing VQA without explicit supervision.

The connectionist approach has the advantage of carrying out VQA for various unexpected environments and questions, but the disadvantage is that it is difficult to explain the process of deducing answers. The symbolic approach can explain the process of inferring answers but entails the limitation of being unable to be used in dynamic environments. The neuro-symbolic approach combines the advantages of all methods in which answers can be deduced for various VQA, and the relevant process can also be explained [16]. However, existing neuro-symbolic-based VQA methods mostly use table databases, which causes difficulties with expanding object information flexibly and performing high-level inference. Therefore, scene information is expressed and saved as a symmetric graph in this study, which can be flexibly extended. Applying the advantages of a graph database enables the simple inference of relation information as well as high-level inference. The proposed framework is shown in Figure 1. In the scene understanding part, objects are first detected using an object detector, and then the objects’ type, color, and position information is extracted through an object feature extractor. In the question understanding part, natural language statements are converted to SPARQL to be applicable to a graph database. In the answer reasoning part, answers are inferred from a graph database through a SPARQL query. The proposed method is verified using a public dataset, CLEVR dataset [17], which is most widely used in neuro-symbolic-based VQA. The results show that answers can be successfully found for various types of queries.

2. Related Work

VQA involves understanding the situation in given images and finding answers to questions. To solve VQA problems, many researchers use VQA datasets consisting of images and complicated natural language questions and answers related to the images. Johnson et al. [17] thus proposed a dataset called CLEVR. CLEVR has details on information, attributes, and position relations of objects constituted of each scene added to the dataset, and thus can be utilized in various VQA algorithms. In this chapter, different methods for solving VQA problems using various VQA datasets are introduced based on the connectionist approach, symbolic approach, and neuro-symbolic approach.

2.1. Connectionist Models for VQA

A large number of VQA algorithms [18,19,20] which train networks using massive datasets have been developed. Li et al. [21] enabled zero-shot transfer learning through a downstream task using architecture in which an image is learned directly from raw text. Yang et al. [22] solved VQA problems by proposing an auto-parsing network (APN) in which a probabilistic graphical model (PGM) is applied to the self-attention layer. Nam et al. [23] proposed dual attention networks (DANs) for elaborate interactions between vision and language. Noh et al. [24] introduced more efficient VQA methods through a method that initializes weighted values by using pre-trained convolutional neural networks (CNN) and gated recurrent units (GRU). The neural network-based connectionist approach has the advantage of executing VQA by robustly extracting features of images and natural language statements, but the disadvantage of being unable to execute complex inference and explain the inference process.

2.2. Symbolic Models for VQA

Diverse methods for solving VQA problems have been proposed whereby symbolic techniques have been used to infer answers from the available knowledge by using inductive and deductive reasoning. Lei et al. [25] proposed a new method capable of continual learning, in which all questions related to images can be answered. Han and Wang [26] introduced a symbolic approach-based graph matching-based reasoner (GMR) that can infer answers by automatically comparing graphs, and evaluated the algorithm’s performance using the bAbI dataset. Malinowski and Fritz [27] applied discrete reasoning with uncertain prediction to automatically answering questions related to images. Symbolic models for VQA are capable of inferring answers but are vulnerable to external changes and difficult to respond to if proper actions are not taken for such changes.

2.3. Neuro-Symbolic Models for VQA

In recent years, several VQA algorithms utilizing the neuro-symbolic approach, which combines the strengths of the connectionist and symbolic approaches, have been actively developed. Amizadeh et al. [28] applied the neuro-symbolic of a logic framework first to infer queries on scenes that are incompletely recognized. Vedantam et al. [29] proposed a probabilistic neural-symbolic model that can respond to queries on the environment through a small learning model. Bosselut et al. [30] developed an algorithm that executes VQA and learns zero-shot, while dynamically generating commonsense knowledge graphs. VQA methods that employ the neuro-symbolic approach can explain the process of inferring answers and respond to queries related to various images and natural language processing. A new algorithm that utilizes the neuro-symbolic architecture is thus proposed in this study. Information on the objects constituting a scene can be extended flexibly by using scenes expressed in a symmetric graph form, and natural language statements can be converted directly into SPARQL for efficiently finding answers and reviewing the process.

3. Methods

In this study, a graph database with objects and object characteristics as nodes is generated from an image, and natural language queries are converted to SPARQL using bidirectional encoder representations from transformers (BERT) models with the transformer architecture. Object class, position, shape, and color can be used as scene information, and the respective symmetric graph can be extended by obtaining related data. Answers are then inferred through a graph database built using environment information and queries converted to SPARQL. For the inference process, Prolog, which uses the first-order logic, is utilized. The overall process of the proposed method is shown in Figure 2.

3.1. Scene Graph Generation

Symmetric graphs are generated using objects consisting of an image and any relation information between those objects. One symmetric graph is generated from one scene, and each graph contains various types of information including the color, size, shape, and texture of each object constituting the scene and the position relations between objects. Since the input of the graph node is text data, even if the property of an object such as color, shape, and other parameters is more than one, all data can be managed by extending the text. Therefore, the entire graph maintains a symmetric graph form. If the number of objects constituting one scene increases, one small symmetric graph composed of the object information is added to the entire graph. Each scene is saved and utilized using Protégé, which is an ontology editor developed at Standard University. An ontology is generated and checked by Protégé, and a plug-in capable of mapping, merging, and applying Pellet inference to graph data is provided [31]. Previous studies saved and used scene information in table databases. Therefore, there is a limitation in terms of freely utilizing the relations of each object [32]. In contrast, a graph database allows for relatively easier access among objects, wherein the nodes of each graph are connected, thus allowing for fast searches for relations. Compared with a table database, a graph database demonstrates the extendibility and stability, which are both beneficial for inference [33,34]. In this study, an ontology was designed for saving the scene information. The design details are provided in the Appendix A.

3.2. Question Understanding

A transformer-based BERT model is used to convert natural language into SPARQL grammar. The transformer architecture comprises an encoder consisting of multi-head attention, feed forward, add/norm layers, and a decoder, in which a masked multi-head attention layer is added to the encoder. BERT is a model built by stacking several layers of encoders in the transformer architecture. The name and performance of the BERT model vary depending on the number of encoders, embedding vectors, and self-attention heads; in this study, in particular, the BERT-Small model is used [35]. To apply our method, it is essential to construct SPARQL datasets corresponding to natural language queries. The BERT-Small model is leveraged for effective learning even in situations with limited data resources. This part encompasses the transformation of natural language queries into SPARQL, a process in which the BERT-Small model’s capability to comprehend context and meaning from smaller datasets proves valuable. As a result, it aids in generating precise and relevant SPARQL queries.

3.3. Answer Reasoning

Answers to the questions are found through natural language statements that have been converted to SPARQL. Natural language statements are converted to sequential SPARQL queries through a query analysis. A graph corresponding to the query is returned as each query is carried out, and ultimately, the results are output in the form of a string. Consequently, the process of finding answers to queries can be demonstrated. For questions requiring additional inference, Prolog [36], which is a logic programming language that supports knowledge expression and inference, is used.

4. Experiment Results

In this study, experiments are conducted using the CLEVR dataset. The CLEVR dataset is related to VQA and consists of images, detailed information on a scene, and queries/responses. Among the detailed information of a scene, information related to objects typically includes colors (gray, blue, brown, yellow, red, green, purple, and cyan), size (large, small), shape (cylinder, sphere, and cube), three-dimensional volume, texture (rubber and metal), three-dimensional center point, and angle of rotation. In addition, position relations (front, back, left, and right) between objects constituting one scene is provided. An example of the dataset is shown in Figure 3.

The graph generated for Figure 3 is shown in Figure 4. A knowledge symmetric graph of each object is expressed as follows. A node of a graph is composed of the index of the image. A symmetric graph’s properties are expressed in relations through domain and range, and the object information is input. Position relation between objects is relative and can thus be set bidirectionally. The sequential SPARQL queries are then sequentially applied to the environment graph stored in the graph database. This process enables the retrieval of answers to complex queries related to environmental information. It allows for the verification and explanation of the process of finding answers to intricate queries concerning the environment.

A searchable database needs to be generated in order to execute a SPARQL query using the CLEVR dataset and the BERT model. The relevant process is as follows. First, the data being used for conversion are extracted by analyzing the existing dataset. Then, the extracted image information is converted to graph data, and the statement information is converted to SPARQL. The converted data need to be verified in terms of whether they function identically to the existing dataset. The converted dataset is used to implement a translation model between the natural language and SPARQL, and then tested to confirm whether it has been properly converted. A logical error test is performed to verify the results of the natural language-SPARQL conversion. The Pellet inference feature of Protégé, which is a knowledge graph visualization tool, is utilized as a test tool. The detailed process is provided in Appendix B.

In this study, the BERT-Small model is used from among various BERT models to convert natural language queries into SPARQL, which is designed by applying the transformer architecture. Natural language query was set as the input and SPARQL query was set as the output of the BERT model for training the network. Figure 5 illustrates the result of converting the given queries into SPARQL queries. As shown in the experimental results, natural language queries have been successfully converted into sequential SPARQL query statements.

The final experimental results are shown in Figure 6. Figure 6a shows the original image and the index image, while Figure 6b shows an overview of the object information. Figure 6c illustrates the process of converting the query “Do the small shiny object and the matte cylinder have the same color?” into SPARQL and sequentially inferring the answer. Additional experimental results are provided in Appendix C. As shown in Figure 6, the proposed method successfully inferred the answer. The method proposed in this study expressed scene information in the form of a graph, converted natural language queries into sequential SPARQL, and inferred answers for high-level queries, and its effectiveness was verified using the CLEVR dataset. Through experiments, the proposed algorithm found answers to 500 untrained test questions with 95.3% accuracy. The approach we propose stands apart from conventional methods in that it enables direct querying to retrieve answers from a graph database. This process allows for a step-by-step verification of the query, offering distinct advantages. Furthermore, leveraging the characteristics of the graph, it becomes feasible to easily expand the semantic information of environmental objects.

5. Discussion and Conclusions

The development of task planning methods capable of performing HRI for high-level task planning is inevitable. The method of VQA is thus introduced in this study for task planning, which is an important element in interactions between humans and robots. Contrary to previous methods, the proposed method efficiently expressed the relation of objects constituting a scene using a graph database, and the answer was inferred using natural language queries that have been converted to sequential SPARQL queries. The proposed method was experimentally verified and successfully executed by VQA. The proposed method can employ extended scene information, thus being utilizable with web data having an extensive range of data. The performance can be further enhanced by designing various forms of ontology structures. Through future research, we plan to expand the semantic information of each object that makes up the environment to be able to answer more complex and difficult questions. Furthermore, we aim to enable high-level task planning by allowing robots and humans to communicate with each other through inquiries.