Relation Extraction from Videos Based on IoT Intelligent Collaboration Framework

Abstract

In the Internet of Things (IoT) era, various devices generate massive videos containing rich human relations. However, the long-distance transmission of huge videos may cause congestion and delays, and the large gap between the visual and relation spaces brings about difficulties for relation analysis. Hence, this study explores an edge-cloud intelligence framework and two algorithms for cooperative relation extraction and analysis from videos based on an IoT system. First, we exploit a cooperative mechanism on the edges and cloud, which can schedule the relation recognition and analysis subtasks from massive video streams. Second, we propose a Multi-Granularity relation recognition Model (MGM) based on coarse and fined granularity features. This means that better mapping is established for identifying relations more accurately. Specifically, we propose an entity graph based on Graph Convolutional Networks (GCN) with an attention mechanism, which can support comprehensive relationship reasoning. Third, we develop a Community Detection based on the Ensemble Learning model (CDEL), which leverages a heterogeneous skip-gram model to perform node embedding and detect communities. Experiments on SRIV datasets and four movie videos validate that our solution outperforms several competitive baselines.

1. Introduction

With the rapid expansion of the Internet of Things (IoT), an increasing number of videos are generated dramatically by various edge devices. Additionally, the emergence of novel intelligence applications requires less network latency and higher accuracy. Therefore, how to efficiently and accurately extract useful information from massive video data made by edge devices has become a hot issue [1,2]. The network transmission and computing capabilities of many camera devices in the IoT are weak and cannot meet the high-bandwidth and low-latency edge computing tasks [3]. Therefore, several challenges arise for relation extraction in the IoT system: (1) limited storage and computing power in the edge nodes; (2) long transmission delays between edge devices and remote cloud server; and (3) great challenge to provide a satisfactory mapping function from the low-level video pixel space to the high-level relationship space.

Intelligent video surveillance systems provide information extraction [4], content analysis [5], and action analysis [6] without human intervention in the cloud. In the traditional method, the limited computational resources of camera nodes, edge devices, and raw data preprocessing cause computation delay [7]. Due to limited network bandwidth, delivering raw videos to a remote cloud node may cause network congestion and transmission delays [8]. In contrast, edge computing has the characteristics of decentralization, and the computing nodes are closer to the data source [9]. Thus, network traffic and latencies can be reduced. Nowadays, camera-edge-cloud architecture is designed as a feasible video analysis method [10,11,12]. In the camera-edge-cloud architecture, edge nodes can share some computing tasks of cloud computing nodes [10]. An edge node is responsible for multiple nearby cameras. Cloud computing is responsible for video processing tasks that require massive computing and storage [11]. However, with the extensive intelligent needs for massive video and data processing, how to use Artificial Intelligence (AI) algorithms on edge nodes has become a difficult research domain [12]. Thus, how to allocate the tasks of multiple edge computing nodes and coordinate with the cloud computing platform is the difficulty of current research.

Relation extraction and analysis are typical for human-centered videos in multimedia IoT systems [13]. It has recently been a popular research focus [14,15,16]. However, this problem has not been fully solved for the following reasons. First, some efforts have focused on relation recognition based on spatial features or interactions from images, i.e., facial, distance, and object features [17,18]. However, these authors highlight the image features of the video and discount the temporal features, such as people’s actions and other valuable cues for social relations. Second, some methods are recognized only as general video classification study [19,20,21,22]. They mainly focused on the coarse-granularity information such as optical flows [19], and temporal features [21]. In other words, it is challenging for machines to “read” social relations from visual content. Vicol et al. [23] manually constructed the MovieGraphs dataset to support related research. Using artificial intelligence to automatically extract relationship graphs from videos has become the main challenge for current research. Meanwhile, there are few studies on further analysis of the extracted relation graph, for example, community detection. Most of the existing community discovery algorithms rely on massive labeled data [24,25].

Therefore, we propose a framework for video analysis by utilizing the collaboration among edge devices, edge nodes, and the cloud. First, we propose a novel framework combining edge and cloud computing using AI algorithms to create more intelligent IoT systems. By designing the cooperative division and data flow transmission between edge and cloud, network traffic and latency can be reduced. Additionally, the video stream is divided into two streams and transmitted to edge and cloud nodes. Second, an MGM model is proposed for relation graph extraction based on multi-granularity features fusion. We employ spatial and temporal networks to learn global features at a coarse granularity and use Mel-Frequency Cepstral Coefficients (MFCC) to extract audio features. Moreover, we construct a knowledge graph to describe interactions among these entities. An attention mechanism based on Graph convolution Network (GCN), named GAT, is adopted to select the most significant fined-granularity features in one video clip. Third, we propose a CDEL model to integrate structure and attribute similarities into a unified framework. Unlike the general fusion strategy, we first detect communities from the topological graph and node attributes. Then, a weighted fusion method is proposed to optimize the communities, which can reduce data inconsistencies from the graph structure and node attributes.

The main contributions of this paper are summarized as follows.

• We propose an edge computing framework to reduce latency by leveraging edge and cloud nodes’ cooperative processing capabilities on AI algorithms;
• To achieve more accuracy for video analysis, two AI algorithms are proposed based on multi-granularity GCN and ensemble learning in the cloud. In addition, some tasks are processed at edge nodes to prevent network congestion;
• We experiment on some video datasets to demonstrate that our framework and AI algorithms can optimize the accuracy and latency of the relation extraction and analysis from massive videos.

The remainder of this paper is organized as follows. Section 2 reviews three categories of related works. In Section 3, we present the edge-cloud intelligence framework. The AI methods for video analysis in edge and cloud are described in Section 4, including relationship graph extraction and analysis. Section 5 describes our experiments and results and presents the essential discussions. Finally, the conclusions are summarized in Section 6.

2. Related Work

This section discusses the related work from three aspects, namely edge computing, relation extraction and recognition, and relation analysis, which are as follows.

2.1. Edge Computing

Due to the advantages of low latency, low energy, and high bandwidth of edge computing, it has become a hot research topic [5,9,26,27]. With more and more video cameras being deployed to obtain rich information for smart IoT, the research of video data analysis have emerged [6,8,9,27,28,29,30]. For example, Yang et al. [27] deployed an end-edge-cloud collaboration to gradually learn the optimal configuration strategy. Recently, AI algorithms with deep learning have been successfully applied in many fields. Most of these algorithms require powerful GPU computing resources and storage capabilities [6]. Therefore, it has become a research challenge to use edge computing to solve delay-sensitive problems and generate accurate results in video processing tasks. Fathy et al. [28] proposed an intelligent adaptive QoS framework to support video streaming in IoT environments. In some industrial fields, the research on porting these algorithms to edge nodes has been realized [9]. For example, Zhang et al. [12] defined edge intelligent computing and reviewed the relevant systems. [9]. In recent years, research on single tasks in the video has been well solved in edge computing, such as object detection. Taghavi et al. [29] proposed the EdgeMask to solve real-time object segmentation. Dave et al. [30] used an event-driven and resource-efficient surveillance system to collect personal information. However, relation analysis consists of multiple tasks, e.g., entity recognition, relation extraction, and graph analysis. Current research finds it difficult to balance the accuracy of video processing results and the processing efficiency of multiple edge nodes in the IoT system.

2.2. Relation Extraction and Recognition

There are several studies on relation extraction and recognition from videos [31,32,33,34]. Yuan et al. [31] used a weighted Gaussian distribution and calculated the weight of each relationship based on both the shots and scenes. Lv et al. [32] divided a video into stories and extracted the relation graph of the persons following the temporal distribution of people in a story. These studies only construct a node-edge graph with no properties of edges. The relation recognition task can solve this problem. Recently, there have been some studies on this issue [18,35]. Li et al. [35] proposed a social relation graph reasoning model driven by Hybrid-Features. Lin et al. [18] proposed a picture reasoning model to achieve relationship classification. However, these studies are based on relation analysis of static images and cannot be applied to videos. Some studies have tried to recognize relations from video data [22,36,37]. Kukleva et al. [22] designed a model for learning and predicting interactions and relationships from videos. Teng et al. [36] proposed a character and relationship joint learning framework to recognize social relations on a selected part of the MovieGraphs dataset. However, their experimental data required certain textual and manual labels. Lv et al. [37] proposed an attentive sequences recurrent network that identifies relationships directly from the original video and had low model operation efficiency. Implementing the works mentioned above requires immense computing power on the server, which is unsuitable for IoT systems. This paper studies how to assign subtasks and optimize the relation extraction model in the IoT system to improve accuracy.

2.3. Relation Graph Analysis

To discover the key people in surveillance videos and to detect communities are important aspects of massive video processing. Lee et al. [38,39] learned representations of relationship graphs extracted from movies and designed a Char2Vec model for node-edge networks. Lv et al. [32] detected communities from node-edge networks extracted from movies. Many studies have proposed a large number of excellent algorithms for finding community structures [40,41,42]. However, they are typically unable to directly solve the problem of analyzing a network containing edges associated with various relationship types. Pan et al. [25] used the features of attributes and structure of the social relations among nodes. Sun et al. [24] proposed a community detection algorithm based on a deep sparse autoencoder and achieved accurate results. However, the premise that these methods can achieve good results is that the proposed models must be trained on large-scale annotated datasets. Recently, many studies have been on graph representation and analysis, including attribute networks [43] and heterogeneous networks [44,45]. These methods can be used for reference in analyzing video relation graphs. This paper uses the above research methods to design a method for analyzing community on the relation graph extracted from wild video data.

Table 1. A summary of main related studies (EC: edge computing; RE: relation extraction; CD: community detection).

Reference	Contribution	Scope
		EC	RE	CD
Fathy et al. [28]	Proposed an intelligent adaptive QoS framework for video stream	yes	no	no
Yang et al. [27]	Deployed an end-edge-cloud collaboration framework for video queries	yes	no	no
Zhang et al. [12]	Defined edge intelligent computing and reviewed the relevant systems	yes	no	no
Taghavi et al. [29]	Proposed an EdgeMask to solve real-time object segmentation	yes	no	no
Dave et al. [30]	Used an event-driven and resource-efficient surveillance system	yes	no	no
Lv et al. [32]	Extracted the relation graph of the persons using coarse-grained features	no	yes	no
Li et al. [35]	Proposed a social relation graph reasoning model based on server	no	yes	no
Teng et al. [36]	Proposed a character and relationship joint learning framework on standard dataset	no	yes	no
Lv et al. [37]	Proposed an attentive sequences recurrent network on standard dataset	no	yes	no
Wang et al. [40]	Proposed an overlapping CD algorithm based on Non-attribute Network	no	no	yes
Sun et al. [24]	Proposed a community detection algorithm based on a deep sparse autoencoder	no	no	yes
Sun et al. [44]	Propose an algorithm named Network Embedding for node Clustering (NEC)	no	no	yes
Our work	Relation extraction and analysis based on edge computing	yes	yes	yes

compares our work and the state-of-art efforts in edge computing and video analysis. Some studies have proposed edge computing frameworks [12,28,29], but they have not researched the relation recognition task. In recent years, there have been some studies on relation recognition and analysis [32,35,36], but they require training massive data on enormous computing power and do not support edge computing tasks. Our work supports the tasks in the above three scopes.

3. The Proposed Edge-Cloud Intelligence Framework

This section first describes the proposed edge-cloud computing architecture for video analysis, depicted in Figure 1. Edge devices, edge nodes, and the cloud will cooperate to complete data transmission and implementation of AI algorithms. Each edge node can serve multiple cameras, communicate with the cloud, and collaborate to complete the different tasks with AI algorithms.

3.1. Framework Overview

In the edge-cloud intelligence framework, we set the edge node to be capable of executing AI algorithms, as shown in Figure 1. The edge node is closer to the edge device. Thereby, it is very necessary to shorten the transmission distance and improve the response time. There are two aspects of collaboration regarding the intelligent processing part: edge-edge and edge-cloud.

In the proposed edge-cloud framework for delay-sensitive relation recognition from videos, shown in Figure 1, edge devices are deployed in every city corner. They can capture video, communicate with edge computing nodes, and support the operation of intelligent algorithms. They transfer data to each other via edge-to-edge communication [46]. The edge devices capture video streams and divide them into different sub-tasks. The distributed video stream is transmitted to the network through the edge and cloud nodes. Then, edge nodes process the collected video sub-tasks, perform some training of intelligent algorithms, and send the computing results via network to the cloud, which can train more extensive intelligent algorithms and store massive amounts of data.

3.2. Definition

The framework includes three main parts: edge devices, edge intelligence, and cloud intelligence.

• Edge devices: These may be static camera devices fixed in the surveillance area such as city streets, public places, mobile phones, or drones, which invoke video tasks, divide them into smaller sub-tasks, compress videos and transmit them to the edge nodes. Besides, essential information and videos are sent to the cloud for high-level planning and decision with Deep Neural Networks (DNN), e.g., relation recognition and video analysis algorithms;
• Edge intelligence: It reflects the core functions of edge computing. The edge nodes can run lightweight AI algorithms, have some storage capacity, and are closer to the edge device to generate faster query responses. It helps execute some sub-tasks, e.g., feature extraction and entity detection, and transmits these results to cloud computing nodes through the network for further analysis;
• Cloud intelligence: The cloud computing platform undertakes the storage and computing capacity of the overall video in the system, which is a fixed device far from the edge node. The cloud node receives videos and operation results from edge devices and edge nodes to perform more detailed analysis and training, such as relation graph construction, relation recognition, and community detection.

3.3. Collaboration and Workflow

In our edge-cloud intelligence framework, two aspects of collaboration are required: edge-edge and edge-cloud. In the edge-edge scenario, many edges need to collaborate to accomplish a task with each other. Training tasks are always completed on the cloud and then downloaded to edge nodes in the edge-cloud scenario. Sometimes, edge nodes need to retrain the AI model by transfer learning based on the video.

Edge device communicates with edge node via device-to-device. One edge node links with multiple edge devices. Video analysis can be divided into subtasks requiring fast query response, such as entity detection and result query. Some require much computation, preprocessing results, and higher accuracy, such as GCN network training and relation recognition. The workflow of our framework has two pipelines, as shown in Figure 2. One pipeline is deployed for tasks that consider latency more important than accuracy, which processes on the edge nodes for the video task. For example, simple video preprocessing and entity detection tasks require computing power and storage space. The other pipeline is edge-cloud; the video that needs further processing is compressed and transmitted from the edge devices or edge nodes to the cloud node. The network monitor manages transmission performance indicators from each edge and cloud node.

4. AI Algorithms for Relation Extraction and Analysis

This section describes our intelligence methods for relation extraction and analysis in edge and cloud nodes. It includes Section 4.1—problem description, Section 4.2—tasks in edge intelligence, and Section 4.3—tasks in cloud intelligence.

Table 2. Important notations.

Notation	Definition
A	The person set, $A=\left\{a_1,a_i,\dots ,a_n\right\}$
Att(A)	The person attributes set
RG(A)	Relation graph extracted from video
$concat(·)$	The concatenation of feature vectors
$normalize(·)$	The normalization operation
$f_E^{V_{a\_c}}$	The entity features in the MGM
$f_G^{V_{a\_c}}$	The global features in the MGM
$f_A^{V_{a\_c}}$	The attribute features in the MGM
$P^{stru}$	The community partition based on structure
$P^{att}$	The community partition based on attribute
M(A)	The adjacency matrix of the entity graph, $M{\left(A\right)}^{\left|m\right|\times \left|m\right|}$
$A_{ei,ej}$	An element of $M\left(A\right)$, $ei,ej=1,2,\cdots ,\left|m\right|$

provides a list of important notations used in this paper.

4.1. Problem Description

We make the fundamental assumption that the relation graph between people consists of five elements: person set (nodes), person attribute set (attributes of the nodes), their relationships (edges), and their relation types among the people (types of the edges), and community detection.

Definition 1

(Person Set). Suppose that n is the number of people that appear in a video $V_a$. The person set is represented as $A=\{a_1,a_2,\dots ,a_n\}$, where $a_i$ is the i-th element of A.

Definition 2

(Person Attribute Set). The person attributes to set the age, sex, and the profile of each person, represented as follows:

$$\begin{array}{c}\hfill Att\left(A\right)=\{Att\left(a_1\right),Att\left(a_2\right),\dots ,Att\left(a_i\right),\dots ,Att\left(a_n\right)\},a_i\in A\end{array}$$

(1)

Definition 3

(Relationship Graph). $RG\left(A\right)$ denotes the relation graph extracted from the video $V_a$. It is represented as $G\left(A\right)=<A,E,R>$, where each node a and each link r are associated with mapping functions, $A\to R_A$ and $E\to R_E$, respectively. $R_A$ and $R_E$ denote the sets of person and relation types and satisfy $|R_A|+|R_E|>2$. The relationship graph not only represents whether relationships exist between pairs of people but also indicates what kinds of relationships they are. $RG\left(A\right)$ can be described as a matrix $RG\left(A\right)$$\in R^{n\times n}$:

$$\begin{array}{c}\hfill RG\left(A\right)=\left[\begin{array}{cccc}{r}_{a_1,a_1}& r_{a_1,a_2}& \cdots & r_{a_1,a_n}\\ r_{a_2,a_1}& r_{a_2,a_2}& \cdots & r_{a_2,a_n}\\ ⋮& ⋮& ⋮& ⋮\\ r_{a_n,a_1}& r_{a_n,a_2}& \cdots & r_{a_n,a_n}\end{array}\right],\end{array}$$

(2)

where each element $r_{a_i,a_j}$ of $RG\left(A\right)$ denotes the corresponding relation type and can be described as $r_{a_i,a_j}\in R_E$, where $R_E$ is a fixed relation type set.

Definition 4

(Community Detection). For the above relationship graph $RG\left(A\right)$, the objective of the community detection task is to divide the person set A into k communities. The set of communities can be expressed as $C\left(A\right)=\{c_1,c_2,\dots ,c_k\}$ in accordance with a model $\Phi (·)$, where the model $\Phi (·)$ is defined in terms of the structural features $\delta \left(G_{stu}\right)$, node attribute features $\phi \left(A_{att}\right)$, and edge category features $\gamma \left(R\right)$ of $RG\left(A\right)$:

$$\begin{array}{c}\hfill \Phi \left(RG\right)=f(\delta \left(G_{stu}\right),\phi \left(A_{att}\right),\gamma \left(R\right)).\end{array}$$

(3)

4.2. Tasks in Edge Intelligence

DNNs partitions are widely used in various fields [10,47] to complete different tasks, e.g., reducing latency or improving accuracy. Our task’s AI algorithms and video analysis methods can be divided into two parts, running on two locations, e.g., one on an edge node and the other in cloud intelligence.

Multiple edge nodes work collaboratively to accomplish a compute-intensive task, e.g., object detection and intelligent query. An AI algorithm may run distributed on one or more edge nodes as the related computation nodes. It is optimally partitioned, offloaded, and executed with the peer sensor nodes [8]. Then the processing results are transmitted to the cloud intelligence.

4.2.1. Partition Schemes for Entity Detection

The significant groundwork of relation recognition is entity detection. In our paper, this task employs a Faster R-CNN partition [10]. Our paper uses Faster R-CNN, including a region proposal network and a classification network. The partition scheme considers how to reduce network transmission and seeks a suitable image size and feature map for transmission. ResNet101 consists of 5 components with 33 building blocks in these components [48]. These building blocks are divided into five components: conv1, conv2, conv3, conv4, and conv5, and the number of building blocks contained in each component is 0, 3, 4, 23, and 3. There are three units in each block, and each unit contains three layers of convolution, batch normalization, and activation, respectively. The partitions can be split after the max and average pooling layer and the RPN, shown in Figure 3a.

Compared with the DNN models, compressing and decompressing image features are executed on edge nodes. So, cloud node only consumes negligible computing resources and storage space. Recent studies have shown that uniform 8-bits quantization has a negligible effect on the accuracy of object detection [49]. We convert 32-bits video feature maps to 8-bits quantized feature maps. A feature map includes many channels; each channel is a 2D integer matrix. Therefore, we tile the channels to convert the 3D matrix into a giant 2D matrix, then the 2D matrix is quantized between 0 and 255, and we use the PNG to compress the 2D matrix. Decompression is the opposite process. Each channel is convolved with a 2D filter shown in Figure 3b. For example, an input image (feature map) to the first layer of a CNN will contain three channels corresponding to the input image’s red, green, and blue components.

4.2.2. Person Identification

In this section, we design a fusion model to obtain the person set A. We use a pre-trained Faster R-CNN model on the Wider Face dataset [50] to recognize faces. All face images are normalized to a size of 112 × 112 pixels. A pre-trained model detects the bodies and is normalized to 128 × 256 pixels. Then, faces and body features are extracted using ResNet101 [10]. Finally, the features are fused, and the fused feature set is obtained as the input to the clustering algorithm presented in Algorithm 1. In the first step of the clustering process, the optimal number of clusters is obtained, using the cosine distance to calculate the similarity of person features [51]. Each node is labeled for the relation graph to get the person set A in the second step. The basic operation of the Algorithm 1 is calculated as $1+2+\cdots +n=\frac{n(n+1)}{2}$. So the time complexity is $O\left(n^2\right)$, where n represents the number of the features $F_{fusion}$.

Algorithm 1 Person Set A

Input: face and body features $F_{face}$ and $F_{body}$. Output: person set A.   //Concatenate feature vectors and perform normalization operations. 1: $F_{fusion}=normalize(concat(F_{face},F_{body})$); 2: initialize the number of clusters k = 0;   // Initialize cluster centers for the first clustering; 3: for$f\in F_{fusion}$do 4:    if the nearest $min\_dis<t$ then 5:      add f into $clu\_cen\left[index\right]$; 6:    else 7:      $k=k+1$; 8:      use f to initialize $\mathrm{clu}\_\mathrm{cen}\left[\mathrm{k}\right]$; 9:    end if 10: end for   // The second clustering gets the cluster centers. 11: $centers,clusterAssent=kmeans\left(F_{fusion},k+1\right)$;   // Set the label for each cluster center and get set A. 12: $\mathrm{label}(\mathrm{clu}\_\mathrm{cen}\in [0,k]$); 13: return the person set A;

4.3. Tasks in Cloud Intelligence

The intelligent cloud server has robust storage and computing capabilities and mainly undertakes the task of improving accuracy. To improve more accurate and thoughtful services for video analysis, we propose a multi-granularity graph convolutional network, which runs in the cloud.

4.3.1. The Architecture of the MGM

The overall architecture of the MGM is shown in Figure 4. It can be divided into coarse-granularity feature extraction and fine-granularity feature extraction. The first part takes as input one video clip $V_{a\_c}$, which is sampled into frames for efficiency, $V_{a\_c}=\left\{I_k\right|k=1,2,\dots ,L\}$. Each frame $I_k$ is resized to 224×224 pixels and fed into a temporal segmentation network, where the global features $f_G^{V_{a\_c}}$ are obtained in the last convolutional layer of the temporal segment network (TSN) network [52] and get 2048 dimension features. The second part uses MFCC to represent audio features (39 dimension), represented as $f_A^{V_{a\_c}}$. In the fine-granularity part, we employ entity detection results from Fast R-CNN and Resnet101 and get 2048 dimension features. We use $f_E^{V_{a\_c}}=\{f_{E_{}}^i\in R^{1\times d}|i=1,2,\dots ,m\}$ to denote the semantic features of fine-granularity entities. Then, an entity graph based on a GCN is established to extract fine-granularity features $f_E^{V_{a\_c}}$. An attention mechanism is adopted to select the most significant entities. Subsequently, multiple features are fused as the input to the next layer. Finally, the sigmoid cross-entropy loss function is utilized to optimize the classifier for relation recognition.

4.3.2. Relation Recognition

The relationship graph indicates whether there is a relationship between persons $a_i$ and $a_j$. However, the relation type, an important and ubiquitous factor in daily life, is not specified. In this section, we study how to solve this problem.

There is a considerable gap between the abstract relationships among persons and the visual pixels depicting those people. Humans can intuitively understand social relationships from videos. However, it poses many challenges for artificial intelligence. Considering fine-granularity object information helps analyze the relationships between people. For example, fine-granularity information such as the presence of desks, computers, and documents may reflect a “colleague” relationship between two people. This paper proposes a Multi-Granularity relation recognition Model (MGM). This model recognizes entities other than people from a fine-granularity perspective.

The set of nodes in a fine-granularity graph is established in order to extract fine-granularity features, denoted by $f_E^{V_{a\_c}}$. $M{\left(A\right)}^{\left|m\right|\times \left|m\right|}$ is an adjacency matrix. $M\left(A\right)$ represents how close together objects and person are in one video clip $V_{a\_c}$. $A_{ei,ej}$ is an element of $M\left(A\right)$, where $ei,ej=1,2,\cdots ,\left|m\right|$. Then, $A_{ei,ej}$ is initialized with the frequencies of co-concurrence of all entity pairs throughout the video clip $V_{a\_c}$. Furthermore, $A_{ei,ej}$ is normalized in the range from 0 to 1 via min-max normalization,

$$\begin{array}{c}\hfill A_{ei,ej}=\frac{A_{ei,ej}-min{A}_{ei,ej}}{max{A}_{ei,ej}-min{A}_{ei,ej}},\end{array}$$

(4)

where $ei,ej=1,2,\cdots ,\left|m\right|$.

We perform a linear transformation on the node feature vector $f_E^i$. Thus, we obtain a new feature vector $h\left(f_E^i\right)=W{f}_E^i$, where $W\in R^{d\times h\left(d\right)}$ and $h\left(d\right)$ is the dimensionality of the new feature vector.

The GAT network employs an attention mechanism to learn the importance of neighboring nodes [53]. Thus, a new feature vector can be obtained:

$$\begin{array}{c}\hfill \begin{array}{cc}& e_{ij}=a(W\overrightarrow{h_i},W\overrightarrow{h_j}),\hfill \\ & {\alpha }_{ij}^k=exp\left(e_{ij}\right)/\left({\sum }_{k\in N_i}exp\left(e_{ik}\right)\right),\hfill \\ & \overrightarrow{h_i^{‘}}=\sigma \left({\sum }_{j\in E}{\alpha }_{ij}^k{W}^k\overrightarrow{h_j}\right),\hfill \end{array}\end{array}$$

(5)

where $e_{ij}$ represents the importance of node j to node i. $\overrightarrow{h_i}$ and $\overrightarrow{h_j}$ represent node feature vectors sequence. The function $a(·)$ represents a shared attention mechanism, which can theoretically calculate the attention between all nodes. If $\overrightarrow{h_i}\in R^F,\overrightarrow{h_i}\in R^F$, $a:R^F\times R^F\to R$.

Finally, entity feature sets $f_E^{V_{a\_c}}=\{f_{E_{}}^i\in R^{1\times h\left(d\right)}|i=1,2,\dots m\}$ are obtained.

For relation type $r_{a_i,a_j}\in R^l$, we concatenate the three types of features.

$$\begin{array}{c}\hfill f_{fusion}=normalize\left(concat(f_E^{V_{a\_c}},f_G^{V_{a\_c}},f_A^{V_{a\_c}})\right),\end{array}$$

(6)

where $concat(·)$ represents the concatenation of the three vectors for each node, and $normalize(·)$ represents the normalization operation.

Then the last $f_c$ layer to generate the output score for the relation type $r_{a_i,a_j}$:

$$\begin{array}{c}\hfill p_{r_{ai,aj}}=W{f}_{fusion}+b.\end{array}$$

(7)

The loss function is defined as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Loss=\frac{1}{N}\sum _{n=1}^N\sum _{i=1}^{I}l\left(r_i\right)log\left(p_{r_{ai,aj}}^i\right)\\ \hfill +(1-l\left(r_i\right))log(1-p_{r_{ai,aj}}^i)),\end{array}\end{array}$$

(8)

where $l\left(r_i\right)$ is the ground truth for the i-th class and $p_{r_{ai,aj}}^i$ is the probability of prediction.

4.3.3. Relationship Graph Analysis

The principal objectives of relationship graph analysis are to discover the grouping of the people to comprise social communities and to find the most influential person within each community [33,39]. The relationship graph extracted may include edge attributes, making it more complex than homogeneous networks. This paper proposes a Community Detection based on the Ensemble Learning model (CDEL) to solve these challenges, as shown in Figure 5.

In the CDEL model, we first divide $RG\left(A\right)$ into two parts, i.e., its topological structure $St{u}_{GR\left(A\right)}$ and its node feature set $At{t}_{GR\left(A\right)}$. Then, we apply community detection and clustering algorithms to $St{u}_{GR\left(A\right)}$ and $At{t}_{GR\left(A\right)}$ separately to obtain candidate community partitions $P^{stu}$ and $P^{att}$ for each branch. Finally, an ensemble learning algorithm is employed to fuse these partitions and obtain the optimal partitions $P=\{P_1,P_2,\cdots ,P_k\}$, which is elaborated in Algorithm 2.

(1) Structure partitioning.

The dense regions of the graph typically correspond to entities that are closely related and therefore are considered to belong to a community. Correspondingly, these structure-based community detection algorithms [54,55] have achieved excellent results. However, such methods are unsuitable for application to graphs with edge attributes.

To overcome the challenge presented by the existence of multiple types of links $GR\left(A\right)$, we leverage metapath2vec [45] to learn a relationship graph representation. Given the relationship graph $RG\left(A\right)$, metapath2vec can learn latent dimensional representations $GR\left(A\right)\in R^{\left|A\right|\times d},d\ll \left|A\right|$, and capture the structural and semantic relations among them. In the meta path based random walks step, a meta path scheme $\rho$ is defined as a path that is expressed in the form $A_1\stackrel{R_1}{\to }{A}_2\stackrel{R_2}{\to }\cdots \stackrel{R_t}{\to }{A}_{t+1}\cdots \stackrel{R_{\mathrm{n}-1}}{\to }{A}_n$, where R defines the composite relations between nodes ${\mathrm{a}}_1$ and ${\mathrm{a}}_2$. The transition probability at step i is defined as follows with the $a^{th}$ row corresponding to the representation of node a,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill p\left(a^{i+1}\right|a_t^i)=\left\{\begin{array}{c}\frac{1}{|N_{t+1}\left(a_t^i\right)|},(a^{i+1},a_t^i)\in E,\varphi \left(a^{i+1}\right)=t+1,\\ 0,(a^{i+1},a_t^i)\in E,\varphi \left(a^{i+1}\right)\ne t+1,\\ 0,(a^{i+1},a_t^i)\notin E,\end{array}\right.\end{array}\end{array}$$

(9)

where $a_t^i\in A_t$ and $N_{t+1}\left(a_t^i\right)$ denote the $A_{t+1}$ type of neighborhood of the node $a_t^i$. In other words, $a^{i+1}\in A_{t+1}$, that is, the flow of the walker, is conditioned on the predefined metapath.

In our work, the meta path scheme has the form “person-relation-person”. The meta path based random walk strategy ensures that the semantic relationships between different nodes are preserved. The output of the network is a low-dimensional matrix. Finally, the node features are used as an input to the k-means algorithm to detect a community partition $P^{stru}=\{c_1^s,c_2^s,\dots ,c_m^s\}$, which satisfies ${\cup }_i{c}_i^s=C$ and $\forall c_i^s,c_j^s\in P,c_i^s\cap c_j^s=\varnothing$.

(2) Attribution partitioning.

The initial node attributes of the relation graph include the age, sex, and the person profile, as obtained by Bidirectional Encoder Representations from Transformers (Bert) [56].

The node features are also input to the k-means algorithm to detect a community partition $P^{att}=\{c_1^a,c_2^a,\dots ,c_m^a\}$, which satisfies ${\cup }_i{c}_i^a=C$ and $\forall c_i^a,c_j^a\in P,c_i^a\cap c_j^a=\varnothing$.

N is the number of candidate partitions P from $GR\left(A\right)$. $P=\{P^1,P^2,\dots ,P^i,\dots ,P^N\}$, where $P^i=\{A_1^i,A_2^i,\dots ,A_{k_i}^i\}$ and $A_j^i$ is the jth community in $P^i$. The main concern of the CDEL model is to determine a fusion strategy that can guarantee that the final community partition $P^c$ is the optimal one.

Suppose that the community partition $P^{att}$ is generated by the community cluster detector from the node attributes and that $P^{stru}$ is generated based on the community structure. $W=(w_1,w_2,\dots ,w_N)$ is the weight vector for the community partition fusion. For each node pair of $GR\left(A\right)$, the weighted co-association matrix M is defined as

$$\begin{array}{c}\hfill \begin{array}{cc}& M_{ij}=\frac{1}{N}{\sum }_k^N{w}_k(a_i,a_j)·\delta (p_k\left(a_i\right),p_k\left(a_j\right))),\hfill \\ & \forall a_i,a_j\in A,p_k\in P,\hfill \\ & w_k(a_i,a_j)\in [0,1],\hfill \end{array}\end{array}$$

(10)

where $p_k\left(a_i\right)$ is the cluster label of $a_i$ in partition $p_k$, and $\delta (p_k\left(a_i\right),p_k\left(a_j\right)))$ is 1, if $p_k\left(a_i\right)=p_k\left(a_j\right)$, and 0 otherwise.

The weighting function $w_k(a_i,a_j)$ in the topological graph $GR\left(A\right)$ can be defined as the Jaccard similarity coefficient, and in the attribute graph it can be defined as the related attribute similarity, which is proposed in the paper [57], i.e.,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill w_k(a_i,a_j)=\left\{\begin{array}{c}\frac{|\Gamma \left(a_i\right)\cap \Gamma \left(a_j\right)|}{|\Gamma \left(a_i\right)\cup \Gamma \left(a_j\right)|},\delta (p_k\left(a_i\right),p_k\left(a_j\right))>0\\ 0,\mathrm{otherwise}\end{array}\right.\end{array}\end{array}$$

(11)

where $\Gamma \left(a_i\right)$ retrieves the neighbors of node $a_i$ in $GR\left(A\right)$. Correspondingly, $w_k(a_i,a_j)$ could also be defined as the related attribute similarity, i.e.,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill w_k(a_i,a_j)=\left\{\begin{array}{c}1-\frac{∥a_i^{att}-a_j^{att}∥}{{max}_{a_p,a_q\in C_i}∥a_p-a_q∥},\delta (p_k\left(a_i\right),p_k\left(a_j\right))>0\\ 0,\mathrm{otherwise}\end{array}\right.,\end{array}\end{array}$$

(12)

where $a_i^{att}$ represents the feature vector of node $a_i$, and $∥·∥$ is the norm operator. $C_i$ represents the cluster that has the label $p_k\left(a_i\right)$, if the $\delta (p_k\left(a_i\right),p_k\left(a_j\right))>0$, then we have $C_i=C_j$.

The matrix M is a similarity matrix for A in the relationship graph $GR\left(A\right)$. The algorithm is shown as Algorithm 2. Our model uses a weighted fusion of the relationship graph structure and attribute information to obtain more accurate communities. The time complexity of the Algorithm 2 is $O(N*m^2)$, where N is the number of candidate partitions, and m is the number of $p^{stru}$ and $p^{att}$, respectively.

Algorithm 2 Fusion algorithm for CDEL

Input: structure and attribute partitions $p^{stru}$ and $p^{att}$. Output: The final community partition $P_{final}$.  //Traverse the two partitions. If the node pair is not in the same partition, we ignore it.  Otherwise, we calculate its weighted co-association matrix M. 1: for each candidate partition $p^s\in p^{stru}$ do 2:    for each candidate partition $p^a\in p^{att}$ do 3:      if $a\ge s$ then 4:         continue; 5:      end if      //initialization the M 6:      $M\left(as\right)\leftarrow 0$; 7:      for i = 1,2,...,N do 8:         $M\left(as\right)\leftarrow M\left(as\right)+w_i(a,s)·\delta (p_i\left(a\right),p_i\left(s\right))$; 9:      end for 10:      $M\left(as\right)\leftarrow \frac{1}{N}·M\left(as\right)$; 11:    end for 12: end for //Obtain the final partitions using the kmeans. 13: $P_{final}\leftarrow kmeans\left(M\left(as\right)\right);$ 14: return the final community partition $P_{final}$.

5. Experiment

The experiment mainly verifies the validity of the edge-cloud intelligence methods. Our experiment environment is comprised of an Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10 GHz processors running at 2.10 GHz with two TITAN X (Pascal) GPUs and two edge nodes. Each edge node is an NVIDIA Jetson TX2 board.

5.1. Dataset Description

Experimental data were selected to support two tasks: relation recognition and community detection.

The dataset that we used for relation recognition is Social Relation in Videos (SRIV) [19]. The SRIV is a personal relationship video dataset containing 3124 videos totaling 25 h. We also test our MGM model on the real four movie videos, i.e., The Shawshank Redemption, Sissi, Kung Fu Hustle, and Forrest Gump. Complex relations exist between people who appear in videos. In total, we analyzed the relationships among 152 roles. We selected 10–20 video clips for each person pair, each with a duration of approximately 5–10 s. The video and audio encoded is H.264 and AC3, respectively, and the video bit rate is 7916 kb/s, and the audio bit rate is 320 kb/s.

For the community detection task, we use the results obtained from the above four movies employing the relation network extraction method of previous work [32].

5.2. Evaluation Metrics

To validate the overall performance of our framework, we use the following evaluation metrics to assess the performance of the method proposed in this paper.

The relation recognition method is evaluated using the $F_1$, BalAccuracy, and subset accuracy as follows.

5.2.1. $F_1$

The $F_1$ score is based on label metrics. $F_1$ for the i-th class is calculated as

$$\begin{array}{c}\hfill F_1\left(i\right)=2\times TP\left(i\right)/(2\times TP\left(i\right)+FP\left(i\right)+FN\left(i\right)),\end{array}$$

(13)

where $TP\left(i\right)$, $FP\left(i\right)$, $TN\left(i\right)$ and $FN\left(i\right)$ denote numbers of true positives, false positives, true negatives, and false negatives, respectively, for the i-th class.

5.2.2. BalAccuracy

To account for the imbalance between the positive and negative classes, a balanced accuracy (BalAcc) is adopted:

$$\begin{array}{c}\hfill BalAcc=\frac{1}{2}(\frac{TP}{N_p}+\frac{TN}{N_n}),\end{array}$$

(14)

where $N_p$ and $N_n$ are the numbers of positive and negative samples, respectively.

5.2.3. Subset Accuracy

This metric evaluates whether the predicted label set exactly matches the ground-truth set [37]. It is defined as follows:

$$\begin{array}{c}\hfill SubAcc\left(h\right)=\frac{1}{n}\sum _{k=1}^{n}I(h\left(x_k\right)=Y_k),\end{array}$$

(15)

where $I\left(true\right)=1$ and $I\left(false\right)=0$.

5.2.4. NMI

The community discovery method is evaluated using the Normalized Mutual Information (NMI), which takes values between 0 and 1 [58].

$T^{*}$ denotes the set of ground truth communities and the set of communities divided by the relationship network division algorithm by T. Additionally, r and k denote the numbers of partitions in $T^{*}$ and T. N is a confusion matrix. The rows in N represent real communities. The columns in N represent the communities obtained by the algorithm. S is the number of nodes. $N_{ij}$ denotes the number of nodes in both real community i and found community j. $N_{i*}$ and $N_{*j}$ sums of i-th row and j-th column in N, respectively. The NMI is calculated as follows:

$$\begin{array}{c}\hfill NMI=\frac{-2{\sum }_{i=1}^r{\sum }_{j=1}^k{N}_{ij}log\frac{N_{ij}S}{N_{i*}{N}_{*j}}}{H\left(T^{*}\right)+H\left(T\right)},\end{array}$$

(16)

where $H\left(T^{*}\right)$ and $H\left(T\right)$ are

$$\begin{array}{c}\hfill H\left(T^{*}\right)={\sum }_{i=1}^r{N}_{i*}log\left(\frac{N_{i*}}{S}\right),\end{array}$$

(17)

$$\begin{array}{c}\hfill H\left(T\right)={\sum }_{j=1}^k{N}_{*j}log\left(\frac{N_{*j}}{S}\right).\end{array}$$

(18)

5.3. Entity Detection in Edge Intelligence

To evaluate the computing capacity and accuracy of the AI task of the edge node, we use the entity detection task to perform experiments on an edge node on a given video clip. We compare feature extraction methods in entity recognition models, Resnet50 and Resnet101—both standard networks. The Resnet50 has five components such as Resnet101, but it contains 16 building blocks allocated to 5 components with the numbers 3, 4, 6, and 3. Figure 6 shows the accuracy of different algorithms on different video image sizes. We can find that the accuracy of entity detection grows with the video quality. It demonstrates that our framework and the edge node can establish a stable mechanism for AI algorithms to run stably on extreme edge devices. The accuracy is still far from 100%, because it is challenging to capture positive and clear entity images with real-world video data. The latency may increase with the tasks increase because there are more cloud pipelines in the cloud.

5.4. Relation Recognition in the Cloud Intelligence

We experimentally verified the accuracy of the MGM model on the cloud server. The relationship graphs for the four movies were constructed as described in a previous work [32]. These graphs show whether a relationship exists between two people; however, what kinds of relations they are is a problem that needs to be solved using the MGM novel relation recognition method.

We compare it with six other methods to validate our MGM based on experiments on the SRIV dataset. The results are shown in

Table 3. Comparison with the state-of-the-art methods on the SRIV dataset.

Method	F1	BalAcc	SubAcc
C3D [59]	0.3886	0.0556	0.0357
LSTM [60]	0.5776	0.6667	0.2797
TSN [52]	0.6142	0.7089	0.3482
Multi-Stream [19]	0.6383	0.6136	0.5291
STMV [37]	0.6492	0.6322	0.5311
TSM [20]	0.7293	0.7125	0.6032
Ours (+GAT)	0.7431	0.7173	0.6114
Ours (+GAT+Audio)	0.7450	0.7183	0.6147

. By aiming at the two innovations of the MGM model, the GAT part and the fusion of multiple features, respectively, ablation experiments were done.

In a video, the relations among people always present complex forms, e.g., through their actions or the items in the same scene. First, the base spatial or temporal base models considered for comparison, i.e., long short-term memory (LSTM) and a TSN, produce relatively low accuracy. In contrast, a fused multi cue model, i.e., the Multi-Stream model, offers improved recognition accuracy, as reflecting the scene and action information are advantageous for relation recognition. These models focus only on coarse information and ignore fine-granularity information. Second, the two-stage model (TSM) is better because it considers detailed appearance characteristics and objects, providing more sophisticated features for relation recognition. Finally, our MGM achieves the best performance by combining the TSN and spatio-temporal multi-view (STMV) models with an attention mechanism. For example, the accuracy achieved is 0.7173 (+8.5%), and $F_1$ achieves 0.7431 (+9.3%), suggesting that the attention mechanism can focus on more crucial entities useful for their relation. More importantly, after audio features are fused, the performances are improved again. For example, the accuracy increased by 1%, demonstrating that the audio feature reflects the emotions at the time and also permeates the relationship.

As shown in Figure 7, the models trained on SRIV were tested on the four movie videos. However, the results vary across the different videos. Our approach yields the worst result on “Kung Fu Hustle” while demonstrating the best performance on “Sissi”. This is because fine-granularity information on relations is more abundant in the latter than in the former.

5.5. Community Detection in the Cloud Intelligence

We validate the effectiveness of the CDEL on the cloud intelligence server and some experiments performed on graphs constructed from real-world datasets.

Figure 8 illustrates the comparative results of the various community detection methods. We can see that the NMI value of our proposed model is higher, indicating that it better characterizes the communities formed by the people in videos. Based on the results, we can observe that the node cannot accurately represent the communities features alone, as seen from the fact that node2Vec [61] and Bert [56] achieve the lowest NMI value. Moreover, models using structural features demonstrate superior performance, i.e., Girvan Newman (GN) [62] and metapath2vec [45]. They will likely reflect the relations with the other nodes more comprehensively in the whole relation graph. Finally, our CDEL model shows the best performance, demonstrating the complementary effect of the weighted fusion of multiple types of features of the relation graph.

The proportion of different features is different by comparing experiments conducted on the four movies, as shown in Figure 9. First, we observe that those division methods based on graph structure occupy a more significant role, accounting for about 30% (GN and Metapath2Vec), respectively. Second, a division based on Node2vec shows the worst performance on “Sissi”. This result demonstrates that simple words cannot express relations. Instead, video can help mine richer semantics.

6. Conclusions

This paper proposes an edge-cloud intelligence framework for cooperative relation extraction and analysis from videos. First, we exploit an edge-cloud intelligence framework, which can schedule the sub-tasks for massive video analysis. Second, we propose a multi-granularity fusion model to recognize the relations. Specifically, edge nodes and the cloud can help cooperatives complete the AI algorithms with computing power and network transmission capacity. Third, we exploit a weighted fusion strategy for community detection, which can provide a comprehensive perspective and innovative service on the relation graph. Finally, experiments on some datasets demonstrate the effectiveness of the proposed framework. However, there are also some limitations. The first problem is that we did not consider the different computing power of different edge nodes. Secondly, the dataset labels are artificially labeled, which may be mixed with some subjective factors.

To deal with the above limitations, we suggest future research directions. First, we can optimize the algorithm by implementing more intelligence algorithms to run on the edge nodes for video analysis. Second, future research can collect more realistic datasets and establish a more objective labeling scheme.