TBRm: A Time Representation Method for Industrial Knowledge Graph

: With the development of the artificial intelligence industry, Knowledge Graph (KG), as a concise and intuitive data presentation form, has received extensive attention and research from both academia and industry in recent years. At the same time, developments in the Internet of Things (IoT) have empowered modern industries to implement large‑scale IoT ecosystems, such as the Industrial Internet of Things (IIoT). Using knowledge graphs (KG) to process data from the Industrial Internet of Things (IIoT) is a research field worthy of attention, but most of the researched knowledge graph technologies are mainly concentrated in the field of static knowledge graphs, which are composed of triples. In fact, many graphs also contain some dynamic information, such as time changes at points and time changes at edges; such knowledge graphs are called Temporal Knowledge Graphs (TKGs). We consider the temporal knowledge graph based on the projection and change of space. In order to combine the temporal information, we propose a new representation of the temporal knowledge graph, namely TBRm, which increases the temporal dimension of the translational distance model and utilizes relational predicates in time add representation in time dimension. We evaluate the pro‑ posed method on knowledge graph completion tasks using four benchmark datasets. Experiments demonstrate the effectiveness of TBRm representation in the temporal dimension. At the same time, it is also practiced on a network security data set of the Industrial Internet of Things. The practical results prove that the TBRm method can achieve good performance in terms of the degree of harm to IIoT network security.


Introduction
With the rapid development of Industrial Internet of Things (IIoT) and Knowledge Graph (KG) in recent years, IIoT applications in the industrial field are highly sensitive and critical, such as Industrial Control System (ICS), which integrates hardware and software to monitor and control the industrial environment operation of the system and its related components [1]. A Knowledge Graph (KG) stores facts about the real world as a collection represented by triples. Each triple in the KG is represented as <s, p, o>, where s and o represent the subject and object, and p represents the predicate connecting the subject s and the object. The problem with link prediction is to find the most suitable triples (s, p, ?) or (?, p, o) to complete knowledge graphs [2]. Our focus is on the temporal knowledge graph, which adds time information to the triples. As shown in Figure 1, the time series knowledge graph essentially wants to extend the traditional static knowledge graph in the time dimension. The form of the problem with link prediction becomes that it is most likely to be completed under the given time information. That is to say, facts in temporal KG can also have the form of (subject, predicate, object, timestamp) or (subject, predicate, object, time predicate, timestamp), which is used to increase the general triple (s, p, o). For example, facts such as (Donald Trump, born, US, 1946) or (Donald Trump, President, US, occurs Since 2017-01) express time information about the facts related to Donald Trump [3]. The former expresses the relation type of the predicate that occurs at a specific point in time, whereas the latter uses the time predicate "occurs Since" to express the disclosed time period (time range).  [3]. The former expresses the relation type of the predicate that occurs at a specific point in time, whereas the latter uses the time predicate "occurs Since" to express the disclosed time period (time range). Recently, a lot of research work has been devoted to the representation learning of TKG. The method of link prediction is generally to embed the subject and predicate of the triple, and then use the scoring function to score. The temporal representation of knowledge graphs remains challenging due to the sparsity and irregularity of dynamic temporal information. The CYGNet model [4] utilizes the historical information of knowledge by designing a special replication module; meanwhile, a generation module is designed to predict the knowledge that appears for the first time. TA-DistMult [5] creates a temporal relationship by treating the characters in r and t as a sequence, and proposes a digit-level LSTM for learning the factual representations that contain temporal information in KGs and can be directly applied to current KGs In the existing scoring function method in the completion task. Through the study of these models, we improved the TransR model and proposed a new TKG representation learning method. The proposed TBRm (Time boundary relationship mapping) represents time information as a single dimension. We map the relationship formed by the TransR model to the time dimension, and by embedding the mapped time predicate into the standard scoring function of knowledge graph completion, Circular neural networks are used to learn the time-aware representation of relationship types. We conduct extensive experiments on four benchmark TKG datasets, and the results show that TBRm can effectively model TKG data through relationships with temporal attributes.
In the direction of IIoT, taking cyberattacks on industrial IOT devices as an example, according to statistics from Kaspersky researchers, the number of cyberattacks on IOT devices jumped from 639 million in 2020 to 1.5 billion in 2021 [6]. In addition to the attack itself, there are other factors that affect the damage of network attacks. In this paper, according to the Edge-IIoT dataset, we use the attacker (represented by IP address) as the subject, the sensor and brake (represented by IP address) as the object, and the attack type as the predicate relationship to establish a static knowledge map. Furthermore, based on the established TBRm method, we take the duration of network attacks as another indicator to judge the degree of harm, form a temporal knowledge graph based on the Recently, a lot of research work has been devoted to the representation learning of TKG. The method of link prediction is generally to embed the subject and predicate of the triple, and then use the scoring function to score. The temporal representation of knowledge graphs remains challenging due to the sparsity and irregularity of dynamic temporal information. The CYGNet model [4] utilizes the historical information of knowledge by designing a special replication module; meanwhile, a generation module is designed to predict the knowledge that appears for the first time. TA-DistMult [5] creates a temporal relationship by treating the characters in r and t as a sequence, and proposes a digit-level LSTM for learning the factual representations that contain temporal information in KGs and can be directly applied to current KGs In the existing scoring function method in the completion task. Through the study of these models, we improved the TransR model and proposed a new TKG representation learning method. The proposed TBRm (Time boundary relationship mapping) represents time information as a single dimension. We map the relationship formed by the TransR model to the time dimension, and by embedding the mapped time predicate into the standard scoring function of knowledge graph completion, Circular neural networks are used to learn the time-aware representation of relationship types. We conduct extensive experiments on four benchmark TKG datasets, and the results show that TBRm can effectively model TKG data through relationships with temporal attributes.
In the direction of IIoT, taking cyberattacks on industrial IOT devices as an example, according to statistics from Kaspersky researchers, the number of cyberattacks on IOT devices jumped from 639 million in 2020 to 1.5 billion in 2021 [6]. In addition to the attack itself, there are other factors that affect the damage of network attacks. In this paper, according to the Edge-IIoT dataset, we use the attacker (represented by IP address) as the subject, the sensor and brake (represented by IP address) as the object, and the attack type as the predicate relationship to establish a static knowledge map. Furthermore, based on the established TBRm method, we take the duration of network attacks as another indicator to judge the degree of harm, form a temporal knowledge graph based on the degree of harm of network attacks, increase the judgment variables to judge the degree of harm of network attacks, and make the judgment basis for the degree of harm more convincing.

Related Work
With the development of artificial intelligence technology, people are more and more interested in KG embedding tasks, among which the most successful method is the embedding-based technique proposed by Nayyeri et al. [7][8][9]. These techniques map entities and relationships into a continuous space and define a scoring function to infer missing information. For static knowledge graphs, i.e., knowledge graphs without temporal dynamic facts, a class of classic models is translational distance models, such as TransE [10] and its extensions [11,12], which represent two entities as vectors and the relation Modeled as a translation vector. The TransH [13] model is improved on the basis of TransE, and the relationship is represented by two vectors, among which, H in TransH represents Hyperplane; this article is also the first to propose the negative sampling method of unif and bern. The TransR [14] model takes the projection of TransH to the hyperplane one step further, that is, to the space. The essence is to convert the projection vector into a projection matrix. The entity is still represented by a vector, and the relationship is represented by a vector and a matrix. The improvement in effect is not large, but the amount of calculation increases significantly. The R in TransR stands for relation space. Another class of classic models are semantic matching models, which represent relations as matrices and combine head and tail entities using multiplication, using triangular norm to determine how plausible a fact is [15,16]. With the continuous iteration and updating of technology, other models based on neural network methods using feedforward or convolutional layers [17,18] have received extensive attention for better performance. However, these models do not incorporate the temporal dynamics of the facts.
In terms of temporal knowledge graph, Jiang et al. improved the TransE model and proposed the TTransE model [19], which embeds temporal information into the score function, captures the temporal order between relation types, and uses common sense to constrain it. to generate more accurate connection predictions. The HyTE [20] model maps each time to a hyperplane, which is equivalent to mapping different time points to different hyperplanes, and then models a relationship on each hyperplane, which is actually a triple relationship. modeling. Although the time series knowledge graph is a very large graph, it can be a subgraph from each time point. In the article of Temp [21], the time series knowledge graph is divided into multiple subgraphs, and then each subgraph is convolved through GNN, and then the direct time sequence of multiple graphs is modeled based on RNN, and GNN's results are concatenated and such methods take into account not only the spatial dimension, but also representation learning in the temporal dimension.
Compared with the traditional static knowledge graph, the temporal knowledge graph has more time information. The knowledge in the knowledge graph is not static, but will change with time. We divide all algorithm models into three categories based on their differences in the way they process time information: temporal knowledge graph representation model with time constraints, time series coding temporal knowledge graph representation model and path reasoning temporal knowledge graph representation model, as shown in Table 1. Table 1. Classification of known temporal knowledge construction methods.

Temporal Knowledge Graph Represents Categories Model Abbreviation Features
Temporal knowledge graph representation model with time constraints

ETA-TransE [22]
On the basis of the TransE model, a time transfer matrix is constructed based on the difference in the time granularity of the application scenario, which can distinguish the impact of the same time on different types of entities.
ATiSE [23] The influence of mining time on the evolution of entities.
TTransE [24] On the basis of the TransE model, time information is used to embed and represent time points in the triplet by relationship-time merging. By constructing an RNN network to update the embedded representation of the entity after it is affected by time changes.

RE-NET [26]
Convert the time information into a sequence of events (triples) with time information, and finally use the RGCN network to aggregate the information of entities at the same time.
Path reasoning temporal knowledge graph representation model Chang2vec [27] Split the time series knowledge graph into multiple static knowledge graphs according to time nodes Spectrum, recalculate the changed node entity representation and update its embedded representation.
xERTE [28] The model can visualize the interpretability of reasoning and show the reasoning path.

Research Methods
To address the fact that TransE and TransR do not mention time in fact, we propose TBRm, which represents a mapping of relational spaces to temporal dimensions bridged by time-specific matrices.

Problem Statement
Compared with traditional knowledge graphs, Temporal Knowledge Graphs have additional information. Each representation in the original knowledge graph is a triple, whereas each representation in the time-series knowledge graph is a quadruple, which not only includes Subject, object and relational predicate, as well as the time point or time range in which the relational predicate is established.  In terms of network security, the requirements knowledge graphs based on the IIoT network security system, as shown in Figure 3. Determining the degree of harm caused by network attacks, and deploying and focusing on prevention of potentially harmful network attacks in advance are conducive to ensuring the network security of IIoT. Clearly, different types of attacks have different degrees of harm, but it is worth thinking about which variables can be added in addition to the types of attacks to make the determination of the degree of harm more accurate.  In TKG, each fact has a relation (or predicate) ∈ ℛ with subject entity ∈ ℰ and object entity ∈ ℰ within time ∈ . where ℰ and ℛ represent the vocabulary sets corresponding to entity and relation predicates, respectively, and T is the set of time periods or timestamps (if they exist). Bold words s, p, o, t represents the embedding vectors of subject entity s, predicate p, object entity o, and time t in the factual events with temporal information. Let denote the snapshot of TKG in t time period, = , , where denotes the time when a relationship starts, and denotes the time when a relationship ends. = , , , represents the quadruple fact in . Both TransE and TransH assume embedding entities and relations in the same space ℝ . In TransR, for each triple (s, p, o), the entity embedding is set to s, o ∈ ℝ , and the relation embedding is set to r ∈ ℝ . Here, the dimensions owned in entity embedding and relationship embedding may not be the same, that is, k ≠ d. However, none of these three classical models contain temporal information. To solve this problem, we propose a new method In TKG, each fact has a relation (or predicate) p ∈ R with subject entity s ∈ E and object entity o ∈ E within time t ∈ T . where E and R represent the vocabulary sets corresponding to entity and relation predicates, respectively, and T is the set of time periods or timestamps (if they exist). Bold words s, p, o, t represents the embedding vectors of subject entity s, predicate p, object entity o, and time t in the factual events with temporal information. Let G t denote the snapshot of TKG in t time period, t = (t b , t e ), where t b denotes the time when a relationship starts, and t e denotes the time when a relationship ends. g = (s, p, o, t) represents the quadruple fact in G t . Both TransE and TransH assume embedding entities and relations in the same space R k . In TransR, for each triple (s, p, o), the entity embedding is set to s, o ∈ R k , and the relation embedding is set to r ∈ R d . Here, the dimensions owned in entity embedding and relationship embedding may not be the same, that is, k ̸ = d. However, none of these three classical models contain temporal information. To solve this problem, we propose a new method that projects triples in relational space onto an extended temporal space and translates them in temporal space, thus was named TBRm.

Model Structures
For each relation predicate p in the triple, TransR sets a projection matrix M p ∈ R k×d , which can project entities from entity space to relation space. Based on this idea, we extend the time dimension on the basis of relational space to form a time-space that is perpendicular to relational space. Define the Vertical projection matrix V t ∈ R d×d , V t can map the entity set and relation set to the time space, and reconstruct the triples under the limitation of the time dimension, so that the relation predicate contains It can be limited by duration. The overall structure of TBRm is shown in Figure 4.
For each relation predicate p in the triple, TransR sets a projection matrix ∈ ℝ , which can project entities from entity space to relation space. Based on this idea, we extend the time dimension on the basis of relational space to form a time-space that is perpendicular to relational space. Define the Vertical projection matrix ∈ ℝ , can map the entity set and relation set to the time space, and reconstruct the triples under the limitation of the time dimension, so that the relation predicate contains It can be limited by duration. The overall structure of TBRm is shown in Figure 4.
where , ∈ ℝ are the embeddings of subject and object entities, and ∈ ℝ are the embeddings of relational predicates. ‖⋅‖ represents the two-norm.
To introduce the temporal information of relational predicates, TBRm uses the projection matrix to define the projection of relational predicates as: The scoring function f is used for the embedding method of KG completion. The function works on the embedding of the subject e s , the object e o , and the predicate e p of the triple in the time dimension, in order to represent the duration (time span) of a relational predicate, let t = t e − t b , it is worth noting that when the time represents the relational predicate in the form of timestamp, only t b is taken as the current time vector, that is, when t e = t b , t = t b . The value of a scoring function is proportional to the probability that a triple is true; a classic example of a scoring function is:

1.
TrasnE [10]: where e s , e o ∈ R d are the embeddings of subject and object entities, and e p ∈ R d are the embeddings of relational predicates. ∥ · ∥ 2 represents the two-norm.
To introduce the temporal information of relational predicates, TBRm uses the projection matrix V t to define the projection of relational predicates as: where p s t represents the projection of the entity (subject) set at the beginning of the relation predicate on the time dimension, and p o t represents the projection of the entity (object) set at the end of the relation predicate on the time dimension.
The scoring function is correspondingly defined as: At the same time, the constraints of the projection matrix are satisfied, that is, ∥s t p ∥ 2 ≤ 1 , ∥o t p ∥ 2 ≤ 1.

Training Target
Predict (object) entities given a query (s, p, ?, t). The learning goal is to minimize all the cross-entropy loss functions L of TKG snapshots that exist during training. By referring to the training method of the paper [29], we use the following margin-based scoring function as the training goal: where max (x, y) aims to obtain the maximum value between x and y, α is the margin, S is the set of correct triples, and S ′ is the set of incorrect triples. Existing knowledge graphs only contain correct triples. It is reasonable to destroy correct triples (s, p, o) ∈ S by replacing entities, and construct incorrect triples (s ′ , p ′ , o) ∈ S ′ . When breaking triples, we follow [30][31][32][33][34][35][36][37][38][39][40][41] and assign different probabilities to head/tail entity replacement.

Experiment and Discussion
In this section, we demonstrate the effectiveness of TBRm using four public IIoT datasets. First of all, we will explain the experimental settings in detail, including a detailed introduction to the baseline and dataset. Then, we analyzed and discussed the experimental results. We also conducted a comparative study to evaluate the advantages of the TBRm method over other baseline methods. Finally, we put the proposed representation method into practice on the Edge-IIoTset data set to prove the feasibility of TBRm on the Industrial Internet of Things. The specific code can be found at https://github.com/Dash69dash/ temporalKG (accessed on 27 October 2022).

Dataset
We conducted experiments on the connection prediction of TBRm on four benchmark data sets related to the industrial Internet.At the same time, in order to see the performance of the proposed method on the Industrial Internet of Things, we used another Industrial Internet of Things data set to test the representation effect of TBRm. Table 2 summarizes the statistics of the dataset. The Bosch production line internal fault data set (BPLP) [42] describes the measurement results of parts as they move in the Bosch production line. Each part has a unique ID and contains a large number of anonymous features. Features are named according to the agreement that tells you the production line, the workstations on the production line, and the feature number. We separate the files according to the types of features contained in the files: numbers, categories, parts and workstations on the production line as subject entities and object entities, and fault types as relational predicates in the triplet. Finally, there is a file with a date feature, and the date function provides a timestamp of each measurement time. MOOC Platform User Behavior Data Set (MOOC-Ub) [43] includes the learning activities of all users on the school's online platform from August 2015 to August 2017. User information is the information of users of the school online, including: gender, year of birth and education level. Course information includes the course start date, course end date, course category, and course type. Extract and filter the user ID as the subject entity, the course category and type as the object entity, and the user information and course information as the relational predicate. Combining the start and end times of the course, the model of the time series knowledge graph is used for experiments. Each json file contains user tracking logs for a specific period of time. The NFT Ethereum transaction data set (NFT) [44] contains the transaction activities of the Ethereum non-homogeneous currency (NFT) from 1 April 2021 to 25 September 2021. It is purely constructed from on-chain data and represents the activities of 9292 NFT smart contracts on the Ethereum blockchain during the period. The Information Exposure from Consumer IoT Devices (IE-IoTD) [45] dataset processed and analyzed the information leakage of 81 devices located in laboratories in the United States and the United Kingdom. We filtered out a total of 23,475 triplet relationships with timestamps.  Ub  NFT  IE-IoTD  Edge-IIoTset   #Entities  3024  24,100  5422  12,564  108,576  #Relation  145  274  186  245  14  #Training  7213  19,151  7274  18,780  24,301  #Validation  5327  7263  4263  4072  19,281  #Test  3348  2854  1000  2349  4820 The Edge-IIoTset [46] data set identifies and analyzes 14 types of network attack methods. These attacks can be summarized as five threats, namely, DoS/DDoS attacks, Information gathering, Man in the middle attacks, Injection attacks, and Malware attacks. At the same time, the data set also indicates the start time and end time of an attack (or the time point of the attack). Edge-IIoTset's IOT data is generated from various IOT devices (more than 10 types), such as Low-cost digital sensors for sensing temperature and humidity, Ultrasonic sensor, Water level detection sensor, pH Sensor Meter, Soil Moisture sensor, Heart Rate Sensor, Flame Sensor, etc.). The Edge-IIoTset dataset also records the IP address of the attacker and the sensor being attacked.
Evaluation Protocol According to the previous work [29], we filtered out the triplet data that meets our experimental requirements from these four Industrial Internet of Things data sets, and divided them into training sets, verification sets, and test sets according to the characteristics of the data, which are 80%/10%/10%, respectively [47][48][49][50]. We report the average countdown ranking (MRR) and Hits@1/3/10 (the proportion of correct test cases in the top 1/3/10) to measure the performance of our model and the comparison model [51][52][53][54][55]. The calculation formula of MRR is as follows: where S is the set of triples, |S| is the number of triples, and 〖rank 〗_i refers to the link prediction ranking of the i-th triplet.
HITS@n refers to the average proportion of triples ranked less than n in the link prediction [56,57]. The specific calculation method is as follows: where the symbols involved in the above formula are the same as those involved in the MRR calculation formula, and I (·) is the indicator function (if the condition is true, the function value is 1, otherwise it is 0). n usually takes the values of 1, 3, and 10. Model configuration Exclude empty accounts that have not been used for a long time in the MOOC-Ub data set, and use the filtered triples and time for verification. The coefficient α is adjusted from 0.1 to 0.9, with a step size of 0.1. The batch size is set to 1024.The training epoch is limited to 50, which is sufficient to converge in most cases. The embedding dimension is set to 200 to be consistent with the setting of Jin et al. [30]. The baseline results are also from Jin et al. [30].
Selection of data in Edge-IIoTset dataset Since the Edge-IIoTset dataset not only contains the data described in Section 4.1, it also contains 61 flow features and two new attributes, which are stored together in a CSV file. We use python programs to extract the data information we need from it, including: The IP addresses of the network attacker and the attacked sensor, the attack methods of different attackers on the sensor, and the time when the network attack started and ended. We regard the attacker's IP address as the subject entity set, the attacked sensor (IP address) as the object entity set, and the attack method as the predicate relationship set. The above data sets are used to form triples of knowledge graphs. Due to the different time units of network attacks (milliseconds, seconds, minutes, and hours), therefore, the time data is dimensionless. Use the Z-score normalization method: where µ b and µ e are the mean values of t b and t e of the same time unit category, respectively. σ b and σ e are the standard deviations of t b and t e in the same time unit category, respectively.

Experimental Results
Tables 3 and 4 report the link prediction results of the TBRm and baseline methods on the four IIoT data sets. On the data set in Table 3, the static KGE method lags far behind TA-DistMult or TTransE, because it cannot capture time dynamics. It can also be observed that the performance of all static KGE methods is usually better than that of HyTE. We believe that this is because HyTE slices the dynamic knowledge map with time into multiple static knowledge maps, which are represented independently on each static map, and lack coherence in time updates. Table 3 also shows that TBRm is significantly better than other baseline methods in BPLP and MOOC-Ub.  The best results are shown in bold.
We also observed in Table 4 that TBRm's performance on NFT and IE-IoTD is not always the best, especially on NFT. In fact, this is due to the excessive concentration of time information participating in the NFT data set. The excessive concentration of time information makes the time information data interval carried smaller, making it more difficult to distinguish triples. However, the Hyte method of slicing the time series knowledge map by time has a good performance on this data set. Although TBRm performs better on other data sets with a more balanced distribution of time information, how to solve this shortcoming of TBRm is a meaningful direction for further research. Table 5 shows the percentage of TBRm's four performance indicators on the Edge-IIoTset dataset. We can see that in addition to Hits@1, TBRm has achieved relatively high scores in the tests of the other three performance indicators. Analyzing the reasons, it was found that in the Edge-IIoTset data set, the number of subject entities of the network attacker is too large, however, there are only 14 kinds of attacks that act as predicates, and the gap between subject entity set and predicate relation set is too obvious. Therefore, in the process of link prediction, the probability of successful prediction for the first time will become very small, and the percentage value of Hits@1 will decrease accordingly.

Conclusions
Describing and inferring knowledge graphs with time constraints is a challenging problem. In this paper, we exploit the mapping mechanism to address this problem, assuming that future facts can be predicted from historical facts. The proposed TBRm selects future facts based on known facts that appeared in the past. The results show that TBRm has good performance in predicting future facts in TKGs. We propose a time-constrained relational mapping method to learn temporal knowledge graph fact representations that can be used in conjunction with current link prediction scoring function methods. Experiments on five temporal knowledge graph data demonstrate the effectiveness of the method and its feasibility on the Industrial Internet of Things.
Funding: This research was funded by Intelligent IOT and Integrated security of industrial information physics systems, Key projects of the National Natural Science Foundation of China, 2022-01-01 to 2026-12-31, grant number 62133014. And The APC was funded by National Natural Science Foundation of China.