Temporal Multimodal Data-Processing Algorithms Based on Algebraic System of Aggregates

Abstract

In many tasks related to an object’s observation or real-time monitoring, the gathering of temporal multimodal data is required. Such data sets are semantically connected as they reflect different aspects of the same object. However, data sets of different modalities are usually stored and processed independently. This paper presents an approach based on the application of the Algebraic System of Aggregates (ASA) operations that enable the creation of an object’s complex representation, referred to as multi-image (MI). The representation of temporal multimodal data sets as the object’s MI yields simple data-processing procedures as it provides a solid semantic connection between data describing different features of the same object, process, or phenomenon. In terms of software development, the MI is a complex data structure used for data processing with ASA operations. This paper provides a detailed presentation of this concept.

1. Introduction

Humans perceive real-world objects through the multiple senses. Semantic fusion of multi-type (multimodal) information about the object of observation received using multi-channel sensing is a natural process for the human brain. Following this natural principle, many scientific and engineering tasks also require complex semantic descriptions of an object of study based on the fusion of multimodal data received from multiple devices.

In many cases, the object is supervised over the course of time. Such timewise observation can help in understanding the dynamics of the object’s behavior and is necessary in many applications. Then, such a timewise complex description requires a collection of multimodal data that are obtained from several sensors measuring certain parameters of the object’s behavior, cameras recording the appearance of the object, etc., over time, not necessarily received simultaneously. This brings the problem of correct multimodal data representation based on data synchronization and aggregation.

Another aspect of using multimodality is its employment for object recognition. The latest advancement in this area concerns multimodal machine learning, which involves integrating and modeling information from multiple heterogeneous sources of data [1]. However, this approach reveals several challenges related to heterogeneity of the data.

According to [2], the first fundamental challenge is comprehensive data representation that takes into consideration the complementarity and redundancy of multiple modalities. The second challenge is multimodal data mapping that enables matching data of different modalities. The third challenge is data alignment that consists of the necessity to identify direct relations between elements from different modalities. The fourth challenge is data fusion, which yields united information of different modalities. The fifth challenge is co-learning, which consists of transferring knowledge based on different modalities. As stated in [1], there are even more challenges that include reasoning, generation, and quantification, along with data representation, alignment, and transference. All these and related issues require correct and comprehensive multimodal data representation, which is key to overcoming challenges related to different aspects of multimodal data analysis and processing.

The rest of the paper is organized as follows. Section 2 presents a related work review. Section 3 formulates the requirements of temporal multimodal data processing and explains the need to employ the Algebraic System of Aggregates (ASA) for the formal specification of an object. Section 4 explains the basic concepts of the ASA. Section 5 provides a detailed presentation of the algorithms of operations on aggregates. Section 6 presents the notion of a multi-image of an object and proposes an algorithm for multi-image formation. Section 7 offers a use case and provides the discussion of ASA operations application for multi-image formation. Finally, Section 8 concludes the paper.

2. Related Work

There are a number of papers presenting different approaches and views on the task of data fusion/data aggregation in various contexts.

Jesus et al. [3] formally defined the concept of aggregation, reviewed distributed data aggregation algorithms, and provided taxonomy of the main aggregation techniques. Comparing different techniques in their survey, the authors showed that among main data aggregation classes, the hierarchical-based approach is the cheapest option; the sketch-based approach could be considered fast but not very precise; the averaging technique gives a higher level of precision but with a much slower execution, which might not be very efficient to operate on dynamic networks, although very few approaches would actually be practical when there are dynamic settings, which presents a challenge for further research on improvements in the efficiency of dynamic networks.

Ribeiro et al. [4] proposed an algorithm for intelligent information fusion in uncertain environments, named fuzzy information fusion (FIF), which is based on multi-criteria decision making and computational intelligence. In this study, data fusion was considered a process of aggregating data from multiple sources into a single composite with higher information quality. The authors analyzed various methods of data fusion. While they considered fuzzy Set theory to be of high applicability, it needs application domain knowledge for data representation, which makes this method not fully universal and creates room for further research in this direction.

Oliveira et al. [5] presented an application of the fuzzy information fusion algorithm for the aggregation of various sources of heterogeneous information to generate value-added maps. In their work, they used operators from different classes of operators, such as algebraic, average, and reinforcement, to fuse, e.g., aggregate data. Validation with specific scenarios allowed authors to compare aggregation operators. The multiplicative FIMICA operator was identified as the most consistent operator that gave the best classification outputs for the scenarios used. It shows, however, that for certain use cases, there is a need for specialized operators that yield efficiently aggregated data depending on a specific scenario.

Lahat et al. [6] considered data fusion to be “the analysis of several data sets such that different data sets can interact and inform each other” and studied multimodality as a form of diversity. In the paper, the authors focused their attention on different aspects of multimodality, considering it in the context of various applications. They also outlined the importance of the development of single-set analysis methods for advanced data fusion.

Marinoni et al. [7] considered data fusion in the context of image pre-processing with relation to transfer learning in remote sensing. In their paper, the authors proposed metrics that quantify the maximum information extraction performance to be achieved by multimodal remote-sensing analysis. Empirical outcomes presented in the paper demonstrated how the accuracy performance of a standard classifier applied to a multimodal data set can be improved using the reliability metric. The approach proposed by the authors can be used to improve multimodal remote-sensing information extraction.

Gaonkar et al. [8] provided a study on advancements in multimodal signal processing. In the paper, focus was given to multimodal data representation and information fusion. The authors considered information fusion based on model-agnostic and model-based approaches.

Oliveira et al. [9] focused on the application of data fusion in a decision support system (DSS). The effectiveness of such a system depends on reliable fusion of data coming from multiple sources. The paper proposed high-level architecture of the DSS.

As was stated in the Introduction, one of the promising applications of multimodality is multimodal machine learning (MML). This topic stimulated a number of resent research works.

Liang et al. [10] discussed the foundational principles in multimodal research and, in particular, the principle of heterogeneity and the principle of interconnection. The authors also provided taxonomy of six core challenges in multimodal machine learning (representation, alignment, reasoning, generation, transference, and quantification) and gave their comprehensive analysis. They outlined different aspects of these challenges, and some of them are of particular interest in the context of the research purposes of this paper. Specifically, modality connections and interconnections, as an essential part of multimodal models, can show how modalities are related and how their elements interact. There is, however, a need, which the authors pointed out as one of the future directions, to formally define the core principles of heterogeneity, connections, and interactions. It requires a mathematical framework to be able to capture causal, logical, and temporal connections and interactions, which formulates a mathematical and algorithmic problem to consider.

Guo et al. [11] studied deep multimodal representation learning frameworks, including modality-specific representations, joint representation and coordinated representation, and encoder–decoder framework. They outlined one of the challenges that still exists in the context of machinery comprehension of information from multiple sensory organs, which is the heterogeneity gap in multimodal data. Aiming to narrow this gap, the authors summarized some typical models in deep multimodal representation learning, including probabilistic graphical models, multimodal autoencoders, deep canonical correlation analysis, generative adversarial networks, and attention mechanisms. Their analysis of different learning frameworks showed that one of the main disadvantages of existing frameworks and models is the difficulty of coordinating more than two modalities. It formulates a practical challenge for further research to overcome the issue of multiple modalities.

Baltrusaitis et al. [2] provided an overview of the recent advances in multimodal machine learning and presented a summary of applications enabled by multimodal machine learning. In their work, they introduced taxonomy of multimodal machine learning, which includes five core challenges: representation, translation, fusion, alignment, and co-learning. Some of these challenges have already been studied quite well. In contrast, there are a few more recent ones (e.g., representation and translation), which are now leading the creation of new multimodal algorithms and are still of interest for further research.

Kline et al. [12] provide a summary of multimodal data fusion applications for solving medical diagnostics problems. The paper shows how medical data of different modalities (e.g., text/image, EHR/genomic/time series) recorded and extracted can be used in a specific use case, giving a firm understanding of the practical importance of multimodal data processing.

3. Approach and Requirements for Temporal Multimodal Data Processing

In our research, we consider an approach to temporal multimodal data processing based on the formal specification of the objects under study. For this purpose, we need a theoretical apparatus to provide the logic of presentation and processing of temporal multimodal data at the level of mathematics, algorithms, and software, taking into account such features of this data:

• Multimodality means that the object is determined by a collection of data of a different nature; the logic of presenting and processing data about the object depends on the qualitative and quantitative composition of the data set.
• Temporality means that the elements of data collection are ordered by the time of their receiving; the order of following individual elements of the data sequence affects the result of processing the entire set of data of the object.

The description of the object and the effective processing of temporal multimodal data require specific mathematical abstractions and mechanisms to operate them. The basic requirements for a mathematical apparatus to describe and process data include the possibility of the following:

• Presenting a data set as a structure of semantically interconnected elements.
• Considering the sequence of data set elements when performing logical operations on them.
• Reordering elements of the data set.

Let us consider several candidates for a mathematical concept to meet these requirements. As the data are temporal, the first possible candidate is the theory of time series (TTS) [13,14]. The TTS is one of the branches of mathematical statistics that deals with the analysis of stochastic processes. The TTS enables the analysis of data sequences, as well as the study of trends, predictions, and other similar data processing. However, time series do not allow any logical operations of the data elements. Therefore, the TTS cannot be employed for logical processing of temporal multimodal data.

Two other candidates are the theory of sets (TS) [15] and the theory of multisets (TMS) [16,17]. Both of them provide flexible mechanisms for the logical processing of data sets; however, they do not consider the sequence of elements and, accordingly, do not provide the possibility of ordering the elements in a set of temporal multimodal data.

Thus, neither of the considered theories fully correspond to the needs of temporal multimodal data processing based on the formal specification of the objects under study. However, the abovementioned requirements are satisfied by the Algebraic System of Aggregates (ASA) [18,19]. This mathematical concept enables data processing with consideration of both required data features, namely multimodality and temporality (

Table 1. ASA comparison.

Features	TTS	TS	TMS	ASA
Ordering	yes	no	no	yes
Logical operations	no	yes	yes	yes
Fuzziness	partly	partly	partly	yes
Data aggregation	no	no	partly	yes
Temporality	yes	no	no	yes

).

Let us consider the main provisions and possibilities of the ASA for the presentation and processing of temporal multimodal data describing an object.

4. Basics Notions of ASA

The ASA [18,19] is an algebraic system whose carrier is a non-empty set of objects, which we call aggregates.

Definition 1. 

An aggregate A is an ordered finite collection of elements defined by a tuple of sets $\left\{A\right\}$ and a tuple of tuples of elements $\langle A\rangle$, and the elements $a_i^j$ of each tuple of elements $\langle a_i^j\rangle {}_{i=1}^{n_j}\in \langle A\rangle$ belong to the corresponding set $M_j\in \left\{A\right\}$, $j=\left[1\dots N\right]$, which specifies a one-to-one relationship between the sequence of sets and the sequence of tuples of elements:

$$A=⟦M_j\mid \langle a_i^j\rangle {{}_{i=1}^{n_j}⟧}_{j=1}^N=⟦\left\{A\right\}\mid \langle A\rangle ⟧,$$

(1)

where $\left\{A\right\}$ is a tuple of sets $M_j$; $\langle A\rangle$ is a tuple of tuples of elements $\langle a_i^j\rangle {}_{i=1}^{n_j}$; and $a_i^j$ is a separate element (value) or a composite element (a tuple of homogeneous values or a tuple of heterogeneous values), $a_i^j\in M_j$.

The defining features of the aggregate, which distinguish this mathematical abstraction from others, are the following:

• An aggregate is a complex mathematical object, all of whose components are ordered;
• Elements of the tuples can be individual values or tuples of values, and the tuples of values can consist of both values of the same type and values of different types, while each tuple contains elements belonging to one set.

According to (1), the elements of the first tuple belong to the first set, the elements of the second tuple belong to the second set, etc. The sequence of sets in an aggregate determines how operations on the aggregate are performed. Sets in a tuple of sets can be repeated; this means that the aggregate includes several tuples consisting of elements of the same type. In software code, an aggregate can be defined as a data structure presented in Listing 1.

Listing 1. Interface of a data structure for representing an aggregate.

Aggregate structure:       collection ← ordered collection of tuples       length ← number of tuples in the aggregate       Initialize Aggregate             define length of the aggregate             create collection of size length             initialize tuples in collection       Insertion (entity) ← add an entity, i.e., a set of values from different tuples       Extraction (entity) ← delete the specific entity       TupleInsertion (position) ← add a new tuple to the specific position in the aggregate       TupleExtraction (position) ← delete the specific tuple       AscendingSorting (primaryTuple) ← sort tuples in the aggregate by the defined primary tuple in asceding order       DecendingSorting (primaryTuple) ← sort tuples in the aggregate by the defined primary tuple in descending order       Singling (primatyTuple) ← remove all non-unique entities by the defined primary tuple       SetsOrdering (orderOfTuples) ← reorder tuples by the defined order

As the order of elements in the aggregate is a crucial point in the ASA, the result of carrying out operations on aggregates depends on the aggregates’ compatibility.

Definition 2. 

Aggregates $A_1$ and $A_2$ are called compatible ($A_1\doteqdot A_2$) if they have the same length, and the type and sequence of sets in them match, that is, the conditions are met:

$$\{\begin{array}{c}\mid A_1\mid =\mid A_2\mid \\ \left\{A_1\right\}\equiv \left\{A_2\right\}\end{array}.$$

(2)

For example, the aggregates defined by (3) and illustrated by Figure 1 are compatible.

$$\begin{array}{c}{A}_1=⟦M_1,M_{2,} M_3\mid \langle a_1^{1,1},a_2^{1,1},a_3^{1,1}\rangle ,\langle a_1^{1,2},a_2^{1,2}\rangle ,\langle a_1^{1,3},a_2^{1,3},a_3^{1,3},a_4^{1,3}\rangle ⟧,\\ A_2=⟦M_1,M_{2,} M_3\mid \langle a_1^{2,1},a_2^{2,1}\rangle ,\langle a_1^{2,2},a_2^{2,2},a_3^{2,2},a_4^{2,2},a_5^{2,2}\rangle ,\langle a_1^{2,3},a_2^{2,3}\rangle ⟧.\end{array}$$

(3)

In Figure 1, the color of an element represents data modality (elements of set $M_1$ are blue, elements of set $M_2$ are brown and elements of set $M_3$ are green), the first value in the element’s number designates the aggregate ($A_1$ or $A_2$), and the second value in the element’s number is an ordering number of the element in the tuple belonging to a certain set. For example, the blue circle, which contains the numbers 1-1, represents the element $a_1^{1,1}$ that belongs to the set $M_1$ from the definition of $A_1$ and the green circle, which contains the numbers 2-1, represents the element $a_1^{2,3}$ that belongs to the set $M_3$ from the definition of $A_2$.

Definition 3. 

Aggregates $A_1$ and $A_2$ are called quasi-compatible ($A_1\doteq A_2$) if the type and sequence order of the sets in them partially coincide, while there is no requirement for the equality of the lengths of these aggregates, i.e., the conditions are fulfilled:

$$\{\begin{array}{c}\left\{A_1\right\}\not\equiv \left\{A_2\right\}\\ \left\{A_1\right\}\cap \left\{A_2\right\}\ne \varnothing .\end{array}$$

(4)

For example, the aggregates defined by (5) and illustrated by Figure 2 are quasi-compatible.

$$\begin{array}{c}{A}_1=⟦M_1,M_{2,} M_3^1\mid \langle a_1^{1,1},a_2^{1,1},a_3^{1,1}\rangle ,\langle a_1^{1,2},a_2^{1,2}\rangle ,\langle a_1^{1,3},a_2^{1,3},a_3^{1,3},a_4^{1,3}\rangle ⟧,\\ A_2=⟦M_1,M_{2,} M_3^2\mid \langle a_1^{2,1},a_2^{2,1}\rangle ,\langle a_1^{2,2},a_2^{2,2},a_3^{2,2},a_4^{2,2},a_5^{2,2}\rangle ,\langle a_1^{2,3},a_2^{2,3}\rangle ⟧.\end{array}$$

(5)

Definition 4. 

Aggregates $A_1$ and $A_2$ are called incompatible ($A_1\circeq A_2$), if the type and sequence of the sets in them do not match, that is, the condition is fulfilled:

$$\left\{A_1\right\}\cap \left\{A_2\right\}=\varnothing .$$

(6)

For example, the aggregates defined by (7) and illustrated by Figure 3 are incompatible.

$$\begin{array}{c}{A}_1=⟦M_1^1,M_2^1, M_3^1\mid \langle a_1^{1,1},a_2^{1,1},a_3^{1,1}\rangle ,\langle a_1^{1,2},a_2^{1,2}\rangle ,\langle a_1^{1,3},a_2^{1,3},a_3^{1,3},a_4^{1,3}\rangle ⟧,\\ A_2=⟦M_1^2,M_2^2, M_3^2\mid \langle a_1^{2,1},a_2^{2,1}\rangle ,\langle a_1^{2,2},a_2^{2,2},a_3^{2,2},a_4^{2,2},a_5^{2,2}\rangle ,\langle a_1^{2,3},a_2^{2,3}\rangle ⟧.\end{array}$$

(7)

A special case of incompatibility is hidden compatibility.

Definition 5. 

Aggregates $A_1$ and $A_2$ are called hiddenly compatible, $A_1 (\doteqdot ) A_2$, if both aggregates have the same set of sets, but their ordering is different, i.e., the conditions are fulfilled:

$$\{\begin{array}{c}\left\{A_1\right\}\not\equiv \left\{A_2\right\}\\ \mid A_1\mid =\mid A_2\mid =N\\ \forall M_j\subset \left\{A_k\right\},\end{array}$$

(8)

where $j=\left[1,\dots ,N\right]$, $k=\left[1, 2\right]$.

For example, the aggregates defined by formulas (9) and illustrated by Figure 4 are hiddenly compatible.

$$\begin{array}{c}{A}_1=⟦M_1,M_{2,} M_3\mid \langle a_1^{1,1},a_2^{1,1},a_3^{1,1}\rangle ,\langle a_1^{1,2},a_2^{1,2}\rangle ,\langle a_1^{1,3},a_2^{1,3},a_3^{1,3},a_4^{1,3}\rangle ⟧,\\ A_2=⟦M_2, M_3, M_1\mid \langle a_1^{2,1},a_2^{2,1}\rangle ,\langle a_1^{2,2},a_2^{2,2}, a_3^{2,2},a_4^{2,2},a_5^{2,2}\rangle ,\langle a_1^{2,3},a_2^{2,3}\rangle ⟧.\end{array}$$

(9)

Hiddenly compatible aggregates can be made compatible by applying certain operations to them.

5. Algorithms of Operations on Aggregates

The operations on aggregates in the ASA include logical operations, ordering operations, and arithmetic operations.

5.1. Logical Operations

The logical operations [18] on aggregates are union, intersection, exclusive intersection, difference, and symmetric difference. The result of any logical operation depends on the aggregates’ compatibility. For example, the rule for the union operation can be mathematically defined as follows.

The union of the aggregates $A_1$ and $A_2$ is the aggregate $R_{\cup }$, which contains elements of the tuples that belong to both aggregates and are ordered in the following way:

1.

If $A_1\doteqdot A_2$, then aggregates $A_1$ and $A_2$ are defined as

$$A_1=⟦M_1,M_2,\dots ,M_N\mid \langle a_1^{1,1},a_2^{1,1},\dots ,a_{n_1^1}^{1,1}\rangle ,\langle a_1^{1,2},a_2^{1,2},\dots ,a_{n_2^1}^{1,2}\rangle ,\dots ,\langle a_1^{1,N},a_2^{1,N},\dots ,a_{n_N^1}^{1,N}\rangle ⟧,$$

$$A_2=⟦M_1,M_2,\dots ,M_N\mid \langle a_1^{2,1},a_2^{2,1},\dots ,a_{n_1^2}^{2,1}\rangle ,\langle a_1^{2,2},a_2^{2,2},\dots ,a_{n_2^2}^{2,2}\rangle ,\dots ,\langle a_1^{2,N},a_2^{2,N},\dots ,a_{n_N^2}^{2,N}\rangle ⟧,$$

and elements of i-tuple of the aggregate $A_2$ are added to the end of i-tuple of the aggregate $A_1$:

$$\begin{array}{c}{R}_{\cup }=A_1\cup A_2=⟦M_1,M_2,\dots ,M_N\mid \langle a_1^{1,1},a_2^{1,1},\dots ,a_{n_1^1}^{1,1},a_1^{2,1},a_2^{2,1},\dots ,a_{n_1^2}^{2,1}\rangle ,\\ \langle a_1^{1,2},a_2^{1,2},\dots ,a_{n_2^1}^{1,2},a_1^{2,2},a_2^{2,2},\dots ,a_{n_2^2}^{2,2}\rangle ,\dots ,\langle a_1^{1,N},a_2^{1,N},\dots ,a_{n_N^1}^{1,N},a_1^{2,N},a_2^{2,N},\dots ,a_{n_N^2}^{2,N}\rangle ⟧.\end{array}$$

(10)

2.

If $A_1\doteq A_2$, then aggregates $A_1$ and $A_2$ are defined as

$$\begin{array}{c}{A}_1=⟦M_1,M_2^1,\dots ,M_k,\dots ,M_{N^1}^1\mid \langle a_1^{1,1},a_2^{1,1},\dots ,a_{n_1^1}^{1,1}\rangle ,\langle a_1^{1,2},a_2^{1,2},\dots ,a_{n_2^1}^{1,2}\rangle ,\dots ,\langle a_1^{1,k},a_2^{1,k},\dots ,a_{n_k^1}^{1,k}\rangle ,\dots ,\\ \langle a_1^{1,N^1},a_2^{1,N^1},\dots ,a_{n_{N^1}^1}^{1,N^1}\rangle ⟧,\end{array}$$

$$\begin{array}{c}{A}_2=⟦M_1,M_2^2,\dots ,M_k,\dots ,M_{N^2}^2\mid \langle a_1^{2,1},a_2^{2,1},\dots ,a_{n_1^2}^{2,1}\rangle ,\langle a_1^{2,2},a_2^{2,2},\dots ,a_{n_2^2}^{2,2}\rangle ,\dots ,\langle a_1^{2,k},a_2^{2,k},\dots ,a_{n_k^2}^{2,k}\rangle ,\dots ,\\ \langle a_1^{2,N^2},a_2^{2,N^2},\dots ,a_{n_{N^2}^2}^{2,N^2}\rangle ⟧,\end{array}$$

(a) elements of the i-tuple of the aggregate $A_2$ are added to the end of the i-tuple of the aggregate $A_1$, if the elements of these tuples belong to the same i-set;

(b) for all i-tuples whose elements belong to different sets, the tuple of tuples of aggregate $A_2$ is added to the end of the tuple of tuples of aggregate $A_1$, and the tuple of sets of aggregate $A_2$ is added to the end of the tuple of sets of aggregate $A_1$, with the exception of tuples subject to rule (a), which are excluded from the tuple of tuples:

$$\begin{array}{c}{R}_{\mathrm{U}}=A_1\cup A_2=⟦M_1,M_2^1,\dots ,M_k,\dots ,M_{N^1}^1,M_2^2,\dots ,M_{N^2}^2\mid \langle a_1^{1,1},a_2^{1,1},\dots ,a_{n_1^1}^{1,1},\\ a_1^{2,1},a_2^{2,1},\dots ,a_{n_1^2}^{2,1}\rangle ,\langle a_1^{1,2},a_2^{1,2},\dots ,a_{n_2^1}^{1,2}\rangle ,\dots ,\langle a_1^{1,k},a_2^{1,k},\dots ,a_{n_k^1}^{1,k},a_1^{2,k},a_2^{2,k},\dots ,a_{n_k^2}^{2,k}\rangle ,\dots ,\\ \langle a_1^{1,N^1},a_2^{1,N^1},\dots ,a_{n_{N^1}^{1,N^1}}^1\rangle ,\langle a_1^{2,2},a_2^{2,2},\dots ,a_{n_2^2}^{2,2}\rangle ,\dots ,\langle a_1^{2,N^2},a_2^{2,N^2},\dots ,a_{n_{N^2}^2}^{2,N^2}\rangle ⟧\end{array}$$

(11)

3.

If $A_1\circeq A_2$, then aggregates $A_1$ and $A_2$ are defined as

$$A_1=⟦M_1^1,M_2^1,\dots ,M_{N^1}^1\mid \langle a_1^{1,1},a_2^{1,1},\dots ,a_{n_1^1}^{1,1}\rangle ,\langle a_1^{1,2},a_2^{1,2},\dots ,a_{n_2^1}^{1,2}\rangle ,\dots ,\langle a_1^{1,N^1},a_2^{1,N^1},\dots ,a_{n_{N^1}^1}^{1,N^1}\rangle ⟧,$$

$$A_2=⟦M_1^2,M_2^2,\dots ,M_{N^2}^2\mid \langle a_1^{2,1},a_2^{2,1},\dots ,a_{n_1^2}^{2,1}\rangle ,\langle a_1^{2,2},a_2^{2,2},\dots ,a_{n_2^2}^{2,2}\rangle ,\dots ,\langle a_1^{2,N^2},a_2^{2,N^2},\dots ,a_{n_{N^2}^2}^{2,N^2}\rangle ⟧,$$

the tuple of tuples of aggregate $A_2$ is added at to end of the tuple of tuples of the aggregate $A_1$, and the tuple of sets of aggregate $A_2$ is added to the end of the tuple of sets of the aggregate $A_1$:

$$\begin{array}{c}{R}_{\cup }=A_1\cup A_2=⟦M_1^1,M_2^1,\dots ,M_{N^1}^1,M_1^2,M_2^2,\dots ,M_{N^2}^2\mid \langle a_1^{1,1},a_2^{1,1},\dots ,a_{n_1^1}^{1,1}\rangle ,\\ \begin{array}{c}\langle a_1^{1,2},a_2^{1,2},\dots ,a_{n_2^1}^{1,2}\rangle ,\dots ,\langle a_1^{1,N^1},a_2^{1,N^1},\dots ,a_{n_{N^1}^1}^{1,N^1}\rangle ,\langle a_1^{2,1},a_2^{2,1},\dots ,a_{n_1^2}^{2,1}\rangle ,\\ \langle a_1^{2,2},a_2^{2,2},\dots ,a_{n_2^2}^{2,2}\rangle ,\dots ,\langle a_1^{2,N^2},a_2^{2,N^2},\dots ,a_{n_{N^2}^2}^{2,N^2}\rangle ⟧.\end{array}\end{array}$$

(12)

The results of applying the union operation to two aggregates with different compatibility (Figure 1, Figure 2 and Figure 3) are shown in Figure 5.

This mathematical definition can be presented as the algorithm for finding a result of the union of two aggregates as shown in Listing 2.

Listing 2: Algorithm of the union operation for two aggregates.

Input: aggregates $A_1$ and $A_2$ Output: aggregate $R$     if $A_1\doteqdot A_2$ then         for $i=1:N$ do             for $k_i=1:n_i^1$do $r_{k_i}=a_{k_i}^{1,i}$             for $k_i=\left(n_i^1+1\right):\left(n_i^1+n_i^2\right)$do $r_{k_i}=a_{k_i}^{2,i}$         end     else if $A_1\circeq A_2$ then         for $i=1:N^1$ do $\langle r_{k_i}\rangle {}_{k_i=1}^{n_i}=\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1}$         for $i=\left(N^1+1\right):\left(N^1+N^2\right)$ do $\langle r_{k_i}\rangle {}_{k_i=1}^{n_i}=\langle a_{k_i}^{2,i}\rangle {}_{k_i=1}^{n_i^2}$     else         for $i=1:\left(N^1+N^2\right)$ do         if $\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1}$ and $\langle a_{k_i}^{2,i}\rangle {}_{k_i=1}^{n_i^2}$ $\in M_i$ then             for $k_i=1:n_i^1$do $r_{k_i}=a_{k_i}^{1,i}$             for $k_i=\left(n_i^1+1\right):\left(n_i^1+n_i^2\right)$do $r_{k_i}=a_{k_i}^{2,i}$         else             $\langle b_{k_i}\rangle {}_{k_i=1}^{n_i^2}=\langle a_{k_i}^{2,i}\rangle {}_{k_i=1}^{n_i^2}$             end         end         for $$\begin{gathered} i=\left(N^1+N^2+1\right):\left(N^1+N^2+N^b\right) \\ \langle r_{k_i}\rangle {}_{k_i=1}^{n_i^2}=\langle b_{k_i}\rangle {}_{k_i=1}^{n_i^2} \end{gathered}$$         end     end

The intersection of the aggregates $A_1$ and $A_2$ is the aggregate $R_{\cap }$, which contains components that are common to these aggregates and are ordered according to the rule defined in [18]. This rule can be presented as the algorithm for finding a result of the intersection of two aggregates as shown in Listing 3.

Listing 3: Algorithm of the intersection operation for two aggregates.

Input: aggregates $A_1$ and $A_2$ Output: aggregate $R$     if $A_1\doteqdot A_2$ then         for $i=1:N$ do             for $k_i=1:n_i^1$do                 if $$\begin{gathered} a_{k_i}^{1,i}\in A_2^i \\ {r}_{k_i}=a_{k_i}^{1,i} \end{gathered}$$                 end             end             for $k_i=\left(n_i^1+1\right):\left(n_i^1+n_i^2\right)$do                 if $$\begin{gathered} a_{k_i}^{2,i}\in A_1^i \\ {r}_{k_i}=a_{k_i}^{2,i} \end{gathered}$$                 end             end         end     else if $A_1\circeq A_2$ then             $\langle r_{k_i}\rangle {}_{k_i=1}^{n_i^1}=\varnothing$     else     for $i=1:\left(N^1+N^2\right)$ do         if $\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1}$ and $\langle a_{k_i}^{2,i}\rangle {}_{k_i=1}^{n_i^2}$ $\in M_i$ then         for $k_i=1:n_i^1$do             if $\begin{array}{c}{a}_{k_i}^{1,i}\in A_2^i\\ r_{k_i}=a_{k_i}^{1,i}\end{array}$             end         end         for $k_i=\left(n_i^1+1\right):\left(n_i^1+n_i^2\right)$do                 if $$\begin{gathered} a_{k_i}^{2,i}\in A_1^i \\ {r}_{k_i}=a_{k_i}^{2,i} \end{gathered}$$                 end             end         end     end

The results of the intersection operation applied to the compatible and quasi-compatible aggregates are shown in Figure 6 and Figure 7, respectively. The elements of the same color marked with the same border color are equal, for example, $a_1^{1,1}=a_3^{1,1}=a_2^{2,1}$ (blue elements with the red border) in Figure 6.

The exclusive intersection of the aggregates $A_1$ and $A_2$ is the aggregate $R_{\neg }$, which contains only components of the $A_1$ aggregate that are common in these aggregates and are ordered according to the rule defined in [18]. This rule can be presented as the algorithm for finding a result of the exclusive intersection of two aggregates as shown in Listing 4.

Listing 4: Algorithm of the exclusive intersection operation for two aggregates.

Input: aggregates $A_1$ and $A_2$ Output: aggregate $R$     if $A_1\doteqdot A_2$ then         for $i=1:N$ do             for $k_i=1:n_i^1$do                 if $$\begin{gathered} a_{k_i}^{1,i}\in A_2^i \\ {r}_{k_i}=a_{k_i}^{1,i} \end{gathered}$$                 end             end         end     else if $A_1\circeq A_2$ then                 $r_{k_i}{}_{k_i=1}^{n_i^1}=\varnothing$     else     for $i=1:N^1$ do         if $\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1}$ and $\langle a_{k_i}^{2,i}\rangle {}_{k_i=1}^{n_i^2}$ $\in M_i$ then             for $k_i=1:n_i^1$do                 if $$\begin{gathered} a_{k_i}^{1,i}\in A_2^i \\ {r}_{k_i}=a_{k_i}^{1,i} \end{gathered}$$                 end             end         end     end

The results of the exclusive intersection operation applied to the compatible and quasi-compatible aggregates are shown in Figure 8 and Figure 9, respectively.

The difference of the aggregates $A_1$ and $A_2$ is the aggregate $R_{\backslash }$, which contains components of the $A_1$ aggregate that are not present in the $A_2$ aggregate and are ordered according to the rule defined in [18]. This rule can be presented as the algorithm for finding a result of the difference of two aggregates as shown in Listing 5.

Listing 5: Algorithm of the difference operation for two aggregates.

Input: aggregates $A_1$ and $A_2$ Output: aggregate $R$     if $A_1\doteqdot A_2$ then         for $i=1:N$ do             for $k_i=1:n_i^1$do                 if $$\begin{gathered} a_{k_i}^{1,i}\notin A_2^i \\ {r}_{k_i}=a_{k_i}^{1,i} \end{gathered}$$                 end             end         end     else if $A_1\circeq A_2$ then         for $i=1:N$ do             $\langle r_{k_i}\rangle {}_{k_i=1}^{n_i^1}=\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1}$         end     else     for $i=1:N^1$ do         if $\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1}$ and $\langle a_{k_i}^{2,i}\rangle {}_{k_i=1}^{n_i^2}$ $\in M_i$ then             for $k_i=1:n_i^1$do                 if $$\begin{gathered} a_{k_i}^{1,i}\notin A_2^i \\ {r}_{k_i}=a_{k_i}^{1,i} \end{gathered}$$                     end                 end             else                 $\langle r_{k_i}\rangle {}_{k_i=1}^{n_i^1}=\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1}$         end     end

The result of the difference operation applied to the compatible, quasi-compatible, and incompatible aggregates is shown in Figure 10, Figure 11 and Figure 12, respectively.

The symmetric difference of the aggregates $A_1$ and $A_2$ is the aggregate $R_{\Delta }$, which contains components of the aggregate $A_1$ that are not in the aggregate $A_2$ and components of the aggregate $A_2$ that are not in the aggregate $A_1$, and are ordered according to the rule defined in [18]. This rule can be presented as the algorithm for finding a result of the symmetric difference of two aggregates as shown in Listing 6.

Listing 6: Algorithm of the symmetric difference operation for two aggregates.

Input: aggregates $A_1$ and $A_2$ Output: aggregate $R$     if $A_1\doteqdot A_2$ then         for $i=1:N$ do             for $k_i=1:n_i^1$do                 if $$\begin{gathered} a_{k_i}^{1,i}\notin A_2^i \\ {r}_{k_i}=a_{k_i}^{1,i} \end{gathered}$$                 end             end             for $k_i=\left(n_i^1+1\right):\left(n_i^1+n_i^2\right)$do                 if $$\begin{gathered} a_{k_i}^{2,i}\notin A_1^i \\ {r}_{k_i}=a_{k_i}^{2,i} \end{gathered}$$                 end             end         end     else if $A_1\circeq A_2$ then         for $i=1:N^1$ do             $\langle r_{k_i}\rangle {}_{k_i=1}^{n_i^1}=\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1}$         end         for $i=1:\left(N^1+1\right):\left(N^1+N^2\right)$do             $\langle r_{k_i}\rangle {}_{k_i=1}^{n_i^2}=\langle a_{k_i}^{2,i}\rangle {}_{k_i=1}^{n_i^2}$         end      else         for $i=1:\left(N^1+N^2\right)$ do         if $\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1}$ and $\langle a_{k_i}^{2,i}\rangle {}_{k_i=1}^{n_i^2}$ $\in M_i$ then             for $k_i=1:n_i^1$do                 if $$\begin{gathered} a_{k_i}^{1,i}\notin A_2^i \\ {r}_{k_i}=a_{k_i}^{1,i} \end{gathered}$$                 end             end         for $k_i=1:n_i^2$do                 if $$\begin{gathered} a_{k_i}^{2,i}\notin A_1^i \\ {r}_{k_i}=a_{k_i}^{2,i} \end{gathered}$$                 end             end         else                 $$\begin{gathered} \langle r_{k_i}\rangle {}_{k_i=1}^{n_i^1}=\langle a_{k_i}^{1,i}\rangle {}_{k_i=1}^{n_i^1} \\ \langle b_{k_i}\rangle {}_{k_i=1}^{n_i^2}=\langle a_{k_i}^{2,i}\rangle {}_{k_i=1}^{n_i^2} \end{gathered}$$             end         end         for $i=\left(N^1+1\right):\left(N^1+N^2\right)$ do                 $\langle r_{k_i}\rangle {}_{k_i=1}^{n_i^2}=\langle b_{k_i}\rangle {}_{k_i=1}^{n_i^2}$         end     end

The results of the symmetric difference operation applied to the compatible, quasi-compatible, and incompatible aggregates are shown in Figure 13, Figure 14 and Figure 15, respectively.

5.2. Ordering Operations

The ordering operations [19] on aggregates are sets ordering, sorting, singling, extraction, and insertion.

Sets ordering is an ordering operation that reorders the tuple of sets and the corresponding tuple of tuples of the aggregate $A$, according to the given hiddenly compatible aggregate $A_T$, which consists of any arbitrary elements, including dummy ones, as shown in Figure 16. The result of sets ordering is the aggregate $A_{\models }=A\models A_T.$

The sets ordering operation can be fulfilled according to the algorithm in Listing 7.

Listing 7. Algorithm of the sets ordering operation.

Input: aggregates $A$ and $A_T$ Output: aggregate $R$     while not ordered do         $j=1$         for $i=1:N$ do             if $\langle a_{k_i}^i\rangle {}_{k_i=1}^{n_i}\in M_j^T$ then                 $$\begin{gathered} \langle r_{k_i}\rangle {}_{k_i=1}^{n_i}=\langle a_{k_i}^i\rangle {}_{k_i=1}^{n_i} \\ j=j+1 \end{gathered}$$                 if $j>N$ then                     sets are ordered                 end             end         end     end

The sets ordering operation has an important practical value that can be discovered through the following theorem and its corollary.

Theorem 1. 

The theorem on compatibility.

If $\widehat{A_1}=A_1\models A_2$ and $\widehat{A_2}=A_2\models A_1$, then $\widehat{A_1}\doteqdot A_2$ and $\widehat{A_2}\doteqdot A_1$ for $\forall A_1, \forall A_2$, such as $A_1(\doteqdot ) A_2$.

 Proof of Theorem 1. 

Let us consider two arbitrary hiddenly compatible aggregates $A_1$ and $A_2$, i. e., $A_1(\doteqdot ) A_2$, and apply the sets ordering operation to these aggregates in two ways as follows:

• $\widehat{A_1}=A_1\models A_2$;
• $\widehat{A_2}=A_2\models A_1$.

Let us consider the aggregates $\widehat{A_1}$ and $A_2$. As $\widehat{A_1}$ is the result of applying the sets ordering operation, then by Definition 11, the result of its application to the given hiddenly compatible aggregates $A_1$ and $A_2$ is the aggregate $\widehat{A_1}=⟦\left\{A_2\right\}\mid \langle a_i^j\rangle {}_{i=1}^{n_j}⟧{}_{j=1}^N$, where $\langle a_i^j\rangle {}_{i=1}^{n_j}\in \langle A_1\rangle$, $\widehat{\langle A_1\rangle }\equiv \langle A_1\rangle$, which means $\widehat{A_1}=⟦\left\{A_2\right\}\mid \langle A_1\rangle ⟧$.

According to Definition 1, the aggregate $A_2$ can be presented as $A_2=⟦\left\{A_2\right\}\mid \langle A_2\rangle ⟧$.

According to Definition 5, two aggregates are hiddenly compatible if $\left\{A_1\right\}\not\equiv \left\{A_2\right\}, \mid A_1\mid =\mid A_2\mid =N$ and $\forall M_j\subset \left\{A_k\right\}$, $j=\left[1,\dots ,N\right]$, $k=\left[1, 2\right]$.

Then, $\widehat{A_1}=⟦\left\{A_2\right\}\mid \langle A_1\rangle ⟧$, $A_2=⟦\left\{A_2\right\}\mid \langle A_2\rangle ⟧$, $\mid \widehat{A_1}\mid =\mid A_2\mid =N$.

According to Definition 2, the aggregates $\widehat{A_1}$ and $A_2$ are compatible because $\mid \widehat{A_1}\mid =\mid A_2\mid$ and $\left\{\widehat{A_1}\right\}\equiv \left\{A_2\right\}$, which we would have to prove.

The compatibility of the aggregates of $A_1$ and $\widehat{A_2}$ can be proved analogously. □

Remark 1.

The corollary of Theorem 1.

The practical significance of applying the sets ordering operation to hiddenly compatible aggregates $A_1$ and $A_2$ is that the components of one aggregate ($A_1$) can be rearranged according to the sequence of the components of the second aggregate ($A_2$); therefore, as a result of applying the sets ordering operation, the hiddenly compatible aggregates become compatible.

The sorting operation [19] yields a new (sorted) sequence of elements of a certain tuple named the primary tuple. The result of applying the sorting operation to an aggregate $A$ is an aggregate $A_{\uparrow }=A\uparrow {\overline{a}}^k$ (for ascending sorting) or $A_{\downarrow }=A\downarrow {\overline{a}}^k$ (for descending sorting) in all tuples, in which the elements are reordered according to the new ordering of the indices of the elements of the sorted primary tuple ${\overline{a}}^k$. If some tuple is shorter than the primary tuple, it is appended by a dummy element and is then sorted. If some tuple is longer than the primary one, the elements with indices that exceed the largest index in the primary tuple, are not sorted (remain on their positions). An example of the result of sorting operation is given in Figure 17.

The sorting operation can be realized using any appropriate sorting algorithm [20], including the one shown in Listing 8.

Listing 8: Algorithm of the ascending sorting operation.

Input: aggregate $A$ and primary tuple $\langle a_{k_p}^p\rangle {}_{k_p=1}^{n_p}$ Output: aggregate $R$     for $k_p=1:n_p$ do         for $k_m=1:\left(n_p-k\right)$ do             swapped is false             if $a_{k_m}^p>a_{k_m+1}^p$ do                 for $i=1:N$ do                 $$\begin{gathered} r_{k_m}^i=a_{k_m+1}^i \\ {r}_{k_m+1}^i=a_{k_m}^i \end{gathered}$$                 end                 swapped is true             end         end         if swapped is false             break loop         end     end

The extraction operation removes a certain element from the primary tuple of an aggregate. The result of extraction of the element $a_m^1$ from the aggregate $A$ is the aggregate $A_{⋉}=A⋉a_m^1$ in all tuples, from which the elements with index $m$ are removed.

An example of the result of the extraction operation for $m=2$ is given in Figure 18.

The extraction operation can be fulfilled according to the algorithm in Listing 9.

Listing 9: Algorithm of the extraction operation.

Input: aggregate $A$, number of set $k$ and position of element $m$ Output: aggregate $R$     for $i=1:N$ do         for $k_i=1:\left(m-1\right)$ do $r_{k_i}=a_{k_i}^i$         for $k_i=m:\left(n_i-1\right)$ do $r_{k_i}=a_{k_i+1}^i$     end

The insertion operation adds a given element to the known position in the primary tuple of an aggregate. The result of insertion of the element $a_m^1$ to the aggregate $A$ is the aggregate $A_{⋊}=A⋊a_m^1$ in all tuples (except the primary one), to which dummy elements with index $m$ are added.

An example of the result of insertion of a new element $x$ to the second position of the primary tuple is shown in Figure 19, where the indices of the shifted elements remain as they numerated in the original tuple for explanatory purposes.

The insertion operation can be fulfilled according to the algorithm in Listing 10.

Listing 10: Algorithm of the insert operation.

Input: aggregate $A$, number of set $k$ and position of element $m$ Output: aggregate $R$     for $i=1:N$ do         for $k_i=1:\left(m-1\right)$ do             $r_{k_i}=a_{k_i}^i$         end         if $k=i$ then             $r_m=a_m^k$         else             $r_m=\varnothing$         end         for $k_i=m:n_i$ do             $r_{k_i+1}=a_{k_i}^i$         end     end

The singling operation removes the duplicate values that are located next to each other from the primary tuple of the aggregate; elements with the same indices as those of the removed duplicates in the primary tuple are simultaneously removed in all other tuples.

An example of the result of the singling operation is given in Figure 20. Elements of the same color marked with the same border color are equal.

The singling operation can be fulfilled according to the algorithm in Listing 11.

Listing 11: Algorithm of the singling operation.

Input: aggregate $A$ Output: aggregate $R$     m = 2     while not all found do         for $k_1=m:n_1$ do             if $a_{k_1}^1=a_{k_1-1}^1$ then                 $m=k_1$                     break loop                 end                 all found             end             for $i=1:N$ do                 extract $a_m^i$         end     end

6. Multi-Image Notion and Formation Algorithm

The multi-image concept [21] is an essential part of the approach presented in this paper as it enables the required formal description of the sequences of multimodal data about the object under study.

Definition 6. 

A multi-image is a non-empty aggregate specified as:

$$I=⟦T,M_1,\dots ,M_N\mid \langle t_1,\dots ,t_{\tau }\rangle ,\langle a_1^1,\dots ,a_{n_1}^1\rangle ,\dots ,\langle a_1^N,\dots ,a_{n_N}^N\rangle ⟧$$

(13)

where $T$ is a set of time values; $\tau \ge n_j,j\in \left[1,\dots ,N\right]$.

From a practical point of view, multi-image refers to a structure of consolidated temporal multimodal data sets that describe different aspects of the same object. Let us formulate an algorithm that enables the formation of multi-image for an arbitrary object described through several temporal multimodal data sets.

The inputs to the algorithm of the object’s multi-image formation are temporal data sets and the data type (modality) of each set. The algorithm consists of seven steps. The result of the algorithm is multi-image, presented in the form of an ordered collection of temporal multimodal data.

The first step of the algorithm is the formalization of the object’s multi-image data structure. This step is performed according to the requirements for the object’s description. Multi-image can be defined mathematically, according to (13), or in any other way that allows a researcher to unambiguously specify the sequence of data sets and their modality.

At the second step of the algorithm, multi-image specification is decomposed into a set of specifications of partial multi-images as seen in (14).

$$I_j=⟦T,M_j\mid \langle t_1,\dots ,t_{{\tau }_j}\rangle ,\langle a_1,\dots ,a_{n_j}\rangle ⟧$$

(14)

where T is the set of time values; $M_j$ is the set of data values of $j$-modality; ${\tau }_j,n_j \in \mathbb{N}; {\tau }_j\ge n_j$; $j\in \left[1,\dots ,N\right]$; and $N$ is the number of data sets.

At the third step, data are obtained. The procedure for receiving data involves determining the method (protocol, format) of data transmission, determining the time period for data transmission, establishing a connection with the data source and receiving data in a specified way and in a specified format.

At the fourth step, the data of each modality are prepared to form the corresponding partial multi-image. A partial multi-image is an aggregate that includes two tuples of elements, namely a tuple of time values and a tuple of elements of a certain modality. Elements are individual values or ordered sets of values (homogeneous or heterogeneous). Data preparation, which is performed at this step, consists of determining (detecting) time values in the set of data obtained at the previous stage. This procedure can be either trivial when the data format requires explicit presentation of time values for the temporal data component, or complex when the time values are presented in an implicit form or are ambiguous. A method for detecting hidden or ill-defined time values must be developed for each specific data format.

At the fifth step, partial multi-images are combined into a single multi-image. The multi-image merging procedure includes two actions: normalization and uniting.

1.

Normalization of partial multi-images (a normalized multi-image (normalized tuple) is a multi-image (tuple) to the elements of which dummy elements are added; a dummy element is a value that is absent in the tuple before its normalization; an empty element ∅ can be used as a dummy element; at other steps of the multi-image processing, dummy elements are ignored):

$$\begin{array}{c}{\widehat{I}}_j=I_j⋊\left({\langle \varnothing \rangle }_{k=1}^{E_i^j}\succ a_{n_j}^j\right)=⟦T,M_j\mid \langle t_i\rangle {}_{i=1}^{{\tau }_j},\langle \langle a_{i_j}^j\rangle {}_{i_j=1}^{n_j},{\langle \varnothing \rangle }_{l=1}^{E_i^j}\rangle ⟧=\\ =⟦T,M_j\mid \langle t_i\rangle {}_{i=1}^{{\tau }_j},\langle {\widehat{a}}_{i_j}^j\rangle {}_{i_j=1}^{n_j+E_i^j}⟧\end{array}$$

(15)

where $⋊$ is the insertion operation; $E_i^j$ is the number of dummy elements; $E_i^j=\left({\sum }_{j=1}^N{\tau }_j\right)-n_j$; ${\widehat{a}}_{i_j}^j$ is an element of a normalized tuple; and $j\in \left[1,\dots ,N\right]$.

2.

Uniting the normalized partial multi-images:

$$I_U=\underset{j=1}{\overset{N}{{\displaystyle \mathsf{U}}}}{\widehat{I}}_j=⟦T,M_1,\dots M_N\mid \langle \langle t_i\rangle {{}_{i=1}^{{\tau }_j}\rangle }_{j=1}^N,\langle {\widehat{a}}_{i_1}\rangle {}_{i_1=1}^{n_1+E_i^1}, \dots \langle {\widehat{a}}_{i_N}\rangle {}_{i_N=1}^{n_N+E_i^N}⟧$$

(16)

At the sixth step, the multi-image obtained at the previous stage is sorted by the tuple of time values:

$$I_S=I\uparrow \overline{t}=⟦T,M_1,\dots M_N\mid \langle \langle t_{\sigma }\rangle {{}_{\sigma =1}^{{\tau }_j}\rangle }_{j=1}^N,\langle {\widehat{a}}_{{\sigma }_1}\rangle {}_{{\sigma }_1=1}^{n_1+E_i^1}, \dots \langle {\widehat{a}}_{{\sigma }_N}\rangle {}_{{\sigma }_N=1}^{n_N+E_i^N}⟧,$$

(17)

where $\sigma$ and ${\sigma }_j$ are indices specifying the order of the elements in the sorted tuple and $j\in \left[1,\dots ,N\right]$.

At the seventh step, the sorted multi-image obtained at the previous step is singled by a tuple of time values:

$$I=I_S\parallel \overline{t}=⟦T,M_1,\dots M_N\mid \langle t_{\sigma }\rangle {}_{\sigma =1}^{\left({\displaystyle \sum }_{j=1}^N{\tau }_j\right)-\delta },\langle {\widehat{a}}_{{\sigma }_1}\rangle {}_{{\sigma }_1=1}^{n_1+E_i^1-\delta }, \dots ,\langle {\widehat{a}}_{{\sigma }_N}\rangle {}_{{\sigma }_N=1}^{n_N+E_i^N-\delta }⟧,$$

(18)

where $\delta$ is the number of discarded time value duplicates.

The obtained I is the final multi-image that represents the consolidated data describing different aspects of the object under study.

7. Discussion

Let us consider a use case in the area of healthcare that demonstrates the practical use of the proposed approach of multimodal data processing (Figure 21).

The healthcare use case assumes that the data describing the patient’s health status can be obtained from multiple sources, including medical investigation devices (MRI scanners, CT scanners, ECG machines, etc.), manual medical measurement tools and devices (pulse oximeters, thermometers, sphygmomanometers, etc.), test systems (e.g., blood testing), and medical documentation records (treatment events). Data sets collected from these sources are temporal multimodal and semantically interconnected as they describe different features of the same object that is a patient’s organism. The multi-image concept and ASA operations can be employed to consolidate such a compound data collection and implement the logic of its processing.

Let us consider an example which demonstrates the formation of multi-image from several timewise data sequences according to the algorithm proposed in Section 6 and the algorithms of the ASA operations presented in Section 5.

Suppose that the patient’s blood pressure readings (systolic and diastolic), pulse rate, and oxygen saturation level are being measured several times a day. A pair of blood pressure values $\left(a_i^{press\left(s\right)},a_i^{press\left(d\right)}\right)$ are to be received from the sphygmomanometer, and the values of pulse rate $a_i^{pulse}$ and oxygen saturation level $a_i^{satur}$ are to be received from a pulse oximeter. The number of measurements conducted by the sphygmomanometer is $n_1$; the number of measurements conducted by the pulse oximeter is $n_2$.

As a result of patient health parameters monitoring, the following multi-image must be obtained:

$$I=⟦T,M_{press\left(s,d\right)},M_{pulse},M_{satur}\mid \langle t_i\rangle {}_{i=1}^{n_1+n_2},\langle {\left(a_i^{press\left(s\right)},a_i^{press\left(d\right)}\right)}_{i=1}^{n_1}\rangle , \langle a_i^{pulse}\rangle {}_{i=1}^{n_2},\langle a_i^{satur}\rangle {}_{i=1}^{n_2}⟧.$$

According to the algorithm of multi-image formation, the received data are represented as two partial multi-images—a partial multi-image $I_1$ containing values received from the first device (sphygmomanometer) and a partial multi-image $I_2$ containing values received from the second device (pulse oximeter):

$$I_1=⟦T,M_{press\left(s,d\right)} \mid \langle 8:30, 12:05,16:10, 21:00\rangle ,\langle \left(160,78\right),\left(172,81\right),\left(155,76\right),\left(158,75\right)\rangle ⟧,$$

$$I_2=⟦T,M_{pulse},M_{satur} \mid \langle 8:30, 15:08, 21:00\rangle ,\langle 63,75,66\rangle ,\langle 95,98,96\rangle ⟧.$$

To obtain the multi-image, these partial multi-images must be properly merged. For this purpose, at first, we need to normalize these partial multi-images, i.e., apply the insertion operation and add dummy elements to the end of the tuple of blood pressure values and to the beginning of both the tuple of heart rate values and the tuple of oxygen saturation level values:

$${\widehat{I}}_1=⟦T,M_{press\left(s,d\right)} \mid \langle 8:30, 12:05,16:10, 21:00\rangle ,\langle \left(160,78\right),\left(172,81\right),\left(155,76\right),\left(158,75\right),\varnothing ,\varnothing ,\varnothing \rangle ⟧,$$

$${\widehat{I}}_2=⟦T,M_{pulse},M_{satur} \mid \langle 8:30, 15:08, 21:00\rangle ,\langle \varnothing ,\varnothing ,\varnothing ,\varnothing ,63,75,66\rangle ,\langle \varnothing ,\varnothing ,\varnothing ,\varnothing ,95,98,96\rangle ⟧.$$

After normalization, we apply the union operation to the normalized aggregates:

$$\begin{array}{c}\begin{array}{c}{I}_U={\widehat{I}}_1\cup {\widehat{I}}_2=⟦T,M_{press\left(s,d\right)},M_{pulse}, M_{satur}\mid \langle 8:30, 12:05,16:10, 21:00,8:30,\\ 15:08, 21:00\rangle ,\langle \left(160,78\right),\left(172,81\right),\left(155,76\right),\left(158,75\right),\varnothing ,\varnothing ,\varnothing \rangle , \end{array}\\ \langle \varnothing ,\varnothing ,\varnothing ,\varnothing ,63,75,66\rangle ,\langle \varnothing ,\varnothing ,\varnothing ,\varnothing ,95,98,96\rangle ⟧.\end{array}$$

Next, we need to sort the obtained multi-image $I_U$ by the time tuple using the corresponding ASA operation:

$$\begin{array}{c}\begin{array}{c}{I}_S=I_U\uparrow \langle t_i\rangle {}_{i=1}^{n_1+n_2}=⟦T,M_{press\left(s,d\right)},M_{pulse}, M_{satur}\mid \langle 8:30,8:30, 12:05,15:08,16:10, \\ 21:00, 21:00\rangle ,\langle \left(160,78\right),\varnothing ,\left(172,81\right),\varnothing ,\left(155,76\right),\left(158,75\right),\varnothing \rangle , \end{array}\\ \langle \varnothing ,63,\varnothing ,75,\varnothing ,\varnothing ,66\rangle ,\langle \varnothing ,95,\varnothing ,98,\varnothing ,\varnothing ,96\rangle ⟧.\end{array}$$

The tuples are now ordered, but they include duplicate time values. To remove them, we apply the singling operation:

$$\begin{array}{c}I=I_S\parallel \langle t_i\rangle {}_{i=1}^{n_1+n_2}=⟦T,M_{press\left(s,d\right)},M_{pulse}, M_{satur}\mid \langle 8:30, 12:05,15:08,16:10,21:00\rangle , \\ \langle \left(160,78\right),\left(172,81\right),\varnothing ,\left(155,76\right),\left(158,75\right)\rangle ,\langle 63,\varnothing ,75,\varnothing ,66\rangle ,\langle 95,\varnothing ,98,\varnothing ,96\rangle ⟧.\end{array}$$

As a result, we obtain a multi-image $I$ that contains data from two independent but semantically interconnected examinations of the patient. This consolidated data structure can be used for further data processing.

This simple example demonstrates that the ASA offers operations which consider both data features defined in Section 3: multimodality (logical operations allow us to implement any logic for modality-wise data processing) and temporality (ordering operations enable timewise data processing). This principle of the proposed approach is valid for any data type because according to Definition 1, an aggregate can include both separate elements and composite elements (homogeneous or heterogeneous values).

The mathematical approach introduced in the ASA is implemented in the domain-specific programming language ASAMPL [22,23,24].

8. Conclusions

Algorithms for performing operations of the Algebraic System of Aggregates (ASA) are proposed in the paper. A feature of this algebraic system is the consideration of the sequence of elements (tuples, aggregates) when performing operations, including brand-new ordering operations. The properties of the ASA make it possible to use it for the formal description of objects under observation and for the development of methods for processing temporal multimodal data.

An algorithm for creating an object’s multi-image is also proposed in the paper. The input data for this algorithm are separate sets of temporal data. The algorithm consists of seven steps, which include the formation of the multi-image data structure of the object under study, decomposition of the multi-image specification into a set of partial multi-image specifications, obtaining and preparing separate data sets, combining partial multi-images into a single multi-image, sorting the multi-image, and thinning the sorted multi-image by tuple time values. The result of algorithm execution is a multi-image of an object containing synchronized and aggregated sequences of temporal multimodal data characterizing this object.

The proposed mathematical approach can be used for a formal description of digital models in various tasks, including a digital twin design, semantic model construction, and temporal multimodal data aggregation and processing. The mathematical concepts offered by the ASA can also be employed for the consolidation of data for multimodal data processing in multimodal machine learning tasks. Further work should aim to increasing the efficiency of the proposed algorithms.