# Temporal Multimodal Data-Processing Algorithms Based on Algebraic System of Aggregates

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Related Work

## 3. Approach and Requirements for Temporal Multimodal Data Processing

- Multimodality means that the object is determined by a collection of data of a different nature; the logic of presenting and processing data about the object depends on the qualitative and quantitative composition of the data set.
- Temporality means that the elements of data collection are ordered by the time of their receiving; the order of following individual elements of the data sequence affects the result of processing the entire set of data of the object.

- Presenting a data set as a structure of semantically interconnected elements.
- Considering the sequence of data set elements when performing logical operations on them.
- Reordering elements of the data set.

## 4. Basics Notions of ASA

**Definition**

**1.**

- An aggregate is a complex mathematical object, all of whose components are ordered;
- Elements of the tuples can be individual values or tuples of values, and the tuples of values can consist of both values of the same type and values of different types, while each tuple contains elements belonging to one set.

Listing 1. Interface of a data structure for representing an aggregate. |

Aggregate structure:collection ← ordered collection of tuples length ← number of tuples in the aggregate Initialize Aggregatedefine length of the aggregate create collection of size length initialize tuples in collection Insertion (entity) ← add an entity, i.e., a set of values from different tuplesExtraction (entity) ← delete the specific entityTupleInsertion (position) ← add a new tuple to the specific position in the aggregateTupleExtraction (position) ← delete the specific tupleAscendingSorting (primaryTuple) ← sort tuples in the aggregate by the defined primary tuple in asceding orderDecendingSorting (primaryTuple) ← sort tuples in the aggregate by the defined primary tuple in descending orderSingling (primatyTuple) ← remove all non-unique entities by the defined primary tupleSetsOrdering (orderOfTuples) ← reorder tuples by the defined order |

**Definition**

**2.**

**1-1**, represents the element ${a}_{1}^{1,1}$ that belongs to the set ${M}_{1}$ from the definition of ${A}_{1}$ and the green circle, which contains the numbers

**2-1**, represents the element ${a}_{1}^{2,3}$ that belongs to the set ${M}_{3}$ from the definition of ${A}_{2}$.

**Definition**

**3.**

**Definition**

**4.**

**Definition**

**5.**

## 5. Algorithms of Operations on Aggregates

#### 5.1. Logical Operations

- 1.
- If ${A}_{1}\doteqdot {A}_{2}$, then aggregates ${A}_{1}$ and ${A}_{2}$ are defined as$${A}_{1}=\u27e6{M}_{1},{M}_{2},\dots ,{M}_{N}\mid \langle {a}_{1}^{1,1},{a}_{2}^{1,1},\dots ,{a}_{{n}_{1}^{1}}^{1,1}\rangle ,\langle {a}_{1}^{1,2},{a}_{2}^{1,2},\dots ,{a}_{{n}_{2}^{1}}^{1,2}\rangle ,\dots ,\langle {a}_{1}^{1,N},{a}_{2}^{1,N},\dots ,{a}_{{n}_{N}^{1}}^{1,N}\rangle \u27e7,$$$${A}_{2}=\u27e6{M}_{1},{M}_{2},\dots ,{M}_{N}\mid \langle {a}_{1}^{2,1},{a}_{2}^{2,1},\dots ,{a}_{{n}_{1}^{2}}^{2,1}\rangle ,\langle {a}_{1}^{2,2},{a}_{2}^{2,2},\dots ,{a}_{{n}_{2}^{2}}^{2,2}\rangle ,\dots ,\langle {a}_{1}^{2,N},{a}_{2}^{2,N},\dots ,{a}_{{n}_{N}^{2}}^{2,N}\rangle \u27e7,$$$$\begin{array}{c}{R}_{\cup}={A}_{1}\cup {A}_{2}=\u27e6{M}_{1},{M}_{2},\dots ,{M}_{N}\mid \langle {a}_{1}^{1,1},{a}_{2}^{1,1},\dots ,{a}_{{n}_{1}^{1}}^{1,1},{a}_{1}^{2,1},{a}_{2}^{2,1},\dots ,{a}_{{n}_{1}^{2}}^{2,1}\rangle ,\\ \langle {a}_{1}^{1,2},{a}_{2}^{1,2},\dots ,{a}_{{n}_{2}^{1}}^{1,2},{a}_{1}^{2,2},{a}_{2}^{2,2},\dots ,{a}_{{n}_{2}^{2}}^{2,2}\rangle ,\dots ,\langle {a}_{1}^{1,N},{a}_{2}^{1,N},\dots ,{a}_{{n}_{N}^{1}}^{1,N},{a}_{1}^{2,N},{a}_{2}^{2,N},\dots ,{a}_{{n}_{N}^{2}}^{2,N}\rangle \u27e7.\end{array}$$

- 2.
- If ${A}_{1}\doteq {A}_{2}$, then aggregates ${A}_{1}$ and ${A}_{2}$ are defined as$$\begin{array}{c}{A}_{1}=\u27e6{M}_{1},{M}_{2}^{1},\dots ,{M}_{k},\dots ,{M}_{{N}^{1}}^{1}\mid \langle {a}_{1}^{1,1},{a}_{2}^{1,1},\dots ,{a}_{{n}_{1}^{1}}^{1,1}\rangle ,\langle {a}_{1}^{1,2},{a}_{2}^{1,2},\dots ,{a}_{{n}_{2}^{1}}^{1,2}\rangle ,\dots ,\langle {a}_{1}^{1,k},{a}_{2}^{1,k},\dots ,{a}_{{n}_{k}^{1}}^{1,k}\rangle ,\dots ,\\ \langle {a}_{1}^{1,{N}^{1}},{a}_{2}^{1,{N}^{1}},\dots ,{a}_{{n}_{{N}^{1}}^{1}}^{1,{N}^{1}}\rangle \u27e7,\end{array}$$$$\begin{array}{c}{A}_{2}=\u27e6{M}_{1},{M}_{2}^{2},\dots ,{M}_{k},\dots ,{M}_{{N}^{2}}^{2}\mid \langle {a}_{1}^{2,1},{a}_{2}^{2,1},\dots ,{a}_{{n}_{1}^{2}}^{2,1}\rangle ,\langle {a}_{1}^{2,2},{a}_{2}^{2,2},\dots ,{a}_{{n}_{2}^{2}}^{2,2}\rangle ,\dots ,\langle {a}_{1}^{2,k},{a}_{2}^{2,k},\dots ,{a}_{{n}_{k}^{2}}^{2,k}\rangle ,\dots ,\\ \langle {a}_{1}^{2,{N}^{2}},{a}_{2}^{2,{N}^{2}},\dots ,{a}_{{n}_{{N}^{2}}^{2}}^{2,{N}^{2}}\rangle \u27e7,\end{array}$$

- 3.
- If ${A}_{1}\circeq {A}_{2}$, then aggregates ${A}_{1}$ and ${A}_{2}$ are defined as$${A}_{1}=\u27e6{M}_{1}^{1},{M}_{2}^{1},\dots ,{M}_{{N}^{1}}^{1}\mid \langle {a}_{1}^{1,1},{a}_{2}^{1,1},\dots ,{a}_{{n}_{1}^{1}}^{1,1}\rangle ,\langle {a}_{1}^{1,2},{a}_{2}^{1,2},\dots ,{a}_{{n}_{2}^{1}}^{1,2}\rangle ,\dots ,\langle {a}_{1}^{1,{N}^{1}},{a}_{2}^{1,{N}^{1}},\dots ,{a}_{{n}_{{N}^{1}}^{1}}^{1,{N}^{1}}\rangle \u27e7,$$$${A}_{2}=\u27e6{M}_{1}^{2},{M}_{2}^{2},\dots ,{M}_{{N}^{2}}^{2}\mid \langle {a}_{1}^{2,1},{a}_{2}^{2,1},\dots ,{a}_{{n}_{1}^{2}}^{2,1}\rangle ,\langle {a}_{1}^{2,2},{a}_{2}^{2,2},\dots ,{a}_{{n}_{2}^{2}}^{2,2}\rangle ,\dots ,\langle {a}_{1}^{2,{N}^{2}},{a}_{2}^{2,{N}^{2}},\dots ,{a}_{{n}_{{N}^{2}}^{2}}^{2,{N}^{2}}\rangle \u27e7,$$$$\begin{array}{c}{R}_{\cup}={A}_{1}\cup {A}_{2}=\u27e6{M}_{1}^{1},{M}_{2}^{1},\dots ,{M}_{{N}^{1}}^{1},{M}_{1}^{2},{M}_{2}^{2},\dots ,{M}_{{N}^{2}}^{2}\mid \langle {a}_{1}^{1,1},{a}_{2}^{1,1},\dots ,{a}_{{n}_{1}^{1}}^{1,1}\rangle ,\\ \begin{array}{c}\langle {a}_{1}^{1,2},{a}_{2}^{1,2},\dots ,{a}_{{n}_{2}^{1}}^{1,2}\rangle ,\dots ,\langle {a}_{1}^{1,{N}^{1}},{a}_{2}^{1,{N}^{1}},\dots ,{a}_{{n}_{{N}^{1}}^{1}}^{1,{N}^{1}}\rangle ,\langle {a}_{1}^{2,1},{a}_{2}^{2,1},\dots ,{a}_{{n}_{1}^{2}}^{2,1}\rangle ,\\ \langle {a}_{1}^{2,2},{a}_{2}^{2,2},\dots ,{a}_{{n}_{2}^{2}}^{2,2}\rangle ,\dots ,\langle {a}_{1}^{2,{N}^{2}},{a}_{2}^{2,{N}^{2}},\dots ,{a}_{{n}_{{N}^{2}}^{2}}^{2,{N}^{2}}\rangle \u27e7.\end{array}\end{array}$$

Listing 2: Algorithm of the union operation for two aggregates. |

Input: aggregates ${A}_{1}$ and ${A}_{2}$Output: aggregate $R$if ${A}_{1}\doteqdot {A}_{2}$ thenfor $i=1:N$ dofor ${k}_{i}=1:{n}_{i}^{1}$do ${r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}$for ${k}_{i}=\left({n}_{i}^{1}+1\right):\left({n}_{i}^{1}+{n}_{i}^{2}\right)$do ${r}_{{k}_{i}}={a}_{{k}_{i}}^{2,i}$endelse if ${A}_{1}\circeq {A}_{2}$ thenfor $i=1:{N}^{1}$ do $\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}}=\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}$for $i=\left({N}^{1}+1\right):\left({N}^{1}+{N}^{2}\right)$ do $\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}}=\langle {a}_{{k}_{i}}^{2,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$elsefor $i=1:\left({N}^{1}+{N}^{2}\right)$ do if $\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}$ and $\langle {a}_{{k}_{i}}^{2,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$ $\in {M}_{i}$ thenfor ${k}_{i}=1:{n}_{i}^{1}$do ${r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}$for ${k}_{i}=\left({n}_{i}^{1}+1\right):\left({n}_{i}^{1}+{n}_{i}^{2}\right)$do ${r}_{{k}_{i}}={a}_{{k}_{i}}^{2,i}$else$\langle {b}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}=\langle {a}_{{k}_{i}}^{2,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$ endendfor $i=\left({N}^{1}+{N}^{2}+1\right):\left({N}^{1}+{N}^{2}+{N}^{b}\right)\phantom{\rule{0ex}{0ex}}\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}=\langle {b}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$endend |

Listing 3: Algorithm of the intersection operation for two aggregates. |

Input: aggregates ${A}_{1}$ and ${A}_{2}$Output: aggregate $R$if ${A}_{1}\doteqdot {A}_{2}$ thenfor $i=1:N$ dofor ${k}_{i}=1:{n}_{i}^{1}$doif ${a}_{{k}_{i}}^{1,i}\in {A}_{2}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}$endendfor ${k}_{i}=\left({n}_{i}^{1}+1\right):\left({n}_{i}^{1}+{n}_{i}^{2}\right)$doif ${a}_{{k}_{i}}^{2,i}\in {A}_{1}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{2,i}$endendendelse if ${A}_{1}\circeq {A}_{2}$ then$\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}=\varnothing $ elsefor $i=1:\left({N}^{1}+{N}^{2}\right)$ do if $\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}$ and $\langle {a}_{{k}_{i}}^{2,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$ $\in {M}_{i}$ thenfor ${k}_{i}=1:{n}_{i}^{1}$doif $\begin{array}{c}{a}_{{k}_{i}}^{1,i}\in {A}_{2}^{i}\\ {r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}\end{array}$endendfor ${k}_{i}=\left({n}_{i}^{1}+1\right):\left({n}_{i}^{1}+{n}_{i}^{2}\right)$doif ${a}_{{k}_{i}}^{2,i}\in {A}_{1}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{2,i}$endendendend |

Listing 4: Algorithm of the exclusive intersection operation for two aggregates. |

Input: aggregates ${A}_{1}$ and ${A}_{2}$Output: aggregate $R$if ${A}_{1}\doteqdot {A}_{2}$ thenfor $i=1:N$ dofor ${k}_{i}=1:{n}_{i}^{1}$doif ${a}_{{k}_{i}}^{1,i}\in {A}_{2}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}$endendendelse if ${A}_{1}\circeq {A}_{2}$ then${r}_{{k}_{i}}{}_{{k}_{i}=1}^{{n}_{i}^{1}}=\varnothing $ elsefor $i=1:{N}^{1}$ do if $\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}$ and $\langle {a}_{{k}_{i}}^{2,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$ $\in {M}_{i}$ thenfor ${k}_{i}=1:{n}_{i}^{1}$doif ${a}_{{k}_{i}}^{1,i}\in {A}_{2}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}$endendendend |

Listing 5: Algorithm of the difference operation for two aggregates. |

Input: aggregates ${A}_{1}$ and ${A}_{2}$Output: aggregate $R$if ${A}_{1}\doteqdot {A}_{2}$ thenfor $i=1:N$ dofor ${k}_{i}=1:{n}_{i}^{1}$doif ${a}_{{k}_{i}}^{1,i}\notin {A}_{2}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}$endendendelse if ${A}_{1}\circeq {A}_{2}$ thenfor $i=1:N$ do$\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}=\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}$ endelsefor $i=1:{N}^{1}$ do if $\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}$ and $\langle {a}_{{k}_{i}}^{2,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$ $\in {M}_{i}$ thenfor ${k}_{i}=1:{n}_{i}^{1}$doif ${a}_{{k}_{i}}^{1,i}\notin {A}_{2}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}$endendelse$\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}=\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}$ endend |

Listing 6: Algorithm of the symmetric difference operation for two aggregates. |

Input: aggregates ${A}_{1}$ and ${A}_{2}$Output: aggregate $R$if ${A}_{1}\doteqdot {A}_{2}$ thenfor $i=1:N$ dofor ${k}_{i}=1:{n}_{i}^{1}$doif ${a}_{{k}_{i}}^{1,i}\notin {A}_{2}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}$endendfor ${k}_{i}=\left({n}_{i}^{1}+1\right):\left({n}_{i}^{1}+{n}_{i}^{2}\right)$doif ${a}_{{k}_{i}}^{2,i}\notin {A}_{1}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{2,i}$endendendelse if ${A}_{1}\circeq {A}_{2}$ thenfor $i=1:{N}^{1}$ do$\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}=\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}$ endfor $i=1:\left({N}^{1}+1\right):\left({N}^{1}+{N}^{2}\right)$do$\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}=\langle {a}_{{k}_{i}}^{2,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$ endelsefor $i=1:\left({N}^{1}+{N}^{2}\right)$ do if $\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}$ and $\langle {a}_{{k}_{i}}^{2,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$ $\in {M}_{i}$ thenfor ${k}_{i}=1:{n}_{i}^{1}$doif ${a}_{{k}_{i}}^{1,i}\notin {A}_{2}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{1,i}$endendfor ${k}_{i}=1:{n}_{i}^{2}$doif ${a}_{{k}_{i}}^{2,i}\notin {A}_{1}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{i}}={a}_{{k}_{i}}^{2,i}$endendelse$\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}=\langle {a}_{{k}_{i}}^{1,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{1}}\phantom{\rule{0ex}{0ex}}\langle {b}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}=\langle {a}_{{k}_{i}}^{2,i}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$ endendfor $i=\left({N}^{1}+1\right):\left({N}^{1}+{N}^{2}\right)$ do$\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}=\langle {b}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}^{2}}$ endend |

#### 5.2. Ordering Operations

Listing 7. Algorithm of the sets ordering operation. |

Input: aggregates $A$ and ${A}_{T}$Output: aggregate $R$while not ordered do$j=1$ for $i=1:N$ doif $\langle {a}_{{k}_{i}}^{i}\rangle {}_{{k}_{i}=1}^{{n}_{i}}\in {M}_{j}^{T}$ then$\langle {r}_{{k}_{i}}\rangle {}_{{k}_{i}=1}^{{n}_{i}}=\langle {a}_{{k}_{i}}^{i}\rangle {}_{{k}_{i}=1}^{{n}_{i}}\phantom{\rule{0ex}{0ex}}j=j+1$ if $j>N$ thensets are orderedendendendend |

**Theorem**

**1.**

**Proof of Theorem**

**1.**

- $\widehat{{A}_{1}}={A}_{1}\models {A}_{2}$;
- $\widehat{{A}_{2}}={A}_{2}\models {A}_{1}$.

**Remark**

**1.**

Listing 8: Algorithm of the ascending sorting operation. |

Input: aggregate $A$ and primary tuple $\langle {a}_{{k}_{p}}^{p}\rangle {}_{{k}_{p}=1}^{{n}_{p}}$Output: aggregate $R$for ${k}_{p}=1:{n}_{p}$ dofor ${k}_{m}=1:\left({n}_{p}-k\right)$ doswapped is falseif ${a}_{{k}_{m}}^{p}>{a}_{{k}_{m}+1}^{p}$ dofor $i=1:N$ do${r}_{{k}_{m}}^{i}={a}_{{k}_{m}+1}^{i}\phantom{\rule{0ex}{0ex}}{r}_{{k}_{m}+1}^{i}={a}_{{k}_{m}}^{i}$ endswapped is trueendendif swapped is falsebreak loopendend |

Listing 9: Algorithm of the extraction operation. |

Input: aggregate $A$, number of set $k$ and position of element $m$Output: aggregate $R$for $i=1:N$ dofor ${k}_{i}=1:\left(m-1\right)$ do ${r}_{{k}_{i}}={a}_{{k}_{i}}^{i}$for ${k}_{i}=m:\left({n}_{i}-1\right)$ do ${r}_{{k}_{i}}={a}_{{k}_{i}+1}^{i}$end |

Listing 10: Algorithm of the insert operation. |

Input: aggregate $A$, number of set $k$ and position of element $m$Output: aggregate $R$for $i=1:N$ dofor ${k}_{i}=1:\left(m-1\right)$ do${r}_{{k}_{i}}={a}_{{k}_{i}}^{i}$ endif $k=i$ then${r}_{m}={a}_{m}^{k}$ else${r}_{m}=\varnothing $ endfor ${k}_{i}=m:{n}_{i}$ do${r}_{{k}_{i}+1}={a}_{{k}_{i}}^{i}$ endend |

Listing 11: Algorithm of the singling operation. |

Input: aggregate $A$Output: aggregate $R$m = 2 while not all found dofor ${k}_{1}=m:{n}_{1}$ doif ${a}_{{k}_{1}}^{1}={a}_{{k}_{1}-1}^{1}$ then$m={k}_{1}$ break loopendall foundendfor $i=1:N$ doextract ${a}_{m}^{i}$endend |

## 6. Multi-Image Notion and Formation Algorithm

**Definition**

**6.**

- 1.
- Normalization of partial multi-images (a normalized multi-image (normalized tuple) is a multi-image (tuple) to the elements of which dummy elements are added; a dummy element is a value that is absent in the tuple before its normalization; an empty element ∅ can be used as a dummy element; at other steps of the multi-image processing, dummy elements are ignored):$$\begin{array}{c}{\widehat{I}}_{j}={I}_{j}\u22ca\left({\langle \varnothing \rangle}_{k=1}^{{E}_{i}^{j}}\succ {a}_{{n}_{j}}^{j}\right)=\u27e6T,{M}_{j}\mid \langle {t}_{i}\rangle {}_{i=1}^{{\tau}_{j}},\langle \langle {a}_{{i}_{j}}^{j}\rangle {}_{{i}_{j}=1}^{{n}_{j}},{\langle \varnothing \rangle}_{l=1}^{{E}_{i}^{j}}\rangle \u27e7=\\ =\u27e6T,{M}_{j}\mid \langle {t}_{i}\rangle {}_{i=1}^{{\tau}_{j}},\langle {\widehat{a}}_{{i}_{j}}^{j}\rangle {}_{{i}_{j}=1}^{{n}_{j}+{E}_{i}^{j}}\u27e7\end{array}$$

- 2.
- Uniting the normalized partial multi-images:$${I}_{U}=\underset{j=1}{\overset{N}{{\displaystyle \mathsf{U}}}}{\widehat{I}}_{j}=\u27e6T,{M}_{1},\dots {M}_{N}\mid \langle \langle {t}_{i}\rangle {{}_{i=1}^{{\tau}_{j}}\rangle}_{j=1}^{N},\langle {\widehat{a}}_{{i}_{1}}\rangle {}_{{i}_{1}=1}^{{n}_{1}+{E}_{i}^{1}},\dots \langle {\widehat{a}}_{{i}_{N}}\rangle {}_{{i}_{N}=1}^{{n}_{N}+{E}_{i}^{N}}\u27e7$$

## 7. Discussion

## 8. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Morency, L.-P.; Liang, P.P.; Zadeh, A. Tutorial on Multimodal Machine Learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts, Seattle, WA, USA, 10–15 July 2022; Association for Computational Linguistics: Stroudsburg, PA, USA, 2022; pp. 33–38. [Google Scholar]
- Baltrušaitis, T.; Ahuja, C.; Morency, L.-P. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Trans. Pattern Anal. Mach. Intell.
**2019**, 41, 423–443. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Jesus, P.; Baquero, C.; Almeida, P.S. A Survey of Distributed Data Aggregation Algorithms. IEEE Commun. Surv. Tutor.
**2015**, 17, 381–404. [Google Scholar] [CrossRef] [Green Version] - Ribeiro, R.A.; Falcão, A.; Mora, A.; Fonseca, J.M. FIF: A fuzzy information fusion algorithm based on multi-criteria decision making. Knowl.-Based Syst.
**2014**, 58, 23–32. [Google Scholar] [CrossRef] - Oliveira, D.; Martins, L.; Mora, A.; Damásio, C.; Caetano, M.; Fonseca, J.; Ribeiro, R.A. Data fusion approach for eucalyptus trees identification. Int. J. Remote Sens.
**2021**, 42, 4087–4109. [Google Scholar] [CrossRef] - Lahat, D.; Adali, T.; Jutten, C. Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects. Proc. IEEE
**2015**, 103, 1449–1477. [Google Scholar] [CrossRef] [Green Version] - Marinoni, A.; Chlaily, S.; Jutten, C. Addressing Reliability of Multimodal Remote Sensing to Enhance Multisensor Data Fusion and Transfer Learning. In Proceedings of the International Geoscience and Remote Sensing Symposium (IGARSS 2020), Waikoloa, HI, USA, 26 September 2020; pp. 3896–3899. [Google Scholar] [CrossRef]
- Gaonkar, A.; Chukkapalli, Y.; Raman, P.J.; Srikanth, S.; Gurugopinath, S. A Comprehensive Survey on Multimodal Data Representation and Information Fusion Algorithms. In Proceedings of the 2021 International Conference on Intelligent Technologies (CONIT), Hubli, India, 25–27 June 2021; pp. 1–8. [Google Scholar] [CrossRef]
- Oliveira, J.P.; Lourenço, M.; Oliveira, L.; Mora, A.; Oliveira, H. A Data Fusion of IoT Sensor Networks for Decision Support in Forest Fire Suppression. In Internet of Things. Technology and Applications. IFIPIoT 2021; Camarinha-Matos, L.M., Heijenk, G., Katkoori, S., Strous, L., Eds.; IFIP Advances in Information and Communication Technology; Springer: Cham, Switzerland, 2022; Volume 641. [Google Scholar] [CrossRef]
- Liang, P.P.; Zadeh, A.; Morency, L.-P. Foundations and Recent Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions. arXiv
**2023**, arXiv:2209.03430. [Google Scholar] [CrossRef] - Guo, W.; Wang, J.; Wang, S. Deep Multimodal Representation Learning: A Survey. IEEE Access
**2019**, 7, 63373–63394. [Google Scholar] [CrossRef] - Kline, A.; Wang, H.; Li, Y.; Dannis, S.; Hutch, M.; Xu, Z.; Wang, F.; Cheng, F.; Luo, Y. Multimodal machine learning in precision health: A scoping review. Npj Digit. Med.
**2022**, 5, 171. [Google Scholar] [CrossRef] [PubMed] - Wei, W. Time Series Analysis; Pearson Addison Wesley: San Francisco, NY, USA, 2006; 614p. [Google Scholar]
- Hannan, E.J. Multiple Time Series; John Wiley and Sons: Hoboken, NJ, USA, 2009; 535p. [Google Scholar]
- Fraenkel, A.A.; Bar-Hillel, Y.; Levy, A. Foundations of Set Theory; Elsevier: Hoboken, NJ, USA, 1973; 415p. [Google Scholar]
- Petrovsky, A.B. Structuring techniques in multiset spaces. In Multiple Criteria Decision Making; Springer: Berlin/Heidelberg, Germany, 1997; pp. 174–184. [Google Scholar]
- Petrovsky, A.B. Multiattribute sorting of qualitative objects in multiset spaces. In Multiple Criteria Decision Making in New Millennium; Springer: Berlin/Heidelberg, Germany, 2001; pp. 124–131. [Google Scholar]
- Dychka, I.A.; Sulema, Y.S. Logical Operations in Algebraic System of Aggregates for Multimodal Data Representation and Processing. Sci. J. KPI Sci. News
**2018**, 6, 44–52. [Google Scholar] [CrossRef] [Green Version] - Dychka, I.A.; Sulema, Y.S. Ordering Operations in Algebraic System of Aggregates for Multi-Image Data Processing. Sci. J. KPI Sci. News
**2019**, 1, 15–23. [Google Scholar] [CrossRef] [Green Version] - Knuth, D. The Art of Computer Programming, Volume 3: Sorting and Searching, 2nd ed.; Addison-Wesley: Boston, MA, USA, 1998; ISBN 0-201-89685-0. Available online: https://dl.acm.org/doi/10.5555/280635 (accessed on 15 March 2023).
- Pester, A.; Sulema, Y. Multimodal Data Representation Based on Multi-Image Concept for Immersive Environments and Online Labs Development; Advances in Intelligent Systems and Computing; Springer: Cham, Switzerland, 2021. [Google Scholar] [CrossRef]
- Sulema, Y. ASAMPL: Programming Language for Mulsemedia Data Processing Based on Algebraic System of Aggregates. In Interactive Mobile Communication Technologies and Learning. IMCL 2017; Advances in Intelligent Systems and Computing; Auer, M., Tsiatsos, T., Eds.; Springer: Cham, Switzerland, 2018; Volume 725. [Google Scholar] [CrossRef]
- Peschanskyi, D.; Budonnyi, P.; Sulema, Y.; Andres, F.; Pester, A. Temporal Data Processing with ASAMPL Programming Language in Mulsemedia Applications. In Artificial Intelligence and Online Engineering. REV 2022; Lecture Notes in Networks and Systems; Auer, M.E., El-Seoud, S.A., Karam, O.H., Eds.; Springer: Cham, Switzerland, 2023; Volume 524. [Google Scholar] [CrossRef]
- ASAMPL Compiler and Library. Available online: https://github.com/orgs/Asampl-development-team/repositories (accessed on 15 March 2023).

**Figure 9.**An example of applying the exclusive intersection operation to two quasi-compatible aggregates.

**Figure 14.**An example of applying the symmetric difference operation to two quasi-compatible aggregates.

**Figure 15.**An example of applying the symmetric difference operation to two incompatible aggregates.

Features | TTS | TS | TMS | ASA |
---|---|---|---|---|

Ordering | yes | no | no | yes |

Logical operations | no | yes | yes | yes |

Fuzziness | partly | partly | partly | yes |

Data aggregation | no | no | partly | yes |

Temporality | yes | no | no | yes |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Pester, A.; Sulema, Y.; Dychka, I.; Sulema, O.
Temporal Multimodal Data-Processing Algorithms Based on Algebraic System of Aggregates. *Algorithms* **2023**, *16*, 186.
https://doi.org/10.3390/a16040186

**AMA Style**

Pester A, Sulema Y, Dychka I, Sulema O.
Temporal Multimodal Data-Processing Algorithms Based on Algebraic System of Aggregates. *Algorithms*. 2023; 16(4):186.
https://doi.org/10.3390/a16040186

**Chicago/Turabian Style**

Pester, Andreas, Yevgeniya Sulema, Ivan Dychka, and Olga Sulema.
2023. "Temporal Multimodal Data-Processing Algorithms Based on Algebraic System of Aggregates" *Algorithms* 16, no. 4: 186.
https://doi.org/10.3390/a16040186