Multi-Flow Complex Event Optimization in the Edge: A Smart Street Scenario

Abstract

Internet of Things (IoT) devices can be used to provide safety, security, and other services that ensure that smart systems work as intended. However, the increasing complexity of the tasks is increasing the potential of performance loss when limited resources are not utilized appropriately. Distributed complex event processing (CEP) applications can be used to execute multiple unique tasks on sensor data. Since these operations can require a variety of data from multiple sensors across separate task steps, non-optimal code and data management can lead to increased processing delays. In this study, a mathematical model for optimizing critical path performance across multiple independent CEP flows is proposed. The model optimally assigns both where codes are executed at, as well as where their respective data should be placed at. The proposed solution is implemented within an open source library with the inclusion of operator placement heuristics from the literature. Approaches are tested within a realistic smart-street scenario. Consumer delays, algorithm runtimes, and delivery ratios within different time windows are reported. The results indicate that the proposed approach can reduce the delivery times for the critical CEP paths better than the heuristic solutions, with the downside of increased optimization runtimes.

1. Introduction

Operations in edge sensor devices tend to have a varying number of purposes. Whereas some sensors only provide raw data readings such as temperature values, other sensors can provide higher-volume data such as image frames. As the capabilities of edge sensors devices increase, with the introduction of more capable IoT hardware, these devices can further execute more complex operations in addition to simple data collection, such as preprocessing some of the collected information to propagate a summarized version to the next data consumer, or broadcasting alarms to nearby devices directly to warn them about possible problems [1]. The complexity further increases when the devices are required to constantly communicate with each other in a distributed environment to share their collected data to obtain more meaningful event output. One of the methodologies that frequently requires such communication is called complex event processing, which in the context of this study is used to identify a set of concurrent and connected tasks that apply different operations on data which are placed across distributed IoT devices. In the literature, complex event processing in the Internet of Things (IoT) is utilized mainly to process a set of commands that are identified via different query languages supported by a variety of frameworks [2,3]. Unlike previous studies, we focus on distributing code scripts using the Python 3.10 programming language [4] to take advantage of being able to execute advanced complex tasks, with the downside of having less ability to decompose operations that would otherwise be easier with query-only operations. Furthermore, this study uses the document database MongoDB [5], which enables the execution of queries that support aggregation and temporal operations. Therefore, a flexible querying language and advanced Python operations are combined together to enable increased functionality within complex event flows. We identify a complex event flow as a series of operations that originate from raw data-producing sensors to final event consumers with intermediate operations that are connected to each other in a directed acyclic graph format, where each operation can be responsible for a variety of tasks such as simple image detection to determining whether a temperature is high.

Whereas the optimal placement of the complex event processing components are important, the negative effect of non-optimal assignments further increase when the number and length of the complex event paths increase. With the introduction of different tasks, such as object detection and simple audio processing, tasks can become more demanding in terms of resources. Furthermore, due to increased hardware capabilities, it is possible for multiple independent flows to run across the IoT devices in parallel, possibly affecting each other’s performance. Assuming that the number of devices is small, by collecting execution statistics and device information in a central device, code and data placements for different flows can be optimized together to reduce performance degradations. Specifically, within this study, the focus is on optimizing the critical path delay, which takes the longest time from the sensor data to the final consumer. Lastly, the environment can have constraints, such as public channel regulations for duty cycle usage or limited bandwidth, which should be taken into account during optimization processes. This study aims to address several of these issues together by developing a constraint programming optimization model which can be used to optimize independent flows together in a centralized optimization system, which also considers bandwidth limitations to better balance the loads between small-scale distributed IoT devices. Additionally, optimization places multiple operators on multiple devices at the same time, considering the effects of executing more than one code script on a device at the same time. Such concurrency scenarios are not widely studied in the literature and could be important in some real-world cases [6].The main problems and research areas that are addressed within this study can be summarized as follows:

• In the real-world, multiple independent CEP flows can co-exist together; focusing on optimizing one of them can harm the performance of other, therefore optimizing them together can prevent the performance degradation of specific paths.
• Different optimization goals exists in the literature, and this study aims to improve the performance of the worse performing path, so that at any time, the worse case delay is reduced as much as possible.
• Enforcing data to be stored at the task location can harm different CEP paths that utilize the same data, and can cause specific devices to be overloaded in terms of bandwidth utilization rate. Therefore, a flexible and shared data placement mechanism is required to adapt dynamic load changes during CEP operations.

A library is written and the proposed solution and other heuristics from the literature are tested on Raspberry Pi [7] devices. Metrics such as minimum and maximum path delays, data-loss ratios, average worker loads, and different time-window delivery rates are reported. It is observed that the library developed and the proposed constrained programming solution can provide satisfactory performance for small-scale smart IoT complex scenarios.

The contributions of this study are as follows:

• Introducing a multi-consumer, multi-flow-oriented complex event flow optimization that can optimize the critical path across independent complex event flows to reduce delays while leveraging the underlying shared database infrastructure. A two-step optimization model is proposed, which minimizes the expected maximum bandwidth usage and further minimizes the delay with consideration of the CEP path relationship.
• The developed approach is experimented on a bandwidth-constrained smart street scenario that aims to distribute multiple flow tasks across distributed small-scale IoT devices. Furthermore, different cost considerations for existing heuristics are experimented on to observe their performances as well. Delay, data loss ratios, expected device loads, and delivery ratios are reported for different quality time windows.
• Providing the proposed solutions as well as the heuristic implementations within an open source library [8] that automates the distribution and execution of CEP codes and related data across small-scale IoT environments.

The remainder of the paper is structured as follows: the second section provides a related literature review, the third section provides the problem foundation and mathematical model, the fourth section provides the CEP infrastructure, the studied scenario, and the heuristics from the literature, the fifth section provides the experiment results, and the paper is concluded with the Conclusion section.

2. Literature Review and Background

Complex event processing in IoT encapsulates a wide range of topics in smart environments with a variety of solutions that address different issues. This study focuses on the execution of complex events within resource-constrained small-scale IoT environments. To keep the scope limited to the focus of this study, the literature review is divided into smart environments, data management, and other related studies.

2.1. CEP in Smart Environments and Edge Computing

Complex event processing can enable the automation of a variety of edge tasks, such as the management of street lights, security cameras, smart streets, smart hospitals, and other applications that can be found everywhere in daily life. Whereas this automation with the support of complex processing enables execution of more functionalities, the solutions that come with it bring together different trade-off relationships. Solutions that focus on coordinating cloud and edge resources together can enable better computing power in the cloud for more analysis. On the other hand, one faces the increasing cost of cloud resources, as well as the delay between cloud and edge devices. When it comes to environments with resource constraints, the use of edge-only solutions can ease the process of managing migrations and operator placements, which reduces costs, but does not perform as well as cloud services [9,10,11]. The number of problems addressed in an edge-only environment also varies greatly, such as managing GPS data of vehicles that enable vehicle-to-vehicle communication, or enabling road-side units (RSUs) to coordinate with each other and other nearby devices to enable the execution of simple tasks. In such environments, requirements are taken into account in relation to different quality attributes, such as delivery times, processing times, and data loss ratios. Functional requirements such as MQTT levels can also affect the focus of the proposed solutions [12]. Further quality requirements and resource constraints can affect the volume of data stored, as well as the number of executions devices can have, and distributed IoT environments try to address these issues [13]. In terms of complex environments, this study focuses on a small smart street case where distributed RSUs are connected to each other in a mesh-like network and can access the shared data of the others. A central management device handles MQTT communication in which data are fetched from databases directly by the required device.

Table 1. Comparison between previous related CEP studies and the current study. × implies no coverage. ✓ implies that the topic is covered. Δ implies that the study has indirect implications in terms of coverage.

Study	Year	Shared Memory	Critical-Path Optimization	Edge-Only Support	Library Provided	Custom Code Execution
[14]	2018	Δ	✓	✓	✓	✓
[15]	2018	×	✓	✓	Δ	×
[16]	2019	Δ	Δ	Δ	✓	Δ
[17]	2020	×	✓	✓	Δ	×
[18]	2021	×	✓	✓	Δ	×
[19]	2021	×	✓	Δ	✓	Δ
[20]	2022	×	✓	Δ	Δ	×
[12]	2023	×	Δ	Δ	Δ	×
This Study	-	✓	✓	✓	✓	✓

shows how this study as well as the related CEP studies in the literature cover different topics researched in this paper. The core differences this study has are the shared data management and resulting better load balancing, and enabling optimization and execution of custom code script execution across multiple independent paths in complex scenarios. Another point is that some studies only have implications and direct support to some of the topics. Lastly, it should be mentioned that not all studies have to cover all topics, as some might focus on improving specific points, whereas our study aims to cover a problem where the listed challenges are addressed all at once while introducing limitations, which are discussed later on in the paper.

2.2. Data Management in IoT

When it comes to handling data, IoT environments have a variety of data sizes with dynamic volumes. For example, the volume of data received by the RSU can change between the times when traffic is convoluted and the times when fewer cars pass by. Data such as laser readings and camera readings change in terms of bytes required to process, which in a non-optimal management can introduce bandwidth problems. Data produced by the sensors are either processed directly where they are produced or sent to another edge device for future processing. In this study, we focus on a system in which devices can store their data on any other IoT device to which they are connected using a Virtually Shared Memory (VSM) system. Especially in event-driven solutions such as the utilization of MQTT, VSM can allow devices to handle data between each other without having to overload the MQTT broker with the data payload. Furthermore, VSM can enable the reduction of the storage load on IoT devices where available resources are restricted [21]. VSM has been used for a variety of purposes not limited to the scope of IoT but for enabling parallel and shared processing of deep learning and machine learning problems, as well as detecting malicious attempts on distributed systems [22,23,24]. The downside of using VSM is the latency that comes from handling remote read/write operations, as well as the potential load on the network that should be considered in optimization solutions [24]. This study utilizes VSM for the aforementioned load reduction on brokers where the broker device is also subject to resource limitations, and also utilizes VSM for better network load balancing. The VSM is defined as an in-memory database location where the sensors can store their data. Including the data itself, the task queries or codes that will be executed for complex tasks should also be handled with care. There exist solutions that download required libraries from the cloud [18]. Some solutions choose to replicate data where multiple processing can be applied, further enabling better scaling in distributed systems [25,26]. The use and improvement of existing CEP frameworks is also an option for better data handing while also utilizing other powerful capabilities of the frameworks [27]. Although data flow is from sources to consumers, the side that initiates the communication is also important and is considered in some studies [16]. Within event-driven environments, there can exist some events that can be discarded to improve important throughput and reduce network load, which is a challenge when deciding which events should be discarded [28]. Studies exist which introduce heuristic solutions for operator placement in complex event processing [2]. In this study, it is always assumed that the communication is initiated by the writers of the data output, which is done by MQTT events.

2.3. Other Related Studies

Complex event processing can be found in many study fields such as machine learning and deep learning, where initial data have to be collected from an arbitrary location, and also tasks can be executed in parallel across different devices for faster preprocessing [29,30,31,32]. Another topic is the deployment of CEP across heterogeneous IoT devices, where devices can have different hardware resources and different operating systems due to differences in vendors [33]. This study addresses this issue by enabling the deployment of CEP applications using docker images with Python library support, enabling the deployment of the applications to devices that have enough capability to run the docker infrastructure. Problem environments such as devices with high GPU capabilities [34], high data-volume solutions [35], and video streaming [36] are not within the scope of this study, but are important to consider due to their additions in event stream processing. Larger network solutions can help improve the scalability of CEP applications [37]. Lastly, future challenges in IoT that directly or indirectly affect CEP performance are widely discussed in the literature [38,39]. Lastly, in terms of taking different approaches to CEP optimization, early resource consumption prevention and one-hop placement optimization [14], and alarming event trigger time evaluation [17] are some of the focuses taken in the literature.

3. Problem Formulation

This section focuses on the mathematical foundation of the problem, as well as the optimization goal, which is to reduce the maximum potential latency across multiple CEP paths so that the critical path with the highest latency has better performance throughout the evaluation period. For this purpose, a constrained-based programming approach is utilized, in which constraint definitions are widely used in the literature [2,17]. Whereas most solutions that focus on operator placement suggest a heuristic approach, we feed our model into a constraint programming solver to reach optimal solutions. We consider our approach viable due to the small-scale focus of the problem. This also further requires all definitions to be valid and strongly defined, as invalid definitions will lead to obtaining non-viable placement suggestions. Such an approach also has its limitations, which will be discussed later in the paper.

3.1. Optimization Goal and Critical Path Problem

As mentioned above, the goal is to optimize the performance of the critical path. Each complex flow or multiple flows running on edge devices at the same time can have different resource requirements and different number of event executions. If the placement of tasks and related data locations is not optimal, what may happen is that some flows will have better execution times, whereas others will be delayed and end up increasing the consumer delivery times. The main purpose of this study is to optimize flow placements so that the flow with the maximum delay has better delivery times compared to related heuristics suggested in the literature. The critical path is the path with the highest latency, and not the path with the lowest throughput. This is so because a path can have a lower event rate while also having higher delays. In related studies, most of the paths include only a few steps from the sources with trivial executions, but within the scenario in this study the number of steps can be more than one, while enabling extended functionalities in the system. Flows, when acyclic, can be represented as steps taken from sources to consumers, and a complex system usually includes multiple sources and consumers, where the data can be required by multiple intermediate events. This creates a directed acyclic graph (DAG) relationship, and then the eventual goal becomes to reduce the delay in the worst path of this DAG. Specifically, DAG is programmed by creating a tree structure where each node is the producers/consumers and actions, and each edge is the input and output topic relationships. However, simply focusing on the path cost can also be harmful, as the overall network load on the devices may very well harm the throughput if not balanced correctly. Therefore, a two-step optimization approach is taken, where in the first step the maximum external load on a device is reduced as much as possible without considering the path relationship to obtain the scenario where all data are transferred with least possible interference. In the second step, this maximum load is set as the constraint, and the delay assumption within the paths is minimized, which eventually optimizes the critical path delay while also keeping the load low.

3.2. Mathematical Formulation

After defining the optimization goal, it is also important to define the mathematical formulation as a constraint-based satisfiability solution which requires exact terms, as without them it will be harder to code the formula itself. These definitions help in traversing through multiple independent flows while also considering costs at different steps and eventually optimizing the worse performing path.

Mathematical notations are given in

Table 2. Notations.

Notation	Definition	Notation	Definition
$CF$	Complex flow cluster	$L{I}_i$	External input load (bytes)
$F_i$	Single flow	$L{O}_i$	External output load (bytes)
$P_i$	Flow path	$T_i$	Worker Task load
$S_i$	Path steps	$t_i$	Task unit cost
$R_i$	Raw data source	$l_i$	Placement choice
$E_i$	Event trigger	n	Number of actions
$A_i$	Action code	$O_i$	Oscillation penalty
$C_i$	Static consumer	$d_s$	Step delay (seconds)
${\succ }_c$	Connection	$d_p$	Path delay (seconds)
$CS$	Cluster starting point	$e_i$	Execution delay (seconds)
$CE$	Cluster ending point	$H_i$	Previous placements
$W_i$	Worker node	$lat$	Delay (seconds)
$Cod{e}_i$	Code placement	$BW$	Bandwidth (Bps)
$Data{S}_i$	Source data placement	$\mu$	BW utilization ratio
$Data{A}_i$	Action data placement	$\nu$	BW utilization penalty
$L_i$	Worker External I/O load (bytes)

, while the definitions are given in Figure 1. Furthermore, a brief description of the equations defined within this section is given in

Table 3. Brief description of each defined equation.

Equation	Description
(1)	Each source data is stored at a single location.
(2)	Each action output is stored at a single location.
(3)	Each CEP step action is executed at a one device.
(4)	Sum of external I/O load cannot exceed devices BW capacity.
(5)	Each worker can support a limited number of task executions.
(6)	Maximum BW load across workers are minimized.
(7)	Data I/O is penalized if the source/target changes from prior location.
(8)	Critical path delay across all CEP paths are minimized.

. The entire CEP system $CF$ is built from one or several flows $F_i$. A flow is built from one or more paths $P_i$, where each path consists of one or more steps $S_i$, and the relationship between these steps ${\succ }_c$. A step $S_i$ can be a data source, code execution, or end consumer. A relationship ${\succ }_c$ originates from a data source $R_i$, or a task action $A_i$, and via the event triggers $E_i$, connects to the end consumers $C_i$, or other tasks $A_i$. An alternative way to consider this is that a relationship connects two steps where each relationship is assumed to be one way from source to consumer, without any loops involved. Furthermore, each path $P_i$ is assumed to be virtually connected to a starting point $CS$, and to end at another virtual point $CE$, which allows optimizing independent flows $F_i$ together, converting the entire $CF$ into a single directed acyclic graph. Lastly, the $S_0\nsucc S_i$ is used to imply that no step precedes the initial step. Furthermore, if a step is further down the flow path, it is not succeed by a prior step, which implies that cyclic relationships are not considered.

Each output data related to data sources and code executions is assumed to be written to a single worker, which is given in Equations (1) and (2). This further implies that each event consumer reads the input data (related to a specific topic, as there may be multiple input topics) from a single location.

$$Data{S}_i=l_i\in {\{0,1\}}^{\left|W\right|\times \left|R\right|}$$

$$\sum _i^{n}Data{S}_i=1$$

(1)

$$Data{A}_i=l_i\in {\{0,1\}}^{\left|W\right|\times \left|A\right|}$$

$$\sum _i^{n}Data{A}_i=1$$

(2)

Each code is assumed to be placed on a single worker only, which is given in Equation (3).

$$Cod{e}_i=l_i\in {\{0,1\}}^{\left|W\right|\times \left|A\right|}$$

$$\sum _i^{n}Cod{e}_i=1$$

(3)

Both task and bandwidth load should not exceed the workers’ capability which is given in Equations (4) and (5). If there exists a case where in the optimal space at least one worker has to surpass its capability due to assignments, a great penalty is induced to give the placement to the least-loaded worker. An important fact to consider here is that due to the complexity of the tasks at hand, heuristics may not be able to directly find results that obey the limits; by introducing penalties, the solutions can at least converge to local optimal points where the limits are not surpassed too much, specifically in scenarios with heavily loaded networks.

$$\forall W_i{L}_{in}=\sum _i^{n}L{I}_i\times Dat{a}_i$$

$$\forall W_i{L}_{out}=\sum _i^{n}L{O}_i\times Dat{a}_i$$

$$\forall W_i{L}_{in},L_{out}<BandwithLimit$$

(4)

$$\forall W_i{T}_i=\sum _i^n{t}_i\times Cod{e}_i$$

$$\forall W_i{T}_i<TaskLimit$$

(5)

The first optimization goal, as mentioned above, is to reduce the external I/O load on the devices. This indicates that given the constraints (4) and (5), we want to minimize the maximum possible load on the devices, which gives us Equation (6). This equation helps the optimization to not overload any specific device with too much bandwidth usage. As mentioned in the Introduction section, this is one of problems that this paper aims to solve, by reducing the possibility of functions that might have too much resource requirement to converge to a single device and end up increasing the delays.

$$L_{max}=\forall W_{i}max(L_{in},L_{out})$$

$$F_1=min\left(L_{max}\right)$$

(6)

After the maximum load is obtained for the given complex flow, the maximum load obtained is fixed and the second optimization part starts. For the second optimization part, the focus is on the path costs. Path costs can be defined as the sum of the costs of individual steps and the relationship between them. It is assumed that the data sources (e.g., sensors) and the end consumers (e.g., road lights) are static in terms of application placement. Therefore, these two have either the data writing or reading costs. It is further assumed that if data is placed where it is produced at, it induces a near-zero delay, assuming that the data can be read without the device’s CPU being overloaded. Therefore, the cost comes from the utilization of the bandwidth of the workers.

$$la{t}_{read}={\displaystyle \frac{L{I}_i}{BW}},la{t}_{write}={\displaystyle \frac{L{O}_i}{BW}}$$

However, there are two problems with the above latency assumption. First, since external data transfer occurs between different devices, both the source device load and the target device load should be taken into account. The device with the highest load can be assumed to have longer delivery times for transfer. Considering that data transfer occurs between a $source$ and a $target$ worker, and that the uplink capacity can be identified as $BWU$, while the downlink capacity can be identified as $BWD$, the following equations can be obtained.

$$la{t}_{read/write}=max({\displaystyle \frac{L{O}_{source}}{BW{U}_{source}}},{\displaystyle \frac{L{I}_{target}}{BW{D}_{target}}})$$

In this way, it can be assumed that for any given data transfer between source and target, worst-case delay can be obtained from the device with the most bandwidth usage according to the capacity direction.

The second issue is the capacity utilization rate: while devices under extreme load may very well utilize all the available bandwidth, this can lead to degradation of services with increased delays and quality of service issues. A simple way to enforce optimization to choose the scenario with less bandwidth utilization is to introduce a penalty multiplier ($\nu$) when the utilization rate is above a certain percentage ($\mu$). For example, if the device should always have around 20% bandwidth availability to process other operations not limited to CEP, any placement delay can be penalized to ensure that, if possible, optimization will decide another path for placement.

$$la{t}_i=\left\{\begin{array}{cc}la{t}_i,\hfill & \mathrm{if}BW<\mu \times B{W}_{limit}\hfill \\ la{t}_i\times \nu ,\hfill & otherwise\hfill \end{array}\right.$$

In addition to the above delays, any oscillations, which are identified as having a different placement location than before, or migrations, which are identified as changes in where data are stored, induce additional penalties that are identified in Equation (7).

$$la{t}_{read/write}=\left\{\begin{array}{cc}la{t}_i,\hfill & \mathrm{if}i==W_{i}i==H_i\hfill \\ la{t}_i\times O_i,\hfill & \mathrm{if}i!=W_i\left|\right|i!=H_i\hfill \end{array}\right.$$

(7)

Furthermore, the operations taken after reading the data also introduce a delay ($e_i$). Internal I/O operations also introduce a delay ($d_i$).

$$la{t}_{step}=la{t}_{read}+e_i+d_i+la{t}_{write}$$

For data producers, the read latency is nonexistent, and for consumers, write latency is nonexistent in the flow.

$$la{t}_{producer}=e_i+d_i+la{t}_{write}$$

$$la{t}_{consumer}=la{t}_{read}+e_i+d_i$$

Including the latency of all individual steps in a path, the entire latency of the path becomes

$$la{t}_P=la{t}_{producer}+\sum _{i=1}^{n}la{t}_{step}+la{t}_{consumer}$$

Since each flow is composed of individual paths and therefore the entire CEP system is composed of individual paths, the critical path latency is the maximum latency from the entire paths.

$$critical=\forall i\in P\left(max\left(la{t}_P\right)\right)$$

Therefore, the optimization goal becomes minimizing the critical path latency while being subject to constraints (1), (2), (3), (4), and (5). As one of the problems this study aims to address is to reduce the critical path delay, and increase the performance of the worse performing path, this function serves as the method to help achieve this.

$$F_2=min\left(critical\right)$$

(8)

In the end, first, Equation (6) is optimized to gain a reduced load, and then the second Equation (8) is optimized to increase the performance of the critical path.

3.3. Complexity of the Optimization

The main complexity of the solution comes from iterating through DAG, as the number of paths and the number of unique steps increase the time it takes to find an appropriate solution. Each individual raw source and consumer must consider whether they will be reading/writing data for a specific worker, introducing costs $O(R\times W)$ and $O(C\times W)$. Each action has to consider each worker for inputs, outputs, and execution locations $O(A\times W\times x+A\times W+A\times W\times y)$ where x and y define the number of input and output event triggers for the specific event. The costs up to this point are enough to define the constraints, whereas the second cost comes from traversing the DAG to define the cost of individual paths. Considering that there exists E number of event I/O relationships between steps, the complexity of the first search becomes $O\left(\right(R+A+C)\times E)$. Lastly, Google ortools [40] is used to solve the constraint programming problem, which has its own runtime duration.

4. Methodology

This section details how the proposed solution is implemented within a library that can be used to distribute task codes and related data placement assignments in a distributed IoT environment. This library enables an easy way to define and experiment with the CEP relationships and includes the algorithms that are experimented within the study. Without the library it would be challenging to experiment the CEP scenarios while coordinating the distributed system. Furthermore, the experimented scenario and its properties are also detailed. The scenario is a smart street case where the sensors are distributed across a small street to cover multiple CEP paths that originate from different sensor data. This helps covering one of the problems this study tries to address, which is optimizing independent and complex flows. Lastly, the heuristics and related improvements that are used in other CEP frameworks in the literature are also explained so that the exact algorithmic steps that are compared are provided clearly.

4.1. CEP IoT Infrastructure

The experimental CEP infrastructure can be seen in Figure 2. The developed library supports the management of CEP applications that are deployed in a small-scale distributed IoT environment. The central device, which is also another RPi4B unit, collects statistics from workers, consumers, and producers to periodically optimize task and data placements. The library supports a variety of heuristics as well as the proposed constraint-based optimization explained in the previous section. Worker applications can produce data or consume events to apply different operations, such as sending alarms to nearby devices. Whereas the event notifications are processed using MQTT, the data itself is always fetched from the shared in-memory database infrastructure. This enables reducing load on the broker by only including where the event data should be fetched. Workers utilize HTTP communication, notifying their existence to the server via periodic heartbeats. In each evaluation period, the workers that are supposed to perform specific tasks download their code from the management device.

4.1.1. CEP Code and Data Management Library

Users of the library can register their code using the provided interfaces. Each registration requires input/output topics, the location of the code at the management device, and the MongoDB query code that is supposed to fetch data from the VSM in-memory database location. At runtime, registered tasks are automatically distributed to connected worker devices, where the worker devices utilize the given configurations to obtain execution information, and placements are updated at each evaluation period if deemed necessary by the optimizer.

4.1.2. Distributed Shared Database Management

An intuitive example of how the shared DB system reduces the load on the network can be seen in Figure 3. As mentioned earlier, the proposed approach aims to balance both code and data placement together by utilizing a shared data storage system. In an only-operator placement solution, changing the location of the task can only help reducing the load of one link, and the writing link between 2–5 could be more costly than the reading link between 2–5. However, the overall link count could be further reduced by allowing the task that works on device 2 to write its output to 5. This will not only reduce the number of links that come to a single device but also allow devices to share the overall load. That is, assuming that the link costs are worth moving, which is what optimization aims to resolve.

4.2. Smart Street Scenario

In order to test the algorithms in a more bandwidth-constrained environment, a realistic CEP scenario is designed, which can be seen in Figure 4. This scenario covers IoT communication and complex event tasks that are executed on distributed sensors along a smart street. Within the given scenario, there exist multiple sensors that are distributed across the road that collect data such as temperature readings, humidity readings, noise levels, air pollutant metrics, and image frames. As stated in the first chapters, the sensors connect to a management application that is responsible for hosting the MQTT broker via wireless communication. Furthermore, each sensor application hosts an in-memory MongoDB database that is used to store and host event data.

Such a scenario can be utilized for different purposes, such as determining whether an ambulance is passing by, by checking the camera footage and running them through a local object detection model deployed in the Raspberry Pi units. A further intuitive example is to determine whether the pedestrian density increases consequently within a time-frame, and combine this knowledge with the average speed of vehicles through the speed detectors to determine when this road faces traffic problems. This knowledge can help improve smart city planning and enable offloading of some of the simple tasks which otherwise would have to be executed on a central cloud device.

Although devices can collect the data mentioned above and execute different tasks, devices such as raspberry Pi units, which are the real physical devices used in our experiments, have the capability to execute multiple tasks concurrently. This capability can bring about two advantages. The first advantage is concurrent execution of tasks that can read the same data but run tasks that output different conclusions towards branching CEP paths. An example is that two concurrent tasks can process the audio data and determine whether an audio signal contains voices from pedestrians or vehicles at the same time, which will also help reducing processing times. The second advantage comes from having the capability to serve multiple CEP flows at the same time. By having the capability of executing multiple tasks, multiple CEP flows that can be independent from each other can be deployed at the edge, considering that the resources can handle the expected load. Lastly, as devices become more capable, they support tools such as Docker, which can make it easier to deploy edge applications.

For serving CEP cases such as the aforementioned different examples, two independent flows are designed to observe the optimization performances. The flows are shown in Figure 5 and Figure 6. In the real world, it can be expected from IoT devices to run multiple independent tasks at the same time, and therefore having two parallel flows aims to satisfy this requirement. The first is a traffic control mechanism in which population density, car speed, object detection, and speed data are used together to reach different conclusions about the current state of the streets. The second flow determines air quality and detects possible cases of problematic pollutants and broadcasts them to nearby listeners.

4.3. Optimization

Any optimization performed within the library can be identified with the steps defined in Algorithm 1. Each algorithm is fed with data from a common interface to keep their algorithmic flow in isolated codes. These data are the statistics collected periodically from the workers, the topology of the task graph, and any historical distribution data if fed by the algorithm in the previous evaluation period. Trivial information, such as worker information, is also provided to the algorithms. After an optimization finishes the assignments, the returned solution is validated by checking that all I/O relationships start from a data producer and end up at a consumer, so that no operation is left open useless.

Algorithm 1: Code and data placement using any optimization method

Data: CEP and worker settings. Result: Code and data I/O assignments. Collect execution statistics; Create DAG via pub/sub topics of CEP; Run the requested optimization using any optimization method; Optimization returns the assignments; return $assignment$

4.4. Heuristic Approaches

Different heuristics from the literature are implemented and improved upon. These are taken from other CEP frameworks that target operator placement. It should be noted that most approaches apply single operator placement; however, in this study, multi task/data placement is applied, which requires modification on existing approaches.

4.4.1. Producer-Oriented

A producer-oriented approach starts assigning tasks in a breadth-first approach starting from producers and takes the assignment target as the source that produces the highest volume of data and is also not resource-limited.

4.4.2. Consumer-Oriented

In contrast to the producer approach, distribution begins with consumers, and tasks are assigned to the available devices that require the highest volume of data.

4.4.3. Genetic Algorithm

The genetic algorithm (GA) provides a randomized semi-heuristic way to generate solutions to problems that otherwise are hard to solve due to the increased cost of the solution space. GA has been used in the literature in the past to solve operator placement problems [41,42]. The simplified version of the algorithm is given in Algorithm 2. First, an initial population is generated by randomly distributing the events to feasible locations. Therefore, every solution in the initial population is feasible and a valid solution to the distribution problem. Next, a specific percentage of this population is kept to be included in the next generation as the highest performing individuals. The remaining percentage is first filled randomly, cross-breeding the entire population, and switching some of the placements so that new solutions are generated. This process favors high-performing parents, and each solution is again kept in the feasible solution space. The last percentage of the next generation is generated by mutating some of the newly bred individuals so that they will introduce random searching into the problem and possibly escape from a local optimum. This process of population generation, cross-breeding, and mutation is applied iteratively over a number of generations, and the best solution across all generations is kept as the distribution solution. An important point about the parameters of the genetic algorithm is that if the population size is too large, the management device may have memory and performance issues. If the number of generations is too large, again the manager will take longer to find a feasible event distribution because of the increased DAG path number and complexity. However, to be fair to other approaches and keep the distribution time in a feasible window, the number of populations and generations is kept at a certain level. Furthermore, it has been noticed that the use of a higher population count may not always benefit the outcome of the genetic algorithm, which will be shown in the evaluation section. This is because utilizing a higher population count as mentioned earlier increases the runtime, which may cause more messages to be transmitted within the non-optimal distribution window.

Algorithm 2: Code and data placement using genetic algorithm approach

4.4.4. Relaxation

The spring relaxation algorithm gets its name from the physical forces in the springs which the flow is given in Algorithm 3. This algorithm is designed from the relaxation steps given in [43]. Since the environment in this study focuses on low-latency mesh connections, instead of latency between devices, the solution here focuses on the bandwidth capacity of the devices. Relaxation has also been used to solve complex event processing problems in the past [19,44]. For example, each connection between the IoT devices in a distributed network can be seen as the springs that connect two objects to each other. Each of these springs represents the network traffic force. Whenever the distance between two devices is large or the data load is high, the spring has higher force. The aim of the relaxation algorithm is to iteratively optimize the placements of the operators so that this force will be reduced. At some point in time, there will be no potential candidates to relax the forces any more, and at that point it is assumed that the system force is stable and an optimal solution is reached. In order to fit this approach to the problem in this study, iteratively, a potential neighbor placement from the consumers and producers is chosen for an event, and it is checked whether placing the event at any of these locations will reduce the overall bandwidth usage of the system. Unfortunately, taking smaller steps and iterations imply that we have to place an upper bound on the number of iterations that can be taken by the algorithm, because if it is unbounded, the algorithm may end up running for several minutes or more, which is not optimal since the collected statistics from the devices can become redundant within that duration.

Algorithm 3: Code and data placement using relaxation approach

4.4.5. Constrained Programming

The proposed constrained programming solution explained in the previous section is given in Algorithm 4.

Algorithm 4: Code and data I/O assignment management using constrained programming

5. Evaluation

The proposed method and heuristics were simulated on 10 real Raspberry Pi 4B devices. One device acts as the central management unit and is responsible for collecting statistics, maintaining the MQTT broker, and distributing code and data placements. The rest of the workers include applications for data producers, final event consumers, and complex event workers. The main methodology of evaluation is that the experiments are run using simulated sensor data on real-world RPi4B devices, where the simulated sensor data production aims to challenge the algorithms in order to observe the performance differences more clearly. The capabilities of the worker device are reduced to using a single processor and 1 GB of RAM to simulate a more restricted environment. Furthermore, the bandwidths of the devices are reduced to 512 KB/s for both upload/download capacities. Such limitations are applied to ensure that the algorithm performances can be compared under restricted environments. If limitations were absent, the capabilities of the RPi4B devices are more than enough to handle basic sensor loads without issues, which can make the algorithm choice redundant under experimented parameters. As mentioned earlier, due to limited resources, each task is assigned to one worker only; similarly, data outputs for an event are allowed to be written to a single location in the distributed database locations at any time. The configurations can be seen in

Table 4. Experiment configurations.

Parameter	Value	Unit
Processor Availability	1.0	count
Production Frequency Per Sensor	3	Hz
Load Change Frequency	90	seconds
Per Task Data Sizes	20–50	KB
Simulation Duration	600	seconds
MQTT QoS Level	1	value
Bandwidth Limit	512	KB/s
Oscillation Penalties	1.1	value
Experiment Repetition	10	value

. Sensors collect data from the environment three times each second, which generates an event traffic between 3 and 15 at consumers. The size of the data produced changes in 90 s intervals to force optimizations to adapt the placements. Data size changes between 20 to 50 KB, to make sure that the approaches that cannot adapt to dynamic workload are penalized. We have specifically chosen this data-size range to ensure that the experiments do not overload the sensors too much when the bandwidth loads are balanced. Increasing the sizes too much causes delays to increase and it becomes harder to compare algorithm optimization performances. Lowering this value causes all algorithms to perform well as the runtimes become too fast. Therefore the values are chosen empirically after doing repeated experiments with different ranges.

Implementations are evaluated using average minimum and maximum delivery times between consumers, the percentage of data that is successfully processed between source and consumers within different time intervals, and data-loss ratios. Algorithm run-times are also provided to gain an insight on how long the solutions need to run on the management device to obtain the given results. Lastly, the average bandwidth utilization percentage for workers is collected and the maximum of this value is reported. This helps in observing how different optimization approaches balance the load and their relationship with the critical path delay.

Before evaluating different heuristics together, it is important to gain insight on how the smart heuristics such as relaxation and genetic algorithm work with different configurations, as the scenario is more complex compared to previous studies and different cost goals can provide different end results. The first evaluation was performed by observing different configurations for the relaxation approach, the results of which can be seen in

Table 5. Performance of different relaxation optimization goals.

Relaxation Approach	Max Consumer Delay (s)	Max Worker Bandwidth Load (%)	1 sec. Delivery Ratio (%)	Algorithm Runtime
SUM_DELAY	1.39	78.62	64.28	1182.02
SUM_DELAY_H	4.09	78.96	31.58	7722.60
MAX_DELAY_H	6.26	98.65	35.06	7836.47
SUM_BW_H	6.82	78.51	21.92	7776.24
SUM_BW	7.06	78.46	14.78	1122.54
MAX_BW_H	19.37	97.90	5.64	7675.95
MAX_DELAY	21.46	94.50	16.24	1180.39
MAX_BW	56.12	86.05	17.83	1160.54

.

Looking at the table, there are four different goals with two different configurations. There are two different configurations: the first one utilizes the algorithm step that enables place swapping between two tasks (results with _H suffix), and the second one is without this algorithm step (results without _H suffix). The reason is that this step introduces additional optimization overhead and increases algorithm runtime, which can be seen from the table, and therefore it is important to observe its effect as well. As the minimization goal, the sum of expected delays (SUM_DELAY), maximum expected delay (MAX_DELAY), the sum of all device loads (SUM_BW), and maximum device load (MAX_BW) is chosen. As can be seen from the results, minimizing the sum of all delays provides the best critical path delay outcome, whereas minimizing the maximum device load ends up giving the least maximum device load, as expected. Using the additional task swapping step not only increases the algorithm runtime from approximately 1 s to 7.7 s, but also increases the critical path delay from 1.39 s to 4.09 s for the sum of delay minimization goal. Therefore, utilizing the additional task swapping step is not worthy for the experimented scenario. Increasing runtimes causes messages to stay within non-optimal placement status more and therefore can increase the overall delay. Focusing on maximum load and maximum delay without the task swapping step are observed to be the worst performing approaches. This could be due to the fact that a path is a combination of several steps, where focusing on maximum delay may still end up with an unbalanced distribution of delay across the paths, and the critical link will end up reducing the throughput, and hence increases the overall delay.

The second evaluation is performed on different minimization goals for the genetic algorithm, the results of which can be seen in

Table 6. Performance of different genetic algorithm goals.

Genetic Algorithm Approach	Max Consumer Delay (s)	Max Worker Bandwidth Load (%)	1 sec. Delivery Ratio (%)	Algorithm Runtime
SUM_DELAY	5.10	85.91	65.74	13,096.74
SUM_DELAY_H	18.04	79.14	28.76	15,802.74
SUM_BW_H	24.02	95.81	15.50	17,521.21
SUM_BW	37.79	80.88	11.72	12,540.10
MAX_BW	59.03	87.80	0.10	10,499.58
MAX_DELAY	66.82	79.24	0.06	12,943.26
MAX_DELAY_H	87.53	100.07	0.24	19,616.72
MAX_BW_H	89.81	97.04	0.04	16,761.02
PATH	90.98	79.12	0.03	14,889.43
PATH_H	116.44	100.79	0.02	20,912.23

. Similarly to relaxation, the genetic algorithm is dependent on different parameters, and specifically the population size and the generation count affect the algorithm runtime. Two different configurations are chosen for this purpose: the first one (results with _H) has 20 generations with a population size of 50, and the second one (results without _H) has 10 generations with a population size of 25. For the minimization goals, the sum of all expected delays (SUM_DELAY), maximum expected delay across all connections (MAX_DELAY), sum of all device loads (SUM_BW), maximum device load (MAX_BW), and sum of delays across paths (PATH) are chosen. One problem in utilizing GA in any dynamic environment with limited resources is that by its nature, GA is a smart randomization approach. This implies that, given enough iterations and population size, it could potentially find the best distribution strategy. However, since it can take a long time to do this, the runtime should be restricted, and hence it obtains less-than-optimal results. The best performance is obtained by minimizing the sum of all expected delays, which was the same result in the relaxation approach as well. However, even with a low population count, overall algorithm running times are longer and can take 10 times longer than the best relaxation approach. A last point about GA is that when it considers path performances, since it needs to traverse paths for each different combination, the runtime becomes longer, and hence fails to adapt completely. This is why it was observed that the path-focused results perform very poorly. Better GA approaches can be used to solve this issue in future studies.

Lastly, three different optimization goals for the proposed constraint programming solution are tested, and the results can be seen in

Table 7. Performance of different constraint programming goals.

Constraint Programming Approach	Max Consumer Delay (s)	Max Worker BW Load (%)	1 sec. Delivery Ratio (%)	Algorithm Runtime
CP_DDAG	0.67	71.80	91.94	4205.20
CP_DAG	0.95	73.37	77.14	5423.58
CP_MAX_LOAD	1.79	78.99	62.54	3219.86

. The first goal considers the two step load and path delay optimization (CP_DDAG), the second goal considers path delays (CP_DAG), and the third only considers the maximum device load (CP_MAX_LOAD). Maximum device load can be thought of as the global optimal optimization without flexible data placement, which aims to compare our results with the operator-placement-only global optimization approach. It focuses solely on placing the tasks so that the maximum load across all workers is minimized where the data are always served from the database of the worker where it is produced. It can be seen from the results that considering both load and path delay together provides the best result with 0.67 s critical path delay, whereas path-only consideration ends up increasing the critical path delay to 0.95 s. One of the interesting outcomes here is that the two-step approach technically optimizes the problem twice, but ends up having reduced optimization runtime. This is because optimizing the maximum load first reduces the problem space for optimizing the path performance in the next step, which helps it reach an optimal solution faster.

The combination of the best approaches with additional metrics can be seen in

Table 8. Performance of different optimization approaches.

Optimization Approach	Min Consumer Delay (s)	Max Consumer Delay (s)	Processing Ratio (%)	Max Worker BW Load (%)
CP_DDAG	0.13	0.67	99.99	71.80
RELAX SUM DELAY	0.13	1.39	99.91	78.62
GA SUM DELAY	0.12	5.10	100.00	85.91
CONS	0.11	95.54	89.03	89.89
PROD	0.28	103.48	78.61	87.20
Optimization Approach	Algorithm Runtime (s)	1 sec. Delivery Ratio (%)	4 sec. Delivery Ratio (%)	10 sec.+ Delivery Ratio (%)
CP_DDAG	4205.20	91.94	97.22	1.15
RELAX SUM DELAY	1182.02	64.28	91.00	0.56
GA SUM DELAY	13,096.74	65.74	73.17	19.00
CONS	84.98	0.00	0.36	94.67
PROD	130.33	0.00	0.22	97.50

. In addition to CP, GA, and Relaxation, producer- and consumer- oriented results are also provided. Looking at the maximum delays, it can be seen that the proposed two-step constraint-based optimization performs better than the other heuristic approaches. However, when it comes to the minimum consumer delay, it can be seen that the consumer-oriented approach, as expected, has the minimum delay. However, while having minimum delay, due to its less heuristic nature, the non-optimal placement of tasks ends up increasing its maximum consumer delay too much. Furthermore, both consumer- and producer-oriented approaches have less than 90% processing ratio, indicating that their data loss ratio is too high. This implies a 10% data-loss and for some critical scenarios, this could be unacceptable.

Within the same table, negative outcome for the producer is more apparent; this is due to the fact that the event stream tends to increase in traffic towards consumers, and focusing task placement towards producers while not considering the intermediate links and paths can end up increasing the overall delay. Looking at the maximum worker loads observed, it can be seen that the proposed two-step approach was able to reach a more optimal distribution placement, thanks to the flexible data management mechanism and consideration of end-to-end delay, where it has on average 7% less observed worker load compared to the second-best performing solution. In terms of algorithm runtimes, Relaxation is the fastest among the top three, as it is neither too convoluted as a genetic algorithm, nor does it have to run an optimization library method for constraints. Checking only critical path delays may not be enough, because since the traffic is dynamic and approaches iteratively to optimize for better placement, some events may end up getting processed later than others. The effect of this can be seen from comparing delivery windows between relaxation and the proposed solution. Relaxation has a minimal ratio when it comes to processing messages more than 10 s. This implies that while it does not process most of its raw data in a second compared to the proposed solution, overall, it can process most of the data in a time of less than 10 s. On the other hand, focusing solely on how fast the data is processed within a 1 s window, the proposed solution performs best with 91.94%.

Lastly,

Table 9. Performance of algorithms under MQTT QoS level 2.

Optimization Method	Max Consumer Delay (s)	Max Worker BW Load (%)	1 sec. Delivery Ratio (%)	10 sec.+ Delivery Ratio
CP_DDAG	1.07	72.86	56.67	1.44
RLX_DELAY	6.89	79.24	0.06	19.22
GA_DELAY	61.07	78.60	0.06	96.61

shows the performance of the algorithms when MQTT QoS level 2 is used instead of 1. It can be seen that, due to the additional messaging overhead introduced by the QoS level 2, all approaches now have increased critical path delays. However, this reduction in performance affects the proposed method less. One reason could be that the proposed method already had less delay than other approaches, and the introduction of additional messages, whereas increasing the critical path delay from 0.67 s to 1.07 s affected the performance less than the heuristic methods.

Whereas the study does not explicitly enforce time constraints across event executions, an implicit limitation and its effects can be observed from the results. For example, if an event across CEP path is supposed to be triggered 10 times per second due to data production frequency, if the executions take longer than approximately 100 milliseconds, it can be said that continuos executions will begin to delay. Therefore, the critical path performance will degrade. Looking at the results, the better performing models tend to reduce this delay, and if a satisfiability condition were to be introduced such as having less than 1 s execution time across paths, only the methods that can have less than 1 s maximum delay will be able to satisfy the condition. The worse case scenario for each method tends to vary, as whereas the data production frequencies are same, the intermediate delays will present different worst-performances for each method. On the other hand, if the goal was to arrange the scheduling of sensor data collection to ensure that methods provide satisfactory performances, it could imply that for methods that have trouble adjusting to system, less data production should occur to help ease load balancing. To explain it more directly, the maximum consumer delay indicates the average worse-case scenario for the data processed within the CEP flow. Lastly, if an artificial event runtime limit were to be introduced to the experiments that run under restricted resources, most of the algorithms would drop events mid-path and this can make it harder to have a healthy comparison scenario.

5.1. Scalability

One of the issues with the scenario studied is that it is a resource-restricted environment with a limited number of IoT devices and data production. As such, it is not as expensive to optimize a very large network with an increasing number of paths. Therefore, the scalability of the solution remains as its limitation, and unless the management device has high CPU capabilities, it is not suggested to use it for larger problems.

5.2. Limitations

In IoT environments, privacy and network security are important properties that have been extensively studied. These two topics are not within the scope of this study and therefore are not mentioned. However, in terms of security, simple API key and certification solutions should be easy to implement. In terms of privacy, library users should keep in mind that they should not violate any regulations. Another limitation is the single point of failure problem; the study includes one management device as the resources are already scarce, which implies that any failure in the broker will end up resulting all workers to halt operations. Implementations are tested using a docker system and software failures can be alleviated easily; however, hardware failures are not considered and remain a potential problem. Lastly, although the problem of deploying CEP to heterogeneous environments can be done easily as long as the underlying application runs on an OS which can run Python, this may not always be the case due to capabilities of the devices. Therefore, the proposed library only supports heterogeneous deployment where the devices support it.

5.3. Threats to Validity

In the proposed system, constraint programming is solved using the google or tools library [40]. One of the problems with using a solver is that as the complexity of the problem increases, the solution time can also increase. This implies that the proposed method, while valid for small complex flows, may fail to provide a solution for very large graph models. For such environments, problems can be separated into smaller subsets to distribute computation loads. The assumption of no data duplication and a single task instance is also not suitable for large-scale networks. In general, the proposed solution is designed to fit in small environments that are resource-constrained to enable higher throughput and QoS performance.

6. Conclusions

Complex event processing frameworks enable us to perform a variety of tasks on IoT sensor data to reach meaningful conclusions. When it comes to edge-only scenarios, IoT devices tend to have resource limitations such as bandwidth capabilities that reduce their ability to perform these tasks. Furthermore, smart environments can have multiple end consumers that require processed data that originate from multiple heterogeneous data sources that increase the complexity of data flow management. Additionally, such complex operations are widely limited to only executing queries on data, which is useful in terms of temporal and aggregations but is lacking when it comes to requiring different tasks such as object detection, or execution of dynamic programming codes. Lastly, within real-world scenarios where devices are capable of running concurrent tasks, multiple independent complex flows may be executed in distributed IoT at the same time. In this study, a constraint-based optimization is proposed which optimizes the critical path performance across independent CEP flows that are distributed across a multi-producer, multi-consumer, small-scale IoT environment. Our approach enables the use of flexible data placement in mesh-based IoT systems to enable better load management. The proposed optimization is implemented in a Python library, which eases the development and deployment of CEP applications and enables the execution of any code that is supported by the programming language. The results showed that the proposed constraint-based solution can increase the processing speed of raw data sources on the critical path compared to other heuristic operator placement optimizations. The main problems that were mentioned in the introduction section are addressed as follows:

• The mathematical model connected independent CEP flows together in order to prevent ignoring specific paths.
• Across all paths, the worst-performing path is optimized, which the proposed model performed better than the approaches in the literature.
• Flexible data placement mechanisms allowed devices to store/fetch data remotely when required, which helped the proposed model to have reduced worker load compared to non-shared approaches.

Future studies will consider the scalability of the solution for larger systems where IoT devices as well as complex flows can be partitioned into smaller pieces for local optimizations. One of the challenges of the current proposed system is that due to the increased number of complex events, multiple simultaneous connections can exist between devices, making it harder to estimate how many of these connections will delay each other within a time frame. Such problems can be addressed with better scheduling estimation models.