Video Processing from a Virtual Unmanned Aerial Vehicle: Comparing Two Approaches to Using OpenCV in Unity

Abstract

Virtual reality (VR) simulators enable the evaluation of engineering systems and robotic solutions in safe and realistic environments. To do so, VR simulators must run algorithms in real time to accurately recreate the expected behaviour of real-life processes. This work was aimed at determining a suitable configuration for processing images taken from a virtual unmanned aerial vehicle developed in Unity using OpenCV. To this end, it was focused on comparing two approaches to integrate video processing in order to avoid potential pitfalls such as delays and bottlenecks. The first approach used a dynamic link library (DLL) programmed in C++, and the second an external module programmed in Python. The native DLL ran internally on the same Unity thread, as opposed to the Python module that ran in parallel to the main process and communicated with Unity through the Message Queue Telemetry Transport (MQTT) protocol. Pre-transmission processing, data transmission and video processing were evaluated for a pair of typical image-processing tasks like colour and face detection. The analysis confirmed that running the Python module in parallel does not overload the main Unity thread and achieves better performance than the C++ plugin in real-time simulation.

1. Introduction

Image and video processing [1,2] have made it possible to create solutions that promote the surveillance of people in need of assistance [3,4,5]. In recent times, progress has been made in home monitoring using small unmanned aerial vehicles (UAVs). However, incorporating autonomous flying vehicles requires excessively costly and hazardous testing in real environments. For this reason, some proposals are directed towards simulation using virtual and augmented environments [6,7,8], allowing scaled prototyping, evaluation and testing at a much lower cost and with no physical risk to people. Our research group is working in this line, developing a virtual reality (VR) simulation platform for small UAVs as future social robots for monitoring dependent people in the home [9,10]. One should note that any VR platform intended to evaluate the performance of robots prior to the live experience must operate with real-time accuracy. Any such platform also allows one to tailor the design to the preferences of potential end users, so virtual environments and simulations must be as realistic as possible, which certainly can be computationally expensive [11].

Unity is one of the most widely used graphics engines for creating virtual simulators. This Unity game engine is widely used for simulation projects and for quickly generating 3D environments [12]. Therefore, the proposed platform uses Unity to recreate home environments where a virtual UAV monitors a dependent person’s avatar, simulating the behaviour of the future aerial assistant. The operation of the UAV will rely heavily on visual information processing. Using the onboard camera as the main sensor, the aerial robot would know where the person is for the monitoring process, determining his or her status and assistance needs [13]. It would also detect and avoid obstacles in the flight path to ensure safe operation. The VR platform will be paramount to programming the computer vision algorithms and evaluating their performance in a safe environment before the physical assistance UAV advances to the next design stage. Therefore, this work aims to study the video-processing capabilities in Unity, even though this engine was originally developed to create and develop video games, and not for image processing. Currently there are several solutions to this interoperability problem, such as client–server architectures, subscribe/publish communication [14] or the development of native libraries for use without the need to adapt external communication [15]. Although they are only usable in the operating systems they were developed for, native plugins allow the generation of functionalities that cannot be realised in the C# language or through the Unity framework.

In this paper, we considered it appropriate to evaluate two approaches to video processing from Unity using OpenCV (Open Source Computer Vision Library), which are summarised as follows. (1) Using a DLL programmed in C++, and (2) programming an external module in Python that communicates through MQTT with the main program. The way of implementing video processing using OpenCV may affect the final performance of the simulator, so it is important to compare and determine the best option to avoid technical problems such as delays and bottlenecks derived from data transmission and/or processing. Therefore, the main contribution of this work is to compare and evaluate the two approaches.

The remainder of this paper is structured as follows. Section 2 introduces some related works. Section 3 presents an overview of the study conducted concerning the integration of OpenCV in Unity and illustrates the two compared approaches, a native C++ DLL and an external Python module communicated via MQTT. It also details the hardware and software used, discusses the main differences between both approaches and describes how images are converted and transferred from Unity. Next, Section 4 details the results of the study, which are divided into the analysis of the processing prior to data transmission, the transmission of the data and finally the processing of the videos using C++ and Python. The colour and face detection algorithms, programmed in both languages, are evaluated in isolation and jointly (sequentially and also in parallel in the case of Python). Finally, Section 5 presents the discussion of the results obtained and the main conclusions of the study.

2. Related Works

Simulation through extended reality is gaining more and more attention in manifold domains requiring computer vision like robotic manufacturing [16,17,18], the automotive industry [19,20], video games [21], fitting rooms [22], mulsemedia setups [23], and IoT-ready WebApps [24], among others. In fact, by creating virtual environments, synthetic training datasets can be generated in a controllable and customizable manner for further solving common computer vision tasks and avoiding the challenges of collecting and annotating data in the real world [25,26]. For instance, such synthetic images have been used in Unity in vision-based detection tasks (e.g., human body parts [27], 3D markers [28]). In the proper context of UAV research, a framework has been developed for accelerating computer vision performance in data collection, detection, passive ranging, and human–robot teaming [29]. Moreover, in a recent a project, a robot with a soft gripper was aimed at grasping a drone in mid-air [30]. The soft hand was featured with a unique colour-based force estimation technology based on computer vision.

In recent times, some comparisons between C++ and Python programming languages have been performed. For example, a recent article reviewed and discussed various applications of object-oriented programming languages, where emphasis mainly was given to differentiating C++, Python and Java [31]. According to this article, C++ offers facility for low-level memory manipulation and efficient management of memory space. However, it does not support dynamic memory allocation, garbage pickup and threads for built-in facilities. On the other hand, the dynamic nature of Python makes it a slow language due to its higher execution time, and hence not suitable for faster applications. Moreover, it is not suitable for memory-optimised processes, and it is also vulnerable for run-time errors and faces problems while accessing huge databases. Another study concluded that the Python programming language is less complex than the C++ programming language [32]. The results offered by a comparative analysis in terms of memory and time indicated that C++ is faster than Python in execution speed but Python serves as a better language for beginners due to its simplicity [33]. The authors concluded that the language should be selected according to the type of project looking for the best results.

OpenCV is one of the most important and productive libraries for implementing computer vision algorithms [34,35,36]. Studies have shown that in terms of time consumption it outperforms other libraries like dlib [35] and tools like MATLAB. OpenCV is written in C++ and its primary interface is in C++. Therefore, the use of OpenCV natively from C++ facilitates the incorporation of video processing. Hence, a first approach to image processing from Unity would be to develop a native C++ dynamic link library (DLL) for use in OpenCV. OpenCV as a reference library for image processing provides bindings in Python, Java and MATLAB/OCTAVE. Although  C++ is itself the most widely used programming language in OpenCV, Python stands out for its ease of use [37]. In this sense, an external module programmed in Python would be a second approach to Unity image processing using OpenCV algorithms. In this way, the external program communicates with Unity to receive the images from the virtual camera and returns the results of video processing.

Among the possible communication options is the Message Queue Telemetry Transport (MQTT) protocol. MQTT is a broker-based publishing/subscribing instant messaging protocol; it is open, lightweight, easy to implement and does not need a big band weight [38]. MQTT has been selected in our approach as it has already shown excellent results in our previous works for MATLAB/Simulink integration [9,39]. Moreover, the combination of an external module programmed in Python for image processing and communicated via MQTT has recently been implemented to evaluate emotion recognition as part of ongoing research [40,41].

3. Materials and Methods

3.1. General Description of the Study

As described before, the objective of this study was to compare and evaluate two approaches to integrating video processing using OpenCV in a VR simulator developed in Unity. Thus, this study confronted two programming languages and two modes of communication with the main Unity process. The C++ DLL ran internally in the same Unity thread as a native plugin that received and sent the information via a static function (see Figure 1a), whereas the Python module ran parallel to the main process and exchanged the data using the MQTT broker (see Figure 1b).

All comparisons were conducted on the same computer, operating system Windows 10 Home edition ×64, with an AMD Ryzen 7 2700× 8-core processor (16 threads), 16 GB of RAM and a Radeon RX Vega 56 video card. The software used was Unity 2020.3.22f1, Visual Studio 2022 community edition and Visual Studio Code. The version of Unity employed was the stable one during the period of the tests performed. All testing was done directly in the Unity editor to ease the modification of the running parameters and debugging of the code. The virtual environment built in Unity corresponded to a simple indoor (home) scenario where a camera monitored a human avatar moving around (see Figure 2). The visual information was obtained through a render texture and images were captured at different time intervals, the most commonly being every 0.2 s (5 frames per second). The Visual Studio 2019 community edition was used for the development of the class library in C++. Visual Studio Code was employed for the development of the program in Python with MQTT. Both languages, C++ and Python (version 3), used the OpenCV version 4.5.2 to implement computer vision algorithms. Finally, Mosquitto was used as the MQTT broker.

3.2. Assessment Approaches

The scenarios or configurations evaluated in this study represent two completely different approaches to integrating computer vision processing into a simulator developed in Unity, as they face different modes of communication between the main program and the computer vision process, different modes of execution and, of course, different programming languages, with all of the pros and cons that this implies.

The first approach takes advantage of one of the functionalities offered by Unity and its C# language, namely the possibility of adapting libraries that implement extra functionalities that run from the C# code independently of the language the library was developed in. Since all the experiments are carried out on the Windows 10 operating system, the native library has been implemented in the programming language that best suits it, so it has been done in C++. It should be mentioned here that the native plugin serves only as a means of information exchange, since video processing would always run on the same Unity thread. In contrast, the second approach uses the MQTT message protocol to exchange information with an external Python program that also runs in parallel to Unity. In this case, an MQTT broker is needed to link the two programs and manage the communication between them.

As for the differences between the programming languages themselves, C++ is a cross-platform compiled language, while Python is also cross-platform but is an interpreted language. The former is widely used thanks to its high performance in real-time applications [42] and also in embedded systems processing images [43]. In addition, C++ allows a better native integration with OpenCV libraries than Python. However, the latter is characterised by its significant number of compatible libraries and its ease of use. In this regard, it is worth noting that, although OpenCV was developed entirely in C and C++, it is much easier to implement OpenCV in Python than in C++. Moreover, there are many OpenCV-based libraries in Python with ready-to-use advanced processing functions that are not available in C++.

3.3. Development

The study analysed the differences in processing the images generated in the Unity environment by using OpenCV via a C++ plugin or an external Python program. In both cases, the images were identical and corresponded to the human avatar moving around the virtual environment presented above (see Figure 2). The images were converted to the appropriate format to be transmitted to both processing alternatives.

We evaluated the implementation of a native DLL plugin for use in Unity for image processing in C++. This approach received the images received from Unity via an external static function that allowed communication between C++ and the DLL compiled in C++. The images were delivered as a pointer to an RGBA pixel structure, so C++ took care of processing the structure directly with OpenCV. The use of the MQTT protocol for transmitting images from Unity, encoded in base64 and submitted as an array of bytes, was evaluated for data processing in Python. In this way, Python received the images through the MQTT event and subsequently processed them using OpenCV. The MQTT protocol has already been evaluated with particularly good results in some Internet of Things (IoT) architectures on remote servers via the cloud [44,45]. However, in this study, the implementation was performed on the same machine to avoid inconsistencies based on network latency. The communication and processing flows of both scenarios are depicted in Figure 3.

3.3.1. Development of the Native C++ Plugin

A native plugin in Unity is a class library developed and compiled externally to Unity but used from within Unity to add extra functionality to the platform that cannot be operated from within the Unity framework itself. In the present case, this was a DLL to run on the Windows 10 × 64 operating system. The native plugin received a pointer to an array of pixels of the captured image along with its width and length values (see Algorithm 1).

Algorithm 1: C++ external function and pixel colour structure.

int process(Color32* raw, int width, int height);

struct Color32 {

uchar r;

uchar g;

uchar b;

uchar a;

};

3.3.2. Development of the Python Program

For this case, the Python program was run independently of Unity to process the images received via the MQTT protocol. Since all tests were performed locally, the Mosquitto broker was used. The broker managed messages between MQTT clients, so that once it received information about a particular topic, it forwarded it to all subscribing clients. Python used the paho.mqtt.client module and Unity the uPLibrary.Networking.M2Mqtt library for this communication.

For the performance tests that will be detailed below, Unity was not required to receive any results from Python, as the detection results of the different algorithms were directly represented in a window that OpenCV generated natively. That is, the processed image was not returned from Python to Unity. However, it was implemented to send a json file from Python to Unity to return some additional processing details or information in the future, e.g., distance estimation. The data exchange process was as follows. The Unity client continuously published (at a specified frame rate) the base64 encoded images in a specific topic the Python client (or clients in the case of a distributed run) was subscribed to. In this way, the Python client(s) received the images via the MQTT event and then processed them using OpenCV. In turn, the Unity client subscribed to those threads into which the Python client(s) published the json file once the processing of a received image was concluded.

4. Results

This section details the results of the different tests carried out to numerically compare the performance of the two options considered for the use of OpenCV in Unity. Initially, the processing requirements needed before transmitting the data were evaluated. In other words, the conversion of an image to the appropriate format for subsequent transmission and processing in C++ and Python was evaluated. Subsequently, the transmission of the previously transformed information was evaluated. Afterwards, the performance of processing with OpenCV in C++ and Python was analysed with two image processing algorithms, one for colour detection and the other for face detection. Finally, a fully featured simulator running both computer vision algorithms was implemented. The results were evaluated in a sequential execution and also in parallel in the case of Python and MQTT.

It should be noted that the two computer vision algorithms presented in this paper were evaluated to determine the maximum rate of frames per second (FPS) that could be processed without failure in the execution of the simulator. The maximum limits for each of the options considered were up to 20 FPS in the case of the native C++ DLL and up to 50 FPS using Python and MQTT. Thus, the use of Python via MQTT, which significantly offloads the processing load of the simulator, has made it possible to achieve a much higher frame rate. Nevertheless, a common and conservative frame rate of 5 FPS was considered for both approaches in the tests detailed below. First of all, we chose to process 5 FPS since this number stresses the simulator very little. In addition, the detection of a person walking through a scene does not require more than 5 FPS for a correct tracking and subsequent activity recognition, if required. On the other hand, for possible future implementations comprising computer vision algorithms using OpenCV with higher computational load, 5 FPS would guarantee no system bottleneck [46]. This number would also allow working with simpler hardware.

4.1. Processing Prior to Transmission

First, the processing time required to adapt each image obtained in Unity to the appropriate format for further processing using the two options studied (DLL C++ and Python with MQTT) was calculated.

4.1.1. Data Conversion in C++

C++ requires to send a structure containing the colour of each pixel, so that the process is as follows:

• The image is taken from the active TextureRender and stored in a Texture2D.
  •     RenderTexture.active = cameraTexture,
  •     tex.ReadPixels(source.Rect, 0, 0);
• The pixel array is extracted from the Texture2D object.
  •     Color32[] RawImg = tex.GetPixels32();
• Red and blue colours are inverted so that the image comes out with the exact colours and the same orientation as seen in the simulator. The GetPixels32() function returns an arrangement of its own Unity Color32 structure. When it is sent as a pointer to C++, it receives the image with the inverted colours and inverted image orientation.
  •     for (int i = 0; i < rawImg.Lenght; i++) {
  •         byte r = rawImg[i].r;
  •         byte g = rawImg[i].g;
  •         byte b = rawImg[i].b;
  •         rawImg[i].r = b; 
  •         rawImg[i].b = r; 
  •         rawImg[i].a = 255;
  •     }
  •     Array.Reverse(rawImg);
• The arrangement is sent as a pointer and also the length and width of the image.
  •     int result = process(rawImg, width, height);

4.1.2. Data Conversion in Python

Python is required to deliver the image in base64 format, but as an array of bytes. The process is as follows:

• The image is taken from the active TextureRender and stored in a Texture2D.
• It is encoded in PNG format and converted to base64 text.
  •     byte[] bytes = tex.EncodeToPNG();
  •     string base64String = Convert.ToBase64String(bytes,
  •                                        0, bytes.Lenght);
• The base64 text is converted to a byte array.
  •     bytes = Encoding.ASCII.GetBytes(base64String);
• The byte array is sent over the MQTT protocol.
  •     client.Publish(topic, bytes);

4.1.3. Performance Analysis of Processing Prior to Transmission

The process of adapting the original image in Unity to the appropriate format for further processing by computer vision algorithms in C++ and Python demands different processing times. The time from capturing the rendered image to obtaining the image in the appropriate format was measured to calculate these pre-transmission processing times. The result of this test is graphically shown in Figure 4.

The figure shows the time spent to convert 300 images to the required format in C++ (represented in blue) and Python (represented in orange). As can be observed, the processing time required to convert the image in Unity into the appropriate format for submission to the C++ processing option was much longer than that required for the Python option in all cases. With much more variable processing times ranging from 35 to 57 ms, the data conversion process for C++ consumed on average about 40 ms. In the case of Python, more regular behaviour was observed with minimum processing time of 4 ms and maximum time of 17 ms, and an average of just over 6 ms.

4.2. Data Transmission

After the conversion process, each transformed image was passed to C++ and Python for further processing. In order to evaluate the two transfer methods from Unity to the video processing algorithms, a test was carried out to measure the time elapsed from sending the converted image in Unity to receiving it in the native C++ plugin or in the Python program via MQTT. The results of the test conducted for sending 300 images are shown in Figure 5.

In this case, irregular behaviour with fluctuations in transmission time was observed for both options, although an upper limit of about 2 ms is apparent in both approaches. The MQTT Unity-to-Python transmission method was slightly superior to the option of forwarding internally to the native plugin. Although it took more extreme minimum and maximum values, 0.14 and 1.99 ms, respectively, passing to Python took about 1 ms on average, compared to almost 1.20 ms for sending to the native DLL.

4.3. Image Processing with OpenCV

Image processing with OpenCV is optimised for fast execution. However, the processing time is highly dependent on the type of algorithm used. Therefore, bottlenecks may occur during processing. In this case, the processing time consumption was evaluated for C++ and Python languages without considering the previously evaluated data transmission time. As described above, two computer vision algorithms with different computational loads were selected, one that limits itself to filtering colours (low computing load) and a second one that uses a human face detection programme (high computing load).

4.3.1. Colour Detection Algorithm

The low-load processing algorithm consisted of recognising a given colour in each image delivered from Unity. For this case, the blue colour was detected in each frame. The algorithm in both languages was the same (see Algorithms 2 and 3). OpenCV provides a mask filter on the hue-saturation-value (HSV) model to obtain only the matched colour to detect. Contours were obtained and a green rectangle was drawn around the detected colour. The processing results were the same in C++ and Python, as shown in Figure 6. Both colour detection algorithms correctly identified the avatar’s blue trousers in all frames over time.

Algorithm 2: Colour detection algorithm in C++.

findContours(redOnly, contours, hierarchy,

cv::RetrievalModes::RETR_TREE,

cv::ContourApproximationModes::CHAIN_APPROX_SIMPLE);

vector<Rect> boundRect(contours.size());

int x0 = 0, y0 = 0, w0 = 0, h0 = 0;

for (int i = 0; i < contours.size(); i++) {

float area = contourArea((Mat)contours[i]);

if (area > 300) {

bpundRect[i] = boundingRect((Mat)contours[i]);

x0 = boundRect[i].x;

y0 = boundRect[i].y;

w0 = boundRect[i].width;

h0 = boundRect[i].height;

rectangle(frame, Point(x0, y0),

Point(x0 + w0, y0 + h0),

Scalar(0, 255, 0), 2, 8);

}

}

Algorithm 3: Colour detection algorithm in Python.

contours, hierarchy = cv2.findContours(blue_mask,

cv2.RETR_TREE,

cv2.CHAIN_APPROX_SIMPLE)

for pic, contour in enumerate(contours):

area = cv2.contourArea(contour)

if (area > 300):

x, y, w, h = cv2.boundingRect(contour)

img = cv2.rectangle(img, (x,y), (x + w, y + h),

(0, 255, 0), 2, 8)

Three-hundred images of the Unity virtual environment were processed at 5 FPS using the two implemented colour algorithms to measure the differences in processing time between C++ and Python. The processing time of each algorithm is plotted in Figure 7.

On the basis of the results, the colour detection algorithm implemented in C++ was found to have a shorter processing time. On average, it took 0.7 ms as compared to more than 2.5 ms for the Python algorithm. In both cases, fluctuations in processing time were observed, something more evident in Python.

4.3.2. Face Detection Algorithm

The Haar cascade algorithm [47] was used for face detection. The particular algorithm adopted [48] made use of an XML file containing the information from previous training for object detection. The methods implemented in C++ and Python are shown in Algorithms 4 and 5, respectively. Figure 8 shows the faces detected in an image in a green circle. Again, the processing time was evaluated in a test with 300 Unity images at 5 FPS. The results are shown in Figure 9.

Algorithm 4: Face detection algorithm in C++.

cvtColor(img, gray, COLOR_BGR2GRAY);

cascade.detectMultiScale(gray, faces, 1.2, 3, 0 |

CASCADE_SCALE_IMAGE, Size(30, 30));

vector<Rect> boundRect(faces.size());

int detected = 0;

for (size_t i = 0; i < faces.size(); i++) {

Rect r = faces[i];

Mat smallImgROI;

vector<Rect> nestedObjects;

Point center;

Scalar colour = Scalar(0, 255, 0);

int radius;

double aspect_ratio = (double)r.width / r.height;

if (0.75 < aspect_ratio && aspect_ratio < 1.3) {

center.x = cvRound(r.x + r.width * 0.5) * scale);

center.y = cvRound(r.y + r.width * 0.5) * scale);

radius = cvRound((r.width + r.height)* 0.25 * scale);

circle(img, center, radius, colour, 3, 8, 0);

detected = 1;

}

}

Algorithm 5: Face detection algorithm in Python.

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

faces = faceCascade.detectMultiScale(gray, scaleFactor = 1.1,

minNeighbors = 3,

minSize = (30, 30))

x1=0

y1=0

for (x, y, w, h) in faces:

aspect_ratio = w / h

if 0.75 < aspect_ratio and aspect_ratio < 1.3:

x1 = round((x + w * 0.5) * 1)

y1 = round((y + h * 0.5) * 1)

radius = round((w + h)* 0.25 * 1);

cv2.circle(img, (x1, y1), radius, (0, 255, 0), 3, 8, 0);

In all processed frames, both algorithms correctly identified the avatar’s face. However, Figure 9 shows that C++ offers slightly faster processing times, but with greater fluctuations. C++ took between 5 and 12 ms per image, while Python did it between 7 and 12 ms. The average run difference for this algorithm was 2.38 ms faster in C++.

4.3.3. Simultaneous Run of Both Algorithms

As a final evaluation, the two computer vision algorithms described above were implemented in a whole simulator run aimed at detecting both the blue colour and the face of a moving avatar. For it, the two possible ways of operating with various types of image processing algorithms on the same image were implemented. The first consisted of running the algorithms sequentially, either from C++ or Python. The second option was to run the image processing algorithms in parallel in different programs. The latter option would have a great advantage in terms of performance, although, in this case, it could only be implemented using MQTT. Indeed, thanks to the use of MQTT, several programs can run at the same time, all receiving the same image but processing it in different threads. However, with the native C++ plugin, each algorithm would necessarily have to run after the other (sequentially) in the same Unity simulator thread.

Both Algorithms Running in the Same Program

Figure 10 describes how the colour and face detection algorithms were executed sequentially in C++ and Python.

Image processing occurs in the native plugin through a call to a DLL function within the execution flow of the simulator. Metrics are evaluated from the time an image is captured until the function returns the processing result. In contrast, data are sent to and received in Python through the MQTT protocol without interfering with the execution flow of the simulator. As can be seen in Figure 10, the function call is synchronous. Therefore, the simulator cannot continue processing until the function returns a value. On the other hand, MQTT provides an asynchronous communication flow, and this means that Python processes the incoming images independently of Unity’s execution flow. This will result in faster communication, processing and data feedback time.

The significant difference between a plugin and MQTT is shown in Figure 11. Although the times were individually quicker in C++, the interoperability offered by MQTT demonstrates better performance in the communication, processing and value returning flow. C++ showed performance in the range of 40 to 65 ms, while MQTT with Python exhibited performance of 13 to 26 ms. This is almost three times faster than the native plugin processing synchronously on the same thread. This finding is also confirmed when comparing the average processing time in the conducted test. The processing time given for Python is 16.30 ms on average versus more than 43.38 ms for C++.

Both Algorithms Running in Different Programs (MQTT Only)

MQTT allows work to be distributed across separate programs providing asynchronous and independent communication. Figure 12 shows a diagram of how a distributed processing system would work in this study. The colour and face detection algorithms ran in two independent programs to evaluate that distributed system (see Figure 13). The two algorithms were executed in separate Python scripts, each exclusively performing its own processing and transmitting the results via MQTT to Unity. It was necessary to add an additional listening channel to differentiate the messages from each of the programs.

The graph of the total processing time revealed a higher computational load of the face detection algorithm compared to the colour detection algorithm, which is obvious. What is relevant in this case is the advantage that Python offers in being able to distribute the load between two programs (or even more if necessary in other more complex developments). In contrast, in C++, it is not possible to build a distributed system through the communication of native plugins. Even if several native plugins were to be developed, each with different processing, function calls would be synchronous.

5. Discussion

In the quest for solutions that enable in-home surveillance of dependent people, we are focused on the use of small assistive UAVs. However, incorporating UAVs indoors requires costly and potentially dangerous testing in real indoor environments. This article has laid out the groundwork for the simulation and evaluation of the capabilities of real-time video processing within a virtual reality environment based on Unity and OpenCV. It was decided to experiment with C++ as the most widely used programming language in OpenCV and Python given its ease of use [37]. In this regard, two approaches to implementing OpenCV-based computer vision algorithms have been studied in this paper. One approach was to use a native DLL written in C++ and the other was to program an external module in Python that communicated with the Unity process through the MQTT protocol.

From the experiments conducted, it can be concluded that interoperability between Unity and other languages is improved in terms of data flow by using a subscribe/publish communication protocol such as MQTT rather than creating a native plugin as this is dependent on running on the same thread as Unity. MQTT had already been used by our research team in recent applications related to simulated UAV indoors (e.g., [9,39,41]). All video processing was done externally with MQTT, allowing more FPS to be processed. Here, the external approach supported 50 FPS compared to only 20 FPS for the native plugin. Considering the tests performed with a common frame rate, it has also been shown that processing time prior to data transmission is significantly faster in Python than in C++.

On the other hand, a slight increase in image processing performance has been found in the C++ language. Although the difference in terms of milliseconds is practically nil compared to Python, it is worth noting that the version of OpenCV used is the same in both languages. Therefore, the difference may be due to language and execution, in line with recent studies comparing both programming languages [31,33]. In Python, the program is executed in an interpreted form, whereas C++ is a compiled class library, which improves access and processing times.

Thus, when a simulator launches a single process that cannot run in a distributed fashion, the use of C++ would be recommended. However, for simulators that require multiple processes that can run in parallel in a distributed manner, it would be recommended to use MQTT to take advantage of multi-threading. Indeed, in the whole run described above, the advantages of MQTT over a native C++ plugin have become apparent. Python via MQTT has proven to offer the best performance, with processing times up to three times shorter.