Passenger Occupancy Estimation in Vehicles: A Review of Current Methods and Research Challenges

Abstract

Passenger detection and occupancy estimation are vital tasks in many fields. The existing literature emphasises that the increasing demand for such systems will continue to grow. This paper reviews the existing literature specializing in the field of transportation safety and efficiency concerning occupancy estimation in vehicles and passenger detection at public transport stations. A comparison between different approaches to passenger estimation is presented. Discussion on the advantages and disadvantages is highlighted. Hence, this paper provides an analysis of 146 papers on the current state of the field. This review paper concludes that invasive methods provide high accuracy with relatively cheap implementation, while noninvasive systems do not violate passenger privacy but lack state-of-the-art accuracy. Future work will include a systematic literature review and a comparative analysis of systems considering the existing window tinting and solar windshields heavily blocking certain parts of the electromagnetic spectrum. Moreover, future work will investigate the critical challenges of noninvasive passenger estimation in different types of vehicles: trucks, buses, or even motorcycles.

1. Introduction

With the advancement of computational power and the improvement of road infrastructure, the development of modern passenger detection and occupancy estimation systems has become the major trend. As a result, many professionals in the field started developing both invasive and noninvasive systems. These systems can be implemented in various sectors: transport systems, surveillance systems, autonomous vehicles, and the military. Different approaches have been presented in the past, starting from simple roadside visual inspection by police officers to complex and expensive video surveillance systems with neural networks. On-site observation brings huge difficulties to the safety of officers, slowing or fully stopping ongoing traffic and not having great accuracy [1].

It is generally easier to implement invasive passenger detection systems with usually very high accuracy. In the case of noninvasive passenger estimation, many environmental aspects affect the final accuracy of such systems. High dust concentration, shades, sun gloss, rain, snowfall, fog, and quality and cleanliness of windows and windshields play important roles in final accuracy [2]. Other factors are distance to the target, speed of a vehicle, occupant position, and more. Glass tint is also a very important attribute to consider in noninvasive passenger detection. Car manufacturers are developing more and more sophisticated ways to mitigate high vehicle temperatures with special types of window tint or special glass to reduce sun energy transmission through windows and windshields.

This paper takes a new look at various methods of passenger detection. PIR sensors, infrared and visible cameras, TOF (time-of-flight) sensors, radar, carbon dioxide sensors or capacitance sensors are a few examples of invasive passenger detection. For noninvasive passenger detection, multispectral cameras or cameras working in the visible part of the spectrum are usually used. The first approach to the detection of passengers was face detection, then technology moved on to methods of extracting other features such as the bodies of passengers, seat belts, or the detection of skin in the foreground versus the seats in the background. With increasing computational power, newer methods are being proposed to detect passengers. Some state-of-the-art methods depend heavily on image pre-processing and HDR (High Dynamic Range) fusion, others depend on new neural network architectures. Many modern vehicles are equipped with new solar windshields. The technology of controlling emitted radiation is relatively young, its global deployment is estimated at 27% in the automotive industry and 60% in the construction industry. This factor might lead to even newer methods for noninvasive passenger estimation.

There are many methods for detecting people, but extremely few reliable methods can be applied to people in vehicles. The development of technologies enables the improvement of transport systems, their efficiency, and the deployment of intelligent transport systems. However, it still remains a challenge to create a fast, robust, and secure algorithm for detecting passengers in the vehicle ecosystem, where there is considerable resource limitation and low computational complexity. The present findings in this review might help to solve several courses of action in order to solve such a difficult problem of occupancy estimation. With this in mind, we tried to select the best references and provide a crucial overview to contribute significantly to the sector.

2. Invasive Passenger Detection Methods

Reliable invasive passenger counting is critical to many existing and future safety systems. Secondly, it might also be used to improve passenger comfort—occupant-dependent control of air-conditioning. Such counting systems must work with top-notch accuracy for safety systems and also in emergency situations. There are many ways to count passengers inside vehicles. This chapter summarizes possible ways to achieve this goal.

2.1. Vision-Based Systems

Vision systems are gaining more and more popularity in security applications. The use of computer vision is also very challenging due to extreme variations in lightning from extremely bright to dark nights. This problem also exists in noninvasive detection. Shadows, both moving and stationary, further complicate the problem. To improve the safety and comfort of the passengers, Gautama et al. proposed a stereo system to observe the cockpit scene and to improve airbag firing control [3]. In this paper, the authors compare different techniques and the influence of random and systematic errors on final parameters (robustness, processing speed). The census transform technique is preferred over zero-mean normalized cross-correlation due to better results in precision and it is also well-suited for real-time applications.

A stereo system by Faber for an intelligent airbag system is proposed in [4,5]. The system classifies seats and tries to estimate the geometry and position of the head. The shape of the human head can be modelled by an ellipsoid. Two monochromatic cameras are mounted on the windshield. No additional lighting was used. The proposed software system consists of the correction of distortions, epipolar rectification of stereo images, feature extraction, feature-based matching, seat occupation detection, and verification with an approximation of the head. Further research is needed to integrate arms and hands.

Devy et al. proposed a stereovision system in their paper [6] and seat situation is recognized with a case-based classification method. A pair of cameras are mounted at the head console combined with infrared illumination. One main issue remains—concern about the reliability of the classification function and computation time. The performance of the proposed stereovision in terms of algorithm execution is set to 250 ms.

Later, in 2003, Schoenmackers and Trivedi discussed a system to determine the position of the head, torso, and arms in front seats inside vehicles [7] with a real-time stereo camera and corresponding depth map of the interior. Firstly, the position of the head is determined, and next is approximated position of the torso.

In 2000, Marcus Klomark in his master’s thesis described and investigated methods for occupant and object detection using computer vision in a car [8]. Autoliv designed an adaptive airbag system with ultra-sonic, weight, buckle, and seat position sensors. Based on data from these sensors, the microprocessor regulates the inflation of the airbag. Stereovision is also discussed is greater detail. Changes in lighting do not affect the system greatly because both images are taken simultaneously. At the same time, the challenge remains in finding corresponding regions in the images and calibration of two cameras. This method is also considered too complex and slow. The structured light method is performed with active special illumination. Deformation caused by an object is extracted by the camera. This method is robust but requires this special lightning that might possibly distract passengers. Moreover, this method is prone to failing under certain conditions. Motion detection is an effective way to distinguish objects in a set scene. This method should be used alongside another method. Sleeping persons or children hidden by a blanket will not be detected. Poor robustness to shadows and rapid light variations is another huge problem. Colour vision with segmentation and binary erosion followed by dilation was also proposed. Hue and saturation space might look promising, but if the background is similar in colour to the skin, this method fails. This method can detect only persons. A neural network can be trained to detect faces and other objects. To correctly train it, the network needs a lot of training data. Training a network might be computationally expensive, especially with large networks or lots of training data, and training can take several months. For an evaluation of the above-mentioned computer vision methods see Table 8.1 in [8]. Some criteria used in this evaluation consisted of robustness, reliability, stand-alone method, difficulty, calibration, etc.

To achieve 100% success, more than one camera or sensor is needed to achieve this and to ensure greater robustness. The main disadvantage of invasive methods is related to the way to communicate a number of passengers to the outside world safely and reliably.

HOV lanes continue to grow in cities as they provide a viable alternative to more efficient transportation and reduction of vehicles with driver only [9,10]. In support of HOV lane monitoring, Schijns presented in his final study ways to detect the number of people using HOV/HOT facilities [11]. The technology used for invasive sensing includes mechanical systems, various forms of photography, LED imaging, infrared sensors, thermal imaging, weight sensors, capacitive and electric field sensors, ultrasonic range sensing, “medical” application sensors (heartbeat, breathing monitors, etc.), smart cards and readers or biometric recognition. At that time, the following major concerns remained: personal privacy, legal changes, cost and economics, and practical issues. Until 2014, there did not exist any system capable of the high efficiency and reliability required for automatic enforcement of HOV lanes apart from invasive ones [12].

In 2004, the camera equipped with a 360° parabola-shaped mirror and NIR LEDs was proposed with Viola’s classifier cascade in [13]. Wender and Loehlein proposed modifications to Viola’s cascade classifier system. The goal of the project was to gather and transmit information about the crash and the state of the car occupants in 2004. This is very similar to the existing e-Call system in the EU. The proposed improvement is to replace the binary output of the weak learners with a floating point output of the sigmoid function. Additionally, special small form-factor IR sensors are proposed by Géczy et al. in [14]. AMG8833 sensor from Panasonic is an 8x8 array of IR thermal sensors and with the Arduino platform, the authors achieved exciting results. It was shown that the system could detect people in the front with a sensor placed at the front. In this scenario, back passengers blend into the ambience due to the large distance and low resolution. More sensor nodes are needed to detect back passengers.

Monocular 360° NIR camera with 7.5 fps, 8-bit grey scale values, and a resolution of 720 × 576 pixels is used in the paper [15]. The vehicle was also equipped with five NIR lighting modules. Makrushin et al. evaluated different approaches to pattern matching, the impact of local normalization, edge detection, multi-algorithm, and temporal matching-score fusion. The authors included all five seats to detect passengers. Frames are stabilized by aligning the centre of the current video to the reference. The steering wheel was used as the search pattern. As expected, the accuracy is not competitive compared to non-optical passenger detection systems but can achieve similar results.

Two computer vision methods (Viola and Jones face detector and Bag of visual words) integrated as a vehicle-to-infrastructure cooperative system are proposed in [16]. A wireless module is used to send images from the in-vehicle camera to the receiver at the infrastructure side. There they are processed with sufficient computing power. Three regions of interest—one for each front passenger and a third for behind passengers—are experimentally separated into each frame. A Bag of Visual Words framework is suggested to improve the performance of the Viola and Jones faces detector during changes in lighting, lens distortion, or head position. A dataset of 1400 photos was divided in half, with 700 patches representing faces and the other half the background. A year later, the authors published another paper [17]. They implemented two classification methods—Naive Bayes and multi-class SVM.

A tiny convolutional model with input from an in-vehicle thermal camera was proposed by Nowruzi et al. [18]. The number of people was estimated with great accuracy, but the main drawback of this solution is the situation when the ambient temperature inside the vehicle is too high. Additionally, all vision-based solutions introduce privacy concerns and sometimes might distract the driver with a bad sensor position.

The front-facing camera mounted on the vehicle capturing images of nearby vehicles was investigated by Amanatiadis et al. in paper [19]. This system is capable of counting the number of passengers in nearby vehicles. Vehicle detection is achieved by applying Haar-like and HOG-SVM (Histogram of Oriented Gradients—Support Vector Machine) techniques. To eliminate the motion, frame registration is performed and ROI (Region of Interest)—windshield is extracted. To detect passengers, CNN (Convolutional Neural Network) was used. The proposed method showed promising results in most cases. Kumar et al. investigated front and side pictures of the car to estimate the occupants using ResNet-50, VGG-19, and GoogLeNet networks [20]. Two seconds are needed for this system to classify a vehicle as violator or not with 96% accuracy.

Vision-based systems still remain the favourite approach for occupancy detection and they bring new possibilities and improved algorithms every year. The only problem is communication with the outside world.

2.2. Pressure Sensing

Zhu et al. studied the characteristic of pressure when the human body gets on or off the stairs in the public transportation environment in paper [21]. This proposed method can distinguish the direction of passengers. Other existing technologies have disadvantages—many of them cannot distinguish passengers’ forward direction and many of them are very expensive. In the next paper, the authors propose a counting method of passenger flow based on human body kinematics and SVM [22]. The pedal on the stair of the bus has four pressure sensors for analogue output. The walking process of a person can be divided into the supporting phase and the swing phase. It is also possible to estimate the number of occupants using indirect methods. Luo et al. proposed to extract the motion signature of boarding occupants in paper [23]. After that, the weights of the occupants are estimated by fitting the response with a transient vehicle dynamic model. Existing onboard motion sensors such as accelerometers, angular rate sensors, and suspension height sensors are utilized for this task.

2.3. Capacitive Sensing

Capacitive occupant sensing is a great contactless method to distinguish between the presence, and position of an object or person. In comparison to pressure sensing, capacitive sensors do not have wearing parts. A new modelling approach for capacitive occupant detection in vehicles is proposed by Satz and Hammerschmidt [24]. A new electrode area variation method is also introduced. The receiver and transmitter electrodes are placed on a vehicle seat. Four capacitances are contributing to the signal and are forming a bridged T-network. The proposed method allows the extraction of a full set of physical model parameters. In [25], a system with employed carrier frequency method and lock-in-amplifier technique is used to measure the capacitances and the influence of external electromagnetic fields is minimal. A single receiving and eleven transmitting electrodes are used. With a single receiving electrode, the calibration of such a system is easier than with a system with multiple electrodes. Electrodes are stitched to a cloth and placed on the sitting and backrest areas of the car seat. To complete a full set of measurements, 200 μs is needed.

The high-frequency measurement principle is introduced in [26]. The electromagnetic emissions of a system may be the issue at high frequencies and signal intensities. Techniques were used by Zangl et al. to lessen these emissions. Fully exposed electrodes must be in the environment for an occupancy detection system to function. One common receiver electrode and a variety of transmitter electrodes are utilized. To decrease emissions and adhere to EMC (Electromagnetic Compatibility) requirements, frequency-hopping spread spectrum and direct sequence spread spectrum are utilized.

If the passenger touches the chassis during the measurement, the capacitive sensors work inaccurately. Tumpold and Satz investigated in detail the effect of a variable passenger grounding in [27]. The proposed system can differentiate between empty and occupied seats and detect passengers out of position. To mitigate the effects of grounded passengers, a combined inductive-capacitive proximity sensor is presented in papers [28,29]. Both inductive and capacitive measurements are achieved by a single sensor. A Signal conditioning unit based on a carrier frequency principle is used.

Sensing of dielectric dispersion effects of biological tissues is used for occupant detection in [30]. Satz et al. proposed an innovative impedance model for seated passengers in vehicles. The impedance is analysed between electrodes as a function of frequency. By mapping the sensor signal onto model parameters, the passenger impedance model allows detectors to be very sensitive to the presence of biological tissues. Measurements are executed between 50 kHz and 5 MHz. Seat occupancy detection based on capacitive sensing is also proposed in papers [31].

Monitoring of vital signs was investigated in paper [32]. Non-contact methods for capacitive ECG monitoring, mechanical heart activity, and magnetic impedance monitoring were used. Results might be used in embedded operator supervision, health care applications, and monitoring of citizens.

2.4. Seat Belt Detection with Vision Systems

An important and mandatory safety measure for passengers is to wear seat belts properly. Many countries enforce wearing them, otherwise, the drivers will be punished. Nowadays, checking is usually done manually during road inspection with high risk and low efficiency. In [33], Huiwen et al. proposed a system to first locate the driver area, secondly detect potential seat belt edges from HSV (hue, saturation, value) colour space, and lastly, judge rules are used for final verification. Many existing methods are based on edge detection. For complex road backgrounds, a deep learning approach is presented in paper [34]. To train CNN (Convolution Neural Network), multi-scale features from the regions of the vehicle, windshield, and seat belt are extracted. To train a classification model through SVM, relative positions of these vehicle components are used. CNN ConvNet trained on the Seatbelt dataset is proposed in [35] and a YOLO (You Only Look Once)-based model is proposed in [36]. A cascade Adaboost classifier-based seat belt detection system is used to find windows and Canny edge detection on a gradient map with Hough transform is proposed in [37]. Straight lines of seat belts are extracted for law enforcement purposes. A seat belt reminder system using IR-UWB radar to check the presence of a person for each seat is proposed in [38]. Elihos et al. developed a model that utilizes both RGB and NIR images in the decision-making process in paper [39].

Seat belt detection may appear to be a suitable system for passenger counting, but not all passengers necessarily have seat belts on, and seat belt detection also fails on black clothing.

2.5. Ultrasonic and Radar

To overcome the limitations of the common occupancy sensors, the quadrature Doppler radar is used in [40]. Yavari et al. measured heart and respiratory signals to improve stationary subject detection with a back-scattered electromagnetic signal. The antennas were 1 to 1.5 m from the subject depending on the intensity of activities. True presence detection is achieved when the human subject is at rest and moving at different activity levels.

Seat occupancy and breathing rate estimation from the amplitude peaks are proposed in paper [41]. A low-cost pulsed mm-Wave radar operating in the 60 GHz band is used due to license-free bandwidth with high data rates. Unlike the previous system, this method does not have high complexity and nor require higher computational costs than simple CW (Continuous Wave) radars or the proposed pulse coherent radar. FMCWs (Frequency-Modulated Continuous-Wave Radar) can also determine the range of multiple persons and angles using beamforming techniques. Novel systems using radar applications have recently been proposed [42,43,44,45,46,47,48].

To reduce the complexity as well as the cost of the overall system, Sterner et al. proposed an RF-transmission system to measure the attenuation of high-frequency radio waves in paper [49]. The amplitude and the phase signals received by the antenna change after a person is seated, thus allowing for passenger counting. A typical memory lapse that could have tragic results is forgetting children. Diewald et al. [50] describe a commercial development for applications inside automobiles. The sensor can also be used to track heartbeat and breathing patterns as vital signs. Noise, traffic, or weather have no effect on the proposed sensor.

The most popular sensors in buildings for individual presence are PIR (Passive infrared) sensors. However, these sensors are fundamentally motion detectors and react to incident radiation variation leading to false negative detections and inaccurate occupancy estimation. Thus, Wu and Wang proposed LAMPIR (Lavet motor PIR) sensor for true presence detection [51]. Classification accuracy of 100% could be achieved for stationary occupants within 4.5 meters and moving up to 10 meters. PIR sensors with an IoT (Internet of Things) system are proposed in [52] with the implementation of a GPS module for localization of the bus. Data are sent to the Firebase database for quick and easy access with mobile devices.

2.6. Smart Cards, Identity Documents, and Cell Phones

For occupancy monitoring, items such as driver’s licenses, cellular phones, or personal identity cards may be used. It is noted that items like smart cards and cell phones have become increasingly widespread. These items may be read and then information sent outside via appropriate communication channels [53]. Many studies have been reported on such occupancy sensing. Smartphones with enabled Wi-Fi send out probe requests containing unique MAC addresses and Pattanusorn et al. investigated the possibility to estimate the number of passengers in the vehicle by monitoring these requests in paper [54]. This method fails if passengers have disabled their Wi-Fi and succeed even in crowded environments with the use of GPS in Raspberry Pi 2 model B. To differentiate between legitimate passengers and non-passenger, the time period and signal strength from each probe request are analysed. In [55], the authors estimate the number of bus passengers with the collection of RSS data from mobile phones that passengers carry on them. To reduce overestimation from nearby vehicles, several Wi-Fi sensors are deployed. Moreover, some passengers might carry several devices or young might carry none.

3. Noninvasive Occupancy Estimation Methods

Noninvasive passenger detection plays an important role in developing better and safer transport infrastructure without sacrificing the privacy of passengers inside vehicles. Many modern vehicles are equipped with an onboard eCall system capable of sending information about the number of passengers or fastened seat belts at the time of the crash. Possible methods are discussed in papers [56,57]. eCall system comes with a microphone installed inside the vehicle, making it vulnerable to eavesdropping and occupants have no control over the remote activation of this microphone. Therefore, there are also noninvasive solutions to estimate the occupancy of a vehicle, and the current state is described in this section. There are already proposed systems for enhancing the eCall system [58]. The most difficult task is to detect sleeping adults and children. The number of missed people was dependent on the camera angle and ranged from 21% to 51% [59]. The reliability of machine vision systems is therefore unlikely to be high enough or groundbreaking compared to invasive methods.

Billheimer et al. proposed the first system to detect passengers inside a vehicle with the use of cameras in 1990 for HOV lane surveillance and enforcement [1]. They also pointed out the high cost of enforcement of 10 HOV lanes in personnel costs and estimated it at 400,000$. Four years later, Mecocci et al. proposed an automatic system utilizing images recorded on two VHS tapes to count persons getting in and out of a bus [60]. Systems based on the idea of counting passengers when entering a public transport vehicle or entering stations are discussed in separate Section 4: Counting passengers at public transport stations. Similar to HOV enforcement systems, ATC (Automatic toll collection) systems have become popular for collecting tolls for their simplicity for the end user and agencies. Information from such systems might be used for multiple tasks, e.g., vehicle classification, passenger counting, etc. [61].

Pavlidis et al. investigated possibilities for passenger counting and detection of vehicle occupants in papers [2,62,63]. The authors investigate the visible and NIR region of the EM spectrum and the unique reflectance characteristics of human skin in the NIR spectrum. With the Mitsubishi Thermal Imager IR-700, authors investigated possibilities to detect people through the windshield and when shooting from the side. They achieved clear images from the side, but could not see anything in the frontal view. The authors then decided to go back from 3 μm back to the range 1–2 μm. The operating range of 1.4–1.7 μm appears as a good choice. It is far from a visible band and cannot distract drivers. Additionally, the transmittance of typical vehicle windows in the NIR region is at least 40% [62]. With car glass with window tint, the transmittance of such a glass drops below 40% and varies depending on the type of tint, but usually stays above 20%. After obtaining high-quality images with two NIR cameras (one in the lower and one in the upper band) captured with the system with filters, and a computer-controlled light source, a fuzzy neural classification was proposed. This classifier scored below 50% in experiments. In 2020, Lee et al. proposed a system for the two-sided camera with only the right side capable of detecting the occupancy in paper [64]. For the binary case in HOV lanes, the system achieved an accuracy of 99% and for detection—87%. A new labelling method with a small amount of data is applied to images captured with two infrared ray cameras with illuminators. Comparison and analysis of the performance of deep learning models are shown in [65].

In 2003, Wood et al. proposed a covert camera system for screening vehicle interiors and HOV enforcement utilizing infrared strobe light to illuminate passenger and cargo compartments through side windows or the windshield [66]. With a high-speed, digital, infrared camera they were able to capture clear, stop-motion images of interiors suitable for human screeners or pattern recognition algorithms to count the number of passengers or identify particular individuals. The main system is divided into three subsystems—imager, illuminator, and trigger subsystem. An 830 nm long-pass filter was added to the camera so it only receives IR. IR illuminator provides light enough energy to overpower the solar background, removing ambient conditions. The energy output was kept at levels that are easily eye-safe. A long-pass filter was located in front of the xenon flash lamp to remove visible light and only IR is presented to the drivers. Tinted windows present a challenge to this system, but it succeeds in getting through common ones. In the severely tinted glass, the occupants could not be seen.

Image segmentation techniques are used to find windshield areas and face detection is executed to count the number of occupants in the paper [59]. Colour CCD cameras were utilized by Birch et al. Images are transformed from RGB (red, green, blue) to HSV (hue, saturation, value) colour space. Following noise removal with a median filter, binary labelling, and a biggest area search for dilatation, erosion, and windscreen, the colour mask is then post-processed. For $sim$ 80% of automobiles and trucks, this method has been proven to be reliable. Only 38% of the faces were discovered because of the inconsistent lighting.

In [67], Lee and Bae experiment with different wavelengths with regard to the transmittance of glass on vehicles and seek to obtain better-quality of image data. This work provides a good theoretical basis for the selection of subsequent components for other papers.

The face detection algorithm proposed by Hao et al. with mathematical morphological operations is employed in paper [68]. The algorithm is effective under normal daytime conditions. Pre-processing of the image consists of coarse detection based on a lighting compensation skin-tone colour model, fine detection based on the correlation of skin colours, and further mathematical morphological operations. Fuzzy ART neural network is used to detect occupants. Hao et al. investigated near-infrared imaging methods since 2006 in papers [69,70,71]. Firstly, the vehicle windshield regions are extracted based on Hough transform methods and humans were detected with AdaBoost-based face detector or HOG (Histogram of Oriented Gradients) descriptors.

In 2008, Tyrer and Lobo published a paper [72] regarding passenger occupation limitations and issues and performed night tests with captured IR images and images in the visible part of the spectrum. The authors display the reflectance spectra of Caucasian, Asian, and African skin as well as the transmission and absorption spectra of typical windscreens. Additionally, experiments were conducted with cars traveling at 110 km/h, on gloomy days, exceptionally bright days, in the middle of the day, etc.

By mixing information from different types of classifiers, Pérez-Jiménez et al. achieved faster and more robust detection through windshield [73]. Different features are searched to characterize people—faces—and safety belts. A cascade of boosted classifiers for fast feature detection is used with a more powerful k nearest neighbor (k-NN) to filter previous results. A face detector based on Haar-like features is used. k-NN classifier is trained to filter the results of the cascade classifier. Safety belts are used as an additional feature to look for and two cascade classifiers are used for the left and right sides. This system achieved an almost 90% success rate with 2% false detection.

US patent by Alves proposes an HOV enforcement system in [74]. The roadside unit includes Ethernet cameras, night-time lighting, and image servers. A laser ranging device is used to detect a car in the HOV lane. A camera is triggered and captures faces through the windshield and a license plate. US patent [75] takes advantage of long contiguous horizontal line segments and curve segments to differentiate between the occupied and empty seats. A test using edge linking, the softness of the edge, the number of lines, and other techniques are used to locate horizontal edges in the image to indicate an unoccupied seat.

In noninvasive passenger estimation, windshield localization plays an important role. Yuan et al. investigated a maximum energy method to extract the windshield regions and HOG descriptors to detect occupants from extracted windshield regions [76]. There are several challenges in windshield detection—it has a different shape, size, and relative position, there might be low contrast between the vehicle and the vehicle body, various capture angles, and distance and complexity of the interior. The proposed method is to combine shape, colour (greyscale), and complexity, and the method can work with colour and NIR images. To detect occupants, HOG descriptors are adopted as the occupant’s features.

Artan Yusuf et al. investigated noninvasive detection with vision systems in papers [39,77,78,79,80,81,82]. A NIR camera pointed at the windscreen of the car was suggested for use in HOV lane occupancy identification utilizing Fisher vectors and a bag of visual words. A system for law enforcement cell phone usage was later proposed, using the same classification techniques. A combined system for mobile phone use, seat belt infractions, and occupancy detection in HOV lanes was suggested by authors in 2016. To determine the type of infraction, classification using local aggregation-based image characteristics is first done on the region of interest, which is the vehicle’s windscreen. Front and side views of a data collection of approximately 4000 photographs, taken on a public road, were employed. Additionally, a system is available for front-seat child occupancy identification, picture classification, and object detection for seat belt infractions.

Distance-based metrics between descriptors to discriminate between images with only the driver and driver with the passenger in the front image of the vehicle are investigated by Xu et al. in the patent [83]. Face detection is explored in the paper [84] for detecting passenger faces using a pixel threshold. A system with two cameras and a comparison of images in early or late fusion to catch an HOV lane violator is proposed in [85]. Two DPM (Deformable part models) are trained to detect the front row and second row. Lastly, for HOV/HOT lane enforcement, the authors investigated three popular CNN architectures in classifying passenger/no-passenger images [86]. Experiments showed that GoogLeNet outperformed two other nets.

An algorithm for cell phone identification used during driving is proposed by Berri et al. [87]. This system based on SVM with a Polynomial kernel achieved a success rate of almost 92%. In pre-processing, three detectors are applied based on Haar-like features for feature extraction. Experiments were performed on a small set of frontal images (100 positives and 100 negatives). Invasive driver cell phone usage detection is also proposed in [88] and noninvasive in [89].

Cornett et al. explored the construction of a multi-unit computational camera system to get consistent face recognition results [90,91]. The system performs HDR (High Dynamic Range) imagining to create a dataset of through-windshield images. Distance to target, bad lightning, strong glare, the pose of occupants, and speed are the main challenges that this system tries to mitigate.

Noninvasive methods present new obstacles and usually do not provide great accuracy compared to noninvasive occupation detection. The main problem is different types of vehicles—trucks, buses, passenger cars, and special cars, where occupancy estimation systems have to be modified for every scenario. The height or position of sensors needs to be adjusted for a robust and universal system.

4. Counting Passengers at Public Transport Stations

An automatic system for monitoring and counting people in various environments is an important task in several fields, where the flow of people who enter, stay and exit an area is an important piece of information. However, monitoring the number of passengers is a difficult task [92].

This chapter contains a specific approach to determining the occupancy of a vehicle by counting passengers before they enter a public transport vehicle at the station or the entrance of vehicles like buses, trains, metro, etc. The number of people entering markets, shows, and exhibitions is also important and provides tools to better optimize, improve quality and create safer environments but in this chapter, we will focus only on counting people in public transport systems.

In articles [93,94], Deparis et al. introduced a new method for counting passengers in public transportation using two active linear cameras. For the purpose of counting the number of passengers passing in front of the cameras, the writers examined image sequences. Compared to the previous image processing paper by Mecocci et al. [60], optical sensors presented additional benefits in the picture acquisition section. With a worldwide error for the morphological algorithm and averaging-thresholding technique of 0.56% and 0.48%, respectively, the suggested device was able to detect pedestrians moving through a 3-metre-wide tunnel.

Three years later, Gerland and Sutter proposed a system utilizing one infra-red sensor with an integrated optic element to accurately count and distinguish passengers [95]. To mitigate climate and weather effects, the sensors are mounted overhead in the door frame. When a person moves under the sensor, signals are generated and transmitted to the analyser unit. This system does not count irrelevant objects. With similar camera placement over bus doors, Bernini et al. presented a stereo vision system in paper [96] with a zenithal position of cameras in a setup similar to [97]. This setup is illustrated in Figure 1. This approach removes the overlapping problem. The zenithal camera is also investigated in [98]. A novel feature-point-tracking and clustering-based passenger counting framework promising better performance than background-modelling and foreground-blob-tracking-based methods are proposed in the paper [99]. When used with a single camera in challenging situations including crowded areas and occlusion among people, the suggested approach achieves an accuracy of up to 96.5%. In a genuine dynamic scenario, it might be challenging to determine the ideal value for a threshold, which is typically used to acquire the moving foreground. By using a KLT (Kanade–Lucas–Tomasi) tracker, the suggested system determines the motion trajectory. A KLT corner detector is also proposed in [100] and shows good results in tracking. Dense stereovision in buses with 99% and 97% accuracy was proposed in reference [101]. Yahiaoui et al. improved the stereo-matching method that can compute precise and noise-free height maps. These maps are segmented to detect the heads of people. Morphological operations and a binarization with multiple thresholds are used. The use of colour images was avoided to reduce processing time. Lengvenis et al. proposed four algorithms to calculate passengers on public transport [102]. Researchers looked into a number of barrier simulation methods, including one based on intensity maximum detection, one for zones, one for object shape correlation, and one for barrier simulation. It was demonstrated that different approaches produced better outcomes under various circumstances. ABSZ performed admirably in extremely complex scenarios, while ABIMD showed promise when only one passenger was in the frame.

A cost-effective method of counting passengers that prevents occlusion issues is the zenithal camera placement scheme. The camera wobbling and changing lighting are two major challenges. To overcome these challenges, Chen et al. deployed a zenithal camera and are counting passengers getting on and off a bus using block motion characteristics [103]. A passenger is constantly travelling in the direction of the door, and the motion vector can be used. Feature pixel selection eliminates erroneous motion estimations. The centre of gravity of moving objects is determined using bounding boxes. When a passenger crosses two baselines, the counting is updated. On average, this counting technique has a 92% accuracy rate.

More early works on the issue of passenger detection are based on simple video cameras. These early works usually use existing surveillance cameras but are vulnerable to ambient illumination changes and complex backgrounds [104]. To reduce mentioned limits, depth data from stereoscopic cameras are presented [105]. Van Oosterhout et al. [106] use a shape-based approach using range data instead of RD voxel. Another common technique is to look for omega-shaped contours or use the partial ellipse fitting technique to find heads. An RGB-D sensor positioned over each door of the bus is used. Automatic head detection with the camera on the bus ceiling is proposed in [107]. The Hough transform and Canny edge detection are combined to find heads. When utilizing the Hough transform, the key benefits include invariance to scale changes, rotations, and robustness to minor occlusions. Similarly, by adopting Haar features and Adaboost algorithm to detect human heads through OpenCV, Yingjie et al. achieved on average 94% correct detection in the paper [108] aimed at subway applications. A background subtraction method and morphology processing for subway passenger flow are proposed in [109].

In 2017, Liciotti et al. proposed a system for counting and monitoring adults and children at the entrance of the bus with the system based on an RGB-D sensor located over the bus door. This approach provides greater reliability and accuracy with real-time processing and low-cost hardware [105]. Invasive passenger detection in the bus also proposed in [110,111,112,113]. In 2018, Bellow et al. proposed a dataset of a full-size laboratory observation of people boarding a public transport vehicle [114]. The authors also provided baseline results for the future. They proved that it is possible to detect most people but there is still much to improve in a final application to be adequately robust.

Sensors working in the visible spectrum have fundamental limitations such as ambient lighting, colour representation, and scene variation. By using low-resolution infrared and visual cameras, Amin et al. assessed the potential accuracy and robustness of such a system [115]. Low-cost Webcam and IRISYS thermal imager were mounted looking vertically down. In the first stage, the image is pre-processed. Resizing and adaptive local thresholding are used. The next step is background identification based on the RAM-based neural network. The average body head is calculated from the infrared data and distinguishable results are achieved in less crowded areas. The 16 by 16 pixel low resolution is regarded as adequate. A practical and accurate method to count individuals in a specific area can be to use a combination of visual and infrared devices. As demonstrated in [116], where authors experimented with a thermal imager in a noninvasive manner, such a system can only be utilized in conjunction with the detection of individuals inside vehicles. The authors tried with closed windows but had unsatisfactory results so they opened automobile windows at various levels to test the accuracy of their suggested technique.

Klauser et al. suggested a time-of-flight-based passenger tracking technique using infrared sensors, image processing, and statistical and probability approaches. The Bayes classification was used by the authors to define a statistical processing-based technique. This method is effective if all objects can be distinguished from one another for at least one frame. As they look for maxima in the height of potential objects, authors are working on a different, entirely algorithmic technique while assuming the same sensors [117].

In [118], Mukherjee et al. suggested a methodology for detection, tracking, and validation to count passengers at the railway station. On the other hand, a top view of a person has few features to work with and little occlusion. A time series of photographs is used as the algorithm’s input, and its output is the number of valid trajectories. The Hough circle is used to identify every individual. In experiments, the Hough circle outperformed the HOG technique.

It is clear that detecting people at different public transport stations or hubs is not an easy task during rush hours or at busy places. To overcome this, many approaches have been investigated.

5. Passenger Discrimination for Airbag Suppression

Airbags can both save and kill. A child may suffer severe injuries or perhaps die in a crash as a result of an airbag that is rapidly inflating. Adults who are not strapped in or who are in an unusual position risk death as well. Additionally, airbag replacement is pricey. To save money on maintenance, an airbag may be disabled if a seat is empty. It is critical to tell whether the front seat is being used by a youngster, an infant in a rear-facing seat, or is empty. A 1996 U.S. Patent by Meister et al. described an innovation for occupancy detection that would stop airbag deployment [119]. System status is indicated to a driver with the possibility to override the system and enable activation of the airbag. In 1997, G. Paula published an article about electronic systems for airbags aimed to reduce injuries and deaths caused by airbags [120] based on ultrasonic, infrared, or piezoelectric sensors. Systems for airbags take into account the state of the seat belt and react accordingly to reduce injuries. The sensor fusion system for sensing the presence, position, and type of an occupant in a passenger seat in vehicles is proposed by Corrado et al. in the United States Patent [121]. The first property to detect is the thermal signature and associated motion, and the second is distance and associated motion detected with an acoustic sensor. Supplementary data, sensor data, and historical references are considered in decision making if the passenger airbag or other types of safety restraint systems will be deployed during a collision.

Jinno et al. proposed occupant sensing while exploiting the perturbation of electric fields. Inside the seat, four electrodes are installed and provide a very weak electric field. The disruption brought on by passengers is detected as matrix data, and the system either enables or disables airbags based on its assessment of the scenario. Ferrous materials may cause the system’s decisions to become muddled [122].

Krumm and Kirk presented a visual occupant identification system for airbag deployment in their paper [123] from 1998. Based on a single monochrome video camera, the first suggested method has a 99.5% correct classification rate using the closest neighbour classifier. With computed stereo disparity, a correct classification rate of 95.1% was attained in the second experiment using two monochrome video cameras. The second method might be more advantageous because it is less sensitive to ambient light. An invasive system with cameras installed inside a vehicle can be used to classify the seat as occupied, containing a child seat, or empty. Range data could also be used to estimate the position of the passenger and vary the inflation rate of the airbag to reduce injuries. Similarly, a stereo system was also proposed in papers [3,4,5,6] in Section 2.

A US patent by Saito et al. aimed to distinguish whether a passenger seat is occupied by an adult, infant, or child seat in the RFIS (Rear Facing Infant Seat) or FFCS (Forward Facing Car Seat) pattern [124]. By generating a weak electric field with an oscillating circuit between electrodes, a corresponding response is detected and the seating pattern is distinguished. Based on output signals, an airbag device for inflating an airbag in case of a collision is presented.

Support vector machines perform admirably when recognizing faces. In their paper [125], Reyna et al. provide modified and standard SVM for head detection inside cars. The authors suggested employing a monocular camera to show where the passenger’s head is in relation to the vehicle. The 144 photos in the image database, which has a resolution of 768 × 576 pixels, were taken from Siemens. The straightforward finding is that heads are less recognizable than faces and that the proportion of false alarms has fallen to almost a third and remains unchanged.

Lu et al. described a new generation of the BMW child seat occupant system consisting of two subsystems—occupant classification with a field detection system with four capacitive plates [126]. Occupant classification is achieved with a force-sensitive sensor array measuring a pressure profile. The system can automatically detect a universal child seat and the system is hidden in the seat. A force-sensitive resistor sensor array and its electronics classify the profile based on pre-programmed characteristics. The SBE2 algorithm combines both subsystems and determines the final classification.

Farmer and Jain introduced a vision-based occupant classification system with a greyscale camera and a digital signal processor to distinguish four classes—rear-facing infant seat, child, adult, and an empty seat. They achieved a classification accuracy of ∼95% [127]. A 400 × 320 px camera with IR illumination and standard IR filter is used to supplement in dark conditions. Edge-based features are used in images with reasonable lighting and silhouette features in low-light scenarios. Three years later, the authors published another paper, where they achieved ∼98% accuracy with the ability to detect dangerous proximity to the airbag within 7 ms [128]. The system uses a wrapper-based approach and the classification result is a priori for segmentation.

Fritzsche et al. proposed a new sensor approach based on the TOF (time-of-flight) principle [129]. A prototype camera was mounted in the overhead console to survey the passenger compartment. Additional active modulated IR illumination is used. The output data were recorded with this 25 × 64 pixel camera. Occupant classification in the horizontal direction is tolerable but in the vertical insufficient with this resolution. Results might improve with the new camera design with 52 × 50 pixels.

Using raw reflectance and stereo disparities, Krotsky and Trivedi proposed a vision-based method of estimating the size, posture, and pose of the occupant [130]. The Digiclops camera is attached to the driver’s side roof rack and images are captured at 320 × 240 pixels at 15 fps. In the next paper [131], the authors propose a thermal long-wavelength infrared video-based real-time vision system. The feasibility of invasive thermal long-wavelength infrared, stereo, and multicamera video-based vision systems to deploy normally, deploy with limiter power, or suppress an airbag is good. Trivedi et al. investigated computer-vision technology for safer automobiles in paper [132]. The aforementioned application estimates the driver’s head pose using elliptical fitting to generate the driver’s vision, which takes up the majority of the computation time. They also take into account technologies for monitoring roads and avoiding collisions with objects or persons.

They came to the conclusion that there are no reliable and affordable driver posture systems that work in actual driving situations. The study is separated into non-vision sensor-based systems and vision-based systems for posture monitoring. The majority of false positives and missed detections are typically produced by monitoring systems that rely on the analysis of partial body parts. A worldwide system must also deal with passengers that have diverse anthropometrics, skin tones, clothing, weights, etc.

Stereo-based systems typically have a poor level of resolution or need a large training batch of data to function well. These flaws are minimized by using data fitting established body models. This approach provides more thorough occupant information without requiring training. The maximal varieties variance threshold splitting method is used to binarize grayscale images. Overall, 96.5% of classes are correctly classified [133].

A night vision camera connected to the Raspberry Pi board is discussed in the paper [134]. The authors used Haar Cascades as a face detection algorithm based on the easiest feature extraction with high accuracy and less computation time compared to other machine learning algorithms. The proposed system achieves great accuracy in hatchback cars. When faces are exposed to direct bright light the system works poorly. The system is used for classifying a person as a child or adult to avoid the deployment of airbags. The calculation is performed when the car speeds from 0 km/h to 20 km/h.

The suppression of the airbag can save lives and a huge amount of money if real-time systems are installed. These systems might be modified in order to count passengers and help with the growth of road transportation.

6. Mathematical Models

6.1. You Only Look Once

In this section, we explain the theoretical framework of the favourite occupancy detection method—the YOLO convolutional network. YOLO is an object detection algorithm. It is based on features learned by a deep convolutional network. YOLO is considered a fully convolutional network (FCN) with 75 convolutional layers with skip connections and upsampling layers. This architecture is considered extremely fast. Prediction is performed using a convolutional layer based on 1 × 1 convolutions, for detecting small objects, the final convolution is 255 × 1 × 1 with batch size 52, 52, 255, for medium objects batch size is 26, 26, 255, for big objects batch size is 13, 13, 255. Output is a feature map, the size of the prediction map is exactly the size of the feature map before. This prediction map is interpreted in the way that each cell can predict a fixed number of bounding boxes. YOLO can achieve fast frame rates [116,135].

The network has $(B*(5+C\left)\right)$ entries in the feature map, where B is the number of bounding boxes each cell can predict, and C represents class confidence for each bounding box. Each bounding box can have $5+C$ attributes that describe dimensions $(w, h)$, centre coordinates $(x, y)$, objectness score $p_0$, and confidence C.

Network output can be presented with the following formulas [116,135]:

$$\begin{array}{c}\hfill b_x=\sigma \left(t_x\right)+c_x\\ \hfill b_y=\sigma \left(t_y\right)+c_y\\ \hfill b_w=p_{w}exp\left(t_w\right)\\ \hfill b_h=p_{h}exp\left(t_h\right),\end{array}$$

(1)

where $b_x$, $b_y$, $b_w$, $b_h$ are the x, y centre coordinates, width and height of prediction, $t_x$, $t_w$, $t_h$ is network output, $c_x$, $c_y$ are top left coordinates of the grid, $p_w$, $p_h$ are anchors dimensions for the box, $\sigma$ represents a sigmoid function.

The loss function indicates the performance of the model. YOLO loss function is defined as equation [116,135]:

$$\begin{array}{c}\hfill {\lambda }_{coord}\sum _{i=0}^{S^2}\sum _{j=0}^B{1}_{ij}^{obj}[{(x_i-\widehat{x_i})}^2+{(y_i-\widehat{y_i})}^2]\\ \hfill +{\lambda }_{coord}\sum _{i=0}^{S^2}\sum _{j=0}^B{1}_{ij}^{obj}[{(\sqrt{w_i}-\sqrt{\widehat{w_i}})}^2+{(\sqrt{h_i}-\sqrt{\widehat{h_i}})}^2]\\ \hfill +\sum _{i=0}^{S^2}\sum _{j=0}^B{1}_{ij}^{obj}{(C_i-\widehat{C_i})}^2+{\lambda }_{noobj}\sum _{i=0}^{S^2}\sum _{j=0}^B{1}_{ij}^{noobj}{(C_i-\widehat{C_i})}^2\\ \hfill +\sum _{i=0}^{S^2}{1}_{ij}^{obj}\sum _{c\in classes}{(p_i\left(c\right)-\widehat{p_i}\left(c\right))}^2,\end{array}$$

(2)

where $C_i$ is the Objectness-confidence score of whether there is an object in the picture or not, $w_i, h_i$ is the width and height of the anchor box, $p_i\left(c\right)$ is classification loss, $x_i, y_i$ is the location of the centre of the anchor box, $1_{ij}^{noobj}$ and $1_{ij}^{obj}$ are masks for each cell that predicts an object in a cell it there is or is not, $\lambda s$ are constants, ${\lambda }_{coord}$ represents the weight of the coordinate error, ${\lambda }_{noobj}$ represents scalar to weight loss in confidence in each bounding box. ${\sum }_{i=0}^{S^2}$ represents a part where we compute losses for each of 13 × 13 cells, ${\sum }_{j=0}^B$ represents a part where we compute losses for each anchor box. $\widehat{C_i}$ represents the confidence score of the j th bounding box in the grid. There are three boxes across three different scales.

6.2. Cascade Classifiers

A successful object recognition approach based on Haar feature-based cascades was proposed by Viola and Jones in their 2001 publication “Rapid Object Detection using a Boosted Cascade of Simple Features” [136]. It is described as a machine learning method. In order to train a cascade function, numerous positive and negative images are required. Haar features are used to extract features from photographs. They are shown below in Figure 2. At the time of publishing their paper, the proposed system was approximately 15 times faster than any previous approach. Many previous papers utilize insights from Haar features. A short list of examples consists of [13,108,134].

Haar features are good at predicting edges and lines. These aspects prove it is really effective in face detection. This algorithm is able to detect objects with clear edges and lines. If we cover parts of the object, a Haar-based classifier might fail. This detector is great for detecting faces in pictures from the front. A large number of false positives is reported in not perfectly direct images. The feature-based system also operates much faster than a pixel-based system.

Rectangle features are computed with the integral image. The integral image at location x, y contains the sum of the pixels above and to the left of x, y.

$$\begin{array}{c}\hfill ii(x, y)=\sum _{\acute{x}\le x,\acute{y}\le y}i\left(\acute{x}, \acute{y}\right),\end{array}$$

(3)

where $ii(x, y)$ is the integral image and $i(x, y)$ is the original. This integral image can be computed from an image using a few operations per pixel. Any Haar-like features can be computed at any scale or location in constant time. To select a small number of important features, AdaBoost is used. AdaBoost provides an effective learning algorithm and strong bounds on generalization performance [136]. The final classifier is a weighted sum of weak classifiers. The weak classifier is not able to classify the image but together creates a strong classifier. The remaining features are grouped into a different stage of classifiers—Cascade of classifiers. After passing all stages, the object is detected.

7. Discussion

Since Billheimer et al. [1] proposed their first attempt to count the number of passengers in a moving vehicle, progress in road transport and computers has been rapid. As the reliability, robustness, and computing power increase, we can expect that passenger estimation systems will be increasingly used in a wide variety of applications including transport systems, surveillance systems, and military or autonomous vehicles. A total of 146 papers were analysed and their weak and strong qualities are discussed below.

7.1. Paper Selection Analysis

In order to systematically identify relevant published papers in this domain, literature research was performed from 1990 up to and including 2021. To acquire as many papers as possible, Google Scholar, Web of Science, and SAE International were searched. The following keywords were chosen: occupancy detection, passenger detection, people detection, and driver detection. Existing patents were also included. This review covers the field of transport, cars, safety, and sensors. A total of 146 papers were analysed in this review alongside advantages and possible obstacles in future development. A graph showing the number of analysed articles per year is shown in Figure 3.

As we can see, interest in this area has increased every year since 1990. This review provides essential information on similar research in passenger estimation systems. It is divided into invasive counting of people, noninvasive estimation of people, and passenger detection in transport hubs due to the nature of each mentioned area of application. We believe that this division is necessary due to different environmental and measurement conditions. A graph showing the occurrence of keywords is shown in Figure 4.

Out of 146 analysed papers, 60 references did not properly list keywords or any index terms for our analysis of the occurrence of keywords. We undertook this study to examine and analyse older, new, and state-of-the-art methods for passenger occupancy detection.

I indicates the invasive measuring method, N indicates noninvasive measuring methods. Based on

Table 1. An overview of different approaches in existing papers.

Method	Advantages	Drawbacks	Reference	Invasivity
NIR systems	No driver distraction. Sees through window tinting. Suitable for both I and N applications.	Harder detection and less robust. A system might get expensive. Creates false positives with pets, luggage, etc.	[2,13,15,63,66,69,70,71,72,80],	I + N
Vision systems	Good execution speed, reliability, and good results during daytime.	Low tolerance to light conditions. A system might become expensive and is influenced by the weather when used noninvasively. Sensitive to colours.	[1,16,60,68,103,115,125,128,133,137],	I + N
Stereo cameras	Insensitive to illumination, and suited for irregular environments.	Trade-off between the reliability and the computation time, incorrect detection during various passenger poses.	[3,4,5,6,7,96,97,101,123,130,131],	I
Thermal images	Can detect also back seating passengers when used invasively.	Not suitable for outdoor applications, problems with similar surrounding temperatures.	[18,116,131],	I *
Radar	Requires less power than camera sensors. Emits fewer EM waves than WiFi. Suitable for low-light conditions. Contactless solution.	Requires additional communication channels for sending results.	[38,40,41,43,44,45,47],	I
WiFi	Does not require interactions with passengers	Requires mobile phones with enabled WiFi. Usually overestimates with many surrounding vehicles.	[48,54,55]	I + N
Seat belts	System is already installed in vehicles.	Requires additional communication channel and access to car electronics.	[56]	I
Capacitive systems	Reliable and fast, robust to external interference.	Requires additional communication channel and has problems when passengers are grounded.	[24,25,26,27,28,29,30,31]	I
PIR	Does not count irrelevant objects, small dimensions.	Requires additional communication channel, is not consistent and has varying accuracy.	[52,95]	I
TOF	Requires less sensors, does not trigger false positives with rucksacks or clothes.	Insufficient results with low resolution, tendency to yield less counting results.	[56,117,129]	I
Electric fields	Low-cost system with hidden electrodes. Is not affected by heat, light, dust, humidity.	Covering electrodes disables the whole system.	[49,122]	I
Floor pressure sensing	Can easily distinguish direction of movement and is relatively cheap.	Suitable only for public transport with difficult sensor placement.	[21,22]	I

* indicates the feasibility of invasive methods but noninvasive were proven unsuccessful. Tests on thermal imaging systems have shown that glass is ineffective in transmitting certain electromagnetic waves [116].

, it is clear that invasive systems require new and reliable channels to transfer diagnostic data to road infrastructure. Without this, more precise invasive systems do not provide fast and very precise data for occupancy monitoring and the only feasible and less precise way to detect occupancy of vehicles remains the noninvasive manner. Moreover, we should not forget about older vehicles, in which the installation of such systems will be very difficult. Finally, we must not forget the infrastructure itself, which could be very financially demanding. Unlike invasive occupancy systems, noninvasive systems provide fast and easy adaptability to various scenarios. Maintenance workers can access sensors and cameras and easily change optics or add or remove pieces of equipment on-site.

7.2. Limitations and Research Challenges

Road transport is an ever-growing field. To improve the quality and safety of transportation, researchers, governments, and companies seek newer and more advanced systems to improve the sustainability of traffic systems, reduce travel times, and greenhouse gas emissions, improve quality of life, response times of rescue services, etc. [138,139,140]. Systems for the detection of transport of dangerous goods, license plate recognition, type of vehicles, stationary vehicle inside a tunnel, etc., are becoming mandatory for new road structures. Distinguishing between hybrid, combustion engine, end EV is also a necessary piece of information when a fire occurs [141].Yet with the rise of such systems, the field of security should also grow [142]. It is important not only to keep the system running but ensure its reliability, integrity, resiliency, security, and data privacy. The current trend in big cities is not to expand the transport infrastructure, but to make existing infrastructure efficient by means of telematics systems leading to changes in traffic scenarios in cities [143].

For noninvasive passenger detection, the main issue is ever-changing automotive glass. New changes in the automotive glass to reduce the solar heat load on the vehicles are new challenges for noninvasive vision, NIR, and thermal systems. To reduce the generation of greenhouse gasses and energy usage on cooling, the total solar transmittance allowance for new vehicles must be lower than 40% by 2014 [144,145]. New types of automotive glass are being constantly developed and manufactured. The transmittance of such glass is changing and also, the possibility of tinting the cars does not make the task of detecting passengers through glass easier. Variations in pose and type of passenger present a new disadvantage of vision-based systems. Sleeping passengers are extremely difficult to detect from the outside world and it is difficult to distinguish whether the child seat is occupied or not. Different colours of clothes and skin present another difficulty. Some papers have investigated the influence of such parameters [2,8,62,63,72,146]. Many environmental aspects affect the final accuracy of these systems. Dust, shades, sun gloss, rain, snowfall, fog, quality and cleanliness of windows and windshields, distance to the target, speed of a vehicle, occupant position, and others greatly affect the final accuracy.

The seat belt approach might be promising, but it is not a usable option without modification. All invasive methods—pressure sensing, capacitive, ultrasonic, radar, PIR, TOF, and vision-based systems have to send acquired pieces of information to the outside infrastructure in order to be effective and useful and help in developing more advanced systems. Additionally, the way to communicate a number of passengers to the outside world must be safe and reliable.

All invasive systems might be very useful in public transport applications. To achieve reliable and low-cost systems, many papers investigated various sensor placements—front windshield, front bonnet, overhead consoles, sensors hidden in seats, etc. Capacitive systems and systems based on electric fields provide a cheap solution. They are not affected by varying weather, temperature, or light. The main downside is that they suffer a lot when the conditions are not right. If the passengers ground themselves, these methods do not work properly. PIR sensors seem great for train or bus applications but suffer like all other invasive methods.

An interesting approach worth mentioning is using gas sensors. In the enclosed compartment, the concentration of gases is changing with time and the most suitable gas is CO₂ [56]. Many systems may present faults when children hold luggage or big luggage is obscuring vision for vision systems.

Furthermore, passenger privacy is also a very important aspect of every invasive system. Every car produced after April 2018 is equipped with an eCall (emergency call) system to bring rapid assistance to motorists involved in road hazards in the EU. This system brought new ways of possible eavesdropping and violation of passenger privacy.

WiFi-based solutions are also presented. The proposed method detects and counts passengers without violating privacy. The system shows good results for use in various applications due to its low cost and simplicity. There are still problems with carrying a mobile phone with WiFi turned on. Not everyone has a phone, and there are people with multiple phones—private and work.

Since 1990, methods have shifted towards a new trend—machine learning and neural networks. Many recent papers propose advanced and improved neural networks for passenger detection tasks [18,113]. They provide great results and robustness for passenger counting. In public transport, crowded places or rush hours present a difficult situation [94,97,99,118]. All final requirements depend on the application and desired field of deployment. Favourite infrared sensors present a number of advantages and are widely used in several systems. The advantages are reduced size, cost, and reliability. However, in crowded situations, their high sensitivity to noise, variations in temperature, and dust and smoke make them unreliable. They also cannot distinguish between one passenger and a group of passengers. Many researchers advise a vision-based system for this task in a bus or public station.

The most promising band for detecting passengers noninvasively appears to be the NIR region of the electromagnetic spectrum from 750 nm to 2500 nm due to the reasonable price of technical equipment. This region of the electromagnetic spectrum is often chosen by researchers involved in noninvasive passenger detection. See

Table 2. An overview of different noninvasive approaches in analysed papers.

Reference	Image Type	Camera Placement and Orientation	Windows Tinting	Use	Notes
[78]	NIR > 750 nm	Only windshield	Not mentioned.	HOV/HOT	Speeds up to 130 km/h.
[1]	Monochrome	Windshield + side images.	They theoretically describe the limitations of tinting in the 1990s.	HOV/HOT and violations	Tinting causes uncertainty in estimation.
[2,62,63]	NIR 1.4-1.7 μm	Windshield + side images.	Graphs of the transmission of wavelengths depending on the used glass, tinting or purity is proposed.	HOV/HOT	An overview of detection methods.
[86]	NIR > 750 nm	Various	Not mentioned.	HOV/HOT	Comparison of different CNN architectures.
[65]	Visible spectrum	Only windshield	Not mentioned.	Violations	Comparison of different CNN architectures.
[77]	NIR > 750 nm	Only windshield	Not mentioned.	HOV/HOT	Binary classification
[64]	NIR > 750 nm	Two side views.	Using NIR to overcome tinting.	HOV/HOT	Detection of 1+, 2+ and 3+.
[73]	Visible spectrum	Only windshield	Not mentioned.	HOV/HOT	Synthesis of face and seat belt detection.
[19]	Visible spectrum	Windshield and rear glass.	Not mentioned.	Emergency situations	Camera placed on the vehicle.
[20]	NIR > 750 nm	Windshield + side images.	They mention the tinting effect, but do not discuss it further.	HOV/HOT and violations	Violations in HOV 3+ scenario.
[75]	Visible spectrum	Only windshield.	Not mentioned.	HOV/HOT	Seat edge detection.
[84]	Multi-band IR	Only windshield.	Not mentioned.	HOV/HOT	Detection in HOV 2+ scenario.
[9]	NIR > 750 nm	Windshield + side images.	SCW (Solar controlled window)	HOV/HOT
[70]	NIR 850 nm	Only windshield.	Experiments with tinting. Required adaptive lighting.	HOV/HOT	Unstable image quality.
[90,91]	Three-camera setup	Only windshield.	Efforts to improve systems with NIR.	Biometric recognition	HDR fusion of images.
[81]	NIR > 750 nm	Windshield + side images.	Not mentioned.	HOV/HOT	Passenger and violation detection.

. The best way seems to design a neural network trained with a special dataset of NIR images, which would be able to detect passengers, children, and objects. Together with the created algorithms, such a system could be applied in various areas of the real world.

7.3. Effects of Window Tinting on Passenger Detection

The unexplored region in noninvasive passenger detection systems is the effects of different window tinting. Various manufacturers and car producers produce different levels of tinting. If very dark window tint is used, it might lead to a reduction in detection accuracy.

This review was created as a part of ongoing research. Figure 5 shows different transmittances of LLumar car window tints. For future research, different camera filters, mounting positions, and various window tinting will be evaluated.

8. Conclusions

In this paper, passenger detection and occupancy estimation systems, solutions, applications, and shortcomings are explained and the studies on them are discussed. Moreover, to address the fundamental issue of invasive systems, considerable attention is paid to noninvasive systems. Accordingly, comparing the present invasive and noninvasive solutions provide more understanding in this field. Both systems have strong and weak points. Many issues which cannot be addressed by one system can be addressed by the other and vice versa. Passenger estimation systems have assisted the sector in many ways, but they can be greatly improved. Therefore, for future research, improving NIR systems and the development of advanced designs are recommended to overcome various window tinting and solar glass. The literature review presented in this paper provides a wide analysis of various methods. Finally, passenger estimation systems are unavoidable in the future and this technology is not limited only to the public sector, but also to military or surveillance systems where they can also contribute significantly.