Methodology for Automatically Detecting Pan–Tilt–Zoom CCTV Camera Drift in Advanced Traffic Management System Networks

Abstract

Many transportation agencies have deployed pan–tilt–zoom (PTZ) closed-circuit television (CCTV) cameras to monitor roadway conditions and coordinate traffic incident management (TIM), particularly in urbanized areas. Pre-programmed “presets” provide the ability to rapidly position a camera on regions of highways. However, camera views occasionally develop systematic deviations from their original presets due to a variety of factors, such as camera change-outs, routine maintenance, drive belt slippage, bracket movements, and even minor vehicle crashes into the camera support structures. Scheduled manual calibration is one way to systematically eliminate these positioning problems, but it is more desirable to develop automated techniques to detect and alert agencies of potential drift. This is particularly useful for agencies with large camera networks, often numbering in the 1000’s. This paper proposes a methodology using the mean Structured Similarity Index Measure (SSIM) to compare images for a current observation to a stored original image with identical PTZ coordinates. Analyzing images using the mean SSIM generates a single value, which is then aggregated every week to generate potential drift alerts. This methodology was applied to 2200 images from 49 cameras over a 12-month period, which generated less than 30 alerts that required manual validation to determine the confirmed drift detection rate. Approximately 57% of those alerts were confirmed to be camera drift. This paper concludes with the limitations of the methodology and future research opportunities to possibly increase alert accuracy in an active deployment.

1. Introduction

State and local agencies have deployed large pan–tilt–zoom (PTZ) closed-circuit television (CCTV) networks to monitor and manage roadway conditions. Modern advanced traffic management systems (ATMSs) are increasingly becoming equipped with enhanced repositioning and monitoring features [1]. All enhanced repositioning and monitoring features require calibrating and georeferencing a camera’s internal images to the physical world. Through the standard use and maintenance of these cameras, it is likely that cameras will drift from a calibrated position over time.

Limited research on the topic and the wide variety of calibration methodologies with various recalibration processes have motivated this research. Drift analysis using LiDAR georeferenced and calibrated cameras has previously been performed but requires driving the corridor with the proper equipment and significant post-processing before implementing any alerting methodology [2]. Additionally, cameras need to be thoroughly calibrated to derive several parameters describing internal and external orientations for these self-recalibration processes. Thus, a drift detection system that can be applied without much initial setup is desired.

This paper proposes a methodology using the mean Structured Similarity Index Measure (mSSIM) to compare a current “comparison image” to an “original image” previously collected. This novel approach to automating PTZ CCTV camera drift detection saves several routine person-hours of manually inspecting cameras. Often, it is the case that when a camera has recently been inspected and deemed fit for operation, shortly after it “drifts”, leaving enhanced features inoperable. Therefore, the proposed automated methodology is crucial to reduce the time needed to find camera drift, alert maintenance personnel, and increase the operable time of enhanced repositioning features of these tools.

The structure of this paper is as follows: a detailed literature review is presented highlighting the past work and research gaps, followed by the objectives of this research and methodologies for assessing images from a single camera, as well as multiple cameras in a network. This is followed by the results and a brief discussion of the findings. This paper concludes with a concise summary of the research and potential future research opportunities.

2. Literature Review

2.1. Early Cameras in Traffic Management

The use of pan–tilt–zoom (PTZ) closed-circuit television (CCTV) cameras in advanced traffic management systems (ATMSs) has its roots in the early 1970s [3,4]. Initially, these systems were fixed-mounted systems routed to Traffic Management Center (TMC) operators located in a central office [3,5]. From these fixed-mounted systems, TMCs shifted to the installation of PTZ cameras in the late 1990s to increase the robustness of their operations. These new PTZ cameras allowed operators to monitor an entire area overhead and facilitate communications in real time [6,7,8,9].

2.2. Developments in Image Analysis

With the newly installed PTZ cameras, all parties (private and public) quickly began experimenting with and developing systems to offer enhanced features. These efforts were a direct result of multi-disciplinary efforts. The fields of computer vision and machine learning concurrently developed several methodologies often incorporated under the topic of “visual information analysis” [10]. Several notable developments with interest in transportation and traffic management are image-difference detection and object detection [11,12,13,14,15]. One notable contribution came from Kodratoff and Moscatelli, who pushed for machine learning in object detection and scene analysis [16]. In the field of image comparison, the early 2000s hosted several foundational publications. Wang developed his “Structural Similarity Index” (SSIM) algorithm [17]. This was later developed into several specialized applications [18,19].

2.3. Preliminary Applications in Transportation

With several tools at their disposal, transportation professionals set out to build principal Intelligent Transportation Systems (ITSs) and Automated Traffic Management Systems (ATMSs). Mak and Fan were among the first to develop an incident detection algorithm using two adjacent video-based prevalence stations [20]. Schoepflin (and others) worked to create algorithms for estimating mean vehicle speed [21] and other traffic characteristics. This trend was a significant focus in early developments, that is, creating video analytic methodologies to characterize traffic flows [22,23].

2.4. The Need for Camera Calibration

Apart from these video analytics, the parallel development branch was focused on assisting those TMC operators by automating specific job tasks. This branch focuses on geographic calibration, or “geo-calibration”, which aligns a camera’s internal coordinates with the environment around it, enabling automated movements derived solely from real-world coordinates. Galego [24] was one of the first publications on this topic, proposing a geometric approach for calibrating traffic cameras. Shortly after, Dadashi proposed a semi-automated approach to geo-calibration [25]. Recent works implementing these foundations have developed fully autonomous movement methodologies, which allow for cameras to reposition onto positional data related to traffic incidents [1,2,26]. Though these methodologies provide high value for traffic management, the source of success is always in the geo-calibration of these cameras; as such, a method of detecting when deviation or drift from a calibrated state takes place is highly desirable.

2.5. Camera Drift Detection

This apparent need has been a topic of research since the implementation of geo-calibration in PTZ traffic cameras. Lisanti proposed a “continuous localization” technique [27], a landmark tracking algorithm utilizing SURF key-point detection [28]. Other approaches utilize multiple cameras to offer the “recalibration” of drifted cameras [29,30]. While these detect “drift”, initial calibration may be cumbersome on larger-scale systems continually under preventative maintenance regimes and upgrades. Others, motivated to simplify the calibration processes that are more suited for more extensive networks, developed procedures that are less capable of automatically adapting to drift [31]. By accepting a simpler model, practitioners take the responsibility of manually locating and correcting drift events.

One of the first published approaches for these scenarios utilizes wide-angle feature point (FP) matching followed by zoom-in FP matching [32]. This approach requires several well-defined feature points, which may not be applicable to many scenarios due to changing traffic conditions. Other approaches suggest the use of laser scanners to generate 3D point clouds to accurately calculate the geometric relations of cameras and their environments in combination with intrinsic and extrinsic parameters [2,33]. While these offer more accurate calibration and drift detection options, for physically large networks (e.g., statewide systems), collecting the necessary data is costly and time-consuming, often only being performed every few years. This lack of updated data can completely exclude its use for newly installed cameras not previously installed in the network at the time of the data collection [2].

In the current literature, these two approaches define the main approaches to camera calibration deviation. Feature matching can be performed by utilizing one or more cameras for continuous localization. These require intensive calibration processes that are not easily scaled to large camera networks. Laser scanning offers a fast and relatively easy calibration process with accurate ground truth to compare to for detecting drift. These methods require significant capital and time investments to collect data for physically large systems. These approaches leave a gap in research. That is where this research proposes a simplified approach based on the comparison of two images, generating a similarity score and generating alerts for where to focus manual maintenance efforts. Although this proposition appears straightforward, several vital decisions must be made. What image comparison algorithm should be used? What similarity index best captures the results of the algorithm? What is the criterion for alerting? What is the criterion for collecting images to compare? These are just a few examples of these decisions.

2.6. Approaches for Image Comparison

2.6.1. SSIM

The first identified approach is the Structural Similarity Index Measure (SSIM) [17]. This approach offers a statistical measure of the luminance, variance, and correlation of image structure [18]. Often, the mean value SSIM (called mean SSIM or mSSIM) is taken to quantify the similarity of the image. A value of 1 is an identical image (based on the structural measure previously mentioned), and 0 is two images that share no commonality [18].

Several adaptations of the SSIM have also been developed to handle several different use cases: Wang’s Multiscale SSIM (MS-SSIM) [34] varies the viewing conditions (e.g., contrast); Sampat et al.’s Complex Wavelet SSIM (CW-SSIM) [19] has been shown to handle minute variations in the image better than the original but is not necessarily as relevant to the stated problem; 3D-SSIM [35] is used for videos; and spherical SSIM (S-SSIM) [36] is used for spherical projections. This family of algorithms offers an easy-to-use algorithm that generates a single, quantified index for comparison.

2.6.2. SIFT

SIFT (Scale Invariant Feature Transformation), first published by Lowe [12], utilizes distinctive image features. SIFT is able to handle feature descriptor matching between several different views. This algorithm offers increased flexibility when combined with geomatic derivations to determine orientation from a separate view (quantifying the approximate drifted amount) [37]. This algorithm offers calculable components of orientation change, which, when compared to a known orientation, could allow for quantifiable drift. However, to implement this, key features need to be identified, requiring an added layer of “calibration.”

2.6.3. SURF

Partially developed from SIFT, SURF (Speeded-Up Robust Features) was proposed by Bay et al. [28] and performs similar actions as SIFT (object recognition and matching), but in many applications is several times faster. Considering its likeness to SIFT, several of its drawbacks are related to the identification of critical features in a rapidly changing environment.

Several studies have proposed various techniques to identify camera drift. However, most of them require additional calibration, which is time-consuming and cumbersome. Additionally, some of the methods involving lasers require extensive data collection and are computationally expensive. This research proposes a systematic and simple methodology that leverages the mSSIM approach to identify potential drift among a network of agency-maintained PTZ cameras without the need for any external recalibration procedures.

3. Study Objectives

The objective of this paper is to develop a framework for assessing if a PTZ CCTV camera has drifted so that it should be recalibrated. This objective is further partitioned into seven sub-objectives:

• Define PTZ CCTV drift using a set of images and recorded PTZ values from one camera;
• Apply the mean SSIM algorithm and determine if it is feasible to detect drift with collected samples;
• Expand and scale the methodology to 49 cameras along a 53-mile route (on Interstate 465) using one sample for each day of the week (seven per week), and show the systematic application of the methodology;
• Reduce the number of samples from seven per week to two per week (Sunday and Saturday) to show that the methodology is still valid at lower sample sizes (important when considering a real-time deployment);
• Establish criteria to classify a potential drift alert;
• Establish analysis criteria to detect and eliminate false positives;
• Analyze generated alerts to establish the drift detection rate recommendations.

4. Motivation

Figure is called “camera drift” or “drift.” Drift is defined by the change of a frame with respect to constant features within the image without a change in PTZ values. The drift event in Figure 1 is easily identified by visual inspection, but to analyze a network of 100’s of cameras on a weekly basis would require many person-hours. Therefore, it is desired to have an automated algorithm to detect and alert to drift events.

5. Methods

Figure 2a is a transparent overlay of Figure 1b onto Figure 1a, where Figure 1b is red-shifted at 50% opacity, and Figure 1a is blue-shifted at 100% opacity. Figure 2a shows many areas of common registration, including the following callouts: I, a mile reference post; iii, the year and month aspects of the time overlay; and v, the lane markings. Objects different between images appear as a “ghost” feature, being see-through. Callout ii shows a tractor-trailer in Figure 1b, but not in Figure 1a. Callout iv shows the day aspect of the timestamp overlay, which is less defined due to conflicting numbers being overlayed. Callout vi shows the shadow from a tree on the left of the image.

Figure 2b is the SSIM difference image produced from the algorithm described in [17]. Common pixels approach a value of 1, appearing lighter. Different pixels approach a value of 0, appearing darker. Callouts I, iii, and v appear as lighter values due to their fixed nature between images. In callout ii, the areas of difference and agreeance are caused by the structure of luminance and contrast, distinguishing objects themselves rather than their general relationships. The tractor-trailer area contains areas resembling the roadway in Figure 1a, causing the mixed similarity. Callout iv shows the day aspect of the timestamp overlay as being different. Callout iv demonstrates that the SSIM algorithm works best with large luminance and contrast differences and defined edges [18]. Finally, callout vi shows mixed results due to the random nature of the shadow. Taking the mean SSIM difference frame, a value of 0.49752 is observed, which is important for determining if drift detection is possible using the SSIM algorithm. This approach ignores the decline the mean SSIM due to passing vehicles and assumes the mean SSIM will vary enough to detect when drift occurs.

Figure 3 performs the same analysis as Figure 2, except comparing Figure 1b-c. In both subfigures, callout I is the mile reference post in the original image, callout ii is the drifted position of the mile reference post in the comparison image, and callout iii is the path of the drift. Drift is apparent through Figure 3 and has a corresponding mean SSIM value of 0.25186. The drop in the SSIM is assumed to be directly caused by a drift in the camera position.

The process applied in Figure 2b and Figure 3b establishes the methodology for drift detection. Next, this methodology is applied to 12 months of historical observations to determine the viability of the methodology for camera 52.

The methodology is summarized by the following numbered list:

• Capture the “original image”.
• Capture the “comparison image”.
• Compare the images using the mSSIM algorithm.
• Store the mSSIM value for the two compared images.
• Compute the 2-day average mSSIM.
• Check if the 2-day average mSSIM is greater than the alerting threshold:

  a.

  If greater than or equal to the alerting threshold, do not send an alert;

  b.

  If less than the alerting threshold, send an alert.

• Repeat 2–6 during the operation of the camera at set intervals.

5.1. Sample Selection Criteria

To apply the developed methodology temporarily, the collected data need to be filtered to common times to limit influences from differential light intensity throughout the day. Data were collected from 1 April 2023 to 1 April 2024 at 5 min intervals and limited by the following:

• Images whose times occurred at 16:00 UTC ± 10 min, best ensuring constant overhead lighting;
• Images that have a common PTZ value with the “original image,” defined in the next section.

5.2. Original Image Selection Criteria

A constant comparison image must be selected, called the “original image.” The original image is selected by meeting the following:

• Images whose times occur at 16:00 UTC ± 1 h;
• Images that have the most common observed combination of PTZ values;
• Images that occur within the first week of August 2023.

The first week of August 2023 was selected, as images then were likely to be on clear, sunny days for the study areas. This helps mitigate the effects of weather on SSIM scores, covered in later sections. The original image does not have to be within the data being compared to consider more possible data if the first criterion did not yield an original image.

5.3. Single Camera

Figure 4a plots the mean SSIM versus time for camera 52, shown in Figure 1. Callout I in Figure 4a identifies the approximate date drift was observed in camera 52. Callout ii is the data point closest to the original, having the greatest mean SSIM. Callout iii corresponds to Figure 1a, callout iv to Figure 1b, and callout v to Figure 1c. Callouts iii and iv occur before the drift and have a higher SSIM score than after drift, both greater than 0.3 mean SSIM. In fact, no observations are less than 0.3 mean SSIM until callout I, from which no value exceeds 0.3 mean SSIM. This shows the drift event being detected for camera 52 at the confirmed time and suggests a value of 0.3 mean SSIM as an early suitable threshold for detecting drift.

Data from Figure 4a are aggregated by the mean observed value for each week, Sunday to Saturday, and plotted in Figure 4b. Callouts between both figures are identical, with the addition of a border for weeks containing points from callouts in Figure 4a. Figure 4b reports the timing of the drift event from Figure 4a as slightly delayed due to the aggregation process.

5.4. Multiple Cameras

This methodology was applied to 49 cameras along Interstate 465, a beltway encircling Indianapolis, Indiana (IN), in the United States of America. Each camera adheres to the selection criterion listed in the Sample Selection Criteria and Original Image Selection Criteria sections of this manuscript. The results are depicted as a heat map for the average mean SSIM per week plotted in Figure 5. Callout I is the drift event of camera 52, the same as Figure 4b. Callout ii is the week in which originals were selected, having a higher aggregated mean SSIM value. Callout iii is a sample space where data were not collected due to inaccessibility, or no data met the selection criteria. Figure 5 uses 1 daily sample aggregated per week, resulting in nearly full coverage over consecutive weeks. However, further analysis is performed to check if the reduction in samples can produce similar results. This will also reduce the data collection efforts and associated computational complexity required for processing the images.

To minimize samples, an additional criterion that samples are taken on either Sunday or Saturday is added, as shown in Figure 6. For comparison, Figure 5, which uses 7 samples per week (1 for each day), has a total sample size of 7934 SSIM values aggregated into 1893 weeks. Out of these, 399 aggregated weeks had an average mean SSIM value of 0.3 or less; Figure 6, which only uses two samples (Sunday and Saturday), has a total of 2203 samples aggregated into 1528 weeks. Out of these, 347 aggregated weeks had an average mean SSIM value of 0.3 or less. Limiting the selection criteria resulted in approximately 28% of original samples, 81% of original aggregated weeks, and 87% of aggregated weeks with an average mean SSIM value of 0.3 or less. It is assumed that 2 weekly samples using the updated criteria are sufficient to detect drift when analyzing the historical data.

Callout I in Figure 7 (same data as Figure 6) represents camera 52. Callouts ii, iii, and iv are additional alerts discussed in later figures for cameras 48, 142, and 7, respectively. Callouts v, vi, vii, and viii are samples from camera 43, which had no alert over the observation period and serves as an example of no drift.

Those points with cyan diamonds are generated with the following algorithm, representing the automated alerting algorithm:

• Extract all aggregated bins with an average mean SSIM value of 0.3 or less;
• Group extracted bins by camera number;
• Select the first occurrence in bins with at least two consecutive alerted bins as a new drift alert.

Figure 8 shows ideal comparisons corresponding to callouts v, vi, vii, and viii in Figure 7. Each subfigure uses dashed registration aligned to the top-most point of a light post within the image frame to visually compare alignment. Figure 8a is the first image frame occurring on 11 June 2023, followed sequentially by Figure 8b–d corresponding to callouts v, vi, vii, and viii of Figure 7. All subfigures of Figure 8 are identically registered to the same light post. Therefore, no drift occurred during the observation period.

Figure 9 compares the original and drifted image of camera 48 (Figure 7, callout ii). Figure 9a is the original with callout I pointing to the top of a light post in the frame, the registration point. Figure 9b shows the drifted image where callout ii is the drifted position of the registration point, visually different between images. This case is a true positive; an alert was reported with an observable drift event.

Figure 10 compares the original and drifted images for Figure 7, callout iii. Figure 10a is the original with callout I pointing to the top of a bridge support, the registration point. Figure 10b is the drifted image. The registration between both images is identical. Therefore, this case is a false positive; an alert was reported with no observable drift event.

Callout ii of Figure 10 is the difference in weather conditions in the comparison image, being overcast or raining. Callout iii is the change in foliage due to the change in seasons. The callouts allude to limitations of the SSIM algorithm from varied luminance and contrast values between images. Therefore, alerts need to be verified manually to determine the likely cause of an alert.

Figure 11 compares the original drifted image for Figure 7, callout iv. Figure 11a is the original with callout I as the top of a sign, the registration point. Figure 11b is the drifted image. This case is a false positive, likely due to additional roadway and riprap along the side slopes, callout ii. Consideration should be made to update the original when considerable construction occurs.

6. Results

Figure 12 displays all 31 alerts generated from the alerting algorithm. Three alerts were caused by errors in image capture and removed from the analysis. The remaining alerts were analyzed with the visual methods from Figure 8, Figure 9, Figure 10 and Figure 11, and the results are summarized in Figure 13.

Figure 13 summarizes the application of the proposed drift detection and alerting methodology on 49 cameras on I-465 from 1 April 2023 to 1 April 2024. For the 49 cameras, 2203 unique, original samples were collected, meeting the selection criteria in the Sample Selection Criteria section. The 2203 original samples also represent samples that occurred on a Sunday or Saturday (being, at most, 2 samples per week per camera). Those samples were then aggregated by week, taking the average mean SSIM value, resulting in 1528 total aggregated weeks for 49 cameras, plotted in Figure 6. The 1528 aggregated weeks represent 69% of the original samples.

From the 1528 aggregated bins, the number of samples being assessed for drift was further reduced using the alerting criteria established earlier in this study. Applying the alerting criteria to the 1528 aggregated weeks resulted in 28 alerts. Importantly, these alerts represent estimated drift occurrence, not drift itself, needing to be manually verified. These 28 alerts represent only 1% of the original 2203 samples, or 2% of the 1528 aggregated bins, and were generated by the presented methodology automatically.

Finally, each of those 28 alerts was analyzed manually to determine if a drift had occurred. Analysis was performed by adding registration lines to a point in the original image, adding identical lines to the alerted image, and comparing the two. This resulted in the discovery of 16 alerts, resulting in a drift event having occurred, that is, a 57% positive drift alert rate. These 16 positive alerts represent 1% of both the original and aggregated samples, which resulted in drift.

7. Discussion

7.1. Limitations

This methodology uses the original variant of the SSIM algorithm without any additional processing. Thus, the comparisons are susceptible to luminance and contrast differences caused by weather and vegetation changes (Figure 10), roadway changes (Figure 11), or other occlusions (e.g., support cables) resulting in false-positive alerts.

The historical observations limited data collection as a passive operation, collecting data wherever the camera was repositioned during normal use. The selection criteria further limited the usable samples for this analysis. An active data collection approach can anticipate larger sample sizes. Higher true-positive alert rates are expected because of larger sample sizes in cases with weather effects or other discussed factors.

PTZ CCTV cameras are limited in the precision of internal PTZ measurements. The PTZ values for this study were limited to 1° of pan precision, 0.1° of tilt precision, and 1× of zoom (integer precision). Thus, images with reported identical PTZ values are not necessarily in the exact same position, which can result in a false positive for “drift”, resulting in calibration disruption.

7.2. Future Research

Future research on this topic has vast opportunity. This work presented the results from a single image comparison algorithm proofed on historic observations. Future work should analyze the application of other image comparison algorithms and approaches such as CW-SSIM, SURF, or SIFT.

Additionally, a more comprehensive examination of analysis techniques for determining the effectiveness of any given technique is required. The proposed qualitative assessment based on quantitative alerts mirrors the approach an operator may use in determining if a drift event has occurred.

Regarding the effectiveness of a technique, future research should study the environmental effects of each technique, such as weather, traffic conditions, lighting conditions, or natural environmental conditions. Environmental effects were shown to play a significant role in the technique, but further work is necessary to quantify what these effects are and the best approach to mitigate their impacts.

An adjacent topic for future research is the application of these techniques for determining when movements occur to better the working relationship between automated systems and operators. Interference between operators and automated systems is of major concern in this field, and thus a solution is highly desirable.

Overall, further analysis of this technique with future developments is a necessity in finding the best possible solution that is easily implemented, computationally light, and performs with few false positives.

8. Conclusions

Automated pan–tilt–zoom CCTV camera drift detection is vital to the implementation and maintenance of enhanced repositioning and monitoring features. Using the mean SSIM algorithm to assign each sampled image is efficient for 49 cameras in a network and easily scaled to much larger networks. Aggregating scores mitigate artificially low scores from external factors (e.g., weather, construction, or traffic changes). For this methodology, 12 months of data for 49 cameras on Interstate 465 with historical data only yielded 2203 samples (automatically collected) and resulted in 28 alerts requiring manual validation. Alerts are derived using a threshold approach, where this work used a 0.3 mSSIM in at least two consecutive days to trigger an alert. The elimination of false positives was performed by a manual analysis of alerts. Future research is required to identify techniques to better the approach in eliminating false positives using this methodology. The resulting 1% of initial samples requiring manual validation is a more manageable size, requiring far fewer person-hours and lessening the potential downtime of enhanced features.

Modifications to the standard SSIM algorithm have the potential to yield higher passing rates. Additionally, applications of preprocessing methodologies or other similarity indexing algorithms could see improved results.

The proposed methodology is an efficient, lightweight approach to monitoring potential drift in ATMS CCTV camera networks and generating alerts for less than 1% of samples over 12 months. This reduction in sample size to alert size will prove powerful in maintaining large ASTM CCTV networks as more enhanced features are deployed in the coming years by limiting the required person-hours for detecting drift.