Loitering Detection Based on Pedestrian Activity Area Classiﬁcation

Featured Application: This paper proposes a loitering detection method based on pedestrian activity area classiﬁcation. Automatic loitering detection can help recognize vulnerable people who need attention and potential suspects that might be harmful to public security. It can be widely used in many important research ﬁelds and industries such as public security and video surveillance. Abstract: Loitering detection can help recognize vulnerable people needing attention and potential suspects harmful to public security. The existing loitering detection methods used time or target trajectories as assessment criteria, and only handled some simple loitering circumstances because of complex track. To solve these problems, this paper proposes a loitering detection method based on pedestrian activity area classiﬁcation. The paper ﬁrst gave loitering deﬁnition from a new perspective using the size of pedestrian activity area. The pedestrian loitering behaviors were divided into three categories. The proposed algorithms dynamically calculate enclosing rectangle, ellipse, and sector of pedestrian activity areas through curve ﬁtting based on trajectory coordinates within given staying time threshold. The loitering is recognized if the pedestrian activity is detected to be constrained in an area within a certain period of time. The algorithm does not need to calculate complex trajectories. The PETS2007 dataset and our own self-collected simulated test videos were tested. The experimental results show that the proposed method accurately detected the pedestrian loitering, not only detected some loitering that the existing methods could not detect, but also distinguishing di ﬀ erent types of loitering. The proposed method is e ﬀ ectiveness, robust, and simpliﬁed in implementation.


Introduction
With the advance of artificial intelligence, pedestrian loitering detection has been more and more widely applied in many fields. As one case of abnormal behavior detection, it plays an important role in the urban video surveillance system. According to IMS Research's research prediction, in the next five years the market demand of domestic security software will increase at the speed of 30% to 50%, which is enough to show that today's video surveillance and management software platform has huge potential market [1][2][3]. The demand of video surveillance systems in public places is large [4,5]. Pedestrian loitering detection is mainly applied to safety monitoring in public places such as public transportation areas to ensure the urban safety [6]. probability and some probability plot models for video surveillance and detection [41]. The likelihood function was used to estimate the probability of each individual [42,43]. Some methods used auxiliary equipment to complete the loitering detection. The most common method was pedestrian loitering detected with the help of GPS [44,45], the target needs to wear a wristband analog, which detection for specified targets cannot be monitored in public. In a method using the stereo camera to analyze abnormal behavior [46], stereo cameras are needed to track people in dense environments such as queues, otherwise this loitering detection technique works in a generic manner on any trajectory data. Our method has no special requirement for equipment.
The above-existing loitering detection methods used time or target trajectories as assessment criteria, and only handled some simple loitering circumstances because of complexity of target tracks. In order to solve these problems of lacking theoretic foundation of loitering detection and failing to detect loitering in complicated situations, this paper proposes a loitering detection method based on pedestrian activity area classification. First, the Gaussian mixture models (GMMs) and MeanShift are used to detect and track the target. The residence time of the target is used as a prejudgment for loitering detection; the complex trajectory is transformed into area using coordinate points of trajectory to describe the geometric figure. Pedestrian loitering behaviors are divided into three categories, i.e., the rectangle loitering, the ellipse loitering, and the sector loitering. The loitering is recognized if the pedestrian activity is detected to be constrained in an area within a certain period of time. The algorithm does not need to consider the complex trajectories, and it is effective and efficient in detecting loitering behaviors. Figure 1 shows the overall framework of our proposed loitering detection method. According to different scenarios, the method analyzes and calculates the size of the target activity area using the best adaptive fitting curve for loitering detection. An automatic alarm will be performed when loitering behavior occurs. The most common method was pedestrian loitering detected with the help of GPS [44,45], the target needs to wear a wristband analog, which detection for specified targets cannot be monitored in public. In a method using the stereo camera to analyze abnormal behavior [46], stereo cameras are needed to track people in dense environments such as queues, otherwise this loitering detection technique works in a generic manner on any trajectory data. Our method has no special requirement for equipment. The above-existing loitering detection methods used time or target trajectories as assessment criteria, and only handled some simple loitering circumstances because of complexity of target tracks. In order to solve these problems of lacking theoretic foundation of loitering detection and failing to detect loitering in complicated situations, this paper proposes a loitering detection method based on pedestrian activity area classification. First, the Gaussian mixture models (GMMs) and MeanShift are used to detect and track the target. The residence time of the target is used as a prejudgment for loitering detection; the complex trajectory is transformed into area using coordinate points of trajectory to describe the geometric figure. Pedestrian loitering behaviors are divided into three categories, i.e. the rectangle loitering, the ellipse loitering, and the sector loitering. The loitering is recognized if the pedestrian activity is detected to be constrained in an area within a certain period of time. The algorithm does not need to consider the complex trajectories, and it is effective and efficient in detecting loitering behaviors. Figure 1 shows the overall framework of our proposed loitering detection method. According to different scenarios, the method analyzes and calculates the size of the target activity area using the best adaptive fitting curve for loitering detection. An automatic alarm will be performed when loitering behavior occurs.  The previous research was based on simple scenarios and lacks reliability. This paper redefines and classifies loitering through the new perspective of pedestrian activity area. In the analysis of loitering in various situations, pedestrian activity areas were used to divide loitering behavior into three categories.

The Overview of Our Proposed Loitering Detection Method
In the spacious and wide scene, normal pedestrians usually walk straight line with a strict goal [47]; the suspects are aimless. They keep loitering with a complex and disordered trajectory in order to not to be noticed. Figure 2 shows a rectangle loitering in the real scene. Through the trajectory map and analysis of suspicious target trajectory, the target's area of motion is completely in the red rectangle. So, we call this type of loitering as rectangle loitering within a rectangle area. The previous research was based on simple scenarios and lacks reliability. This paper redefines and classifies loitering through the new perspective of pedestrian activity area. In the analysis of loitering in various situations, pedestrian activity areas were used to divide loitering behavior into three categories.
In the spacious and wide scene, normal pedestrians usually walk straight line with a strict goal [47]; the suspects are aimless. They keep loitering with a complex and disordered trajectory in order to not to be noticed. Figure 2 shows a rectangle loitering in the real scene. Through the trajectory map and analysis of suspicious target trajectory, the target's area of motion is completely in the red rectangle. So, we call this type of loitering as rectangle loitering within a rectangle area. In some small spaces, suspects with abnormal behavior loiter in a narrow area with back-andforth movement. However, when the area of activity decreases, the distribution of target trajectory is dispersed, and the trajectory coordinate change greatly and discrete. The rectangular analysis method not satisfy this kind of small area movement, which will miss the judgment of loitering behavior. Figure 3 shows the different detection methods used for fitting the loitering in the same situation. The rectangle frame and ellipse frame were used to fit the same area of activity, and the shaded area was the difference of the area. So, it is necessary to propose ellipse detection in order to improve the accuracy of detection. Aiming at the narrow moving scene and the large return distance of target trajectory, an ellipse area fitting method is proposed. The trajectory of the suspect will constantly change. We first detected the convex hulls of all the trajectory points, and then used the ellipse fitting to solve the problem of uncertainty in the active area and the irregularity of the trajectory.  In some occasions, the suspects, such as thieves or drug dealers, are purposefully loitering around points of interest. Candamo [13] the author claims that the suspect's motion is purposeful, for example, drug dealers focusing on the goals of interest, purposeful round trip loitering, to achieve drug dealing or stealing. This is very similar to the reality of the detection of dropped objects [48]. From the perspective of the mathematical model for analysis, this kind of activity area is similar to a sector. A sector loitering example is shown in Figure 4. The pedestrians' activity area is fitted with a In some small spaces, suspects with abnormal behavior loiter in a narrow area with back-and-forth movement. However, when the area of activity decreases, the distribution of target trajectory is dispersed, and the trajectory coordinate change greatly and discrete. The rectangular analysis method not satisfy this kind of small area movement, which will miss the judgment of loitering behavior. Figure 3 shows the different detection methods used for fitting the loitering in the same situation. The rectangle frame and ellipse frame were used to fit the same area of activity, and the shaded area was the difference of the area. So, it is necessary to propose ellipse detection in order to improve the accuracy of detection. Aiming at the narrow moving scene and the large return distance of target trajectory, an ellipse area fitting method is proposed. The trajectory of the suspect will constantly change. We first detected the convex hulls of all the trajectory points, and then used the ellipse fitting to solve the problem of uncertainty in the active area and the irregularity of the trajectory. The previous research was based on simple scenarios and lacks reliability. This paper redefines and classifies loitering through the new perspective of pedestrian activity area. In the analysis of loitering in various situations, pedestrian activity areas were used to divide loitering behavior into three categories.
In the spacious and wide scene, normal pedestrians usually walk straight line with a strict goal [47]; the suspects are aimless. They keep loitering with a complex and disordered trajectory in order to not to be noticed. Figure 2 shows a rectangle loitering in the real scene. Through the trajectory map and analysis of suspicious target trajectory, the target's area of motion is completely in the red rectangle. So, we call this type of loitering as rectangle loitering within a rectangle area. In some small spaces, suspects with abnormal behavior loiter in a narrow area with back-andforth movement. However, when the area of activity decreases, the distribution of target trajectory is dispersed, and the trajectory coordinate change greatly and discrete. The rectangular analysis method not satisfy this kind of small area movement, which will miss the judgment of loitering behavior. Figure 3 shows the different detection methods used for fitting the loitering in the same situation. The rectangle frame and ellipse frame were used to fit the same area of activity, and the shaded area was the difference of the area. So, it is necessary to propose ellipse detection in order to improve the accuracy of detection. Aiming at the narrow moving scene and the large return distance of target trajectory, an ellipse area fitting method is proposed. The trajectory of the suspect will constantly change. We first detected the convex hulls of all the trajectory points, and then used the ellipse fitting to solve the problem of uncertainty in the active area and the irregularity of the trajectory.  In some occasions, the suspects, such as thieves or drug dealers, are purposefully loitering around points of interest. Candamo [13] the author claims that the suspect's motion is purposeful, for example, drug dealers focusing on the goals of interest, purposeful round trip loitering, to achieve drug dealing or stealing. This is very similar to the reality of the detection of dropped objects [48]. From the perspective of the mathematical model for analysis, this kind of activity area is similar to a sector. A sector loitering example is shown in Figure 4. The pedestrians' activity area is fitted with a In some occasions, the suspects, such as thieves or drug dealers, are purposefully loitering around points of interest. Candamo [13] the author claims that the suspect's motion is purposeful, for example, drug dealers focusing on the goals of interest, purposeful round trip loitering, to achieve drug dealing or stealing. This is very similar to the reality of the detection of dropped objects [48]. From the perspective of the mathematical model for analysis, this kind of activity area is similar to a sector. A sector loitering example is shown in Figure 4. The pedestrians' activity area is fitted with a red sector which contains the motions of the pedestrian. We call this type of loitering as a sector loitering around a point in a sector area.

Target Detection and Tracking
Loitering detection video is usually shot statically, and the background is changed slowly, mainly considering the impact of illumination, occlusion, and so on. In this paper, in order to achieve the purpose of detecting moving objects, GMM [49] is used to detect the target. The probability density of image gray level is estimated by the gray level difference between target and background, this method has high accuracy and objectivity. Multiple Gaussian models are selected to represent the feature distribution, then compared with the GMM. The EM iterative algorithm is used to solve the parameters of the GMM. Based on the existing library implementation algorithm, some parameter values are default values and complete the classification of target and background.
We established GMM fusion models in each pixel, the probability density function of a pixel at time is shown in Equation (1).
where is the weight of the Gaussian model, is the mean vector of the Gaussian model, and ∑ is the covariance vector of n GMM fusion models. We used the EM algorithm to calculate the parameters of the GMM and posterior probability, set a new latent variable Z, and then transformed Equation (1) by variable Z to Equation (2).
Attain the posterior probability ( ) by the prior probability ( ), and ( | ) is likelihood probability. The posterior probability ( ) is shown in Equation (3).
From the Equation (3), the derivative expressions of three variables are shown in Equation (4).
where Q is the number of all the pixels, we update the parameters to convergence by using Equation (3) and Equation (4) so as to achieve the purpose of detecting moving objects; this method has high accuracy and objectivity.

Target Detection and Tracking
Loitering detection video is usually shot statically, and the background is changed slowly, mainly considering the impact of illumination, occlusion, and so on. In this paper, in order to achieve the purpose of detecting moving objects, GMM [49] is used to detect the target. The probability density of image gray level is estimated by the gray level difference between target and background, this method has high accuracy and objectivity. Multiple Gaussian models are selected to represent the feature distribution, then compared with the GMM. The EM iterative algorithm is used to solve the parameters of the GMM. Based on the existing library implementation algorithm, some parameter values are default values and complete the classification of target and background.
We established k GMM fusion models in each pixel, the probability density function of a pixel X at time T is shown in Equation (1).
where W is the weight of the Gaussian model, µ is the mean vector of the Gaussian model, and n is the covariance vector of n GMM fusion models. We used the EM algorithm to calculate the parameters of the GMM and posterior probability, set a new latent variable Z, and then transformed Equation (1) by variable Z to Equation (2).
Attain the posterior probability δ(z n ) by the prior probability P(z), and P(x z) is likelihood probability. The posterior probability δ(z n ) is shown in Equation (3).
where Q is the number of all the pixels, we update the parameters to convergence by using Equations (3) and (4) so as to achieve the purpose of detecting moving objects; this method has high accuracy and objectivity. After targets are detected, we used the MeanShift method to find the most likely target location in the current video frame [50]. The process of object tracking is to find a target position to make the Bhattacharyya coefficient maximize, it is defined in the following Formula (5).
p(x) and q(x) represent the relative density functions of the discrete position and the target model, respectively. The specific flowchart of target tracking is shown in Figure 5. After targets are detected, we used the MeanShift method to find the most likely target location in the current video frame [50]. The process of object tracking is to find a target position to make the Bhattacharyya coefficient maximize, it is defined in the following formula (5).
p( ) and q( ) represent the relative density functions of the discrete position and the target model, respectively. The specific flowchart of target tracking is shown in Figure 5. Through the above method, we can track the pedestrian and get a series of tracking coordinates defined in Equation (6).
where ( , ), ( , ), ⋯ , ( , ), respectively, represent the position coordinates ( , ) at the corresponding time. Figure 6 shows the motion trajectory. represents the trajectory of the target. We use Equation (7) to calculate the change of the pedestrian angle to select the feature points: where ∆ ⃗ represents the direction vector's angle value of the target at the different moment. is the angle threshold set to 50° in our experiment if it meets the condition in the Equation (7) at a point, then we record this point as feature point .
The moment of the initial direction vector is the m moment. Once a feature point is found, the direction vector will be replaced by the direction vector . The eigenvector of the trajectory is marked. As the number of eigenvectors appearing in the time (0, ⋯ , ) exceeds 4, it needs to find corresponding loitering detection based on different activity area characteristics.

The Rectangle Loitering
We use Equation (8) to establish an active rectangle frame to determine whether the pedestrian's Through the above method, we can track the pedestrian and get a series of tracking coordinates defined in Equation (6).
where (x 1 , y 1 ), (x 2 , y 2 ), · · · , (x n , y n ), respectively, represent the position coordinates (x, y) at the corresponding time. Figure 6 shows the motion trajectory. s i represents the trajectory of the i target. We use Equation (7) to calculate the change of the pedestrian angle to select the feature points: where ∆ → v represents the direction vector's angle value of the target at the different moment. θ is the angle threshold set to 50 • in our experiment if it meets the condition in the Equation (7)  After targets are detected, we used the MeanShift method to find the most likely target location in the current video frame [50]. The process of object tracking is to find a target position to make the Bhattacharyya coefficient maximize, it is defined in the following formula (5).
p( ) and q( ) represent the relative density functions of the discrete position and the target model, respectively. The specific flowchart of target tracking is shown in Figure 5. Through the above method, we can track the pedestrian and get a series of tracking coordinates defined in Equation (6).
where ( , ), ( , ), ⋯ , ( , ), respectively, represent the position coordinates ( , ) at the corresponding time. Figure 6 shows the motion trajectory. represents the trajectory of the target. We use Equation (7) to calculate the change of the pedestrian angle to select the feature points: where ∆ ⃗ represents the direction vector's angle value of the target at the different moment. is the angle threshold set to 50° in our experiment if it meets the condition in the Equation (7) at a point, then we record this point as feature point .
The moment of the initial direction vector is the m moment. Once a feature point is found, the direction vector will be replaced by the direction vector . The eigenvector of the trajectory is marked. As the number of eigenvectors appearing in the time (0, ⋯ , ) exceeds 4, it needs to find corresponding loitering detection based on different activity area characteristics.

The Rectangle Loitering
We use Equation (8) to establish an active rectangle frame to determine whether the pedestrian's activity area is in a rectangle frame. A rectangle trajectory is defined as a trajectory contained within a rectangle, as shown in Figure 7. The moment of the initial direction vector is the m moment. Once a feature point is found, the direction vector v m will be replaced by the direction vector v f . The eigenvector of the trajectory is marked. As the number of eigenvectors appearing in the time (0, · · · , t) exceeds 4, it needs to find corresponding loitering detection based on different activity area characteristics.

The Rectangle Loitering
We use Equation (8) to establish an active rectangle frame to determine whether the pedestrian's activity area is in a rectangle frame. A rectangle trajectory is defined as a trajectory contained within a rectangle, as shown in Figure 7.
In Equation (8), (x i , y i ) represents any coordinate in the (t 1 , · · · , t 2 ) period, (x 1 , y 1 ) represents the coordinates of t 1 moments, (x 2 , y 2 ) represents the coordinates of t 2 moments, and rect x 1 , y 1 , x 2 , y 2 represents all coordinate points in the rectangular domain with (x 1 , y 1 ) and (x 2 , y 2 ) as diagonals. We propose a dynamic rectangular frame to judge loitering, which avoids judging the motion trajectory of each frame in the video. The detailed of our algorithm is described in the Algorithm 1, where represents the initial threshold. In our experiments using PETS 2007 dataset, the time threshold is set to 60 seconds. The time threshold should be set according to different scenarios. Compare the area, if ＜ , a rectangle loitering is detected.

The Ellipse Loitering
An ellipse trajectory is defined as a trajectory contained within an ellipse. We use the following steps to determine whether the pedestrian's active area is in an ellipse area. Firstly, the Graham scan [51] method is used to find convex hulls for all track points in time , then ellipse fitting is performed. Figure 8 shows the process of finding a convex hull of a set of point. We propose a dynamic rectangular frame to judge loitering, which avoids judging the motion trajectory of each frame in the video. The detailed of our algorithm is described in the Algorithm 1, where T 0 represents the initial threshold. In our experiments using PETS 2007 dataset, the time threshold T 0 is set to 60 s. The time threshold should be set according to different scenarios.
Extract the trajectory coordinates p i (x i , y i ) 2.
when T i > T 0 , calculate the distance between P i and P i−T , i > 0 Update P i , calculate the size of activity area: Compare the area, if S < S threshold , a rectangle loitering is detected.

The Ellipse Loitering
An ellipse trajectory is defined as a trajectory contained within an ellipse. We use the following steps to determine whether the pedestrian's active area is in an ellipse area. Firstly, the Graham scan [51] method is used to find convex hulls for all track points in time T i , then ellipse fitting is performed. Figure 8 shows the process of finding a convex hull of a set of point.

The Ellipse Loitering
An ellipse trajectory is defined as a trajectory contained within an ellipse. We use the following steps to determine whether the pedestrian's active area is in an ellipse area. Firstly, the Graham scan [51] method is used to find convex hulls for all track points in time , then ellipse fitting is performed. Figure 8 shows the process of finding a convex hull of a set of point.  In order to fit the ellipse, we use the general elliptic equation in Equation (9).
We used some constraints [52] so that we can directly use the coordinates of the trajectories to do the least square. The coefficients of the equation can be obtained using Equation (10).
Each coefficient is determined by its minimum value. Equation (11) can be obtained by the principle of extreme value. ∂ Thus, we can get A, B, C, D, and E values, so that we can perform ellipse fitting on the given coordinates, calculate the optimal elliptic equation with least squares method. The detailed algorithm is described in the Algorithm 2.
Extract the trajectory coordinates P i (x i , y i ).

3.
Sort the clockwise direction through the polar angle θ and distance D i , then cycle points; select six points in the calculate coordinates, T i−t < T < T i ; calculate parameters: Record num-match and num-index, when num match > num index , update ellipse parameter.

5.
Calculate the size of activity area: S = πAC , A > 0 , C > 0, compare area, if S < S threshold , an ellipse loitering is detected.

The Sector Loitering
When suspicious person loiters around the point of interest, suspect's activity area revolves around one point. The displacement calculating cannot accurately determine whether pedestrians Appl. Sci. 2019, 9, 1866 9 of 18 have abnormal behavior. Therefore, a sector trajectory is defined as a trajectory contained within a sector. As shown in Figure 9. Equation (12) is the loitering definition based on the sector.
The D t is the displacement between the initial detection coordinate and the current coordinate and M 1 and M 2 are the set thresholds. D max is the maximum value of the coordinate point displacement, while D min is the minimum value of the coordinate point displacement in the time (t i , · · · , t). The detailed algorithm is described in the Algorithm 3. 4． Compare the area, when ＜ , then a sector loitering is detected.

Experimental Results and Analysis
Most of the public abnormal behavior dataset are abnormal behaviors for public safety. There are only a few datasets for loitering. The most classic dataset is PETS2007 [53]. This dataset was taken by four cameras at four different angles. The shooting scenes were changeable and complex. The PETS2007 dataset has specialized experimentation and testing for loitering. Loitering is defined as a person who enters the scene, and remains within the scene for more than 60 seconds.
We also collected our own dataset by using video cameras. About our collected dataset, we used HIKVISON DS-2CD5152F-IZ camera, choose different scenarios for experiments. The video was saved in AVI format. Videos in the dataset were captured at 15 frames per second. To ensure the quality of the experiment, each video has at least 800 frames. We collected 120 videos (90 videos of loitering and 30 videos of normal walking), including loitering detection in many scenarios from both indoor and outdoor scenes.
In order to ensure the loitering experiment, we tested PETS2007 and our collected dataset which contain different angles, different illumination, target occlusion, multitarget, and so on. Figure 10 shows the results of test PETS2007 dataset and our self-collected dataset by using the GMM fusion model. The background of the PETS 2007 dataset is complex and many challenges exist while detecting objects such as dynamic environment, illumination, occlusion of objects, etc. Figure 10a,b Figure 9. Demonstrate a sector trajectory and its corresponding calculation.
Record the initial position P 0 , extract the trajectory coordinates P i (x i , y i ) 2.
Calculate the distance D i between P i and P 0 , when T i > T 0 , then update D i into the scope [D max , D min ]

3.
Calculate the size of activity area, the sector angle is α: α = 180 • π * 2arctan y p 0 − y p i x p 0 − x p i , So, the area is:

4.
Compare the area, when S < S threshold , then a sector loitering is detected.

Experimental Results and Analysis
Most of the public abnormal behavior dataset are abnormal behaviors for public safety. There are only a few datasets for loitering. The most classic dataset is PETS2007 [53]. This dataset was taken by four cameras at four different angles. The shooting scenes were changeable and complex. The PETS2007 dataset has specialized experimentation and testing for loitering. Loitering is defined as a person who enters the scene, and remains within the scene for more than 60 s.
We also collected our own dataset by using video cameras. About our collected dataset, we used HIKVISON DS-2CD5152F-IZ camera, choose different scenarios for experiments. The video was saved in AVI format. Videos in the dataset were captured at 15 frames per second. To ensure the quality of the experiment, each video has at least 800 frames. We collected 120 videos (90 videos of loitering and 30 videos of normal walking), including loitering detection in many scenarios from both indoor and outdoor scenes.
In order to ensure the loitering experiment, we tested PETS2007 and our collected dataset which contain different angles, different illumination, target occlusion, multitarget, and so on. Figure 10 shows the results of test PETS2007 dataset and our self-collected dataset by using the GMM fusion model. The background of the PETS 2007 dataset is complex and many challenges exist while detecting objects such as dynamic environment, illumination, occlusion of objects, etc. Figure 10a,b shows the detection effect of the PETS 2007 dataset. The detection scene is complex background in public places, and there are many external factors such as illumination change and occlusion. Although the experimental results provided by PETS 2007 dataset were inevitably incoherent and some residual, the overall effect was still relatively accurate. Figure 10c-d shows the results of our collected dataset including outdoor and indoor scenes. The dataset collected had accurate detection results and distinct human contours were extracted. Figure 11 shown the experimental results of tracking and detecting pedestrians. The trajectory of pedestrians was plotted, the green frame is used as the detection frame, and the center of the frame is used as the trajectory plotting point. For the PETS 2007 dataset, mobile pedestrians could be detected. The test results are valid for our collected dataset. Table 1    After the detection and tracking algorithm achieves high accuracy and good robustness, a loitering detection method is added to the framework. Through the analysis and testing of the dataset   After the detection and tracking algorithm achieves high accuracy and good robustness, a loitering detection method is added to the framework. Through the analysis and testing of the dataset we selected 30 videos of each algorithm for area threshold testing. Those videos are all from loitering  After the detection and tracking algorithm achieves high accuracy and good robustness, a loitering detection method is added to the framework. Through the analysis and testing of the dataset we selected 30 videos of each algorithm for area threshold testing. Those videos are all from loitering dataset collected by ourselves. In order to be applicable for video surveillance scenarios, all videos are converted to the same resolution 704 × 576, which can reduce the influence of different resolution on detection results. Figure 12 shows the accuracy of loitering detection algorithms with different area thresholds. The abscissa is the number of pixels which represents the area of geometric figure while the ordinate represents the number of correctly detected videos containing loitering instances. The change of curve represents the ability of the algorithm to detect correctly under different area thresholds. The higher the curve is, the better the detection effect is. instances. The change of curve represents the ability of the algorithm to detect correctly under different area thresholds. The higher the curve is, the better the detection effect is.  Figure 13 shows the rectangle loitering detection results and corresponding fitting area changed. The horizontal coordinate represents frames of videos, and the vertical coordinate is the size of fitting rectangle area. The curve change corresponds to the change of fitted area. When the area is over the threshold area it will stop alarm, until the pedestrian out of the camera field of views which can reduce misjudgment. We can clearly see that the rectangle loitering detection algorithm was accurate and stable throughout the fitting process.  Figure 13 shows the rectangle loitering detection results and corresponding fitting area changed. The horizontal coordinate represents frames of videos, and the vertical coordinate is the size of fitting rectangle area. The curve change corresponds to the change of fitted area. When the area is over the threshold area it will stop alarm, until the pedestrian out of the camera field of views which can reduce misjudgment. We can clearly see that the rectangle loitering detection algorithm was accurate and stable throughout the fitting process. Figure 14 shows the detected result of ellipse loitering in different scenarios and the activity area fitting curve. Once it exceeds the time, it immediately alarms. From the results, the entire fitting process is continuous without interruption, so it has good robustness and real-time performance. Figure 13 shows the rectangle loitering detection results and corresponding fitting area changed. The horizontal coordinate represents frames of videos, and the vertical coordinate is the size of fitting rectangle area. The curve change corresponds to the change of fitted area. When the area is over the threshold area it will stop alarm, until the pedestrian out of the camera field of views which can reduce misjudgment. We can clearly see that the rectangle loitering detection algorithm was accurate and stable throughout the fitting process.    Figure 15 shows the experimental results of the sector loitering detected in three different scenarios and activity area fitting. From these experimental results, it can be seen that the sector loitering detected can perfectly fitted the suspect's activity area for both indoor and outdoor. The whole process of sector fitting was very smooth and stable.  Figure 15 shows the experimental results of the sector loitering detected in three different scenarios and activity area fitting. From these experimental results, it can be seen that the sector loitering detected can perfectly fitted the suspect's activity area for both indoor and outdoor. The whole process of sector fitting was very smooth and stable. In order to directly demonstrate the superiority of the algorithm presented in this paper, Table  2 shows the real experimental record and result analysis of this paper. Our collected dataset was accurately classified according to different scenarios. Each scenario has loitering and normal walking. Here, TP (True Positive) means the true loitering can be detected. TN (True Negative) means normal walking can be detected. FP (False Positive) is the normal behavior detected as abnormal. FN (False Negative) is loitering detected as normal. In the Table 2, P stands for precision ( = ), R stands for recall rate ( = ), and A stands for accuracy ( = ). Ellipse 30 30 Figure 16 shows the advantages and differences of the three algorithms. From the results, an unsuitable algorithm does not only consumes time but also fails to detect loitering correctly. Therefore, the detection of loitering should be achieved based on scene and motivation to choose the correct algorithm. Table 3 shows the results of the three algorithms applied to our collected dataset.  In order to directly demonstrate the superiority of the algorithm presented in this paper, Table 2 shows the real experimental record and result analysis of this paper. Our collected dataset was accurately classified according to different scenarios. Each scenario has loitering and normal walking. Here, TP (True Positive) means the true loitering can be detected. TN (True Negative) means normal walking can be detected. FP (False Positive) is the normal behavior detected as abnormal. FN (False Negative) is loitering detected as normal. In the Table 2   In order to directly demonstrate the superiority of the algorithm presented in this paper, Table  2 shows the real experimental record and result analysis of this paper. Our collected dataset was accurately classified according to different scenarios. Each scenario has loitering and normal walking. Here, TP (True Positive) means the true loitering can be detected. TN (True Negative) means normal walking can be detected. FP (False Positive) is the normal behavior detected as abnormal. FN (False Negative) is loitering detected as normal. In the Table 2, P stands for precision ( = ), R stands for recall rate ( = ), and A stands for accuracy ( = ). Sector

28
Ellipse 30 30 Figure 16 shows the advantages and differences of the three algorithms. From the results, an unsuitable algorithm does not only consumes time but also fails to detect loitering correctly.  Figure 16 shows the advantages and differences of the three algorithms. From the results, an unsuitable algorithm does not only consumes time but also fails to detect loitering correctly. Therefore, the detection of loitering should be achieved based on scene and motivation to choose the correct algorithm. Table 3 shows the results of the three algorithms applied to our collected dataset.

28
Ellipse 30 30 Figure 16 shows the advantages and differences of the three algorithms. From the results, an unsuitable algorithm does not only consumes time but also fails to detect loitering correctly. Therefore, the detection of loitering should be achieved based on scene and motivation to choose the correct algorithm. Table 3 shows the results of the three algorithms applied to our collected dataset. Figure 16. The different algorithms to test the same scenarios to distinguish the advantages and differences of the three algorithms.  Figure 16. The different algorithms to test the same scenarios to distinguish the advantages and differences of the three algorithms.  Figure 17 shows the detected results of the PETS2007 dataset (S00, S01, S03, S05, and S08) based on our method. The experimental results show that our proposed algorithms can also be used to detect suspects in crowded scenarios. Our method was compared with the method of Nam 2015 [54]. Our method can detect loitering in videos from multiple angles, while Nam 2015 only detected loitering in a video from one angle where the number of pedestrians is small. For dataset S00 without containing loitering behavior, our algorithms successfully detected them as normal. It is good to pick a rectangle algorithm to detect suspects in emptiness and high activity areas, learning from the result of dataset S01. Dataset S05 is a video with a thief in public: the suspect loitered around the interest point of the package; we used the sector loitering detection algorithm. The suspicious loitering in a small area in dataset S03, we choose the ellipse loitering algorithm. All the loitering detecting algorithms can effective and accurately detect the suspects. Table 4 is the loitering detection results using the PETS2007 dataset compared with Dalley's method [55]. First of all, our algorithms can accurately detect loitering in the PETS2007 dataset. All our three algorithms performed much better and achieved much higher detection accuracy than the Dalley's algorithm. Our algorithms successfully detected the loitering which could not be detected by the Dalley's method, such as those in dataset S05 and S08. algorithms can effective and accurately detect the suspects. Table 4 is the loitering detection results using the PETS2007 dataset compared with Dalley's method [55]. First of all, our algorithms can accurately detect loitering in the PETS2007 dataset. All our three algorithms performed much better and achieved much higher detection accuracy than the Dalley's algorithm. Our algorithms successfully detected the loitering which could not be detected by the Dalley's method, such as those in dataset S05 and S08.

Conclusions
In this paper, we first gave a loitering definition from a new perspective using pedestrian activity areas. We propose a loitering detection method based on pedestrian activity areas classification. The pedestrian loitering behaviors are divided into three categories, i.e., the rectangle loitering, the ellipse loitering and the sector loitering. Unlike other loitering detection algorithms judging the loitering only by trajectory angle or time, this paper converts complex trajectory processing into activity area fitting. The loitering is recognized if the pedestrian activity is detected and to be constrained in an area within a certain period of time. From the experimental results, the proposed three algorithms are very effective for detecting various loitering. No matter whether it is indoors or outdoors, it has good accuracy and robustness in loitering detection for both the PETS2007 dataset and our self-collected dataset. The proposed method can detect loitering in videos captured from different angles, and can be applied to crowded scenes also. Compared with the existing methods, the proposed method not only can detect some loitering that the existing methods could not detect, but also is practical to handle complicated situations, efficient, robust, and simplified in implementation. Self-adaptive time threshold setting for loitering detection in different scenarios is worth of further research. We will further research on this in the future.
Author Contributions: All authors of the paper have made significant contributions to this work. T.H. and Q.H. contributed equally to the paper, conceived the idea of work, wrote the manuscript, and analyzed the experiment data. W.M. led the project, directed, and revised the paper writing. X.L., Y.Y., and Y.Z. collected the original data of the experiment and participated in programming.

Conflicts of Interest:
The authors declare no conflict of interest.