Real-time Multiple Object Tracking Using Virtual Shells

This paper presents a new multiple object tracking framework where arbitrarily shaped complex objects having severe occlusions are successfully tracked in real-time using stationary cameras. The proposed method utilizes the concept of a virtual shell that encloses each moving object in the scene and simplifies analysis of object interactions. Object's state transitions are handled by an event resolution analysis. Such a regional analysis boosts the tracking process. Finally, a pixel level evaluation is employed for all objects to establish successful pixel memberships. The proposed tracking algorithm is experimentally tested on some public databases and our own challenging aquarium setup that contains many different fish interactions.


INTRODUCTION
Object tracking has been a focus of research for decades due to its promising potential for real-time applications such as intelligent user interfaces, navigation systems and surveillance.A successful tracking algorithm is expected to be robust against environmental changes.Main difficulties of the single object tracking problem are appearance changes, abrupt motion and object non-rigidity.One can view multiple object tracking as the problem of running multiple tracker instances on each object.However such an approach is likely to fail because of not exploiting the dependence between tracker instances and object models, running each tracker independently, causing tracker to get stuck on one object and lose the others (see MacCormick and Blake [1]).One needs a more reliable tracker which uses holistic information exploiting the tracking correlation between objects.
Main problem of the multi object tracking systems is object interactions which give rise to occlusions (see Papadourakis et al. [2] and Bose et al. [3]).While simple scenarios usually include interaction of two objects, real world examples may include several objects interacting with each other that give rise to complex events and cause the tracker to fail.This necessitates development of more robust, reliable and easily scalable object tracking formulations.Argyros et al. [4] propose online learning of color for tracking skin.Khan et al. [5] employ particle filtering for tracking of multiple objects.Sullivan et al. [6] exploit continuity of depth and motion direction for object labeling problem.Yu and Medioni [7] propose a Markov Chain Monte Carlo formulation for multiple object tracking.
Most of the multi object tracking methods, regardless of the number of cameras installed, requires a background subtraction algorithm to detect motion, a prerequisite step for object representation.The occlusion problem is mostly handled in two different ways: merge-split approach and straight-through approach as noted in work of Gabriel et al. [8].In the merge-split based methods (McKenna et al. [9] and Bremond et al. [10] use appearance cues, Haritaoglu's W4 system [11] uses appearance and motion cues), separate blobs are updated as long as no occlusion is detected.In the case of occlusion, a joint blob is formed and tracked until the objects are splitted.The detection of beginning and termination of occlusion requires an occlusion and split predicate.Straight-through approaches as in Khan's method [12] and Haritaoglu's Hydra system [13] do not handle split and merge cases separately.These approaches continue to track each individual object in the presence of occlusions using various image features.No joint blob, or hypothesis, is formed during the occlusion phase.These methods require an occlusion predicate which acts as a trigger for ongoing occlusion event.
Regardless of the approach that may be taken, a robust and reliable multiple object tracking framework necessitates formal definition of the interaction problem that is boosted by merge and split predicates.Motivated by these observations, in this work, we propose a novel method for solving multiple object tracking problem under severe occlusions.First, occlusion and split predicates are implemented using virtual shells and blobs.Second, a split-merge level analysis is performed using an event resolution step.This step provides information about object interactions with the help of temporal consistence.Third, a straight-through approach is adopted and pixel level analysis is carried out on shells by considering updated events without forming joint objects.
The rest of the paper is organized as follows: In section 2, we present our proposed multiple object tracking framework.In section 3, we provide experimental results on well-known video databases and videos generated from our aquarium setup.The paper is finally concluded with several remarks in section 4 and future directions are indicated.

TRACKING USING VIRTUAL SHELLS
Our tracking approach has three important ingredients: a virtual shell model, an event resolution analysis and a pixel membership evaluation.By a virtual shell it is meant a closed-bounded curve or surface that encloses each moving object and handles complex interactions between objects that may have arbitrary shapes and motion.An event resolution analysis based on state transitions is performed using geometric relationships between object shells.This resolution step provides a macro level evaluation for multi-object tracking process.Finally, a micro level analysis, namely a pixel membership evaluation is carried out where all interesting pixels are evaluated and assigned to corresponding objects using a probabilistic approach that utilizes virtual shells and event resolutions.

Background Subtraction
In most tracking systems, background subtraction is a preprocessing step for motion detection.In the first step of our algorithm, the background subtraction technique detailed in Sheikh's work [14] is applied to the input frame to obtain background and foreground pixels.In the cited work, a three step algorithm is employed to obtain pixel labels.First, a background model is created and maintained.Second, a foreground model is created.Third, both models are used under a MAP-MRF (Maximum A Posteriori-Markov Random Fields) decision framework.Inspired by the performance evaluations in Benezeth's comparison [15], we have replaced the third step with morphological filtering to achieve real-time performance.
After probabilistic evaluation of background and foreground pixels, each pixel is labeled using a weighted median filter.Outputs of Parzen classifiers for each pixel, indicated by  in the original algorithm, are fed into the filtering process as weights.Using such a filter guarantees reconstruction of the disconnected parts of an object.Finally, a connected component labeling algorithm is employed to obtain labeled blobs in the input image.This procedure is then continued with a geometric filtering step which is obligatory to avoid false positive regions.Geometric filtering can be utilized in many different ways.One can create a filter based on angle, shape or size constraints.In a scenario where objects with arbitrary and highly complex shapes that may undergo abrupt motion, a size filter will be adequate to remove false positives of background subtraction process.
Enhanced blobs are main regions to create, track and update object hypothesis.However, due to high shape variations in temporal domain, blobs themselves are insufficient for object representation.To overcome this problem, we introduce virtual shells for further processing.

Virtual Shell Model
In shell model, each object possesses a unique enclosing shell (Fig. 1(a)).Shells provide spatial relaxation, allow prediction of object interactions and possible merge and split events.Object interactions can be interpreted as shell interactions.Object merge events are described as merging of multiple shells.Shell radius defines the interaction range for an object.An object's interaction range is highly correlated to its speed.Thus, rapid objects will have bigger shells whereas slowing down objects' shells will shrink in time as shown in Fig. 1

Event Resolution
Object to object relationships in the scene can be in one of three possible states: INDEPENDENT, INTERACTION and JOINT as shown in Fig. 1(c)-(d Two objects in JOINT state can split in time (see Fig. 2).On split event, their shells will have a non-empty intersection set and their blobs will be spatially distinct.The event list must be updated to hold incoming INTERACTION event and discard previous JOINT event between objects.
Two interacting objects end their interaction which is detected by an empty intersection set between shells.In this case both objects will go to an INDEPENDENT state and isolate themselves from each other as shown in Fig. 2. Previous interaction record must be erased from the event list.
Figure 2. Demonstration of Case IV (green and red) and Case V (purple and orange).

Pixel Membership Evaluation
After performing an event resolution analysis at object level, each object's nearby pixels in a region of interest must be evaluated for pixel membership.This region of interest is selected as object's shell.As in Fig. 2, there can be both the object pixels and the interacted objects' pixels inside the shell radius of an object.Each pixel is then evaluated for possible membership with each object   using the following optimization problem.
= arg max c     ), ∀  ∈   ,  ∈ ℵ(  , ) (6) where ℵ(. ) is defined as: Note that the set for an object   is defined as the union of the object and other objects   that are in relations with   .All elements of   , are checked to determine for possible ownership of the pixel   .Then the probability term (|  ) can be computed by utilizing Bayes theorem; i.e.
=    ) () (8) where the likelihood term    ) is calculated using the color histogram of pixels, and the prior term   is modeled as a kernel function corresponding to the shell   .In a scenario where frequent occlusions take place and objects undergo abrupt motion, the prior term provides robustness to the pixel evaluation process by imposing the constraint of spatial proximity (to the shell center) on pixels.Pixels far away from the center will have less attraction to belong to that shell.This kernel function of each object is centered at    , their shell center.Intuitively this saturation introduces some constraint where an object   cannot enter membership voting for a pixel if the pixel is outside of its own shell, even if it is in interaction with   .Selection of the kernel is a question of accuracy and computational constraints.In our implementation, Epanechnikov kernel [16] has been selected as the kernel function.The pixels that are far away from shell center have lower probabilities to belong to that object.
As an example, in Fig. 3, we consider an occlusion scenario between two objects.Purple colored shells indicate interaction where blue color indicate JOINT event between objects.After utilizing pixel evaluation step, green colored pixels indicate pixels of the occluded object while the blue pixels belong to the occluder.

Position Update
After assigning each pixel to the corresponding object, each object's center is updated using member pixels; i.e.
,+1 =    + 1 −    ,+1 |  , where   represents all member pixels of   .The updated position is a linear combination of the current first order moment of   , i.e. [  ], and the Kalman filter prediction, i.e.   ,+1 |  , .Integration of filtering into the process helps smooth transitions of object centers in temporal domain avoiding instantaneous noisy estimations.

EXPERIMENTS
We present experiments on PETS, Caviar and aquarium sequences.Aquarium images has been acquired using Imaging Source DFK21BF04H Firewire CCD cameras and resized to 320x240 resolution under RGB color space.The ground level of aquarium contains shiny colorful pebbles which can act as false positives to many detection algorithms based on appearance variation.They are suppressed successfully using our background subtraction step as shown in Fig. 3(b).First aquarium sequence contains rapid motion and self-occlusion examples.Second aquarium sequence focuses on several objects' interaction.
In Fig. 4 tracker's results on PETS video is shown.A car and a person is moving towards each other, merges and splits after their interaction ends.In our quantitive analysis, we have used the MOTA metric described in Bernardin's work [17] for evaluation of multiple object tracking accuracy.In Fig. 5, quantitative results from PETS sequence are shown where three objects (shown by blue, green, red bars) reside occasionally on the scene.We defined the tracking accuracy for each object as the distance of computed object center to ground truth object center.In Fig. 7, we show a tracking sequence from our experimental setup; an aquarium containing several fish.While in many scenarios split and merge events are solved with the help of linear motion assumptions, this assumption is not valid in this environment due to abrupt and rapid motion.This is observable in Fig. 7(e) and (f) which shows that small fish in pink bounded box performs a sudden turn.Moreover, self-occlusion and object interactions are very frequent.In Fig. 7(a) fish in blue bounded box severely self-occludes which reduces number of pixels associated with it and suffers from a noticeable shape variation.Interaction of two fish is observable in Fig. 7(g) and (h).Fish in blue box appears in front and with maneuver of neighbor fish they change roles, it appears behind of its neighbor in a very short time.(e) (f) (g) (h) Figure 7.A tracking sequence from aquarium where three fish interact.
In Fig. 8, we show a second sequence from aquarium environment.This sequence has a total of 350 frames.In this sequence we track six fish.Four of them go into an interaction eventually (Fig. 8(i)-(m)).Note that objects' interaction complexity is high as many shells collide with each other and pixel classification task extends to several objects.At the end of the interaction, classification succeeds and each fish's location is preserved (Fig. 8(n)-(p)).This sequence shows that arbitrary number of objects' interaction can be coped with using our framework.

CONCLUSION AND FUTURE WORKS
We have now presented a real-time multiple object tracking algorithm.Based on virtual shell modeling, the algorithm uses event resolution analysis and pixel class evaluation to achieve robust tracking.The algorithm has been tested on some wellknown databases and also on our challenging aquarium setup where multiple objects interact with each other and create very complicated occlusion scenarios.Algorithm is fast, repeatable and robust.The experimental results are quite promising.
As a future work, we plan to integrate several robust cues along with color to improve pixel discrimination, devise better handling mechanisms for disputed pixels, and develop soft assignment rules for pixels to increase accuracy.
(b).Dynamically sized shells facilitate detection of multiple object interactions.

Figure 1
Figure 1.a) An object   with its minimum shell   and instantaneous shell   is depicted.b) Shells dynamically grow and shrink with time c) State transitions of object to object relations d) State graph where thin edges represent INTERACTION and thick edges represent JOINT relationship between objects.

Figure 3
Figure 3. a) Aquarium image b) Background subtraction c) Output of the event resolution and pixel membership evaluation where pixels of each object are colored differently for illustration purposes.

Figure 4 .
Sequence from PETS database.

Figure 5 .
Figure 5. Tracking accuracy for PETS Sequence where each color represents a tracked object.

Figure 6 .
Sequence from Caviar database.

Figure 8 .
A tracking sequence from aquarium where several fish interact.
). Object state transitions follow a simple but crucial assumption: no two objects can make transition directly from INDEPENDENT to JOINT state or vice versa.An object in JOINT state must first move to the INTERACTION state before becoming an INDEPENDENT object.Thus, the tracking problem can be expressed as keeping track of the states and the positions of each object.To be more specific, let an object   has an event list denoted by   , at time , that keeps record of INTERACTION ( , , ) and JOINT ( , , ) relationships with object   .An object having no relationship with other objects has an empty list   , = ∅.Proposed shell model provides an infrastructure for the determination of possible state transitions.Let   and   be the shells of the objects   and   , respectively.Also let   and   denote the blobs corresponding to   and   .In what follows, we shall consider several possible scenarios and analyze each in detail. Case I.   ∩   = ∅,  , ,−1 ,  , ,−1 ∉  ,−1 (1) This condition says that if object shells do not intersect and they had no INTERACTION or JOINT relationship at time  − 1, then each object is INDEPENDENT of each other.
 Case II.∩   ≠ ∅,  , ,−1 ,  , ,−1 ∉  ,−1   , =  ,−1 ∪  , ,(2)When two object shells has a non-empty intersection and had no INTERACTION and JOINT relation in their list, one can conclude that this is a start of an INTERACTION event between objects   and   .Volume of the intersection set accumulates when objects move toward each other.The INTERACTION event must be added to both objects' event list   , and   , .