Review on Computer Aided Sewer Pipeline Defect Detection and Condition Assessment

Abstract

Physical and operational inspection of sewer pipelines is critical to sustaining an acceptable level of system serviceability. Emerging inspection tools in addition to developments in sensor and lens technologies have facilitated sewer condition assessment and increased the quality and consistency of provided data. Meanwhile, sewer networks are too vast to be adequately investigated manually so the development of innovative computer vision techniques for automation applications has become an interest point of recent studies. This review paper presents the current state of inspection technology practices in sewer pipelines. An overall inspection tool comparison was conducted and the advantages and disadvantages of each method were discussed. This was followed by a comprehensive review of recent studies on visual inspection automation using computer vision and machine learning techniques. Finally, current achievements and limitations of existing automation methods were debated to outline open challenges and future research for both infrastructure management and computer science researchers.

1. Introduction

Operating infrastructures in a qualified level of service is the primary concern of infrastructures asset management. To achieve a cost efficient operating system, a proper asset assessment needs to be conducted to evaluate the infrastructure condition and consequently, to make a decision regarding whether to rehabilitate or replace assets. Inspection of the infrastructure’s physical and functional condition can be considered part of infrastructure asset management and plays a significant role in extending an asset’s service life. Infrastructure condition information can be considered the most valuable tool for conducting asset management planning strategies since this information not only describes current situations but is required to predict future asset performance.

Sewer pipeline networks have become the main concern of modern municipalities around the world as these networks are too old and they are reaching their design lifetime; meanwhile, increasing environmental and health requirements, growing demands, and tight budgets have all made the problem harder to deal with [1]. Based on the American Society of Civil Engineers (ASCE) Report Card for America’s Infrastructure [2], wastewater conditions in the United States (US) are graded as D: poor. Meanwhile, the Canadian Infrastructure Report Card [3] reports sewer pipelines conditions in Canada as being poor and very poor, comprising of 52% of systems in small responding municipalities, 30% in medium-sized responding municipalities, and 26% in large responding municipalities. Moreover, because of the degradation of sewer networks over their service life, even if they were in very good condition today, they would still require increasingly larger investments as they age [3]. Therefore, to prevent further severe and costly damage, sewer system conditions need to be monitored through a timely and comprehensive periodic assessment [4].

Before the 1960s, inspecting sewer pipelines was a challenging task [5] and access and inspection was difficult due to pipe size. Sewer inspection methods have rapidly advanced with technologies such as visual and infrared cameras, sonar/ultrasonic techniques, sewer scanning and evaluation technology (SSET), laser-based scanning, focused electrode leak location (FELL), and ground penetrating radars (GPR) enhancing inspection approaches. Inspection technology developments have paralleled advances in sensor and computer technologies making it easier to capture rapid high-quality data with high-speed data interpretation capabilities. However, all of the invented technologies are not practical for sewer pipeline inspection and comparing the advantages and disadvantages of each can help inspectors to choose the most proper method considering pipe type, the amount of available budget, etc.

Currently, to prepare a comprehensive agenda that asset managers can use to plan investments and develop strategies, manual inspection is common practice in the inspection and assessment of sewer networks. Visual inspection requires hundreds of hours of data processing by certified inspectors to detect defects (i.e., crack, joint offset, roots, deposit, infiltration, etc.) and assess defect severity (i.e., length, number, consequences, etc.). However, manual inspection used in the assessment of extensive sewer systems is error-prone, subjective, and time-consuming [6]. In the last decade, considering the availability of optical sensing technologies, automation model development using computer vision techniques for sewer condition assessment has increased and become an attractive field for researchers. The objective of this paper is to introduce common practices for sewer pipelines inspection and assessment and comparing advantages and drawbacks of each method. Moreover, proposed computer vision models for the automation of defect detection and condition assessment in recent studies are reviewed to answer the following questions: (1) what are the common defects in sewer pipelines; (2) what are the current manual procedures in sewer visual assessment and their limitations; (3) what techniques are used to automate defect detection; and (4) what is the future of computer vision application in the field.

This review begins by describing the most common inspection technologies available today and compares their advantages and disadvantages. This section investigates various defects in sewer pipelines, conventional tools to detect them, and the limitation of manual inspection. This is followed by an inquiry into the models for sewer inspection automation proposed in recent studies found through different databases. These studies try to introduce methodologies based on various computer vision techniques, including template matching, high and low pass filtering, edge detection, and so forth. This section illustrates how these algorithms are deployed to detect and classify different types of defects in sewer pipeline visual assessment. Moreover, the advantages and drawbacks of each method are discussed. Finally, achievements and challenges in the field of automation of sewer assessment are presented and open research questions are introduced for future research.

2. Data Collection

In order to review the available literature in sewer inspection, a comprehensive search was conducted through accessible databases. The databases mainly focused on engineering and managerial topics and included peer-reviewed journals and conference proceedings. Also, a survey of available reports related to sewer systems and inspection technologies was used. The publications collected were papers and reports published between 1998 and 2018. Figure 1 shows the number of papers reviewed in each database. These databases contain hundreds of papers related to sewer asset management. However, the literature review was carried out only on sewer pipelines inspection technologies and proposed sewer pipeline assessment automation models using computer vision techniques. More specifically, the primary focus was publications containing innovative models for inspection automation and papers such as case studies and prediction and prioritization models were excluded.

The review was conducted through peer-reviewed journals from a variety of fields including computer science, civil engineering, and construction and infrastructure management. A search of available conference proceedings was also performed and the results filtered taking into account the above-mentioned criteria; those that met the criteria were included in the literature review.

Table 1. Publications in popular journals over time.

Journal	1998–2000	2001–2003	2004–2006	2007–2009	2010–2012	2013–2015	2016–2018	Total
Automation in Construction	2	0	3	1	2	1	5	14
Computer-Aided Civil and Infrastructure Engineering	0	1	1	0	0	0		2
Journal of Computing in Civil Engineering	1	1	0	2	0	1		5
Expert Systems with Applications	0	0	0	2	1	0		3
Journal of Infrastructure Systems	1	0	1	0	0	1		3
Journal of Pipeline Systems Engineering and Practice	0	0	0	1	1	0		2
Pipelines Conference	0	1	1	0	0	1	1	4

represents the number of papers published in popular journals and conferences in the area of sewer inspection and evaluation.

Some valuable contributions were published in journals and conference proceedings that are not mentioned in this table and will be pointed out in the next sections. Based on the conducted review, it was obvious that sewer condition assessment automation has always been a topic of interest for some journals such as Automation in Construction and Journal of Computing in Civil Engineering, while not for others like Expert Systems with Applications who only started publishing in the area in last decade.

After conducting a comprehensive literature review through the above-mentioned databases and excluding the non-related studies, the paper pool was classified into two categories: (1) publications which introduced or compared sewer pipeline inspection technologies and (2) publications which presented contributions in automating sewer inspection methods. This categorization was made because, in addition to publications introducing and comparing different methods in sewer pipeline assessment, most of the research also proposed models to automate inspection using visual methods or a combination of a visual-based tool with one or more sensors. Each of these categories is described in the next sections and related studies are synthesized through statistical analysis.

3. Sewer Pipeline Inspection Tools

Underground infrastructures are challenging to inspect due to their hidden conditions. Moreover, that they are buried has made them expensive to fix and replace. Direct human inspection, unlike other infrastructures such as road pavements and tunnels, is simply impractical for underground infrastructures because of the vast sets of buried pipelines and networks, unsafe and unhealthy sewer environment, low visibility, and small pipe size. Based on the Pipeline Assessment Certification Program (PACP) [7], there are a wide range of defects in the sewer system and detecting all of them using regular methods might be unfeasible. Sewer pipeline defects can be classified into structural, construction, and operation and maintenance defects (Figure 2).

The above-mentioned challenges have always been the key attributes motivating the development of more complex inspection tools for sewer pipelines assessment. The developments of sensor and lens technologies have made it easier and faster to introduce and improve new detection techniques. In this paper, sewer pipeline inspection techniques have been categorized into four groups as follows. The first is vision-based technologies which technically employ a camera and totally depend on what it can capture. The second is structural and bedding tools used to check pipe wall integrity and soil conditions around the pipeline. The third is defect-specific technologies developed to detect a particular type of defect and lastly, hybrid technologies that are a combination of two or more tools (Figure 3). This paper illustrates each category entirely through the description of each sub-category method and the advantages and disadvantages of each one. It concludes by comparing the explained technologies considering different criteria.

3.1. Subsection Vision-Based Technologies

Internal sewer wall inspection can be done through physical man entry in the pipeline or closed circuit television cameras [8]. As described in the previous section, man entry inspection is impractical considering the aforementioned reasons. Thereby, camera-based inspection tools definitely can be introduced as one of the most applicable visual inspection methods in condition assessment for wastewater collection systems. The most appropriate technologies based on camera inspection methods are conventional closed-circuit television (CCTV) inspection, zoom camera inspection, and digital scanning.

The use of CCTV for the inspection of pipe interiors was first introduced in the 1960s. The system consists of a television camera mounted on a tractor and remotely controlled by an operator. The distinct advantage of this method is that it provides evidence by directly illuminated images of pipe defects, which can be examined in detail by zooming the camera or viewing the defect from different angles by controlling the tractor [9]. Data obtained from CCTV inspection includes information of existing defects such as cracks, debris, roots, deflections, etc., and also location and condition of service connections [10]. The primary disadvantage of this technology is that it only provides a view of the pipe surface above the waterline and does not provide any structural pipe wall integrity data or a view of the soil envelope supporting the pipe [11]. Regarding benefits, CCTV technology is a cost-effective technology that provides the broadest base level data used in condition assessment.

Zoom cameras are used to perform manhole inspections by a camera mounted on the end of a telescopic pole. The primary function is the same as traditional CCTV method including the generation of still images or recorded video imagery of a pipe of any material. The main difference between a zoom camera and CCTV is that the zoom camera is stationarily mounted on a truck, crane, pole, or tripod positioned at a manhole and does not pass through the pipe segment being inspected. Zoom cameras can be used as a primary inspection tool for pipelines since the sewer pipe does not need to be cleaned before the zoom camera inspection is conducted and the inspection crew can move quickly through a service area and highlight pipe segments that require more detailed inspection [11]. Moreover, zoom camera inspection is not subject to delays caused by obstructions in the pipe as often occurs with a crawler-mounted CCTV cameras. Thus, zoom cameras can prioritize the pipes that need to be cleaned in order to do a more detailed inspection by traditional CCTV camera.

Although zoom camera inspection is a very efficient, cost-effective inspection method, there are some drawbacks. Like all camera technologies, it is only useful for inspecting gravity sewers because force mains and service laterals do not have manholes for access. Zoom camera inspection has the same limitation as traditional CCTV pipe inspection in that the camera cannot see the pipe below the water surface. Also, if the pipe deviates from a straight line due to sagging or deficient installation, the zoom camera will not “see” the hidden defects. Furthermore, the zoom camera does not provide the same detailed visual evaluation as conventional CCTV. Some zoom cameras lack pan and tilt viewing and cannot accurately measure and locate defects. Limitations in image resolution, lighting, and optical zoom also pose challenges.

Digital Scanning, another visual inspection method, employs a 360-degree fisheye digital scanner which produces digital side scan images of a pipe wall instead of the forward-facing continuous video provided by conventional CCTV cameras. A digital scanner examines the pipe wall, millimeter by millimeter, to produce a continuous image of the inside of the pipe, with front view images captured every ten centimeters. During the digital scanning process, data is transmitted to a surface station for real-time viewing and recording for later evaluation. Additional information on digital scanning capabilities and models can be found in reference [10].

3.2. Structural and Bedding Inspection Technologies

Because visual techniques are not able to evaluate pipe wall integrity and bedding conditions, other technologies have been employed to explore subsurface features. Ground penetrating radar (GPR) is presently the most available alternative to inspect bedding and void conditions. GPR is known as a non-destructive method for detecting subsurface materials using electromagnetic waves. For more information about GPR, please refer to [9]. GPR can detect location of pipes independent of material. GPR reports can produce precise data about pipe wall and condition of soil around pipe. However, penetration depth depends on the ground material and magnetic pulses lose strength in conductive materials. Moreover, interpreting GPR data requires trained and high skilled operators [10].

Sonar is another inspection tool for detecting defects of invisible areas. Sonar works by sending high frequency sound waves through the pipe and depending on the material condition of the pipeline, signals changes. Sonar can detect defects under the waterline as well as pipe geometry, joint displacement, and pipe deflection. This method can be employed without shutting down the sewer system. Sonar technology depends heavily on acoustic frequency to develop a sonar image. Different frequencies might be required based on pipe diameter, amount of water sediment, and turbulences in the pipeline. Report interpretation, however, is not straight forward and needs explanation by trained operators.

3.3. Defect-Specific Technologies

Other techniques have been introduced to detect specific defects since visual methods are only marginally successful in scaling some defects such as infiltration and exfiltration. Electrical leak detection systems can be employed to find and evaluate the magnitude of infiltration or exfiltration. The basic premise of these methods is that by measuring the electrical resistance of the pipe wall and amount of current flowing the defect can be detected. Electro Scan Inc. developed a tool to evaluate infiltration defects in the pipeline, based on measuring pipe wall strength [12]. This method can be implemented on non-conductive pipe materials which are resistant to electric current, such as plastic, concrete, reinforced concrete, clay, and brick.

3.4. Hybrid Technologies

In recent years, new inspection methods have emerged by integrating two or more technologies to detect various types of defects in sewer systems. This strategy may cause to offset the limitations of other technologies such as vision-based. Sewer Scanner and Evaluation Technology (SSET) utilizes a fisheye camera lens with an optical scanner and gyroscope technology to provide a view of the total pipeline surface. The scanned images are digitized for generation of color-coded computer images for further analysis [13]. RedZone combined laser and CCTV methods to inspect large pipes. Therefore, more accurate and comprehensive information of pipe conditions can be produced [14]. KARO and PIRAT systems are other multi-sensor inspection methods that can automatically interpret and categorize defects in sewer pipelines [15]. INNOKANIS project introduces a novel tool by combining zoom camera and the acoustic technology SewerBatt that integrates optical and acoustic device as an alternative to conventional CCTV [16].

3.5. Sewer Inspection Technologies Comparison

Municipalities are now able to employ a broad range of inspection tools. However, tool selection depends on available budget, type of assessment required, and pipeline material. A number of studies have compared sewer inspection technologies from different aspects and presented an overview of the current methods and new developments for sewer pipeline inspection techniques [8,9,10,11,17,18]. The authors reviewed inspection methods such as CCTV and GPR, as well as introduced some advanced systems in sewer inspection like KARO, PIRAT, and SSET. The advantages and drawbacks of each method are described below.

Camera-based technologies are the most common tools in sewer inspection and can provide evidence of most sewer pipe defects. These methods are productive and cost efficient and the provided information can be analyzed easily. However, they have deficiencies in supplying backfill condition information and defects under the waterline, in addition to their high dependency on image quality, lighting quality, and illumination. Also, it may be difficult to come up with precise information on the intensity of defects such as the depth of cracks or amount of pipe deflection. Structural and bedding inspection technologies can present a cross section profile of pipe walls and also determine the situation of invisible pipe parts like the underwater line or outer pipe wall. They can prepare clear and perceptible data about defect geometry and location. The drawbacks of these methods are the complexity of data interpretation and particular working condition demands.

Defect-specific techniques are well known because of the quantitative data on the detected defect they provide. Besides their limited application in detecting of one or two defects, they are also expensive to employ and require highly skilled operators to interpret provided data. Hybrid systems (KARO, PIRAT, and SSET) can cover the limitations of other methods, especially camera-based tools, since using two or more complementary technologies results in high quality and reliable information of a wide range of defects in sewer pipeline assessment. However, these technologies are in the prototype stage and have high initial operating costs. Specialized training for operating and data analysis is necessary, while additional equipment for field work is required.

Table 2. Sewer inspection technologies comparison.

	Camera Based		Structural & Bedding	Hybrid Technology	Defect Specific
	CCTV	Zoom camera	GPR	SSET	Electro scanning
No. of defects *	6	6	3	8	3
Complexity	Low	Low	Medium	High	Medium
Cost	Medium	Low	Medium	High	Medium
Downtime	High	Low	High	High	Low
Data analysis	Low	Low	Medium	High	Low
Operational equipment	Low	Low	Medium	High	Medium
Data quality	Medium	Low	High	High	High

* Defects which are inspected in sewer pipeline: Sediment, Debris, Roots, Pipe sags and deflections, Corrosion and metal loss, Off-set joints, Pipe cracks, Leakage, Bedding Condition.

presents a comparison of the above-mentioned technologies.

Introducing and comparing sewer pipeline inspection methods has always been a highly popular topic as new technologies are introduced by companies on a regular basis. Moreover, camera-based methods such as CCTV are the most commonly used tools in sewer pipeline inspection by municipalities and automating the inspection process by CCTV is the most interesting research subject in the area. Researchers try to improve CCTV performance by combining it with one or more sensors. Recent studies have proposed innovative algorithms to automate sewer pipe inspection by the above-mentioned hybrid techniques like SSET [19,20,21,22,23,24], introduced as a superior alternative to CCTV for accurate condition assessment [25]. Meanwhile, other algorithms proposed for automating other inspection technologies such as Laser profiler and CCTV [26] and sonar techniques to provide enhanced 3D graphs which represent the inner sewer surfaces [27]. Based on this review, the recent research in sewer inspection automation can be divided into three groups of studies: (1) introduction and comparison of inspection technologies, (2) CCTV inspection automation, and (3) hybrid technology inspection automation. Figure 4 illustrates the research in each discipline and each year.

4. Automated Sewer Defect Detection

As mentioned before, although CCTV has some deficiencies including lack of geometric references, subjective judgment and dependency on operator skill, image quality, and illumination effect, it is still the most common practice in sewer pipeline assessment. Recent advancements in imaging technology have led to a dramatic reduction in the cost of using cameras for municipalities. Municipalities are usually interested in inspection reports including realizable evidence and visual data to perform condition assessment and decision making. In each visual evidence a large amount of information such as texture, shape, location, severity, etc., can be delivered. In addition to advances in digital imaging technology, improvements in computer hardware capabilities make it feasible to employ machine learning algorithms and computer vision techniques. In recent years, innovative algorithms have been brought forward by researchers in order to reduce the level of ambiguity and improve CCTV inspection speed by automated assessment.

Methods used by the researchers are mainly involve defect detection using image processing techniques such as morphology, feature extraction, and object detection and using a machine learning algorithm for classification part. However, recent studies employed artificial neural networks and mainly deep learning algorithms for direct feature sampling among huge sets of data. These methods provide reliable accuracy results and generalization capability. In the following section, studies in both areas are reviewed and critiqued.

4.1. Computer Vision Techniques

Computer vision techniques applied in sewer pipeline defect detection mostly involve image processing techniques for image representation and then machine learning tools for defect classification. Based on which computer vision technique is used, recent studies in sewer pipeline defect detection automation fall into three categories: preprocessing, image representation, and learning. Studies which classified in the preprocessing category generally utilized morphological operation such as image enhancement, image registration, image restoration, and image segmentation to analyze image data. Studies that classified as image representation category employed feature extraction methods including template matching, feature matching, and pattern matching to understand image data and recognition of image regions. Other research which is classified as part of the learning category proposed object detection and recognition methods that employed tools such as deep learning algorithms to recognize instances of defects in images. Figure 5 presents various computer vision methods which are applied in the reviewed studies in this paper. It should be mentioned that, technically, these methods cannot be explicitly separated and there is always collaboration among them. Each of these methods is discussed completely below.

4.1.1. Morphology

Morphological operators define the shape or morphology of features in an image using non-linear mathematical operations. These operators encode the characteristics of objects in the input image of a structuring element. Morphological operators determine the value of pixels in the output image based on the condition specified by the operator and regarding a comparison of the corresponding pixel in the input image with its neighbors.

A number of studies have employed sequences of morphological operations to accurately segment pipe defects [20,21,22,23,28,29,30,31]. Also, morphological segmentation based on edge detection (MSED) is used to identify the morphology representatives for sewer pipe defects on CCTV images such as cracks, joints, and holes [24,32,33,34,35,36] and maximally stable extremal regions (MSER) to detect texts in frames and cracks [37].

Most of the sewer pipe inspection images are based on binary (light/dark) primitive shapes therefore image morphology can be considered as a natural tool for image processing [38]. In addition, image segmentation methods are easy to apply when the shapes are necessary. Techniques such as edge detection and thresholding are applicable where structural defects like cracks need to be detected. On the other hand, morphological methods rely heavily on image quality. Lighting conditions inside sewer pipelines, illumination, and low contrast images may result in inaccuracy in defect detection.

4.1.2. Feature Extraction

Analyzing different regions of the image is a significant step in image recognition and the image needs to be properly defined for the classification stage. Therefore, feature extraction generates a set of measured data and builds a numeric feature vector of the properties of the region. The approach of using computer vision for feature extraction and trained neural networks for classification has been employed in studies in sewer pipeline defect detection [19,39,40,41,42]. A number of contributions in engineering of novel features such as wavelet transform and co-occurrence matrices [43], Scale invariant feature transform (SIFT) [4], histograms of oriented gradients (HOG) [44], GIST descriptor [45], and Maximally stable external regions (MSER) [37] have also been made.

Feature extraction is useful in the reduction of dimensions and number of calculations since analyzing and classifying thousands of pixels in an image seems to be impossible. The highly patterned and poorly lit environment of sewer pipelines causes infinite templates of defects. Camera position and lighting condition can affect the appearance of defects and patterns may look like any sewer pipe failure class. Thereby, defining a feature-based approach for classification of defects is too complicated because of unknown features or a large variety of patterns in sewer pipes. Besides, data labeling in training the classifier is usually performed manually (supervised) and considering a high variety of patterns, that method is highly time-consuming and prones to errors.

4.1.3. Object Detection and Recognition

Figure 6 presents recent research in automated sewer defect inspection using each introduced method of computer vision.

Improved quality and quantity of produced sewer inspection data made it feasible to employ deep learning algorithms such as deep models including Deep Auto-encoders (DAEs), Deep Belief Networks (DBNs), Convolutional Neural Networks (CNNs), Restricted Boltzmann Machines (RBMs), Recurrent neural networks (RNNs), and Generative Adversarial Nets (GANs) are widely used for image classification in recent years. Unlike the other machine learning algorithms, in deep learning algorithms the network learns how to extract and subsample the features automatically during the training. In fact, deep learning omits the critical and time consuming feature engineering step used in conventional machine learning techniques. Deep learning algorithm trains the network using raw image pixels in the input layer to the defect detection in the output layer. Figure 7 shows the difference of feature extraction in conventional machine learning approaches and the deep learning approach.

Different network layers learn various features as low-level features in first layers which are learned directly from input image pixels. In next layers, medium-level features are learned from the features extracted in first layer. The higher-level features are learned in the same way in the next layers of deep network. In other words, the first layers learn to represent the features on their simpler attributes and deeper layers represent more complex features and concepts using learned features from previous layers.

In recent years deep convolutional neural networks (CNN) [46] and faster region-based convolutional neural networks (Faster-RCNN) [47] are employed in sewer defect detection. CNNs can get images directly in any size and they make a link among pixels of images for feature extraction and image classification. CNNs are proven to have higher classification accuracy and better generalization capability [48]. CNN incorporates constraints and achieves some degree of shift and deformation invariance using three ideas: local receptive fields, shared weights, and spatial subsampling [49]. Using deep neural networks seems to be a practical approach in future years in image classification tasks as is used in sewer defect classification [6,46] and defect detection and classification [47].

With all advantages of deep neural networks, they require huge amount of data for training and they are computationally expensive specially when the network has many layers and is relatively deep. However, these limitations are partially covered using data augmentation algorithms in generating training datasets and employing graphic processing units (GPUs) [48].

5. Discussion

5.1. Achievements

Inspection of underground infrastructure has always been a challenging task for municipalities and infrastructure construction companies. This matter persuades the industry to develop new technologies for pipe inspections. Looking into the accomplishments introduced in previous sections, developments in inspection tools and methods provide enough data for extracting the related information in order to detect and localize nearly all defects. Most of them may be identified visually and detection can be automated using computer vision techniques. Recent studies in automation of sewer pipeline mostly focus on structural defects. However, a number of innovative algorithms have been proposed to detect defects such as deformation and infiltration. Other defects including deformation, deposit, hole, and root intrusion were studied because the required data was available (Figure 8).

Although visual technologies like conventional CCTV are unable to inspect defects under water level or behind pipe wall, improvements in hybrid technologies have made it possible to assess invisible defects. Moreover, innovative technologies such as SSET have dramatically improved the quality of collected data, resulting in higher precision in defect detection, while on the other hand, researchers can develop algorithms to automate the defect detection process with better accuracy.

Furthermore, developments in image analysis and computer vision techniques have made the perfect conditions for proposing methods for automation of sewer inspection. Detection and localization of defects such as joint displacement and cracks have been automated to some extent. Also, evaluation and assessment of these defects has partially been automated by algorithms proposed in several studies. Some other computer vision-based models which proposed to automate further defects like deformation, root intrusion, infiltration, and deposit succeeded in detection and localization of defects by virtue of new technologies application in sewer inspection. Deep neural networks represented remarkable performance in recent years according to their recent application in sewer defect detection. Their generalizability can be improved by training with larger data sets (

Table 3. Automation of sewer defects assessment.

Defect Type	Detection	Localization	Evaluation	Algorithm Generalization
Cracks	◙	◙	◘	◘
Joint	◙	◘	○	◘
Deformation	◘	◘	◘	○
Holes	◙	◙	◘	◘
Root	◙	◙	○	○
Infiltration	◙	◙	○	○
Deposit	◘	◘	○	○

(◙) achieved, (◘) partially achieved, (○) not achieved

).

5.2. Limitations

Contrary to the accomplishments mentioned in the previous section, there are still challenges in the automation of sewer inspection using computer vision methods. When it comes to underground visual inspection, at first glance, varying illumination and poor lighting conditions emerge as the main problems for preparing data with an acceptable quality. In addition, discontinuous background and numerous patterns and shapes of defects considering pose and distance to the camera also arise as the most critical problems in applying computer vision algorithms on image data for defect detection. Therefore, methods based on limited historical data can barely be generalized since unknown conditions are expected in each sewer pipeline inspection operation. Inspection is not still entirely automated and, in the most recent studies, localizing the region of interest is still operator-based and most proposed classifiers are based on manual labeling (supervised learning) for training.

One of the efforts made in the last decade has been the development of efficient and well-organized methods for feature extraction from inspection data for setting up a robust classifier. Feature selection is the most important step in classifier design since features need to provide enough discrimination, and at the same time, be easy to compute. Previous studies have mainly focused on the geometric shape and size of objects present in sewer images while there are infinite unknown patterns and shapes for each type of defect in sewer pipes. Therefore, developing feature selection algorithms for maximizing performance and generalization of classifiers may be the most demanding task for the field. Another challenge is the use of color in computer vision techniques. Previous efforts have mostly converted images to grayscale before performing image analysis to simplify the calculations and speed up computation. However, in computer vision, sometimes color may streamline object recognition. Color can affect the detection of some defects like infiltration due to pipe wall being stained by waste water. Therefore, a significant number of color shades and intensities can be considered in defect detection and classification in comparison with a limited number of shades of grayscale images.

Although introduced deep neural networks proven to be robust in dealing with feature extraction challenges in sewer video frames, they still need too much time for training and tuning hyper-parameters of the network layers. Efforts for preparing the data sets time and model training computational in previous research show the drawbacks in developing a generalized defect detection and classification model. Also, the time required for classification and detection made it still far from the real time defect detection standards in sewer CCTV inspection videos.

In summary, computer vision methods mainly rely on information that visual inspection tools (i.e., CCTV) provide and as such, what cannot be captured by CCTV cannot be analyzed by the algorithm. To provide sufficient data for inspection and defect detection, the pipe needs to be illuminated properly and also a standard range for angle of camera pose needs to be defined. Although current approaches are still human interactive, advances in computer vision algorithms and improvements of hardware capacity may assist the researchers to develop more efficient models by decreasing training and detection time as well as increasing detection accuracy and performance of the models. This leads to a considerable reduction in time and cost of inspection of vast pipeline networks.

6. Conclusions

To sustain an acceptable level of service in sewer infrastructure systems, it is critical to inspect and assess physical and operational conditions of pipelines. Sewer condition assessment can be done using different inspection tools and varying levels of data quality. The condition assessment process is performed by certified operators manually, which is time consuming, subjective, and error prone. Therefore, many researchers have tried to automate the process using computer vision methods. This article introduced available practices in sewer inspection and condition assessment. The introduced technologies were compared thoroughly based on their superiorities and drawbacks. This review paper found that some tools like CCTV and zoom camera were the most used methods in sewer inspection due to cost efficiency and their ability to detect more types of defects. Computer vision techniques and neural networks were then illustrated and critiqued. Following this, a comprehensive literature review on automation models proposed in the recent studies was conducted and the advantages and disadvantages of each category were discussed. This was concluded with the achievements made in in automating sewer defect detection and condition assessment and existing challenges were summarized as open questions for future research in the field.