Urban Built Environment Assessment Based on Scene Understanding of High-Resolution Remote Sensing Imagery

Abstract

A high-quality built environment is important for human health and well-being. Assessing the quality of the urban built environment can provide planners and managers with decision-making for urban renewal to improve resident satisfaction. Many studies evaluate the built environment from the perspective of street scenes, but it is difficult for street-view data to cover every area of the built environment and its update frequency is low, which cannot meet the requirement of built-environment assessment under rapid urban development. Earth-observation data have the advantages of wide coverage, high update frequency, and good availability. This paper proposes an intelligent evaluation method for urban built environments based on scene understanding of high-resolution remote-sensing images. It contributes not only the assessment criteria for the built environment in remote-sensing images from the perspective of visual cognition but also an image-caption dataset applicable to urban-built-environment assessment. The results show that the proposed deep-learning-driven method can provide a feasible paradigm for representing high-resolution remote-sensing image scenes and large-scale urban-built-area assessment.

1. Introduction

The linkage between urban built environment and life quality has attracted a lot of attention in recent years. Many studies have focused on ways to establish high-quality communities through the design of built environment that can significantly benefit public health and well-being [1,2,3,4]. Marans and Stimson described how to measure and assess the connection between the urban setting and life quality [5]. Kent and Thompson pointed out that the built environment can improve health and well-being in three ways: physical exercise, community social cohesion, and fair access to wholesome food [6]. Pfeiffer and Cloutier believed the key elements of community well-being are open, green, and natural spaces and urban layouts that promote safety and social contact [7]. Wang and Wang summarized theoretical and empirical data on how geography affects subjective well-being [8]. Mouratidis found that the built environment at community size has an impact on subjective well-being in four different ways, including social connections, leisure, physical health, and emotional experiences [9]. The impact of urban design on the quality of the built environment has been validated on many topics, such as human health [10], pedestrian activity [11,12,13,14,15], and travel behavior [15,16].

Several studies have examined how citizens perceive the built environment visually or acoustically through surveys, in-person interviews, and on-site inspections [17]. The most popular methods for conducting surveys, which capture amorphous perceptions of the built environment, are questionnaires and interviews. To measure respondents’ subjective assessment of the survey area, researchers often construct numerical-rating scales and open-ended questions. However, there have been persistent issues of response bias associated with these two approaches [18,19]. The main way to describe human impressions or emotional responses about urban design at various scales is face-to-face interviews [20]. However, due to face-to-face interviews being expensive and time-consuming, most scholars engaged in urban studies choose to conduct small-scale empirical studies [18,21].

Freely available, high-spatial-resolution crowdsourced map-service data and geotagged images have become increasingly important data sources. For example, Google Street View (GSV) can provide a panoramic view of streets and neighborhoods. Several GSV-related studies have been presented for the assessment of visual perception of urban streetscapes [18,21]. Thanks to the combination of a large amount of street-image data and deep-learning algorithms, means of analyzing the perception and understanding of urban landscapes has emerged and attracted much attention. Due to the outstanding performance in urban-street-view classification and mapping, these approaches play an important role in automatic semantic-information extraction, visual-element classification, and scene-feature representation [22,23,24,25,26,27,28,29]. However, because the street view is captured along the road, the assessment of the environment is confined to the region along both sides of the road, which is not effective for assessing the internal-area environment of the block area. Therefore, it is difficult to implement a full-coverage, continuous measurement of the built urban environment with this assessment means. Fortunately, the large coverage capability of high-resolution remote-sensing imagery allows it to objectively monitor the urban built environment from a top-down perspective. Moreover, remote-sensing images have a shorter observation period and faster update frequency, making them better able to adapt to the changing urban built environment. Due to these benefits, remote-sensing imagery has long been an important data source for urban land use/cover mapping [30,31], but its potential for assessing the urban built environment has not been fully realized mainly because the rich detailed information in high-resolution remote-sensing imagery has not been fully exploited.

The use of quantitative indicators and the calculation of indicator weights have become the main content of urban-built-environment assessment studies. These indicators come from social statistics, surveys, Street View, and GIS data; however, these data are updated less frequently and are not easily available [32,33,34,35,36,37,38]. Therefore, rather than emphasizing the selection and weighting of quantitative indicators, this study focuses on the development of an evaluation method based on the visual semantics of remote-sensing images of the urban built environment. The basic unit of urban built environment in our study is a remote-sensing-image scene within a spatial extent. Firstly, evaluation criteria are established according to their visibility in remote-sensing images and the role of different geo-objects in the environment. Secondly, the image-caption algorithm is used to explore the semantics and relationships of geo-objects in the scene image to obtain a visual semantic description of the urban built environment. Then, based on the environmental evaluation criteria, the environmental-evaluation score of the scene unit is determined according to the visual semantic description of the image. Finally, the inverse-distance weighted-interpolation algorithm is used to carry out an urban built environment.

The main contributions of this paper are as follows: (1) According to the characteristics of high-resolution remote-sensing images, the representative indicators of the urban-built-environment criteria of geographic scenes are established. (2) A natural-language dataset for urban-built-environment evaluation is established. (3) The image-caption model is employed to evaluate the urban built environment for the first time.

The remainder of this paper is organized as follows. Section 2 introduces the methodology, including the constructed-scene-assessment criteria, dataset, and assessment model. Section 3 presents the experimental results and analysis. Section 4 discusses and explores the applicability of this research method in different cities and mentions the strengths and limitations of the proposed method. Section 5 concludes with some remarks and potential future research directions.

2. Methodology

2.1. Overview

Vision is the main form of human perception, and language is the most powerful tool for human communication with the world [39]. Humans can utilize language to transmit information according to their perception of the geographical environment [40]. This study intends to carry out an environmental assessment from the perspective of spatial understanding of remote-sensing-image scenes. The image-caption model is used to create language descriptions of geographic scenes. The general framework of the proposed methodology is shown in Figure 1. Firstly, it establishes criteria for assessing the urban built environment from remote-sensing images from the perspective of human visual perception. Secondly, we construct an image-caption dataset of the built environment based on the assessment criteria and utilize the dataset to train an image-caption model to support the assessment. Finally, the image-caption model and the assessment criteria are jointly utilized to assess the built environment of Changsha.

2.2. Assessment Criteria

The visual elements in the built-environment scene play a fundamental role in evaluating the built environment from a visual-perception perspective, as shown in Figure 2. Therefore, considering the visibility of distinct urban visual elements in high-resolution remote-sensing images, the following five assessment indicators were established. The specific description of each indicator is given below.

Building density. As the urban population increases, the impact of building on the assessment of the urban built environment is increasingly worth considering. Density is the most immediate and representative factor of the urban built environment [41]. The space between buildings is important for comfortable outdoor time [42]. Studies have shown that building density is directly correlated with community satisfaction [43,44]. Low-density settlements are appreciated for their spacious living environment [45]. High building density can also lead to the heat-island effect, which reduces the wind speed among buildings and increases the ambient temperature [46]. This paper refers to the criteria of Stewart and Oke [47], namely, based on whether the density is greater than 40%, the building area is divided into an open-building area and a compact-building area. In this criteria, open buildings were positively correlated with the assessment results, whereas dense buildings were negatively correlated with the assessment results.

Road connectivity. Roads are not only places for the transportation activities of residents in residential areas but also used as spaces for social interaction, which has an important influence on environmental quality. Wei et al. [48] and Duan et al. [49] identified road connectivity as one of the factors that determine the walkability of a neighborhood. Malambo et al. [50] also showed that urban attributes such as road connectivity improve the walkability of the community, thereby promoting the development of physical activities such as walking and recreation, which can help reduce the incidence of disease. Xu et al. [51] mentioned that road connectivity has an effect on community vitality. In this paper, good and bad connectivity is distinguished by the road structure in the geographic scene. The road structure of the geographic scene is a grid structure or a tree structure. The grid structure has a homogeneous distribution, is usually straight, and has good connectivity; on the contrary, the tree-structured road network is dendritic, mainly in some old communities, and has poor connectivity [52].

Vegetation coverage. Vegetation coverage has always been an important indicator of the quality of the living environment and plays a very important ecological role in residential areas, such as oxygen release, cooling, humidity control, shading, wind protection, dust blocking, sun protection, noise reduction, acting as an insecticide, and other functions [51]. Urban vegetation cover is one of the reasons for comfort, improves air quality, reduces the effects of noise pollution and extreme weather events, and promotes mental health by reducing stress [53]. Hur et al. [44] found that vegetation is an important factor in space use, security and adaptability, and informal social contact between neighbors. Numerous studies [54,55,56,57] have shown that the vegetation cover in residential areas often leads to positive evaluations of the environment. Therefore, the presence of vegetation cover around buildings is positively correlated with the assessment results.

Distribution of industrial areas. With the development of industry, residential areas and industrial areas coexist in large cities, and sometimes the distance between industrial areas and residential areas is less than 100 m [55]. However, industrial areas have a negative impact on the surrounding area and affect the aesthetic quality of the environment [58]. People are less likely to engage in recreational activities near industrial areas [59]. Existing studies [58,60] have shown that the presence of an industrial area often leads to a negative assessment of the environment. Therefore, the presence of industrial areas is negatively correlated with the evaluation results.

Distribution of activity areas. Open activity can enhance human comfort. Research [60] has shown that open, green spaces have a positive impact on the quality of urban life. Douglas et al. [61] surveyed 483 residents living in three communities in Dublin, Ireland, and found that open, green spaces are important predictors of high community satisfaction. The presence of neighborhood green spaces, such as parks and woodlands, is associated with better physical health, lower rates of certain categories of disease, and lower levels of depression. In addition, the study by Ambrey and Fleming [62] showed that a neighborhood surrounded by water is an exciting place to live. Malambo et al. [50] found that the presence of sports and recreational facilities can increase the physical activity of residents. Therefore, the presence of an activity area often leads to people’s positive assessment of the environment. In this paper, rivers, lakes, green spaces, and playgrounds, which are common factors of urban-life satisfaction, were selected as activity-space elements. In our defined criteria, the presence of these activity areas was positively correlated with the evaluation results.

In this paper, 0 and 1 represent the evaluation results for each evaluation metric. The score was set to 1 and 0 for positively correlated results and negatively correlated results, respectively.

After clarifying the performance of each indicator, the final evaluation score was obtained by summing the scores of each indicator. The image-caption model was used in this paper to mine remote-sensing-scene information. Commonly used weight-setting methods such as AHP [63] and principal-component analysis [64] were not suitable for this paper. In our method, the indicators were expressed in natural language, and the quantitative results of each indicator were not calculated, so the traditional weight-setting method was not applicable. Therefore, after comprehensive consideration, the total score was obtained by adding each indicator to show the comprehensive performance of each indicator. This method can more easily correspond to the content of natural-language description and provide the most direct understanding of the assessment results of urban scenes, which can be calculated as follows:

$$X={\displaystyle \sum }_{i=1}^5{x}_i,$$

(1)

where X represents the total score of the evaluation results, reflecting the comprehensive performance results of the five indicators; and x_i represents the score of each indicator and takes a value of 0 or 1. Based on the above, the assessment criteria were determined, as shown in

Table 1. Assessment criteria for remote-sensing-image scenes.

Indicator	Criteria	Comprehensive Score
Building density	1: Low density and open layout	$X={\displaystyle {\displaystyle \sum }_{i=1}^5}{x}_i$ The range is 0–5 points
	0: High density and compact layout
Road connectivity	1: Grid structure
	0: Tree structure
Vegetation coverage	1: Vegetation coverage
	0: No vegetation coverage
Distribution of industrial area	1: No distribution of industrial area
	0: Distribution of industrial area
Distribution of activity area	1: Distribution of forest
	1: Distribution of lake
	1: Distribution of river
	1: Distribution of playground
	0: No distribution of activity area

.

2.3. Dataset and Model

2.3.1. Dataset Establishment

A large number of studies have used image-caption models for spatial understanding of remote-sensing-image scenes in recent years [65,66,67,68,69]. Given a high-resolution remote-sensing image, the computer mines the visual-feature information of the image and expresses the information in the form of natural language. The automatic generation of an image-content description that matches human cognition and expression habits has become a new way to transfer remote-sensing-image information.

Existing image-caption datasets mainly include the UCM-caption dataset [67], Sydney-caption dataset [67], and GRIUD dataset [65]. The UCM-caption and Sydney-caption dataset simply describe the objects in the image, with labels such as “there are two planes on the airfield”, and “this is a beach”. It is only suitable for spatial cognition of simple geographic scenes. The GRIUD dataset proposes a semantic understanding of remote-sensing images from the perspective of geospatial relationships. Although this dataset can mine richer spatial information, it cannot achieve the goal of environmental assessment.

The existing image-caption datasets are limited to expressing the objects and their relationships and cannot meet the requirements of this paper. Therefore, based on the above assessment criteria for spatial understanding of remote-sensing-image scenes, this work deeply explores the composition of remote-sensing scenes, determines the relevant vocabulary and grammatical structure, and establishes a dataset dedicated to this study.

With the improvement of earth-observation capabilities, massive high-resolution remote-sensing-image data can provide large-scale, rich, and updated ground information. In this paper, high-resolution remote-sensing images of the central urban area of Changsha were selected to build a dataset. The high-resolution remote-sensing images were collected from Google Earth with a spatial resolution of 0.5 m, containing three visible bands of red, green, and blue. To meet the input requirements of the image-caption model, this paper cropped these images to obtain image patches, and each image patch corresponds to a geographic-scene unit containing different geographic elements.

Cropping at different scales results in geographic-scene cells of different spatial extents. If the spatial extent of the scene was too small (Figure 3a), the geographic elements were uniform and could not reflect the surrounding environmental information, resulting in very small differences between such scene units. Conversely, if the spatial extent of the scene was too large (Figure 3c), it was difficult to focus on the local spatial environment because it contained many geographic elements. In order to determine the appropriate scene-unit size, this study carried out scene-unit division with different scale sizes and finally decided to use an image block with a size of 250 × 250 as the basic unit. The scene unit of this scale can include a reasonable number of geographic elements and information about the surroundings, thus facilitating the urban-built-environment assessment in this study. After cropping the remote-sensing image of the main urban area of Changsha, we obtained a total of 3874 images with a size of 250 × 250.

Since this work analyzed the environmental assessment of urban scenes based on remote-sensing images, further filtering of the cropped images was required after obtaining remote-sensing scenes. The principle of filtering is to delete the images that did not contain any building areas and the images that contained architectural areas but did not belong to urban areas. For example, due to the main elements in Figure 4a being water bodies and Figure 4b showing that its main elements are water bodies and forests, both of these images should be filtered out accordingly. In addition, Figure 4c contains many buildings but is located in a rural area; thus, it also does not meet the requirement. Finally, 836 images of these types were discarded.

Next, the remaining 3038 remote-sensing-scene images needed to be labeled to establish the Environmental Assessment Dataset. To this end, 10 senior GIS students were invited as volunteers to manually describe and assign scores to each image. The newly recruited volunteers were trained in two steps. The first step was sketch construction, which focused on identifying geo-objects and their spatial relationships. The second step was sentence expression, which mainly included elimination of irrelevant geo-objects and natural-language description. After the training, all of the volunteers were assigned to practice describing the same images many times until they could achieve the same perception in most cases.

Sketch construction: Spatial relationships can represent the interdependent distribution of geo-objects in the spatial environment and are the key content of spatial understanding in geographic scenes. In this study, we referred to the spatial-relationship expression in [65], which provides a technical basis for generating descriptions of various topics. The sketch construction involves both identifying geographic entities in a scene and extracting sketches based on the spatial relationships among these entities, as shown in Figure 5.

Sentence description. After sketch construction, irrelevant geo-objects such as parking lots and golf courses were eliminated from the sketch and then the simple geographic relationship of the sketch was expressed as a natural-language description. Geographic-scene description is a collection of geographic entities, spatial relations, and attributes. This paper referred to the description rules in [65] and converted the sketches into natural-language descriptions containing geographic knowledge. For example, we used nouns to represent spatial entities, such as “building area”, “industrial area”, “trees”, “roads”, “playgrounds”, “rivers”, and “lakes”. Spatial relationships were represented by prepositions, such as “around”, “near”, and “adjacent”. Adjectives were used to describe the characteristics or attributes of geographic entities; e.g., “open” and “compact” were used to characterize the properties of building density (Figure 6).

2.3.2. Image Caption

In this study, the image-caption model was adopted to provide a potential way to mine geographic knowledge in remote-sensing images and provide strong support for understanding scenes. Despite many studies having developed remote-sensing image-caption models [65,66,67,68,69], these models only output one sentence to describe the whole image. This study conducted built-environment assessment based on remote-sensing-image scenes from five perspectives, i.e., it aimed to generate sentences on five different topics by using image caption. Thus, to enable adaptation of the task, a multi-task hierarchical model [70] was improved in this study to automatically generate descriptions of five different topics from remote-sensing images (Figure 7).

Given a high-resolution remote-sensing image, a CNN model was used to learn the visual features ${\left\{{\mathit{v}}_n\right\}}_{n=1}^N\in {ℝ}^D$. The visual features were then fed into a multi-label classification network (MLC) to predict the tag labels:

$$p_{l,pred}\left({\mathit{l}}_i=1|{\left\{{\mathit{v}}_n\right\}}_{n=1}^N\right)\propto \exp \left({\mathrm{MLC}}_{\mathrm{i}}\left({\left\{{\mathit{v}}_n\right\}}_{n=1}^N\right)\right),$$

(2)

where $\mathit{l}\in {ℝ}^{\mathit{L}}$ is a label vector, l_i = 1/0 represents the presence and absence of the ith label, and MLC_i represents the ith output of the MLC network. The obtained result was used to output the semantic features on five different topics, where the semantic features ${\left\{{\mathit{a}}_m\right\}}_{m=1}^M\in {ℝ}^E$ are the embedding-vector representations of the labels.

Then, the visual features ${\left\{{\mathit{v}}_n\right\}}_{n=1}^N\in {ℝ}^{\mathit{D}}$ and semantic features ${\left\{{\mathit{a}}_m\right\}}_{m=1}^M\in {ℝ}^E$ were fed into an attention-mechanism model to generate a context vector $\mathit{c}\mathit{t}{\mathit{x}}^{\left(s\right)}\in {ℝ}^{\mathit{C}}$. The attention model employed a single-layer feed-forward network to compute visual and semantic attention on input image features and labels.

$${\mathit{\alpha }}_{a,m}\propto \exp \left({\mathit{W}}_{a_{att}}\tan \mathrm{h}\left({\mathit{W}}_a{\mathit{a}}_m+{\mathit{W}}_{a,h}{\mathit{h}}_{sent}^{\left(s-1\right)}\right)\right),$$

(3)

$${\mathit{\alpha }}_{v,n}\propto \exp \left({\mathit{W}}_{v_{att}}\tan \mathrm{h}\left({\mathit{W}}_v{v}_n+{\mathit{W}}_{v,h}{\mathit{h}}_{sent}^{\left(s-1\right)}\right)\right),$$

(4)

where ${\mathit{W}}_v$, ${\mathit{W}}_{v,h}$, and ${\mathit{W}}_{v_{att}}$ are the parameter matrices of the visual-attention network; ${\mathit{W}}_{v_{att}}$, ${\mathit{W}}_{a,h}$, and ${\mathit{W}}_{a_{att}}$ are the parameter matrices of the semantic-attention network; and ${\mathit{h}}_{sent}^{\left(s-1\right)}\in {ℝ}^{\mathit{H}}$ are the hidden states of the sentence LSTM at time-step s − 1. I context vector $\mathit{c}\mathit{t}{\mathit{x}}^{\left(s\right)}\in {ℝ}^{\mathit{C}}$ is calculated as follows:

$${\mathit{v}}_{att}^{\left(s\right)}={\displaystyle \sum }_{n=1}^N{\mathit{\alpha }}_{v,n}{\mathit{v}}_n, {\mathit{a}}_{att}^{\left(s\right)}={\displaystyle \sum }_{n=1}^N{\mathit{\alpha }}_{a,m}{\mathit{a}}_m,$$

(5)

$$\mathit{c}\mathit{t}{\mathit{x}}^{\left(s\right)}={\mathit{W}}_{fc}\left[{\mathit{v}}_{att}^{\left(s\right)};{\mathit{a}}_{att}^{\left(s\right)}\right],$$

(6)

Next, starting from the context vector, decoding generated a textual description. The model produced sentences hierarchically: It first generated high-level topic vectors representing different sentences and then generated a sentence from each topic vector. Specifically, the context vector was input into a sentence LSTM, which is a single-layer LSTM. The input context vector was used to generate a sequence of hidden states ${\mathit{h}}_{sent}^1,\cdots \cdots ,{\mathit{h}}_{sent}^s\in {ℝ}^{\mathit{H}}$, each sentence corresponding to a hidden state, and the hidden state was used to generate a topic vector $t^{\left(s\right)}\in {ℝ}^{\mathit{P}}$ for each sentence, as the input to the word LSTM.

$${\mathit{t}}^{\left(s\right)}=\tan \mathrm{h}\left({\mathit{W}}_{t,h_{sent}}{\mathit{h}}_{sent}^{\left(s\right)}+{\mathit{W}}_{t,ctx}\mathit{c}\mathit{t}{\mathit{x}}^{\left(s\right)}\right),$$

(7)

where ${\mathit{W}}_{t,h_{sent}}$ and ${\mathit{W}}_{t,ctx}$ are the parameter matrices, and ${\mathit{h}}_{sent}^{\left(s\right)}$ is the hidden state of the sentence LSTM at the moment.

Given a topic vector ${\mathit{t}}^{\left(s\right)}$, the word LSTM took it as input and progressively generated sentences. The initial input to the word LSTM was the topic vector and an embedding vector ${\mathit{W}}_e{s}_0$ of the START input indices $s_0$, and the subsequent inputs were the topic vector and the embedding vector ${\mathit{W}}_e{s}_1,\cdots \cdots ,{\mathit{W}}_e{s}_{N-1}$ of true word labels $s_1,\cdots \cdots ,s_{N-1}$. At each time step, the word LSTM hidden state ${\mathit{h}}_{word}^1,\cdots \cdots ,{\mathit{h}}_{word}^N\in {ℝ}^{\mathit{H}}$ was used to predict words ${\mathit{p}}_1,\cdots \cdots ,{\mathit{p}}_N$. After each word LSTM generated the words of the respective sentences, the sentences were finally concatenated to form the generated paragraphs.

$${\mathit{x}}_t={\mathit{W}}_e{s}_t,$$

(8)

$${\mathit{h}}_{word}^{t+1}=\mathrm{LSTM}\left({\mathit{t}}^{\left(s\right)},{\mathit{x}}_t,{\mathit{h}}_{word}^t\right),$$

(9)

$${\mathit{p}}_{t+1}\propto exp\left({\mathit{W}}_{out}{\mathit{h}}_{word}^{t+1}\right),$$

(10)

where ${\mathit{W}}_{out}$ is the parameter matrix, and ${\mathit{h}}_{word}$ is the hidden state of the word LSTM.

The model used MLC to perform multilabel classification on images, encoding the presence and absence of labels. The training loss for this step was the cross-entropy loss $L_{tag}$ between the multiclass predicted values and the true labels. Next, the topic vector S was generated on the sentence LSTM, and then, the five generated topic vectors were input into the word LSTM step by step to generate words. The training loss for this step was the cross-entropy loss $L_{word}$ between the generated word p_t and the ground-truth label s_t. Therefore, the overall training loss was as follows:

$$L\left(\mathrm{I},\mathrm{l},w\right)={\lambda }_{tag}{L}_{tag}+{\lambda }_{word}{L}_{word},$$

(11)

where ${\lambda }_{tag}$ and ${\lambda }_{word}$ correspond to the weights of the multilabel-classification loss L_tag and the sentence-output loss L_word, respectively. The overall training procedure is summarized in Algorithm 1.

Algorithm 1: Training Image-Caption Model

1. Input: Dataset $D=\left\{x^n=\left(I^n,l^n,w^n\right),n=1\cdots N\right\}$; //$I^n$ is a remote sensing image, $w^n$ are the ground-truth sentences. $l^n$ denotes the ground-truth tag vector, which is extracted from $w^n$. 2. Output: $f_{CNN}(·),f_{COatt}(·), f_{MLC}(·),f_{sent-LSTM}(·), f_{word-LSTM}(·); // f_{CNN}(·),f_{COatt}(·), f_{MLC}(·),f_{sent-LSTM}(·), f_{word-LSTM}(·)$ respectively represent the weight parameters of the CNN module, co-attention module, MLC module, sentence LSTM, and word LSTM. 3. Hyperparameter: ${\lambda }_{tag}, {\lambda }_{word};$ 4. For each epoch: 5.  Extract image features $v_n\leftarrow f_{CNN}\left(I^n\right);$ 6.  Predict the tag labels $p_l\leftarrow exp\left(f_{MLC}\left(v_n\right)\right);/$/Equation (2). 7.  Embed semantic features $a_m\leftarrow p_{l.pred};/$/$a_m$ are the embedding vectors of the labels. 8.  Compute the context vectors $ct{x}^{\left(s\right)}\leftarrow f_{COatt}\left(v_n,a_m\right);/$/Equations (3)–(6). 9.  Compute the topic vectors $t^{\left(s\right)}\leftarrow f_{sent-LSTM}\left(ct{x}^{\left(s\right)}\right);/$/Equation (7). 10.  Predict the words $p_{word}\leftarrow f_{word-LSTM}\left(t^{\left(s\right)}\right);/$/Equations (8)–(10). 11.  Compute the loss $L\leftarrow {\lambda }_{tag}CE\left(p_l,l^n\right)+{\lambda }_{word}CE\left(p_{word},w^n\right);/$/$CE(·)$ is the cross-entropy   loss function, and this step refers to Equation (11). 12.  End.

In this study, visual features were extracted using ResNet152 in the encoding stage to obtain visual features of the size 14 × 14 × 512. The visual features were fed into the MLC to obtain the classification results, which were converted into an embedding vector of 512 dimensions. For the hierarchical LSTM, the dimension of both the sentence-LSTM and the word-LSTM hidden states was set to 512 and the size of the initial learning rate was 0.001. The model was optimized by the Adam method. The parameters ${\lambda }_{tag}$ and ${\lambda }_{word}$ were 0.5 and 2, respectively.

2.4. Environmental-Assessment Mapping

As the basic analysis unit of this paper was a remote-sensing scene of the size 250 × 250, the assessment result of the built environment may have been affected by the cropping of remote-sensing images. For example, when the spatial-grid unit is relocated to the left or right, the main elements contained in it may change. The spatial-grid unit in Figure 8a,b are spatially adjacent. Figure 8a contains an industrial area. Although the residential area in Figure 8b is quite close to this industrial area, the industrial area cannot be included in Figure 8b due to the effect of cropping. Although the proposed evaluation method can be analyzed well in a single remote-sensing scene, the assessment of the entire urban built environment must not consider individual scene units in isolation.

Therefore, this paper adopted the interpolation method after sampling to reduce the effect of cropping. At first, we increased the sampling in two steps: (1) The initial sampling spatial-grid unit was moved to the four directions of east, south, west, and north, and the moving distance was 1/2 of the range of the spatial grid unit, that is, 125 pixels. Then, we used the image-caption model to describe the moved spatial-grid unit and obtained the assessment score of each unit. (2) The spatial-grid unit was converted into a center point, which was convenient for subsequent interpolation operations. The attribute of the point was the environmental-assessment score, which was equal to the assessment score of the spatial-grid unit centered on the point. This process aimed to increase the coverage sampling of the original spatial-grid unit and reduce the effect of cropping. The method of moving the spatial-grid unit and obtaining the center point is shown in Figure 9.

Next was the interpolation of the obtained center points. In this study, the inverse distance-weighting method was used to interpolate the obtained sample points. The inverse distance-weighting method is based on the first law of geography. Therefore, the influence of a known point decreases as the distance increases, and the weight is inversely proportional to the square of the distance [71]. In this study, the maximum distance of the point search radius in the inverse distance-weighting method was set to 125. The assessment map generated by interpolation can retain the characteristics of the point itself, and it can integrate the mutual influence between the point and the surrounding points, which can reduce the influence caused by cropping to a certain extent.

3. Results

3.1. Study Area and Data

Changsha is the capital of Hunan Province. It has a large undulating terrain, diverse landform types, and developed surface-water systems. The Xiang River runs through the city from south to north. On the east bank of the river is the main urban area of Changsha City, where the density of buildings and roads is relatively high, and on the west bank of the river are some new urban areas, with low building density and large areas of vegetation cover. In this study, the central urban area of Changsha, which is distributed on both sides of the Xiang River, was selected as the study area (Figure 10). The data used in this study were the high-resolution remote-sensing images covering the central urban area of Changsha.

3.2. Implementation Details

All experiments were performed in the PyTorch framework with one NVIDIA 2080Ti GPU. The experiments were carried out using the established dataset, in which 80% and 20% of the images were used as training and test sets, respectively.

3.3. Model Performance

Table 2. Assessment accuracy for pictures.

i	${\mathit{N}}_{\mathbf{i}}$	Proportion1
5	527	85.69%
4	77	12.52%
3	9	1.46%
2	2	0.33%
1	0	0
0	0	0

shows the performance results of our trained model on the test set (615 images). The correctness of the sentence for each image in the test set was computed using the following formula:

$${\mathrm{Proportion}}_1=\frac{N_i}{N},$$

(12)

where N_i represents the correct number of pictures for i sentences, the value range of i is 0–5, and N represents the total number of pictures.

Statistical analysis showed that the model accurately predicted the assessment sentences of 85.69% of the images (all five indicators were accurately predicted), 12.52% of the images had one indicator incorrectly predicted (four indicators were accurately predicted), 1.46% of the images had two indicators incorrectly predicted (the prediction of three indicators was accurate), and no prediction error of more than two indicators.

To better evaluate the model, the assessment results of the model on five different indicators are shown. For each image in the test set, the prediction accuracy was calculated using the following formula:

$${\mathrm{Proportion}}_2=\frac{N_j}{N},$$

(13)

where N_j represents the number of pictures with the correct jth indicator, j corresponds to indicators of different topics, and N represents the total number of pictures.

Table 3. Assessment accuracy for indicators.

j	${\mathit{N}}_{\mathit{j}}$	Proportion2
Building	573	93.17%
Vegetation	597	97.07%
Road	601	97.72%
Industrial area	608	98.86%
Activity space	595	96.75%

shows that the image-caption model achieved more than 93% prediction accuracy on the five indicators and performed well in discriminating between high and low quality for each indicator.

3.4. Assessment Results

Using the image-caption model trained on the test set of the Environmental Assessment Dataset, the model generated five sentences for each remote-sensing image for description. The generated sentences were then analyzed and the built-environment scores were calculated with reference to the evaluation criteria.

This section shows the sentence results for each image patch (scene unit), although the images cannot fully reflect all the complex scenes in the city and all the performances in each score. However, through these typical images, this work found that for the scenes with the same score, their performances were similar, which may reflect some representative scene characteristics.

These results also show that our method can evaluate geographic scenes from a visual perspective and can simulate human cognition and discriminate environmental quality. (1) For scene units with scores of 5, the quality of the five evaluation indicators was relatively high, which was represented by an open-building layout, high vegetation coverage, good road connectivity, no industrial interference, and space for leisure activities. (2) For scene units with scores of 4, most of them performed well in the four indicators of building density, vegetation coverage, road connectivity, and industrial area, but it was difficult to meet the indicator of the activity area. (3) For scene units with scores of 3, the performance was relatively more diverse; some geographic scenes showed a lack of vegetation coverage but the buildings were more open, and some geographic scenes showed higher vegetation coverage but denser buildings. (4) For scene units with scores of 0, 1, and 2, the indicator of building density performed poorly and most of them lacked vegetation coverage. Among them, scenes with scores of 2 had more open-building density than scenes with scores of 0 and 1, and performed better on the road-connectivity indicator. Scenes with scores of 0 and 1 had dense and irregular buildings, poor road connectivity, and lack vegetation coverage and activity space. Scenes with scores of 0 had industrial areas. The examples in Figure 11, Figure 12, Figure 13, Figure 14, Figure 15 and Figure 16 show that the transition from the highest score to the lowest score is consistent with the results of people’s spatial cognition of geographic scenes, indicating that our method can simulate human perception. The accurate expression of natural language also demonstrates the effectiveness of our assessment method, which can fully describe the specific environmental elements in high-resolution remote sensing images through natural language. These results can fully demonstrate the representativeness of our assessment indicator. Through the organization of five different indicators, the organization of high-resolution remote-sensing-image scene elements can be fully exploited to carry out environmental assessment.

3.5. Built-Environment Assessment

The sampling-point data (Figure 17) were obtained according to the steps described in Section 2.4, and the attribute of the point data was the corresponding environmental-quality-assessment score. After interpolation, the quality-assessment map of the central urban area of Changsha was obtained (Figure 18).

Figure 18 shows the spatial heterogeneity of the environment in the central urban area of Changsha. The figure shows that there is a small blue region on the east bank of the Xiang River. This area is the old urban district of Changsha, where the construction period is relatively old, the residential environment is relatively poor, the commercial activities are rich, the population density is high, and the houses are close to each other. The performance in the five indicators was high building density, less vegetation coverage, narrow roads, and poor connectivity, so the evaluation score was low. The newly developed areas on the west bank of the Xiang River are more orange–red areas because they have low building density, good greenery, and sufficient leisure facilities. This assessment result can simulate the evaluation of human cognition, reflect environmental factors such as building density and vegetation coverage that people pay attention to when choosing a residential area, and provide scientific references for people.

Here, the effectiveness of our generated environment-quality-assessment map is demonstrated by presenting the results on a finer scale. This paper took the road as the boundary and selected six representative regions, a, b, c, d, e, and f, to display the results. The location of each region is shown in Figure 19.

The specific assessment results for the six different location regions are shown in the following figures. Figure 20 shows that the red areas were mainly concentrated in the upper left and lower-right areas, and the middle-left area had lower assessment results. Near the reddest area in the lower right of the figure is a playground, which is one of the few activity areas in this area. In addition, this area has high vegetation coverage, so the assessment result was high.

For region b, most of the areas were warm in color, and most of these areas are emerging residential areas (Figure 21). The buildings are open, the vegetation cover is high, and the road connectivity is good. The reddest area corresponds to the location of the playground in the remote-sensing imagery, and it scored higher than the others because of this activity area. The cool-colored area near the bottom corresponds to the area under construction, which is mostly bare land and lacks vegetation coverage, so the score was low.

For region c, the warm-colored areas were mainly concentrated in the middle and right regions, whereas the cool-colored areas were mainly concentrated in the left region (Figure 22). The buildings in the left region are cluttered, with high density, less vegetation coverage, and very poor road connectivity. In contrast, the middle and right areas are relatively neatly arranged, and the road connectivity is slightly better. In general, region c has a high building density and no activity areas, so its overall score was not high.

Region d had three warm-color areas (Figure 23). The warm-color area with the highest score was concentrated at the bottom. In the remote-sensing images, buildings in this area are well arranged, the vegetation is sufficient, and lakes are present, so the scores were higher. The other two warm-color areas correspond to well-arranged residential areas with playgrounds. The score was higher than the surrounding area because of this activity-space element. Other areas have a higher building density, so the assessment result was lower.

For region e, the color was mostly blue (Figure 24). This result is consistent with the actual situation shown by remote-sensing images. This area is densely built and lacks vegetation cover.

For region f (Figure 25), the warm-color area was mainly concentrated in the upper- and lower-right positions, which correspond to the high-grade residential areas with low building density, orderly houses, and good green coverage. The cool-color area was mainly concentrated in the upper-left area. Industrial areas are around this area and vegetation coverage is lacking, so the assessment result was low.

The results show that high-end residential areas scored higher, and this part of the area was developed later, such as region b, which has better greening conditions, building arrangement, and road connectivity. Region e is an area rich in commercial activities. This type of area was developed earlier and is mostly located in the old city. The building density is very high and the greening conditions are poor, so the overall assessment result was low. Although our results do not have clear boundaries, they are good on a fine scale, which can help demonstrate the differences in adjacent locations, discover spatial heterogeneity, and realize assessment at the micro-scale. Moreover, from the overall distribution of the environment-quality-assessment map in Changsha, most of the areas were warm colors and only a few areas were cool colors.

4. Discussion

4.1. Transferability of the Model

To analyze whether our approach can be effectively applied to other cities, we conducted a built-environment assessment in Chaoyang District of Beijing. Through screening, the urban streets of Chaoyang District, Beijing, were finally selected as the study area (with an area of about 92 km²). Following the steps described above, the final assessment results were obtained. Here, the model trained in Changsha City as described above was used. The final assessment results are shown in Figure 26.

To evaluate the performance of this model in Beijing, blocks a and b were selected for a more detailed analysis. In region a (Figure 27), most of the regions were yellow, and the orange–red areas were mainly concentrated in the lower position. This position corresponds to the residential area with open density and good vegetation coverage in the remote-sensing image, which is also consistent with human cognition. The warm-color areas in region b were mainly concentrated in the left and middle positions (Figure 28). The remote-sensing images show that the distribution of this warm color is consistent with the distribution of the rivers. The orange areas on the right and center correspond to open-plan residential areas close to the river. The red area in the middle corresponds to a relatively open built environment and good vegetation coverage and is close to the river and the playground. In addition, a relatively evident cool-colored area was observed, which corresponds to the industrial area on the right side of the remote-sensing image. According to the above analysis, our method performed well in Chaoyang District and generated assessments that are consistent with human cognition.

4.2. Effectiveness of the Method

The image-description sentences obtained by the image-caption model in this paper is an important base for urban-built-environment assessment. The co-attention module in this model can integrate visual and semantic information and be able to focus on key semantic regions using the attention mechanism. Therefore, we performed an ablation experiment on the co-attention module to evaluate its efficiency.

The experimental results are shown in

Table 4. Assessment accuracy of the ablation experiment on the co-attention module.

i	$\mathit{P}\mathit{r}\mathit{o}\mathit{p}\mathit{o}\mathit{r}\mathit{t}\mathit{i}\mathit{o}{\mathit{n}}_{\mathit{i}}$ (Co-Attention)	$\mathit{P}\mathit{r}\mathit{o}\mathit{p}\mathit{o}\mathit{r}\mathit{t}\mathit{i}\mathit{o}{\mathit{n}}_{\mathit{i}}$(No Co-Attention)
5	85.69%	74.32%
4	12.52%	17.20%
3	1.46%	4.64%
2	0.33%	2.88%
1	0	0.80%
0	0	0.16%

, where the i in this table indicates the number of correct sentences among the five description sentences for each image and the $Proportio{n}_i$ indicates the proportion of images with i correct sentences to the total number of images in the test set. It can be seen that the number of images with five correct sentences accounted for 85.69% of the number of images in the test set when the co-attention module was used and 74.32% when the co-attention module was not used. Meanwhile, it can also be observed that all other $Proportio{n}_i$(i = 4, 3, 2, 1, 0) increased when the co-attention module was not used, which means that the number of images containing incorrect descriptions increased. For an image scene, if the number of correct description sentences output by the model was not five, the image-caption sentence could not represent the environment objectively enough, resulting in the inability to correctly evaluate the built-up area environment. This shows that the co-attention module plays an important role in generating correct descriptive sentences and obtaining an objective assessment of the built-up area environment.

In addition, this paper provides a new idea for studying environmental-quality assessment from a visual perspective. Owing to the increasing availability of open satellite data and related products, the method presented in this paper has significant potential. This paper also has the following special features:

Firstly, this paper uses natural language to represent geographic entities and their relationships that reflect environmental quality, which is the first attempt in the field of urban-built-environment assessment. The computer is endowed with environmental cognition and realizes natural-language expression. As a medium of human expression, natural language is related to human cognition and is a powerful tool for communicating objective reality and subjective feelings [72]. For the general public, most of the information involved in life is also expressed in natural language. By expressing professional geographic data in natural language, it can be better understood by the public and better serve people without relevant professional backgrounds. Moreover, people’s attention can be limited to a specific fine scale, and local areas can be localized according to needs.

Secondly, the assessment criteria in this paper can better consider the spatial layout and physical characteristics. For example, in the road-connectivity indicators, the road connectivity is considered by the road-layout structure in the scene [49]. Traditional road-related indicators often use road-network density to represent connectivity, but this method has limitations. For example, in the case of excessive intersections or many dead-ends and T-roads, although the road-network density is high, the road-network connectivity is not good, which cannot accurately reflect the real situation within the scene. In addition, for vegetation, which is a common element of residential areas, the assessment criteria in this study make a more detailed distinction, distinguishing between the trees around the building and the forest near the building area. Previous studies [73] generally used values such as NDVI to reflect vegetation coverage. Although it can reflect the overall vegetation coverage of the region, it cannot reflect the vegetation distribution. For example, there may be cases where there is no vegetation coverage in the building area, but the calculated value of the overall vegetation coverage is high due to reasons such as proximity to the forest. The evaluation method in this paper can most intuitively reflect the human perspective and directly reflect the distribution and organization of physical objects such as buildings, roads, and open spaces in small urban scenes, bringing new insights into environmental evaluation.

Lastly, this paper expands the boundaries of urban remote-sensing applications based on the image-caption technology. Urban remote sensing is an important research content in the two fields of remote-sensing application and urban science. High-spatial-resolution remote-sensing images provide detailed surface-coverage information and rich local-structure information for urban monitoring, analysis, evaluation, and management. Existing urban research based on remote-sensing-image data mainly includes the extraction, identification, and classification of ground objects in remote-sensing images [74,75,76]. Realizing the knowledgeable description and expression of geoscience objects is a difficulty in current applications. This paper creatively starts from the perspective of simulating human cognition, constructs the assessment criteria of geographic scenes under the guidance of expert knowledge, and uses image-caption models to express cognition.

4.3. Limitations of the Method

This paper still has certain limitations. First, it only considers the physical environment and assesses the local environmental quality from a visual perspective, ignoring the effect of social, ecological, economic, and cultural conditions. In the future, a more comprehensive discussion should be carried out by combining remote-sensing images with crowd-sourced perception data, social-survey data, and open-source geographic information. Second, this work focuses on the simulation of human expressions. From the perspective of natural-language assessment, the indicators cannot be calculated to obtain quantitative values, which may lead to situations where details cannot be obtained. Future work needs to determine how to perform a more detailed assessment while considering the advantages of natural language. Third, the dataset was labeled by 10 GIS students, and although training and strict quality control were conducted, the limitations of the labeling group may have brought a subjective effect to the data, which may have affected the experimental results.

5. Conclusions

This paper uses high-resolution remote-sensing images as data sources to carry out urban-built-environment assessment from a new perspective. It uses the image-caption model to extract objective scene information and develop a large-scale automatic assessment. It reduces the influence of cropping by adopting an inverse distance-weighted interpolation method. This paper is the first work to apply an image-caption model to urban-built-environment assessment. The contributions of this paper are as follows: (1) According to the characteristics of high-resolution remote-sensing images, it selects representative indicators and defines spatial-assessment criteria of geographic scenes based on high-resolution remote-sensing images. (2) According to the established criteria, it designs an expression framework based on natural language and establishes a dataset, which is the first natural-language dataset established for the application of urban-built-environment assessment. (3) It performs the automatic assessment of large-scale urban built environment by applying an advanced image-caption model. The experimental results demonstrate that our method is effective and transferable for rapid assessment of the urban built environment. This work has also generated an environmental-quality-assessment map in the study area, from which local residents can learn about the high quality of the urban built environment. By looking at the map, planners and city managers can gain a more detailed and comprehensive understanding of the city’s environment and support policy formulation in areas such as urban renewal and community revitalization.

The spatial-assessment criteria of geographic scenes are determined by referring to expert knowledge and visual representation of geographical elements in remote-sensing images. In future work, the setting of evaluation criteria will take into account the requirements of environmental psychology, that is, the establishment of evaluation criteria based on residents’ perceptual feedback on image scenes.