Multimodal Deep Learning Framework for Profiling Socio-Economic Indicators and Public Health Determinants in Urban Environments

Abstract

Urbanization significantly enhances socio-economic conditions, health, and well-being for many by improving access to services, education, and economic opportunities. However, socio-economic and public health disparities are also being exacerbated by urbanization. The reliable data required to monitor these conditions are often unavailable, outdated, or inconsistent. This study introduces a multimodal deep learning framework that integrates satellite imagery with street network datasets to predict urban socio-economic indicators and public health determinants at the sector level as a political administrative unit of public health planning in Rwanda. We extracted latent visual and topological embeddings of the urban built environment, using a Convolutional Neural Network (CNN) and Graph Neural Network (GNN). These embeddings were fused through an attentional mechanism to train a multi-task regression model that simultaneously predicts multiple socio-economic indicators and public health determinants. This framework was applied to the City of Kigali in Rwanda. Overall, the multimodal fusion model achieved the best average performance across targets, with an average correlation of 0.68 and MAE of 1.26 for socio-economic indicators, and 0.68 and 1.46 for public health determinants, demonstrating the benefit of integrating visual and topological information. The learned fused embedding space arranges socio-economic indicators and public health determinant deciles along a continuous morphological gradient from sparsely built rural settings to dense urban settings, demonstrating that the urban form encodes latent signals that capture socio-economic indicators and health determinants. Moreover, the study reveals a strong relationship between socio-economic indicators and the public health index, with education, cooking materials, and floor materials exhibiting a correlation above 0.96. This work demonstrates the utility of an integrated framework for socio-economic indicator profiling and public health planning in data-scarce urban contexts, offering a scalable approach for monitoring the indicators of Sustainable Development Goals in rapidly changing urban environments.

1. Introduction

The current trend of urbanization is enhancing the socio-economic conditions, health, and general well-being of urban residents [1,2]. However, urbanization also poses significant socio-economic and health challenges, including social exclusion of the poor dwellers, insecurity, diseases, and an inadequate sanitation environment [3,4]. These challenges induce inequalities [5] and expose vulnerable urban dwellers to socio-economic and health challenges [4,6]. These challenges are particularly severe in the global south, where governments often lack resources to implement urbanization policies and strategies that improve the well-being of urban dwellers [7]. Addressing these challenges has been at the forefront of the Sustainable Development Goals (SDGs): specifically, goal 11 on sustainable settlements and cities, and its target 11.3, which attempts to improve inclusive and sustainable urbanization [8]. These challenges also increase the need to monitor sustainable urban development while ensuring the protection and improvement of well-being [9,10].

More accurate data and insights are needed to support and make comprehensive, informed policy decisions to address urbanization challenges. There is a need for reliable and updated information regarding urban conditions, as it is critical for effective management and monitoring. In addition, cities need structured methods to collect and analyze data into insights that are necessary for policies and actions [11,12]. These are data and information which are essential for understanding urbanization patterns for profiling urban socio-economic indicators and public health determinants. This requires integrated data and methods that make it possible to holistically comprehend the urban socio-economic environment, public health, and their complex interrelations. By leveraging data, policymakers can create an inclusive, sustainable, and healthy urban environment that caters to the needs of all residents through ensuring that resources are efficiently allocated, interventions are targeted effectively, and urban development promotes both socio-economic well-being and public health [11]. The available data are often obtained through censuses, surveys, health information systems, and administrative records. However, obtaining these data is costly, time-consuming, and complex, usually leading to the data being absent, incomplete, or outdated [13].

Moreover, variability in data collection methods, inconsistency in spatial resolution, limited temporal coverage, and difference in definitions across various regions and disciplines hinder their usage [14]. Thus, robust socio-economic and public health data, which are critical for identifying needs, planning interventions, monitoring progress, and evaluating impacts, are missing. Emerging technologies like satellite imagery and machine learning provide promising solutions to data-related problems. Satellite imagery provides large-scale data on housing conditions, infrastructure, and environmental factors [15,16,17,18], offering a cost-effective and scalable data source for monitoring urban development and living conditions.

In this study, the city is considered as a complex urban system where socio-economic, health and environmental processes interact, as depicted in [1,19]. Accordingly, the study introduces a multimodal deep learning framework that integrates high-resolution satellite imagery and street network data to model urban socio-economic indicators and public health determinants in data-scarce environments. By combining visual and topological information, the framework captures the complex interactions underlying urban inequalities and provides interpretable, actionable insights for profiling urban socio-economic and health conditions. This approach not only addresses the challenges of limited and inconsistent urban data but also enables the quantification, monitoring, and modeling of urban dynamics to inform evidence-based planning and policy. We demonstrate the applicability of the framework in Kigali, Rwanda, a rapidly urbanizing city in the global south.

2. Related Works

The combination of satellite imagery and machine learning models has created a new opportunity for the extraction of insights at a large scale. For instance, Yeh et al. [20], in their breakthrough study, utilized CNN to measure asset wealth across Africa by analyzing multispectral satellite images. Their model demonstrated high accuracy, capturing 70% of the wealth variation, as validated against ground-based data. Li et al. [21] proposed an approach for predicting various socio-economic indicators, including population and consumption, utilizing street view and satellite imagery, which outperformed baseline models in accuracy by over 10%. Castro and Álvarez [22] estimated parameters like average income, Gross Domestic Product (GDP) per capita, and water index for Brasilia at the city level, using transfer learning on both day- and nighttime satellite imageries, with a 64% prediction accuracy. A study by Jean et al. [23] proposed a deep neural network model-based approach that estimates poverty levels in Sub-Saharan countries using day- and nighttime satellite imageries. It establishes a relationship between the image characteristics and economic conditions, emphasizing areas that are illuminated as important indicators.

Satellite imagery can also be integrated with diverse datasets, offering a holistic view of socio-economic and health conditions. For instance, Liu et al. [24] proposed a machine learning-based approach that used satellite images, street-view images, and street networks to predict the population, education, crime, and other economic activities. Their model demonstrated a superior performance, with 30% improvements compared to the baseline models. Xi et al. [25] applied satellite images, points of interest, and an unsupervised learning model to predict indicators such as the population and number of takeaway orders. Their model was able to estimate socio-economic indicators with an R² of 0.874. Fan et al. [26] applied street view images to model and measure travel behavior at 83%, crime occurrence at 64%, and 68% of the population lacking physical activities at their variations. Chong et al. [27] extracted information on various amenities from open points of interest to predict the mobility and economic productivity. The above studies, alongside the existing studies, leverage satellite images and machine learning to enable large-scale monitoring of urban socio-economic conditions. However, limitations persist in modeling complex urban environments in most of the global south at a fine spatial scale due to sparse and outdated data [13]. The existing studies are rooted in unimodal learning frameworks, where a single data modality, satellite imagery, is used to capture specific urban features such as population [28,29], urbanization [30], poverty and well-being [17,20], and disparities [31]. While they were proven to be effective for capturing specific urban features, they are limited in their abilities to represent the complex multidimensional nature of the urban environment holistically, since they cannot learn and integrate more socio-economic indicators and public health determinants [32]. In addition, quantitative assessment of how socio-economic well-being influences public health is often ignored. Prior studies treat socio-economic modeling and public health analysis as separate modeling tasks, rather than integrating them into a cohesive modeling framework. This results in a critical gap and limits the actionable value of current integrated urban planning, health equity, and policy interventions [4,32].

3. Socio-Economic Indicators, Public Health Determinants and Their Associations

Socio-economic indicators and public health determinants evolve from socio-economic development and public health, which are two distinct yet interrelated fields [33]. On one side, socio-economic development involves enhancing quality of life by addressing various indicators such as education, employment, income, housing, and access to clean energy [33,34,35,36]. It strives toward improved living standards, reduced poverty and inequality, and enhanced social inclusion and cohesion, as well as promoting sustainable economic growth. For instance, high employment rates reflect economic stability and job opportunities and suggest a thriving economy, which is a critical factor influencing an individual’s quality of life and economic well-being [37]. Educational attainment is another crucial indicator, whereby it plays a significant role in social and economic development by enhancing individuals’ skills and knowledge, which, in turn, contributes to better job prospects, higher incomes, and social stability [38]. Housing conditions, including the type and quality of housing and ownership status, are essential for understanding living standards in urban areas whereby adequate housing reduces exposure to environmental hazards and improves mental and physical health [39,40]. Access to clean cooking fuels reflects the reduced indoor air pollution, which is linked to respiratory diseases, which is crucial for reducing health issues related to smoke and toxins from solid fuels [41].

On the other side, public health aims to reduce health disparities and enhance the quality of healthcare services, targeting entire populations rather than individuals [42]. While public health is observed by considering outcomes, such as life expectancy, mortality rates and disease prevalence, public health determinants consider conditions or factors that influence these outcomes, and include determinants such as sanitation and water access, health insurance coverage and disease prevalence [31,34]. For instance, access to improved sanitation facilities and safe drinking water is critical for maintaining public health and is essential for preventing waterborne diseases such as cholera, typhoid, and hepatitis, and promoting hygiene [43]. Health insurance coverage is a crucial indicator of public health, as it reflects the population’s ability to access medical services without facing financial hardship. Higher health insurance coverage rates indicate better access to healthcare services, which can lead to improved health outcomes [44]. Disease prevalence measures the number of cases of a particular disease present in a population at a given time [45]. Focusing on malaria, this indicator provides insights into the health challenges faced by urban populations, especially in regions where malaria is endemic. Malaria is a significant public health concern, particularly in regions where high malaria prevalence requires targeted interventions to control and reduce the disease burden [46].

While socio-economic development and public health have distinct goals and components, they are highly interconnected and understanding their associations is crucial for designing policies that improve both fields simultaneously, as shown by several studies [34,47,48,49,50]. For instance, higher employment rates lead to better income and job security, contributing to better health outcomes because employed individuals are more likely to afford healthcare, live in healthier environments, and experience lower stress levels [37]. Similarly, higher educational attainment is linked to better health literacy and healthier lifestyle choices, as educated individuals are more likely to understand health information, engage in preventive care, and have lower rates of chronic diseases [38]. Quality housing prevents overcrowding, reduces exposure to pollutants, and provides a stable environment that is conducive to good health. Urban poor communities are disproportionately exposed to health risks, due to the lack of proper infrastructure and poor access to healthcare, information and knowledge networks [39]. Furthermore, access to clean cooking fuels reduces indoor air pollution, leading to lower respiratory diseases [41].

4. Materials and Methods

4.1. Study Area

The study proposed, tested and evaluated a framework for profiling socio-economic indicators and public health determinants. Testing and evaluation of the framework was done based on Kigali, Rwanda’s capital and largest city. Kigali occupies an area of around 730 km² and is home to roughly 1.7 million people [51]. The city consists of 3 administrative districts, Gasabo, Kicukiro and Nyarugenge, comprising urbanized areas and areas under urbanization, which are all surrounded by rural areas (See Figure 1). The city’s urbanized areas and areas under urbanization are characterized by a blend of planned neighborhoods, transitional mixed neighborhoods, and informal settlements [52,53,54]. Transitional mixed neighborhoods are made of a mixture of both planned and informal settlements, and are created by growing planned communities or progressively enhancing informal settlements. Informal settlements are made of cramped and tiny housing units that lack proper access to urban facilities and services. Planned neighborhoods exhibit well-planned road systems and enhanced access to urban facilities and services. In addition, the City of Kigali presents a significant number of inadequate high-density informal settlement areas, which make up 60% of residential buildings [55,56,57].

4.2. Proposed Framework for Urban Socio-Economic and Public Health Profiling

Building upon a review of recent studies in earth observation, computer vision, spatial graph learning, and urban analytics, we proposed a multimodal and multi-model deep learning framework (illustrated in Figure 2) for profiling socio-economic indicators and public health in data-scarce urban environments [17,20,23,39,50,58,59,60,61,62,63]. The framework explores and integrates the complementary strengths of satellite imagery and street network data. The satellite imagery is processed through a CNN [64] encoder to capture morphological and visual cues from the built urban environment, such as settlement density, rooftop texture, and neighborhood structure [20,31,65]. The street network, represented as graphs of nodes and edges, is processed through GNN [66] encoder to learn the latent topological features that reflect street connectivity, spatial accessibility, and urban movement [67,68,69,70,71]. All learned embeddings are fused using an attention-based mechanism, which adaptively learns the relative importance of each modality during joint encoding [72]. The resulting multimodal embeddings are passed into a downstream feedforward neural network (MLP) to perform multi-target regression on urban socio-economic indicators and public health determinants. In addition, the framework features interpretation and visualization that assess associations between socio-economic indicators and public health determinants (see details in Section 4.6).

4.3. Data

The study used high-resolution optical satellite imagery and street networks to train the foundational models. The spatial resolution of the used imagery was 0.5 m, allowing for the capture of urban surface details. It was acquired from the Rwanda National Land Authority. The imagery was composed of three spectral bands corresponding to the visible spectrum: red, green, and blue. It was sliced into a fixed size of 512 × 512 pixels, equivalent to about 256 × 256 m on the ground. This patch size allowed for easily identifying visual features such as buildings, roads, and capturing their surrounding environment. Street network datasets were obtained from OpenStreetMap (OSM), https://www.openstreetmap.org/ (accessed on 22 June 2024). OSM is a global open-source database that maps the world’s geographical features using volunteers [62]. The street network was used to capture additional human factors, such as connectivity, which might not be fully captured by satellite imagery. In addition, the study used nine socio-economic indicators and six public health determinants from the 2022 national population census data available from the National Institute of Statistics of Rwanda [51]. They were available at a sector level for a sampled population of 10% of the entire population of Kigali, which is 1.7 million inhabitants. The socio-economic indicators used in this study are: education, school attendance, employment, occupation, urbanization, floor materials, wall material, cooking energy, and tenure type. At the same time, public health determinants were health insurance, access to drinking water, access to water for general usage, sewage mode, toilet facilities, and waste disposal. Detailed information on each socio-economic indicator and public health determinant is presented in Appendix A.1.

4.3.1. Satellite Image Data Preparation

A total of 10,236 satellite image patches, each representing a distinct area, were used in this study, covering all 35 administrative sectors of the City of Kigali. This total was obtained after a preliminary analysis through which patches overlapping more than one sector were removed. These patches were then labeled based on socio-economic indicators and public health determinant data. For each indicator and determinant, we computed its proportion to obtain the continuous scores, based on the available data per sector. We then transformed these continuous scores into deciles to enable standardized comparisons. The deciles rank each sector on a scale of 1–10, indicating the worst-off to better-off socio-economic indicators and public health determinants. Following Suel et al. [49] and Yeh et al. [20], deciles were used to standardize heterogeneous socio-economic and public health indicators, reduce sensitivity to extreme values, and allow comparisons across variables with different units. Although deciles are ordinal, they were treated as approximately continuous, because the ten-level scale provides sufficient granularity to model gradual socio-economic and public health gradients, enabling the regression models to capture the relative differences between sectors while preserving their ranking, rather than their class boundaries.

Therefore, systematic labeling was applied to associate satellite image patches with sector-level deciles of socio-economic indicators and public health determinants. For each image patch whose centroid fell within a sector boundary, a corresponding label was assigned using non-geometric attributes (e.g., deciles of education, employment, water access). A visualization was implemented to display the image patch and a stacked bar of decile scores across all the outcomes to qualitatively inspect the distribution of socio-economic indicators and public health determinants at the image level. Figure 3 illustrates a sample of image patches alongside the labels.

4.3.2. Street Network Data Preparation

The street network dataset was interpreted based on its structure as a graph, where each road junction is represented by a node, and nodes are connected by their edges as a road segment. The street network dataset has attributes including length, speed limit, surface type, and functional class [62]. Figure 4 illustrates the road type, surface type, and maximum speed for the street network.

We applied a graph-based learning pipeline [66,67] integrating geospatial street network features (attributes) with census-derived socio-economic and health outcomes. The street network dataset was split based on the size of the image patch, whereby each was considered as a patch-specific graph. Thus, we have 10,236 subgraphs corresponding to the total number of image patches. Categorical variables (network type and surface) were encoded using one-hot encoding, and the continuous variable (maximum speed) was normalized using min–max normalization before aggregation into patch-specific graphs. The min–max normalization is among the greatest normalization techniques that have been found to enhance model performance by applying linear transformation of input data to generate a balance of value comparisons between data before and after the process [73,74]. Each patch-specific graph was represented as a sub-graph G = (V, E), where V represents street segments as nodes and E encodes topological and semantic relationships among segments (e.g., connectivity, intersection). Each node $V_i$ ∈ V was associated with a feature vector $x_i$ ∈ R^d capturing encoded attributes such as type and surface type, linked to patch-level target vectors containing socio-economic indicators and public health determinants.

4.4. Extraction of Embeddings

The extraction of embeddings followed a developed deep learning pipeline. To extract embeddings from satellite imagery, each image patch was resized to 224 × 224 pixels ($X_i$ ∈ R^224×224) and served as an input to a CNN, which was trained to predict a vector of socio-economic indicators and public health determinants. We used a pretrained ResNet50 model that was trained on over 1.3 million ImageNet images [75] as a CNN feature extractor. We used global average pooling followed by a fully connected layer of 256 units with a rectified linear unit (ReLU) activation and a dropout rate of 0.3 to extracted 256-unit dimensions. This set-up enabled the model to distinguish high-dimensional visual information into a lower-dimensional embedding space that retains relevant spatial patterns that are important for downstream applications. This task was accomplished as a multi-output regression problem, enabling the model to jointly predict multiple dimensions of urban socio-economic indicators and public health determinants simultaneously. Given a training set of N-labeled image patches with corresponding decile targets $y_i$ ∈ [1, 10]^T, training was performed using the Adam optimizer with a batch size of 32, and we optimized the mean squared error, due to its robustness to outliers [76], across all outputs using a loss function defined in Equation (1), as in Brooks [77]:

$$L\left(\theta \right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N||y_i - f_{\theta }{\left(x_i\right)||}_2^2$$

(1)

where $f_{\theta }$ denotes the CNN parameterized by weights θ mapping input $x_i$to the final predicted outputs $y_i$; $x_i$is the input, $y_i$ are decile targets in T dimensions for sample i, and the norm sums square differences over all T output dimensions. We extracted intermediate semantic embeddings from the trained CNN to support multimodal learning. Specifically, the output from the final dense layer prior to regression, $z_i \in { \mathrm{R}}^{256},$ was used as the embedding of the 256 dimension for each image patch, as captured in Equation (2):

$$z_i= g_{\theta }\left(X_i\right)$$

(2)

where $g_{\theta }$ denotes the CNN up to the last dense layer before the regression output.

We also extracted embeddings from the street network. We used two Graph Convolutional Network (GCN) layers, followed by a global mean pooling operator and a fully connected output layer. Each GCN layer used 32 hidden channels with ReLU activation. The applied GCN is an extended version of ordinal CNN for graph processing [69], where the feature propagation at each layer follows the rule, as depicted in Equation (3):

$$H^{(l+1)}= \delta ({D^{\prime }}^{-{\displaystyle \frac{1}{2}}}{A}^{\prime }{D^{\prime }}^{-1/2}{H}^{\left(l\right)}{W}^{\left(l\right)})$$

(3)

where A′ = A + I is the adjacency matrix with self-loops, D′ is the corresponding degree matrix of A′, W^(l) is a learnable weight matrix at layer l, H^(l) is the input node feature matrix to layer l, and $\delta$(⋅) denotes a ReLU activation. At each layer, node features are updated by aggregating information from neighboring nodes, applying a linear transformation, and passing the result through the non-linear activation, yielding embeddings that capture both the local topological structure and node attributes.

The model was trained using the Adam optimizer, with a learning rate of 0.01 and mean squared error (MSE) loss, following Equation (1). After model training, we extracted 32-dimensional graph-level embeddings $z_r\text{∈}{\mathrm{R}}^{\mathrm{k}}$by modifying the forward pass to return pooled node features, rather than decile prediction outputs. This was obtained by taking the feature matrix at the final layer of GCN, H^(l) = [$h_1, h_1, \dots , h_N]$^T, and applying a global mean pooling operation to form a single vector representing the whole graph, $z_r$. Therefore, from Equation (3), the final overall embeddings, $z_r$, are given by Equation (4):

$$z_r= {\displaystyle \frac{1}{\left|V\right|}} {\sum }_{v_i\in V}{h}_i$$

(4)

where V is the number of nodes and $z_r$ is used as the embedding for downstream tasks.

4.5. Fusion of Embeddings

Next, we merged embeddings extracted from satellite imagery and street network data to construct a fused embedding dataset for predicting socio-economic indicators and public health determinants. Inspired by Xi et al. [25], we used a modality-aware attentional fusion technique that learns to adaptively weight and combine embeddings from each modality [72]. While other fusion methods such as simple concatenation or weighted average can sometimes achieve good predictive performance, the attention-based fusion provides per-sample interpretability by quantifying the relative contribution of each modality [25], which is an advantage for urban analytics (see Appendix E). We denoted $z_i$ and $z_r$ for image and street network-based embeddings, respectively. We then have projected 32-dimensional graph-level embeddings to 256-dimentional embeddings to allow fusion. Finally, we applied a non-linear transformation to project both embeddings into a shared latent space, as denoted in Equation (5):

$${ \mathrm{h}}_{\mathrm{i}}=\tan \left(\mathrm{W}{z}_i + \mathrm{b}\right), {\mathrm{h}}_{\mathrm{r}}=\tan \left(\mathrm{W}{z}_r + \mathrm{b}\right)$$

(5)

where $\mathrm{W} \in { \mathrm{R}}^{\mathrm{d}\text{×}\mathrm{d}}$ and $\mathrm{b} \in { \mathrm{R}}^{\mathrm{d}}$ are shared learnable parameters. An attention context vector $\mathrm{c} \in {\mathrm{R}}^{\mathrm{d}}$ is then used to compute relevance scores for each modality ${\alpha }_{\mathrm{i}} ={ \mathrm{c}}^{\mathrm{T}}{\mathrm{h}}_{\mathrm{i}}$, ${\alpha }_{\mathrm{r}} = {\mathrm{c}}^{\mathrm{T}}{\mathrm{h}}_{\mathrm{r}}$, which were normalized through a softmax function to obtain attention weights, as shown in Equation (6):

$${\beta }_{\mathrm{i}} = {\displaystyle \frac{{\mathrm{e}}^{{\alpha }_{\mathrm{i}}}}{{\mathrm{e}}^{{\alpha }_{\mathrm{i}}}+{\mathrm{e}}^{{\alpha }_{\mathrm{r}}}}}, {\beta }_{\mathrm{r}} = {\displaystyle \frac{{\mathrm{e}}^{{\alpha }_{\mathrm{r}}}}{{\mathrm{e}}^{{\alpha }_{\mathrm{i}}}+{\mathrm{e}}^{{\alpha }_{\mathrm{r}}}}}$$

(6)

The final fused embeddings $z_f$ are computed as a convex combination, as illustrated in Equation (7), based on Equations (2), (4) and (6):

$$z_f = {\beta }_{\mathrm{i}}·z_i + {\beta }_{\mathrm{G}} · z_G$$

(7)

4.6. Downstream Tasks

The fused embeddings $z_f$ were used as an input to a multilayer perceptron (MLP) [78] for the regression of multiple decile-based socio-economic indicators and public health determinants. The MLP architecture comprised two hidden layers with ReLU activation and dropout regularization, trained with mean squared error loss. We investigated the interpretability of the learned embedding space by applying Principal Component Analysis (PCA), a multivariate analysis that maintains data covariance while simplifying the data [79], to reduce the high-dimensional embeddings (256 dimensions) into two dimensions. To qualitatively assess this alignment, we selected ten anchor points corresponding to different decile levels of a representative socio-economic outcome. For each anchor point, we displayed the associated satellite image and decile label, illustrating how distinct urban textures, such as dense informal settlements, intermediate peri-urban zones, and formal neighborhoods, are embedded within the learned space.

Next, we assessed the association between public health determinants and socio-economic indicators, using statistical analysis through Spearman correlation coefficient and Ordinary Least Squares (OLS) regression, due to its computational tractability and its natural extension to multi-output regression [80]. This choice was also influenced by the socio-economic and public health data that were available only at the sector level, which produce unstable estimates in spatially localized models such as Geographically Weighted Regression (GWR). Furthermore, we used PCA on public health determinants to create a composite health index, using the first principal component. Then, we applied OLS regression to assess the influence of socio-economic indicators on the obtained health index.

4.7. Measurement of Prediction Performance

For all the models (CNN, GCN and downstream MLP), we have used 80% of the data from all 35 sectors of City of Kigali for training and 20% for evaluation, ensuring that no patches from the same sector appeared in both sets, which reflected a common practice in machine learning training [81]. We trained models for 100 iterations, balancing performance and computational resources to allow for the research settings and application in computational resource-constrained areas. The Appendix B contains graphs showing mean loss and mean absolute error for both the training and validation sets over training epochs (see Figure A1, Figure A2 and Figure A3).

We then evaluated the performance of the models by using five-fold cross validation with shuffled splits and a fixed random seed (42), which is commonly used in machine learning to test model robustness and generalizability [49]. In each fold, 80% of the data were used for training and 20% were held out for testing. We trained each fold for 30 epochs to balance the limited computational resources available for the study (Appendix C represents the hyperparameters used). The performance was reported using the MAE and Spearman correlation coefficient (r) aggregated as mean ± standard deviation across folds, due to their robustness on ranked data [82]. The results are presented in

Table A4. Cross-validation performance, using fused embeddings across socio-economic indicators and public health determinants.

		MAE_mean	MAE_std	r_mean	r_std
Socio-economic indicators	Education	1.20	0.03	0.78	0.01
	School attendance	1.26	0.04	0.77	0.01
	Employment	1.60	0.04	0.67	0.01
	Occupation	1.63	0.03	0.57	0.02
	Urbanization	1.99	0.05	0.61	0.02
	Floor materials	1.25	0.02	0.78	0.01
	Wall material	1.50	0.03	0.71	0.01
	Tenure type	1.16	0.03	0.73	0.01
Public health determinants	Health insurance	1.61	0.03	0.67	0.00
	Drinking water	1.84	0.03	0.58	0.02
	Water for general usage	1.29	0.03	0.76	0.01
	Cooking energy	1.18	0.04	0.79	0.01
	Sewage mode	1.75	0.03	0.59	0.02
	Toilet facility	1.68	0.03	0.50	0.01
	Waste disposal	1.11	0.04	0.77	0.01

in Appendix D.

5. Results

5.1. Predictions of Socio-Economic Indicators and Public Health Determinants

We present the performance evaluation metrics of unimodal and multimodal models across socio-economic indicators and public health determinants through the MAE and Spearman correlation coefficient (r) in

Table 1. Predictions metrics for socio-economic indicators and public health determinants.

		Satellite Imagery		Street Network		Fused
		MAE		r		MAE
Socio-economic indicators	Education	1.81	0.65	1.39	0.71	1.18	0.75
	School attendance	2.09	0.56	1.45	0.72	1.23	0.72
	Employment	1.76	0.65	1.66	0.64	1.58	0.65
	Occupation	2.06	0.26	1.65	0.58	1.61	0.59
	Urbanization	2.10	0.49	2.06	0.57	1.99	0.61
	Floor materials	1.78	0.68	1.51	0.70	1.22	0.77
	Wall material	1.65	0.70	1.70	0.63	1.45	0.70
	Cooking energy	1.79	0.65	1.38	0.72	1.13	0.75
	Tenure type	1.85	0.47	1.32	0.70	1.57	0.66
Public health determinants	Health insurance	1.68	0.68	1.70	0.62	1.18	0.75
	Drinking water	2.05	0.39	1.87	0.55	1.23	0.72
	Water for general usage	1.74	0.65	1.44	0.70	1.58	0.65
	Sewage mode	2.15	0.32	1.75	0.59	1.61	0.59
	Toilet facility	2.04	0.49	1.76	0.45	1.99	0.61
	Waste disposal	1.81	0.59	1.24	0.72	1.22	0.77

. The best performance metrics are represented in bold. Overall, the multimodal fusion model achieves the best average performance across socio-economic indicators and public health determinants, demonstrating the benefit of integrating visual and topological information. For socio-economic determinants, there was an average correlation of 0.68 and an MAE of 1.26 between the actual and predicted deciles. The average correlation and MAE for public health determinants were 0.68 and 1.46, respectively. The model shows strong predictive performance for education, cooking energy, and health insurance (r = 0.75, MAE = 1.18, 1.13, and 1.18). The weakest performances were occupation (r = 0.59, MAE = 1.61) and sewage mode (r = 0.59, MAE = 1.61). However, performance gains are not uniform across all targets (socio-economic indicators and public health determinants). For tenure type, water for general usage, and toilet facility, the street-network-only model attains a marginally lower MAE. This suggests that these outcomes are more strongly influenced by infrastructural topology and connectivity patterns than by visual urban morphology alone.

The scatter plots (Figure 5) indicate that the occupation and urbanization predictions are more dispersed and tend to underestimate higher observed values, which is reflected in the weaker correlations. By contrast, in education, cooking energy, and floor materials, larger bubbles cluster tightly along the diagonal, indicating consistent agreement across categories.

A similar pattern is evident for public health determinants (Figure 6). While health insurance and waste disposal show tight clustering along the diagonal line and relatively low error, sewage mode and toilet facility display greater dispersion, with a clear tendency to underestimate higher observed values.

5.2. Spatial Patterns of Urban Socio-Economic Indicators and Public Health Determinants

The observed data from national statistics reveal that the city center of the City of Kigali has the best socio-economic indicators and public health determinants, generally. The outskirts are primarily home to the impoverished, while the city’s east side is gentrifying and has pockets of modern neighborhoods. However, this observation was different for tenure type (housing) and school attendance, which are dominant in the outskirts and rural settings of the city. Importantly, the fused embeddings from both the satellite image and road network were able to capture these variations. Figure 7 and Figure 8 visualize spatial patterns for the observed and predicted socio-economic indicators and public health determinants across the city of Kigali, respectively.

5.3. Interpretation of Fused Embeddings

The fused embeddings are shown in two dimensions, and reveal the location of the satellite images of several public health determinants and socio-economic variables in the embedding space (Figure 9). Points in the scatter plot (center) correspond to the distribution of fused embeddings. Based on the observed employment data, black-labeled markers denote the centroids of employment deciles, ranging from 1 (lowest employment; worst off) to 10 (highest employment; better off). Example patches from each decile are shown around the embedding space, illustrating a visual progression from sparsely built, rural environments (deciles 1–4) to densely developed urban and peri-urban areas (deciles 7–10). The embedding reveals a continuous feature space aligned with the employment conditions, suggesting that spatial and structural characteristics in satellite imagery capture salient patterns associated with local socio-economic status.

5.4. Influences of Socio-Economic Conditions on Public Health Determinants

The findings illustrating the influence of socio-economic indicators and public health are captured in Figure 10A,B. The predicted indicators consistently exhibit stronger and more homogeneous monotonic correlations with determinants of public health than the observed indicators. This pattern suggests that the model predictions smooth the local noise and inconsistencies that are present in the observed data, yielding clearer sector-level relationships. While the observed indicators reflect real-world variability, they are also affected by reporting errors, aggregation effects, and missingness, which may weaken apparent associations with health outcomes. Some negative or weak correlations indicate localized or domain-specific mismatches that are not fully captured by socio-economic decile scores. The predicted matrix (left) reveals even stronger and more uniformly positive correlations than the observed matrix. For instance, predicted education shows a near-perfect correlation with several health-related indicators (r > 0.9), while observed education correlations are slightly lower (e.g., r = 0.82 with health insurance). This suggests that the model captures and may amplify monotonic relationships between these domains.

We also evaluate the extent to which socio-economic indicators generally influence public health, using its corresponding index obtained by us. The observed and predicted indices showed a strong association (r = 0.91) and low prediction error (MAE = 0.92; RMSE = 1.19). The OLS regression indicated that socio-economic indicators are significant predictors of the health index (adjusted R² = 0.97). The data analysis results illustrated in Figure 11 reveal a close alignment between observed and predicted indices from sparsely built rural environments to dense urban cores, aligning with the observed employment and infrastructure conditions.

Scatter plots (Figure 12) reveal that education (r = 0.98), cooking energy (r = 0.98), floor materials (r = 0.96), wall materials (r = 0.84), and employment (r = 0.86) were the strongest predictors of the health index, reflecting the combined influence of infrastructure quality, energy access, and labor participation (occupation) on population health. Urbanization showed a moderate association (r = 0.63), while school attendance (r = −0.96) and tenure type (r = −0.72) exhibited strong negative correlations, suggesting complex socio-demographic effects that run counter to the expected trends.

6. Discussion

This study introduces a multimodal deep learning framework that integrates satellite imagery and street network data to model urban socio-economic indicators and public health determinants in a data-scarce urban environment. The findings (see Table 1 and Figure 5 and Figure 6) highlight that multimodal and fusion approaches improve the predictive performance and interpretability for modeling complex urban conditions. This aligns with prior studies that demonstrated the benefit of multimodal fusion in urban analytics [24,25]. For instance, prior studies used CNN and satellite imagery to predict poverty, living conditions, and well-being [22,31], while street networks were used to predict economic status and deprivation [21]. Our findings align with the above studies, but extend them by showing that fusing these data modalities through an attention mechanism allows us to derive additional information and interpretation that would not be captured using a single data modality. This enabled more accurate prediction for most of the socio-economic indicators and public health determinants, compared to single-modality models. This is particularly significant in contexts where traditional data such as surveys and censuses are unavailable or outdated. This use of an attentional mechanism approach is also supported by recent multimodal machine learning studies [25,68].

The findings (Figure 9) also revealed that fused embeddings highlight the structural, morphological, and topological attributes captured in high-resolution satellite imagery and street networks, which retain urban spatial visual and topological proxies that are crucial for predictive analysis [17,25]. The patterns of fused embedding highlight that beyond their predictive performance, they also provide a lens into the spatial logic of urban inequalities by capturing transitions between dense urban cores, peri-urban zones, and sparsely developed areas. This reflects how the settlement morphology and infrastructure networks shape the living conditions. Such representations can support planners and public health practitioners by identifying emerging zones of deprivation, infrastructure gaps, and spatial discontinuities in service provision. In rapidly urbanizing cities where formal data collection falls behind physical expansion, these embeddings provide a mechanism for timely situational awareness. They translate raw geospatial signals into interpretable patterns that can inform infrastructure prioritization, targeted interventions, and equity-oriented planning.

Findings illustrated by Figure 10A,B reveal a strong relationship between socio-economic indicators and public health determinants. Observed sector-level associations highlight how urban morphology, infrastructure, and service access collectively shape public health. The stronger correlations observed between predicted socio-economic indicators and public health determinants reveal that model predictions act as spatial proxies that capture latent urban structural patterns, rather than direct socio-economic measurements. Therefore, the results reflect noise reduction, spatial smoothing, or representation of underlying environmental gradients. This supports previous claims that deep learning methods can uncover latent spatial processes linking built environment features to social outcomes, particularly in urban contexts where formal data collections do not evolve with rapid urban changes, which would allow for urban monitoring [23,31]. In addition, findings on the influence of socio-economic indicators on public health (Figure 11) demonstrate that urban form and infrastructural characteristics encode strong socio-economic indicators that influence public health, and can be extracted through multimodal approaches. The high predictive power of education, energy access, and housing materials reveal the interconnectedness of living conditions and health, supporting long-standing urban planning theory while quantifying these effects at an unprecedented spatial resolution [4,32,50]. Some counterintuitive patterns in school attendance and tenure type arise from the specific spatial organization of City of Kigali, where peri-urban areas exhibit higher school attendance and home ownership, but still lack an adequate health-supporting infrastructure. These relationships should be understood as aggregate, place-based dynamics, rather than individual-level causal mechanisms. Urban sectors function as socio-spatial units shaped by historical development, infrastructure investment, and demographic processes. As a result, correlations at this level may reflect environmental gradients, service clustering, and settlement patterns, rather than direct behavioral or household-level effects.

In the proposed framework depicted in Figure 2, we hypothesize that satellite imagery encodes the visual and morphological cues of urban areas, while street networks encode the latent accessibility and connectivity patterns, and that their integration should yield an interpretable representation of socio-economic indicators and public health determinants. The observed performance of the multimodal model and the spatial coherence of predicted socio-economic indicators and public health determinants are broadly consistent with this hypothesis. In particular, the complementary strengths of CNN-derived morphological embeddings and GNN-derived topological embeddings suggest that inequality and health vulnerability are structurally embedded in both the physical form and network organization of the city. Partial divergences between predicted and census-based indicators do not contradict the framework; rather, they indicate that administrative statistics and learned spatial representations capture different observational layers of the same urban system. While the census data used summarize the discrete sector-level categories, the multimodal embeddings approximate the continuous spatial gradients of the infrastructure and environmental conditions. This convergence between modalities supports the conceptual premise that urban socio-economic indicators and public health determinants are multi-dimensional and cannot be fully explained by a single data source.

Some methodological considerations should be noted. First, using sector-level labels introduces aggregation effects that may obscure intra-sector heterogeneity and exposes the analysis to the Modifiable Areal Unit Problem (MAUP). As a result, statistical relationships can vary, depending on the spatial zoning system. While this approach is necessary in data-constrained contexts and enables multimodal learning, it smooths fine-scale variability in socio-economic and environmental conditions. Consequently, correlations observed at the sector level may differ under alternative aggregation schemes. Moreover, these correlations represent aggregated spatial associations and should not be interpreted as individual-level relationships. The study also assumes temporal stationarity, representing socio-economic and health conditions at a single point in time. Urban systems are dynamic, so future work should incorporate time-resolved data, such as satellite imagery, mobility traces, or longitudinal administrative records. Additionally, multi-scale or hierarchical modeling could capture both fine-grained spatial variation and temporal dynamics. Expanding the framework to include other urban datasets, such as points of interest or mobile phone metadata, could further improve the predictive accuracy and interpretability.

7. Conclusions

This study proposed a multimodal deep learning framework, integrating high-resolution satellite imagery with street network datasets alongside deep learning models, to predict the spatial patterns of socio-economic indicators and public health determinants and their inter-relationships in urban environments. The findings highlight that the framework achieved strong predictive performance, which confirmed that fused embeddings portray visual and topological proxies that are essential to approximate urban living conditions. These embeddings encode signals portraying socio-economic indicators and public health determinants and its dimensional space illustrate a continuous gradient from sparsely built rural areas to densely developed urban core. This reveals how the built urban environment reflects the underlying disparities. The findings indicate that multimodal deep learning approaches can generate disaggregated and policy-relevant information that is essential for urban planning and the decision-making process, which is highly relevant in data scarce settings where conventional socio-economic and health surveys are costly, outdated or unavailable. Thus, operationalizing the proposed framework at scale can support evidence-based planning and targeted interventions to address disparities, ultimately enabling more equitable and health-conscious urban development for the achievement of the Sustainable Development Goals, mainly target 3 of goal 11.