Review Reports - Predicting Concentrations of PM2.5, PM10, CO, VOC, and NOx on the Urban Scale Using Machine Learning-Based Surrogate Models

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The topic is interesting. The following problems must be handled.

(a) The paper needs a comprehensive grammar and language correctness review. Some sentences are too long to be understood. Please rewrite these sentences.

(b) From the technology aspect, it is difficult to judge the significance of the contribution.

(c) The conclusion part: A discussion of the advantage and underlying drawbacks of the proposed method is welcome in the conclusion part.

(d) Try to analyze from Fig.3 from the theory aspect.

(e)Title is so long.

(f) Why lgbm is used？ It is not new.

Author Response

a)The paper needs a comprehensive grammar and language correctness review. Some sentences are too long to be understood. Please rewrite these sentences.

We have attempted to shorten sentences that were too long and therefore unclear; however, the entire text in the final version will be subject to linguistic proofreading provided by MPDI.

b) From the technology aspect, it is difficult to judge the significance of the contribution.

Yes. Information about technology and implementation details has been expanded.

c) The conclusion part: A discussion of the advantag and underlying drawbacks of the proposed method is welcome in the conclusion part.

Yes, the conclusion section has been significantly expanded and extended with a summary of the conclusions included in the Results section.

d) Try to analyze from Fig.3 from the theory aspect.

This is an empirical fact. Fig. 3 has been removed and replaced with a simple comment. The complete drawings contain all the determined approximations, i.e., more than 97 $10^6$ points for the entire map and slightly fewer for the roads themselves.

(e)Title is so long.

Yes. The title is working only, and we changed it to:
New machine-learning-based surrogate models to predict $PM_{2.5}$, $PM_{10}$, CO, VOC, NOx concentrations in the urban scale.

(f) Why lgbm is used？ It is not new.

The LGBM method was used with a form of input data that was undoubtedly unprecedented, as described in more detail in this paper.
The relationships with other ways of using LGBM in spatial pollution prediction were also discussed in the paper, as motivation for our approach.
LGBM is, in fact, a single-output regression method, and in this article we utilize it to solve a high-dimensional output regression problem (d = 194,477).

Reviewer 2 Report

Comments and Suggestions for Authors

This article addresses the pressing and practically significant problem of predicting urban air pollution with high spatial and temporal resolution. The topic is fully consistent with the scope of the journal Applied Sciences, as it combines machine learning methods with environmental monitoring and sustainable urban development. The proposed approach significantly reduces computational costs compared to physical models such as ADMS-Urban and enables the generation of pollution maps in near real time, which is important for environmental alert systems and air quality management.

Comments and questions to the authors

The introduction provides a good overview of the literature, but it is not entirely clear what problem remained unresolved in previous studies and why the need for developing this particular surrogate model arose. A more precise formulation of the scientific gap that this work fills would be appreciated.
The research results presented in the introduction (lines 96–116) would be more appropriately moved to the "Conclusion" section. It would be more logical to conclude the introduction with a clear statement of the study's purpose (lines 61–65), without first presenting the results.
Section 2. Lines 66–95 of the introduction (description of data and methods) should be moved to the "Materials and Methods" section. Reference to project funding should be removed from this section and placed in an appropriate location (e.g., in the "Acknowledgments" section).
Rationale for using PM5. One of the key ideas of the paper is the use of PM2.5 measurements to predict other pollutants. The authors note that the success of this approach may be region-specific. It would be useful to clarify why PM2.5 particles were chosen over other metrics, and how generalizable this method might be for other cities and types of pollution.
Selection of Machine Learning Models. Section 2.3 lacks a rationale for the choice of MLP+PCA and LGBM. It would be helpful to briefly explain why these models were chosen and their advantages for solving the problem at hand compared to other approaches.
The "Discussion and Conclusions" section contains useful observations, but it lacks a structured conclusion outlining the main results, scientific novelty, practical significance, and prospects for further research. Currently, some of these elements are placed in the introduction, where they are out of place.

Overall, the work is of interest and can be recommended for publication after revision of the text to improve its clarity, structure, and the validity of the proposed solutions.

Comments for author File: Comments.pdf

Author Response

The introduction provides a good overview of the literature, but it is not entirely clear what problem remained unresolved in previous studies and why the need for developing this particular surrogate model arose. A more precise formulation of the scientific gap that this work fills would be appreciated.

and) We more precisely explained the main goal and the motivation in the Introduction. Briefly, our primary objective in this work is to explore the feasibility of utilizing machine learning regression models to generate approximate pollution maps at the urban scale. We compare two types of model: multi-output regression models, which directly predict pollution values at all map points, and simple regression models, which predict pollution values at single map points. These approaches have been considered in previous works. Based on this analysis, we propose an effective machine learning surrogate model to predict air pollution maps in urban areas with high temporal resolution (e.g., hourly).

2. The research results presented in the introduction (lines 96–116) would be more appropriately moved to the "Conclusion" section. It would be more logical to conclude the introduction with a clear statement of the study's purpose (lines 61–65), without first presenting the results.

ans) We reorganized the paper in accordance with this suggestion: we moved the indicated paragraph to the Conclusions section. We added an extended paragraph to the Introduction to clarify the main objective of the work more extensively.

3. Section 2. Lines 66–95 of the introduction (description of data and methods) should be moved to the "Materials and Methods" section. Reference to project funding should be removed from this section and placed in an appropriate location (e.g., in the "Acknowledgments" section).

ans) We reorganized the paper as suggested: the indicated paragraph is now in the Materials and Methods section, and the information on project funding is now at the end of the work.

4. Rationale for using PM5. One of the key ideas of the paper is the use of PM2.5 measurements to predict other pollutants. The authors note that the success of this approach may be region-specific. It would be useful to clarify why PM2.5 particles were chosen over other metrics, and how generalizable this method might be for other cities and types of pollution.

ans) The fact that we can use measurements from PM2.5 monitoring stations to model
The levels of various air pollutants are not a general property, but rather an indication of the specificity of the pollutants present in a given area. For example, in the paper [3], it is demonstrated that in the London area, PM2.5 measurements in random forest models are strongly correlated with PM10 values and, to a lesser extent, with NOx levels.

5. Selection of Machine Learning Models. Section 2.3 lacks a rationale for the choice of MLP+PCA and LGBM. It would be helpful to briefly explain why these models were chosen and their advantages for solving the problem at hand compared to other approaches.

ans) We explain this in more detail in the Materials and Methods section and also signal it in the Introduction. Briefly, we selected the MLP+PCA and LGBM regression models as computationally feasible implementations of the two approaches we compare in the work: multi-output and single-output regression models. Our extensive studies on alternative approaches (not included here, conducted during the project with Lemitor) indicated that these two methods are efficient and effective representatives of the two approaches to regression that we compare in the task of generating pollution maps in an urban scale. We clarified the rationale for selecting regression methods as candidates for the surrogate map generators in Section 2.3. We more clearly formulated the research question related to this problem in the Introduction. Regarding linear models, they were tested in a controlled manner and yielded larger errors in each case, particularly in terms of RMSE. We write about this in the extended motivation for choosing the MLP-PCA method.

6. The "Discussion and Conclusions" section contains useful observations, but it lacks a structured conclusion outlining the main results, scientific novelty, practical significance, and prospects for further research. Currently, some of these elements are placed in the introduction, where they are out of place.

ans) We restructured the Conclusion section according to this suggestion, providing a more detailed explanation of the main results and their significance.

Reviewer 3 Report

Comments and Suggestions for Authors

This study employs machine learning models, historical data, and monitoring station measurements of actual PM2.5 pollution levels to predict the concentrations of nitrogen oxides, carbon monoxide, volatile organic compounds, PM2.5, and PM10 at the urban scale. Specific suggestions are as follows:

(1) The description of the study area is relatively brief, lacking detailed geographical boundary coordinates or map visualization.

(2) The study area covers the city center, residential areas, and parks, but the proportion of these areas or their typical characteristics (e.g., population density) are not specified. Air pollution is closely related to population density and land use.

(3) The timestamp data is discontinuous (only specific days and hours), which may not represent annual variations.

(4) Table 1 provides basic statistical values for the pollutants, but the outlier handling or data cleaning steps are not explained.

(5) In the MLP-PCA method, what is the rationale for setting the variance explanation threshold for PCA dimensionality reduction to >0.99?

(6) The MLP network structure (two hidden layers with 200 neurons each) is only mentioned as "experimentally selected," lacking theoretical justification or details on hyperparameter tuning.

(7) Figure 1 shows individual fluctuations in MAE values, but the reasons for these fluctuations (e.g., seasonality or multicollinearity) are not discussed in depth.

(8) Figures 2 and 3 use scatter plots to show the relationship between predicted and true values, but the points overlap severely; density plots or box plots could be used instead.

(9) PM10 prediction is based on PM2.5 measurements (Table 5), but the universal applicability of this substitution is not sufficiently justified (relying only on local data similarity).

(10) The CO prediction error is large (Tables 8 and 9), but the influence of the unit scale (inherently larger CO values) is not discussed.

(11) The LGBM model shows extremely high prediction error for NOx (Table 16), but the reasons are not explored.

(12) The extensive use of tables for statistical prediction accuracy compromises readability; concise graphical representations should be used instead.

Author Response

(1) The description of the study area is relatively brief, lacking detailed geographical boundary coordinates or map visualization.

ans) Now Figure 1 shows a map of the area of interest, indicating roads, green areas, buildings, waterways, and railway lines, and the general coordinates.

ans) Right. But the baseline data that we use, provided by LEMITOR, comes from a model that predicted pollution levels in 2016. We did not use demographic information because that was the assumption behind the surrogate model.

(3) The timestamp data is discontinuous (only specific days and hours), which may not represent annual variations.

ans) Yes. We had no influence on the data set. Each observation is assigned a specific date and time. The date also indicates the day of the week. With the exception of December, the data come evenly from each subsequent month of the year. In this sense, they are seasonal. The reported results show that such selectively selected data, supplemented with weather data, allow for the construction of a meta-model, a surrogate model that easily approximates the complex physical model from which the training data, supplemented with measurements from monitoring stations and weather data, are derived. Seasonal variability is also visible in the prediction error tables, as the time-averaged errors exhibit seasonal variability (see Tables 2-16), which justifies our training data selection methodology.

(4) Table 1 provides basic statistical values for the pollutants, but the outlier handling or data cleaning steps are not explained.

ans) The data received were free of gaps and did not exceed the ranges listed in the table. It is highly likely that the company that built the source model provided us with only complete and accurate data.

(5) In the MLP-PCA method, what is the rationale for setting the variance explanation threshold for PCA dimensionality reduction to >0.99?

ans) In the manuscript, we have extended the explanation regarding the choice of the number of PCs.
"The number of the largest PC used for the reduction in dimensionality of the output map was determined on the basis of the first 200 observations and depends on the type of pollution. The number of principal components considered was 20, 30 40, 40, 50, or 60. The Five-fold cross-validation method was applied to the chosen data set with MAE taken as the criterion. The number of components by cross-validation procedure ensured that the cumulative explained variance ratio captured by the given number of principal components was, in our experiments, greater than 0.99."

(6) The MLP network structure (two hidden layers with 200 neurons each) is only mentioned as "experimentally selected," lacking theoretical justification or details on hyperparameter tuning.

ans)

First, we sought a simple, yet nonlinear, regression model that would learn quickly and be insensitive to over-parameterization. This was due to the very limited dataset, 696 maps divided differently into training and test data. It was important not to have to fine-tune the model's hyper-parameters each time. Therefore, we did not use any hyperparameter optimization procedures.

As is well known, MLP networks are very resistant to overfitting, even in the version with only two hidden layers. With 13 inputs and 30-60 outputs, a reasonable number of neurons is 100, 200, or 300. Probably not more. With 200 neurons in each layer, this results in approximately 50,000 weights. Applying the Adam optimizer with optional parameters, as used in scikit-learn, to several random tasks proved to be a robust solution that is resilient to changes in input and test data.

(7) Figure 1 shows individual fluctuations in MAE values, but the reasons for these fluctuations (e.g., seasonality or multicollinearity) are not discussed in depth.

ans) This phenomenon occurs in all approximation models. It is a result of the limited amount of historical data and the errors resulting from the least squares criterion used in regression. It is worth noting that this variability results from the summing of error modules occurring in specific grid maps, the time intervals of which may differ.

(8) Figures 2 and 3 use scatter plots to show the relationship between predicted and true values, but the points overlap severely; density plots or box plots could be used instead.

ans) Indeed, the full drawings contain all the determined approximations, i.e., more than 97 $10^6$ points for the entire map and slightly fewer for the roads themselves. The figures have been removed and replaced with a simple comment and other images.

(9) PM10 prediction is based on PM2.5 measurements (Table 5), but the universal applicability of this substitution is not sufficiently justified (relying only on local data similarity).

We never claimed that this was a universal property, as we pointed out in our work. In contrast, in a different environment (different sources of pollution, different combustion process structure), the relationship between PM2.5 and PM10 and other pollutants may be completely different (see citation [3]: Analitis A, Barratt B, Green D, Beddows A, Samoli E, Schwartz J, Katsouyanni K. "Prediction of PM2.5 concentrations at the locations of monitoring sites measuring PM10 and NOx, using generalized additive models and machine learning methods: a case study in London..."). However, it was difficult to abandon this observation, and we decided to write about it, especially since measurements from PM25 monitoring stations have improved the accuracy of spatial CO predictions. The mentioned possible relationships form an interesting research problem that should be addressed interdisciplinarily.

(10) The CO prediction error is large (Tables 8 and 9), but the influence of the unit scale (inherently larger CO values) is not discussed.

ans) We considered this fact obvious. We have included a corresponding note in the text when discussing the results of the CO pollution approximation.

(11) The LGBM model shows extremely high prediction error for NOx (Table 16), but the reasons are not explored.

Generally, NOx prediction proved to be the most challenging problem for both approaches, likely because of the very high data variability. In the case of LGBM, an additional challenge is that this model performs a univariate regression. In the absence of terrain decomposition, the model provides an average response for all points on the grid map. However, with terrain decomposition, it requires the creation of several separate models (as in our case). For similar reasons, LGBM was significantly more difficult to train in longer training sequences. The advantages and disadvantages of the LGBM approach are discussed in more detail in this paper.

(12) The extensive use of tables for statistical prediction accuracy compromises readability; concise graphical representations should be used instead.

ans) The tables have been moved to the appendix and replaced by graphic illustrations. It should be emphasized that the tables contain very precise information regarding prediction errors with respect to space and time, which additionally expand the possibilities of using algorithms. They demonstrate how to select the training set and the applicability of the methods in various time-space situations.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The revision is improved. However, no special contribution can be found.

Author Response

The revision is improved. However, no special contribution can be found.

Ans) We've rewritten the Discussion and Conclusions section to highlight new aspects of our contributions.

The entire work has been revised for greater precision of expression, including explanations for figures and tables. All drawings created in Matplotlib were reviewed and, where necessary, improved for quality. Figure 1 is supplemented with a link to a publicly available online map of the city, where all the details can be viewed.

Thank you for all your comments and observations

Reviewer 3 Report

Comments and Suggestions for Authors

Based on the uncertainty in the model results from the previous comments and other factors, the authors still need to expand the discussion section by incorporating relevant literature to thoroughly explore the reasons behind changes in urban air pollutants.

Additionally, formatting issues such as unclear standards in the figures must be revised. The term "New" in the paper’s title is not appropriate, as the authors did not propose a new or improved model but merely applied traditional machine learning models. The title and related descriptions in the text need to be modified.

The headings of many subsections are too vague in meaning. For example, "3.2 PM10" is not a complete or meaningful title that reflects the content of the subsection.

The authors must include accuracy evaluation metrics such as R² and RPD, and discuss the strengths and weaknesses of the model’s performance.

Author Response

Based on the uncertainty in the model results from the previous comments and other factors, the authors still need to expand the discussion section by incorporating relevant literature to thoroughly explore the reasons behind changes in urban air pollutants.

Ans) We expanded on this topic in the introduction and additionally cited works [16] and [22]. This issue is also extensively represented in the works [24] and [27].

2a. Additionally, formatting issues such as unclear standards in the figures must be revised.

Ans) All drawings created in Matplotlib were reviewed and, where necessary, improved for quality. Figure 1 is supplemented with a link to a publicly available online map of the city, where all the details can be viewed. On the pollution maps in Figures 2, 3, 5, and 8, we refer to this area using the 112 geodetic coordinate reference system for Poland, with the EPSG code 2180 and this information is added in the text.

The term "New" in the paper’s title is not appropriate, as the authors did not propose a new or improved model but merely applied traditional machine learning models. The title and related descriptions in the text need to be modified.

Ans) To avoid controversy, we changed the word "model" to "approach" in the paper's title.

We believe the problem stems from differences in how the term "model" is treated across disciplines. In computer science, a model is a broader concept that encompasses a method of defining input and output, the type of activation function, and often the learning algorithm. We expand the novelty aspect in the method description and in the Discussion and Conclusions section.

3) The headings of many subsections are too vague in meaning. For example, "3.2 PM10" is not a complete or meaningful title that reflects the content of the subsection.

Ans) We have expanded the subsection titles to: "Spatio-temporal prediction results for...".

4). The authors must include accuracy evaluation metrics such as R² and RPD, and discuss the strengths and weaknesses of the model’s performance.

ANS) We have supplemented the results with Y2 (and the RPD value, which is directly related to Y2). In fact, we monitored the Y2 results throughout all testing. We also determined the correlation coefficient, because in some works on pollution modeling it was used instead of Y2.

5) We've rewritten the Discussion and Conclusions section to highlight aspects of our contributions. We have made every effort to improve the precision of the results presentation.

Round 3

Reviewer 3 Report

Comments and Suggestions for Authors

The paper does not propose a novel method; the word "new" is inappropriate for use in the title. Additionally, the labels in all figures are unclear.

Author Response

1) The paper does not propose a novel method; the word "new" is inappropriate for use in the title.

ANs. We changed the title of the work to "Predicting Concentrations of PM2.5, PM10, CO, VOC, NOx in the
Urban Scale Using Machine Learning-Based Surrogate Models"

2) Additionally, the labels in all figures are unclear.

Ans. The labels in the drawings have been updated, and the new ones have been carefully prepared.