Testing and Validation of Automotive Point-Cloud Sensors in Adverse Weather Conditions
Round 1
Reviewer 1 Report
Overall, I found the paper interesting and I like the way snow conditions were addressed.
About the results,
Fog chamber
Showing the results with 90% is fine however I would appreciate to see also what happened with less reflective target. I may imagine that variations are higher.
Seeing decreasing variations as LiDAR are loosing the view on target is rather counter intuitive. I would expect to see increasing variations. How do you explain that ?
I am interested to know whether you have configured the LiDARs in a particular way. For example, did you use any gating ?
Snow test
I do not understand figures 7, 8 and 9. Are they intensity images ? In which part of the image should we see some parts of the leading vehicle ? What is the size of the field-of-view displayed ? Where is the LiDAR in these images ? What are the distances measured ?
Author Response
Added the images from the 5% reflective target.
When observing the intensity images of the tests, there appear more random outliers in front of the targets as the fog density decreases. It is not entirely clear to us why this happens. One reason could be that the "wall" the fog creates is still present at these distances and visibilities even though it is not visible and not as thick and unpenetrable than in the densest scenarios.
LiDARs were used "as is" and we did not configure them in any special way.
Yes, the snow test images are intensity images. We've tried to be more specific about what is seen in these images according to your suggestions and added these changes to the manuscript.
Reviewer 2 Report
the author presents the measurement results of different LiDAR sensors in adverse weather conditions. The results are interesting and can be a guidance for LiDAR sensor selection.
Some comments are following:
1. In figure 4, 5, 6, it seems the sensors still can detect the object even at visibility of about 5m. Can you please explain the reason ?
2. There are two types of targetswith 90% and 5% refletivity. Only the 90% results are presented in detail, but the 5% is just mentioned in text without figures. It is better to plot and compare the results with figures.
3. All these sensors have different specs, such as central wavelength, frame rate, laser power, FOV, system power consumption, etc. It is not convincing to present the results without comparing the specs. It is highly recommended to make a comparison table of the sensors. And based on the table, the author could make a prediction to the performance in the measurements.
4. Some figures are not referred correctly, e.g. figures in line 247, 253, 259
Author Response
Apparently, the figures in question were imprecise in this matter. There are no measurements done with a visibility set to 5 m. The performed tests were done with visibilities of 10 m, 15 m, 20 m, 25 m, 30 m, 40 m and 50 m. We decided to put all the variation values in the same graph for each target distance in order to make the comparison easy. We've updated the graphs a bit to make them more understandable, hopefully succeeding.
The results from 5% target are now added to the manuscript.
Table containing the sensor specifications has been added to the manuscript, and some light comparison of the sensors is described.
Checked the references, they ought to be in place now.
Reviewer 3 Report
Review of Testing and Validation of Point-Cloud Sensors in Adverse Weather Conditions by Jokel et al., 2019
The manuscript fits within the journal scope, as it describes and analyzes different lidar sensors in adverse meteorological weather conditions for range sensing in unmanned vehicles. However, authors work point out something very obvious in atmospheric physics: lidars don’t work well in a turbid atmosphere. This is related to the atmospheric optical depth: when rain, fog, clouds or snow are present, the simplified lidar equation below shows that the received power reduces with a factor depending on the exponential of the optical depth, that is the product of the extinction by the traveled distance
Where P(r)is the received power r from range r, P0is the initial power and a(r)is the extinction coefficient in m-1. Now the extinction coefficient can be the sum of different contributions as specified in Eq. 2 in Lolli, S., P. Di Girolamo, B. Demoz, X. Li, and E.J. Welton, 2017: Rain Evaporation Rate Estimates from Dual-Wavelength Lidar Measurements and Intercomparison against a Model Analytical Solution. J. Atmos. Oceanic Technol., 34, 829 839.
, as molecules, aerosols, raindrops and clouds.
In case of fog, it should be taken into account that the presence of a strong absorption band between 900 and 950 nm. In this case those instruments using this wavelength will have very poor performance with respect to those using longer wavelengths (e.g. 1550nm). This is shown in https://www.atmos-meas-tech.net/11/2459/2018/. In Table 1 it should be added a column reporting this information.
As a general comment, it can be very interesting to quantitatively assess thresholds for rain, fog or snow (in terms of optical depth) beyond which the lidar instrument can’t detect the target anymore. However, for this work I think is very difficult to assess as the tests have been performed already. I suggest to the authors to add a paragraph on this topic saying that the research will be performed in future.
How much is the minimum Signal-To-Noise ratio for each different target to be detected ? It can be useful to add a paragraph about it.
Specific comments can be found in the attached file
Comments for author File: 
Comments.pdf
Author Response
It is not entirely clear what you mean about adding a column for wavelength information. In Table 1, the last row contains information of each sensor's laser wavelength. Is this what you intended or did you wish us to highlight their expected performance in some manner?
We have added paragraphs concerning the future research and SNR to the manuscript.
We've read and considered your comments in the PDF you provided and have edited the manuscript accordingly. We've modified it so that it would be more clear that this paper concentrates on the automotive LiDARs and their performance and testing. Of course, development for different applications using LiDARs and experience from them could be more carefully considered in future research.
Reviewer 4 Report
Referee Report
Title: Testing and Validation of Point-Cloud Sensors in Adverse Weather Conditions
Manuscript ID: applsci-484688
By Jokela et al
Submitted to Applied Sciences
Comment
This work compared the performances of different LiDARs sensors under the fog and snow weather conditions. From the results, the authors concluded that all performances of the sensors degraded as expected. In this work, I have the following suggestions for further improvement
The concept, characteristics and features of each sensors should be mentioned, compared and discussed in the manuscript. So that readers would know how and why they performed badly under the poor weather condition.
The authors should make a conclusion on which senior has the best performance on which weather conditions.
The turbulent snow test is on the road so it is not standardized. The authors should try their best to describe the environment condition in the test.
There is no error bars in Figs. 4-6.
Some schematic drawings on Figs. 7-9 would be helpful to understand the positions of the sensors and vehicles.
Author Response
We added table containing specifications of the sensors and preceding it some small discussion regarding their differences and similarities to the second version of the paper. I wonder if this is enough or should there be even more detailed description of them. I also considered adding more details to the conclusions part but felt that all the necessary had been said already.
Additionally, the second version contained turbulent snow images where the sensors were marked with a red dot. We've now added a leading vehicle to the pictures as well.
From turbulent snow tests we removed the part mentioning tests performed on the road. They were not presented in the results so mentioning them in the paper seemed actually irrelevant.
Round 2
Reviewer 2 Report
The author has fixed the all points. I don't have further comments.
Author Response
Wonderful!
Reviewer 3 Report
I understand that the lidar technology is applied for automotive purposes, but the authors, having probably a different background, ignore the physics principles that are valid, independently on the purposes. As an example, in lines 362 363 it is stated:
There are no ideal fog, rain or snow conditions which create a single threshold when the objects are not visible anymore. Moreover, this highly depends on LiDAR type.
it is not the rain or fog that it is ideal (ideal definition even doesn't apply in this case), but there exist physical parameters to characterize the two processes, i.e. rain intensity or fog optical depth. And the detection threshold is then a characteristic of each single instrument (e.g. Cepton can detect a target with precipitation lighter than XX mm/h)
As computer vision and automotive it is not my domain, and all the other reviewers seem to be happy, I will overlook my concern, even if this journal is called "applied science" and I accept the manuscript in present form
Reviewer 4 Report
I can see in this revision that the authors added Table 1 and modified Figs 10-12 as per my comments in the first revision. The quality and presentation of this work are greatly improved. I have no further question in this revision.
Author Response
Wonderful, thank you!
