Extending Light Direction Reconstruction to Outdoor and Surface Datasets

Abstract

Immersive augmented reality (AR) requires consistent illumination between synthetic scene content and the real world. Recent deep neural networks achieve this by reconstructing important illumination parameters, though the performance was measured in clean laboratory environments. This study shows that the reconstruction performance of a recently proposed network architecture remains satisfying in more complex real-world outdoor scenarios. The labelled outdoor scenario dataset used in this study is provided as part of this publication. The study further reveals how auxiliary surface (in this scope, depth and normal vector) information affects the reconstruction performance of the network architecture.

1. Introduction

The registration problem in AR applications [1] describes the issue of achieving both consistent spatial placement of virtual content, known as the geometric registration problem, and coherent illumination between the real and synthetic world, known as the photometric registration problem. Diverging illumination between the real and the synthetic world may disrupt immersion in AR applications. When rendering virtual content in AR applications, it is important to apply suitable illumination parameters so that the virtual illumination resembles the real world situation. Immersion may be disrupted particularly easily, however, if the shading and the direction of cast shadows in the real and synthetic world significantly mismatch. In the past years, several deep neural network (DNN) approaches have been published to address this issue. Kán and Kaufmann [2] reconstructed the dominant light direction from red-green-blue-depth (RGB-D) images using a ResNet, yet figured that sole use of red-green-blue (RGB) data did not work for their approach. Miller et al. [3] modified the VGG-16 classification architecture [4] by introducing ${\mathrm{Net}}_{s_x,s_y}$, which regresses the direction of the dominant point light source in stereographic coordinates from RGB input and eliminated the need for additional depth information. Later, they improved the prediction performance of ${\mathrm{Net}}_{s_x,s_y}$ by analysing its internal decision process [5], leading to architectural adjustments and an optimized synthetic training dataset.

Apparently, reconstructing the light direction is feasible both with regular RGB data and RGB data with additional depth information. Consider the reflective light term

$${\int }_{\Omega }{f}_r(x,{\omega }_i,{\omega }_o)L_i(x,{\omega }_i)\left({\omega }_i^{T}n\right)d{\omega }_i$$

(1)

of the rendering Equation [6], which incorporates the following components:

• f_r = bidirectional reflectance distribution function (BRDF);
• L_i = incident light radiance;
• x = location to be lit;
• ω_i/ω_o = incident/outgoing light direction;
• Ω = entirety of ω_i from the hemisphere above x.

The projective term (cyan) incorporates information about the reflective surface, particularly the angle between ${\omega }_i$ and the orientation n of the surface. Commonly, n is represented by depth or normal vector information (hereafter referred to as surface information, which may generally include BRDF data but is explicitly excluded). This lets one assume that feeding additional surface data as auxiliary information to a network, though not essential, may ease the reconstruction task, as only $f_r$ and $L_i$ need to be estimated, which may help to achieve better reconstruction performance. RGB and depth information are sensor mappings of the same scene image and can be considered as multimodal data [7], which can be fed to a DNN either as early-, intermediate- or late-fusion. In this study, it is investigated whether the described multimodal data fusion indeed eases the illumination reconstruction task, leading to more precise illumination reconstruction.

Reconstruction approaches that are based on more recent computer vision architectures, such as ResNet [8] and ConvNeXt [9], perform slower or less accurate than ${\mathrm{Net}}_{s_x,s_y}$ (Appendix A.1). Therefore, this study takes up the architectural adjustments suggested by Miller et al. [5], introducing the fully convolutional dense network (FCDN) architecture and evaluating its performance on the previously used test dataset ${\mathrm{T}}_{\mathrm{REAL}}$. As ${\mathrm{T}}_{\mathrm{REAL}}$, though containing real-world images, constitutes a benevolent dataset due to its photos taken under controlled lab conditions, the generalisability of FCDN is demonstrated by using a newly generated labelled outdoor dataset, which is published alongside this paper (https://sites.hm.edu/ccbv/projekte_6/projekte_detail_49024.de.html, accessed on 25 November 2025). While capturing real outdoor illumination conditions, the outdoor dataset is limited in that sense that the images of the dataset so far were recorded during high summer in Central Europe, meaning the images mostly depict cloudless, sunny scenes with little to no wind influence.

The main contributions of this study to the current state of scientific knowledge is summarized as follows:

• Improvement of reconstruction error to $3.8°$ on reference test data via FCDN architecture.
• Publishment of a labelled outdoor dataset suitable for illumination reconstruction that can be further expanded in the future.
• Demonstration of the presented architecture’s ability to generalise to real scenarios (with rudimentary domain adaptation).
• Investigation on whether multimodal (RGB-D, RGB-N) network architectures have any effect on the reconstruction performance.

2. Related Work

In the past, several DNN approaches to reconstruct illumination have been published, and they can be categorised into two research branches. Approaches in the first branch aim to generate illumination textures for image-based lighting (IBL) renderers by using DNNs to generate such textures directly or from reconstructed parameters to approximate illumination in indoor [10,11,12], outdoor [13,14] or both scenarios [15]. The second branch focuses on reconstructing specific illumination parameters, which is what this study targets.

Sommer et al. [16] employed a DenseNet-121 to reconstruct the illumination parameters $\{\overrightarrow{d},c,a,o\}$, with $\overrightarrow{d}$ being the direction towards the light source, c being the light colour, a being the ambient colour and o being the opacity value of a shadow casting texture, which is created pixel-wise by another fully connected DNN and used in their approach to generate soft shadows. The DenseNet-121 is trained on a modified variant of the Laval indoor HDR dataset [10], which is frequently used in IBL-based approaches, and receives RGB input images with a resolution of $256\times 192$. The approach requires the shadow casting DNN to be specifically trained for a certain model and is limited to planar shadow receivers. Providing an additional depth channel, Kán and Kaufmann [2] feed RGB-D images to a residual DNN and aim to predict the direction to the dominant light source in a given scene. The output light direction is given as a tuple of azimuth $\varphi$ and elevation $\theta$. The residual DNN is trained on synthetic RGB-D data and tested in real-world scenarios, resulting in an overall average error of $28.4°$ between the reconstructed light direction and the ground truth. Miller et al. [3] proposed a VGG-16-like ${\mathrm{Net}}_{s_x{s}_y}$ to reconstruct the dominant light direction in stereographic coordinates from RGB images. They conducted experiments with ${\mathrm{Net}}_{s_x{s}_y}$ being trained on solely synthetic, solely real and mixed data and determined that mixed training data performed the best with an average angular error of $7.1°$ on the used test dataset ${\mathrm{T}}_{\mathrm{real}}$, which contained solely real-world indoor images. By analysing meta information, which arise as secondary data when applying explainable artificial intelligence (XAI) approaches, Miller et al. [5] derived the linear feature aggregation network (LFAN) and fully convolutional network (FCN) architectures, improving the average angular error when reconstructing the dominant light direction to $6.2°$ and $5.7°$ on the reference test set ${\mathrm{T}}_{\mathrm{real}}$, respectively. Optimizing the synthetic training dataset, Miller et al. further improved the average angular error to $4.4°$ on ${\mathrm{T}}_{\mathrm{real}}$ with the proposed ${\mathrm{Net}}_{\mathrm{RTS}}$, which is based on ${\mathrm{Net}}_{s_x,s_y}$. The input data is optimized by investigating the influence of synthetic training data, rendered with different illumination models and shadow algorithms, on the reconstruction performance. The best results were achieved with the Oren–Nayar [17] and Cook–Torrance [18] illumination models and ray-traced soft shadows [19], used in ${\mathrm{Net}}_{\mathrm{RTS}}$. Combining the findings from architecture and dataset improvements, Miller et al. suggested adjustments to their proposed architectures to reduce potential over-fitting and rebuild its ability to generalise.

3. FCDN Evaluation on RGB Data

The FCN architecture [5] consists of the VGG-16 convolutional section $C_1$ to $C_5$, which is extended by two convolutional blocks $C_6$ and $C_7$ to expand the extracted feature regions to the entire input image. Each additional block consists of one convolutional layer and a max-pooling operation. The entire convolutional section is followed by a single linear output layer $L_o$. The FCN architecture, however, suffers from an inherent tendency to over-fit and presumed poor generalizability due to its inability to train well with augmented training data. Therefore, the FCDN architecture introduces a single dense layer D with a rectified linear unit (ReLU) activation function between $C_7$ and $L_o$ (Figure 1) to investigate if these adjustments overcome FCN’s architectural limits, as postulated by Miller et al. [5].

Like the training process of the predecessor models, the training of the FCDN architecture follows the two-phased Keras transfer learning and fine-tuning process. In the first phase, the model’s hyperparameters are optimized with KerasTuner [20] on a reduced dataset of 20,000 images over five epochs. In this phase, the convolutional section $C_1$ to $C_5$ is initialized with the weights of ${\mathrm{Net}}_{\mathrm{RTS}}$ as it showed best performance on ${\mathrm{T}}_{\mathrm{REAL}}$. During the optimization process, the weights of $C_1$ to $C_5$ are not updated after each training epoch (the trainable flag is set to false). KerasTuner optimizes the filter size of $C_6$ and $C_7$, the number of neurons of D, the dropout value, the learning rate, the batch size and the optimizer. In the second phase, the most promising variant is fine-tuned on the full dataset of 100,000 images for up to 250 epochs. During the fine-tuning training, the weights of $C_5$ are updated each epoch (trainable flag is set to true) and the learning rate is set to a tenth of the original rate, so $C_5$ gradually adjusts to the tail section and the new task. If the validation metric has not changed over a period of 12 epochs, the learning rate is reduced by a Keras learning rate reducer module. If the validation metric has not changed after another 20 epochs, the training is stopped early by a Keras early stopper module. In both phases, the datasets are split with a split ratio of 80:20 (ignoring rounding) for training and evaluation.

Miller et al. [5] used different datasets: one synthetic training and evaluation dataset ${\mathrm{DS}}_{\mathrm{syn}}$ from the same domain as the synthetic test dataset ${\mathrm{T}}_{\mathrm{syn}}$, one real training and evaluation dataset ${\mathrm{DS}}_{\mathrm{real}}$ from the domain of the real indoor test dataset ${\mathrm{T}}_{\mathrm{real}}$ and several synthetic datasets that were generated with different illumination models. As combinations of synthetic and real data showed to produce the best results [3,5], the best-performing synthetic datasets (${\mathrm{DS}}_{\mathrm{syn}}$ and ${\mathrm{DS}}_{\mathrm{RTS}}$) described by Miller et al. [5] are mixed with ${\mathrm{DS}}_{\mathrm{real}}$. Besides ImageNet [21], pre-trained base weights for $C_1$ to $C_5$, base weights of ${\mathrm{Net}}_{s_x{s}_y}$ and ${\mathrm{Net}}_{\mathrm{RTS}}$ are used in the transfer learning process. Each combination is used to train a dedicated FCDN model. Eventually, all FCDN models are evaluated on ${\mathrm{T}}_{\mathrm{real}}$, revealing the model using pre-trained weights of ${\mathrm{Net}}_{s_x{s}_y}$ and ${\mathrm{DS}}_{\mathrm{RTS}}$ to perform best with an average angular prediction error of $3.8°$ (Appendix A.4).

4. Real-World Generalisation

DNNs used to reconstruct illumination only provide real benefit if they perform satisfyingly under real conditions. To evaluate DNNs regarding their performance to generalise on real-world data, a test dataset of reasonable size is required. However, the major drawback of ${\mathrm{T}}_{\mathrm{real}}$ is its limited size. The Laval indoor HDR panorama dataset contains more than 2100 images and the Laval outdoor HDR panorama dataset [14] contains 205 images. Both datasets could be used to extend ${\mathrm{T}}_{\mathrm{real}}$. Unfortunately, both Laval datasets are unlabelled, so labelled training images are not easily obtainable, especially from panoramas which do not clearly show the dominant light source, making most of the Laval indoor HDR panorama dataset and a quarter to half of the images in the Laval outdoor HDR dataset unusable for this purpose. Moreover, earlier iterations [3,5] were trained on data with anchor objects in the foreground, simulating common setups of AR environments. This particular image domain is not captured in the Laval datasets. In consequence, the proposed FCDN architecture needs to be trained on training data of similar domain, to remain comparable to earlier iterations. Hence, the Laval datasets are not suitable for this particular approach. ${\mathrm{T}}_{\mathrm{real}}$ could have been extended; however, indoor data generation would have required high levels of automation for both camera and light placement to effectively extend the dataset, which were not available.

Instead, a labelled outdoor dataset is generated, as the generation process for this kind of data can be performed by simply deploying a self-written data gathering application to a mobile device with a suitable camera and taking photographs in specified time intervals from an outdoor scene, which is illuminated by the sun, reducing the automation effort to measuring required references and placing the camera once a day. The image data used in this study is gathered mostly under fair weather conditions due to geographical and seasonal circumstances at the time and location of the recording. Cloudy weather conditions would cause the perceived illumination to be very diffuse and, thus, the light direction is difficult to determine from illumination effects under such weather conditions for both humans and the introduced DNN. Nevertheless, the introduced DNN aims to reconstruct the dominant light direction in a given scene that is illuminated by one main light source—ideally one point light source. Given that scenes with very diffuse illumination, e.g., under said conditions, do not have typical dominant light directions, the limitation of the outdoor dataset to mostly fair weather conditions does not constitute a problem for this study.

4.1. Recording Setup

The device is mounted to a tripod and focused on the captured scene. In the foreground of the scene, one or more different 3D-printed objects are placed on a table, which is covered by differently coloured tablecloths (Figure 2).

In the foreground scenes, five different models (a white Stanford bunny, a textured Stanford bunny, a white Stanford Buddha, a purple Stanford Buddha and a white Stanford dragon) and four different tablecloths (a yellow, a blue, a green and a textured one in a Bavarian style) are used. In total, 2955 images are captured and organised (Appendix A.2.3) into a usable fraction D of 2425 images, a fraction of landscape photos S with 189 images and a fraction U containing 341 unused images (e.g., the scene was fully shaded by a tree, the camera tripod was blown over by the wind or similar).

This dataset structure not only allows the investigation of how well DNN models generalise on real-world outdoor data, but also how robust those models are with respect to slight changes to the otherwise same domain.

4.2. Labelling Outdoor Photographs

To label each image with the direction towards the light source, in this context the sun, the bearing in which the camera is pointing and the position of the sun is required. The position of the sun can be computed from a given point in time and the location on earth. The DNNs of previous approaches [3,5] assumed clockwise labelling with light directions originating from the left side in an image to be light azimuth angles of $90°$ and light directions coming from the opposite side of the scene (casting a shadow towards the camera) to be caused by light azimuth angles of $180°$. In the context of earth locations and following this labelling convention, the computed sun’s position could be directly used as a label if the device’s orientation is facing southwards (Figure 3).

To capture scene images illuminated from as many light directions as possible, the images are taken with the device facing various cardinal directions, which needs to be incorporated in the labelling process. For this, the device’s bearing is required, which could be measured with a compass. However, determining the bearing with a compass or gyroscope sensor may not produce measurements accurate enough for the labelling process, due to scale resolution when reading a compass and technical inaccuracies of the gyroscope sensor. To increase the accuracy, the device’s bearing $B_D$ is computed between a distant ($1.0 \mathrm{km}$ to $1.4 \mathrm{km}$) reference point $P_r$, previously located with the global positioning system (GPS) sensor, and the device’s GPS location $P_d$ (Appendix A.2.1).

With $B_D$ and the sun’s azimuth and elevation, the actual image labels indicating the direction to the sun can be computed. As $B_D$ and the sun’s azimuth and elevation are given in the context of spherical coordinates, the dataset labels are computed in this representation and later converted into stereographic coordinates (as described in Miller et al. [3]). Since the elevation of the sun ${\theta }_{\mathrm{sun}}$ is independent from $B_D$, ${\theta }_{\mathrm{sun}}$ can be directly assigned to the elevation label $L_{\theta }$. To compute the azimuth label $L_{\varphi }$, the azimuth of the sun ${\varphi }_{\mathrm{sun}}$ requires to be adjusted by $B_D$ (Appendix A.3.1), resulting in the final spheric label $\left(L_{\varphi },L_{\theta }\right)$, which is eventually converted into the stereographic label $\left(L_{s_x},L_{s_y}\right)$. The ground truth labels of the dataset itself incorporate an inherent measurement error between $0.82°$ and $1.15°$ (Appendix A.2.2), which is important to keep in mind when evaluating a DNN’s performance.

4.3. Evaluation Strategy

Predicting illumination in an unknown domain is a difficult task. With high certainty, models trained on different domains, especially without any incorporated domain adaptation measures, are going to perform worse in unknown, more complex domains. With this in mind, each subset of the aforementioned dataset D with reasonable size is split into a test dataset and a training/validation dataset, intended to be used for an adaptation training process. Due to their small size, the data subsets S, $D_{001}$ and $D_{010}$ are used in their entirety in the test datasets $T_{\mathrm{S}}$, $T_{\mathrm{D}001}$ and $T_{\mathrm{D}010}$. The remaining subsets are split into test subsets $T_{xy}$ and training/validation subsets $D{S}_{xy}$ with $xy$ indicating the specifically matching subset (

Table 1. Subset split with number of contained images in parentheses.

Origin		Test	Training/Validation
$D_{10}$	↦	$T_{\mathrm{D}10}$ (701)	$D{S}_{\mathrm{D}10}$ (1000)
$D_{100}$	↦	$T_{\mathrm{D}100}$ (195)	$D{S}_{\mathrm{D}100}$ (278)
$D_{101}$	↦	$T_{\mathrm{D}101}$ (506)	$D{S}_{\mathrm{D}101}$ (722)
$D_{11}$	↦	$T_{\mathrm{D}11}$ (200)	$D{S}_{\mathrm{D}11}$ (440)
$D_{110}$	↦	$T_{\mathrm{D}110}$ (80)	$D{S}_{\mathrm{D}110}$ (180)
$D_{112}$	↦	$T_{\mathrm{D}112}$ (120)	$D{S}_{\mathrm{D}112}$ (260)

).

Before usage, the training/validation subsets $D{S}_{\mathrm{Dxyz}}$ are split with a ratio of 80:20 into respective sets.

The adaptation training is performed using the FCDN architecture with ImageNet pre-trained weights for $C_1$ to $C_5$. Different models are trained with specific training subsets or combinations to investigate the architecture’s ability to generalise in certain situations; for instance, how well a model that is trained for outdoor scenes with single objects performs on the aforementioned $T_{\mathrm{REAL}}$ of indoor images, or on outdoor scenes with multiple objects. The training process is conducted as described earlier (Section 3, second paragraph).

5. Influence of Surface Data

Kán and Kaufmann [2] found their network to be unable to properly reconstruct the dominant light direction when only using RGB instead of RGB-D information. In contrast, Miller et al. [3] showed in their approach that it is possible to reconstruct the dominant light direction solely from RGB data. From a theoretical point of view, one can argue that a DNN is capable of performing an end-to-end transformation of the illumination situation, given by a single RGB image, to a directional vector representing the dominant light direction, but having additional information about the shape of the surfaces in the RGB image may help to ease the task of doing so. Considering the incidental light part of the reflective light term (Equation (1), cyan part) in the rendering Equation [6], the angle between the orientation of the surface, denoted by the normal vector n, and the light inciding from a certain direction ${\omega }_i$ determines how much light from ${\omega }_i$ is actually received at the investigated location x. So, when trying to reconstruct the dominant light direction, a DNN is trying to map the reflected light $L_o$ of each pixel visible in the input image to the most significant inciding light direction ${\omega }_i$ without knowledge about each pixel’s material ($f_r$), radiance ($L_i$) and surface (n). Considering the mentioned surface orientation term $\left({\omega }_i^{T}n\right)$, it is a reasonable attempt to improve a DNN’s prediction result by providing additional information about the orientation of the surface, either as depth information from a sensor or as normal vector information, which may be reconstructable, yet non-trivially, from depth data [22]. With the provided surface orientation information, a DNN would only have to estimate values for $f_r$ and $L_i$ when reconstructing the dominant light direction. This study investigates if providing additional surface information indeed improves the prediction performance or if DNNs may inherently learn surface structures, not requiring additional information input, when confronted with a light direction reconstruction task. The investigation is conducted on synthetic data, as the labels of this data do not incorporate any measurement errors. Further, additional surface information is not expected to improve the reconstruction performance in real-world scenarios if it does not improve the results under synthetic laboratory conditions.

5.1. Fusing Data

Surface information, either as depth or normal vector data, can be passed to a DNN either as early, late or intermediate fusion (Figure 4).

In intermediate and late fusion settings, the feature extraction needs to extract meaningful information in respect to the data sample label only from surface information. However, since the shape of the surface is not influenced by light angles, surface information is entirely independent from illumination and, therefore, a DNN in principle cannot infer illumination from surface information alone, as empirically confirmed. By fusing colour and surface information right after the input layer in an early fusion approach, meaningful features are extracted from both multimodal sensor data, so that the decision part may infer the direction to the dominant light source more accurately. After passing the data augmentation stage, which applies random zoom and shifting to both information types and random brightness to the colour features, colour and surface information are concatenated and eventually passed to the first convolutional layer of the DNN, leading to a kernel depth from three (RGB only) to four in case of RGB-D and six for red-green-blue-normal (RGB-N) information, respectively.

5.2. Training with RGB-D and RGB-N

To investigate the influence of surface information on the prediction performance, DNNs of the aforementioned FCDN architecture are trained for RGB-D and RGB-N information with laboratory data, i.e., synthetic data (80,000 for training, 20,000 for validation and 10,000 for testing) with depth and normal information, and compared to a baseline FCDN network ${\mathrm{Net}}_{\mathrm{base}}$, which uses pre-trained weights of ${\mathrm{Net}}_{\mathrm{RTS}}$ and is trained solely on RGB information. Since multimodal input features require all layers to be trained, all layers beginning with $C_1$ in ${\mathrm{Net}}_{\mathrm{base}}$ are trained, which differs from the described training procedure (Section 3). Throughout all datasets, the RGB information remains the same and is rendered with the parameters of the best-performing dataset ${\mathrm{DS}}_{\mathrm{RTS}}$ from Miller et al. [5]. In summary, three datasets are compiled, consisting of solely RGB information (${\mathrm{SDS}}_{\mathrm{RGB}}$), RGB-D information (${\mathrm{SDS}}_{\text{RGB-D}}$) and RGB-N information (${\mathrm{SDS}}_{\text{RGB-N}}$). Corresponding test datasets are called ${\mathrm{T}}_{\mathrm{SDS}}$ without additional surface data indication, as this information can be derived from the context. Depth and normal information is generated simultaneously with the RGB information and stored in separate files, containing information as viewed by the recording camera (Figure 5).

Before being stored into a file, the normal information is packed into an interval of $[0, 1]$ (Appendix A.3.2). When being passed to a DNN, the normal information remains unconverted, as converting it back into an interval of $[-1, 1]$ would most likely lead to incompatibility in regards of the used ReLU activation function, considering the range $[0, 1]$ of RGB data, which the normal data is supposed to be merged with. Apart from that, normal data is usually computed from depth data, commonly recorded by depth sensors, and may consequently be mapped into the interval $[0, 1]$ while doing so.

To profit from pre-trained model weights in the given early fusion setup, the weights of the convolutional section in ${\mathrm{Net}}_{\mathrm{RTS}}$ are loaded into $C_1$ to $C_5$. However, these pre-trained models do not possess weights for additional layer weights that are required to process the concatenated surface information of the input. Facing a similar problem in a near-infrared (NIR) context, Liu et al. [23] initialized missing layer weights with the average of existing RGB weights. Yet, NIR is more closely related to RGB than surface information, leaving random initialization as another approach. Both methods are investigated, yet average initialization consistently ranked behind random initialization. The weights in the remaining layers are initialized randomly.

Hyperparameters commonly influence the prediction performance of DNNs significantly. Thus, two hyperparameter strategies $S_A$ and $S_B$ are investigated to isolate the effects of additional surface information on the reconstruction performance. The first strategy $S_A$ uses the same hyperparameters as ${\mathrm{Net}}_{\mathrm{base}}$ to solely investigate the influence of additional surface information. This favours RGB information, as feature extraction and decision making section are not adapted to additional surface information. Consequently, strategy $S_B$ optimizes the full range of hyperparameters: the optimizer, the learning rate, the batch size, the filter size of $C_6$ and $C_7$, the number of neurons in the dense layer, the dropout rate and whether to use a bias in the output layer, allowing all layers to adjust to the additional surface information.

In addition to the transfer learning methods used to train ${\mathrm{Net}}_{\mathrm{FCD}}$, the fine-tuning is conducted with an unmodified learning rate to gain further insights and to rule out possible unnoticed side effects when investigating only one of these methods. For instance, the effect sizes could be below the significance level if investigated only with a reduced learning rate.

5.3. Statistical Investigation

The comparison between surface information models and ${\mathrm{Net}}_{\mathrm{base}}$ on the same test samples constitutes a test with paired samples. Therefore, to determine whether surface information improves the average illumination reconstruction error, Wilcoxon signed-rank tests [24] between the multimodal FCDN models and ${\mathrm{Net}}_{\mathrm{base}}$ are conducted with a significance level of $\alpha =0.01$. However, due to the large sample size of the test dataset (10,000 samples), the test is likely to show significant differences between the multimodal FCDN models and ${\mathrm{Net}}_{\mathrm{base}}$. Thus, the Pearson correlation coefficient r [25] is computed in addition, to determine the effect size $\left|r\right|$ and its classification [26] in small, medium and large effect sizes (Appendix A.3.3) as a more descriptive metric. In that sense, the effect size tells how strongly surface information is influencing the illumination reconstruction performance.

Those tests and factors, however, do not reflect if certain multimodal FCDN models may have higher average prediction errors but in return may suffer less from extreme statistical outliers. Therefore, the resulting error distributions are additionally investigated with box plots.

6. Results

In this section, results regarding the FCDN architecture performance, the generalisability on real outdoor data and the effects of surface information on the reconstruction performance are presented.

6.1. Results: FCDN Architecture

The earlier published reference DNNs ${\mathrm{Net}}_{s_x,s_y}$ [3], LFAN, FCN and ${\mathrm{Net}}_{\mathrm{RTS}}$ [5], to which the FCDN architecture is compared, achieved average angular errors $e_{\measuredangle }$ on ${\mathrm{T}}_{\mathrm{real}}$ of $7.1°$, $6.2°$, $5.7°$ and $4.4°$ (

Table 2. Prediction performance of the reference DNNs on ${\mathrm{T}}_{\mathrm{REAL}}$.

	${\mathit{e}}_{\mathit{\measuredangle }}$ on ${\mathbf{T}}_{\mathbf{REAL}}$	Base Weights
${\mathrm{Net}}_{s_x,s_y}$	$7.1°$	ImageNet
LFAN	$6.2°$	${\mathrm{Net}}_{s_x,s_y}$
FCN	$5.7°$
${\mathrm{Net}}_{\mathrm{RTS}}$	$4.4°$	ImageNet
${\mathrm{Net}}_{\mathrm{FCD}}$	$3.8°$	${\mathrm{Net}}_{s_x,s_y}$

, upper section), respectively.

The lowest $e_{\measuredangle }=3.8°$ is achieved by the FCDN architecture ${\mathrm{Net}}_{\mathrm{F}\mathrm{C}\mathrm{D}}$ (Table 2, lower section) trained with a combination of ${\mathrm{DS}}_{\mathrm{RTS}}$ and ${\mathrm{DS}}_{\mathrm{real}}$ and pre-trained base weights from ${\mathrm{Net}}_{s_x,s_y}$. ${\mathrm{Net}}_{\mathrm{F}\mathrm{C}\mathrm{D}}$ uses 1205 filters in $C_6$ and 2466 filters in $C_7$. The dense layer D contains 4769 neurons and a dropout rate of $0.1$ is applied. The output layer $L_o$ does not use a bias. A Nadam optimizer is used with a learning rate of $3.6\times {10}^{-5}$ and the dataset is chunked into batches of size 16.

Considering the distributions of $e_{\measuredangle }$ (Figure 6) achieved by ${\mathrm{Net}}_{s_x,s_y}$, LFAN, FCN, ${\mathrm{Net}}_{\mathrm{RTS}}$ and ${\mathrm{Net}}_{\mathrm{F}\mathrm{C}\mathrm{D}}$, the error range of each distribution gets narrower with each architectural adjustment.

6.2. Results: Outdoor Data Generalisation

As expected, the selection of unadapted models (${\mathrm{Net}}_{s_x,s_y}$, ${\mathrm{Net}}_{\mathrm{RTS}}$ and ${\mathrm{Net}}_{\mathrm{F}\mathrm{C}\mathrm{D}}$) achieved unsatisfying prediction performance (

Table 3. Results on real outdoor data, depicting the prediction performance of unadapted DNNs in the upper and adapted DNNs in the lower section.

DNN	${\mathbf{T}}_{\mathbf{REAL}}$	S	${\mathbf{T}}_{\mathbf{D}0}$	${\mathbf{T}}_{\mathbf{D}10}$	${\mathbf{T}}_{\mathbf{D}11}$
${\mathrm{Net}}_{s_x,s_y}$	$7.1°$	$61.3°$	$63.5°$	$54.2°$	$45.9°$
${\mathrm{Net}}_{\mathrm{RTS}}$	$4.4°$	$60.6°$	$41.5°$	$52.4°$	$44.4°$
${\mathrm{Net}}_{\mathrm{FCD}}$	$3.8°$	$65.7°$	$47.5°$	$54.8°$	$49.2°$
${\mathrm{Net}}_{{\mathrm{DS}}_{\mathrm{D}10}}$	$33.3°$	$63.4°$	$48.2°$	$5.0°$	$36.3°$
${\mathrm{Net}}_{{\mathrm{DS}}_{\mathrm{REAL}}+{\mathrm{DS}}_{\mathrm{D}10}}$	$7.0°$	$62.5°$	$41.4°$	$4.8°$	$36.1°$
${\mathrm{Net}}_{{\mathrm{DS}}_{\mathrm{REAL}}+{\mathrm{DS}}_{\mathrm{D}10/\mathrm{D}11}}$	$6.7°$	$61.8°$	$39.6°$	$5.8°$	$5.7°$

, upper section) on the outdoor test datasets with angular errors $e_{\measuredangle }$ between $44.4°$ and $65.7°$.

Investigating the generalisability of the FCDN architecture (Table 3, lower section), ${\mathrm{Net}}_{{\mathrm{DS}}_{\mathrm{D}10}}$, trained on images of $D{S}_{\mathrm{D}10}$ (single textured and uni-coloured objects on uni-coloured surfaces), achieved $5.0°$ on the respective test dataset ${\mathrm{T}}_{\mathrm{D}10}$, though did not perform satisfyingly on ${\mathrm{T}}_{\mathrm{D}11}$ (multiple objects—uni-coloured and mixed—on uni-coloured surfaces), ${\mathrm{T}}_{\mathrm{D}0}$ (single and multiple uni-coloured objects on textured surface), S (scenery images) and ${\mathrm{T}}_{\mathrm{REAL}}$. Adding ${\mathrm{DS}}_{\mathrm{REAL}}$ to the training data in ${\mathrm{Net}}_{{\mathrm{DS}}_{\mathrm{REAL}}+{\mathrm{DS}}_{\mathrm{D}10}}$, its prediction error improved to $4.8°$ on ${\mathrm{T}}_{\mathrm{D}10}$ and $7.0°$ on ${\mathrm{T}}_{\mathrm{REAL}}$, yet remained unsatisfying on the remaining test datasets. ${\mathrm{Net}}_{{\mathrm{DS}}_{\mathrm{REAL}}+{\mathrm{DS}}_{\mathrm{D}10/\mathrm{D}11}}$, trained with $D{S}_{\mathrm{D}10}$, $D{S}_{\mathrm{D}11}$ and ${\mathrm{DS}}_{\mathrm{REAL}}$, eventually achieved $5.8°$ on ${\mathrm{T}}_{\mathrm{D}10}$, $5.7°$ on ${\mathrm{T}}_{\mathrm{D}11}$ and $6.7°$ on ${\mathrm{T}}_{\mathrm{REAL}}$, but did not perform satisfyingly on ${\mathrm{T}}_{\mathrm{D}0}$ and S. Further adjustment trainings for the domains of ${\mathrm{T}}_{\mathrm{D}0}$ and S have not been conducted due to small dataset sizes.

6.3. Results: Effects of Surface Data

When investigating the influence of surface information (

Table 4. Results of the trained strategies for RGB-D and RGB-N data with unmodified learning rate during fine-tuning and random initialization. Column ${\mathrm{T}}_{\mathrm{SDS}}$ contains the average angular error $e_{\measuredangle }$ and column p the p-value in relation to $\alpha =0.01$.

	${\mathbf{T}}_{\mathbf{SDS}}$	Strategy	p	r
${\mathrm{Net}}_{\mathrm{base}}$	$0.9°$	n/a	n/a	n/a
RGB-D	$0.9°$	$S_A$	<$\alpha$	$0.053$
RGB-N	$0.8°$		<$\alpha$	$0.167$
RGB-D	$0.7°$	$S_B$	<$\alpha$	$0.281$
RGB-N	$0.6°$		<$\alpha$	$0.461$

), the baseline DNN ${\mathrm{Net}}_{\mathrm{base}}$ achieved an error $e_{\measuredangle }$ of $0.9°$.

Applying $S_A$, the RGB-D experiment yielded a significant ($p<\alpha$), yet negligible (effect size $\left|r\right|<0.1$) difference. The RGB-N experiment achieved a significant difference ($p<\alpha$) with a small effect size of $\left|r\right|=0.167$. The RGB-D and RGB-N experiments applying $S_B$ both yielded significant differences ($p<\alpha$) with RGB-D showing a small effect size with $\left|r\right|=0.281$ and RGB-N showing a medium effect size of $\left|r\right|=0.461$. This behaviour is reflected in the distribution of the angular error $e_{\measuredangle }$ (Figure 7):

Graphs of the RGB-N experiments appear notably narrower with less pronounced outliers than graphs of the RGB-D experiments compared to the grey graph of ${\mathrm{Net}}_{\mathrm{base}}$.

The described results are based on experiments with randomly initialized missing multimodal pre-trained weights. Further, only the results of the transfer learning method with unmodified learning rate are depicted, as the modified learning rate results showed the same behaviour with higher $e_{\measuredangle }$. Additional results can be found on the project’s webpage.

7. Discussion

The average angular prediction error $e_{\measuredangle }$ indicating the dominant light direction in a scene could be improved to $3.8°$ on ${\mathrm{T}}_{\mathrm{REAL}}$ with ${\mathrm{Net}}_{\mathrm{FCD}}$. In known domains, the FCDN architecture performs satisfyingly, even under real-world conditions as demonstrated on ${\mathrm{T}}_{\mathrm{REAL}}$ and the real outdoor test datasets. However, it is limited in principle on unknown domains (for instance, ${\mathrm{Net}}_{{\mathrm{DS}}_{\mathrm{D}10}}$ achieved $5.0°$ on images with single objects on uni-coloured surfaces but deteriorated to $36.3°$ on images with multiple objects on the same surface) as it does not incorporate domain adaptation mechanisms. Adding the dense Layer D did not seemingly contribute to improve this issue either.

To put the achieved angular errors on the real outdoor and on the synthetic RGB laboratory datasets into perspective, one needs to consider that the labels of the outdoor dataset incorporate measurement errors, whereas a similar error is in principle not present in the labels of the synthetic laboratory dataset. Therefore, the achievable minimum error on synthetic test data can be in principle lower than on real-world data with measured label information. The minimum measurement error in the outdoor dataset labels is estimated to lie between $0.82°$ and $1.15°$ (Appendix A.2.2). Due to geographical limitations at the place of recording, the dataset is missing data points of light inclining from the north-eastern sky region. So far, the dataset does not contain images that were taken under cloudy or rainy weather conditions as the dataset was gathered during high summer in Central Europe. The missing weather conditions are no issue for this study, as the proposed DNN aims to reconstruct the direction to a dominant point light source. The missing weather conditions need to be recorded in a future dataset extension. Nonetheless, the outdoor dataset contains a small number of images that show the scene to be partially or fully shaded by a single cloud, a tree or other light obstacles.

So, does surface information improve the light direction prediction performance? It does, yet with a low impact. The experiments with depth information showed small effect sizes at best, making additional depth information in relation to the expected performance gain not worth the additional effort. Adding normal vector information resulted in small to medium effect sizes and less pronounced outliers. Considering the effort required to achieve normal vector information, however, their impact on the performance gain may be too small to be practically applicable. Given that these results were achieved under laboratory conditions, we would not recommend incorporating additional surface information when aiming to improve the prediction performance as, considering the complexity, their impact is expected to be even lower under real conditions.

Summary

In summary, the proposed architectural changes of the FCDN architecture were partially successful. With almost 3000 labelled images, the generated outdoor dataset constitutes a good foundation to extend the presented work. Since each image is labelled with a variety of recorded information, such as METAR data and weather data, the dataset could be useful in other scientific disciplines, too. The investigation of the influence of surface information clarified that the additional required computing power outweighs the benefits of adding additional surface information.

The real outdoor dataset is publicly available (https://sites.hm.edu/ccbv/projekte_6/projekte_detail_49024.de.html). Source code for the DNN and the data gathering application along with documentation is also available there, so the software may be used by the community to create their own datasets. Detailed information about the dataset and further details on the results can be found on the project webpage as well.

8. Future Work

In the future, we intend to gradually extend the labelled outdoor dataset provided with this study with images from different places on earth with different weather conditions to offer a large labelled dataset that can be used to teach deep neural networks how to reconstruct illumination parameters. It would be interesting to see, if the proposed architecture—with specific domain adjustment training—is able to reconstruct the dominant light direction in cloudy or rainy weather situations. Since software and documentation is publicly available, this idea could also benefit from community participation.

The prediction performances of the investigated networks are prone to unknown domains. Introducing mechanisms that enable the networks to adapt to unknown domains in some way could reduce the effort required to tailor training datasets and increase the performance on unknown data overall. Adaptation to unknown domains could be achieved by training an encoder to produce core features from synthetic and real images and having a discriminator to decide whether the produced core features originate from synthetic or real images. Once the origin of the core features is indistinguishable, they may be used in a succeeding dense section to derive illumination parameters from it.

Considering illumination reconstruction from scenery images (S), it may be interesting to investigate if a specifically trained FCDN instance may perform well or if this particular problem may be an entirely new research topic, as scenery images may incorporate a whole new range of perturbing effects, such as shadows from invisible objects like clouds. In this context, the further investigation of methods that are capable of reconstructing more complex illumination parameters to approximate these illumination conditions may be an interesting research topic.

Most illumination reconstruction approaches are compared based on how accurate they reconstruct a light position or direction. However, from what we could find, it is unclear how accurately reconstructed illumination parameters have to resemble real-world illumination situations to successfully deceive human perception. We, therefore, intend to conduct a study in the near future to quantify the noticeable deviation of illumination effects.