CBEN—A Multimodal Machine Learning Dataset for Cloud-Robust Remote Sensing Image Understanding

Clouds frequently degrade optical satellite imagery, limiting the reliability of remote sensing models. However, in the literature, cloud-free analyses are often performed by excluding cloudy images from machine learning datasets and methods. This restricts their usefulness in time-critical scenarios such as disaster response, where waiting for cloud-free imagery is impractical. Cloud removal can mitigate this issue, but methods remain imperfect and may introduce visual artifacts. Therefore, it is desirable to develop cloud-robust methods by combining optical imagery with radar data, a modality unaffected by clouds. While datasets for machine learning combine optical and radar data, most researchers exclude cloudy images from training and evaluation. We identify this exclusion as a limitation that reduces applicability to cloudy scenarios and address it by introducing CloudyBigEarthNet (CBEN), a dataset of paired optical and radar images containing cloud occlusions for land-use and land-cover classification. Using average precision (AP), we show that state-of-the-art methods trained on clear-sky optical and radar data suffer performance drops of between 23.8 and 33.4 AP points when tested on cloudy imagery. We adapt these methods using cloudy images during training and improve AP on cloudy test cases by 17.2 to 28.7 AP points. Code and dataset have been published.