Soil moisture is an important factor for vegetation restoration and ecosystem sustainability in the Loess Plateau of China. The strong spatial heterogeneity of soil moisture is controlled by many environmental factors, including topography and land use. Moreover, the spatial patterns and soil hydrological processes depend on the scale of the site being investigated, which creates a challenge for soil moisture forecasts. This study was conducted at two scales: watershed and small watershed. The goal of the study was to investigate the spatial variability in soil moisture and the scale effect of its controlling factors, as well as to provide references for soil moisture forecasting and studies of scale transformation. We took samples at 76 sites in the Ansai watershed and at 34 sites in a typical small watershed within the Ansai watershed in August. Next, we measured the soil moisture in five equal layers from a depth of 0–100 cm and recorded the land use type, location on the hill slope, slope, aspect, elevation and vegetation cover at the sampling sites. The results indicated that soil moisture was negatively correlated with relative elevation, slope and vegetation cover. As depth increased, the correlations among slope, aspect and soil moisture increased. At the small watershed and watershed scales, the soil moisture was highest in cultivated land, followed by wild grassland and lowest in garden plots, woodland and shrubland. The soil moisture was distributed similarly with respect to the location on the hill slope at both scales: upper slope < middle-upper slope < middle slope < middle-lower slope < lower slope. The deep layer soil moisture value of the slope top was high, being close to the soil moisture in the lower slope. Therefore, wild grassland or low-density woodland should be prioritized for farmland recovery in the Ansai watershed, and the locations on the hill slope, slope and elevation should be combined to configure different mosaic patterns. For example, low-density woodland or wild grassland would be appropriate for sites with low soil moisture content, such as upper slope, high elevation and steep slope sites. A stepwise regression analysis indicated that the dominant factor controlling the spatial variability of soil moisture values varied at different scales. At the small watershed scale, the order of significance for the influence of environmental factors on soil moisture values was as follows: land use type, slope, relative elevation and vegetation cover. The order of significance at the watershed scale was also determined: location on the hill slope, vegetation cover, slope, relative elevation and sine of the aspect. This result indicated that the influence of different environmental factors on soil moisture variability was dependent on the scale. The forecasting capability of regression models for soil moisture decreases from the small watershed scale to the watershed scale. This study could provide a reference for relevant scale transformation studies and offer guidance for water resource management and vegetation restoration approaches on the Loess Plateau.