In this paper, a knowledge-based algorithm is proposed for automatically generating three-dimensional (3D) building models in dense urban areas by using airborne light detection and ranging (LiDAR) data and aerial images. Automatic 3D building modeling using LiDAR is challenging in dense urban areas, in which houses are typically located close to each other and their heights are similar. This makes it difficult to separate point clouds into individual buildings. A combination of airborne LiDAR and aerial images can be an effective approach to resolve this issue. Information about individual building boundaries, derived by segmentation of images, can be utilized for modeling. However, shadows cast by adjacent buildings cause segmentation errors. The algorithm proposed in this paper uses an improved segmentation algorithm (Susaki, J. 2012.) that functions even for shadowed buildings. In addition, the proposed algorithm uses assumptions about the geometry of building arrangement to calculate normal vectors to candidate roof segments. By considering the segmented regions and the normals, models of four common roof types—gable-roof, hip-roof, flat-roof, and slant-roof buildings—are generated. The proposed algorithm was applied to two areas of Higashiyama ward, Kyoto, Japan, and the modeling was successful even in dense urban areas.