A new breast cancer detection algorithm, named the “Gabor Cancer Detection” (GCD) algorithm, utilizing Gabor features is proposed. Three major steps are involved in the GCD algorithm, preprocessing, segmentation (generating alarm segments), and classification (reducing false alarms). In preprocessing, a digital mammogram is down-sampled, quantized, denoised and enhanced. Nonlinear diffusion is used for noise suppression. In segmentation, a band-pass filter is formed by rotating a 1-D Gaussian filter (off center) in frequency space, termed as “Circular Gaussian Filter” (CGF). A CGF can be uniquely characterized by specifying a central frequency and a frequency band. A mass or calcification is a space-occupying lesion and usually appears as a bright region on a mammogram. The alarm segments (suspicious to be masses/calcifications) can be extracted out using a threshold that is adaptively decided upon the histogram analysis of the CGF-filtered mammogram. In classification, a Gabor filter bank is formed with five bands by four orientations (horizontal, vertical, 45 and 135 degree) in Fourier frequency domain. For each mammographic image, twenty Gabor-filtered images are produced. A set of edge histogram descriptors (EHD) are then extracted from 20 Gabor images for classification. An EHD signature is computed with four orientations of Gabor images along each band and five EHD signatures are then joined together to form an EHD feature vector of 20 dimensions. With the EHD features, the fuzzy C-means clustering technique and k-nearest neighbor (KNN) classifier are used to reduce the number of false alarms. The experimental results tested on the DDSM database (University of South Florida) show the promises of GCD algorithm in breast cancer detection, which achieved TP (true positive rate) = 90% at FPI (false positives per image) = 1.21 in mass detection; and TP = 93% at FPI = 1.19 in calcification detection.