Bone mass is a major determinant of fracture, but there have been few comprehensive studies of the correlates of bone mineral density (BMD) in older men. The objective of the current cross-sectional analysis was to determine the factors associated with BMD of the lumbar spine and proximal femur in a large population-based sample of older men enrolled in The Osteoporotic Fractures in Men Study, Mr.OS. We enrolled 5,995 men 65 years of age or older, 89% Caucasian, in Mr.OS at six US clinical centers. Demographic, medical and family history and lifestyle information was obtained by interview and physical function and anthropometric data by examination. Spine and hip BMD was measured using dual-energy X-ray absorptimetry. The multivariable linear regression models predicted 19 and 10% of the overall variance in BMD of the femoral neck and spine, respectively. African-American men had 6 to 11% higher BMD than Caucasian men independent of multiple factors. Hip BMD declined with advancing age, while spine BMD increased. Body weight (per 10 kg) and self report of diabetes were each associated with 2 to 4% higher BMD, while history of a non-trauma fracture and current use of selective serotonin reuptake inhibitors, but not other antidepressants, were associated with at least 4% lower BMD. Both maternal and paternal histories of fracture were associated with 1.4–1.7% lower BMD. Osteoarthritis, physical activity, grip strength, alcohol intake, and dietary calcium were positively related to BMD, while a history of chronic lung disease, prostate cancer, and kidney stones was associated with lower BMD. Smoking, caffeine intake, and thiazide diuretics were not related to BMD in older men. A number of lifestyle and behavioral characteristics and medical conditions were associated with BMD in older men. Identification of these correlates could improve methods to identify men at risk for fracture and improve our understanding of fracture etiology.