Background: A reduced heart rate variability (HRV) is associated with worse prognosis, increased incidence of cardiovascular disease (CVD) and mortality. There are conflicting results and a lack of population-based data regarding the association of HRV with CVD risk factors and its potential role as independent cause or mediator of CVD risk. Methods: Cross-sectional data of a population-based cohort including 1,779 women and men aged 45–83 years were used to analyse associations of time and frequency domain measures of HRV (derived from 5-min ECG segments) with age, behavioural and biomedical risk factors and disease in the whole sample and in a “healthy” subgroup. Results: Age was inversely associated with all measures of HRV (mean standard deviation of normal intervals across 10-year age-groups 32.1, 26.9, 27.1 and 24.8 ms in women, 29.3, 25.9, 23.8 and 25.7 ms in men). There was no association of physical activity, current smoking or alcohol with HRV. In age-adjusted models, triglycerides, glucose, waist-to-hip ratio and diabetes were inversely associated with HRV in men and women, and low/high density cholesterol and hypertension in men only (up to 43% difference across risk factor quartiles). Multivariable adjustment and restriction to the “healthy” subgroup attenuated the associations. Conclusions: We found only weak and inconsistent associations of HRV with cardiovascular risk factors. However, these results as well as those from previous studies are still compatible with the hypothesis that short-term HRV may be a marker of ill health or a mediator of the effect of selected biomedical risk factors on CVD.