Background: To examine the relationship between reduced heart rate variability (HRV) and cognitive function in middle-aged adults in the general population. Methods: HRV, in both time and frequency domains, and cognitive functioning were measured twice in 5,375 male and female participants of the UK Whitehall II study (mean ages = 55 and 61 years, respectively). Logistic regression was used to model associations between HRV and cognition [short-term verbal memory, reasoning (Alice Heim 4-I), vocabulary, phonemic and semantic fluency]. Cross-sectional associations were assessed at both waves, and longitudinal associations were measured as changes in cognition over the 5-year follow-up. Results: No consistent associations were found in men or women, either in the cross-sectional, prospective or the longitudinal analyses of declines in cognition. Conclusion: Reduced cardiovascular autonomic function does not contribute to cognitive impairment in this middle-aged population. Further studies are needed to verify the potential role of HRV measures in predicting the degeneration of cognitive function at older ages.