Gonadal hormones may influence cognitive function. Postmenopausal midlife women in the population-based Melbourne Women's Midlife Health Project cohort were administered a comprehensive battery of neuropsychological tests on two occasions 2 years apart. Participants (n = 148, mean age 60 years) had undergone natural menopause and were not using hormone therapy. Estrone, total and free estradiol, and total and free testosterone levels were measured at time of the first testing. Principal-component analysis identified four cognitive factors. In multiple linear regression analyses, better semantic memory performance was associated with higher total (p = 0.02) and free (p = 0.03) estradiol levels and a lower ratio of testosterone to estradiol (p = 0.007). There were trends for associations between better verbal episodic memory and lower total testosterone (p = 0.08) and lower testosterone/estradiol ratio (p = 0.06). Lower free testosterone levels were associated with greater 2-year improvement on verbal episodic memory (p = 0.04); higher testosterone/estradiol predicted greater semantic memory improvement (p = 0.03). In postmenopausal midlife women, endogenous estradiol and testosterone levels and the testosterone/estradiol ratio are associated with semantic memory and verbal episodic memory abilities.