Structural imaging studies suggest gender differences in brain volumes; however, whether hormone treatment (HT) can protect against age-related structural changes remains unknown, and no prior neuroimaging study has investigated potential interactions between HT and estrogen receptor (ESR) polymorphisms. Magnetic resonance imaging was used to measure gray and white matter, hippocampal volume, corpus callosum, cerebrospinal fluid (CSF), total intracranial volume (ICV) and white matter lesions (WML) in 582 non-demented older adults. In multivariable analysis, when compared to women who had never used HT, men and women currently on treatment, but not past users, had significantly smaller ratios of gray matter to ICV and increased atrophy (CSF/ICV ratio). Hippocampal and white matter volume as well as the corpus callosum area were not significantly different across groups. ESR2 variants were not significantly associated with brain measures, but women with the ESR1 rs2234693 C allele had significantly smaller WML. Furthermore this association was modified by HT use. Our results do not support a beneficial effect of HT on brain volumes in older women, but suggest the potential involvement of ESR1 in WML.