Although it has been hypothesized that the association of physical activity with depressive and anxiety symptoms is bidirectional, few studies have examined this issue in a prospective setting. We studied this bidirectional association using data on physical activity and symptoms of anxiety and depression at three points in time over 8 years. A total of 9,309 participants of the British Whitehall II prospective cohort study provided data on physical activity, anxiety and depression symptoms and 10 covariates at baseline in 1985. We analysed the associations of physical activity with anxiety and/or depression symptoms using multinomial logistic regression (with anxiety and depression symptoms as dependent variables) and binary logistic regression (with physical activity as the dependent variable). There was a cross-sectional inverse association between physical activity and anxiety and/or depressive symptoms at baseline (ORs between 0.63 and 0.72). In cumulative analyses, regular physical activity across all three data waves, but not irregular physical activity, was associated with reduced likelihood of depressive symptoms at follow-up (OR = 0.71, 95 % CI 0.54, 0.99). In a converse analysis, participants with anxiety and depression symptoms at baseline had higher odds of not meeting the recommended levels of physical activity at follow-up (OR = 1.79, 95 % CI 1.17, 2.74). This was also the case in individuals with anxiety and/or depression symptoms at both baseline and follow-up (OR = 1.70, 95 % CI 1.10, 2.63). The association between physical activity and symptoms of anxiety and/or depression appears to be bidirectional.