Fat and lean body mass have important implications for health and physical functioning in older age, and physical activity is purported to be an important modifiable determinant. However, our evidence-based understanding of its role is limited. We examined the associations of physical activity, assessed both by self-report (using data on leisure time physical activity (LTPA) collected on 4 occasions over a 28-year period) and objectively (using 5-day heart rate and movement monitoring), with fat and lean mass at ages 60–64 years in 1,162 British participants from the Medical Research Council National Survey of Health and Development in 1946–2010. Higher objectively assessed physical activity energy expenditure (PAEE) at ages 60–64 years was associated with lower fat mass and android (abdominal):gynoid (hip) fat ratio (mean differences in fat mass per 1–standard deviation increase in PAEE were −0.79 kg/m1.2 in men (95% confidence interval: −1.08, −0.50) and −1.79 kg/m1.2 (95% confidence interval: −2.15, −1.42) in women). After adjustment for fat mass, higher PAEE was associated with higher appendicular lean mass. Both light and moderate-to-vigorous intensities of activity were associated with fat mass, and the latter was associated with lean mass. More frequent LTPA across adulthood was associated with lower fat mass (in women only) and higher appendicular lean mass (in both sexes, after adjustment for fat mass). These results support the promotion of LTPA across adulthood, as well as both light and moderate-to-vigorous intensities of activity among older adults.