The regular practice of physical activity (PA) can reduce the chance of aggravation of the disease and lower rates of hospitalization and mortality from COVID-19, but few studies have analyzed the association of PA with the risk of infection by SARS-CoV-2. The aim of the study was to analyze the association between PA and self-reported SARS-CoV-2 infection. A longitudinal study was conducted with data from 4476 ELSA-Brasil participants who had their PA analyzed twice, once in 2016–2018 and again in 2020. PA was identified using the IPAQ at both follow-up moments and categorized into four groups: (a) remained physically inactive (reference); (b) remained physically active; (c) became physically active in the second moment; and (d) became physically inactive in the second moment. The variables of age, sex, obesity, hypertension, diabetes and specific protective practices against COVID-19 were tested as possible confounders. Data were analyzed by logistic regression. A 95% confidence interval (CI) was used. Remaining physically active was associated with a 43% reduction in the risk of SARS-CoV-2 infection only among those who used specific practices to protect against COVID-19, OR = 0.57 and CI = 0.32-0.99. The results suggested that regular practice of PA can reduce the risk of SARS-CoV-2 infection, especially among those who have used specific practices to protect against COVID-19 during the pandemic.