Background Work from home (WFH) can impact workers´ sedentary behaviors and levels of physical activity. The aim of this study was to estimate the association between WFH and workers´ sedentary behaviors, leisure-time and domestic physical activities during the COVID-19 pandemic and verify whether age and sex may act as effect modifiers. Methods We conducted a cross-sectional study of 2544 participants in the supplementary study on COVID-19 in the Longitudinal Study of Adult Health (ELSA-Brasil) from July 2020 to February 2021. We assessed screen time (≤ 8 h/day versus > 8 h/day), accumulated sitting time (≤ 8 h/day versus > 8 h/day) as sedentary behaviors on a typical day, and leisure-time (active versus inactive, according to World Health Organization recommendations) and domestic (low versus high, according to median) physical activity, using the International Physical Activity Questionnaire (IPAQ), before and during social distancing. Logistic regression models were used. Results Participants that were working from home during social distancing showed increased odds of screen time and sitting time greater than 8 h/day (OR = 3.12; 95%CI: 2.32–4.20 and OR = 2.68; 95%CI: 2.02–3.56, respectively) and higher odds of high domestic physical activity (OR = 1.29; 95%CI: 0.99–1.67) when compared to those not working from home. There was no association between WFH and leisure-time physical activity (OR = 0.99, 95%CI: 0.75,1.31). Age was an effect modifier in the association between WFH and leisure-time physical activity and domestic activity. Older people working from home showed higher odds of physical inactivity (OR = 1.84, 95%CI: 1.07,3.16) and high domestic physical activity (OR = 1.92, 95%CI: 1.12,3.27) compared to older people not working from home. Conclusion WFH was associated with sedentary behavior > 8 h/day and high domestic physical activity. In the older people, WFH was associated with physical inactivity and high domestic physical activity. As sedentary behavior and physical inactivity are consistently negatively associated with health, it is important to discuss policies to manage WFH that allow pauses from physical activities and performance of hours of work within preestablished limits to reduce sedentary behavior. In addition, individuals working from home, especially the older people, should be encouraged to engage in leisure-time physical activity as a form of health promotion.