Background: Disadvantaged populations, especially those living in rural and informal settlements, constitute the most affected during the COVID-19 pandemic. However, there is limited information on the health-related quality of life (HRQOL) and indirect consequences of non-pharmaceutical interventions implemented during the pandemic. Methods: We leveraged on an ongoing prospective open-cohort survey and performed a cross-sectional analysis of data collected between November 2021 and July 2022 among 793 residents above age 5 in a large slum community in the city of Salvador, Brazil. We describe the HRQOL of residents, explored participants’ perception about the pandemic, and the relationship between sociodemographic, economic and employment data on physical and mental health scores using both χ2 statistics and separate mixed-effects regression models. Results: Most participants were female (58.9%), aged 18–45 years (45%), with nearly half (49.7%) employed before pandemic, of whom 38.8% lost jobs during the pandemic. Food insecurity was 69.6%, and only 27.1% received government aid. Those retaining employment during the pandemic had better physical (β: 4.02, 95% CI 1.34 to 6.69, p<0.001) and mental (β: 3.08, 95% CI 0.60 to 5.56, p<0.001) health. Females had lower physical health scores than males (β: −2.44, 95% CI −3.94 to −0.94, p=0.002). Older participants had lower physical health scores (β: −9.11, 95% CI −12.14 to −6.07, p<0.001), but higher schooling improved physical health (p<0.001). Females and older adults faced more COVID-impacted challenges (p<0.001) related to health, education, family, social relationships, work, finances and employment. Conclusion: We found lower HRQOL among adults, females, the unemployed and those with lower school attainment. In addition, women and individuals in older age groups reported experiencing COVID-impacted mental challenges more frequently than others. These findings highlight the need to prioritise creation of economic opportunities and expansion of existing assistance programmes for marginalised populations residing in these slums.