Background: This study aimed to verify the association between Sedentary Behavior (SB) and performance on cognitive function tests in middle-aged and elderly adults. Methods: This cross-sectional study included 6505 participants (55.2% women) of ELSA-Brasil, with a median age of 61 years. The different types of SB considered were sitting time and screen time. The scores obtained in the memory, language, and executive function tests were used to assess cognitive performance (CP). The association between SB and CP was assessed using linear regression. Results: For men, sitting time was associated with better performance in memory, language, and executive function tests. Screen time on the weekend, showed a favorable association with performance in the executive function test. Occupational screen time on weekdays was positively associated with language test performance. For women, sitting time and occupational screen time were positively associated with performance on memory tests. SB was favorably associated with performance in language tests and executive function tests. Conclusions: SB seems to favor CP in this population without evident dementia and with a high level of education. The type of SB (mentally active or passive) and the schooling seem to be of particular interest for cognitive performance.