Hypertension, particularly in middle age, has been associated with worse cognitive function, but evidence is inconclusive. This study investigated whether hypertension, prehypertension, age, and duration of diagnosis, as well as blood pressure control, are associated with a decline in cognitive performance in ELSA-Brasil participants. This longitudinal study included 7063 participants, mean age 58.9 years at baseline (2008–2010), who attended visit 2 (2012–2014). Cognitive performance was measured in both visits and evaluated by the standardized scores of the memory, verbal fluency, trail B tests, and global cognitive score. The associations were investigated using linear mixed models. Hypertension and prehypertension at baseline were associated with decline in global cognitive score; being hypertension associated with reduction in memory test; and prehypertension with reduction in fluency test. Hypertension diagnose ≥55 years was associated with lower global cognitive and memory test scores, and hypertension diagnose <55 years with lower memory test scores. Duration of hypertension diagnoses was not associated with any marker of cognitive function decline. Among treated individuals, blood pressure control at baseline was inversely associated with the decline in both global cognitive and memory test scores. In this relatively young cohort, hypertension, prehypertension, and blood pressure control were independent predictors of cognitive decline in distinct abilities. Our findings suggest that both lower and older age of hypertension, but not duration of diagnosis, were associated with cognitive decline in different abilities. In addition to hypertension, prehypertension and pressure control might be critical for the preservation of cognitive function.