The COVID-19 pandemic has led to a reorganization of health systems to prioritize the fight against the virus. The adoption of social distancing interfered with the flow of existing policies, and may thus negatively affect the most vulnerable groups, such as the rare disease community. Aimming at characterizing the perception of the impact of COVID-19 on the health care of the Brazilian rare disease community, an online questionnaire addressed to patients with rare diseases and their caregivers was disseminated in the Brazilian territory between June 1st to July 5th, 2020. The questions dealt with the sanitary measures adopted; access to medical services; and mental suffering during the pandemic. The survey was answered by 1,466 participants (<18 yo = 53.3%) representing 192 rare diseases. Regarding physical distancing, 1,372 (93.6%) participants did not leave their residence, or did so only when essential; 1,321 (90.1%) always wore masks when leaving home. 1,042 (71.1%) and 995 (67.9%) participants, respectively, referred medical genetics appointments and rehabilitation therapies were postponed/canceled. Telemedicine was experienced by 1,026 (70%), and 68.3% agreed this is a good strategy for health care. Patients with Inborn Errors of Metabolism (IEM, n = 624, 42.5%) appear to have more access to information and ability to overcome difficulties, and feel less threatened, lonely and depressed than the non-IEM group (p < .05). There was an increment of the rare disease patients' vulnerability in the pandemic scenario. The cooperation of patients/caregivers along with adaptation of the health system is crucial and may be so even post-pandemic.