Post-lingual deafness induces a decline in the ability to process phonological sounds or evoke phonological representations. This decline is paralleled with abnormally high neural activity in the right posterior superior temporal gyrus/supramarginal gyrus (PSTG/SMG). As this neural plasticity negatively relates to cochlear implantation (CI) success, it appears important to understand its determinants. We addressed the neuro-functional mechanisms underlying this maladaptive phenomenon using behavioral and functional magnetic resonance imaging (fMRI) data acquired in 10 normal-hearing subjects and 10 post-lingual deaf candidates for CI. We compared two memory tasks where subjects had to evoke phonological (speech) and environmental sound representations from visually presented items. We observed dissociations in the dynamics of right versus left PSTG/SMG neural responses as a function of duration of deafness. Responses in the left PSTG/SMG to phonological processing and responses in the right PSTG/SMG to environmental sound imagery both declined. However, abnormally high neural activity was observed in response to phonological visual items in the right PSTG/SMG, i.e., contralateral to the zone where phonological activity decreased. In contrast, no such responses (overactivation) were observed in the left PSTG/SMG in response to environmental sounds. This asymmetry in functional adaptation to deafness suggests that maladaptive reorganization of the right PSTG/SMG region is not due to balanced hemispheric interaction, but to a specific take-over of the right PSTG/SMG region by phonological processing, presumably because speech remains behaviorally more relevant to communication than the processing of environmental sounds. These results demonstrate that cognitive long-term alteration of auditory processing shapes functional cerebral reorganization.