Recognizing a musical excerpt without necessarily retrieving its title typically reflects the existence of a memory system dedicated to the retrieval of musical knowledge. The functional distinction between musical and verbal semantic memory has seldom been investigated. In this fMRI study, we directly compared the musical and verbal memory of 20 nonmusicians, using a congruence task involving automatic semantic retrieval and a familiarity task requiring more thorough semantic retrieval. In the former, participants had to access their semantic store to retrieve musical or verbal representations of melodies or expressions they heard, in order to decide whether these were then given the right ending or not. In the latter, they had to judge the level of familiarity of musical excerpts and expressions. Both tasks revealed activation of the left inferior frontal and posterior middle temporal cortices, suggesting that executive and selection processes are common to both verbal and musical retrievals. Distinct patterns of activation were observed within the left temporal cortex, with musical material mainly activating the superior temporal gyrus and verbal material the middle and inferior gyri. This cortical organization of musical and verbal semantic representations could explain clinical dissociations featuring selective disturbances for musical or verbal material.