Recent research has suggested that only linguistically relevant sources of variability, or those which affect the perception of spoken words, are retained in long-term memory. The present study sought to determine if the linguistic relevance of surface characteristics such as speaking rate, overall amplitude, and vocal effort would differentially affect memory retention. For each source of variability, a continuous recognition memory task was used in which listeners were presented with a list of spoken words and asked to judge whether each word in the list was ``old'' (had occurred previously on the list) or ``new.'' Results showed that listeners were better able to identify a word as ``old'' if the word was repeated at the same speaking rate (condition 1), overall amplitude (condition 2), or vocal effort (condition 3), suggesting that the individual surface characteristics which comprise each type of variability were encoded into memory. However, speaking rate and vocal effort produced a greater effect on recognition memory than overall amplitude, indicating that all sources of variability may not be processed and encoded to the same extent. Rather, memory for surface characteristics may be related to the amount of linguistically relevant information each source of variability contains.