In order to confirm the factors influencing the consistency of the effect of valence and arousal of emotional stimuli on memory performance, the author conducted a recognition memory test using auditory stimuli, which were considered rarely studied. The experiments were conducted on 31 participants, and the sounds employed in the tests were collected from the newly developed affective digitized sounds database (IADS-E). The results indicated that positive and high awareness sounds accrued the highest scores on recognition memory tasks. These sounds also demonstrated a higher rating when employed to accompany episodes. By comparing the results of this investigation with those obtained from previous studies, we find that the effect of arousal is relatively stable and is only minimally affected by the type of memory task (free recall/recognition) or the stimulus modalities (visual/auditory). Further, we insist that dividing the condition of stimuli appropriately may also affect the effect of valence and arousal on memory performance. Moreover, in accordance with the Yakies–Dodson law, experiments that create a more viable control for the values of stimuli, the promoting effect of positive stimuli and the inhibitory effect of negative stimuli are deemed suspect.