English education in Japan has been changing recently. While second-language teachers aim to help their students increase their English communication skills, Japanese English as a foreign language (EFL) learners tend to recognize their English skill as poor, especially in listening and speaking, which are important for English communication. This article reviewed why Japanese learning English tend to believe that they are not good at communicating in English. Working memory, i.e., verbal working memory, is important in listening to and understanding spoken English. However, the mora, which is a rhythmic unit in the Japanese language creates bigger cognitive loads on verbal working memory, which is the memory component involved in temporary information maintenance and processing. This is one reason that Japanese EFL learners do not have good listening ability in English. Another factor is that English speakers must pass through three steps; i.e., conceptualize, formulate, and articulate. However, these steps cause bigger cognitive loads on the verbal working memory in speaking English than in speaking the native language. Moreover, Japanese students feel more anxious when speaking the second language compared with reading, listening, and writing in the second language. Thus, the cognitive load may be one reason that Japanese EFL learners hesitate to speak English. However, we must consider these features of Japanese students in English listening and speaking to ensure that English education is more effective and improve Japanese English communication skills in Japan.