An assessment system for academic speaking in Japanese as a second language (L2) was developed. The L2 English academic speaking ability is tested using high-stage tests such as Test of English as a Foreign Language (TOEFL) and International English Language Testing System (IELTS). However, no such tests are available for L2 Japanese learners. To fill this gap, the present study aims to develop assessment criteria for argumentative speech by nonnative Japanese speakers. In so doing, we developed a rating scale using the TOEFL Independent Speaking Rubrics and the IELTS Speaking Assessment Criteria, and analyzed the fit between the descriptions in the rating scale and the linguistic characteristics of the data, such as fluency, complexity, and accuracy. The results revealed that overall fluency was positively correlated with complexity and accuracy, but negatively correlated with fluency. Therefore, there is a trade-off in effects between fluency and accuracy and between fluency and complexity. Examinees who received high scores tended to be more accurate with a high degree of complexity, but their speech rate was slow. In this sense, the rating criteria were accurate for complexity and accuracy, but the fluency description may need to be modified.