This study discusses the validity and reliability of the evaluation criteria conducted as part of preparing the analytic scoring evaluation criteria for research papers on learners of Japanese as a second language. With the publication of the “National Standard for Education of Undergraduate Specialty in Colleges and Universities” (a new national standard) and “Bachelor thesis (design) lottery inspection method (trial) ” by the Chinese Ministry of Education, the importance of writing has been recognized more than before. However, there are still no writing evaluation standards that can be shared in Japanese language education. Using the Delphi method , the author conducted a research study, on a collection of evaluation items constructed based on the abilities required by the new national standard Subsequently, the author created a draft of the analytic scoring evaluation criteria for Japanese learners’ research papers based on the survey results. This study comprises conducting the research, using the draft, examining the validity and reliability of the draft for the evaluation criteria, and revising the draft for the evaluation criteria based on the results. The revised draft of the evaluation criteria was named the “Japanese research paper evaluation criteria”. From the results of the factor analysis and Cronbach’s alpha, “Japanese research research treatise evaluation criteria” was proven to be a high reliability and validity scale.