Upon attaining the Japanese Ministry of Education, Culture, Science, and Technology’s (MEXT) designation of the so-called “Super Global” university status, universities must necessarily shift their focus from merely setting aggressive goals to the actual implementation of policies and practices which will move them toward meeting those lofty goals. In many cases a significant portion of the benchmarks these universities are aiming to reach involve increasing the enrollment of foreign students, the number of foreign faculty, the courses taught in English, and the frequency with which their faculty and students present at international conferences and publish in international journals. The following research illustrates how one department at a major research university in Japan has taken the initial steps in this direction by developing and distributing a brief survey exploring faculty members’ feelings of self-efficacy in English for academic purposes (EAP). By employing a Rasch-based analysis to validate the items on that survey, the department was then able to begin taking the initial steps towards meeting their Super Global goals by designing a series of workshops targeting the academic English skills identified by the faculty as most critically needed. The focus of the research presented here, however, is limited to a discussion of the challenges involved in the process of designing and validating an instrument capable of quickly measuring what faculty members believe they can and cannot do academically in English. Possible improvements for the next iteration of this instrument, which may be deployed across the entire university, are also included.