Efficacy of Corpus-driven Scoring Methods for Computer-based Dictation Tests

Abstract

The present research is a follow-up research of an exploratory validation study of computer-based dictation tests utilizing TTS (text-to-speech) (Choi & Park, 2015). The study attempted to explore the efficacy of different scoring methods utilizing corpus linguistic techniques. To this end, the data collected from a prior study was employed in the research. 117 Korean EFL learners participated in this research and were divided into three groups depending on educational levels as well as their proficiency levels: 37 UNI (advanced), 40 HSU (intermediate), and 40 HSL (low). Participants' responses were scored using six different methods: GS (graduated scoring), AM (appropriate meaning-based scoring), EW (exact-word scoring), WM (wordlist match scoring), WML (wordlist match scoring including lemmas), and WMS (wordlist match scoring including synonyms). Comparative analysis of the efficacy of the scoring methods revealed that the four methods were highly correlated with each other and that corpus-based wordlist matching was found to be the most effective and robust scoring method. Among the three wordlist match scoring methods producing robust results, WM was proven to be the most effective semi-automated method with high practicality, requiring the least amount of time and energy for scoring.