교육적 활용을 위한 자동 영어 발음 평가 시스템의 점수 신뢰도 평가Evaluating Score Reliability of Automatic English Pronunciation Assessment System for Education
- Other Titles
- Evaluating Score Reliability of Automatic English Pronunciation Assessment System for Education
- Authors
- 홍연정; 남호성
- Issue Date
- 2021
- Publisher
- 한국외국어대학교 외국어교육연구소
- Keywords
- Automatic pronunciation assessment; CAPT; CAPT; Inter-rater agreement; Machine score reliability; 기계 점수 신뢰도; 자동 발음 평가; 평가자간 일치
- Citation
- 외국어교육연구, v.35, no.1, pp.91 - 104
- Indexed
- KCI
- Journal Title
- 외국어교육연구
- Volume
- 35
- Number
- 1
- Start Page
- 91
- End Page
- 104
- URI
- https://scholar.korea.ac.kr/handle/2021.sw.korea/137952
- DOI
- 10.16933/sfle.2021.35.1.91
- ISSN
- 1225-4975
- Abstract
- The purpose of this study was to evaluate the pronunciation score reliability of an automatic pronunciation assessment system for education, SpeechPro, a commercially released and patented system but without a score reliability test. So it is necessary to ensure the commercial system’s reliability. The method is to measure score agreement between SpeechPro and human raters. The database used is a paid English speech corpus of the native speakers and non-native speakers with score annotations of the three English raters. First, the inter-rater agreement was measured, and then the agreement between SpeechPro’s scores and the raters’ average scores were measured. The following 5 metrics were used: Pearson correlation coefficient, standardized mean difference, quadratic weighted kappa, exact percentage agreement, and 1-point adjacent percentage agreement. The results are that human-machine agreement is significantly identical to human-human agreement according to all the metrics used, proving the score reliability of SpeechPro. This provides a logical justification required before the comparison with other automatic pronunciation assessment systems.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - College of Liberal Arts > Department of English Language and Literature > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.