교육적 활용을 위한 자동 영어 발음 평가 시스템의 점수 신뢰도 평가

Abstract

The purpose of this study was to evaluate the pronunciation score reliability of an automatic pronunciation assessment system for education, SpeechPro, a commercially released and patented system but without a score reliability test. So it is necessary to ensure the commercial system’s reliability. The method is to measure score agreement between SpeechPro and human raters. The database used is a paid English speech corpus of the native speakers and non-native speakers with score annotations of the three English raters. First, the inter-rater agreement was measured, and then the agreement between SpeechPro’s scores and the raters’ average scores were measured. The following 5 metrics were used: Pearson correlation coefficient, standardized mean difference, quadratic weighted kappa, exact percentage agreement, and 1-point adjacent percentage agreement. The results are that human-machine agreement is significantly identical to human-human agreement according to all the metrics used, proving the score reliability of SpeechPro. This provides a logical justification required before the comparison with other automatic pronunciation assessment systems.