Measuring the Degree of Convergence between Informal and Formal Acceptability Judgments for Korean

Abstract

Song, Sanghoun, Lee Sang-Geun , Choe Jae-Woong & Oh Eunjeong. 2017. Measuring the Degree of Convergence between Informal and Formal Acceptability Judgments for Korean. Korean Journal of Linguistics, 42-4, 761-804. The present study attempts to quantify the extent to which the acceptability judgments of naïve native speakers coincide with those of linguists in the domain of Korean syntax. We excerpted 1,554 pairs from articles published in a theoretical syntax journal on Korean, Studies in Generative Grammar, from 1991 to 2014, and then randomly chose 287 pairs. Employing PsychoPy as the toolkit and a five-point Likert scale as the task, we collected 68,388 data points from 328 naïve participants. Two statistical analyses, a one-tailed T-test and a κ-test, were then run on the Z-score transformed data points. The T-test, which determines the directionality of difference, yielded a convergence rate of 88.15% between the two types of judgments. The κ-test, which estimates both the size and directionality of difference, obtained a convergence rate of 82.93%. 16 pairs that passed the T-test in a measure of directionality failed to pass the κ-test when testing both directionality and size of difference by degree of match/mismatch between the two types of judgments. Such consideration leads us to believe that the κ-test is more appropriate for detecting the size of the difference between the two types of judgments, as well as the directionality of the difference. We also classified convergence results into 4 patterns: (i) convergence for both sentences in the pair, (ii) divergence for only the bad sentences in the pair, (iii) divergence for only the good sentences in the pair, and (iv) divergence for both sentences in the pair. These patterns play a role in interpreting different convergence rates obtained in the study. We discuss the implications of the κ-test convergence rate for our research question and the use of κ-tests and patterns in experimental syntax studies. (Incheon National University, Korea University, Sangmyung University)