언어의 공기관계 분석을 위한 임의화검증의 응용

Abstract

Contingency tables are used to compare counts of n-grams to determine if the n-gram is a true collocation, meaning that the words that make up the n-gram are highlyassociated in the text.Some statistical methods for identifying collocation are used. They are Kulczinskycoecient, Ochiai coecient, Frager and McGowan coecient, Yule coecient, mutualinformation, and chi-square, and so on.But the main problem is that these measures are based on the assumption of a nor-mal or approximately normal distribution of the variables being sampled. While thisassumption is valid in most instances, it is not valid when comparing the rates ofoccurrence of rare events, and texts are composed mostly of rare events.In this paper we have simply reviewed some statistics about testing association oftwo words. Some randomization tests to evaluate the signicance level in analyzing collocation in large corpora are proposed. A related graph can be used to compare dierent test statistics that can be used to analyze the same contingency table.