An information-theoretic distance for k-NN classification: Enhancing mixed-type data analysis with Jensen-Shannon divergence

초록

Distance-based classifiers such as k-nearest neighbors (KNN) often struggle with categorical variables due to the absence of a meaningful distance metric. Simple overlap or one-hot encodings are commonly used but are insufficient. They fail to capture differences in the underlying class-conditional distributions, especially for attributes with many categories. We address this problem by proposing an information-theoretic mixed-type dis tance for KNN classification: the square root of the Jensen-Shannon divergence (JSD) for categorical attributes is combined with Euclidean distance on normalized numeric attributes, and each feature is further scaled by its mutual information (MI) with the class label. This construction supplies a principled distance for categori cal levels, aligns heterogeneous scales, and enhances discriminative power. Across 27 benchmark datasets, our JSD-based method achieved the highest mean accuracy of 0.8433, outperforming standard baselines (Dummy and numeric-only KNN) and established heterogeneous metrics (HEOM and HVDM). These results demonstrate that an information-theoretic, distribution-aware distance provides an effective and interpretable solution to mixed-type KNN classification.

키워드