Predicting the Difficulty of Korean College Scholastic Ability Test Reading Comprehension Questions Using LLM-Based Qualitative Features

초록

Accurate estimation of item difficulty is essential to ensure fairness, reliability, and discriminative power in high-stakes standardized tests. However, existing approaches rely primarily on surface-level linguistic features or implicit representations learned by neural models, which may not explicitly capture the cognitive and reasoning processes underlying perceived difficulty. In this study, we propose a feature-based framework for predicting the difficulty of Korean College Scholastic Ability Test (KCSAT) reading comprehension items by integrating quantitative item features ( F-1 ) and large language model (LLM)-based qualitative features ( F-2 ). Although F-1 captures traditional linguistic and structural properties, F-2 is designed to quantify deeper aspects of item difficulty, including abstraction, logical complexity, cognitive load, and distractor effectiveness, using an LLM. Extensive experiments were conducted on KCSAT reading data collected from 2016 to 2024 under both time-based and stratified evaluation settings. The results indicate that feature-only models using F-1+F-2 achieve a performance comparable to, and in some cases, exceeding, that of models that utilize contextual textual embeddings. In the classification task, the proposed approach achieved a Macro F1-score of 0.6215 under a time-based split, highlighting the importance of temporal structure in difficulty prediction. Ablation studies further demonstrated that LLM-based features provide complementary information beyond traditional features, with different components contributing differently depending on the task. These findings indicate that structured feature representations, particularly those incorporating LLM-derived qualitative factors, provide an effective and interpretable approach for modeling item difficulty in high-stakes reading assessments.

키워드