Automated Evaluation and Feedback of Argumentative Essays Using LLM and Argument Analysis Tool

초록

This study proposes a novel Automated Writing Evaluation (AWE) system that integrates a fine-tuned DeBERTa model and ChatGPT within the LangGraph framework to enhance the evaluation of argumentative essays. Traditional AWE systems have primarily focused on surface-level features such as grammar and vocabulary, while demonstrating limitations in evaluating the logical structure and depth of argumentative elements. To address these shortcomings, the proposed system leverages DeBERTa for high-accuracy detection of argumentative elements and employs ChatGPT’s flexible learning capabilities for essay classification and feedback generation. The system’s performance was validated through three experiments. The fine-tuned DeBERTa model achieved superior performance in detecting argumentative elements, recording an F1 score of 0.74, while ChatGPT demonstrated strong adaptability in essay type classification, showing higher accuracy in few-shot learning environments compared to zero-shot settings. Furthermore, the feedback generated by the system significantly improved students’ use of argumentative elements, with notable enhancements observed in challenging components such as Counterclaims (+28%) and Rebuttals (+30%). The LangGraph-based approach efficiently manages the dynamic flow of tasks, offering scalability and adaptability across diverse educational contexts. This system makes a substantial contribution to the analysis of argumentative structures and the generation of personalized feedback, highlighting its potential for expansion to various writing genres in the future.

키워드