Lifelong Language Learning With the Most Forgotten Knowledge
- Authors
- Choi, Heejeong; Kang, Pilsung
- Issue Date
- 2021
- Publisher
- IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
- Keywords
- Task analysis; Training; Natural language processing; Neural networks; Data models; Knowledge discovery; Measurement; Lifelong language learning; natural language processing; catastrophic forgetting; a stream of text data; generative replay
- Citation
- IEEE ACCESS, v.9, pp.57941 - 57948
- Indexed
- SCIE
SCOPUS
- Journal Title
- IEEE ACCESS
- Volume
- 9
- Start Page
- 57941
- End Page
- 57948
- URI
- https://scholar.korea.ac.kr/handle/2021.sw.korea/130179
- DOI
- 10.1109/ACCESS.2021.3071787
- ISSN
- 2169-3536
- Abstract
- Lifelong language learning enables a language model to accumulate knowledge through training on a stream of text data. Recent research on lifelong language learning is based on samples of previous tasks from an episodic memory or generative model. LAMOL, a representative generative model-based lifelong language learning model, preserves the previous information with the generated pseudo-old samples, which are suboptimal. In this paper, we propose an improved version of LAMOL, MFK-LAMOL, which constructs a generative replay using a more effective method. When a new task is received, MFK-LAMOL replays sufficient previous data and retrieves important examples for training alongside the new task. Specifically, it selects the examples with the most forgotten knowledge learned from previous tasks based on the extent to which they include knowledge that has been forgotten after learning new information. We showed that the proposed method outperforms LAMOL on a stream of three different natural language processing tasks.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - College of Engineering > School of Industrial and Management Engineering > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.