Text Embedding Augmentation Based on Retraining With Pseudo-Labeled Adversarial Embedding
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Kim, M. | - |
dc.contributor.author | Kang, P. | - |
dc.date.accessioned | 2022-08-14T22:40:51Z | - |
dc.date.available | 2022-08-14T22:40:51Z | - |
dc.date.created | 2022-08-12 | - |
dc.date.issued | 2022 | - |
dc.identifier.issn | 2169-3536 | - |
dc.identifier.uri | https://scholar.korea.ac.kr/handle/2021.sw.korea/143202 | - |
dc.description.abstract | Pre-trained language models (LMs) have been shown to achieve outstanding performance in various natural language processing tasks; however, these models have a significantly large number of parameters to handle large-scale text corpora during the pre-training process, and thus, they entail the risk of overfitting when fine-tuning for small task-oriented datasets is conducted. In this paper, we propose a text embedding augmentation method to prevent such overfitting. The proposed method applies augmentation to a text embedding by generating an adversarial embedding, which is not identical to original input embedding but maintaining the characteristics of the original input embedding, using PGD-based adversarial training for input text data. A pseudo-label that is identical to the label of the input text is then assigned to adversarial embedding to conduct retraining by using adversarial embedding and pseudo-label as input embedding and label pair for a separate LM. Experimental results on several text classification benchmark datasets demonstrated that the proposed method effectively prevented overfitting, which commonly occurs when adjusting a large-scale pre-trained LM to a specific task. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ | - |
dc.language | English | - |
dc.language.iso | en | - |
dc.publisher | Institute of Electrical and Electronics Engineers Inc. | - |
dc.title | Text Embedding Augmentation Based on Retraining With Pseudo-Labeled Adversarial Embedding | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Kang, P. | - |
dc.identifier.doi | 10.1109/ACCESS.2022.3142843 | - |
dc.identifier.scopusid | 2-s2.0-85123310058 | - |
dc.identifier.wosid | 000838501200001 | - |
dc.identifier.bibliographicCitation | IEEE Access, v.10, pp.8363 - 8376 | - |
dc.relation.isPartOf | IEEE Access | - |
dc.citation.title | IEEE Access | - |
dc.citation.volume | 10 | - |
dc.citation.startPage | 8363 | - |
dc.citation.endPage | 8376 | - |
dc.type.rims | ART | - |
dc.type.docType | Article | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | Y | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalResearchArea | Telecommunications | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Information Systems | - |
dc.relation.journalWebOfScienceCategory | Engineering, Electrical & Electronic | - |
dc.relation.journalWebOfScienceCategory | Telecommunications | - |
dc.subject.keywordAuthor | Data models | - |
dc.subject.keywordAuthor | Extrapolation | - |
dc.subject.keywordAuthor | Interpolation | - |
dc.subject.keywordAuthor | Semantics | - |
dc.subject.keywordAuthor | Task analysis | - |
dc.subject.keywordAuthor | Training | - |
dc.subject.keywordAuthor | Transformers | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
(02841) 서울특별시 성북구 안암로 14502-3290-1114
COPYRIGHT © 2021 Korea University. All Rights Reserved.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.