On the Redundancy in the Rank of Neural Network Parameters and Its Controllability
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Lee, Chanhee | - |
dc.contributor.author | Kim, Young-Bum | - |
dc.contributor.author | Ji, Hyesung | - |
dc.contributor.author | Lee, Yeonsoo | - |
dc.contributor.author | Hur, Yuna | - |
dc.contributor.author | Lim, Heuiseok | - |
dc.date.accessioned | 2021-08-30T05:04:34Z | - |
dc.date.available | 2021-08-30T05:04:34Z | - |
dc.date.created | 2021-06-18 | - |
dc.date.issued | 2021-01 | - |
dc.identifier.issn | 2076-3417 | - |
dc.identifier.uri | https://scholar.korea.ac.kr/handle/2021.sw.korea/50623 | - |
dc.description.abstract | In this paper, we show that parameters of a neural network can have redundancy in their ranks, both theoretically and empirically. When viewed as a function from one space to another, neural networks can exhibit feature correlation and slower training due to this redundancy. Motivated by this, we propose a novel regularization method to reduce the redundancy in the rank of parameters. It is a combination of an objective function that makes the parameter rank-deficient and a dynamic low-rank factorization algorithm that gradually reduces the size of this parameter by fusing linearly dependent vectors together. This regularization-by-pruning approach leads to a neural network with better training dynamics and fewer trainable parameters. We also present experimental results that verify our claims. When applied to a neural network trained to classify images, this method provides statistically significant improvement in accuracy and 7.1 times speedup in terms of number of steps required for training. Furthermore, this approach has the side benefit of reducing the network size, which led to a model with 30.65% fewer trainable parameters. | - |
dc.language | English | - |
dc.language.iso | en | - |
dc.publisher | MDPI | - |
dc.title | On the Redundancy in the Rank of Neural Network Parameters and Its Controllability | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Lim, Heuiseok | - |
dc.identifier.doi | 10.3390/app11020725 | - |
dc.identifier.scopusid | 2-s2.0-85099418323 | - |
dc.identifier.wosid | 000610919500001 | - |
dc.identifier.bibliographicCitation | APPLIED SCIENCES-BASEL, v.11, no.2, pp.1 - 15 | - |
dc.relation.isPartOf | APPLIED SCIENCES-BASEL | - |
dc.citation.title | APPLIED SCIENCES-BASEL | - |
dc.citation.volume | 11 | - |
dc.citation.number | 2 | - |
dc.citation.startPage | 1 | - |
dc.citation.endPage | 15 | - |
dc.type.rims | ART | - |
dc.type.docType | Article | - |
dc.description.journalClass | 1 | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Chemistry | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalResearchArea | Materials Science | - |
dc.relation.journalResearchArea | Physics | - |
dc.relation.journalWebOfScienceCategory | Chemistry, Multidisciplinary | - |
dc.relation.journalWebOfScienceCategory | Engineering, Multidisciplinary | - |
dc.relation.journalWebOfScienceCategory | Materials Science, Multidisciplinary | - |
dc.relation.journalWebOfScienceCategory | Physics, Applied | - |
dc.subject.keywordAuthor | matrix rank | - |
dc.subject.keywordAuthor | neural network | - |
dc.subject.keywordAuthor | pruning | - |
dc.subject.keywordAuthor | redundancy | - |
dc.subject.keywordAuthor | regularization | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
(02841) 서울특별시 성북구 안암로 14502-3290-1114
COPYRIGHT © 2021 Korea University. All Rights Reserved.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.