Classification of web robots: An empirical study based on over one billion requests
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Lee, Junsup | - |
dc.contributor.author | Cha, Sunydeok | - |
dc.contributor.author | Lee, Dongkun | - |
dc.contributor.author | Lee, Hyungkyu | - |
dc.date.accessioned | 2021-09-08T12:02:00Z | - |
dc.date.available | 2021-09-08T12:02:00Z | - |
dc.date.created | 2021-06-11 | - |
dc.date.issued | 2009-11 | - |
dc.identifier.issn | 0167-4048 | - |
dc.identifier.uri | https://scholar.korea.ac.kr/handle/2021.sw.korea/119014 | - |
dc.description.abstract | Many studies on detection and classification of web robots have focused their attention mostly on text crawlers, and empirical experiments used relatively small data collected at universities. In this paper, we analyzed more than one billion requests to www.microsoft.com in 24 h. Web logs were made anonymous to eliminate potential privacy concerns while preserving essential characteristics (e.g., frequency, queries, etc). We have developed an effective characterization metrics, based on workload characteristics and resource types, in detecting and classifying various web robots including text crawlers, link checkers, and icon crawlers. As expected, web robot behavior was clearly different from that of typical interactive users, and different types of web robots also exhibited different characteristics. However, comparison of the similar type of web robots, text crawlers in particular, revealed different characteristics, thereby enabling characterization with reasonably high confidence level. we divided various feature metrics into five groups, and effectiveness of each group in classification is shown in polar diagram in the decreasing order of effectiveness in the clockwise direction. One can use the findings to classify likely identify of unknown web robots, and organizations can develop appropriate measures to deal with them. Our analysis is based on recent web log data collected at one of the best known site which offers truly global service. Crown Copyright (C) 2009 Published by Elsevier Ltd. All rights reserved. | - |
dc.language | English | - |
dc.language.iso | en | - |
dc.publisher | ELSEVIER ADVANCED TECHNOLOGY | - |
dc.subject | CRAWLER BEHAVIOR | - |
dc.title | Classification of web robots: An empirical study based on over one billion requests | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Cha, Sunydeok | - |
dc.identifier.doi | 10.1016/j.cose.2009.05.004 | - |
dc.identifier.scopusid | 2-s2.0-71849091131 | - |
dc.identifier.wosid | 000272742700007 | - |
dc.identifier.bibliographicCitation | COMPUTERS & SECURITY, v.28, no.8, pp.795 - 802 | - |
dc.relation.isPartOf | COMPUTERS & SECURITY | - |
dc.citation.title | COMPUTERS & SECURITY | - |
dc.citation.volume | 28 | - |
dc.citation.number | 8 | - |
dc.citation.startPage | 795 | - |
dc.citation.endPage | 802 | - |
dc.type.rims | ART | - |
dc.type.docType | Article | - |
dc.description.journalClass | 1 | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Information Systems | - |
dc.subject.keywordPlus | CRAWLER BEHAVIOR | - |
dc.subject.keywordAuthor | Web robot classification | - |
dc.subject.keywordAuthor | Web robot detection | - |
dc.subject.keywordAuthor | Web robot characterization | - |
dc.subject.keywordAuthor | Web security | - |
dc.subject.keywordAuthor | Web usage mining | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
(02841) 서울특별시 성북구 안암로 14502-3290-1114
COPYRIGHT © 2021 Korea University. All Rights Reserved.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.