Web robot detection based on pattern-matching technique
- Authors
- Kwon, Shinil; Kim, Young-Gab; Cha, Sungdeok
- Issue Date
- 4월-2012
- Publisher
- SAGE PUBLICATIONS LTD
- Keywords
- web robot detection; web robot pattern; human pattern; pattern analysis
- Citation
- JOURNAL OF INFORMATION SCIENCE, v.38, no.2, pp.118 - 126
- Indexed
- SCIE
SSCI
SCOPUS
- Journal Title
- JOURNAL OF INFORMATION SCIENCE
- Volume
- 38
- Number
- 2
- Start Page
- 118
- End Page
- 126
- URI
- https://scholar.korea.ac.kr/handle/2021.sw.korea/108839
- DOI
- 10.1177/0165551511435969
- ISSN
- 0165-5515
- Abstract
- In web robot detection it is important is to find features that are common characteristics of diverse robots, in order to differentiate between them and humans. Existing approaches employ fairly simple features (e.g. empty referrer field, interval between successive requests), which often fail to reflect web robots' behaviour accurately. False alarms may therefore occur unacceptably often. In this paper we propose a fresh approach that expresses the behaviour of interactive users and various web robots in terms of a sequence of request types, called request patterns. Previous proposals have primarily targeted the detection of text crawlers, but our approach works well on many other web robots, such as image crawlers, email collectors and link checkers. In empirical evaluation of more than 1 billion requests collected at www.microsoft.com, our approach achieved 94% accuracy in web robot detection, estimated by F-measure. A decision tree algorithm proposed by Tan and Kumar was also applied to the same data. A comparison shows that the proposed approach is more accurate, and that real-time detection of web robots is feasible.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - Graduate School > Department of Computer Science and Engineering > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.