Classification and indexing scheme of large-scale image repository for spatio-temporal landmark recognition

Abstract

In this paper, we propose a classification and indexing scheme of large-scale image repository for spatio-temporal landmark recognition using the local features, GPS data and user tags of images. For spatio-temporal landmark image classification, we first divide Earth's entire surface into unit grid cells and collect pictures taken in each cell through Flickr. The collected images contain information such as location, titles and other user tags. Usually, the titles or user tags of landmark images include landmark names. Hence, by analyzing such tags, we can identify promising landmark names in the region and create a collection of images for each landmark using Flickr API. Even though each landmark class contains images of the same landmark, their spatio-temporal features could be different depending on shooting time, distance or angle. Therefore, we further divide the images in each landmark class into several subclasses according to their spatio-temporal characteristics using their color and local features. Especially, we detect the interest points of the images in the class, construct their feature descriptors using SURF and perform statistical analysis to select their representative points. Similar representative points are merged for fast comparison. Finally, we construct an index on the representative points using k-d tree. To identify the landmark in a user query image, we extract its SURF features and search for them in the index. Most similar matches are returned, along with descriptive text and GPS information. We implemented a prototype system based on a client-server architecture and performed various experiments to demonstrate that our scheme can achieve reasonable precision and scalability and provide a new browsing experience to the user.