简介: |
Content-based search for feature-rich or non-text data has been a challenging problem in information retrieval, data mining and data analysis. The main reason is that feature-rich data are noisy and their feature vectors are high dimensional. Comparing noisy, feature-rich data requires fast similarity search instead of exact match, and thus searching for noisy data requires similarity search instead of exact search. Because of the difficulty of similarity search in high-dimensional spaces (the so called curse of dimensionality), practical advanced search solutions, such as database tools and search engines, have been limited to searching for exact matches and they tend to work only for text documents or small feature-rich datasets.
This talk presents the design, implementation, and applications of a toolkit for constructing content-based similarity search tools or search engines for feature-rich data. The toolkit includes a core content-based similarity search engine, a metadata manager, an integrated attribute-based search mechanism, a command-line and client web interface, and a performance evaluation tool. The design of the core content-based similarity search mechanisms was inspired by several recent theoretical results on dimension reduction and sketch construction for similarity searches. The toolkit has been used to construct content-based similarity search engines for four data types: audio data, digital images, 3D shape models and genomic micro-array data. Our experiments with these search engines showed that the toolkit is efficient comparing with existing domain-specific search tools. It can typically reduce the metadata storage requirement by an order of magnitude while achieving high speed and high quality.
|