Abhimanyu Lad and Daniel Tunkelang
In the Fifth Workshop on Human-Computer Interaction and Information Retrieval (HCIR 2011)
Commonly used click-based measures like abandonment and mean reciprocal rank (MRR) present an incomplete, and often misleading, picture of search performance, especially in rich user interfaces that support a wide range of search behaviors. We propose a search utility framework that is based on a holistic view of the information-seeking process. First, we go beyond the use of clicks as indicators of relevance, taking into account actions performed on search results as more reliable indicators of completion of the user’s underlying task. Second, instead of looking only at individual queries, we consider the entire search session comprising multiple queries that are meant to address a single information need. We argue that an evaluation metric that combines these two features more accurately reflects the effectiveness of the system as perceived by the user. Finally, we propose future experiments to operationalize as well as validate this framework.