A User-Centric Evaluation Framework for Recommender Systems

This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users’ point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems’ Quality of user experience), which aimed at measuring the qualities of the recommended items, the system’s usability, usefulness, interface and interaction qualities, users’ satisfaction with the systems, and the influence of these qualities on users’ behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources.