Evaluating Preference-based Search Tools: a Tale of Two Approaches

People frequently use the world-wide web to find their most preferred item among a large range of options. We call this task preference-based search. The most common tool for preference-based search on the WWW today obtains users' preferences by asking them to fill in a form. It then returns a list of items that most closely match these preferences. Recently, several researchers have proposed tools for preference-based search that elicit preferences from the critiques a user actively makes on examples shown to them. We carried out a user study in order to compare the performance of traditional preference-based search tools using form-filling with two different versions of an example-critiquing tool. The results show that example critiquing achieves almost three times the decision accuracy, while requiring only slightly higher interaction effort.