Taxonomies of concepts are important across many application domains, for instance online shopping portals use catalogs to help users navigate and search for products. Task-dependent taxonomies, e.g., adapting the taxonomy to a specific cohort of users, can greatly improve the effectiveness of navigation and search. However, taxonomies are usually created by domain experts and hence designing task-dependent taxonomies can be an expensive process: this often limits the applications to deploy generic taxonomies. Crowdsourcing based techniques have the potential to provide a cost-efficient solution to building task-dependent taxonomies. In this paper, we present the first quantitative study to evaluate the effectiveness of these crowdsourcing based techniques. Our experimental study compares different task-dependent taxonomies built via crowdsourcing and generic taxonomies built by experts. We design randomized behavioral experiments on the Amazon Mechanical Turk platform for navigation tasks using these taxonomies resembling real-world applications such as product search. We record various metrics such as the time of navigation, the number of clicks performed, and the search path taken by a participant to navigate the taxonomy to locate a desired object. Our findings show that task-dependent taxonomies built by crowdsourcing techniques can reduce the navigation time up to 20%. Our results, in turn, demonstrate the power of crowdsourcing for learning complex structures such as semantic taxonomies.

@inproceedings{sun16taxonomies,
	author = {Yuyin Sun and Adish Singla and Tori Yan and Andreas Krause and Dieter Fox},
	booktitle = {AAAI Conference on Human Computation and Crowdsourcing (HCOMP)},
	month = {October},
	title = {Evaluating Task-Dependent Taxonomies for Navigation},
	year = {2016}}