Crowds, Learning and Incentives

How can machines and people work together to accomplish what neither can alone? Our research focuses on problems at the interplay of learning and incentives in crowd-powered systems. Motivated by applications in community sensing, human computation, citizen science and crowdsourcing, we focus on particular technical challenges such as: How to optimally design incentives for learning / information gathering? How can one piece together a global picture from large numbers of noisy observations of a priori unknown quality?

We address these fundamental questions by building on state of the art results in machine learning, probabilistic modeling and game theory. Our research spans various application domains, including community sensing systems for earthquake detection and air-quality monitoring, rebalancing bike sharing systems by steering the behavior of the users, incentivizing users for privacy-tradeoff in information elicitation, teaching and training crowd workers, exploring social aspects of crowdsourcing, and learning optimal pricing policies for online crowdsourcing marketplaces.

Publications

2022

Incentive-Compatible Forecasting Competitions J. Witkowski, R. Freeman, J. W. Vaughan, D. Pennock, A. Krause In Management Science, 2022 [bibtex] [abstract] [pdf]

2018

Incentive-Compatible Forecasting Competitions J. Witkowski, R. Freeman, J. W. Vaughan, D. Pennock, A. Krause In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI’18), 2018 [bibtex] [abstract] [pdf]

2017

A Geometric Perspective on Minimal Peer Prediction R. Frongillo, J. Witkowski In ACM Transactions on Economics and Computation (TEAC), volume 5, 2017 [bibtex] [abstract] [pdf]

Learning and Incentives in Crowd-Powered Systems A. Singla PhD thesis, ETH Zurich, 2017 [bibtex] [abstract] [pdf]

Proper Proxy Scoring Rules J. Witkowski, P. Atanasov, L. H. Ungar, A. Krause In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI’17), 2017 forthcoming [bibtex] [abstract] [pdf]

2016

Evaluating Task-Dependent Taxonomies for Navigation Y. Sun, A. Singla, T. Yan, A. Krause, D. Fox In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2016 [bibtex] [abstract] [pdf]

Learning and Feature Selection under Budget Constraints in Crowdsourcing B. Nushi, A. Singla, A. Krause, D. Kossmann In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2016 [bibtex] [abstract] [pdf]

Actively Learning Hemimetrics with Applications to Eliciting User Preferences A. Singla, S. Tschiatschek, A. Krause In Proc. International Conference on Machine Learning (ICML), 2016 [bibtex] [abstract] [pdf] [long]

Noisy Submodular Maximization via Adaptive Sampling with Applications to Crowdsourced Image Collection Summarization A. Singla, S. Tschiatschek, A. Krause In Proc. Conference on Artificial Intelligence (AAAI), 2016 [bibtex] [abstract] [pdf] [long]

2015

Crowd Access Path Optimization: Diversity Matters B. Nushi, A. Singla, A. Gruenheid, E. Zamanian, A. Krause, D. Kossmann In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2015 [bibtex] [abstract] [pdf] [long]

Learning to Hire Teams A. Singla, E. Horvitz, P. Kohli, A. Krause In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), short paper, 2015 [bibtex] [abstract] [pdf] [long]

Information Gathering in Networks via Active Exploration A. Singla, E. Horvitz, P. Kohli, R. White, A. Krause In International Joint Conference on Artificial Intelligence (IJCAI), 2015 [bibtex] [abstract] [pdf] [long]

Building Hierarchies of Concepts via Crowdsourcing Y. Sun, A. Singla, D. Fox, A. Krause In International Joint Conference on Artificial Intelligence (IJCAI), 2015 [bibtex] [abstract] [pdf] [long]

Robot Navigation in Dense Crowds: Statistical Models and Experimental Studies of Human Robot Cooperation P. Trautman, J. Ma, R. Murray, A. Krause In International Journal on Robotics Research (IJRR), 2015 [bibtex] [abstract] [doi]

Incentivizing Users for Balancing Bike Sharing Systems A. Singla, M. Santoni, G. Bartok, P. Mukerji, M. Meenen, A. Krause In Proc. Conference on Artificial Intelligence (AAAI), 2015 Selected for AAAI Open House [bibtex] [abstract] [pdf]

2014

Mechanism Design for Crowdsourcing Markets with Heterogeneous Tasks G. Goel, A. Nikzad, A. Singla In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2014 [bibtex] [abstract] [pdf] [long]

Quantifying Web-Search Privacy A. Gervais, R. Shokri, A. Singla, S. Capkun, V. Lenders In Proc. ACM Conference on Computer and Communications Security (CCS), 2014 [bibtex] [abstract] [pdf]

Community Sense and Response Systems: Your Phone as Quake Detector M. Faulkner, R. Clayton, T. Heaton, K. M. Chandy, M. Kohler, J. Bunn, R. Guy, A. Liu, M. Olson, M. Cheng, A. Krause In Communications of the ACM, volume 57, 2014 Cover Feature [bibtex] [abstract] [doi]

Enhancing Personalization via Search Activity Attribution A. Singla, R. W. White, A. Hassan, E. Horvitz In Proc. Special Interest Group On Information Retrieval (SIGIR), 2014 [bibtex] [abstract] [pdf]

Near-Optimally Teaching the Crowd to Classify A. Singla, I. Bogunovic, G. Bartók, A. Karbasi, A. Krause In Proc. International Conference on Machine Learning (ICML), 2014 [bibtex] [abstract] [pdf] [long]

Stochastic Privacy A. Singla, E. Horvitz, E. Kamar, R. W. White In Proc. Conference on Artificial Intelligence (AAAI), 2014 [bibtex] [abstract] [pdf] [long]

Community sense and response systems M. Faulkner PhD thesis, California Institute of Technology, 2014 [bibtex] [abstract] [pdf]

Allocating Tasks to Workers with Matching Constraints: Truthful Mechanisms for Crowdsourcing Markets G. Goel, A. Nikzad, A. Singla In Proc. International World Wide Web Conference (WWW), 2014 [bibtex] [abstract] [pdf]

From Devices to People: Attribution of Search Activity in Multi-User Settings R. W. White, A. Hassan, A. Singla, E. Horvitz In Proc. International World Wide Web Conference (WWW), 2014 [bibtex] [abstract] [pdf]

2013

Incentives for Privacy Tradeoff in Community Sensing A. Singla, A. Krause In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2013 [bibtex] [abstract] [pdf] [long]

Truthful Incentives in Crowdsourcing Tasks using Regret Minimization Mechanisms A. Singla, A. Krause In International World Wide Web Conference (WWW), 2013 [bibtex] [abstract] [pdf]

A Fresh Perspective: Learning to Sparsify for Detection in Massive Noisy Sensor Networks M. Faulkner, A. Liu, A. Krause In ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), 2013 Best Paper Award [bibtex] [abstract] [pdf] [talk]

Robot Navigation in Dense Human Crowds: the Case for Cooperation P. Trautman, J. Ma, R. Murray, A. Krause In Proc. International Conference on Robotics and Automation (ICRA), 2013 Best Paper Award Finalist [bibtex] [abstract] [pdf]

2012

Robot Navigation in Dense Crowds: Statistical Models and Experimental Studies of Human Robot Cooperation P. Trautman PhD thesis, California Institute of Technology, 2012 [bibtex] [abstract] [pdf]

2011

Community Seismic Network R. Clayton, T. Heaton, M. Chandy, A. Krause, M. Kohler, J. Bunn, M. Olson, M. Faulkner, M. Cheng, L. Strand, R. Chandy, D. Obenshain, A. Liu, M. Aivazis, R. Guy In Annals of Geophysics, volume 54, 2011 [bibtex] [abstract] [pdf]

The Next Big One: Detecting Earthquakes and other Rare Events from Community-based Sensors M. Faulkner, M. Olson, R. Chandy, J. Krause, K. M. Chandy, A. Krause In Proc. ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), 2011 [bibtex] [abstract] [pdf] [talk]

Randomized Sensing in Adversarial Environments A. Krause, A. Roper, D. Golovin In Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2011 [bibtex] [abstract] [pdf] [talk]

Crowdclustering R. Gomes, P. Welinder, A. Krause, P. Perona In Proc. Neural Information Processing Systems (NeurIPS), 2011 [bibtex] [abstract] [pdf] [long]

2010

A Utility-Theoretic Approach to Privacy in Online Services A. Krause, E. Horvitz In Journal of Artificial Intelligence Research (JAIR), volume 39, 2010 [bibtex] [abstract] [pdf] [talk]

Unfreezing the Robot: Navigation in Dense, Interacting Crowds P. Trautman, A. Krause In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010 [bibtex] [abstract] [pdf] [talk]

Online Distributed Sensor Selection D. Golovin, M. Faulkner, A. Krause In Proc. ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), 2010 [bibtex] [abstract] [pdf] [long] [talk]

2008

A Utility-Theoretic Approach to Privacy and Personalization A. Krause, E. Horvitz In Proc. 23rd Conference on Artificial Intelligence (AAAI), 2008 [bibtex] [abstract] [pdf] [talk]

Toward Community Sensing A. Krause, E. Horvitz, A. Kansal, F. Zhao In Proc. ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), 2008 [bibtex] [abstract] [pdf] [talk]