Existing mathematical notions of fairness fail to account for the context of decision-making. We argue that moral consideration of contextual factors is an inherently human task. So we present a framework to learn context-aware mathematical formulations of fairness by eliciting people's situated fairness assessments. Our family of fairness notions corresponds to a new interpretation of economic models of Equality of Opportunity (EOP), and it includes most existing notions of fairness as special cases. Our human-in-the-loop approach is designed to learn the appropriate parameters of the EOP family by utilizing human responses to pair-wise questions about decision subjects' circumstance and deservingness, and the harm/benefit imposed on them. We illustrate our framework in a hypothetical criminal risk assessment scenario by conducting a series of human-subject experiments on Amazon Mechanical Turk. Our work takes an important initial step toward empowering stakeholders to have a voice in the formulation of fairness for Machine Learning.

@inproceedings{yaghini21human,
	author = {Mohammad Yaghini and Andreas Krause and Hoda Heidari},
	booktitle = {In Proc. AAAI/ACM Conference on AI, Ethics, and Society (AIES)},
	month = {July},
	title = {A Human-in-the-loop Framework to Construct Context-aware Mathematical Notions of Outcome Fairness},
	video = {https://www.youtube.com/watch?v=46abDfhb7jw},
	year = {2021}}