Economic and algorithmic truth serum

The capability of collecting high quality input information from a diverse population of human subjects for building a machine learning system is crucial and fundamental to implement the idea of involving society in the loop algorithmically. Strategic human agents who hold private observation can choose to report their information (data) arbitrarily; yet the mechanism designer (e.g., the practitioner collecting training data for a machine learning system) has no ground-truth verification to validate the reports.

"You've got to change incentives for good behavior as opposed to just disincentivizing bad behavior." -- Gavin Newsom

"Call it what you will, incentives are what get people to work harder." -- Nikita Khrushchev

I'm interested in reading people's mind via developing economic and algorithmic truth serum. Existing solutions designed payment and scoring functions to better engage rational human agents to contribute high quality data. There the solution concept builds on soliciting redundant answers for each information from different peer agents, to serve as references answers to score or pay each agent. I explore the design of incentive aligned system for data collection from machine learning and computation's perspective.

Relevant works

Yang Liu and Yiling Chen
Surrogate Scoring Rules and a Dominant Truth Serum for Information Elicitation
in preparation.

  • Equilibrium notion, that is it is agent's best interest to truthfully reveal their answer, if all other agents are truthfully reporting, as typically adopted in the game theory context, renders its solution not robust for many practical applications, for several good reasons. We propose the first dominant truth serum for such information elicitation problems.

Yang Liu and Yiling Chen
Machine Learning aided Peer Prediction.
ACM EC 2017, Cambridge, United States

Yang Liu and Yiling Chen
Sequential Peer Prediction: Learning to Elicit Effort using Posted Prices.
AAAI 2017, San Francisco, United States.

Yang Liu and Yiling Chen
A Bandit Framework for Strategic Regression.
NIPS 2016, Barcelona, Spain.

Zehong Hu, Yang Liu, Yitao Liang and Jie Zhang
A Reinforcement Learning Framework for Eliciting High Quality Information.
Machine Learning in the Presence of Strategic Behavior at NIPS'17, Long Beach, United States.

  • We build a reinforment learning framework to push peer prediction mechanisms towards more practical settings that (1) agents may not be fully rational (2) the right scaling of payment is learned through interacting with agents.

Yang Liu and Chien-Ju Ho
Incentivizing High Quality User Contributions: New Arm Generation in Bandit Learning.
AAAI 2018, New Orleans, United States.

Yang Liu and Yiling Chen
Learning to Incentivize: Eliciting Effort via Output Agreement.
IJCAI 2016, New York City, Uniteed States

Shang-Pin Sheng, Yang Liu and Mingyan Liu
A Regulated Oligopoly Multi-Market Model for Trading Smart Data.
IEEE SDP 2015, in conjunction with IEEE INFOCOM 2015, Hong Kong, China.