There is empirical evidence that the data collected from a potentially smaller but smarter crowd can benefit a machine learning system. For instance, the Good Judgment Open project reported the existence of a small subset of super-forecasters. A general difficulty of learning and identifying a smarter crowd is the lack of ground-truth, be either in an objective or a subjective manner. I rigorously study this question and quantify the best sub-crowd for performing certain tasks.

A Gossiping Approach to Distributed Bandit Decision Making, with Ji Liu and Tamer Basar

Yang Liu and Mingyan Liu
An Online Learning Approach to Improving the Quality of Crowdsourcing.
ACM SIGMETRICS 2015, Portland, United States.
Extended version to appear at IEEE/ACM Transaction on Networkings, 2017.

Shuran Zheng, Bo Waggoner, Yang Liu and Yiling Chen
Active Information Acquisition for Linear Optimization.
arXiv:1709.10061.

Yang Liu and Mingyan Liu
Crowd Learning: Improving Online Decision Making Using Crowdsourced Data.
IJCAI 2017, Melbourne, Australia.

Yang Liu and Mingyan Liu
Finding One's Best Crowd: Online Learning By Exploiting Source Similarity.
AAAI 2016, Phoenix, United States.

Yang Liu and Yining Wang
Doubly Active Learning: when Active Learning meets Active Crowdsourcing.
CrowdML'16 (NIPS'16), Barcelona, Spain.

Ji Liu, Yang Liu, Angelia Nedich and Tamer Basar
An Approach to Distributed Parametric Learning with Streaming Data.
CDC 2017 (invited paper), Melbourne, Australia.

“If small groups are included in the decision-making process, then they should be allowed to make decisions. If an organization sets up teams and then uses them for purely advisory purposes, it loses the true advantage that a team has: namely, collective wisdom.” -- James Surowiecki, The Wisdom of Crowds

"The wisdom of crowds works when the crowd is choosing the price of an ox, when there's a single numeric average. But if it's a design or something that matters, the decision is made by committee, and that's crap. You want people and groups who are able to think thoughts before they share." Jaron Lanier