Papers: [ACII 2017] [IEEE TAC 2019] [Psychiatry Research 2018] [IJCV 2019]
Jianbo Ye, Jia Li, Michelle G. Newman, Reginald B. Adams, Jr., James Z. Wang
The Pennsylvania State University, USA
We proposed a probabilistic approach to joint modeling of
participants' reliability and humans' regularity in crowdsourced
affective studies. Reliability measures how likely a subject will
respond to a question seriously; and regularity measures how often a
human will agree with other seriously-entered responses coming from a
targeted population. Crowdsourcing-based studies or experiments, which
rely on human self-reported affect, pose additional challenges as
compared with typical crowdsourcing studies that attempt to acquire
concrete non-affective labels of objects. The reliability of
participants has been massively pursued for typical non-affective
crowdsourcing studies, whereas the regularity of humans in an
affective experiment in its own right has not been thoroughly
considered. It has been often observed that different individuals
exhibit different feelings on the same test question, which does not
have a sole correct response in the first place. High reliability of
responses from one individual thus cannot conclusively result in high
consensus across individuals. Instead, globally testing consensus of a
population is of interest to investigators. Built upon the agreement
multigraph among tasks and workers, our probabilistic model
differentiates subject regularity from population reliability. We
demonstrate the method's effectiveness for in-depth robust analysis of
large-scale crowdsourced affective data, including emotion and
aesthetic assessments collected by presenting visual stimuli to human
subjects.
Full Paper
Dataset: 90% confident high/low valence images from Amazon Mechanical Turkamt_validated_valence_90.zip (524MB, 4415 images; due to copyright please contact Dr. James Wang to obtain for research; jwang at ist.psu.edu)
Dataset: 95% confident high/low valence images from Amazon Mechanical Turkamt_validated_valence_95.zip (11MB, 94 images; due to copyright please contact Dr. James Wang to obtain for research; jwang at ist.psu.edu)
Citation: Jianbo Ye, Jia Li, Michelle G. Newman, Reginald B. Adams, Jr. and James Z. Wang, ``Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data,'' IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 115-128, 2019.
© 2017 All rights reserved.