Affective Computing / Emotion Modeling

The group has been working on affective computing since 2005. They pioneered in computational modeling of visual aesthetics, predicting evoked emotions from visual content, and automated recognition of bodily expression of emotion, all for in-the-wild situations. We have also worked on other topics such as microexpression analysis, photo composition analysis, and computational psychiatry.

The following is an archive of our publications in this area. They are in reverse chronical order. You can also check out information on a past NSF project.

  1. Yu Luo, Jianbo Ye, Reginald B. Adams, Jr., Jia Li, Michelle G. Newman and James Z. Wang, ``ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wild,'' International Journal of Computer Vision, vol. 128, no. 1, pp. 1-25, 2020. [A version was posted in August 2018 at http://arxiv.org/abs/1808.09568] (download) (g-scholar)
  2. Jianbo Ye, Jia Li, Michelle G. Newman, Reginald B. Adams, Jr. and James Z. Wang, ``Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data,'' IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 115-128, 2019. [A version was posted in January 2017 at http://arxiv.org/abs/1701.01096] (download) (g-scholar)
  3. Hanjoo Kim, Xin Lu, Michael Costa, Baris Kandemir, Reginald B. Adams, Jr., Jia Li, James Z. Wang and Michelle G. Newman, ``Development and Validation of Image Stimuli for Emotion Elicitation (ISEE): A Novel Affective Pictorial System with Test-Retest Repeatability,'' Psychiatry Research, vol. 261, pp. 414-420, Elsevier, 2018. (download) (g-scholar)
  4. Siqiong He, Zihan Zhou, Farshid Farhat and James Z. Wang, ``Discovering Triangles in Portraits for Supporting Photographic Creation,'' IEEE Transactions on Multimedia, vol. 20, no. 2, 496-508, 2018. [A version was posted in May 2016 at http://arxiv.org/abs/1605.09559] (download) (g-scholar)
  5. Zihan Zhou, Farshid Farhat and James Z. Wang, ``Detecting Dominant Vanishing Points in Natural Scenes with Application to Composition-Sensitive Image Retrieval,'' IEEE Transactions on Multimedia, vol. 19, no. 12, pp. 2651-2665, 2017. (download) (g-scholar)
  6. Feng Xu, Junping Zhang and James Z. Wang, ``Microexpression Identification and Categorization using a Facial Dynamics Map,'' IEEE Transactions on Affective Computing, vol. 8, no. 2, pp. 254-267, 2017. (download) (g-scholar)
  7. Xin Lu, Reginald B. Adams, Jr., Jia Li, Michelle G. Newman and James Z. Wang, ``An Investigation into Three Visual Characteristics of Complex Scenes that Evoke Human Emotion,'' Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 440-447, San Antonio, Texas, October 2017. (download) (g-scholar)
  8. Baris Kandemir, Zihan Zhou, Jia Li and James Z. Wang, ``Beyond Saliency: Assessing Visual Balance with High-level Cues,'' Proceedings of the Thematic Workshops of ACM Multimedia, in conjunction with the ACM Multimedia Conference, pp. 26-34, Mountain View, California, October 2017. (download) (g-scholar)
  9. Farshid Farhat, Mohammad M. Kamani, Sahil Mishra and James Z. Wang, ``Intelligent Portrait Composition Assistance - Integrating Deep-learned Models and Photography Idea Retrieval,'' Proceedings of the Thematic Workshops of ACM Multimedia, in conjunction with the ACM Multimedia Conference, pp. 17-25, Mountain View, California, October 2017. (download) (g-scholar)
  10. Ye Zhou, Xin Lu, Junping Zhang and James Z. Wang, ``Joint Image and Text Representation for Aesthetics Analysis,'' Proceedings of the ACM Multimedia Conference, pp. 262-266, Amsterdam, The Netherlands, ACM, October 2016. (download) (g-scholar)
  11. Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang and James Z. Wang, ``Rating Image Aesthetics using Deep Learning,'' IEEE Transactions on Multimedia, vol. 17, no. 11, pp, 2021-2034, 2015. (download) (g-scholar)
  12. Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech and James Z. Wang, ``Deep Multi-Patch Aggregation Network for Image Style, Aesthetics, and Quality Estimation,'' Proceedings of the International Conference on Computer Vision, pp. 990-998, Santiago, Chile, IEEE, 2015. (download) (g-scholar)
  13. Zihan Zhou, Siqiong He, Jia Li and James Z. Wang, ``Modeling Perspective Effects in Photographic Composition,'' Proceedings of the ACM Multimedia Conference, pp. 301-310, Brisbane, Australia, ACM, October 2015. (download) (g-scholar)
  14. Jia Li, Lei Yao and James Z. Wang, ``Photo Composition Feedback and Enhancement --- Exploiting Spatial Design Categories and the Notan Dark-Light Principle,'' Mobile Cloud Visual Media Computing, G. Hua and X.-S. Hua (eds.), Springer Int. Publ. Switzerland, Chapter 5, pp. 113-144, 2015. (download) (g-scholar)
  15. Hanjoo Kim, Xin Lu, Michael Costa, Baris Kandemir, Reginald B. Adams, Jr., Jia Li, James Z. Wang and Michelle G. Newman, ``Development and Validation of the Image Stimuli for Emotion Elicitation (ISEE),'' Association for Psychological Science Annual Convention, poster, New York City, May 2015. (download) (g-scholar)
  16. Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang and James Z. Wang, ``RAPID: Rating Pictorial Aesthetics using Deep Learning,'' Proceedings of the ACM Multimedia Conference, pp. 457-466, Orlando, Florida, ACM, November 2014. (download) (g-scholar)
  17. Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Xin Lu, Quang-Tuan Luong, James Z. Wang, Jia Li and Jiebo Luo, ``On Aesthetics and Emotions in Scene Images: A Computational Perspective,'' Scene Vision: Making Sense of What We See, K. Kveraga and M. Bar (eds.), MIT Press, pp. 241-272, Chapter 12, November 2014. (download) (g-scholar)
  18. Lei Yao, Poonam Suryanarayan, Mu Qiao, James Z. Wang and Jia Li, ``OSCAR: On-Site Composition and Aesthetics Feedback through Exemplars for Photographers,'' International Journal of Computer Vision, vol. 96, no. 3, pp. 353-383, 2012. [DOI: 10.1007/s11263-011-0478-3] (download) (g-scholar)
  19. Xin Lu, Poonam Suryanarayan, Reginald B. Adams, Jr., Jia Li, Michelle G. Newman and James Z. Wang, ``On Shape and the Computability of Emotions,'' Proceedings of the ACM Multimedia Conference, pp. 229-238, Nara, Japan, ACM, October 2012. (download) (g-scholar)
  20. Dhiraj Joshi, Ritendra Datta, Quang-Tuan Luong, Elena Fedorovskaya, James Z. Wang, Jia Li and Jiebo Luo, ``Aesthetics and Emotions in Images: A Computational Perspective,'' IEEE Signal Processing Magazine, vol. 28, no. 5, pp. 94-115, September 2011. (download) (g-scholar)
  21. Razvan Orendovici and James Z. Wang, ``Training Data Collection System for a Learning-based Photographic Aesthetic Quality Inference Engine,'' Proceedings of the ACM Multimedia Conference, Demonstration, pp. 1575-1578, Florence, Italy, ACM, October 2010. (download) (g-scholar)
  22. Ritendra Datta and James Z. Wang, ``ACQUINE: Aesthetic Quality Inference Engine - Real-time Automatic Rating of Photo Aesthetics,'' Proceedings of the ACM International Conference on Multimedia Information Retrieval, Demonstration, pp. 421-424, Philadelphia, Pennsylvania, ACM, March 2010. (download) (g-scholar)
  23. Ritendra Datta, Jia Li and James Z. Wang, ``Algorithmic Inferencing of Aesthetics and Emotion in Natural Images: An Exposition,'' Proceedings of the IEEE International Conference on Image Processing (ICIP), Special Session on Image Aesthetics, Mood and Emotion, pp. 105-108, San Diego, California, IEEE, October 2008. [invited but peer-reviewed] (download) (g-scholar)
  24. Ritendra Datta, Jia Li and James Z. Wang, ``Learning the Consensus on Visual Quality for Next-Generation Image Management,'' Proceedings of the ACM Multimedia Conference, pp. 533-536, ACM, Augsburg, Germany, September 2007. (download) (g-scholar)
  25. Ritendra Datta, Dhiraj Joshi, Jia Li and James Z. Wang, ``Studying Aesthetics in Photographic Images Using a Computational Approach,'' Lecture Notes in Computer Science, vol. 3953, Proceedings of the European Conference on Computer Vision, Part III, pp. 288-301, Graz, Austria, Springer, May 2006. (download) (g-scholar)

In the midst of difficulty lies opportunity. - Albert Einstein

COPYRIGHT © 2000- James Z. Wang Research Group, Penn State, University Park .