Affective Computing / Emotion Modeling

The group has been working on affective computing since 2005. They pioneered in computational modeling of visual aesthetics, predicting evoked emotions from visual content, and automated recognition of bodily expression of emotion, all for in-the-wild situations. We have also worked on other topics such as microexpression analysis, photo composition analysis, and computational psychiatry.

Our ongoing research projects include an NSF-funded data infrastructure project and Amazon Research Awards gift-funded project.

The following is an archive of our publications in this area. They are in reverse chronical order. You can also check out information on a past NSF project.

  1. Benjamin Wortman and James Z. Wang, ``HICEM: A High-Coverage Emotion Model for Artificial Emotional Intelligence,'' IEEE Transactions on Affective Computing, vol. , no. , pp. -, 2022, under revision for second-round review. [A version was posted in June 2022 at https://arxiv.org/abs/2206.07593] (download) (g-scholar)
  2. James Z. Wang, Sicheng Zhao, Chenyan Wu, Reginald B. Adams, Jr., Michelle G. Newman, Tal Shafir and Rachelle Tsachor, ``Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion,'' Proceedings of the IEEE, 2023, under second-round review. (download) (g-scholar)
  3. Sitao Zhang, Yimu Pan and James Z. Wang, ``Learning Emotion Representations from Verbal and Nonverbal Communication,'' Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. -, Vancouver, Canada, June 2023. (download) (g-scholar)
  4. James Z. Wang and Reginald B. Adams, Jr., editors, Modeling Visual Aesthetics, Emotion, and Artistic Style, Springer, 2023, to be published. (download) (g-scholar)
  5. Farshid Farhat, Mohammad M. Kamani and James Z. Wang, ``CAPTAIN: Comprehensive Composition Assistance for Photo Taking,'' ACM Transactions on Multimedia Computing, Communications and Applications, vol. 18, no. 1, article 14, pp. 14:1-24, 2022. (download) (g-scholar)
  6. Yu Luo, Jianbo Ye, Reginald B. Adams, Jr., Jia Li, Michelle G. Newman and James Z. Wang, ``ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wild,'' International Journal of Computer Vision, vol. 128, no. 1, pp. 1-25, 2020. [A version was posted in August 2018 at http://arxiv.org/abs/1808.09568] (download) (g-scholar)
  7. James Z. Wang, ``Modeling Aesthetics and Emotions in Visual Content: From Vincent van Gogh to Robotics and Vision,'' Proceedings of the Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends, in conjunction with the ACM International Conference on Multimedia, 2 pages, Virtual, October 2020. (download) (g-scholar)
  8. James Z. Wang, Norman Badler, Nadia Berthouze, Rick O. Gilmore, Kerri L. Johnson, Agata Lapedriza, Xin Lu and Nikolaus Troje, ``Bodily Expressed Emotion Understanding Research: A Multidisciplinary Perspective,'' Proceedings of the First International Workshop on Bodily Expressed Emotion Understanding, in conjunction with the European Computer Vision Conference, pp. 1-14, Virtual, August 2020. (download) (g-scholar)
  9. Jianbo Ye, Jia Li, Michelle G. Newman, Reginald B. Adams, Jr. and James Z. Wang, ``Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data,'' IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 115-128, 2019. [A version was posted in January 2017 at http://arxiv.org/abs/1701.01096] (download) (g-scholar)
  10. Hanjoo Kim, Xin Lu, Michael Costa, Baris Kandemir, Reginald B. Adams, Jr., Jia Li, James Z. Wang and Michelle G. Newman, ``Development and Validation of Image Stimuli for Emotion Elicitation (ISEE): A Novel Affective Pictorial System with Test-Retest Repeatability,'' Psychiatry Research, vol. 261, pp. 414-420, Elsevier, 2018. (download) (g-scholar)
  11. Siqiong He, Zihan Zhou, Farshid Farhat and James Z. Wang, ``Discovering Triangles in Portraits for Supporting Photographic Creation,'' IEEE Transactions on Multimedia, vol. 20, no. 2, 496-508, 2018. [A version was posted in May 2016 at http://arxiv.org/abs/1605.09559] (download) (g-scholar)
  12. Zihan Zhou, Farshid Farhat and James Z. Wang, ``Detecting Dominant Vanishing Points in Natural Scenes with Application to Composition-Sensitive Image Retrieval,'' IEEE Transactions on Multimedia, vol. 19, no. 12, pp. 2651-2665, 2017. (download) (g-scholar)
  13. Feng Xu, Junping Zhang and James Z. Wang, ``Microexpression Identification and Categorization using a Facial Dynamics Map,'' IEEE Transactions on Affective Computing, vol. 8, no. 2, pp. 254-267, 2017. (download) (g-scholar)
  14. Xin Lu, Reginald B. Adams, Jr., Jia Li, Michelle G. Newman and James Z. Wang, ``An Investigation into Three Visual Characteristics of Complex Scenes that Evoke Human Emotion,'' Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 440-447, San Antonio, Texas, October 2017. (download) (g-scholar)
  15. Baris Kandemir, Zihan Zhou, Jia Li and James Z. Wang, ``Beyond Saliency: Assessing Visual Balance with High-level Cues,'' Proceedings of the Thematic Workshops of ACM Multimedia, in conjunction with the ACM Multimedia Conference, pp. 26-34, Mountain View, California, October 2017. (download) (g-scholar)
  16. Farshid Farhat, Mohammad M. Kamani, Sahil Mishra and James Z. Wang, ``Intelligent Portrait Composition Assistance - Integrating Deep-learned Models and Photography Idea Retrieval,'' Proceedings of the Thematic Workshops of ACM Multimedia, in conjunction with the ACM Multimedia Conference, pp. 17-25, Mountain View, California, October 2017. (download) (g-scholar)
  17. Ye Zhou, Xin Lu, Junping Zhang and James Z. Wang, ``Joint Image and Text Representation for Aesthetics Analysis,'' Proceedings of the ACM Multimedia Conference, pp. 262-266, Amsterdam, The Netherlands, ACM, October 2016. (download) (g-scholar)
  18. Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang and James Z. Wang, ``Rating Image Aesthetics using Deep Learning,'' IEEE Transactions on Multimedia, vol. 17, no. 11, pp, 2021-2034, 2015. (download) (g-scholar)
  19. Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech and James Z. Wang, ``Deep Multi-Patch Aggregation Network for Image Style, Aesthetics, and Quality Estimation,'' Proceedings of the International Conference on Computer Vision, pp. 990-998, Santiago, Chile, IEEE, 2015. (download) (g-scholar)
  20. Zihan Zhou, Siqiong He, Jia Li and James Z. Wang, ``Modeling Perspective Effects in Photographic Composition,'' Proceedings of the ACM Multimedia Conference, pp. 301-310, Brisbane, Australia, ACM, October 2015. (download) (g-scholar)
  21. Jia Li, Lei Yao and James Z. Wang, ``Photo Composition Feedback and Enhancement --- Exploiting Spatial Design Categories and the Notan Dark-Light Principle,'' Mobile Cloud Visual Media Computing, G. Hua and X.-S. Hua (eds.), Springer Int. Publ. Switzerland, Chapter 5, pp. 113-144, 2015. (download) (g-scholar)
  22. Hanjoo Kim, Xin Lu, Michael Costa, Baris Kandemir, Reginald B. Adams, Jr., Jia Li, James Z. Wang and Michelle G. Newman, ``Development and Validation of the Image Stimuli for Emotion Elicitation (ISEE),'' Association for Psychological Science Annual Convention, poster, New York City, May 2015. (download) (g-scholar)
  23. Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang and James Z. Wang, ``RAPID: Rating Pictorial Aesthetics using Deep Learning,'' Proceedings of the ACM Multimedia Conference, pp. 457-466, Orlando, Florida, ACM, November 2014. (download) (g-scholar)
  24. Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Xin Lu, Quang-Tuan Luong, James Z. Wang, Jia Li and Jiebo Luo, ``On Aesthetics and Emotions in Scene Images: A Computational Perspective,'' Scene Vision: Making Sense of What We See, K. Kveraga and M. Bar (eds.), MIT Press, pp. 241-272, Chapter 12, November 2014. (download) (g-scholar)
  25. Lei Yao, Poonam Suryanarayan, Mu Qiao, James Z. Wang and Jia Li, ``OSCAR: On-Site Composition and Aesthetics Feedback through Exemplars for Photographers,'' International Journal of Computer Vision, vol. 96, no. 3, pp. 353-383, 2012. [DOI: 10.1007/s11263-011-0478-3] (download) (g-scholar)
  26. Xin Lu, Poonam Suryanarayan, Reginald B. Adams, Jr., Jia Li, Michelle G. Newman and James Z. Wang, ``On Shape and the Computability of Emotions,'' Proceedings of the ACM Multimedia Conference, pp. 229-238, Nara, Japan, ACM, October 2012. (download) (g-scholar)
  27. Dhiraj Joshi, Ritendra Datta, Quang-Tuan Luong, Elena Fedorovskaya, James Z. Wang, Jia Li and Jiebo Luo, ``Aesthetics and Emotions in Images: A Computational Perspective,'' IEEE Signal Processing Magazine, vol. 28, no. 5, pp. 94-115, September 2011. (download) (g-scholar)
  28. Razvan Orendovici and James Z. Wang, ``Training Data Collection System for a Learning-based Photographic Aesthetic Quality Inference Engine,'' Proceedings of the ACM Multimedia Conference, Demonstration, pp. 1575-1578, Florence, Italy, ACM, October 2010. (download) (g-scholar)
  29. Ritendra Datta and James Z. Wang, ``ACQUINE: Aesthetic Quality Inference Engine - Real-time Automatic Rating of Photo Aesthetics,'' Proceedings of the ACM International Conference on Multimedia Information Retrieval, Demonstration, pp. 421-424, Philadelphia, Pennsylvania, ACM, March 2010. (download) (g-scholar)
  30. Ritendra Datta, Jia Li and James Z. Wang, ``Algorithmic Inferencing of Aesthetics and Emotion in Natural Images: An Exposition,'' Proceedings of the IEEE International Conference on Image Processing (ICIP), Special Session on Image Aesthetics, Mood and Emotion, pp. 105-108, San Diego, California, IEEE, October 2008. [invited but peer-reviewed] (download) (g-scholar)
  31. Ritendra Datta, Jia Li and James Z. Wang, ``Learning the Consensus on Visual Quality for Next-Generation Image Management,'' Proceedings of the ACM Multimedia Conference, pp. 533-536, ACM, Augsburg, Germany, September 2007. (download) (g-scholar)
  32. Ritendra Datta, Dhiraj Joshi, Jia Li and James Z. Wang, ``Studying Aesthetics in Photographic Images Using a Computational Approach,'' Lecture Notes in Computer Science, vol. 3953, Proceedings of the European Conference on Computer Vision, Part III, pp. 288-301, Graz, Austria, Springer, May 2006. (download) (g-scholar)

In the midst of difficulty lies opportunity. - Albert Einstein

COPYRIGHT © 2000- James Z. Wang Research Group, Penn State, University Park .