Affective Computing / Emotion Modeling

The group has been working on affective computing since 2005. They pioneered in computational modeling of visual aesthetics, predicting evoked emotions from visual content, and automated recognition of bodily expression of emotion, all for in-the-wild situations. We have also worked on other topics such as microexpression analysis, photo composition analysis, and computational psychiatry.

Our ongoing research projects include an NSF-funded data infrastructure project and Amazon Research Awards gift-funded project.

The following is an archive of our publications in this area. They are in reverse chronical order. You can also check out information on a past NSF project.

  1. Chenyan Wu, Yandong Li, Xianfeng Tang, Zhe Huang and James Z. Wang, ``MUG: Multi-human Graph Network for 3D Mesh Reconstruction from 2D Pose,'' International Journal of Computer Vision, vol. 134, no. 8, article 134, pp. 1-21, 2026. [A version was posted in May 2022 at https://arxiv.org/abs/2205.12583] (download) (g-scholar)
  2. Damith C. Herath, Amy LaViers, Sitao Zhang, Nipuni Hansika Wijesinghe, Sharni Konrad, Stelarc, Janie Busby Grant and James Z. Wang, ``The Body in Affective Robotics: A Survey and Conceptual Positioning Using the Performing Arts as a Scaffold for Understanding Bodily Expressed Emotion,'' IEEE Transactions on Affective Computing, vol. 17, no. 2, pp. -, 2026. (download) (g-scholar)
  3. Sree Bhattacharyya, Samarth Khanna, Leona Chen, Lucas Craig, Tharun Dilliraj and James Z. Wang, ``Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs,'' Advances in Neural Information Processing Systems, vol. , pp. -, Sydney, Australia, December 2026, under review. (download) (g-scholar)
  4. Sitao Zhang, Yimu Pan and James Z. Wang, ``TRACER: End-to-End Context-Aware Emotion Recognition with Transformers,'' Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. -, Puebla, Mexico, September 2026. (download) (g-scholar)
  5. Sree Bhattacharyya, Yaman Kumar Singla, Sudhir Yarram, Somesh Kumar Singh, Harini S I and James Z. Wang, ``Unsupervised Memorability Modeling from Tip-of-the-Tongue Retrieval Queries,'' Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. -, Tucson, Arizona, March 2026. (download) (g-scholar)
  6. Sree Bhattacharyya and James Z. Wang, ``Evaluating Vision-Language Models for Emotion Recognition,'' Findings of the Association for Computational Linguistics: Nations of the Americas Chapter of the Association for Computational Linguistics, pp. 1798-1820, Albuquerque, New Mexico, April to May 2025. (download) (g-scholar)
  7. Benjamin Wortman and James Z. Wang, ``HICEM: A High-Coverage Emotion Model for Artificial Emotional Intelligence,'' IEEE Transactions on Affective Computing, vol. 15, no. 3, pp. 1136-1152, 2024. [A version was posted in June 2022 at https://arxiv.org/abs/2206.07593] (download) (g-scholar)
  8. Sree Bhattacharyya, Shuhua Yang and James Z. Wang, ``A Heterogeneous Multimodal Graph Learning Framework for Recognizing User Emotions in Social Networks,'' Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 327-336, Glasgow, U.K., September 2024. (download) (g-scholar)
  9. Amy LaViers, Cat Maguire, James Z. Wang and Rachelle Tsachor, ``Tutorial on Movement Notation: An Interdisciplinary Methodology for HRI to Reveal the Bodily Expression of Human Counterparts via Collecting Annotations from Dancers in a Shared Data Repository,'' Companion of the Annual ACM/IEEE International Conference on Human Robot Interaction, pp. 1332-1134, 2024. (download) (g-scholar)
  10. Feng Xu, Yifan Yuan, Junping Zhang and James Z. Wang, ``High-Speed Joint Learning of Action Units and Facial Expressions,'' Modeling Visual Aesthetics, Emotion, and Artistic Style, James Z. Wang and Reginald B. Adams, Jr. (eds), Springer, Chapter 7, pp. 105-126, 2024. (download) (g-scholar)
  11. Feng Xu, Yifan Yuan, Junping Zhang and James Z. Wang, ``ExpressionFlow: A Microexpression Descriptor for Efficient Recognition,'' Modeling Visual Aesthetics, Emotion, and Artistic Style, James Z. Wang and Reginald B. Adams, Jr. (eds), Springer, Chapter 8, pp. 127-145, 2024. (download) (g-scholar)
  12. Baris Kandemir, Hanjoo Kim, Michelle G. Newman, Reginald B. Adams, Jr., Jia Li and James Z. Wang, ``Demographic Differences and Biases in Affect Evoked by Visual Features,'' Modeling Visual Aesthetics, Emotion, and Artistic Style, James Z. Wang and Reginald B. Adams, Jr. (eds), Springer, Chapter 19, pp. 331-347, 2024. (download) (g-scholar)
  13. James Z. Wang and Reginald B. Adams, Jr., editors, Modeling Visual Aesthetics, Emotion, and Artistic Style, Springer, Cham, Switzerland, 400 pages, ISBN-10: 303150268X, February 2024. (download) (g-scholar)
  14. James Z. Wang, Sicheng Zhao, Chenyan Wu, Reginald B. Adams, Jr., Michelle G. Newman, Tal Shafir and Rachelle Tsachor, ``Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion,'' Proceedings of the IEEE, vol. 111, no. 10, pp. 1236-1286, 2023. (download) (g-scholar)
  15. Chenyan Wu, Dolzodmaa Davaasuren, Tal Shafir, Rachelle Tsachor and James Z. Wang, ``Bodily Expressed Emotion Understanding Through Integrating Laban Movement Analysis,'' Patterns, vol. 4, no. 10, article 100816, pp. 1-14, 2023. Featured cover article. [A version was posted in April 2023 at https://arxiv.org/abs/2304.02187] (download) (g-scholar)
  16. Sitao Zhang, Yimu Pan and James Z. Wang, ``Learning Emotion Representations from Verbal and Nonverbal Communication,'' Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 18993-19004, Vancouver, Canada, June 2023. (download) (g-scholar)
  17. Farshid Farhat, Mohammad M. Kamani and James Z. Wang, ``CAPTAIN: Comprehensive Composition Assistance for Photo Taking,'' ACM Transactions on Multimedia Computing, Communications and Applications, vol. 18, no. 1, article 14, pp. 14:1-24, 2022. (download) (g-scholar)
  18. Yu Luo, Jianbo Ye, Reginald B. Adams, Jr., Jia Li, Michelle G. Newman and James Z. Wang, ``ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wild,'' International Journal of Computer Vision, vol. 128, no. 1, pp. 1-25, 2020. [A version was posted in August 2018 at http://arxiv.org/abs/1808.09568] (download) (g-scholar)
  19. James Z. Wang, ``Modeling Aesthetics and Emotions in Visual Content: From Vincent van Gogh to Robotics and Vision,'' Proceedings of the Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends, in conjunction with the ACM International Conference on Multimedia, pp. 15-16, Virtual, October 2020. (download) (g-scholar)
  20. James Z. Wang, Norman Badler, Nadia Berthouze, Rick O. Gilmore, Kerri L. Johnson, Agata Lapedriza, Xin Lu and Nikolaus Troje, ``Bodily Expressed Emotion Understanding Research: A Multidisciplinary Perspective,'' Proceedings of the First International Workshop on Bodily Expressed Emotion Understanding, in conjunction with the European Computer Vision Conference, Lecture Notes in Computer Science, vol. 12535, A. Bartoli and A. Fusiello (eds.), pp. 733-746, Virtual, August 2020. (download) (g-scholar)
  21. Jianbo Ye, Jia Li, Michelle G. Newman, Reginald B. Adams, Jr. and James Z. Wang, ``Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data,'' IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 115-128, 2019. [A version was posted in January 2017 at http://arxiv.org/abs/1701.01096] (download) (g-scholar)
  22. Hanjoo Kim, Xin Lu, Michael Costa, Baris Kandemir, Reginald B. Adams, Jr., Jia Li, James Z. Wang and Michelle G. Newman, ``Development and Validation of Image Stimuli for Emotion Elicitation (ISEE): A Novel Affective Pictorial System with Test-Retest Repeatability,'' Psychiatry Research, vol. 261, pp. 414-420, Elsevier, 2018. (download) (g-scholar)
  23. Siqiong He, Zihan Zhou, Farshid Farhat and James Z. Wang, ``Discovering Triangles in Portraits for Supporting Photographic Creation,'' IEEE Transactions on Multimedia, vol. 20, no. 2, 496-508, 2018. [A version was posted in May 2016 at http://arxiv.org/abs/1605.09559] (download) (g-scholar)
  24. Feng Xu, Junping Zhang and James Z. Wang, ``Microexpression Identification and Categorization using a Facial Dynamics Map,'' IEEE Transactions on Affective Computing, vol. 8, no. 2, pp. 254-267, 2017. (download) (g-scholar)
  25. Zihan Zhou, Farshid Farhat and James Z. Wang, ``Detecting Dominant Vanishing Points in Natural Scenes with Application to Composition-Sensitive Image Retrieval,'' IEEE Transactions on Multimedia, vol. 19, no. 12, pp. 2651-2665, 2017. (download) (g-scholar)
  26. Xin Lu, Reginald B. Adams, Jr., Jia Li, Michelle G. Newman and James Z. Wang, ``An Investigation into Three Visual Characteristics of Complex Scenes that Evoke Human Emotion,'' Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 440-447, San Antonio, Texas, October 2017. (download) (g-scholar)
  27. Baris Kandemir, Zihan Zhou, Jia Li and James Z. Wang, ``Beyond Saliency: Assessing Visual Balance with High-level Cues,'' Proceedings of the Thematic Workshops of ACM Multimedia, in conjunction with the ACM Multimedia Conference, pp. 26-34, Mountain View, California, October 2017. (download) (g-scholar)
  28. Farshid Farhat, Mohammad M. Kamani, Sahil Mishra and James Z. Wang, ``Intelligent Portrait Composition Assistance - Integrating Deep-learned Models and Photography Idea Retrieval,'' Proceedings of the Thematic Workshops of ACM Multimedia, in conjunction with the ACM Multimedia Conference, pp. 17-25, Mountain View, California, October 2017. (download) (g-scholar)
  29. Ye Zhou, Xin Lu, Junping Zhang and James Z. Wang, ``Joint Image and Text Representation for Aesthetics Analysis,'' Proceedings of the ACM Multimedia Conference, pp. 262-266, Amsterdam, The Netherlands, ACM, October 2016. (download) (g-scholar)
  30. Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang and James Z. Wang, ``Rating Image Aesthetics using Deep Learning,'' IEEE Transactions on Multimedia, vol. 17, no. 11, pp, 2021-2034, 2015. (download) (g-scholar)
  31. Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech and James Z. Wang, ``Deep Multi-Patch Aggregation Network for Image Style, Aesthetics, and Quality Estimation,'' Proceedings of the International Conference on Computer Vision, pp. 990-998, Santiago, Chile, IEEE, 2015. (download) (g-scholar)
  32. Zihan Zhou, Siqiong He, Jia Li and James Z. Wang, ``Modeling Perspective Effects in Photographic Composition,'' Proceedings of the ACM Multimedia Conference, pp. 301-310, Brisbane, Australia, ACM, October 2015. (download) (g-scholar)
  33. Jia Li, Lei Yao and James Z. Wang, ``Photo Composition Feedback and Enhancement --- Exploiting Spatial Design Categories and the Notan Dark-Light Principle,'' Mobile Cloud Visual Media Computing, G. Hua and X.-S. Hua (eds.), Springer Int. Publ. Switzerland, Chapter 5, pp. 113-144, 2015. (download) (g-scholar)
  34. Hanjoo Kim, Xin Lu, Michael Costa, Baris Kandemir, Reginald B. Adams, Jr., Jia Li, James Z. Wang and Michelle G. Newman, ``Development and Validation of the Image Stimuli for Emotion Elicitation (ISEE),'' Association for Psychological Science Annual Convention, poster, New York City, May 2015. (download) (g-scholar)
  35. Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang and James Z. Wang, ``RAPID: Rating Pictorial Aesthetics using Deep Learning,'' Proceedings of the ACM Multimedia Conference, pp. 457-466, Orlando, Florida, ACM, November 2014. (download) (g-scholar)
  36. Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Xin Lu, Quang-Tuan Luong, James Z. Wang, Jia Li and Jiebo Luo, ``On Aesthetics and Emotions in Scene Images: A Computational Perspective,'' Scene Vision: Making Sense of What We See, K. Kveraga and M. Bar (eds.), MIT Press, pp. 241-272, Chapter 12, November 2014. (download) (g-scholar)
  37. Lei Yao, Poonam Suryanarayan, Mu Qiao, James Z. Wang and Jia Li, ``OSCAR: On-Site Composition and Aesthetics Feedback through Exemplars for Photographers,'' International Journal of Computer Vision, vol. 96, no. 3, pp. 353-383, 2012. [DOI: 10.1007/s11263-011-0478-3] (download) (g-scholar)
  38. Xin Lu, Poonam Suryanarayan, Reginald B. Adams, Jr., Jia Li, Michelle G. Newman and James Z. Wang, ``On Shape and the Computability of Emotions,'' Proceedings of the ACM Multimedia Conference, pp. 229-238, Nara, Japan, ACM, October 2012. (download) (g-scholar)
  39. Dhiraj Joshi, Ritendra Datta, Quang-Tuan Luong, Elena Fedorovskaya, James Z. Wang, Jia Li and Jiebo Luo, ``Aesthetics and Emotions in Images: A Computational Perspective,'' IEEE Signal Processing Magazine, vol. 28, no. 5, pp. 94-115, September 2011. (download) (g-scholar)
  40. Razvan Orendovici and James Z. Wang, ``Training Data Collection System for a Learning-based Photographic Aesthetic Quality Inference Engine,'' Proceedings of the ACM Multimedia Conference, Demonstration, pp. 1575-1578, Florence, Italy, ACM, October 2010. (download) (g-scholar)
  41. Ritendra Datta and James Z. Wang, ``ACQUINE: Aesthetic Quality Inference Engine - Real-time Automatic Rating of Photo Aesthetics,'' Proceedings of the ACM International Conference on Multimedia Information Retrieval, Demonstration, pp. 421-424, Philadelphia, Pennsylvania, ACM, March 2010. (download) (g-scholar)
  42. Ritendra Datta, Jia Li and James Z. Wang, ``Algorithmic Inferencing of Aesthetics and Emotion in Natural Images: An Exposition,'' Proceedings of the IEEE International Conference on Image Processing (ICIP), Special Session on Image Aesthetics, Mood and Emotion, pp. 105-108, San Diego, California, IEEE, October 2008. [invited but peer-reviewed] (download) (g-scholar)
  43. Ritendra Datta, Jia Li and James Z. Wang, ``Learning the Consensus on Visual Quality for Next-Generation Image Management,'' Proceedings of the ACM Multimedia Conference, pp. 533-536, ACM, Augsburg, Germany, September 2007. (download) (g-scholar)
  44. Ritendra Datta, Dhiraj Joshi, Jia Li and James Z. Wang, ``Studying Aesthetics in Photographic Images Using a Computational Approach,'' Lecture Notes in Computer Science, vol. 3953, Proceedings of the European Conference on Computer Vision, Part III, pp. 288-301, Graz, Austria, Springer, May 2006. (download) (g-scholar)