Affective Computing and Multimodal Systems Lab (ACMS)

With vast amounts of data being generated on a daily basis, advanced data analytics techniques are essential to understand and learn from this data. Research scientists relied mostly on a specific attribute of the data of interest to make inferences. However, most of the generated data is multimodal in nature. The ACMS lab focuses on human-centered modeling in particular. In order to improve different aspects of our life and develop enhanced, personalized technologies, we need to understand human behavior. However, human behaviors are complex, vary significantly, and are difficult to model and predict.

In our lab, we are developing multimodal frameworks that allow us to capture multiple diverse signals that are reflective of human behaviors, thereby enabling us to understand several human-centric phenomena such as deception, stress, discomfort, alertness, and affect. In order to build reliable systems, the modeling process encompasses multimodal inputs covering vision, language, physiological signals, and thermal maps in addition to demographic information, and aims at analyzing how these features interact, complement each other, and integrate to train our behavior detection systems. Our research is funded by Ford Motor Company (Ford), Educational Testing Service (ETS), Toyota Research institute (TRI), and Procter & Gamble (P&G).

Current Research Projects

Circadian Rhythm Modeling

To be able to develop smart vehicles that are capable of understanding their occupants’ states, assessing their general health and well-being, and adjusting the settings accordingly, we model the circadian rhythm, behaviors, and traits of each person.

1. K. Riani, S. Sharak, M. Abouelenien, M. Burzo, R. Mihalcea, J. Elson, C. Maranville, K. Prakah-Asante, and W. Manzoor, “Non-Contact Based Modeling of Enervation,” in IEEE 17th International Conference on Automatic Face and Gesture Recognition, FG, USA, January 2023. [PDF]
   
2. K. Riani, S. Sharak, K. Das, M. Abouelenien, M. Burzo, R. Mihalcea, J. Elson, C. Maranville, K. Prakah-Asante, and W. Manzoor, “Towards Classifying Human Circadian Rhythm Using Multiple Modalities” in Proceedings of the 2021 9th International Conference on Affective Computing and Intelligent Interaction, ACII, October 2021. [PDF]
   

Non-contact Extraction of Physiological Signals

This project explores reliable supervised and unsupervised methods for extracting physiological signals using thermal imaging. These signals provide a reliable method to identify the physical and mental state of a person at any given point in time without the need for contact-based sensors.

1. M. Lilley, K. Das, K. Riani and M. Abouelenien, "A Topological Approach for Facial Region Seg-mentation in Thermal Images," in IEEE International Symposium on Multimedia, ISM, Italy, December 2022. [PDF]
   
2. C. Hessler, M. Abouelenien, and M. Burzo “A Non-contact Method for Extracting Heart and Respiration Rates,” in the 2020 17th Conference on Computer and Robot Vision (CRV), IEEE, Ottawa, ON, Canada, 2020. [PDF]
   
3. C. Hessler, M. Abouelenien, and M. Burzo. “A Survey on Extracting Physiological Measurements from Thermal Images”, in the 10th ACM Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2018, Corfu, Greece, June 2018. [PDF]
   
4. C. Hessler and M. Abouelenien. “Using Thermal Images and Physiological Features to Model Human Behavior: A Survey”, in Multimedia Pragmatics, IEEE International Conference on Multimedia Information Processing and Retrieval, MIPR 2018, Miami, Florida, April 2018. [PDF]
   
5. M. Abouelenien, V. Perez-Rosas, R. Mihalcea, and M. Burzo. “Multimodal Gender Detection”, in Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI 2017, Glasgow, Scotland, November 2017. [PDF]
   

Multimodal Deception Detection

The goal of this research is to explore a new generation of computational tools for joint modeling of physiological and linguistic signals of human behaviour. Our work is the first to investigate physio-linguistic models for deception analysis.

1. U. M. Sen, V. Perez-Rosas, B. Yanikoglu, M. Abouelenien, M. Burzo and R. Mihalcea, "Multimodal Deception Detection using Real-Life Trial Data," in IEEE Transactions on Affective Computing, vol. 13, no. 1, pp. 306-319, 2022. [PDF]
   
2. M. Kamboj, C. Hessler, P. Asnani, K. Riani and M. Abouelenien, "Multimodal Political Deception Detection," in IEEE MultiMedia, vol. 28, no. 1, pp. 94-102, 2021. [PDF]
   
3. M. Abouelenien, M. Burzo, V. Pérez-Rosas, R. Mihalcea, H. Sun and B. Zhao. "Gender Differences in Multimodal Contact-Free Deception Detection". IEEE MultiMedia, Vol. 26, no. 3, pages 19-30, 2019. [PDF]
   
4. V. Perez-Rosas, Q. Davenport, A. M. Dai, M. Abouelenien, and R. Mihalcea. “Identity Deception Detection”, in Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, IJCNLP 2017, Taipei, Taiwan, November 2017. [PDF]
   
5. M. Abouelenien, V. Perez-Rosas, R. Mihalcea, and M. Burzo. "Detecting Deceptive Behavior Via Integration of Discriminative Features from Multiple Modalities". IEEE Transactions on Information Forensics and Security, Vol. 12, no. 5, pages 1042-1055, 2017. [PDF]
   
6. M. Abouelenien, V. Perez-Rosas, B. Zhao, R. Mihalcea, and M. Burzo. “Gender-based Multimodal Deception Detection” in the 32nd ACM Symposium on Applied Computing, SAC 2017, Marrakesh, Morocco, April 2017. [PDF]
   
7. M. Abouelenien, R. Mihalcea, M. Burzo. "Analyzing Thermal and Visual Clues of Deception for a Non-Contact Deception Detection Approach", in Proceedings of the 9th ACM Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2016, Corfu, Greece, July 2016. [PDF]
   
8. M. Abouelenien, R. Mihalcea, and M. Burzo. "Trimodal analysis of deceptive behavior", in Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection, WMDD 2015, pages 9-13, Seattle, WA, USA, November, 2015. [PDF]
   
9. V. Perez-Rosas, M. Abouelenien, R. Mihalcea, and M. Burzo. "Deception detection using real-life trial data", in Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI 2015, pages 59-66, Seattle, WA, USA, November 2015. [PDF]
   
10. V. Perez-Rosas, M. Abouelenien, R. Mihalcea, Y. Xiao, C.J. Linton, and M. Burzo. "Verbal and nonverbal clues for real-life deception detection", in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, pages 2336-2346, September 2015. [PDF]
    
11. M. Abouelenien, V. Perez-Rosas, R. Mihalcea, and M. Burzo. "Deception Detection Using a Multimodal Approach" 16th ACM International Conference on Multimodal Interaction, ICMI 2014, Istanbul, Turkey, November 12-16, 2014.
    

Alertness Detection for Drivers' Safety Applications

Drowsy and distracted driving have a strong influence on the road traffic safety. Relying on improvements of sensorial technologies, a multimodal approach can provide features that can be more effective in detecting the level of alertness of the drivers.

1. K. Das, M. Papakostas, Kais Riani, A. Gasiorowski, M. Abouelenien, M. Burzo, and R. Mihalcea, “Detection and Recognition of Driver Distraction Using Multimodal Signals” in ACM Transaction on Interactive Intelligent Systems, vol. 12, no. 4, December 2022. [PDF]
   
2. S. Sharak, K. Das, K. Riani, M. Abouelenien, M. Burzo, R. Mihalcea, “Contact Versus Noncontact Detection of Driver's Drowsiness” in the 26th International Conference on Pattern Recognition, ICPR, Canada, August 2022. [PDF]
   
3. K. Das, M. Abouelenien, M. Burzo, and R. Mihalcea, “Towards Imbalanced Multiclass Driver Distraction Identification” in Proceedings of the 2022 35th International FLAIRS Conference, FLAIRS, May 2022. [PDF]
   
4. M. Papakostas, K. Das, M. Abouelenien, R. Mihalcea, M. Burzo, “Distracted and Drowsy Driving Modeling Using Deep Physiological Representations and Multitask Learning” in Applied Sciences, 11(1):88, 2021. [PDF]
   
5. K. Das, S. Sharak, K. Riani, M. Abouelenien, M. Burzo, and M. Papakostas, “Multimodal Detection of Drivers Drowsiness and Distraction” in Proceedings of the 2021 International Conference on Multimodal Interaction, ICMI, October 2021. [PDF]
   
6. M. Papakostas, K. Riani, A. Gasiorowski, Y. Sun, M. Abouelenien, R. Mihalcea, and M. Burzo “Understanding Driving Distractions: A Multimodal Analysis on Distraction Characterization”, in the 26th International Conference on Intelligent User Interfaces, ACM IUI, April 2021. [PDF] (Honorable mention for outstanding paper)
   
7. K. Riani, M. Papakostas, H. Kokash, M. Abouelenien, M. Burzo, and R. Mihalcea, “Towards Detecting Levels of Alertness in Drivers Using Multiple Modalities,” in the ACM Conference on PErvasive Technologies Related to Assistive Environments, Greece, PETRA 2020. [PDF]
   
8. M. Abouelenien, M. Burzo, and R. Mihalcea. "Cascaded Multimodal Analysis of Alertness Related Features for Drivers Safety Applications" The 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2015, Corfu, Greece, July 1-3, 2015. [PDF]
   

Modeling of Affective States

Seeking to understand how consumers' behaviors connect to their feelings and actions towards products, this work aims to develop multimodal methods to capture the consumer’s affective responses.

1. Y. Yao, V. Perez-Rosas, M. Abouelenien, and M. Burzo. “MORSE: MultimOdal sentiment analysis for Real-life Settings”, in Proceedings of the 2020 International Conference on Multimodal Interaction, ICMI 2020, Netherlands, October 2020. [PDF]
   

Multimodal Stress Detection

The capability to automatically detect human stress can benefit artificial intelligent agents involved in affective computing and human-computer interaction. This work aims to understand and analyze the multimodal nature of stress to aid in reducing its negative consequences.

1. Y. Yao, M. Papakostas, M. Burzo, M. Abouelenien, and R. Mihalcea, “MUSER: MUltimodal Stress detection using Emotion Recognition as an Auxiliary Task” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL, June 2021. [PDF]
   
2. M. Abouelenien, M. Burzo, R. Mihalcea. "Human Acute Stress Detection via Integration of Physiological Signals and Thermal Imaging", in Proceedings of the 9th ACM Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2016, Corfu, Greece, July 2016. [PDF] (best paper award)
   

Automated Scoring of Students' Progression

In this work, we develop an automated approach that relies on image processing to detect and classify shapes in visual models drawn by students, extract relevant features, and, ultimately, assign a learning progression score to each model in order to guide students in their learning pathways in science education.

1. A. Sagherian, S. Lingaiah, M. Abouelenien, C. Leong, L. Liu, M. Zhao, B. Lafuente, S. Chen, and Y. Qi, “Learning Progression-based Automated Scoring of Visual Models” in ACM Conference on PErvasive Technologies Related to Assistive Environments, PETRA, Greece, June 2022. [PDF]
   

Multimodal Detection of Thermal Discomfort

This research lays the grounds for a new methodology for detecting thermal discomfort by integrating physiological and thermal modalities, which can potentially reduce the energy consumption of buildings and vehicles while improving the comfort of their occupants.

1. M. Burzo, R. Mihalcea, and M. Abouelenien. “Multimodal Sensing of Thermal Comfort for Adaptable Climate Control.” US Patent 11631259, April 2023.
   
2. M. Abouelenien and M. Burzo, "Detecting Thermal Discomfort of Drivers Using Physiological Sen-sors and Thermal Imaging," in IEEE Intelligent Systems, vol. 34, no. 5, pp. 3-13, 2019. [PDF]
   
3. M. Burzo, M. Abouelenien, D. Van Alstine, and K. Rusinek. “Thermal Discomfort Detection Using Thermal Imaging”, in ASME 2017 International Mechanical Engineering Congress and Exposition, American Society of Mechanical Engineers, Tampa, Florida, November 2017. [PDF]
   
4. M. Abouelenien, M. Burzo, R. Mihalcea, K. Rusinek, and D. Van Alstine. “Detecting Human thermal Discomfort via Physiological Signals”, in Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2017, Rhodes, Greece, June 2017. [PDF]
   
5. M. Burzo, M. Abouelenien, V. Perez-Rosas, C. Wicaksono, Y. Tao, and R. Mihalcea "Using Infrared Thermography And Biosensors To Detect Thermal Discomfort In A Buildings Inhabitants" ASME 2014 International Mechanical Engineering Congress & Exposition, Montreal, Canada, November 14-20, 2014.
   
6. M. Burzo, C. Wicaksono, M. Abouelenien, V. Perez-Rosas, R. Mihalcea, and Y. Tao. "Multi-modal Sensing of Thermal Discomfort for Adaptive Energy Saving in Buildings" iiSBE Net Zero Built Environment 2014 Symposium, Florida, USA, March 7-8, 2014.
   

Previous Research Projects

Ensemble Learning from Large Imbalanced Datasets

In this project, we proposed a novel ensemble learning approach that combines stratified downsampling strategies with a new loss function that focuses on repeatedly misclassified instances in order to reduce the error and enable efficacy and efficiency of ensemble learning with large and imbalanced data.

1. X. Yuan, L. Xie, and M. Abouelenien. "A Regularized Ensemble Framework of Deep Learning for Cancer Detection from Multi-class, Imbalanced Training Data". Pattern Recognition, Vol. 77, pages 160–172, 2018.
   
2. X. Yuan, M. Abouelenien, and M. Elhoseny "A Boosting-based Decision Fusion Method for Learning from Large, Imbalanced Face Data Set", in Quantum Computing: An Environment for Intelligent Large Scale Real Application, Springer, 2018.
   
3. X. Yuan, M. Abouelenien. "A Multi-class Boosting Method for Learning from Imbalanced Data". International Journal of Granular Computing, Rough Sets and Intelligent Systems, Vol. 4, No. 1, 2015. [PDF]
   
4. M. Abouelenien and X. Yuan. "Boosting for Learning from Multiclass Data Sets via a Regularized Loss Function". IEEE International Conference on Granular Computing, GRC 2013, Beijing, China, December 13-15, 2013.
   
5. M. Abouelenien and X. Yuan. "Study on Parameter Selection using SampleBoost". The AAAI 26th International FLAIRS Conference, St. Pete Beach, Florida, USA, May 22-24, 2013.
   
6. M. Abouelenien and X. Yuan. "Incremental SampleBoost for Efficient Learning from Multi-class Data Sets" Data Mining for Medicine and Healthcare in SIAM - International Conference on Data Mining, Austin, Texas, USA, May 2-4, 2013. [PDF]
   
7. X. Yuan and M. Abouelenien. "A Boosting Method for Learning from Imbalanced Data toward Improved Face Recognition" In Proceedings of 11th International Conference on Machine Learning and Applications ICMLA 2012, Workshop on Class Imbalances: Past, Present, Future, Boca Raton, Florida, December 12-15, 2012.
   
8. M. Abouelenien and X. Yuan "SampleBoost for Capsule Endoscopy Categorization and Abnormality Detection", Springer - Communications in Computer and Information Science, CCIS, International Conference on Advanced Machine Learning Technologies and Applications, Cairo, Egypt, December 8-10, 2012.
   
9. M. Abouelenien and X. Yuan "Improved Action Recognition Using an Efficient Boosting Method", Springer - Communications in Computer and Information Science, CCIS, International Conference on Advanced Machine Learning Technologies and Applications, Cairo, Egypt, December 8-10, 2012.
   
10. M. Abouelenien and X. Yuan "SampleBoost: Improving Boosting Performance by Destabilizing Weak Learners Based on Weighted Error Analysis" In Proceedings of 21st International Conference on Pattern Recognition ICPR 2012, Tsukuba, Japan, November 11-15, 2012.
    
11. M. Abouelenien, X. Yuan, P. Duraisamy, and X. Yuan "Improving Classification Performance for the Minority Class in Highly Imbalanced Dataset using Boosting". In Proceedings of IEEE International Conference on Computing, Communication, and Networking Technologies, ICCCNT 2012, Coimbatore, India, July 26-28, 2012.