Data Analytics and Multimodal Systems Lab (DAMS)
With vast amount of data being generated on daily basis, advanced data analytics techniques are essential to understand and learn from this data.
Research scientists relied mostly on a specific attribute of the data of interest to make inferences.
However, most of the generated data is multimodal in nature. DAMS lab focuses on human-centered modeling in particular.
In order to improve different aspects of our life and develop enhanced, personalized technologies, we need to understand human behavior.
However, human behaviors are complex, vary significantly, and are difficult to model and predict.
In our lab, we are developing multimodal frameworks that allow us to capture multiple diverse signals that are reflective of human behaviors, thereby enabling us to understand several human-centric phenomena such as deception, stress, discomfort, alertness, and affect.
In order to build reliable systems, the modeling process encompasses multimodal inputs covering vision, language, physiological signals, and thermal maps in addition to demographic information, and aims at analyzing how these features interact, complement each other, and integrate to train our behavior detection systems.
Current Research Projects
Non-contact Extraction of Physiological Signals
This project explores reliable methods for extracting physiological signals using thermal imaging. These signals provide a reliable method to identify the physical and mental state of a person at any given point in time.
- C. Hessler, M. Abouelenien, and M. Burzo “A Non-contact Method for Extracting Heart and Respiration Rates,” in the 2020 17th Conference on Computer and Robot Vision (CRV), IEEE, Ottawa, ON, Canada, 2020.
- C. Hessler, M. Abouelenien, and M. Burzo. “A Survey on Extracting Physiological Measurements from Thermal Images”, in the 10th ACM Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2018, Corfu, Greece, June 2018. [PDF]
- C. Hessler and M. Abouelenien. “Using Thermal Images and Physiological Features to Model Human Behavior: A Survey”, in Multimedia Pragmatics, IEEE International Conference on Multimedia Information Processing and Retrieval, MIPR 2018, Miami, Florida, April 2018. [PDF]
- M. Abouelenien, V. Perez-Rosas, R. Mihalcea, and M. Burzo. “Multimodal Gender Detection”, in Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI 2017, Glasgow, Scotland, November 2017. [PDF]
Multimodal Deception Detection
The goal of this research is to explore a new generation of computational tools for joint modeling of physiological and linguistic signals of human behaviour. The project is the first to investigate physio-linguistic models for deception analysis.
- U. M. Sen, V. Perez-Rosas, B. Yanikoglu, M. Abouelenien, M. Burzo and R. Mihalcea, "Multimodal Deception Detection using Real-Life Trial Data," in IEEE Transactions on Affective Computing, 2020. [PDF]
- M. Abouelenien, M. Burzo, V. Pérez-Rosas, R. Mihalcea, H. Sun and B. Zhao. "Gender Differences in Multimodal Contact-Free Deception Detection". IEEE MultiMedia, Vol. 26, no. 3, pages 19-30, 2019. [PDF]
- V. Perez-Rosas, Q. Davenport, A. M. Dai, M. Abouelenien, and R. Mihalcea. “Identity Deception Detection”, in Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, IJCNLP 2017, Taipei, Taiwan, November 2017. [PDF]
- M. Abouelenien, V. Perez-Rosas, R. Mihalcea, and M. Burzo. "Detecting Deceptive Behavior Via Integration of Discriminative Features from Multiple Modalities". IEEE Transactions on Information Forensics and Security, Vol. 12, no. 5, pages 1042-1055, 2017. [PDF]
- M. Abouelenien, V. Perez-Rosas, B. Zhao, R. Mihalcea, and M. Burzo. “Gender-based Multimodal Deception Detection” in the 32nd ACM Symposium on Applied Computing, SAC 2017, Marrakesh, Morocco, April 2017. [PDF]
- M. Abouelenien, R. Mihalcea, M. Burzo. "Analyzing Thermal and Visual Clues of Deception for a Non-Contact Deception Detection Approach", in Proceedings of the 9th ACM Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2016, Corfu, Greece, July 2016. [PDF]
- M. Abouelenien, R. Mihalcea, and M. Burzo. "Trimodal analysis of deceptive behavior", in Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection, WMDD 2015, pages 9-13, Seattle, WA, USA, November, 2015. [PDF]
- V. Perez-Rosas, M. Abouelenien, R. Mihalcea, and M. Burzo. "Deception detection using real-life trial data", in Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI 2015, pages 59-66, Seattle, WA, USA, November 2015. [PDF]
- V. Perez-Rosas, M. Abouelenien, R. Mihalcea, Y. Xiao, C.J. Linton, and M. Burzo. "Verbal and nonverbal clues for real-life deception detection", in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, pages 2336-2346, September 2015. [PDF]
- M. Abouelenien, V. Perez-Rosas, R. Mihalcea, and M. Burzo. "Deception Detection Using a Multimodal Approach" 16th ACM International Conference on Multimodal Interaction, ICMI 2014, Istanbul, Turkey, November 12-16, 2014.
Multimodal Detection of Thermal Discomfort
This research lays the grounds for a new methodology for detecting thermal discomfort by integrating physiological and thermal modalities, which can potentially reduce the energy consumption of buildings and vehicles while improving the comfort of their occupants.
- M. Burzo, R. Mihalcea, M. Abouelenien "Multimodal Sensing Of Thermal Comfort For Adaptable Climate Control," US Patent App. 16/671,841.
- M. Abouelenien and M. Burzo, "Detecting Thermal Discomfort of Drivers Using Physiological Sen-sors and Thermal Imaging," in IEEE Intelligent Systems, vol. 34, no. 5, pp. 3-13, 2019. [PDF]
- M. Burzo, M. Abouelenien, D. Van Alstine, and K. Rusinek. “Thermal Discomfort Detection Using Thermal Imaging”, in ASME 2017 International Mechanical Engineering Congress and Exposition, American Society of Mechanical Engineers, Tampa, Florida, November 2017. [PDF]
- M. Abouelenien, M. Burzo, R. Mihalcea, K. Rusinek, and D. Van Alstine. “Detecting Human thermal Discomfort via Physiological Signals”, in Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2017, Rhodes, Greece, June 2017. [PDF]
- M. Burzo, M. Abouelenien, V. Perez-Rosas, C. Wicaksono, Y. Tao, and R. Mihalcea "Using Infrared Thermography And Biosensors To Detect Thermal Discomfort In A Buildings Inhabitants" ASME 2014 International Mechanical Engineering Congress & Exposition, Montreal, Canada, November 14-20, 2014.
- M. Burzo, C. Wicaksono, M. Abouelenien, V. Perez-Rosas, R. Mihalcea, and Y. Tao. "Multi-modal Sensing of Thermal Discomfort for Adaptive Energy Saving in Buildings" iiSBE Net Zero Built Environment 2014 Symposium, Florida, USA, March 7-8, 2014.
Alertness Detection for Drivers' Safety Applications
Drowsy driving has a strong influence on the road traffic safety. Relying on improvements of sensorial technologies, a multimodal approach can provide features that can be more effective in detecting the level of alertness of the drivers.
- K. Riani, M. Papakostas, H. Kokash, M. Abouelenien, M. Burzo, and R. Mihalcea, “Towards Detecting Levels of Alertness in Drivers Using Multiple Modalities,” in the ACM Conference on PErvasive Technologies Related to Assistive Environments, Greece, PETRA 2020.
- M. Abouelenien, M. Burzo, and R. Mihalcea. "Cascaded Multimodal Analysis of Alertness Related Features for Drivers Safety Applications" The 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2015, Corfu, Greece, July 1-3, 2015. [PDF]
Previous Research Projects
Ensemble Learning from Large Imbalanced Datasets
In this project, we proposed a novel ensemble learning approach that combines stratified downsampling strategies with a new loss function that focuses on repeatedly misclassified instances in order to reduce the error and enable efficacy and efficiency of ensemble learning with large and imbalanced data.
- X. Yuan, L. Xie, and M. Abouelenien. "A Regularized Ensemble Framework of Deep Learning for Cancer Detection from Multi-class, Imbalanced Training Data". Pattern Recognition, Vol. 77, pages 160–172, 2018.
- X. Yuan, M. Abouelenien, and M. Elhoseny "A Boosting-based Decision Fusion Method for Learning from Large, Imbalanced Face Data Set", in Quantum Computing: An Environment for Intelligent Large Scale Real Application, Springer, 2018.
- X. Yuan, M. Abouelenien. "A Multi-class Boosting Method for Learning from Imbalanced Data". International Journal of Granular Computing, Rough Sets and Intelligent Systems, Vol. 4, No. 1, 2015. [PDF]
- M. Abouelenien and X. Yuan. "Boosting for Learning from Multiclass Data Sets via a Regularized Loss Function". IEEE International Conference on Granular Computing, GRC 2013, Beijing, China, December 13-15, 2013.
- M. Abouelenien and X. Yuan. "Study on Parameter Selection using SampleBoost". The AAAI 26th International FLAIRS Conference, St. Pete Beach, Florida, USA, May 22-24, 2013.
- M. Abouelenien and X. Yuan. "Incremental SampleBoost for Efficient Learning from Multi-class Data Sets" Data Mining for Medicine and Healthcare in SIAM - International Conference on Data Mining, Austin, Texas, USA, May 2-4, 2013. [PDF]
- X. Yuan and M. Abouelenien. "A Boosting Method for Learning from Imbalanced Data toward Improved Face Recognition" In Proceedings of 11th International Conference on Machine Learning and Applications ICMLA 2012, Workshop on Class Imbalances: Past, Present, Future, Boca Raton, Florida, December 12-15, 2012.
- M. Abouelenien and X. Yuan "SampleBoost for Capsule Endoscopy Categorization and Abnormality Detection", Springer - Communications in Computer and Information Science, CCIS, International Conference on Advanced Machine Learning Technologies and Applications, Cairo, Egypt, December 8-10, 2012.
- M. Abouelenien and X. Yuan "Improved Action Recognition Using an Efficient Boosting Method", Springer - Communications in Computer and Information Science, CCIS, International Conference on Advanced Machine Learning Technologies and Applications, Cairo, Egypt, December 8-10, 2012.
- M. Abouelenien and X. Yuan "SampleBoost: Improving Boosting Performance by Destabilizing Weak Learners Based on Weighted Error Analysis" In Proceedings of 21st International Conference on Pattern Recognition ICPR 2012, Tsukuba, Japan, November 11-15, 2012.
- M. Abouelenien, X. Yuan, P. Duraisamy, and X. Yuan "Improving Classification Performance for the Minority Class in Highly Imbalanced Dataset using Boosting". In Proceedings of IEEE International Conference on Computing, Communication, and Networking Technologies, ICCCNT 2012, Coimbatore, India, July 26-28, 2012.