Title : Detecting leadership stress in virtual reality: A multimodal analysis of speech emotion and heart rate variability

Background: Stressful interpersonal interactions can challenge emotional regulation and may contribute to destructive leadership behaviours. However, the physiological and behavioural processes associated with leaders’ responses to such interactions remain insufficiently understood. This study investigated whether combining speech emotion recognition (SER) and heart rate variability (HRV) could help characterise stress-related responses during immersive virtual reality (VR) leadership scenarios.

Methods: Twenty-nine participants completed a within-subject experimental study involving VR-based interactions with virtual followers exhibiting escalating or de-escalating behaviour. Cardiac autonomic activity was assessed using the natural logarithm of the root mean square of successive differences (lnRMSSD), the standard deviation of NN intervals (SDNN), and the low-frequency/high-frequency power ratio (LF/HF). Participants’ speech was analysed using pretrained Wav2Vec2-based models to estimate discrete emotions and continuous emotional dimensions, including valence, arousal, and dominance. Condition-related differences and associations between speech and physiological measures were examined.

Results: Escalating interactions were associated with a trend towards lower speech valence (β = −0.079, p = .060; Cohen’s d = −0.439), although the condition effect did not reach conventional statistical significance. No clear condition-related effects were observed for arousal, dominance, or the HRV measures. Associations between speech and physiological measures were generally weak; however, lower lnRMSSD was significantly associated with greater sadness in speech (Spearman’s ρ = −0.315, Holm-adjusted p = .01).

Conclusion: Speech valence showed the clearest condition-related pattern among the measures examined, whereas HRV effects were small or inconsistent. These exploratory findings demonstrate the feasibility of combining wearable physiological monitoring, automated speech emotion analysis, and immersive VR to investigate stress-related responses during social interaction. Larger studies and further validation of speech-emotion models are needed to establish the reliability and generalisability of these measures.

Keywords: Heart rate variability; speech emotion recognition; virtual reality; psychophysiology; emotional valence; stress; multimodal neuroscience.

Bisma Faisal holds an MSc in Neuroengineering from the Technical University of Munich (TUM), Germany, and a Master’s degree in Biomedical Engineering from NED University of Engineering and Technology, Pakistan. Her research interests include cognitive neuroscience, neurophysiology, psychophysiology, and multimodal analysis of human behaviour. Her research experience includes EEG-based cognitive neuroscience at the National Centre of Artificial Intelligence (NCAI), Pakistan, and interdisciplinary research at TUM. Her Master’s thesis investigated stress responses during virtual reality interactions by combining speech emotion recognition with heart rate variability analysis, exploring the potential of multimodal approaches to understand emotional and physiological responses to challenging social situations.