Heart-Tune: Teaching Emotional Regulation and Awareness Through a VR-based Rhythm Game
My Master’s research project explored how virtual reality and music-based gaming can be used to teach social-emotional skills like self-awareness and self-management to teenagers. Using a design thinking methodology, we worked with educators in Colombia to define their needs when addressing these skills with teenagers, combined this with pertinent literature in emotional education methodologies, emotional design, and emerging technologies and developed the game HeartTune.
The game seeks to help students develop skills such as relaxation techniques, emotional expression, and emotional literacy through culturally-situated rhythm gaming and performance.
Master Research Project completed for the Master of Digital Media, Toronto Metropolitan University.
ROLE
Research, user experience and interaction design
RESEARCH PARTNER
Maria Camila Bohorquez
TOOLS
Unity, Photoshop, Illustrator,
YEAR
2020
Overview and Research Problem
Programs designed to promote social-emotional learning (SEL) such as identifying and managing emotions are increasingly being used in schools to facilitate academic and life success, with particular relevance to at-risk youth in challenging environments.
Through online interviews and surveys with key stakeholders at Coschool, an organization in Cali, Colombia that teaches Social-Emotional Learning (SEL) skills to at-risk youth, we learned these educators facilitate emotional regulation and awareness through role-play of challenging situations and games that lead to frustration, followed by group discussions on how to regulate it. They also draw on stories, anecdotes, videos, guided meditations, and exercises to identify and name emotions. They carry out reflections on body signals and responses to emotions, like breathing rhythms.
However, teachers perceive students struggle to focus and engage in passive activities like writing, long reflections, or meditations. They notice students don’t express their emotions freely mainly because they fear rejection and have trouble accepting all emotions without judging them, especially those seen as “negative”.
How can technology for emotional wellbeing and learning maximize current SEL methodologies to make it more appealing to tennage students?
Coschool’s students and teachers – coschool.com
Research
We learned SEL and emotion-regulation schemes based on music-listening and rhythmic-performance are particularly compelling programming for teens, due to the popularity of music among adolescents, the inherent emotional expressivity of the form, and our natural ability to connect emotions to music
Interactivity through embodied body movements in VR, tied to visual and audio feedback while engaging in rhythm gameplay, could help us build an immersive serious game to explain the qualities of different emotional experiences and their bodily reactions and sensations. The high level of immersion and sense of presence possible in VR and its effectiveness in generating intense emotions could promote the self-reflection and introspection needed for emotional awareness and regulation and help maintain the student’s attention.
Ideation and prototyping
We ideated a VR rhythm game where the user plays an instrument while attempting to hit the notes in synch with a song. The game has different songs that represent emotions like joy, sadness, anger and fear, changing the tempo according to each one. The 3D environment would react to the emotion being played by the user.
We designed an early prototype using storyboards to explain the concept to Coschool and our peers.
User playing a music track in the marimba representing joy
User playing a music track in the marimba representing sadness
Following their feedback and research on emotional regulation, we built a second prototype in Unity that better integrated known techniques for emotional regulation and identification.
In this prototype, as the difficulty in the music tempo and difficulty quickly escalates, both the music and environment aim to create a heightened emotional state representing anger. Dissonant noises in the background give a sense of unpleasantness and the colors in the environment change from a calm, cold tone to a bright and warm color. This emotional peak is followed by a prompt to practice deep breathing exercises while the tempo decreases, teaching the player how to calm the physiological arousal and regain emotional balance.
Ideation and prototyping
We ideated a VR rhythm game where the user plays an instrument while attempting to hit the notes in synch with a song. The game has different songs that represent emotions like joy, sadness, anger and fear, changing the tempo according to each one. The 3D environment would react to the emotion being played by the user.
We designed an early prototype using storyboards to explain the concept to Coschool and our peers.
User playing the drums in a low arousal state
User playing the drums in a high-arousal state
After testing this prototype with our peers, we developed a third prototype with an improved environment and bongos instead of drums to facilitate the interaction.
We also added a second part to the experience that allows the player to improvise and play the bongos to a particular emotion, facilitating further reflection and understanding of its qualities. Several secondary emotions that could describe the primary emotion more specifically appear in the environment, allowing the player to name the feeling experienced, a crucial process in self-awareness.
User playing the drums in a low arousal state
