Physiological-based Dynamic Difficulty Adaptation in a Theragame for Children having Cerebral Palsy

Defended by: VERHULST Adrien
Internship co-ordinator: YAMAGUCHI Takehiko (TUS)
Internship supervisor: RICHARD Paul (UA)

This research have been done in the context of the Children Rehabilitation Project (CRP), for LARIS (UA), at Harada’s laboratory (TUS)

Abstract

The purpose of this research is to provide a physiological-based Dynamic Difficulty Adaptation (DDA) able to be used in a CVA rehabilitation application, and more especially in the Children Rehabilitation Project (CRP). The CRP is an adaptive modular interface destined to CVA’s rehabilitation for Children, and has for purpose to offer a state-of-the-art rehabilitation platform to improve CVA’s rehabilitation through rehabilitation applications.
During this research, we successfully managed to implement a physiological-based DDA (using Anxiety state and Boredom state) able to direct the user toward the Flow state. We still need to experiment it on children having CVA’s rehabilitation.

In this report, we go through all the steps of this research, by going through (1) the state of the art done on DDA and on physiological-based DDA, (2) the approach based on physiological signals, (3) the acquisition of physiological signals, (4) the extraction of the physiological signals’ features and (5) the training of a learning algorithm able to dynamically change the difficulty.

Keywords: Theragame, Cerebral Palsy, Rehabilitation, DDA, Children Rehabilitation Project.

One-page report

This one-page report just go through the important steps of the research. It has not for purpose of being exhaustive in any ways.

The approach

To do a physiological-based DDA we decided to focus on the emotional state of the user, and more especially on the Anxiety and Boredom emotional state. Through Anxiety / Boredom states (and the Flow theory) it is possible to know "where" the user is regarding the Excitement state. Our approach was to propose a DDA able to (1) recognize, in real-time, if the user was in Anxiety / Boredom; (2) adjust the difficulty to avoid Anxiety / Boredom and meet Excitement.

Table to know how to adjust the difficulty if anxiety / boredom
How to adjust the difficulty depending of the Anxiety / Boredom. The "Maybe flow" means that, if the difficulty is above a threshold value (15%), there is no need to change the difficulty because the user may be in flow.
example of difficulty adjustment over time
Every x seconds, Anxiety or Boredom are identified. The difficulty is then adjusted to decrease or increase over time, in order to meet excitement.

Physiological signals acquisition

The step #1 of our research was to capture physiological in real-time, in Unity3D. We decided to use the BITalino Board, and developed a C# SDK (.NET 2.0) to use BITalino in Unity3D. We successfully implemented it in Unity3D. We show bellow some of the physiological signals captured by our Unity3D scripts.

EDA data captured by our Unity3D scripts
EDA data.
ECG data captured by our Unity3D scripts
ECG data

Physiological signals' features extraction

The step #2 of our research was to extract the physiological signals' features, in real-time, in Unity3D. Based on our state-of-the-art, we developed several MATLAB scripts to extract EDA / ECG features from EDA / ECG physiological signals. We then exported the MATLAB scripts in a C-shared library and successfully implemented it in Unity3D. We show bellow a sample of the features.

Features extracted by our Unity3D scripts (from the MATLAB scripts)
14 EDA / ECG features.

Classification of labelled features in affect-state

The step #3 of our research was to (1) label the physiological signals; (2) train a learning classifier to classify non-labelled physiological signals. We captured the users' physiological signals when they were in the Anxiety or Boredom state (through ProvokeAffect). We then extracted the labelled physiological signals' features and used them to train a SVM to recognize Anxiety / Boredom state (we managed to have a success rate of 67%). We successfully implemented it in Unity3D.

Example of provokeAffect, the anxiety scene.
The ProvokeAffect application, in the Anxiety scene.
eda in Boredom scene
EDA captured in the Boredom scene of ProvokeAffect (60 seconds, raw data).
eda in Anxiety scene
EDA captured in the Anxiety scene of ProvokeAffect (60 seconds, raw data).
The SVM results in Unity3D
Every 40 sec, the SVM guess the class (1 for Boredom, 2 for Anxiety) in Unity3D. Here, it detect Anxiety.

Implementation in the CRP

The last step of our research was to develop a physiological-based DDA, and to implement it in the CRP. We decided to develop a simple DDA (which basically increases the difficulty when there is Boredom and decrease the difficulty when there is Anxiety) able to "move" the difficulty from 0% to 100%. When then simply linked the difficulty's variable of the CRP to the difficulty's value.
For example, if the user is anxious at a difficulty of 50%, the difficulty decreases from 50% to 40%, over 30 seconds. While the difficulty decreases the: (1) speed of the object decreases, (2) size of the object increase, (3) interval decreases, (4) distance increases.

The difficulty change over time
Change of the difficulty over time. When the learning classifier identify a state (every 30 seconds), the DDA adjust the difficulty. The difficulty then changes over 30 seconds.
the difficulty's parameters of the CRP after about 30 seconds
The difficulty's parameters of the CRP after about 30 seconds of decreasing difficulty.
the difficulty's parameters of the CRP after about 90 seconds
The difficulty's parameters of the CRP after about 90 seconds of decreasing difficulty.

We saw how we choose to implement our AFFECT-DDA into the CRP. We successfully managed to links all our steps in the application, and therefore to do our approach. However, in order to answer our research question (what is the incidence of a physiological-based DDA on a Children-with-Cerebral-Palsy rehabilitation application?) and to validate our approach it is necessary to test the system with Children with Cerebral Palsy, probably in the Hospital of Angers.

Annexes