user research
The Context
Brain injuries (BI) are recognized as a major public health concern by the Centers for Disease Control. There are multiple and varied repercussions of experiencing a BI, including lifelong physical and/or cognitive disabilities, shorter lifespan, and lower quality of life. It is estimated that 6.4 million children and adults in the US live with a lifelong disability as a result of a traumatic BI.
Many clinicians include games, both commercial (e.g., Nintendo® Wii™, Microsoft Xbox® with Kinect®) and specialized platforms (e.g. Jintronix), to help motivate people who have had a BI perform exercises similar to those prescribed for rehabilitation. Because of their affordability and availability, commercial games are a popular choice even though they are often considered too difficult for this audience.
The Challenge
While there are multiple commercial games that therapists rate as effective for physical goals, there are very few that addressed the Speech-Language Pathologists’ (SLP) cognitive-related goals. Particularly, SLPs have limited commercial game options to address their top therapy goals. Goals we specially considered in this study were: (1) attention/concentration, (2) short-term visual memory and (3) processing speed.
The Objective
The aim of this exploratory user research was to examine how a driving simulation game might provide Speech Language Pathologists (SLPs) with an additional evidence-based commercial game option when working with their patients who have had a brain injury (BI). Research has indicated that cognitive skills required for safe driving are aligned with top SLP goals.
The Approach
Participants considered for inclusion were identified as:
1. Capable of understanding informed consent
2. Having the ability to play the driving game and complete our assessments
3. Expressing interest in the study
4. Not having permission to drive a ‘real’ car during the study
Clients entered the New Focus program with varied needs and abilities. Therapists evaluated clients as they were admitted to the program and on a bi-yearly basis. This is the demographics of our seven participants including their cognitive scores:
Data collection occurred between February to September. We were on site Mondays and Wednesdays during the 12:30 - 1:15 pm and 1:30 - 2:15 pm sessions.To create an engaging driving experience, we set up two game carts with 50” televisions, racing seats that were designed for the Forza Xbox One game and racing wheels for Xbox One (by HORI) with a gas pedal.
Clients drove in ‘Free-play’ mode, which afforded multiple different racetracks of varying degrees of difficulty. Additionally, we took some additional steps to simplify gameplay: (a) we limited the number of competitors to five; (b) we set the competition to the easiest setting; and (c) we chose to have all of the assists selected that resulted in easier handling where the clients did not need to use the brake.
We used three interactive programs (in the same sequence) in the Brain Baseline app on iPad minis to assess our three dependent variables (attention/ concentration, visual working memory, and processing speed).
1) To assess processing speed, we used the ‘Speed’ program. The interactive program displayed a reddot on screen for about 5-seconds at random intervals. Users could press either the left or right button with the goal of pressing the buttons as soon as they saw the red dot.
2) To assess short-term visual memory, we used the ‘Visual Short Term Memory’ (VSTM) program. The program first flashed four square color blocks for about 2-seconds on screen and then displayed one color block indefinitely. The program required users to determine if one block matched any one of the four previously displayed blocks.
3) To assess concentration/attention, we used the Flanker program. Flanker flashed five arrows for about 2-seconds on screen and required users to identify the direction of the middle arrow.
Participants were assigned into four driving cohorts. Each cohort drove for three 2-week periods with six weeks off between driving periods. We assessed the dependent variables 12 times:
1) One week before each driving period (3 baseline assessments)
2) On Wednesdays after a shortened driving session (6 driving assessments)
3) In the third week after a driving session (3 post-study assessments)
Beginning with the second driving session, we also noted two subjective observations:
1) Level of engagement (from 1-5)
2) Level of help needed in playing the game (from 1-5)
We focused on three hypotheses.
1) We hypothesized an overall improvement in cognitive scores from the original baseline to the last post-study in the three goals; we expected that all the participants would show some improvement, in part because they would improve with increased familiarity of the cognitive assessment tasks regardless of driving.
2) We expected higher cognitive scores during the driving treatment periods compared to before or after the treatment periods.
3) We hypothesized that the use of driving simulations would benefit people who were less cognitively impacted by their BI more than those who were highly impacted.
While we scheduled 12 assessments, two participants missed at least one Wednesday driving/data collection sessions. To compensate, we averaged the driving session assessments.
1) To test the first hypothesis (overall improvement), we conducted a Wilcoxon signed-ranked test for a non-parametric two group comparison with related samples between the original baseline and the last post-study assessment for each assessment type (speed, VTSM and attention).
2) To test the second hypothesis (peak performance during driving sessions), we created three scores for each assessment type: (1) average driving assessments combined; (2) average baseline assessments combined; and (3) average post-study assessments combined. We conducted individual Wilcoxon Signed Ranked tests to compare (a) driving to baseline assessments and (b) driving to post-study assessments.
3) To test the third hypothesis (driving will benefit people who were less cognitively impacted), we conducted a non-parametric correlation using Spearman’s rho that assessed the correlations among clients’ cognitive scores (as assessed by Anixter) to their overall average assessments.
While we found that the participants enjoyed the driving sessions and did not find the game overly difficult (subjective measures), the results from our objective measures were somewhat mixed.
1) From the first baseline assessments to the final post-study assessments, we only saw improvement in the speed processing scores, and those were not significant. We expected that just learning effects of the assessment programs alone would result in overall improvement. However, as the figure below indicates, the total baseline scores (taken one week before the three driving sessions) demonstrated higher scores than the post-study scores (taken three weeks after the three driving sessions).
2) Our second hypothesis, that the assessment scores would peak during driving sessions when compared to baselines and post-study, was somewhat supported for speed processing and attention. The highest assessment scores were during driving sessions, followed by post-study for both measures. However, the scores were essentially flat for visual short-term memory, see the figure below. As a result, this finding indicated that driving simulations may have benefits for improving speed processing and attention for people who have had a brain injury, but that a larger study is needed to investigate the full potential.
3) Finally, our last hypothesis that the clients’ cognitive scores (as assessed by Anixter) would correlate with their assessment scores was not supported. This finding indicated that any potential benefits of the driving simulation on speed processing and attention do not appear to be related to overall cognitive functioning.
Overall, our findings were promising but somewhat disappointing. Our small sample size limited the strength of our findings; i.e., our initial power analysis found that for a medium effect size of .28, and collecting assessments at 12 intervals that we would need a sample size of 15 to use repeated measures ANOVA for each assessment (p<.05 assumed). Unfortunately, we could not recruit enough people from the clientele at the Anixter Center. Moving forward, we suggest several modifications to reproduce this study:
1) Use alternative instruments for assessing the dependent measurements. The Brain Baseline app had limitations that included: (1) some participants had difficulty holding the iPad and using their thumbs due to partial hand paralysis – we worked around this by using alternative methods, e.g. placing the iPad on a table and holding the iPads for the clients; and (2) participants became bored with the Brain Baseline apps over the 12 assessments.
2) Use a control group that is not driving. While we did not see any significant learning effects as reflected by the original baseline and final post-study scores, a closely matched control group who took the assessments without driving would have been a good addition to the study. However, a matched control group is always a challenge in such a diverse audience.
Recognition
the end
Want to learn more?There are still some details I didn't show here. If you are interested, feel free to view the full document here.
view full document