This paper is in the following e-collection/theme issue:

Children and Youth with Disabilities (53) Human Factors and Usability Case Studies (787) mHealth for Wellness, Behavior Change and Prevention (2871) Usability of Apps and User Perceptions of mHealth (974) Parent and Child Education on Healthy Eating and Nutrition (169) Parent and Child Education on Physical Activity (136)

Published on in Vol 2, No 2 (2019): Jul-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/15153, first published .
Exploring an Existing Weight Management App for Use With Adolescents and Young Adults With Spina Bifida: Usability Study

Exploring an Existing Weight Management App for Use With Adolescents and Young Adults With Spina Bifida: Usability Study

Exploring an Existing Weight Management App for Use With Adolescents and Young Adults With Spina Bifida: Usability Study

Authors of this article:

Colleen Stiles-Shields1, 2 Author Orcid Image ;   Brittney Garcia2 Author Orcid Image ;   Kimberly Villota2 Author Orcid Image ;   Elicia Wartman2 Author Orcid Image ;   Adrien M Winning2 Author Orcid Image ;   Grayson N Holmbeck2 Author Orcid Image
Article Authors Cited by (5) Tweetations (5) Metrics

    Original Paper

    1Population Behavioral Health, Department of Psychiatry and Behavioral Sciences, Rush University Medical Center, Chicago, IL, United States

    2Department of Psychology, Loyola University Chicago, Chicago, IL, United States

    Corresponding Author:

    Colleen Stiles-Shields, PhD

    Population Behavioral Health

    Department of Psychiatry and Behavioral Sciences

    Rush University Medical Center

    1645 W Jackson Blvd

    Chicago, IL, 60612

    United States

    Phone: 1 3125631579

    Email: elizabeth_stiles-shields@rush.edu


    Background: Adolescents and young adults with spina bifida (AYA-SBs) have unique user needs, given their variable and complex symptom profile. Owing to multiple barriers to prevention and intervention treatments for secondary conditions (eg, obesity), AYA-SBs may benefit from the use of behavioral intervention technologies (BITs). However, as BITs are often designed and tested with typically developing individuals, it is unclear if existing BITs may be usable for AYA-SBs.

    Objective: This study aimed to evaluate the usability of a high-quality, publicly available, weight management–focused mobile BIT (smartphone app) for AYA-SBs.

    Methods: Overall, 28 AYA-SBs attending a Young Men’s Christian Association–based summer camp completed 4 structured usability tasks using a weight management app designed for the general public called My Diet Coach (Bending Spoons). Learnability was measured by (1) time to complete task, (2) number of user errors, and (3) correct entry of data when requested by the app. Satisfaction and general usability were measured via self-reported questionnaires and qualitative feedback following interactions with the app.

    Results: The majority of the sample were able to complete the tasks, with increased completion rates and improved times on second attempts of the tasks (Ps<.05). Errors were common, and discrepancies emerged between quantitative and qualitative feedback such that self-reported measures indicated dissatisfaction but qualitative feedback was generally positive. Suggested improvements to the app included (1) tutorials, (2) simplifying the design, (3) more activity options for those who ambulate by wheelchair, and (4) notifications to prompt use.

    Conclusions: AYA-SBs were able to learn how to complete specific tasks independently on a weight management app, but design changes consistent with previously proposed user needs were recommended. Rather than designing entirely new BITs, it may be possible to adapt existing technologies to personalize BITs for specific populations such as AYA-SBs.

    JMIR Pediatr Parent 2019;2(2):e15153

    doi:10.2196/15153

    Keywords

    spina bifida occulta; mHealth; mobile apps; usability testing; adolescent; young adult; weight reduction programs; body weight maintenance 


    Background

    Spina bifida (SB) is the most common congenital birth defect affecting the central nervous system and requires the management of both a complex medical treatment regimen and a variety of cognitive and psychosocial comorbidities [1,2]. Complicating the management of this condition, adolescents and young adults (AYAs) with SB (hereafter referred to as AYA-SBs) are disproportionately impacted by obesity [3]. Beyond the typical risk factors associated with obesity as a secondary condition, obesity poses a greater risk for other outcomes, such as muscle loss, pressure sores, depression, complications of surgery, and decreased social and physical activities [3-5]. In addition, people with SB and obesity are doubly at risk for social rejection, given the combination of 2 visible vulnerabilities: physical disability and obese status [6,7]. Therefore, preventing and reducing obesity in individuals with SB is a critical goal.

    Multiple emotional (eg, low motivation and depressive symptoms) and practical barriers (eg, ambulation status and transportation) to addressing obesity and healthy lifestyles have been identified for AYA-SBs [3,5,8-10]. Behavioral intervention technologies (BITs), the use of technology to deliver behavioral health interventions [11], have demonstrated promising effects on health outcomes for pediatric samples [12]. AYA-SBs report using a variety of technology and media [13]. Furthermore, a self-management behavioral intervention technology (BIT) for people with disabilities (ie, SB, cerebral palsy, and spinal cord injuries), Interactive Mobile Health and Rehabilitation (iMHere), has demonstrated feasibility and benefits to high users of the system [14]. Technology usage, in combination with the barriers faced to addressing obesity, implicates the use of BITs as a delivery mechanism for obesity and healthy lifestyle monitoring and interventions for AYA-SBs.

    Given their unique user needs, calls for personalizing BITs for people with disabilities have been made [15]. For example, AYA-SBs are unique in that they have symptoms that overlap with chronic medical conditions and physical and intellectual disabilities. For this reason, a user needs model for BITs that support self-management in SB has been created [16]. The model includes (1) behavioral skills– and evidence-based change strategies that avoid abstract concepts and help to categorize behavior, (2) elements that are multisensory (eg, text and audio) and use multiple methods (eg, visualizations and passive data collection), (3) being capable of being used across multiple platforms, (4) linear and user-driven workflows, and (5) expert and/or peer support. However, before using this model to create new BITs designed for AYA-SBs, currently available BITs should be evaluated. This is for multiple reasons. First, although the previously mentioned iMHere platform already exists, it is (1) currently only available for Android devices and (2) designed to promote self-management across a variety of issues pertinent to those with disabilities. AYA-SBs without an Android and/or those who might wish to exclusively focus on an issue such as weight management face barriers to utilizing this BIT. Therefore, AYA-SBs who face such barriers are likely to search the publicly available marketplace for another BIT to suit their needs. Second, given the high number of BITs already available for health and body image management, it is possible that elements of existing BITs may be appropriate for AYA-SBs. Therefore, evaluating how currently available BITs function for AYA-SBs is necessary to potentially avoid reinventing the wheel in terms of some BIT tools.

    To accomplish the task of evaluating how well currently available BITs may serve this population, usability testing was utilized. Usability testing is the systematic observation of planned tasks by potential end users to improve the design of a product or technology [17]. Learnability, or how well a user can complete tasks during the first interaction(s) on an app [18], was the usability attribute of interest. Indeed, if AYA-SBs independently download a publicly available app, their ability to learn to use the app is essential for correct use. On average and based on evaluations of healthy adults, users report spending about 5 min or less learning how to use a new app [19,20], and AYAs have been found to quickly dismiss technological tools that misalign with their expectations [21,22]. Furthermore, AYA-SBs tend to have difficulties with assembled processing (ie, learning to construct and digest information) [23]. This difficulty means that even the most ideal app (eg, grounded within evidence-based behavioral change theories and high usability) might have different learnability for a user with SB. Therefore, in assessing the usability of BITs for AYA-SBs, how quickly and well the users learn to use the BIT is important.

    Purpose

    The purpose of this study was to evaluate the usability attribute of learnability for a high-quality [18], publicly available, weight management–focused mobile BIT (smartphone app) for AYA-SBs. We hypothesized that, in evaluating an app rated as having high quality for typically developing populations [18], the majority of users would be able to complete tasks and do so in 5 min or less by their second attempt of a task [19,20]. However, given the variable symptom and functioning profile of AYA-SBs [16], it was anticipated that high variability across the sample would be observed for all evaluated usability metrics.


    Participants

    Participants were recruited from the Young Men’s Christian Association (YMCA)-sponsored Camp Independence during the 2018 summer sessions. Camp Independence is located in Illinois (USA) and is a sleep-away camp designed for AYA-SBs. Programming includes (1) a 1-week stay with similarly aged campers, (2) typical camp-based activities (eg, swimming) with accommodations for camper needs, and (3) camp-based interventions to promote medical and social independence [24-26]. Participants were eligible for study inclusion if they (1) had SB, (2) were aged between 13 and 30 years, (3) attended Camp Independence during the summer of 2018, (4) had previously used a mobile app independently, and (5) could read and write in English.

    My Diet Coach

    The app that was selected for usability testing was the My Diet Coach app. My Diet Coach was selected as it is (1) highly rated for quality using the Mobile App Rating Scale [18,27], (2) publicly available on both iTunes [28] and the Google Play Store [29], and (3) cost free (a version is also available for purchase). The app includes features such as a meal and activity journal, tips, and user-selected challenges.

    Procedure

    This study was approved by the Loyola University Chicago institutional review board. Participants aged 18 years and older provided informed consent. Participants younger than 18 years provided informed assent, and their parents provided consent.

    Usability Testing

    To assess the usability attribute of learnability, the participants were asked to complete specific tasks (eg, log a food item) with an app on a mobile device. These tasks were related to food intake and activity monitoring, which were selected because (1) they are common tasks for weight loss and management and (2) they could be evaluated relatively quickly, so as to not keep participants from their camp activities. Learnability was measured by (1) time to complete task, (2) number of user errors (tracked on a standardized paper measure by the moderator), and (3) correct entry of data when requested by the app. Improvements across these measurements were hypothesized to occur across attempts (eg, logging a food item attempt 1 versus attempt 2). Satisfaction and general usability were measured via self-reported questionnaires (please see Measures section).

    Participants were brought to a private area of the main camp building to complete testing. Testing was conducted by research and graduate assistants with experience in interviewing AYA-SBs. A think-aloud protocol was used [30]; however, the majority of participants did not say what they were thinking during the usability tasks. To avoid distracting the participants, those who did not speak during tasks were given time to share any relevant thoughts after each interaction. Before the testing of My Diet Coach, participants engaged in a card-sorting task to identify barriers to the use of apps for health management. The card sorting results are reported elsewhere [13]. Next, participants were verbally queried about their typical platform for accessing apps (ie, iOS or Android) and then provided a description of My Diet Coach from the Google Play Store (ie, “My Diet Coach helps you find your inner motivation, stay on track, make healthy lifestyle changes, resist food cravings, and avoid exercise laziness and other weight loss difficulties.”). Participants were then provided an Android phone (Moto G5s Plus, 5.5-inch screen). If they were unable to utilize this device because of vision or motor dexterity problems, they were offered the option to complete the testing on an iPad (6th generation), which was encased in a handle-stand cover to improve the participant’s ability to hold the device. The verbal instructions for the first task were as follows:

    Now, please imagine that you want to eat healthier. You see this description of My Diet Coach on the Google Play store and decide to download it. Let’s pretend that you just ate one piece of pepperoni pizza for lunch. Please open the app, which is already on the home page, and select “the lightning bolt” to log the pizza you just ate. Feel free to say out loud what you are thinking while you do this. Tell me when you are finished.

    This interaction was timed and audiotaped, and any observed user errors or alternative paths to complete the task were noted by the research or graduate assistant. If a participant stopped working on the task, they were prompted with “What’s going through your mind right now?”, followed by “Please do your best to complete the task and let me know when you are finished.” To avoid participant frustration, the task was ended if a user stopped working for 90 consecutive seconds. Once the task was completed, participants were asked to share their thinking with regard to alternative paths taken to complete the task. Participants completed 3 more tasks following this same methodology: (1) exercise (ie, entering a 30-min activity of their choice), (2) second food (ie, entering in eating an apple for a snack), and (3) second exercise (ie, entering a different 15-min activity of their choice). Finally, participants were asked open-ended questions about (1) their impressions of the app (eg, “What are your overall impressions of the logging features of My Diet Coach?” and “Is there anything that you feel is missing?”) and (2) the designs of technology more generally for AYA-SBs (eg, “How could technology work better for you in terms of managing your health?”).

    Data Collection Approaches

    The following traditional data collection methodologies that have been used in the testing of other apps [19,31,32] were selected to evaluate My Diet Coach for AYA-SBs: (1) audio recordings of the testing, (2) standardized interview questions, (3) providing the option for the research/graduate assistants to prompt participants following specific behaviors, (4) validated questionnaires (see Measures section), (5) timing of the tasks with a stopwatch, and (6) research/graduate assistant recording of errors or path deviations (on a standardized paper form).

    Measures

    All measures were administered following completion of the interactions with My Diet Coach. Participants were given the opportunity to answer questionnaires via paper and pencil or via an electronic version administered through Opinio [33], licensed and administered by Loyola University Chicago.

    Demographics

    Participants were asked to report the following information: age, sex, race/ethnicity, and SB characteristics, including type, shunt status, and lesion level. Full Scale Intelligence Quotient (FSIQ) was measured and collected for those who also participated in another camp-based study [24-26].

    System Usability Scale

    The System Usability Scale (SUS) is a 10-item self-reported instrument measuring a user’s rating of a product’s usability [34]. Items are rated on a 5-point Likert scale (ie, 1=strongly disagree to 5=strongly agree). Total scores are derived by converting the responses (ie, subtracting 1 from odd-numbered items and 5 from even-numbered items), summing the converted numbers, and multiplying the total by 2 and a half. Although this scoring method yields total scores ranging from 0 to 100, this number is not meant to be interpreted as a percentage. A SUS total score of 68 is considered the cut point for an average score or grade of C; higher scores are considered above average and lower scores are considered below average [35]. The SUS has been utilized in previous research with youth and adults with SB [36] and had adequate reliability in the current sample (alpha=.86).

    After-Scenario Questionnaire

    The After-Scenario Questionnaire (ASQ) is a 3-item self-reported instrument measuring a user’s satisfaction with a product [37]. Items are rated on a 7-point Likert scale (ie, 1=strongly agree to 7=strongly disagree). Respondent answers to the items are averaged to create a total score, with higher scores indicating higher dissatisfaction following a specific task. To the best of our knowledge, this is the first use of the ASQ with a sample of AYA-SBs. The ASQ demonstrated an adequate Cronbach alpha coefficient (.66) for this sample.

    Health Questionnaire

    The Health Questionnaire is a modified and abbreviated (17 out of 87 original items) version of the 1999 Youth Risk Behavior Survey by the Centers for Disease Control and Prevention (CDC) [38]. Items used for this study address self-reporting of weight, height, desire to change weight, diet, food, and exercise questions appropriate for youth with SB. Data from the Health Questionnaire include categorical (ie, “Which of the following are you trying to do about your weight? Lose weight; Gain weight; Stay the same weight; or I am not trying to do anything about my weight”) and continuous (ie, frequency of behavior) variables. Responses on the Health Questionnaire were used to calculate body mass index (BMI) based on self-reported height and weight, which were calculated using the CDC BMI calculators for children and teens [39] and adults [40]. However, some missing data for BMI were anticipated, as there are established difficulties in obtaining valid measurements of height and weight in people with physical disabilities [41]. The Health Questionnaire was administered to characterize the current health and weight management behaviors of the current sample and has previously been used in studies involving AYA-SBs [42].

    Data Analysis

    The demographic and SB characteristics (eg, type of SB), usability testing measurements (eg, time to complete and number of user errors), and questionnaire data were analyzed using descriptive statistics. The t test was used to compare differences between the first and second attempts of tasks (eg, time to complete the first food entry compared with time to complete the second food entry). All analyses were run in Statistical Package for the Social Sciences version 24 (IBM Corp), with the 0.05 type I error rate.


    Participants

    A total of 29 participants agreed to participate; however, 1 participant only completed questionnaires as she reported feeling too overwhelmed from the card sorting task [13] to complete usability testing. Participants were primarily young adults (mean 18.11, SD 4.55), female (59%, 17/29), and non-Hispanic white (72%, 21/29), with myelomeningocele (69%, 20/29), with a lumbar (41%, 12/29) or unknown lesion level (41%, 12/29), and with a shunt (79%, 23/29). The BMI of the sample ranged from underweight to obese, with the average BMI falling within the normal range (mean 21.84, SD 4.19). It should be noted that 7 participants (24%) did not have BMI data because of difficulty in reporting current height and/or weight (obtaining accurate height and weight measurements in people with disabilities can pose challenges) [41]. The majority of the sample endorsed wanting to lose (39%, 11/28; one participant did not answer) or maintain (32%, 9/28; one participant did not answer) weight. Table 1 displays the demographic, SB, and health characteristics.

    Table 1. Demographic, spina bifida, and health characteristics (N=29).
    CharacteristicsAdolescents and young adults with spina bifida
    Age (years), mean (SD); range18.11 (4.55); 13-30
    Sex,n(%)

    		Male12 (41.4)

    		Female17 (58.6)
    Race/ethnicity,n(%)

    		African American2 (6.9)

    		Asian2 (6.9)

    		Caucasian21 (72.4)

    		Hispanic4 (13.8)

    		Other—a
    Spina bifida type,n(%)

    		Myelomeningocele20 (69.0)

    		Other9 (31.0)

    		Meningocele—

    		Lipomeningocele1 (3.4)

    		Occulta—

    		Unsure8 (27.6)
    Lesion level,n(%)

    		Thoracic1 (3.4)

    		Lumbar12 (41.4)

    		Sacral4 (13.8)

    		Unsure12 (41.4)
    Shunt present, n (%)23 (79.3)
    Full Scale Intelligence Quotientb, mean (SD); range84.67 (19.51); 55-132
    Personal mobile device,n(%)

    		Android10 (34.5)

    		iOS18 (62.1)

    		Did not report1 (3.4)
    Body mass indexc, mean (SD); range21.84 (4.19); 14.50-32.00
    Current weight change attempts,n(%)

    		Gain6 (21.4)

    		Lose11 (39.3)

    		Maintain9 (32.1)

    		No attempts to change2 (7.1)

    aNot applicable.

    bData missing for 5 participants because of not participating in the larger camp-related intervention.

    cData missing for 7 participants because of not responding about height and/or weight.

    Health Behaviors

    To contextualize the usability outcomes within the sample’s health behaviors, participants reported their current frequency of healthy food consumption, physical activity, screen time, and sleep. Tables 2 to 4 display the health behavior frequencies of the current sample. For dietary behaviors, the CDC recommends a daily minimum intake of (1) 2 fruits/100% juice servings, (2) 2 and a half servings of vegetables, and (3) 3 servings of milk/dairy [43]. Consistent with previous reports of health behaviors in AYA-SBs [42], the majority of the sample reported consuming fruits/100% juice servings (86%, 23/28), vegetables (82%, 22/28), and milk/dairy (89%, 25/28) below the recommended frequencies. For physical activity, teens younger than 18 years are recommended to get at least 60 min of physical activity daily; adults 18 years and older are recommended to get at least 150 min of physical activity weekly (including aerobic, muscle strengthening, and bone strengthening activities) [43]. Given that only (1) 21% (6/28) of the sample endorsed engaging in at least 30 min of strenuous exercise daily, (2) 14% (4/28) endorsed engaging in at least 30 min of nonstrenuous exercise daily, and (3) 4% (1/28) endorsed daily strength exercises, it is likely that the sample is falling short of CDC recommendations for physical activity. However, falling in line with current recommendations of 2 hours or less of screen time per day [44], the majority (69%, 17/26) of the sample reported engaging in 2 hours or less of screen time on weekdays (No time: 11.5%, 1 hour: 11.5%, <1 hour: 11.5%, 2 hours: 34.6%, 3 hours: 15.4%, 4 hours: 3.8%, >5 hours: 11.5%). Finally, and contrary to previous findings of youth and AYA-SBs [42,45],exactly half of the sample endorsed typically sleeping 8 to 9 hours per night [46] and having no or very little difficulty falling and staying asleep.

    Table 2. Health behavior frequencies.
    Frequency of daily healthy food intake and screen timeNo times1-3 times in past 7 days4-6 times in past 7 daysOnce per dayTwice per day3 times per day>4 times per day
    100% fruit juicea (%)17.935.710.721.47.17.10
    Fruit (%)17.928.617.917.917.900
    Vegetables (%)14.332.121.410.73.6017.9
    Milk (%)25.017.914.317.914.33.67.1

    aFrequencies reported from adapted version of the Youth Risk Behavior Surveillance System, as reported in the study by Kolbe et al [38].

    Table 3. Frequency of physical activity.
    Physical activity0 days1 day2 days3 days4 days5 days6 days7 days
    ≥30 min of strenuous exercisea (%)7.1%10.714.317.910.714.33.621.4
    ≥30 min of nonstrenuous exercise (%)21.414.317.910.714.33.63.614.3
    Strength exercises (%)53.67.114.33.610.73.63.63.6

    aFrequencies reported from adapted version of the Youth Risk Behavior Surveillance System, as reported in the study by Kolbe et al [38].

    Table 4. Sleep quality.
    Ability to fall asleepNot at allVery littleModerately oftenVery oftenAlmost always
    Trouble falling asleepa (%)26.942.315.47.77.7
    Trouble staying asleepa (%)50.034.611.53.81.2

    aTwo participants did not report their sleep information.

    Equipment

    The majority of the sample reported using iOS (62%, 18/28) for their personal mobile devices; however, most of them denied using apps to help manage their SB in any way (75%, 21/28). Most participants completed the usability tasks on the Android mobile phone (86%, 24/28); however, 4 participants requested to use the iPad to complete testing because of vision or motor dexterity issues. Given the small sample size and uneven number in the groups, Mann-Whitney U tests were run to compare those who reported owning an iPhone with those who reported owning an Android on the task measurements (ie, completion and error rates and time to complete tasks) to ensure that the device platform did not impact the results. There was no evidence to suggest significant differences between those who owned an iPhone and an Android (Ps>.09), with the exception of time to complete the second exercise entry (P=.04). For this task, those who owned an iPhone (mean 38, SD 32) were significantly faster than those who owned an Android (mean 58, SD 29). Therefore, the data suggest that it is unlikely that those who owned an iPhone were at a disadvantage if they completed the tasks on an Android device.

    Usability of My Diet Coach for Adolescents and Young Adults With Spina Bifida

    Completion

    About two-thirds of participants (n=20) were able to independently complete a food entry on their first attempt (ie, pizza), with an increase in completion on the second attempt (ie, apple; n=22). For activity entries, about two-thirds of participants (n=20) were able to independently complete an entry on the first attempt, with another increase in completion on the second attempt (n=24). Significance testing was not performed on the completion rates because of lower than expected counts in at least one cell of the contingency table for both activities. The activity entries could be selected by the user, with the most frequently chosen activities being lifting weights, sled hockey, and walking/running. Most participants (86%) were able to come up with their own activities without suggestions from the research or graduate assistant moderating the sessions.

    Time

    Participants significantly decreased their time to complete tasks across the 2 attempts for both food entries (mean time 2:00, SD 1:38 vs mean 1:02, SD :56; P=.01) and activity entries (mean time 1:11, SD :50 vs mean :45, SD :31; P=.002).

    Errors and Deviations

    The most common user errors involved (1) entering incomplete or inaccurate data (16 out of 112 attempted tasks, 14.3%), (2) being unsure of how to proceed to the next step without being able to recover and complete the task (ie, a fatal error; n=13, 11.6%), or (3) believing the task to be complete when it was not (n=8, 7%). The most common deviation was accidentally selecting an option to upgrade the app (n=5, 5%). There was no evidence to suggest a significant difference in the number of errors or deviations across food entry attempts (mean 2.42, SD 2.09 vs mean 1.88, SD 2.14; P=.09). However, a decrease in errors or deviations from the expected path across activity entries occurred (mean 1.46, SD 1.96 vs mean 0.58, SD 0.76; P=.03).

    Usability and Satisfaction

    Participants completed the SUS and ASQ following their interactions with My Diet Coach to evaluate usability and satisfaction, respectively. SUS ratings for My Diet Coach were highly variable (range 2.50-100). The average SUS rating was 64.17 (SD 29.59; a below average SUS score). The average ASQ rating for My Diet Coach was 5.55 (SD 1.36; total scores range from 3 to 21, with higher scores indicating greater dissatisfaction).

    Sensitivity Analyses by Age

    Given the wide range of AYA-SB participants (13-30), sensitivity analyses were conducted to explore any differences that were driven by age. Specifically, exploratory chi-square and t tests were conducted to compare demographic and SB-related factors, phone usage, and usability outcomes for those aged 17 years and younger (n=17) with those aged 18 years and older (n=11). There was no evidence to suggest differences in FSIQ, sex, race/ethnicity, type of SB, lesion level, shunt status, type of phone ownership, device used for testing, questionnaire responses, or any usability testing outcomes (Ps≥.1), with the exception of adolescents being more likely to have successfully completed the first exercise entry (P=.03).

    Qualitative Feedback

    Although the usability tasks were audiotaped, the majority of participants did not engage in the suggested think-aloud method of completing tasks. Therefore, qualitative feedback occurred through responses to structured questions that were administered to all participants after completing the usability testing (please see Procedures section).

    Despite the variable usability ratings, the majority of participants stated that My Diet Coach would be useful for them (64%, 18/28). When queried as to why this app might be useful, the most common response was related to the app reminding and encouraging them to change their eating and activity choices. For example:

    I could see how many calories I'm burning and consuming, and balance those. I definitely see me using in the future.
    [17-year-old female]
    [This app] would help me make healthy choices.
    [17-year-old female]
    It would help me realize I need more fruits and vegetables.
    [21-year-old male]

    Although the current usability testing evaluated initial learnability, qualitative feedback also suggested that learnability might improve with long-term use (eg, “It was a little tricky at first. As I used it more, it became easier to use.” [13-year-old male]; “It seems pretty easy to use once you get used to it.” [18-year-old female]). Suggested improvements specific to AYA-SBs included (1) tutorials (eg, “Make it more self-explanatory. Have practice stuff, give step-by-step directions to enter stuff.” [14-year-old female] and “There’s a lot of information in it with no instructions.” [20-year-old female]), (2) fewer logging options and/or simplifying the design throughout (eg, “It’s too confusing with too many steps” [17-year-old female] and “Make it easier by not making it so heavy in content and choices.” [19-year-old female]), (3) more activity options for those who ambulate by wheelchair (eg, “It needs to add exercise for people in wheelchairs.” [23-year-old male]), and (4) notifications to prompt use (eg, “I need reminders [to do this] on my phone…Remind me to do my exercises and eat healthy.” [20-year-old female]).


    Principal Findings

    The purpose of this study was to evaluate the usability attribute of learnability for My Diet Coach, a publicly available, high-quality app [18] designed for the general public, for AYA-SBs. The majority of the sample endorsed wanting to maintain or lose weight and reported dietary and exercise behaviors that fell short of the CDC recommendations [43]. These characteristics, combined with the increased risk for obesity in people with SB [3], make this app a potentially appropriate tool for aiding in weight management for this sample. The majority of the sample was able to complete the tasks of entering foods and exercises into the app, with improved performance on the second attempts for both tasks. Consistent with the hypotheses, AYA-SBs were able to complete initial tasks with the app in under 5 min [19,20] and demonstrated decreased task time on the second attempts. Despite this success, but also consistent with our hypotheses, high variability of usability was observed, suggesting that added tutorial features for users with special needs might be beneficial. To the best of our knowledge, there is no established metric for the ideal number of user errors on initial interactions with an app [47,48]; however, the error rates (1) appeared high for a limited number of required task actions and (2) raise questions about the initial learnability of this app for subsets of AYA-SBs. Finally, responses to validated usability questionnaires and open-ended interview questions suggest variable usability for this group and also emphasize the importance of querying AYA-SBs in a mixed-methods fashion.

    A user needs model for AYA-SBs was recently created for BITs aiming to improve self-management [16]. This user needs model was framed within the BIT model, which includes conceptual (why the BIT is needed and how it may achieve such aims) and technical (what is delivered to BIT users and how and when the delivery may occur) aspects for designing and deploying BITs [49]. In utilizing a high-quality app already publicly available (ie, My Diet Coach), this study evaluated the usability of existing BIT elements (ie, what is delivered) and characteristics (ie, how the delivery may occur) for AYA-SBs. The findings suggest that in applying the user needs model for AYA-SBs to the design of BITs specific to this population, design teams do not need to reinvent the wheel and create entirely new BITs for AYA-SBs (a cost-heavy venture from a financial and time perspective). Indeed, it appears that the majority of AYA-SBs in the current sample can independently learn to complete specific data entry tasks within a reasonable time frame using an existing BIT targeting weight management [19,20].

    Although the current findings suggest that My Diet Coach is learnable for AYA-SBs, it also implicates personalizing existing elements and characteristics for AYA-SBs and related users (eg, youth with physical and/or motor disabilities) [16]. The evaluated elements of My Diet Coach involved active, text-based data entry. Consistent with the literature on AYAs with similar symptom profiles [16], the AYA-SB participants reported that these elements could be improved for their use by having tutorials, being simpler, and having less logging options. Possibly because of the lack of such features and/or the executive functioning problems associated with SB [23], participants also displayed several errors for a relatively simple data entry path. In addition, the entry options were not optimized for variability in accessibility (eg, for those who ambulate with leg braces or by wheelchair). Therefore, although AYA-SBs demonstrated that they are capable of using elements featuring text-based data entry, they may be unlikely to persist in using such elements in real-world conditions. Moreover, the data collected might not be as accurate as compared with the use of other means of collection (eg, voice-to-text entry or passive data collection via an accelerometer). These findings suggest that a multisensory and multimethod approach to BIT elements is likely warranted for AYA-SBs [16] but that text-based data entry is a learnable element for AYA-SBs.

    Given the variable levels of impairment in motor dexterity, coordination, hearing, vision, and visuospatial processing in people with SB [23], BIT characteristics likely also need to demonstrate flexibility. My Diet Coach demonstrated multiple platform capabilities in testing, which was important and necessary for AYA-SBs (ie, participants accessed the app via an Android phone or an iPad). The text-based data entry tasks involved elements of the app that had limited graphics, which falls in line with user needs for simple and/or limited graphics [16] because of posterior attention difficulties (which impact the ability to focus and shift attention) [23]. My Diet Coach is also designed to include customized reminders around user goals (ie, “Drink water” and “Always be prepared with healthy snacks”). This characteristic was reported as desired following testing and also falls in line with the user needs model [16]. However, it was not evaluated in this study. Therefore, the design characteristics of My Diet Coach appeared to align well with the needs of AYA-SBs, but future research will need to evaluate how usable such characteristics are when users are confronted with other common tasks (eg, interacting with notification reminders).

    The majority of participants were able to complete the tasks and stated that the app would be personally useful. However, questionnaire responses were not indicative of high usability and satisfaction, and the majority of the sample reported that they do not use apps to manage their SB (75%). These discrepancies may have multiple explanations. First, young people with SB have variable cognitive profiles, with many falling within the category of having a mild intellectual disability [2]. The differences between qualitative and quantitative feedback may reflect the importance of feedback methods when assessing AYA-SBs. It also highlights the need for continued validation of usability measures for individuals with disabilities and/or special needs. Second, AYA-SBs may use technology less frequently and/or in selective ways compared with the general population [50]. It is likely that BITs may sound appealing in theory but are anticipated to have multiple barriers (being unintuitive, not specialized for the needs of people with SB, etc) or are in conflict with time already allotted to TV viewing or social networking [3,13]. Finally, it is possible that participants believed that the app had been developed by those conducting the usability testing. Therefore, qualitative feedback may have been driven by a desire to please the evaluators.

    Limitations

    This study builds upon previous work establishing the importance and feasibility of conducting usability testing with AYA-SBs [51,52]. However, the findings should be considered in light of specific limitations. First, the sample was recruited from the YMCA-sponsored Camp Independence [24-26]. The sample consisted of a wide age range of AYA-SBs who have the support and ability to attend a sleep-away summer camp session and who were also primarily non-Hispanic whites. More usability testing of apps for self-management is required, targeting AYA-SBs with greater diversity and within more real-world conditions. Second, testing was focused solely upon data entry of monitoring behaviors that are common in weight management strategies (ie, food intake and physical activity monitoring). Furthermore, the usability tasks were designed to be brief (ie, 4 entry tasks that took an average of about 2 min per task), so as to not keep participants from the activities of the camp for a significant amount of time. It is unclear how the current findings extend to (1) the entry and monitoring of other behavioral change strategies, (2) long-term use, and (3) other types of app elements and other apps, more generally. Third, the majority of participants reported using iOS for their personal devices, yet the majority opted to complete usability testing on an Android device. Comparisons of usability outcomes across these 2 groups did not suggest a disadvantage for iPhone users. However, to avoid this confound in future research, facilitating acclimation to the operating system before usability testing would be ideal. Finally, it is unclear if the presence of the research/graduate assistants facilitating the testing sessions had any impact on performance.

    Conclusions

    For clinicians and engineers designing BITs for pediatric and AYA populations, the findings of this study suggest that it may be possible to iterate from existing technologies to personalize apps for specific populations. Doing so may decrease both the financial and time burden associated with designing and building a new technology. However, the use of appropriate user-centered design principles and the use of formative usability testing is still critical [49,53]. Indeed, AYA-SBs were able to learn how to complete specific tasks independently on a weight management app, but design changes consistent with previously proposed user needs are still recommended [16].

    Acknowledgments

    This research was supported in part by a Research Support Grant from Loyola University Chicago and grants from the Kiwanis Neuroscience Research Foundation: Illinois-Eastern Iowa District of Kiwanis International, National Institute of Nursing Research and the Office of Behavioral and Social Sciences Research (R01 NR016235), National Institute of Child Health and Human Development (R01 HD048629), and the March of Dimes Birth Defects Foundation (12-FY13-271). The authors thank the YMCA-sponsored Camp Independence and the Chicago Healthy Adolescent Transition Study (CHATS) research team, without whom this research would not have been possible.

    Conflicts of Interest

    None declared.

    1. Copp AJ, Adzick NS, Chitty LS, Fletcher JM, Holmbeck GN, Shaw GM. Spina bifida. Nat Rev Dis Primers. Apr 30, 2015;1:15007. [FREE Full text] [CrossRef] [Medline]
    2. Wasserman RM, Holmbeck GN. Profiles of neuropsychological functioning in children and adolescents with spina bifida: associations with biopsychosocial predictors and functional outcomes. J Int Neuropsychol Soc. Sep 2016;22(8):804-815. [CrossRef] [Medline]
    3. Polfuss M, Bandini LG, Sawin KJ. Obesity prevention for individuals with spina bifida. Curr Obes Rep. Jun 2017;6(2):116-126. [CrossRef] [Medline]
    4. Liou TH, Pi-Sunyer FX, Laferrère B. Physical disability and obesity. Nutr Rev. Oct 2005;63(10):321-331. [CrossRef] [Medline]
    5. Rimmer JH, Rowland JL, Yamaki K. Obesity and secondary conditions in adolescents with disabilities: addressing the needs of an underserved population. J Adolesc Health. Sep 2007;41(3):224-229. [CrossRef] [Medline]
    6. Roux G, Sawin KJ, Bellin MH, Buran CF, Brei TJ. The experience of adolescent women living with spina bifida. Part II: peer relationships. Rehabil Nurs. 2007;32(3):112-119. [CrossRef] [Medline]
    7. Harrist AW, Swindle TM, Hubbs-Tait L, Topham GL, Shriver LH, Page MC. The social and emotional lives of overweight, obese, and severely obese children. Child Dev. Sep 2016;87(5):1564-1580. [CrossRef] [Medline]
    8. Bellin MH, Zabel TA, Dicianno BE, Levey E, Garver K, Linroth R, et al. Correlates of depressive and anxiety symptoms in young adults with spina bifida. J Pediatr Psychol. Aug 2010;35(7):778-789. [FREE Full text] [CrossRef] [Medline]
    9. Buffart LM, Westendorp T, van den Berg-Emons RJ, Stam HJ, Roebroeck ME. Perceived barriers to and facilitators of physical activity in young adults with childhood-onset physical disabilities. J Rehabil Med. Nov 2009;41(11):881-885. [FREE Full text] [CrossRef] [Medline]
    10. Bush RA, Pérez A, Connelly CD, Jorquera E, Leveque NL, Chiang GJ. Community-academic partnership: identifying patient perceptions related to obesity in a spina bifida population. J Particip Med. 2016;8:e6. [FREE Full text]
    11. Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry. 2013;35(4):332-338. [FREE Full text] [CrossRef] [Medline]
    12. Fedele DA, Cushing CC, Fritz A, Amaro CM, Ortega A. Mobile health interventions for improving health outcomes in youth: a meta-analysis. JAMA Pediatr. May 1, 2017;171(5):461-469. [FREE Full text] [CrossRef] [Medline]
    13. Stiles-Shields C, Anderson L, Driscoll CF, Ohanian D, Starnes M, Yunez J. Technology usage and barriers to the use of behavioral intervention technologies in adolescents and young adults with spina bifida. Disabil Rehabil. 2020:-. (forthcoming).
    14. Dicianno BE, Fairman AD, McCue M, Parmanto B, Yih E, McCoy A, et al. Feasibility of using mobile health to promote self-management in spina bifida. Am J Phys Med Rehabil. Jun 2016;95(6):425-437. [CrossRef] [Medline]
    15. Yu D, Parmanto B, Dicianno B. An mhealth app for users with dexterity impairments: accessibility study. JMIR Mhealth Uhealth. Jan 8, 2019;7(1):e202. [FREE Full text] [CrossRef] [Medline]
    16. Stiles-Shields C, Crowe AN, Driscoll CF, Ohanian DM, Stern A, Wartman E, et al. A systematic review of behavioral intervention technologies for youth with chronic health conditions and physical and intellectual disabilities: implications for adolescents and young adults with spina bifida. J Pediatr Psychol. Apr 1, 2019;44(3):349-362. [CrossRef] [Medline]
    17. Albert W, Tullis T. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington, MA. Morgan Kaufmann; 2019.
    18. Bardus M, van Beurden SB, Smith JR, Abraham C. A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management. Int J Behav Nutr Phys Act. Mar 10, 2016;13:35. [FREE Full text] [CrossRef] [Medline]
    19. Harrison R, Flood D, Duce D. Usability of mobile applications: literature review and rationale for a new usability model. J Interact Sci. 2013;1(1):1. [CrossRef]
    20. Flood D, Harrison R, Iacob C, Duce D. Evaluating mobile applications: a spreadsheet case study. Int J Mob Hum Comp Interact. 2012;4(4):37-65. [CrossRef]
    21. Moran K. Nielsen Norman Group. 2016. URL: https://www.nngroup.com/articles/young-adults-ux/ [accessed 2019-08-27]
    22. Loranger H, Nielsen J. Nielsen Norman Group. 2013. URL: https://blog.experientia.com/teenage-usability-designing-teen-targeted-websites/ [accessed 2019-08-27]
    23. Dennis M, Landry SH, Barnes M, Fletcher JM. A model of neurocognitive function in spina bifida over the life span. J Int Neuropsychol Soc. Mar 2006;12(2):285-296. [CrossRef] [Medline]
    24. Driscoll CF, Murray CB, Holbein CE, Stiles-Shields C, Cuevas G, Holmbeck GN. Camp-based psychosocial intervention dosage and changes in independence in young people with spina bifida. Dev Med Child Neurol. Apr 13, 2019:-. (epub ahead of print). [CrossRef] [Medline]
    25. Holbein CE, Murray CB, Psihogios AM, Wasserman RM, Essner BS, O'Hara LK, et al. A camp-based psychosocialiIntervention to promote independence and social function in individuals with spina bifida: moderators of treatment effectiveness. J Pediatr Psychol. May 2013;38(4):412-424. [FREE Full text] [CrossRef] [Medline]
    26. O'Mahar K, Holmbeck GN, Jandasek B, Zukerman J. A camp-based intervention targeting independence among individuals with spina bifida. J Pediatr Psychol. Sep 2010;35(8):848-856. [FREE Full text] [CrossRef] [Medline]
    27. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. Mar 11, 2015;3(1):e27. [FREE Full text] [CrossRef] [Medline]
    28. App Store - Apple. URL: https://itunes.apple.com/us/app/my-diet-coach-weight-loss/id552341639?mt=8 [accessed 2018-06-01]
    29. Google Play. URL: https://play.google.com/store/apps/details?id=com.dietcoacher.sos [accessed 2018-06-01]
    30. Cooke L. Assessing concurrent think-aloud protocol as a usability test method: a technical communication approach. IEEE Trans Profess Commun. Sep 2010;53(3):202-215. [CrossRef]
    31. Mohr DC, Stiles-Shields C, Brenner C, Palac H, Montague E, Kaiser SM, et al. Medlink: a mobile intervention to address failure points in the treatment of depression in general medicine. Int Conf Pervasive Comput Technol Healthc. May 2015;2015:100-107. [FREE Full text] [CrossRef] [Medline]
    32. Stiles-Shields C, Montague E, Lattie EG, Schueller SM, Kwasny MJ, Mohr DC. Exploring user learnability and learning performance in an app for depression: usability study. JMIR Hum Factors. Aug 11, 2017;4(3):e18. [FREE Full text] [CrossRef] [Medline]
    33. Object Planet. URL: http://www.objectplanet.com/opinio/ [accessed 2019-08-27]
    34. Brooke J. SUS: a 'quick and dirty' usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B, editors. Usability Evaluation In Industry. London, UK. Taylor & Francis; 1996:189-194.
    35. Sauro J. A Practical Guide To The System Usability Scale: Background, Benchmarks & Best Practices. Denver, CO. Createspace Independent Publishing Platform; 2011.
    36. Thirumalai M, Kirkland WB, Misko SR, Padalabalanarayanan S, Malone LA. Adapting the Wii Fit balance board to enable active video game play by wheelchair users: user-centered design and usability evaluation. JMIR Rehabil Assist Technol. Mar 6, 2018;5(1):e2. [FREE Full text] [CrossRef] [Medline]
    37. Lewis JR. An after-scenario questionnaire for usability studies: psychometric evaluation over three trials. SIGCHI Bull. Oct 1, 1991;23(4):79. [CrossRef]
    38. Kolbe LJ, Kann L, Collins JL. Overview of the youth risk behavior surveillance system. Public Health Rep. 1993;108(Suppl 1):2-10. [FREE Full text] [Medline]
    39. Centers for Disease Control and Prevention. 2019. URL: https://www.cdc.gov/healthyweight/bmi/calculator.html [accessed 2019-03-31]
    40. Centers for Disease Control and Prevention. 2019. URL: https://www.cdc.gov/healthyweight/assessing/bmi/adult_bmi/english_bmi_calculator/bmi_calculator.html [accessed 2019-04-01]
    41. Liusuwan RA, Widman LM, Abresch RT, Styne DM, McDonald CM. Body composition and resting energy expenditure in patients aged 11 to 21 years with spinal cord dysfunction compared to controls: comparisons and relationships among the groups. J Spinal Cord Med. 2007;30(Suppl 1):S105-S111. [FREE Full text] [CrossRef] [Medline]
    42. Stiles-Shields C, Holmbeck GN. Health behaviors and disordered eating in adolescents and young adults with spina bifida: results from a national survey. Disabil Rehabil. Apr 12, 2019:1-7. (epub ahead of print). [CrossRef] [Medline]
    43. Health.gov. 2015. URL: https://health.gov/dietaryguidelines/2015/guidelines/ [accessed 2019-04-01]
    44. Centers for Disease Control and Prevention. 2017. URL: https://www.cdc.gov/nccdphp/dch/images/infographics/getmoving_15-18.png [accessed 2019-04-01]
    45. Murray CB, Palermo TM, Holmbeck GN. A multimethod, case-controlled study of sleep-wake disturbances in adolescents with spina bifida. J Pediatr Psychol. Jul 1, 2018;43(6):601-612. [FREE Full text] [CrossRef] [Medline]
    46. Centers for Disease Control and Prevention. 2017. URL: https://www.cdc.gov/sleep/about_sleep/how_much_sleep.html [accessed 2019-04-01]
    47. Sauro J. MeasuringU. 2012. URL: https://measuringu.com/errors-ux/ [accessed 2019-04-01]
    48. Norman D. The Design of Everyday Things. New York. Basic Books; 2002.
    49. Mohr DC, Schueller SM, Montague E, Burns MN, Rashidi P. The behavioral intervention technology model: an integrated conceptual and technological framework for ehealth and mhealth interventions. J Med Internet Res. Jun 5, 2014;16(6):e146. [FREE Full text] [CrossRef] [Medline]
    50. Anderson M, Jiang J. Pew Research Center. 2018. URL: https://www.pewinternet.org/2018/05/31/teens-social-media-technology-2018/ [accessed 2019-04-01]
    51. Fairman AD, Yih ET, McCoy DF, Lopresti EF, McCue MP, Parmanto B, et al. Iterative design and usability testing of the IMHere system for managing chronic conditions and disability. Int J Telerehabil. 2016;8(1):11-20. [FREE Full text] [CrossRef] [Medline]
    52. Yu DX, Parmanto B, Dicianno BE, Pramana G. Accessibility of mhealth self-care apps for individuals with spina bifida. Perspect Health Inf Manag. 2015;12:1h. [FREE Full text] [Medline]
    53. Bendixen RM, Fairman AD, Karavolis M, Sullivan C, Parmanto B. A user-centered approach: understanding client and caregiver needs and preferences in the development of mhealth apps for self-management. JMIR Mhealth Uhealth. Sep 26, 2017;5(9):e141. [FREE Full text] [CrossRef] [Medline]

    ‎
    ASQ: After-Scenario Questionnaire
    AYA: adolescents and young adults
    AYA-SB: adolescents and young adults with spina bifida
    BIT: behavioral intervention technology
    BMI: body mass index
    CDC: Centers for Disease Control and Prevention
    FSIQ: Full Scale Intelligence Quotient
    SB: spina bifida
    SUS: System Usability Scale
    YMCA: Young Men’s Christian Association

    Edited by O Perski; submitted 25.06.19; peer-reviewed by K Mateo, A Fairman, S Comulada; comments to author 02.08.19; revised version received 13.08.19; accepted 16.08.19; published 10.10.19.

    Copyright

    ©Colleen Stiles-Shields, Brittney Garcia, Kimberly Villota, Elicia Wartman, Adrien M Winning, Grayson N Holmbeck. Originally published in JMIR Pediatrics and Parenting (http://pediatrics.jmir.org), 10.10.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on http://pediatrics.jmir.org, as well as this copyright and license information must be included.