This paper is in the following e-collection/theme issue:

Parent and Caregiver Education and Behavior Change for Vaccination (74) HPV Vaccination (55) Vaccination and Immunization in the Digital Age (374) Consumer & Patient Education and Shared-Decision Making (679) Innovations and Technology for Cancer Prevention and Screening (147) Web-based and Mobile Health Interventions (3297) Mobile Apps for Cancer Care and Cancer Prevention and Screening (206) mHealth for Wellness, Behavior Change and Prevention (2871)

Published on in Vol 5, No 1 (2022): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/30340, first published .
Parents’ Experience With a Mobile Health Intervention to Influence Human Papillomavirus Vaccination Decision Making: Mixed Methods Study

Parents’ Experience With a Mobile Health Intervention to Influence Human Papillomavirus Vaccination Decision Making: Mixed Methods Study

Parents’ Experience With a Mobile Health Intervention to Influence Human Papillomavirus Vaccination Decision Making: Mixed Methods Study

Authors of this article:

Elisabeth RB Becker1 Author Orcid Image ;   Ross Shegog1 Author Orcid Image ;   Lara S Savas1 Author Orcid Image ;   Erica L Frost1 Author Orcid Image ;   Sharon P Coan1 Author Orcid Image ;   C Mary Healy2 Author Orcid Image ;   Stanley W Spinner3 Author Orcid Image ;   Sally W Vernon1 Author Orcid Image
Article Authors Cited by (8) Tweetations (4) Metrics

    Original Paper

    1University of Texas Health Science Center at Houston, Houston, TX, United States

    2Baylor College of Medicine, Houston, TX, United States

    3Texas Children's Pediatrics, Houston, TX, United States

    Corresponding Author:

    Elisabeth RB Becker, MPH, PhD

    University of Texas Health Science Center at Houston

    7000 Fannin St.

    Houston, TX, 77030

    United States

    Phone: 1 (713) 500 9677

    Email: elisabethrbecker@gmail.com


    Background: Human papillomavirus (HPV)-attributed cancers are preventable, yet HPV vaccination rates severely lag behind other adolescent vaccinations. HPVcancerFree (HPVCF) is a mobile health (mHealth) intervention developed to influence parental HPV vaccination decision making by raising awareness of HPV, reducing HPV vaccination barriers, and enabling HPV vaccination scheduling and reminders through a smartphone app. Evaluating the user experience of mHealth interventions is a vital component in assessing their quality and success but tends to be underreported in mHealth intervention evaluation.

    Objective: We aimed to evaluate the user experience of HPVCF, an HPV cancer prevention app designed for a pediatric clinic network, using mixed methods data collected from log files, survey measures, and qualitative feedback.

    Methods: Study data were evaluated from parents in a large US pediatric clinic network using HPVCF in the treatment study condition of a group randomized controlled trial. Log data captured HPVCF retention and use. Postintervention rating scales and items assessed HPVCF utility, usefulness, understandability, appeal, credibility, and perceived impact. Overall quality was evaluated using the user version of the Mobile Application Rating Scale (uMars). Open-ended responses assessed parent recommendations for HPVCF enhancement.

    Results: The 98 parents were mainly female (n=94, 96%), 41 (5.67) years of age, college educated (n=55, 56%), and White and non-Hispanic (n=55, 56%) and had private health insurance for their children (n=75, 77%). Parents used HPVCF 197 times, with the average visit duration approximating 3.5 minutes. The uMARS app quality score was positively skewed (4.2/5.0). Mean ratings were highest for information (4.46 [SD 0.53]) and lowest for engagement (3.74 [SD 0.69]). In addition, of 95 parents, 45 (47%) rated HPVCF as helpful in HPV vaccination decision making and 16 (17%) attributed HPV vaccine initiation to HPVCF. Parents reported that HPVCF increased their awareness (84/95, 88%), knowledge (84/95, 88%), and HPV vaccination intentions (64/95, 67%). Most of the 98 parents rated the 4 HPVCF components as useful (72-92 [73%-94%]). Parents also agreed that HPVCF is clear (86/95, 91%), accurate (86/95, 91%), and more helpful than other HPV vaccine information they had received (89/95, 94%) and that they would recommend it to others (81/95, 85%). In addition, parents suggested ways to increase awareness and engagement with the app, along with opportunities to enhance the content and functionality.

    Conclusions: HPVCF was well received by parents and performed well on indicators of quality, usefulness, utility, credibility, and perceived impact. This study contributes a multimethod and multimeasure evaluation to the growing body of literature focused on assessing the user experience of patient-focused technology-mediated applications for HPV education.

    JMIR Pediatr Parent 2022;5(1):e30340

    doi:10.2196/30340

    Keywords

    human papillomavirus; vaccination; user experience; parent; mHealth; HPV; vaccine; HPV vaccine; parenting; pediatrics; sexual health; cervical cancer; adolescents; mHealth; app; application 


    Background

    Human papillomavirus (HPV) is a sexually transmitted infection that causes anogenital cancers and oropharyngeal cancers in men and women [1]. HPV is attributed to 630,000 new cancer cases per year worldwide [2] and 44,000 cases per year in the United States [3]. The majority of HPV-attributed cancers can be prevented with a 2-dose 9-valent HPV vaccine [4]. HPV vaccination is recommended for adolescents 11-12 years old, but rates severely lag behind other adolescent vaccinations, such as Tdap and meningococcal vaccines [5]. The Healthy People 2030 goal for HPV series completion is 80% of youth; however, only 60% of 13-17-year-olds have initiated and 40% have completed the HPV series in the United States [6].

    National samples in the United States have found that 28% of parents have refused or decided not to get the HPV vaccine for their child and 8% of parents have delayed or put off getting the vaccine [7]. Refusal is associated with parental perceptions that the HPV vaccine is ineffective and harmful, and delay is associated with the parental need for more information [7]. Frequent reasons for HPV vaccine hesitancy also include perceptions that the vaccination is not necessary, a lack of provider recommendation, and a lack of parental knowledge [8]. Despite this, parent intervention can persuade HPV vaccination initiation, as over 85% of parents with a history of delay have reported initiating HPV vaccination or intending to do so after continued counseling and recommendation [7].

    Factors at the individual, provider, and clinic levels have been positively associated with HPV vaccination outcomes. Interventions that address parental psychosocial factors (ie, knowledge, beliefs, and outcome expectations), provider behavior (ie, HPV vaccine recommendation), patient-targeted systems (ie, reminder systems), and provider-targeted systems (ie, assessment and feedback) can positively influence HPV vaccination rates [9]. Multimethod strategies demonstrate the highest rates of maintaining increases in HPV vaccination [10].

    The use of parent- and patient-focused apps to promote HPV education and vaccination is on the rise [11-14]. Mobile health (mHealth) is rapidly becoming a dominant mode to deliver health education and health promotion interventions [15]. Evaluating the user experience of mHealth interventions is important in assessing their quality, acceptability to users, and application in real-world clinical settings [16-20] but is often underreported [21-23]. User experience is broadly defined as a person’s perceptions and responses resulting from the use or anticipated use of a product, system, or service [24]. There is a lack of consensus on the best methods, measures, or scales to use when assessing mHealth user experience, although this field is maturing, and a number of scales have been developed that focus exclusively on usability [25,26], quality [27,28], and clinically meaningful risks and benefits [29]. The most frequently evaluated domains in assessments of commercially available mHealth apps include the scientific and clinical basis, functionality, usability, accountability, impact, and popularity [23]. Nouri et al’s [30] systematic review of mHealth evaluation criteria found 7 domains that are commonly used: design, content, usability, functionality, ethical issues, security and privacy, and user-perceived value. In addition, government bodies, such as the UK National Health Service, have developed their own mHealth evaluation standards, which include usability and accessibility to ensure they meet the needs of a diverse set of users, including people with disabilities or those with limited technical knowledge, for their health app marketplace [31]. Of 18 scales developed to evaluate the quality of mHealth apps, the Mobile Application Rating Scale (MARS) is the most frequently applied and is the only scale that has a user version for evaluation by nonhealth professionals [32]. A few studies have evaluated the aspects of user experience for patient-focused technology-mediated HPV interventions, including evaluating the usability of a conversational agent for HPV vaccine counseling of parents and college students using a Wizard of Oz methodology [11,33]; evaluating the feasibility, acceptability, and usability of a cervical cancer and HPV educational virtual agent for Hispanic women [34]; and evaluating the usability of a HPV information website for parents and adolescents [35]. These usability evaluations were done as part of the formative design process and did not include users enrolled in a randomized controlled trial (RCT), using the intervention longitudinally on their own. As HPV vaccination misinformation remains a significant public health problem, there is a need for HPV education apps that are usable, useful, and scalable to motivate parents to vaccinate their adolescent children.

    mHealth Intervention: HPVcancerFree

    HPVcancerFree (HPVCF) is an iOS- and Android-compatible smartphone app designed for parents of patients aged 10-17 years who have not initiated HPV vaccination. HPVCF is part of a multilevel intervention aimed at increasing HPV vaccination initiation and completion rates in a large US pediatric clinic network [36]. HPVCF was designed to (1) raise awareness of HPV and its prevention, (2) reduce barriers to HPV vaccination, and (3) enable parents to initiate HPV vaccination scheduling and reminders through their smartphone. Preliminary findings have demonstrated potential of HPVCF in changing parental knowledge and perceptions of HPV vaccination [37].

    HPVCF was created using user-centered design principles and Intervention Mapping, an evidence- and theory-based systematic framework for developing behavior change interventions [38]. The design steps included (1) literature review and online synchronous text-based focus groups with parents from the pediatric clinic network to assess HPV attitudes, barriers, beliefs, and needs related to a digital behavior change solution [39]: (2) matrices describing target behaviors, psychosocial determinants of behavior, and change objectives; (3) delineation of theoretical methods and practical applications; (4) prototype build; (5) heuristic evaluation and in-house alpha testing; and (6) iterative user testing to assess app content, function, delivery channel, usability, value, desirability, and adoptability for both design and content.

    HPVCF contains 4 self-tailored components: (1) HPV A-Z, a compendium of 9 content domains providing facts about HPV and the HPV vaccine; (2) Bust-a-Myth, 7 educational modules, including peer and health care provider testimonials addressing the most salient HPV vaccination barriers; (3) Notes 4 Doc, a medium to facilitate communication with health care providers about the HPV vaccine; and (4) Get the Vax, a feature to schedule HPV vaccination appointments and receive tailored reminders (Figure 1). There were 77 app pages and links that parents had unlimited access to. HPVCF was designed for user-centric navigation and so did not prescribe an intended user path.

    The purpose of this study is to evaluate the user experience of HPVCF, an HPV cancer prevention app designed for a pediatric clinic network, using mixed methods data collected from log files, survey measures, and qualitative feedback.

    ‎
    Figure 1. HPVcancerFree (HPVCF) components. HPV: human papillomavirus.
    View this figure

    Study Design and Ethics Approval

    Study data were drawn from parents who used HPVCF in a group RCT assessing HPVCF effectiveness within a large pediatric clinic network in Texas, USA, and completed a postintervention experience survey [37]. The study occurred between September 2017 and March 2019, where the 51 network clinics were randomized to either the treatment (HPVCF with usual care) or a comparison (usual care only) study condition. Parents in the 26 treatment clinics represented the analytic sample for this user experience evaluation. These parents were given instruction and links to download HPVCF from the Apple App Store (iPhone users) or Google Play Store (Android users). They were given a personal ID to enter the first time they launched HPVCF for tracking purposes. Study protocols were approved by the institutional review board at the University of Texas Health Science Center at Houston (HSC-SPH-15-0202).

    Study Inclusion Criteria and Recruitment

    Eligibility for the study included (1) having a 10-to-17-year-old child who was a patient in the clinic network, (2) having a child that had not initiated HPV vaccination, and (3) the ability to speak and write in English. Parents who had an eligible child were invited to participate in the study via patient health record portal invitations, flyers in the clinic waiting rooms, and posts on the clinic network Facebook page. Recruitment for the study took place on a rolling basis from September 2017 to September 2018. Each parent participated in the intervention for 5 months between September 2017 and March 2019, depending on when they were recruited and enrolled. Parent completed a presurvey before they were given access to the intervention and a postsurvey, which included an experience assessment, at the conclusion of their intervention time frame.

    HPVCF Onboarding, Use, and Retention

    HPVCF use data were gathered over the course of the 5-month intervention from log files, including total number of visits, number of visits per participant, actions (viewing an app page or link) per visit, and visit duration. A back-end data capture system (Matomo) [40] collected time-stamped use by participant. Actions were triggered when the parent visited a new app page.

    Experience

    HPVCF user experience was assessed with a postintervention survey using a quality rating scale, survey items, and an open-ended response item for recommended enhancements. The survey items included utility, perceived impact, component usefulness, clarity, credibility, and motivational appeal.

    App Quality

    App quality was assessed by the user version of the Mobile Application Rating Scale (uMARS) [28]. uMARS is a reliable mHealth quality measure comprising 3 separate components: an app quality mean score, an app subjective quality scale, and perceived impact items. The app quality mean score contains 16 items evaluating 4 subscales: engagement (5 items), functionality (4 items), aesthetics (3 items), and information (4 items) on a 5-point response from 1 for “inadequate” to 5 for “excellent” and N/A if an app component is not used. uMARS has consistent internal consistency (Cronbach α=.90) for all subscales (engagement α=.80; functionality α=.70; aesthetics α=.71; information α=.78) [28]. The app quality score was calculated by averaging the combined scores for each of the 4 subscales (engagement, functionality, aesthetics, and information).

    Survey Items

    Utility

    Two utility items assessed whether HPVCF information helped parents decide to get the HPV vaccine for their child (no, yes, no opinion) and whether parents got their child the HPV vaccine as a result of using HPVCF (no, yes, no opinion). These items were adapted from prior surveys used with patient-focused digital behavior change interventions in clinic and school settings [41-43].

    Perceived Impact

    Perceived impact was assessed with 5 items on user perceptions of HPVCF. These modified perceived impact uMARS items measured perceptions of change in awareness of HPV and the HPV vaccine, knowledge of HPV and the HPV vaccine, attitudes of HPV and the HPV vaccine, intentions to get their child the HPV vaccine, and communication with the child’s pediatrician about the HPV vaccine. These items were evaluated on a 4-point scale with response options “strongly disagree,” “somewhat disagree,” “somewhat agree,” and “strongly agree” [28]. For analysis, “somewhat agree” and “strongly agree” response options were collapsed into an “agreement” category.

    Usefulness

    Usefulness was assessed using ratings of 4 HPVCF components (HPV A-Z, Bust-a-Myth, Notes 4 Doc, and Get the Vax) with response options “did not use,” “not very useful,” “somewhat useful,” “very useful,” and “do not recall.” For analysis, “very useful” and “somewhat useful” were collapsed into an “agreement” category.

    Clarity

    Clarity was assessed with a single item on whether the goal of HPVCF was clear (no, yes, no opinion).

    Credibility

    Credibility was assessed using 1 rating of accuracy of HPVCF content (inaccurate, accurate, no opinion) and 1 rating of trustworthiness of HPVCF information (cannot be trusted, can be trusted, no opinion).

    Motivational Appeal

    Motivational appeal was assessed using 3 ratings: whether parents would use HPVCF again (no, yes, no opinion), a comparison of the helpfulness of HPVCF content against other HPV content received (less helpful, as helpful, more helpful), and the extent to which parents would recommend HPVCF to others who might benefit from it (few people, several people, many people, everyone). These items were adapted from prior surveys used with patient-focused digital behavior change interventions in clinic and school settings [41-43].

    Recommended Enhancements

    Recommended enhancements were solicited from an open-ended question, “What would make the HPVCF app more appealing so that parents would want to use it?” adapted from prior surveys used with patient-focused digital behavior change interventions in clinic and school settings [41-43].

    Demographics

    Parent sociodemographic variables were gathered from preintervention survey items at the start of the 5-month intervention. The parent sociodemographic variables included age, number of adolescent children, sex, race, ethnicity, education, child’s health insurance status, and baseline HPV vaccination intention.


    HPVCF Onboarding, Use, and Retention

    In total, 168 parents completed the postintervention survey, of whom 98 (58.3%) were included in this experience analysis as they also downloaded and used the intervention (viewed at least 1 page past the home screen on any visit; Figure 2).

    Parents had a mean age of 41 years, and the majority were female (94/98, 96%), college graduates (55/98, 56%), and White and non-Hispanic (55/98, 56%) and had private health insurance for their children (75/98, 77%); see Table 1. Most parents had 1 child between 10 and 17 years of age, with a range of 1-4 children in that age group. At baseline, of 98 parents, 12 (12%) reported that they “don’t intend” to vaccinate their child for HPV, 40 (41%) “definitely” planned to, and 46 (46%) were unsure (“haven’t thought of it,” “considering,” and “will probably get”). These demographics reflected the RCT sample where parents had a mean age of 41 years, were majority female (358/375, 95.5%), were college graduates (233/375, 62.1%), and identified as White and non-Hispanic (210/375, 56%). Further, these results approximate the demographic characteristics of the clinic network population where among children 10-17 years old, 45% are White and non-Hispanic and 80% have private health insurance.

    Parents visited HPVCF 197 times during the study period (Table 2). Most parents used HPVCF once (45/98, 46%) or twice (28/98, 29%) with a range of 1-8 visits. During a single visit, 2-84 actions occurred with a mode of 3 actions. The average visit duration was 3 minutes and 27 seconds with a mode of 24 seconds and a range from 3 seconds to just under 27 minutes. Of the 4 HPVCF main component pages, HPV A-Z (370 views) was visited most often by parents, followed by Bust-a-Myth (273 views), Get the Vax (173 views), and Notes 4 Doc (110 views).

    ‎
    Figure 2. Recruitment and retention. HPV: human papillomavirus; HPVCF: HPVcancerFree.
    View this figure
    Table 1. Parent demographics (N=98).
    CharacteristicValue
    Age (years), mean (SD); range41.23 (5.67); 26-54
    Number of adolescent childrena, mean (SD); range; mode1.42 (0.62); 1-4; 1
    Parent sex, n (%)

    		Male4 (4)

    		Female94 (96)
    Parent race and ethnicity, n (%)

    		White, non-Hispanic55 (56)

    		Black or African American, non-Hispanic7 (7)

    		Hispanic30 (31)

    		Asian4 (4)

    		Other2 (2)
    Parent education, n (%)

    		Some high school1 (1)

    		 High school graduate or General Educational Development (GED)7 (7)

    		Some college35 (36)

    		College graduate25 (25)

    		Graduate or professional degree30 (31)
    Child/children’s health insurance statusb, n (%)

    		Private health insurance75 (77)

    		Medicaid/Medicare/State Children\'s Health Insurance Program20 (20)

    		Uninsured; no coverage of any type3 (3)
    Parent baseline HPVc vaccination intention, n (%)

    		Haven’t thought of it8 (8)

    		Considering19 (19)

    		Will probably get19 (19)

    		Definitely40 (41)

    		Don’t intend12 (12)

    a10-17 years old.

    bResponse options are inclusive.

    cHPV: human papillomavirus.

    Table 2. HPVcancerFree (HPVCF) use (N=98).
    Visit detailsValue
    Total number of separate visitsa (all parents)197
    Number of visitsa per parent, mean (SD); range2 (1.25); 1-8
    Distribution of parent visits, n (%)

    		1 visit45 (46)

    		2 visits28 (29)

    		3 visits13 (13)

    		4 visits10 (10)

    		5 visits2 (2)

    		8 visits1 (1)
    Actionsb per visit, mean (SD); range; mode11 (10); 2-84; 3
    Visit duration (seconds), mean (SD); range; mode207 (249); 3-1601; 24
    Total views by main component pagec, n

    		HPVd A-Z370

    		Bust-a-Myth273

    		Notes 4 Doc110

    		Get the Vax173

    aA visit was defined as viewing at least 1 page past the home screen.

    bAn action was defined as viewing an app page or link. There were 77 app pages and links in total, which could be viewed unlimited times.

    cThe main component pages could be visited unlimited times while visiting the app.

    dHPV: human papillomavirus.

    App Quality

    The uMARS app quality rating was 4.2/5.0 (Table 3). The uMARS information subscale had the highest mean rating (4.46 [SD 0.53]), followed by functionality (4.32 [SD 0.65]), aesthetics (4.30 [SD 0.57]), and engagement (3.74 [SD 0.69]).

    Table 3. User version of the Mobile Application Rating Scale (uMARS) mean scores (N=95).
    SubscaleaMean (SD)
    Engagement (5 items)3.74 (0.69)
    Functionality (4 items)4.32 (0.65)
    Aesthetics (3 items)4.30 (0.57)
    Information (4 items)4.46 (0.53)
    Overall qualityb (from subscales)4.20 (0.48)

    aItems in the subscale measured on a 5-point response scale from 1 for “inadequate” to 5 for “excellent” and N/A if an app component was not used.

    bCalculated by averaging the combined scores for each of the 4 subscales (engagement, functionality, aesthetics, and information).

    Utility, Perceived Impact, Usefulness, Clarity, Credibility, and Appeal
    Utility

    Overall, 45 (47%) of 95 parents rated HPVCF as helping them decide to get their child the HPV vaccine, and 16 (17%) responded that they got their child the HPV vaccine as a result of HPVCF (Table 4).

    Table 4. Parent agreement on utility, perceived impact, usefulness, clarity, credibility, and appeal (N=95).
    User experience survey parameterAgreement, n (%)
    Utilitya

    		The information I got from HPVCFb helped me decide to get my child the HPVc vaccine.45 (47)

    		I got my child the HPV vaccine as a result of using the HPVCF app.16 (17)
    Perceived impact of HPVCFd

    		Increased my awareness of HPV and HPV vaccine.84 (88)

    		Increased my knowledge of HPV and HPV vaccine.84 (88)

    		Changed my attitudes of HPV and HPV vaccine.54 (57)

    		Increased my intentions to get my child the HPV vaccine.64 (67)

    		Encouraged me to talk to my child’s pediatrician about the HPV vaccine.65 (68)
    Usefulness by componente

    		HPV A-Z92 (94)

    		Bust-a-Myth88 (90)

    		Notes 4 Doc72 (73)

    		Get the Vax73 (75)
    Clarityc

    		The goal/purpose of the HPVCF app was clear.86 (91)
    Credibility

    		I think information I got from the HPVCF app was accurate.f86 (91)

    		I think the information I got from the HPVCF app can be trusted.g85 (90)
    Appeal

    		I would use HPVCF again.c63 (66)

    		Compared to other information I have seen about the HPV the HPVCF app is as or more helpful.h89 (94)

    		I would recommend HPVCF to others.d81 (85)

    aResponded “yes” as opposed to “no” or “no opinion.”

    bHPVCF: HPVcancerFree.

    cHPV: human papillomavirus.

    dIncludes “somewhat agree” and “strongly agree” response options.

    eN=98; combined responses of “very useful” and “somewhat useful.” Combined percentage responses of “did not use” and “do not recall” were as follows: HPV A-Z (5/98, 5%), Bust-a-Myth (5/98, 5%), Notes 4 Doc (19/98, 19%), and Get the Vax (20/98, 20%).

    fRated as “accurate” as opposed to “inaccurate” or “no opinion.”

    gRated as “can be trusted” as opposed to “cannot be trusted” or “no opinion.”

    hRated as “as helpful” or “more helpful” as opposed to “less helpful.”

    Perceived Impact

    Most parents (64/95, 67%) agreed that HPVCF increased their intentions to get their child the HPV vaccine. Parents reported that HPVCF positively impacted their awareness (84/95, 88%), knowledge (84/95, 88%), and attitudes (54/95, 57%) about HPV and the HPV vaccine and encouraged them to discuss the HPV vaccine with their child’s pediatrician (65/95, 68%); see Table 4.

    Component Usefulness

    Most parents rated the 4 HPVCF components as useful (Table 4). HPV A-Z (92/98, 94%) scored the highest, followed by Bust-a-Myth (88/98, 90%), Get the Vax (73/98, 75%), and Notes 4 Doc (72/98, 73%). Most parents used Bust-a-Myth and HPV A-Z (both ≥95%); however, 20 (20%) and 19 (19%) of 98 parents reported not using the Get the Vax and Notes 4 Doc components, respectively.

    Clarity, Credibility, and Appeal

    The majority of parents rated the purpose of HPVCF as clear (86/95, 91%) and that the information in HPVCF was accurate (86/95, 91%) and can be trusted (85/95, 89%). Parents also agreed that they would use HPVCF again (63/95, 66%), that it was more helpful than other information they had seen about HPV and the HPV vaccine (89/95, 94%), and that they would recommend HPVCF to others (81/95, 85%); see Table 4.

    Recommended Enhancements

    Qualitative feedback gathered on how to improve HPVCF included themes of increasing awareness and engagement with the app and enhancing the content and functionality.

    Increasing Awareness and Engagement

    Parents commented that they forgot to use the app after the initial download and needed a reminder from the app, pediatrician, or clinic to use HPVCF:

    It's been a long time since I used the app, so I don't remember if there was a function to remind the user in the future to make an appointment etc. That would prompt the user to reopen the app. I read all the information the first time I opened it and didn't open it again, so I've forgotten much of it now.

    Due to a lack of repeated engagement, some parents reported they had forgotten much of the information they had originally reviewed. Parents suggested improving engagement by having push notifications with HPV facts instead of having to open and use the app to obtain information. Further, to improve marketability, one parent recommended incorporating HPVCF content into a broader app that included topics outside of HPV.

    Parents also suggested making HPVCF more interactive and entertaining, especially by providing opportunities to engage and speak with adolescents about HPV and sexual health. To make HPVCF more adolescent friendly, parents suggested adding animations and games.

    Enhancing Content and Functionality

    To improve the content, parents suggested offering parent testimonials highlighting their struggle to decide to vaccinate and how they used HPVCF to make an informed decision:

    I think offering parent testimonials about their struggle to decide to vaccinate and how they used the info offered to help make an educated decision.

    Some parents were dissatisfied with the presentation of HPV’s long-term effects, expecting to see more in-depth information and studies about complications and side effects of the vaccine:

    This app, as a parent, did not give me the type of information I would want and need about HPV. I would prefer more information on the age of the vaccine. Credible studies completed. New research and side effects. More of that information would help. If the vaccine is less than 10 years old, I want more information on studies.

    Parents wanted a tailored reminder system that conveyed information regarding their child’s HPV vaccination status and recommendation:

    [I would suggest] some type of electronic reminder from the doctor to review the app. I downloaded the app for the survey, reviewed it, but then forgot about it. Maybe the doctor's office can send email to parents of 9-, 10-, and 11-year-old patients. Also, [a] reminder should have information on [the] child's status (ie, for informational purposes only; needs first dose; received first dose, time for second; etc).

    Parents noted some issues with the available functionality, such as the app not storing their appointment information caused by usability issues with the design. For future iterations, parents suggested the ability to share information in the app with existing social media outlets:

    [It would be helpful if the app would] allow for flagging/sharing individual items to foster organic awareness through existing social outlets?

    Principal Findings, Strengths, and Limitations

    This study evaluated the user experience of HPVCF, an HPV cancer prevention app designed for parents with children belonging to a large urban pediatric clinic network in the United States. Parents viewed HPVCF as having high quality, utility, and perceived impact. HPVCF quality ratings were robust (4.2) compared to quality scores of 2.4-4.6 for mHealth apps focused on prostate cancer risk [44], Alzheimer disease [45], alcohol use [46], occupational therapy [47], orthopedic rehabilitation [48], and medication adherence [49]. HPVCF quality was rated the highest for information and lowest for engagement subscales. This is consistent with the parents’ perceptions of HPVCF information as accurate and trustworthy, while also acknowledging the need to enhance functions and features. The positively skewed quality rating is consistent with a considerable number of parents (16/95, 17%) attributing their child’s HPV vaccination to HPVCF and the majority of parents (64/95, 67%) attributing HPVCF to increasing their intentions to vaccinate their child. This is promising, considering that the period of participation in the study was only 5 months, which is inadequate to fully track vaccinations through annual well visits. The study design was insufficient to determine whether these perceptions and vaccinations were significantly different from trends in parents not exposed to HPVCF, but it does appear that HPVCF was sufficiently persuasive to move at least a sample of parents to action. It is also unclear whether, in these instances, HPVCF was directly associated with vaccination or mediated through greater pediatrician dialogue that promoted vaccination.

    Interestingly, 63 (66%) of 95 parents reported that they would use the app again, but log data indicated that 45 (46%) of 98 parents only used the app once. Triangulating these findings with information obtained in the qualitative feedback suggests possible reasons for this, including forgetting about the app after initial download, only using HPVCF on an as-needed basis, or no longer needing the app since it fulfilled its intended use after a single visit. Future iterations could be strengthened to help parents reengage by utilizing pediatricians and clinic staff to incorporate reminders as part of their standard communication. Exploring adjunct functionality that offers HPVCF information in more compact ways (ie, push notifications, text messages) may prove beneficial as most parents only looked at a few pages or links for a brief time (under 3.5 minutes).

    Parents perceived HPVCF as more impactful for increasing awareness and knowledge than in changing attitudes about HPV. Knowledge is necessary but not necessarily sufficient to elicit behavior change. Negative attitudes around HPV vaccine safety are particularly pervasive [9], and strategies that are personalized, tailored, and require engagement beyond passive education may be needed to modify attitudes [50]. Concerns about HPV vaccination may have extended to HPVCF itself, with some parents feeling that HPVCF is biased in its portrayal of the long-term risks and safety of the HPV vaccine, despite high credibility ratings. A further behavioral impact was that the majority of parents also reported that HPVCF prompts greater communication with their pediatrician. This is an important adjunct function that helps the parent engage more competently with the pediatrician and provides the pediatrician with an opportunity to educate the parent during “teachable moments” at the clinic visit.

    Importantly, parent engagement with the intervention was low, with about 70 (41.7%) of 168 parents choosing not to use HPVCF, making it difficult to generalize results to the clinic network. The low engagement is partially a reflection of the real-world nature of the study and accompanying challenges of competing for attention in an open market. Future iterations might adopt a more assertive approach by having parents download HPVCF during clinic visits and having clinic staff be more involved in providing reminders for its use. Future studies can explore promotional strategies to motivate parents to use HPVCF.

    Additional limitations of this study should be considered. The intervention timeline did not include the back-to-school vaccination period (generally June-September) for many adolescent children, as the intervention took place on a 5-month rolling basis over the course of 1.5 years. This may have affected the parents’ decision to get their adolescents vaccinated. The 5-month intervention timeline also meant that some parents answered postintervention questions weeks after using HPVCF, possibly affecting their ability to accurately recall and report on some survey measures. A further limitation of the study was that the English-only content and the smartphone-based application may have excluded participation from parents with lower socioeconomic status or those who do not speak English. Although smartphone ownership among Americans is high (85% White, 85% Hispanic, 83% Black), there are disparities between Americans who are college educated (93%) and those with a high school diploma or less (75%) [51], and health app usage among low-income, racial minority, and ethnic minority patients in Texas remains low [52]. Future research addressing non-English-speaking and populations with a lower socioeconomic status is recommended. Finally, this analysis did not examine user experience by participant characteristics. However, subsequent analyses explored content-specific patterns of use that underlie psychosocial characteristics of parents [53].

    As digital technologies continue to evolve, they stand to provide a paradigmatic shift in how health education and health behavior research are conducted [54], but doing so will require them to be perceived by users as being usable and useful applications. This study contributes a multimethod and multimeasure evaluation to the growing body of literature focused on assessing the user experience of patient-focused technology-mediated applications for HPV education [11,33-35].

    Conclusion

    HPVCF was well received by parents and performed well on indicators of quality, usefulness, utility, credibility, and perceived impact. HPVCF contributes to a multimethod and multimeasure evaluation strategy for user experience, which remains underreported in mHealth apps.

    Acknowledgments

    The authors thank the parents who participated in the study for their time and valuable insights. This research was supported by the Cancer Prevention and Research Institute of Texas (RP150014).

    Conflicts of Interest

    None declared.

    1. Saraiya M, Unger ER, Thompson TD, Lynch CF, Hernandez BY, Lyu CW, HPV Typing of Cancers Workgroup. US assessment of HPV types in cancers: implications for current and 9-valent HPV vaccines. J Natl Cancer Inst 2015 Jun;107(6):djv086 [FREE Full text] [CrossRef] [Medline]
    2. de Martel C, Plummer M, Vignat J, Franceschi S. Worldwide burden of cancer attributable to HPV by site, country and HPV type. Int J Cancer 2017 Aug 15;141(4):664-670 [FREE Full text] [CrossRef] [Medline]
    3. Cancers Associated with Human Papillomavirus, United States, 2012-2016.   URL: https://www.cdc.gov/cancer/uscs/about/data-briefs/no10-hpv-assoc-cancers-UnitedStates-2012-2016.htm [accessed 2022-02-13]
    4. Senkomago V, Henley SJ, Thomas CC, Mix JM, Markowitz LE, Saraiya M. Human papillomavirus-attributable cancers: United States, 2012-2016. MMWR Morb Mortal Wkly Rep 2019 Aug 23;68(33):724-728 [FREE Full text] [CrossRef] [Medline]
    5. Walker TY, Elam-Evans LD, Yankey D, Markowitz LE, Williams CL, Fredua B, et al. National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2018. MMWR Morb Mortal Wkly Rep 2019 Aug 23;68(33):718-723 [FREE Full text] [CrossRef] [Medline]
    6. Walker TY, Elam-Evans LD, Singleton JA, Yankey D, Markowitz LE, Fredua B, et al. National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2016. MMWR Morb Mortal Wkly Rep 2017 Aug 25;66(33):874-882 [FREE Full text] [CrossRef] [Medline]
    7. Gilkey M, Calo W, Marciniak M, Brewer N. Parents who refuse or delay HPV vaccine: differences in vaccination behavior, beliefs, and clinical communication preferences. Hum Vaccin Immunother 2017 Mar 04;13(3):680-686 [FREE Full text] [CrossRef] [Medline]
    8. Hirth JM, Fuchs EL, Chang M, Fernandez ME, Berenson AB. Variations in reason for intention not to vaccinate across time, region, and by race/ethnicity, NIS-Teen (2008-2016). Vaccine 2019 Jan 21;37(4):595-601 [FREE Full text] [CrossRef] [Medline]
    9. Rodriguez S, Mullen P, Lopez D, Savas L, Fernández ME. Factors associated with adolescent HPV vaccination in the U.S.: a systematic review of reviews and multilevel framework to inform intervention development. Prev Med 2020 Feb;131:105968 [FREE Full text] [CrossRef] [Medline]
    10. Holloway GL. Effective HPV vaccination strategies: what does the evidence say? An integrated literature review. J Pediatr Nurs 2019;44:31-41. [CrossRef] [Medline]
    11. Amith M, Zhu A, Cunningham R, Lin R, Savas L, Shay L, et al. Early usability assessment of a conversational agent for HPV vaccination. Stud Health Technol Inform 2019;257:17-23 [FREE Full text] [Medline]
    12. Ruiz-López T, Sen S, Jakobsen E, Tropé A, Castle P, Hansen B, et al. FightHPV: design and evaluation of a mobile game to raise awareness about human papillomavirus and nudge people to take action against cervical cancer. JMIR Serious Games 2019 Apr 08;7(2):e8540 [FREE Full text] [CrossRef] [Medline]
    13. Baldwin AS, Denman DC, Sala M, Marks EG, Shay LA, Fuller S, et al. Translating self-persuasion into an adolescent HPV vaccine promotion intervention for parents attending safety-net clinics. Patient Educ Couns 2017 Apr;100(4):736-741 [FREE Full text] [CrossRef] [Medline]
    14. Gockley AA, Pena N, Vitonis A, Welch K, Duffey-Lind EC, Feldman S. Tablet-based patient education regarding human papillomavirus vaccination in colposcopy clinic. J Low Genit Tract Dis 2019 Apr 10;23(3):188-192. [CrossRef]
    15. Marcolino MS, Oliveira JAQ, D'Agostino M, Ribeiro AL, Alkmim MBM, Novillo-Ortiz D. The impact of mHealth interventions: systematic review of systematic reviews. JMIR Mhealth Uhealth 2018 Jan 17;6(1):e23 [FREE Full text] [CrossRef] [Medline]
    16. Sucala M, Ezeanochie N, Cole-Lewis H, Turgiss J. An iterative, interdisciplinary, collaborative framework for developing and evaluating digital behavior change interventions. Transl Behav Med 2020 Dec 31;10(6):1538-1548 [FREE Full text] [CrossRef] [Medline]
    17. Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (integrate, design, assess, and share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behavior. J Med Internet Res 2016 Dec 16;18(12):e317 [FREE Full text] [CrossRef] [Medline]
    18. Patel S, Arya M. The BUS framework: a comprehensive tool in creating an mHealth app utilizing behavior change theories, user-centered design, and social marketing. J Mob Technol Med 2017 Apr;6(1):39-45 [FREE Full text] [CrossRef] [Medline]
    19. Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al. Evaluating digital health interventions: key questions and approaches. Am J Prev Med 2016 Nov;51(5):843-851 [FREE Full text] [CrossRef] [Medline]
    20. Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res 2015 Jan 30;17(1):e30 [FREE Full text] [CrossRef] [Medline]
    21. Floch J, Vilarinho T, Zettl A, Ibanez-Sanchez G, Calvo-Lerma J, Stav E, et al. Users' experiences of a mobile health self-management approach for the treatment of cystic fibrosis: mixed methods study. JMIR Mhealth Uhealth 2020 Jul 08;8(7):e15896 [FREE Full text] [CrossRef] [Medline]
    22. Oliver-Williams C, Brown E, Devereux S, Fairhead C, Holeman I. Using mobile phones to improve vaccination uptake in 21 low- and middle-income countries: systematic review. JMIR Mhealth Uhealth 2017 Oct 04;5(10):e148 [FREE Full text] [CrossRef] [Medline]
    23. Grundy Q, Wang Z, Bero L. Challenges in assessing mobile health app quality: a systematic review of prevalent and innovative methods. Am J Prev Med 2016 Dec;51(6):1051-1059. [CrossRef] [Medline]
    24. ISO 9241-210:2010 Ergonomics of Human-System Interaction—Part 210: Human-Centred Design for Interactive Systems.   URL: https://www.iso.org/standard/52075.html [accessed 2022-02-13]
    25. Zhou L, Bao J, Setiawan IMA, Saptono A, Parmanto B. The mHealth App Usability Questionnaire (MAUQ): development and validation study. JMIR Mhealth Uhealth 2019 Apr 11;7(4):e11500 [FREE Full text] [CrossRef] [Medline]
    26. Schnall R, Cho H, Liu J. Health Information Technology Usability Evaluation Scale (Health-ITUES) for usability assessment of mobile health technology: validation study. JMIR Mhealth Uhealth 2018 Jan 05;6(1):e4 [FREE Full text] [CrossRef] [Medline]
    27. Stoyanov S, Hides L, Kavanagh D, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27 [FREE Full text] [CrossRef] [Medline]
    28. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the user version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016 Jun 10;4(2):e72 [FREE Full text] [CrossRef] [Medline]
    29. Levine DM, Co Z, Newmark LP, Groisser AR, Holmgren AJ, Haas JS, et al. Design and testing of a mobile health application rating tool. NPJ Digit Med 2020 May 21;3(1):74 [FREE Full text] [CrossRef] [Medline]
    30. Nouri R, Niakan Kalhori SR, Ghazisaeedi M, Marchand G, Yasini M. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inform Assoc 2018 Aug 01;25(8):1089-1098 [FREE Full text] [CrossRef] [Medline]
    31. NHS Digital: Digital Assessment Questionnaire V2.1.   URL: https://developer.nhs.uk/wp-content/uploads/2018/09/Digital-Assessment-Questions-V2.1-Beta-PDF.pdf [accessed 2022-02-13]
    32. Azad-Khaneghah P, Neubauer N, Miguel Cruz A, Liu L. Mobile health app usability and quality rating scales: a systematic review. Disabil Rehabil Assist Technol 2021 Oct;16(7):712-721. [CrossRef] [Medline]
    33. Amith M, Lin R, Cunningham R, Wu QL, Savas LS, Gong Y, et al. Examining potential usability and health beliefs among young adults using a conversational agent for HPV vaccine counseling. AMIA Jt Summits Transl Sci Proc 2020;2020:43-52 [FREE Full text] [Medline]
    34. Mendu S, Boukhechba M, Gordon J, Datta D, Molina E, Arroyo G, et al. Design of a culturally-informed virtual human for educating Hispanic women about cervical cancer. Int Conf Pervasive Comput Technol Healthc 2018 May;2018:360-366 [FREE Full text] [CrossRef] [Medline]
    35. Starling R, Nodulman JA, Kong AS, Wheeler CM, Buller DB, Woodall WG. Usability testing of an HPV information website for parents and adolescents. Online J Commun Media Technol 2015 Oct;5(4):184-203 [FREE Full text] [Medline]
    36. Vernon S, Savas L, Shegog R, Healy C, Frost E, Coan S, et al. Increasing HPV vaccination in a network of pediatric clinics using a multi-component approach. J Appl Res Child 2019;10(2):11 [FREE Full text] [Medline]
    37. Shegog R, Savas L, Healy C, Frost E, Spinner S, Vernon S. HPVcancerFree (HPVCF): a cluster randomized trail to examine a digital behavior change intervention to impact parent decision making about HPV vaccine. 2021 Presented at: National Foundation for Infectious Diseases Annual Conference on Vaccinology Research; April 2021; Virtual.
    38. Bartholomew LK, Parcel GS, Kok G, Gottlieb NH, Fernandez ME. Planning Health Promotion Programs: An Intervention Mapping Approach. San Francisco: Jossey-Bass; 2011.
    39. Becker E, Shegog R, Savas LS, Frost EL, Healy CM, Spinner SW, et al. Informing content and feature design of a parent-focused human papillomavirus vaccination digital behavior change intervention: synchronous text-based focus group study. JMIR Form Res 2021 Nov 19;5(11):e28846 [FREE Full text] [CrossRef] [Medline]
    40. Matomo.   URL: https://tinyurl.com/2p98zekw [accessed 2022-02-13]
    41. Shegog R, Peskin M, Markham C, Thiel M, Karny E, Addy R, et al. It's your game-tech: toward sexual health in the digital age. Creat Educ 2014 Aug;5(15):1428-1447 [FREE Full text] [CrossRef] [Medline]
    42. Shegog R, Markham C, Peskin M, Dancel M, Coton C, Tortolero S. "It's your game": an innovative multimedia virtual world to prevent HIV/STI and pregnancy in middle school youth. Stud Health Technol Inform 2007;129(Pt 2):983-987. [Medline]
    43. Begley C, Chong J, Shegog R, Sepulveda R, Halavacs N, Addy R, et al. MINDSET: clinical feasibility of utilizing the revised epilepsy self-management tool for Spanish speaking patients. Epilepsy Behav 2018 Nov;88:218-226. [CrossRef] [Medline]
    44. Adam A, Hellig J, Perera M, Bolton D, Lawrentschuk N. 'Prostate Cancer Risk Calculator' mobile applications (apps): a systematic review and scoring using the validated user version of the Mobile Application Rating Scale (uMARS). World J Urol 2018 Apr;36(4):565-573. [CrossRef] [Medline]
    45. Choi SK, Yelton B, Ezeanya VK, Kannaley K, Friedman DB. Review of the content and quality of mobile applications about Alzheimer's disease and related dementias. J Appl Gerontol 2020 Jun;39(6):601-608 [FREE Full text] [CrossRef] [Medline]
    46. Hides L, Quinn C, Cockshaw W, Stoyanov S, Zelenko O, Johnson D, et al. Efficacy and outcomes of a mobile app targeting alcohol use in young people. Addict Behav 2018 Feb;77:89-95. [CrossRef] [Medline]
    47. LeBeau K, Huey LG, Hart M. Assessing the quality of mobile apps used by occupational therapists: evaluation using the user version of the Mobile Application Rating Scale. JMIR Mhealth Uhealth 2019 May 01;7(5):e13019 [FREE Full text] [CrossRef] [Medline]
    48. Argent R, Slevin P, Bevilacqua A, Neligan M, Daly A, Caulfield B. Wearable sensor-based exercise biofeedback for orthopaedic rehabilitation: a mixed methods user evaluation of a prototype system. Sensors (Basel) 2019 Jan 21;19(2):A [FREE Full text] [CrossRef] [Medline]
    49. Carmody JK, Denson LA, Hommel KA. Content and usability evaluation of medication adherence mobile applications for use in pediatrics. J Pediatr Psychol 2019 Apr 01;44(3):333-342 [FREE Full text] [CrossRef] [Medline]
    50. Hekler E, Tiro J, Hunter C, Nebeker C. Precision health: the role of the social and behavioral sciences in advancing the vision. Ann Behav Med 2020 Nov 01;54(11):805-826 [FREE Full text] [CrossRef] [Medline]
    51. Mobile Fact Sheet.   URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2022-02-13]
    52. Kumar D, Hemmige V, Kallen MA, Giordano TP, Arya M. Mobile phones may not bridge the digital divide: a look at mobile phone literacy in an underserved patient population. Cureus 2019 Feb 20;11(2):e4104 [FREE Full text] [CrossRef] [Medline]
    53. Becker ERB, Myneni S, Shegog R, Fujimoto K, Savas LS, Frost EL, et al. Parent engagement with a self-tailored cancer prevention digital behaviour change intervention: exploratory application of affiliation network analysis. In: Studies in Health Technology and Informatics (forthcoming). Amsterdam: IOS Press; 2021.
    54. Riley WT, Oh A, Aklin WM, Wolff-Hughes DL. National Institutes of Health support of digital health behavior research. Health Educ Behav 2019 Dec;46(2_suppl):12-19. [CrossRef] [Medline]

    ‎
    HPV: human papillomavirus
    HPVCF: HPVcancerFree
    mHealth: mobile health
    RCT: randomized controlled trial
    uMARS: user version of the Mobile Application Rating Scale

    Edited by S Badawy; submitted 11.05.21; peer-reviewed by O Ilozumba, G Sabben, T Ntalindwa; comments to author 27.07.21; revised version received 25.10.21; accepted 17.11.21; published 21.02.22

    Copyright

    ©Elisabeth RB Becker, Ross Shegog, Lara S Savas, Erica L Frost, Sharon P Coan, C Mary Healy, Stanley W Spinner, Sally W Vernon. Originally published in JMIR Pediatrics and Parenting (https://pediatrics.jmir.org), 21.02.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.