Human papillomavirus (HPV) is a sexually transmitted infection that causes anogenital cancers and oropharyngeal cancers in men and women [1]. HPV is attributed to 630,000 new cancer cases per year worldwide [2] and 44,000 cases per year in the United States [3]. The majority of HPV-attributed cancers can be prevented with a 2-dose 9-valent HPV vaccine [4]. HPV vaccination is recommended for adolescents 11-12 years old, but rates severely lag behind other adolescent vaccinations, such as Tdap and meningococcal vaccines [5]. The Healthy People 2030 goal for HPV series completion is 80% of youth; however, only 60% of 13-17-year-olds have initiated and 40% have completed the HPV series in the United States [6].

National samples in the United States have found that 28% of parents have refused or decided not to get the HPV vaccine for their child and 8% of parents have delayed or put off getting the vaccine [7]. Refusal is associated with parental perceptions that the HPV vaccine is ineffective and harmful, and delay is associated with the parental need for more information [7]. Frequent reasons for HPV vaccine hesitancy also include perceptions that the vaccination is not necessary, a lack of provider recommendation, and a lack of parental knowledge [8]. Despite this, parent intervention can persuade HPV vaccination initiation, as over 85% of parents with a history of delay have reported initiating HPV vaccination or intending to do so after continued counseling and recommendation [7].

Factors at the individual, provider, and clinic levels have been positively associated with HPV vaccination outcomes. Interventions that address parental psychosocial factors (ie, knowledge, beliefs, and outcome expectations), provider behavior (ie, HPV vaccine recommendation), patient-targeted systems (ie, reminder systems), and provider-targeted systems (ie, assessment and feedback) can positively influence HPV vaccination rates [9]. Multimethod strategies demonstrate the highest rates of maintaining increases in HPV vaccination [10].

The use of parent- and patient-focused apps to promote HPV education and vaccination is on the rise [11-14]. Mobile health (mHealth) is rapidly becoming a dominant mode to deliver health education and health promotion interventions [15]. Evaluating the user experience of mHealth interventions is important in assessing their quality, acceptability to users, and application in real-world clinical settings [16-20] but is often underreported [21-23]. User experience is broadly defined as a person’s perceptions and responses resulting from the use or anticipated use of a product, system, or service [24]. There is a lack of consensus on the best methods, measures, or scales to use when assessing mHealth user experience, although this field is maturing, and a number of scales have been developed that focus exclusively on usability [25,26], quality [27,28], and clinically meaningful risks and benefits [29]. The most frequently evaluated domains in assessments of commercially available mHealth apps include the scientific and clinical basis, functionality, usability, accountability, impact, and popularity [23]. Nouri et al’s [30] systematic review of mHealth evaluation criteria found 7 domains that are commonly used: design, content, usability, functionality, ethical issues, security and privacy, and user-perceived value. In addition, government bodies, such as the UK National Health Service, have developed their own mHealth evaluation standards, which include usability and accessibility to ensure they meet the needs of a diverse set of users, including people with disabilities or those with limited technical knowledge, for their health app marketplace [31]. Of 18 scales developed to evaluate the quality of mHealth apps, the Mobile Application Rating Scale (MARS) is the most frequently applied and is the only scale that has a user version for evaluation by nonhealth professionals [32]. A few studies have evaluated the aspects of user experience for patient-focused technology-mediated HPV interventions, including evaluating the usability of a conversational agent for HPV vaccine counseling of parents and college students using a Wizard of Oz methodology [11,33]; evaluating the feasibility, acceptability, and usability of a cervical cancer and HPV educational virtual agent for Hispanic women [34]; and evaluating the usability of a HPV information website for parents and adolescents [35]. These usability evaluations were done as part of the formative design process and did not include users enrolled in a randomized controlled trial (RCT), using the intervention longitudinally on their own. As HPV vaccination misinformation remains a significant public health problem, there is a need for HPV education apps that are usable, useful, and scalable to motivate parents to vaccinate their adolescent children.

HPVcancerFree (HPVCF) is an iOS- and Android-compatible smartphone app designed for parents of patients aged 10-17 years who have not initiated HPV vaccination. HPVCF is part of a multilevel intervention aimed at increasing HPV vaccination initiation and completion rates in a large US pediatric clinic network [36]. HPVCF was designed to (1) raise awareness of HPV and its prevention, (2) reduce barriers to HPV vaccination, and (3) enable parents to initiate HPV vaccination scheduling and reminders through their smartphone. Preliminary findings have demonstrated potential of HPVCF in changing parental knowledge and perceptions of HPV vaccination [37].

HPVCF was created using user-centered design principles and Intervention Mapping, an evidence- and theory-based systematic framework for developing behavior change interventions [38]. The design steps included (1) literature review and online synchronous text-based focus groups with parents from the pediatric clinic network to assess HPV attitudes, barriers, beliefs, and needs related to a digital behavior change solution [39]: (2) matrices describing target behaviors, psychosocial determinants of behavior, and change objectives; (3) delineation of theoretical methods and practical applications; (4) prototype build; (5) heuristic evaluation and in-house alpha testing; and (6) iterative user testing to assess app content, function, delivery channel, usability, value, desirability, and adoptability for both design and content.

HPVCF contains 4 self-tailored components: (1) HPV A-Z, a compendium of 9 content domains providing facts about HPV and the HPV vaccine; (2) Bust-a-Myth, 7 educational modules, including peer and health care provider testimonials addressing the most salient HPV vaccination barriers; (3) Notes 4 Doc, a medium to facilitate communication with health care providers about the HPV vaccine; and (4) Get the Vax, a feature to schedule HPV vaccination appointments and receive tailored reminders (Figure 1). There were 77 app pages and links that parents had unlimited access to. HPVCF was designed for user-centric navigation and so did not prescribe an intended user path.

The purpose of this study is to evaluate the user experience of HPVCF, an HPV cancer prevention app designed for a pediatric clinic network, using mixed methods data collected from log files, survey measures, and qualitative feedback.

Study data were drawn from parents who used HPVCF in a group RCT assessing HPVCF effectiveness within a large pediatric clinic network in Texas, USA, and completed a postintervention experience survey [37]. The study occurred between September 2017 and March 2019, where the 51 network clinics were randomized to either the treatment (HPVCF with usual care) or a comparison (usual care only) study condition. Parents in the 26 treatment clinics represented the analytic sample for this user experience evaluation. These parents were given instruction and links to download HPVCF from the Apple App Store (iPhone users) or Google Play Store (Android users). They were given a personal ID to enter the first time they launched HPVCF for tracking purposes. Study protocols were approved by the institutional review board at the University of Texas Health Science Center at Houston (HSC-SPH-15-0202).

Eligibility for the study included (1) having a 10-to-17-year-old child who was a patient in the clinic network, (2) having a child that had not initiated HPV vaccination, and (3) the ability to speak and write in English. Parents who had an eligible child were invited to participate in the study via patient health record portal invitations, flyers in the clinic waiting rooms, and posts on the clinic network Facebook page. Recruitment for the study took place on a rolling basis from September 2017 to September 2018. Each parent participated in the intervention for 5 months between September 2017 and March 2019, depending on when they were recruited and enrolled. Parent completed a presurvey before they were given access to the intervention and a postsurvey, which included an experience assessment, at the conclusion of their intervention time frame.

HPVCF use data were gathered over the course of the 5-month intervention from log files, including total number of visits, number of visits per participant, actions (viewing an app page or link) per visit, and visit duration. A back-end data capture system (Matomo) [40] collected time-stamped use by participant. Actions were triggered when the parent visited a new app page.

Parent sociodemographic variables were gathered from preintervention survey items at the start of the 5-month intervention. The parent sociodemographic variables included age, number of adolescent children, sex, race, ethnicity, education, child’s health insurance status, and baseline HPV vaccination intention.

In total, 168 parents completed the postintervention survey, of whom 98 (58.3%) were included in this experience analysis as they also downloaded and used the intervention (viewed at least 1 page past the home screen on any visit; Figure 2).

Parents had a mean age of 41 years, and the majority were female (94/98, 96%), college graduates (55/98, 56%), and White and non-Hispanic (55/98, 56%) and had private health insurance for their children (75/98, 77%); see Table 1. Most parents had 1 child between 10 and 17 years of age, with a range of 1-4 children in that age group. At baseline, of 98 parents, 12 (12%) reported that they “don’t intend” to vaccinate their child for HPV, 40 (41%) “definitely” planned to, and 46 (46%) were unsure (“haven’t thought of it,” “considering,” and “will probably get”). These demographics reflected the RCT sample where parents had a mean age of 41 years, were majority female (358/375, 95.5%), were college graduates (233/375, 62.1%), and identified as White and non-Hispanic (210/375, 56%). Further, these results approximate the demographic characteristics of the clinic network population where among children 10-17 years old, 45% are White and non-Hispanic and 80% have private health insurance.

Parents visited HPVCF 197 times during the study period (Table 2). Most parents used HPVCF once (45/98, 46%) or twice (28/98, 29%) with a range of 1-8 visits. During a single visit, 2-84 actions occurred with a mode of 3 actions. The average visit duration was 3 minutes and 27 seconds with a mode of 24 seconds and a range from 3 seconds to just under 27 minutes. Of the 4 HPVCF main component pages, HPV A-Z (370 views) was visited most often by parents, followed by Bust-a-Myth (273 views), Get the Vax (173 views), and Notes 4 Doc (110 views).

Qualitative feedback gathered on how to improve HPVCF included themes of increasing awareness and engagement with the app and enhancing the content and functionality.

This study evaluated the user experience of HPVCF, an HPV cancer prevention app designed for parents with children belonging to a large urban pediatric clinic network in the United States. Parents viewed HPVCF as having high quality, utility, and perceived impact. HPVCF quality ratings were robust (4.2) compared to quality scores of 2.4-4.6 for mHealth apps focused on prostate cancer risk [44], Alzheimer disease [45], alcohol use [46], occupational therapy [47], orthopedic rehabilitation [48], and medication adherence [49]. HPVCF quality was rated the highest for information and lowest for engagement subscales. This is consistent with the parents’ perceptions of HPVCF information as accurate and trustworthy, while also acknowledging the need to enhance functions and features. The positively skewed quality rating is consistent with a considerable number of parents (16/95, 17%) attributing their child’s HPV vaccination to HPVCF and the majority of parents (64/95, 67%) attributing HPVCF to increasing their intentions to vaccinate their child. This is promising, considering that the period of participation in the study was only 5 months, which is inadequate to fully track vaccinations through annual well visits. The study design was insufficient to determine whether these perceptions and vaccinations were significantly different from trends in parents not exposed to HPVCF, but it does appear that HPVCF was sufficiently persuasive to move at least a sample of parents to action. It is also unclear whether, in these instances, HPVCF was directly associated with vaccination or mediated through greater pediatrician dialogue that promoted vaccination.

Interestingly, 63 (66%) of 95 parents reported that they would use the app again, but log data indicated that 45 (46%) of 98 parents only used the app once. Triangulating these findings with information obtained in the qualitative feedback suggests possible reasons for this, including forgetting about the app after initial download, only using HPVCF on an as-needed basis, or no longer needing the app since it fulfilled its intended use after a single visit. Future iterations could be strengthened to help parents reengage by utilizing pediatricians and clinic staff to incorporate reminders as part of their standard communication. Exploring adjunct functionality that offers HPVCF information in more compact ways (ie, push notifications, text messages) may prove beneficial as most parents only looked at a few pages or links for a brief time (under 3.5 minutes).

Parents perceived HPVCF as more impactful for increasing awareness and knowledge than in changing attitudes about HPV. Knowledge is necessary but not necessarily sufficient to elicit behavior change. Negative attitudes around HPV vaccine safety are particularly pervasive [9], and strategies that are personalized, tailored, and require engagement beyond passive education may be needed to modify attitudes [50]. Concerns about HPV vaccination may have extended to HPVCF itself, with some parents feeling that HPVCF is biased in its portrayal of the long-term risks and safety of the HPV vaccine, despite high credibility ratings. A further behavioral impact was that the majority of parents also reported that HPVCF prompts greater communication with their pediatrician. This is an important adjunct function that helps the parent engage more competently with the pediatrician and provides the pediatrician with an opportunity to educate the parent during “teachable moments” at the clinic visit.

Importantly, parent engagement with the intervention was low, with about 70 (41.7%) of 168 parents choosing not to use HPVCF, making it difficult to generalize results to the clinic network. The low engagement is partially a reflection of the real-world nature of the study and accompanying challenges of competing for attention in an open market. Future iterations might adopt a more assertive approach by having parents download HPVCF during clinic visits and having clinic staff be more involved in providing reminders for its use. Future studies can explore promotional strategies to motivate parents to use HPVCF.

Additional limitations of this study should be considered. The intervention timeline did not include the back-to-school vaccination period (generally June-September) for many adolescent children, as the intervention took place on a 5-month rolling basis over the course of 1.5 years. This may have affected the parents’ decision to get their adolescents vaccinated. The 5-month intervention timeline also meant that some parents answered postintervention questions weeks after using HPVCF, possibly affecting their ability to accurately recall and report on some survey measures. A further limitation of the study was that the English-only content and the smartphone-based application may have excluded participation from parents with lower socioeconomic status or those who do not speak English. Although smartphone ownership among Americans is high (85% White, 85% Hispanic, 83% Black), there are disparities between Americans who are college educated (93%) and those with a high school diploma or less (75%) [51], and health app usage among low-income, racial minority, and ethnic minority patients in Texas remains low [52]. Future research addressing non-English-speaking and populations with a lower socioeconomic status is recommended. Finally, this analysis did not examine user experience by participant characteristics. However, subsequent analyses explored content-specific patterns of use that underlie psychosocial characteristics of parents [53].

As digital technologies continue to evolve, they stand to provide a paradigmatic shift in how health education and health behavior research are conducted [54], but doing so will require them to be perceived by users as being usable and useful applications. This study contributes a multimethod and multimeasure evaluation to the growing body of literature focused on assessing the user experience of patient-focused technology-mediated applications for HPV education [11,33-35].

HPVCF was well received by parents and performed well on indicators of quality, usefulness, utility, credibility, and perceived impact. HPVCF contributes to a multimethod and multimeasure evaluation strategy for user experience, which remains underreported in mHealth apps.