This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.
Human papillomavirus (HPV)-attributed cancers are preventable, yet HPV vaccination rates severely lag behind other adolescent vaccinations. HPVcancerFree (HPVCF) is a mobile health (mHealth) intervention developed to influence parental HPV vaccination decision making by raising awareness of HPV, reducing HPV vaccination barriers, and enabling HPV vaccination scheduling and reminders through a smartphone app. Evaluating the user experience of mHealth interventions is a vital component in assessing their quality and success but tends to be underreported in mHealth intervention evaluation.
We aimed to evaluate the user experience of HPVCF, an HPV cancer prevention app designed for a pediatric clinic network, using mixed methods data collected from log files, survey measures, and qualitative feedback.
Study data were evaluated from parents in a large US pediatric clinic network using HPVCF in the treatment study condition of a group randomized controlled trial. Log data captured HPVCF retention and use. Postintervention rating scales and items assessed HPVCF utility, usefulness, understandability, appeal, credibility, and perceived impact. Overall quality was evaluated using the user version of the Mobile Application Rating Scale (uMars). Open-ended responses assessed parent recommendations for HPVCF enhancement.
The 98 parents were mainly female (n=94, 96%), 41 (5.67) years of age, college educated (n=55, 56%), and White and non-Hispanic (n=55, 56%) and had private health insurance for their children (n=75, 77%). Parents used HPVCF 197 times, with the average visit duration approximating 3.5 minutes. The uMARS app quality score was positively skewed (4.2/5.0). Mean ratings were highest for information (4.46 [SD 0.53]) and lowest for engagement (3.74 [SD 0.69]). In addition, of 95 parents, 45 (47%) rated HPVCF as helpful in HPV vaccination decision making and 16 (17%) attributed HPV vaccine initiation to HPVCF. Parents reported that HPVCF increased their awareness (84/95, 88%), knowledge (84/95, 88%), and HPV vaccination intentions (64/95, 67%). Most of the 98 parents rated the 4 HPVCF components as useful (72-92 [73%-94%]). Parents also agreed that HPVCF is clear (86/95, 91%), accurate (86/95, 91%), and more helpful than other HPV vaccine information they had received (89/95, 94%) and that they would recommend it to others (81/95, 85%). In addition, parents suggested ways to increase awareness and engagement with the app, along with opportunities to enhance the content and functionality.
HPVCF was well received by parents and performed well on indicators of quality, usefulness, utility, credibility, and perceived impact. This study contributes a multimethod and multimeasure evaluation to the growing body of literature focused on assessing the user experience of patient-focused technology-mediated applications for HPV education.
Human papillomavirus (HPV) is a sexually transmitted infection that causes anogenital cancers and oropharyngeal cancers in men and women [
National samples in the United States have found that 28% of parents have refused or decided not to get the HPV vaccine for their child and 8% of parents have delayed or put off getting the vaccine [
Factors at the individual, provider, and clinic levels have been positively associated with HPV vaccination outcomes. Interventions that address parental psychosocial factors (ie, knowledge, beliefs, and outcome expectations), provider behavior (ie, HPV vaccine recommendation), patient-targeted systems (ie, reminder systems), and provider-targeted systems (ie, assessment and feedback) can positively influence HPV vaccination rates [
The use of parent- and patient-focused apps to promote HPV education and vaccination is on the rise [
HPVcancerFree (HPVCF) is an iOS- and Android-compatible smartphone app designed for parents of patients aged 10-17 years who have not initiated HPV vaccination. HPVCF is part of a multilevel intervention aimed at increasing HPV vaccination initiation and completion rates in a large US pediatric clinic network [
HPVCF was created using user-centered design principles and Intervention Mapping, an evidence- and theory-based systematic framework for developing behavior change interventions [
HPVCF contains 4 self-tailored components: (1) HPV A-Z, a compendium of 9 content domains providing facts about HPV and the HPV vaccine; (2) Bust-a-Myth, 7 educational modules, including peer and health care provider testimonials addressing the most salient HPV vaccination barriers; (3) Notes 4 Doc, a medium to facilitate communication with health care providers about the HPV vaccine; and (4) Get the Vax, a feature to schedule HPV vaccination appointments and receive tailored reminders (
The purpose of this study is to evaluate the user experience of HPVCF, an HPV cancer prevention app designed for a pediatric clinic network, using mixed methods data collected from log files, survey measures, and qualitative feedback.
HPVcancerFree (HPVCF) components. HPV: human papillomavirus.
Study data were drawn from parents who used HPVCF in a group RCT assessing HPVCF effectiveness within a large pediatric clinic network in Texas, USA, and completed a postintervention experience survey [
Eligibility for the study included (1) having a 10-to-17-year-old child who was a patient in the clinic network, (2) having a child that had not initiated HPV vaccination, and (3) the ability to speak and write in English. Parents who had an eligible child were invited to participate in the study via patient health record portal invitations, flyers in the clinic waiting rooms, and posts on the clinic network Facebook page. Recruitment for the study took place on a rolling basis from September 2017 to September 2018. Each parent participated in the intervention for 5 months between September 2017 and March 2019, depending on when they were recruited and enrolled. Parent completed a presurvey before they were given access to the intervention and a postsurvey, which included an experience assessment, at the conclusion of their intervention time frame.
HPVCF use data were gathered over the course of the 5-month intervention from log files, including total number of visits, number of visits per participant, actions (viewing an app page or link) per visit, and visit duration. A back-end data capture system (Matomo) [
HPVCF user experience was assessed with a postintervention survey using a quality rating scale, survey items, and an open-ended response item for recommended enhancements. The survey items included utility, perceived impact, component usefulness, clarity, credibility, and motivational appeal.
App quality was assessed by the user version of the Mobile Application Rating Scale (uMARS) [
Two utility items assessed whether HPVCF information helped parents decide to get the HPV vaccine for their child (no, yes, no opinion) and whether parents got their child the HPV vaccine as a result of using HPVCF (no, yes, no opinion). These items were adapted from prior surveys used with patient-focused digital behavior change interventions in clinic and school settings [
Perceived impact was assessed with 5 items on user perceptions of HPVCF. These modified perceived impact uMARS items measured perceptions of change in awareness of HPV and the HPV vaccine, knowledge of HPV and the HPV vaccine, attitudes of HPV and the HPV vaccine, intentions to get their child the HPV vaccine, and communication with the child’s pediatrician about the HPV vaccine. These items were evaluated on a 4-point scale with response options “strongly disagree,” “somewhat disagree,” “somewhat agree,” and “strongly agree” [
Usefulness was assessed using ratings of 4 HPVCF components (HPV A-Z, Bust-a-Myth, Notes 4 Doc, and Get the Vax) with response options “did not use,” “not very useful,” “somewhat useful,” “very useful,” and “do not recall.” For analysis, “very useful” and “somewhat useful” were collapsed into an “agreement” category.
Clarity was assessed with a single item on whether the goal of HPVCF was clear (no, yes, no opinion).
Credibility was assessed using 1 rating of accuracy of HPVCF content (inaccurate, accurate, no opinion) and 1 rating of trustworthiness of HPVCF information (cannot be trusted, can be trusted, no opinion).
Motivational appeal was assessed using 3 ratings: whether parents would use HPVCF again (no, yes, no opinion), a comparison of the helpfulness of HPVCF content against other HPV content received (less helpful, as helpful, more helpful), and the extent to which parents would recommend HPVCF to others who might benefit from it (few people, several people, many people, everyone). These items were adapted from prior surveys used with patient-focused digital behavior change interventions in clinic and school settings [
Recommended enhancements were solicited from an open-ended question, “What would make the HPVCF app more appealing so that parents would want to use it?” adapted from prior surveys used with patient-focused digital behavior change interventions in clinic and school settings [
Parent sociodemographic variables were gathered from preintervention survey items at the start of the 5-month intervention. The parent sociodemographic variables included age, number of adolescent children, sex, race, ethnicity, education, child’s health insurance status, and baseline HPV vaccination intention.
In total, 168 parents completed the postintervention survey, of whom 98 (58.3%) were included in this experience analysis as they also downloaded and used the intervention (viewed at least 1 page past the home screen on any visit;
Parents had a mean age of 41 years, and the majority were female (94/98, 96%), college graduates (55/98, 56%), and White and non-Hispanic (55/98, 56%) and had private health insurance for their children (75/98, 77%); see
Parents visited HPVCF 197 times during the study period (
Recruitment and retention. HPV: human papillomavirus; HPVCF: HPVcancerFree.
Parent demographics (N=98).
Characteristic | Value | |
|
41.23 (5.67); 26-54 | |
|
1.42 (0.62); 1-4; 1 | |
|
||
|
Male | 4 (4) |
|
Female | 94 (96) |
|
||
|
White, non-Hispanic | 55 (56) |
|
Black or African American, non-Hispanic | 7 (7) |
|
Hispanic | 30 (31) |
|
Asian | 4 (4) |
|
Other | 2 (2) |
|
||
|
Some high school | 1 (1) |
|
High school graduate or General Educational Development (GED) | 7 (7) |
|
Some college | 35 (36) |
|
College graduate | 25 (25) |
|
Graduate or professional degree | 30 (31) |
|
||
|
Private health insurance | 75 (77) |
|
Medicaid/Medicare/State Children's Health Insurance Program | 20 (20) |
|
Uninsured; no coverage of any type | 3 (3) |
|
||
|
Haven’t thought of it | 8 (8) |
|
Considering | 19 (19) |
|
Will probably get | 19 (19) |
|
Definitely | 40 (41) |
|
Don’t intend | 12 (12) |
a10-17 years old.
bResponse options are inclusive.
cHPV: human papillomavirus.
HPVcancerFree (HPVCF) use (N=98).
Visit details | Value | |
|
197 | |
|
2 (1.25); 1-8 | |
|
||
|
1 visit | 45 (46) |
|
2 visits | 28 (29) |
|
3 visits | 13 (13) |
|
4 visits | 10 (10) |
|
5 visits | 2 (2) |
|
8 visits | 1 (1) |
|
11 (10); 2-84; 3 | |
|
207 (249); 3-1601; 24 | |
|
||
|
HPVd A-Z | 370 |
|
Bust-a-Myth | 273 |
|
Notes 4 Doc | 110 |
|
Get the Vax | 173 |
aA visit was defined as viewing at least 1 page past the home screen.
bAn action was defined as viewing an app page or link. There were 77 app pages and links in total, which could be viewed unlimited times.
cThe main component pages could be visited unlimited times while visiting the app.
dHPV: human papillomavirus.
The uMARS app quality rating was 4.2/5.0 (
User version of the Mobile Application Rating Scale (uMARS) mean scores (N=95).
Subscalea | Mean (SD) |
Engagement (5 items) | 3.74 (0.69) |
Functionality (4 items) | 4.32 (0.65) |
Aesthetics (3 items) | 4.30 (0.57) |
Information (4 items) | 4.46 (0.53) |
Overall qualityb (from subscales) | 4.20 (0.48) |
aItems in the subscale measured on a 5-point response scale from 1 for “inadequate” to 5 for “excellent” and N/A if an app component was not used.
bCalculated by averaging the combined scores for each of the 4 subscales (engagement, functionality, aesthetics, and information).
Overall, 45 (47%) of 95 parents rated HPVCF as helping them decide to get their child the HPV vaccine, and 16 (17%) responded that they got their child the HPV vaccine as a result of HPVCF (
Parent agreement on utility, perceived impact, usefulness, clarity, credibility, and appeal (N=95).
User experience survey parameter | Agreement, n (%) | |
|
||
|
The information I got from HPVCFb helped me decide to get my child the HPVc vaccine. | 45 (47) |
|
I got my child the HPV vaccine as a result of using the HPVCF app. | 16 (17) |
|
||
|
Increased my awareness of HPV and HPV vaccine. | 84 (88) |
|
Increased my knowledge of HPV and HPV vaccine. | 84 (88) |
|
Changed my attitudes of HPV and HPV vaccine. | 54 (57) |
|
Increased my intentions to get my child the HPV vaccine. | 64 (67) |
|
Encouraged me to talk to my child’s pediatrician about the HPV vaccine. | 65 (68) |
|
||
|
HPV A-Z | 92 (94) |
|
Bust-a-Myth | 88 (90) |
|
Notes 4 Doc | 72 (73) |
|
Get the Vax | 73 (75) |
|
||
|
The goal/purpose of the HPVCF app was clear. | 86 (91) |
|
||
|
I think information I got from the HPVCF app was accurate.f | 86 (91) |
|
I think the information I got from the HPVCF app can be trusted.g | 85 (90) |
|
||
|
I would use HPVCF again.c | 63 (66) |
|
Compared to other information I have seen about the HPV the HPVCF app is as or more helpful.h | 89 (94) |
|
I would recommend HPVCF to others.d | 81 (85) |
aResponded “yes” as opposed to “no” or “no opinion.”
bHPVCF: HPVcancerFree.
cHPV: human papillomavirus.
dIncludes “somewhat agree” and “strongly agree” response options.
eN=98; combined responses of “very useful” and “somewhat useful.” Combined percentage responses of “did not use” and “do not recall” were as follows: HPV A-Z (5/98, 5%), Bust-a-Myth (5/98, 5%), Notes 4 Doc (19/98, 19%), and Get the Vax (20/98, 20%).
fRated as “accurate” as opposed to “inaccurate” or “no opinion.”
gRated as “can be trusted” as opposed to “cannot be trusted” or “no opinion.”
hRated as “as helpful” or “more helpful” as opposed to “less helpful.”
Most parents (64/95, 67%) agreed that HPVCF increased their intentions to get their child the HPV vaccine. Parents reported that HPVCF positively impacted their awareness (84/95, 88%), knowledge (84/95, 88%), and attitudes (54/95, 57%) about HPV and the HPV vaccine and encouraged them to discuss the HPV vaccine with their child’s pediatrician (65/95, 68%); see
Most parents rated the 4 HPVCF components as useful (
The majority of parents rated the purpose of HPVCF as clear (86/95, 91%) and that the information in HPVCF was accurate (86/95, 91%) and can be trusted (85/95, 89%). Parents also agreed that they would use HPVCF again (63/95, 66%), that it was more helpful than other information they had seen about HPV and the HPV vaccine (89/95, 94%), and that they would recommend HPVCF to others (81/95, 85%); see
Qualitative feedback gathered on how to improve HPVCF included themes of increasing awareness and engagement with the app and enhancing the content and functionality.
Parents commented that they forgot to use the app after the initial download and needed a reminder from the app, pediatrician, or clinic to use HPVCF:
It's been a long time since I used the app, so I don't remember if there was a function to remind the user in the future to make an appointment etc. That would prompt the user to reopen the app. I read all the information the first time I opened it and didn't open it again, so I've forgotten much of it now.
Due to a lack of repeated engagement, some parents reported they had forgotten much of the information they had originally reviewed. Parents suggested improving engagement by having push notifications with HPV facts instead of having to open and use the app to obtain information. Further, to improve marketability, one parent recommended incorporating HPVCF content into a broader app that included topics outside of HPV.
Parents also suggested making HPVCF more interactive and entertaining, especially by providing opportunities to engage and speak with adolescents about HPV and sexual health. To make HPVCF more adolescent friendly, parents suggested adding animations and games.
To improve the content, parents suggested offering parent testimonials highlighting their struggle to decide to vaccinate and how they used HPVCF to make an informed decision:
I think offering parent testimonials about their struggle to decide to vaccinate and how they used the info offered to help make an educated decision.
Some parents were dissatisfied with the presentation of HPV’s long-term effects, expecting to see more in-depth information and studies about complications and side effects of the vaccine:
This app, as a parent, did not give me the type of information I would want and need about HPV. I would prefer more information on the age of the vaccine. Credible studies completed. New research and side effects. More of that information would help. If the vaccine is less than 10 years old, I want more information on studies.
Parents wanted a tailored reminder system that conveyed information regarding their child’s HPV vaccination status and recommendation:
[I would suggest] some type of electronic reminder from the doctor to review the app. I downloaded the app for the survey, reviewed it, but then forgot about it. Maybe the doctor's office can send email to parents of 9-, 10-, and 11-year-old patients. Also, [a] reminder should have information on [the] child's status (ie, for informational purposes only; needs first dose; received first dose, time for second; etc).
Parents noted some issues with the available functionality, such as the app not storing their appointment information caused by usability issues with the design. For future iterations, parents suggested the ability to share information in the app with existing social media outlets:
[It would be helpful if the app would] allow for flagging/sharing individual items to foster organic awareness through existing social outlets?
This study evaluated the user experience of HPVCF, an HPV cancer prevention app designed for parents with children belonging to a large urban pediatric clinic network in the United States. Parents viewed HPVCF as having high quality, utility, and perceived impact. HPVCF quality ratings were robust (4.2) compared to quality scores of 2.4-4.6 for mHealth apps focused on prostate cancer risk [
Interestingly, 63 (66%) of 95 parents reported that they would use the app again, but log data indicated that 45 (46%) of 98 parents only used the app once. Triangulating these findings with information obtained in the qualitative feedback suggests possible reasons for this, including forgetting about the app after initial download, only using HPVCF on an as-needed basis, or no longer needing the app since it fulfilled its intended use after a single visit. Future iterations could be strengthened to help parents reengage by utilizing pediatricians and clinic staff to incorporate reminders as part of their standard communication. Exploring adjunct functionality that offers HPVCF information in more compact ways (ie, push notifications, text messages) may prove beneficial as most parents only looked at a few pages or links for a brief time (under 3.5 minutes).
Parents perceived HPVCF as more impactful for increasing awareness and knowledge than in changing attitudes about HPV. Knowledge is necessary but not necessarily sufficient to elicit behavior change. Negative attitudes around HPV vaccine safety are particularly pervasive [
Importantly, parent engagement with the intervention was low, with about 70 (41.7%) of 168 parents choosing not to use HPVCF, making it difficult to generalize results to the clinic network. The low engagement is partially a reflection of the real-world nature of the study and accompanying challenges of competing for attention in an open market. Future iterations might adopt a more assertive approach by having parents download HPVCF during clinic visits and having clinic staff be more involved in providing reminders for its use. Future studies can explore promotional strategies to motivate parents to use HPVCF.
Additional limitations of this study should be considered. The intervention timeline did not include the back-to-school vaccination period (generally June-September) for many adolescent children, as the intervention took place on a 5-month rolling basis over the course of 1.5 years. This may have affected the parents’ decision to get their adolescents vaccinated. The 5-month intervention timeline also meant that some parents answered postintervention questions weeks after using HPVCF, possibly affecting their ability to accurately recall and report on some survey measures. A further limitation of the study was that the English-only content and the smartphone-based application may have excluded participation from parents with lower socioeconomic status or those who do not speak English. Although smartphone ownership among Americans is high (85% White, 85% Hispanic, 83% Black), there are disparities between Americans who are college educated (93%) and those with a high school diploma or less (75%) [
As digital technologies continue to evolve, they stand to provide a paradigmatic shift in how health education and health behavior research are conducted [
HPVCF was well received by parents and performed well on indicators of quality, usefulness, utility, credibility, and perceived impact. HPVCF contributes to a multimethod and multimeasure evaluation strategy for user experience, which remains underreported in mHealth apps.
human papillomavirus
HPVcancerFree
mobile health
randomized controlled trial
user version of the Mobile Application Rating Scale
The authors thank the parents who participated in the study for their time and valuable insights. This research was supported by the Cancer Prevention and Research Institute of Texas (RP150014).
None declared.