We used a mixed methods research design with a sample of caregivers (n=12) to examine the usability of the PLAY high-fidelity prototype. A high-fidelity prototype is a simplified version representing the skeletal framework of a website or application’s interface [31]. Specifically, we used an expedited iterative design process to advance the study timeline. Rapid usability analysis has been used to speed the interpretation of user feedback and maximize identified codes and themes via reviewing conversations (ie, audio and video recording) instead of transcriptions [32-34]. This approach is known for its efficiency in distilling user feedback, accelerating the identification of codes and themes by analyzing audio and video recordings directly, thereby bypassing the labor-intensive transcription process. This method ensures swift progression from user feedback to design iteration, aligning with user-centered design principles [23]. By engaging a diverse cross-section of caregivers, including a significant representation from underserved rural or Appalachian regions (over 30%), we aimed to rapidly pinpoint and implement essential modifications to tailor the platform to the needs of these communities.

A high-fidelity prototype of the PLAY website, based on GROW (Figure 1), was developed and shared with caregivers in this study to (1) determine usability among caregivers of young survivors of childhood cancer and (2) identify recommendations for tailoring the program to their unique needs. The PLAY website was based on the GROW website, a fully developed digital health intervention program originally designed for caregivers of young children ages 0‐4 years following TBI, to improve responsive parenting skills and cognitive stimulation [27,28]. Participants were instructed to review the content, as generally about having a child with a chronic medical condition. The GROW intervention involves internet-based learning modules (ie, GROW platform), as well as live one-on-one coaching via videoconferencing and weekly homework. GROW was based on a previously developed and tested responsive parenting program called PLAY and Learning Strategies (PALS) [35] and informed by caregivers and experts in special education and TBI. All 3 programs (ie, PALS, GROW, and PLAY) are rooted in attachment theory [36] and sociocultural models of child development [37]. The core treatment target is responsive parenting, which involves increasing caregiver warmth and contingent reactions to their child’s cues, while also providing verbal stimulation that enriches development.

The PLAY website, such as GROW, has six learning modules: (1) interactions, (2) language development, (3) attention/play, (4) parent self-care, (5) relationships, and (6) working with schools. Each module includes brief text content on the topic area, videos highlighting examples of caregivers interacting with their children and practicing the skills, and exercises to apply learned information, such as matching and quizzes (refer to previous publications) [27]. The interface was built using the popular content management platform WordPress (Automattic, Inc), augmented with the LearnDash plugin to provide learning management system functionality. This system is compatible with PC, mobile (Android and iOS), and tablet browsing due to an inbuilt responsive design.

Purposive sampling was used to identify potentially eligible caregivers of young survivors of childhood cancer from the cancer registry of a Midwest pediatric academic medical center. Inclusion criteria for eligible participants included caregivers of a child who (1) was diagnosed with cancer before the age of 7 years, (2) was between the ages of 3‐12 years at the time of study participation, (3) did not have a history of a neurodevelopmental disorder prior to their cancer diagnosis (eg, neurofibromatosis, tuberous sclerosis, Down syndrome, and autism), (4) had completed treatment for cancer at least 6 months prior or was on maintenance oral chemotherapy only, without evidence of recurrent disease or current hospitalization. Caregivers also were required to be (1) the child’s legal guardians, (2) living with the identified child at least 50% of the time, (3) fluent in English, and (4) residing in the state of the institution or surrounding states (ie, Ohio, Kentucky, Pennsylvania, or West Virginia). Note that families at different time points post treatment were recruited to provide a range of perspectives, and that caregivers of children currently aged 7‐12 years were asked to retrospectively consider unmet needs and the use of the PLAY program during the planned target age range of 3‐6 years.

Stratified purposive sampling was conducted by using random sampling within subgroups to ensure representation by diagnosis type (ie, brain tumor, leukemia/lymphoma, and solid tumor), geographic location (ie, rural, nonrural, Appalachian, and non-Appalachian as defined by the Rural Urban Commuting Codes [38] and Appalachian Regional Commission [14]), and age of the young survivors of childhood cancer (ie, either 3‐6 or 7‐12 years; to include those at various time points postcancer diagnosis/treatment). We also intentionally oversampled individuals from racial and ethnic minority groups to increase representation in our sample and reflect the institutional cancer registry (ie, 9% Black and 6% Latinx).

Caregivers were invited to participate with a mailed letter from their oncologist or neurosurgeon, which was followed by attempts to contact the caregiver by phone. Caregivers could opt out of future contact with the research team. In phase 1 of this study, caregivers were initially invited to participate in a mixed methods assessment of their unmet psychosocial and educational needs involving internet-based REDCap (Research Electronic Data Capture; Vanderbilt University) surveys and an interview conducted by phone or secure videoconferencing. Caregivers could also elect to participate in a monthly Community Advisory Board (CAB) and future research. Research staff discussed the CAB and usability testing with potentially interested members during their scheduled phone/videoconference interview until the desired sample size for planned usability testing was acquired (n of at least 10) [39] and saturation in rapid data analysis was attained.

Usability testing visits were scheduled at the convenience of caregivers on a secure videoconferencing platform with screen sharing and recording capabilities. All visits were recorded and transcribed automatically by the secure videoconferencing platform for cross-referencing during the coding process. During the usability testing visit, a trained research staff member who conducted phase 1 with the caregiver provided brief context on the platform (ie, currently designed for families of young children following TBI). Next, caregivers were provided with the link to the platform and were instructed to engage in unstructured navigation of the platform on their own for 5 minutes while sharing their screen (ie, hereafter “unguided use”). Caregivers were encouraged to “think aloud” about any of their feedback. Pages accessed and verbal comments were noted for analysis.

Following this, caregivers were instructed to perform a series of structured, task-based scenarios involving one specific module of the platform (refer to Figure 2; minimodule: “How Do I Connect with My Child?” and File S1 in Multimedia Appendix 1) for standardization across caregivers. This minimodule introduced the key concept of warmth, including how to use specific, labeled praises and the skill of diaphragmatic breathing for caregiver stress management. On each page of this module, research staff members provided standardized prompts (eg, “start the module,” “go to the next page,” “watch the video,” and “complete the exercise”) and encouraged caregivers to continue to “think aloud.” Total time to complete the scenario, navigation challenges, and verbal comments were noted for analysis. Following the completion of the scenario, caregivers completed brief online surveys (refer to the “Measures” section below) to indicate the device used (ie, mobile, tablet, or PC) and to provide feedback about the platform. Caregivers also engaged in a semistructured interview based on the technology acceptance model [40] about their overall impressions of the platform, other potential components of the intervention (ie, coaching by videoconferencing and text messaging), and suggestions for improvement. A copy of the semistructured interview guide is included in File S2 in Multimedia Appendix 1.

Descriptive statistics (means and SDs) and frequency counts were calculated for demographic and medical variables, device usage, SUS scores, reaction cards, and documented data from the sessions (ie, pages accessed and length of time for scenario). For the qualitative data, a rapid analysis approach [32-34] was implemented with 2 trained research staff members, who served as interviewers and coders (EM, a postdoctoral fellow and licensed clinical psychologist, and ED, a clinical research coordinator). They attended usability testing sessions live and took detailed field notes. Recordings and field notes were reviewed a second time, with general codes extracted into summary templates organized by interview question in Microsoft Excel spreadsheets. Double coding was conducted for 25% of the interviews, with consensus reached prior to independent coding for the remainder of the data. A second consensus meeting was held to identify themes endorsed by a majority of participants across interviews using a matrix. Member checking was conducted with Caregiver Advisory Board members, who confirmed the findings and coding themes.

This study was reviewed and approved by the Nationwide Children’s Hospital’s institutional review board (00002939). As a minimal risk study, participants verbally agreed to participate after reviewing a written information sheet detailing risks and potential benefits. Participants also had an opportunity to ask questions of the study staff.

Of 524 families in the cancer registry, 137 families were randomly selected as part of the stratification process in the “Procedure” section and were attempted to be contacted. Of those, 84 caregivers were reached by phone, and 32 caregivers agreed to participate in phase 1. A total of 25 agreed to be recontacted to participate in the CAB and other future studies, and the first 12 caregivers were sequentially invited for usability testing. There were no significant differences in demographic characteristics of the caregiver or child with cancer (ie, geographic region, sex, income, race, ethnicity, and diagnostic type) between the subsample of 12 participants who completed usability testing and the 32 participants from phase 1 (Fisher exact tests; P>.05), except that caregivers who completed usability testing were slightly older (mean 42.67 and SD 6.5 years) than those who did not participate in the usability phase (mean 36.8 and SD 6.8 years; t₃₀=–2.41, P=.02).

A total of 12 usability sessions were conducted by videoconference. However, there was 1 technical difficulty in saving the video for 1 father, and thus, only audio was available for that visit. Detailed field notes were also documented for each visit by 2 staff members in attendance. The sample of caregivers who participated was mostly White (n=10, 83%) and married (n=8, 67%). They were equally dispersed amongst metropolitan, rural, and Appalachian areas (n=4 each, 33% each). Although most caregivers were biological parents (n=9, 75%), there were 2 grandfathers and 1 legal guardian/adoptive parent. At the time of the interview, young survivors of childhood cancer were between the ages of 3 and 9 years. Demographic characteristics are presented in Tables 1 and 2.

This mixed methods study involved initial usability testing of a high-fidelity prototype of an internet-based responsive parenting intervention program called PLAY, designed to support caregivers of young survivors of childhood cancer. Overall, all but one caregiver found PLAY usable (92% reported SUS scores ≥68). Acknowledging the limitations of the SUS as a summative measure, we supplemented this analysis with qualitative feedback and the use of reaction cards [42,43]. Caregivers primarily selected positively balanced adjectives (eg, useful, valuable, easy to use, accessible, and appealing) to describe the program. These results are promising and comparable to other internet-based support and educational resources for caregivers of children with various chronic and developmental conditions and for those from rural and remote areas (eg, autism and asthma) [44-48]. Similarly, in the previous pilot of the 6-session intervention with caregivers of young children with TBI, all 11 families who completed the full 6-session intervention reported high satisfaction with the program [27].

This study, strengthened by a mixed methods approach to usability testing, yielded valuable and detailed information related to the innovation design (ie, CFIR) [24] of PLAY. Caregivers identified key aspects to tailor the program for the unique needs of the young survivors of childhood cancer, such as resources specific to cancer and late effects, examples and resources related to advocacy for their child in health care and school settings, and the importance of the caregiver managing their own stress and uncertainty (eg, “scanxiety”) in their capacity to implement responsive parenting skills. Other general suggestions included increasing usability, offering coaching and texting as optional supports, and considering intervention timing, which differed by gender of the caregiver. With respect to usability, caregivers repeatedly underscored the necessity of PLAY being highly mobile-friendly, practical, and brief, given multiple competing demands on caregivers’ time and attention. Most caregivers agreed that the length of the module was appropriate (approximately 15‐20 minutes), and all caregivers reported having access to a smartphone and a preference for mobile usability. This is comparable to other studies with caregivers of young children with autism, in which mobile usability was a top priority [45]. The Pew Center also highlights that most adults living in both metropolitan (89%) and rural (80%) areas own a smartphone [49]. Still, both quantitative (ie, SUS) and qualitative data indicated that improvements were needed in the mobile usability and design of the platform, such as easier navigation to the main menu on the mobile version. Other suggestions to improve accessibility included closed captioning on videos and options for the text to be read aloud on each module page. These suggestions are in line with similar studies of digital health interventions (DHI) for caregivers [50,51]. Furthermore, these adaptations would assist in bridging the “disability digital divide” [52].

Most caregivers described receptiveness to coaching from a trained professional and occasional text message reminder, in addition to the self-guided platform resource. Caregivers emphasized that these supports should be optional and individually tailored in terms of frequency. They also recommended that coaching should be flexibly scheduled to increase accessibility for busy families with young children (ie, evenings and weekends). In studies of telehealth interventions for caregivers of children with autism [46], caregiver retention and positive outcomes were superior following weekly contact with a therapist, as compared to self-guided. Multimodal interventions may be especially needed, given the documented decline in caregiver participation over the course of internet-based positive parenting programs [53]. One consideration for future intervention development is that videoconference coaching may require high-speed broadband coverage, which may reduce accessibility for this benefit for rural families, as approximately 73% of rural households have home subscriptions [49].

Unexpectedly, there were some gender differences in the preferred modality and timing of this type of responsive parenting resource. Specifically, women caregivers preferred resources in the immediate period after diagnosis, while men caregivers preferred resources after cancer treatment had concluded. Although not universal even amongst our participants, this may be attributed to the common division of responsibility and typical gender roles after a child is diagnosed with cancer. Mothers often directly support the child in the hospital, while fathers and grandfathers often maintain their work schedule and household responsibilities [54,55]. Fathers may prefer to wait until after treatment is concluded when they feel more involved and included in care. Furthermore, there may be gender differences in how caregivers cope and adjust to the diagnosis and treatment [56], which may affect readiness for intervention. Still, this difference in preferred timing of resources may also be attributed to the difference in mean age and caregiver type, such that there were two grandfathers but no grandmothers in our small sample. At least one grandfather reported that he preferred to receive information about responsive parenting through the self-guided platform as opposed to coaching due to feeling defensive or “judged.” This aligns with previous work suggesting that fathers and other caregivers who are men may face unique barriers in help-seeking [57] and are often excluded from parenting intervention research [58]. Given that preferences in timing and modality may differ across family members, DHI such as PLAY might be offered to families earlier and more periodically throughout the illness trajectory, with careful framing of the parenting intervention to encourage father engagement.

Although this usability testing study offers several insights for the refinement of digital health platforms for caregivers of young children with chronic health conditions, several limitations should be considered. First, although we attempted to conduct stratified sampling, our post hoc testing indicated possible ascertainment bias with our subsample, including slightly older caregivers, who may have differing usability and information preferences than younger caregivers. Similarly, our recruitment rate of approximately 40% was relatively low, yet higher than a similar study with caregivers of young children following TBI (25%) [27]. This highlights the inherent challenges of engaging with caregivers of young children with complex medical needs, requiring future work to identify additional ways to engage and provide accessible psychosocial care to this underserved and sometimes hard-to-reach group. Second, although our sample size was sufficient to assess usability experience [39], we were not powered to examine subgroup differences based on type of caregiver or diagnostic group. Third, we used a rapid analysis approach [34], in which we were unable to calculate interrater reliability between coders. The interviewers and coders were part of the study team and, thus, their probing during interviews and observations in rapid analysis may have been biased. Still, the usability was assessed with multiple methods (ie, standardized measures, reaction cards, and “think aloud” procedures), in line with current research recommendations [43]. Finally, the single-session usability testing limited the review of only one module in detail (ie, responsive parenting). Future work will need to obtain perspectives of the PLAY intervention comprehensively (ie, other modules listed in File S3 in Multimedia Appendix 1).

Overall, internet-based responsive parenting interventions to support caregivers of young children may have promising transdiagnostic applications. However, creative problem-solving may be needed to improve future engagement, uptake, and implementation. Guided by the CFIR framework and designing with implementation in mind, our next step (ie, phase 3) will involve an iterative co-design process [59] with a CAB (n=13) and a multidisciplinary Healthcare Provider Advisory Board (HPAB; n=11). Co-design involves close collaboration between researchers, designers, and users, who serve as the experts due to their experiences [59]. For our proposed study, the research study team will partner with both the CAB and HPAB during monthly meetings held via videoconference to review and discuss feedback on each module of the PLAY program to allow for in-depth feedback and tailoring of each page of the intervention. For example, the self-care and relationships (refer to File S3 in Multimedia Appendix 1) modules focus on how the caregiver and their relationships have been affected, as well as strategies that may help (eg, relaxation strategies, and “I statements”). Both advisory boards will collaborate with the study team to develop and iteratively refine the graphics, videos, exercises, and text-based content to reflect their lived experiences and clinical expertise. Additional single-session user-experience testing with another group of caregivers will be conducted to ensure adequate usability (SUS ≥68) with the refined and tailored PLAY program. A pilot of the PLAY intervention will test feasibility and acceptability with caregivers of young survivors of childhood cancer, with a small randomized controlled trial to follow. A future larger randomized controlled trial will determine whether the program improves responsive parenting behaviors over several months and leads to downstream effects on neurodevelopmental outcomes for young survivors of childhood cancer. Future work will also identify barriers and facilitators in the inner and outer settings from the CFIR framework to support the adoption of the PLAY intervention in clinical practice.

Designing equitable digital health interventions demands the active and ongoing engagement of individuals with lived experience across all stages of design, particularly ensuring the inclusion of underrepresented groups to optimize success.