This paper is in the following e-collection/theme issue:

Parenting (264) Pediatrics (201) General Parenting Education and Training (61) Behavior Change (712)

Published on in Vol 4, No 4 (2021): Oct-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/27551, first published .
Engaging Parents and Health Care Stakeholders to Inform Development of a Behavioral Intervention Technology to Promote Pediatric Behavioral Health: Mixed Methods Study

Engaging Parents and Health Care Stakeholders to Inform Development of a Behavioral Intervention Technology to Promote Pediatric Behavioral Health: Mixed Methods Study

Engaging Parents and Health Care Stakeholders to Inform Development of a Behavioral Intervention Technology to Promote Pediatric Behavioral Health: Mixed Methods Study

Authors of this article:

Sean M O'Dell1, 2 Author Orcid Image ;   Heidi R Fisher3 Author Orcid Image ;   Victoria Schlieder4 Author Orcid Image ;   Tracey Klinger4 Author Orcid Image ;   Rachel L Kininger1 Author Orcid Image ;   McKenna Cosottile1 Author Orcid Image ;   Stacey Cummings5 Author Orcid Image ;   Kathy DeHart5 Author Orcid Image
Article Authors Cited by (2) Tweetations (2) Metrics

    Original Paper

    1Department of Psychiatry and Behavioral Health, Geisinger, Danville, PA, United States

    2Department of Population Health Sciences, Geisinger, Danville, PA, United States

    3Autism and Developmental Medicine Institute, Geisinger, Lewisburg, PA, United States

    4Investigator Initiated Research Operations, Geisinger, Danville, PA, United States

    5Department of Pediatrics, Geisinger, Danville, PA, United States

    Corresponding Author:

    Sean M O'Dell, PhD

    Department of Psychiatry and Behavioral Health

    Geisinger

    100 N. Academy Ave

    Danville, PA

    United States

    Phone: 1 570 214 5236

    Email: sodell@geisinger.edu


    Background: Despite effective psychosocial interventions, gaps in access to care persist for youth and families in need. Behavioral intervention technologies (BITs) that apply psychosocial intervention strategies using technological features represent a modality for targeted prevention that is promising for the transformation of primary care behavioral health by empowering parents to take charge of the behavioral health care of their children. To realize the potential of BITs for parents, research is needed to understand the status quo of parental self-help and parent-provider collaboration to address behavioral health challenges and unmet parental needs that could be addressed by BITs.

    Objective: The aim of this study is to conduct foundational research with parents and health care stakeholders (HCS) to discover current practices and unmet needs related to common behavioral health challenges to inform the design, build, and testing of BITs to address these care gaps within a predominantly rural health system.

    Methods: We conducted a convergent mixed-parallel study within a large, predominantly rural health system in which the BITs will be developed and implemented. We analyzed data from parent surveys (N=385) on current practices and preferences related to behavioral health topics to be addressed in BITs along with focus group data of 48 HCS in 9 clinics regarding internal and external contextual factors contributing to unmet parental needs and current practices. By comparing and relating the findings, we formed interpretations that will inform subsequent BIT development activities.

    Results: Parents frequently endorsed several behavioral health topics, and several topics were relatively more or less frequently endorsed based on the child’s age. The HCS suggested that BITs may connect families with evidence-based guidance sooner and indicated that a web-based platform aligns with how parents already seek behavioral health guidance. Areas of divergence between parents and HCS were related to internalizing problems and cross-cutting issues such as parenting stress, which may be more difficult for health care HCS to detect or address because of the time constraints of routine medical visits.

    Conclusions: These findings provide a rich understanding of the complexity involved in meeting parents’ needs for behavioral health guidance in a primary care setting using BITs. User testing studies for BIT prototypes are needed to successfully design, build, and test effective BITs to empower parents to take charge of promoting the behavioral health of their children.

    JMIR Pediatr Parent 2021;4(4):e27551

    doi:10.2196/27551

    Keywords

    primary care; parenting; targeted prevention; behavioral intervention technology; behavioral health 


    High Behavioral Health Need for Youth and Intractable Gaps in Access to Care

    Behavioral health problems are common among children and adolescents [1,2]. More than 13% of preschool–age children present with disruptive behavioral problems [3], and the onset of approximately half of all lifetime cases of clinically diagnosable disorders occurs by the age of 14 [4]. Short-term consequences associated with behavioral health problems include significant impact on family functioning [2,5,6] and educational achievement [3,7]. In the long term, children with behavioral health problems have a higher lifetime risk for conduct problems, antisocial behavior, early pregnancy, drug use, and school failure [7-9]. Symptoms and impairment falling below the cutoff for diagnosis or treatment also carry a significantly higher risk for psychopathology years later. This is especially concerning considering that the prevalence of subclinical cases is twice that of those reaching clinical thresholds [10,11].

    Despite the increased risk for short- and long-term negative outcomes, most children who would benefit from behavioral health care do not receive services [12,13]. Barriers to service use include structural barriers, such as shortage of behavioral health care providers, particularly in rural areas, and barriers related to stigma and negative perceptions regarding mental health problems and accessing mental health services [14,15]. In the pediatric health care setting, primary care clinicians (PCCs) often do not make appropriate referrals [16], and even when referrals are placed, many families never engage with the services [17].

    Furthermore, initiatives directly aimed at increasing access to services often fail to accomplish this goal. For example, despite the efficacy of school-based programs in preventing and decreasing aggressive behavior [18,19], ongoing efforts to provide services in schools are mitigated by a variety of factors including availability of trained staff [20], stakeholder attitudes about services [21], and the attendance and participation of those students who may benefit the most [22]. Similar or higher rates of behavioral health problems in rural communities [23-25] are compounded by even lesser access to and use of behavioral health services than those in urban communities [26].

    Leveraging Innovations in Service Delivery and Technology Can Help to Close Access Gaps

    Behavioral intervention technologies (BITs) have emerged as an option that may expand access to individuals for whom structural and consumer-level barriers prevent engagement with traditional face-to-face (FTF) therapy and telehealth services [27]. Most adults have a mobile phone and home internet access, far outreaching the number of individuals who live in areas with accessible behavioral health care [28]. BITs have the potential to provide better access to underserved populations and eliminate distance or transportation barriers, and they are not necessarily subject to shortages of trained staff [29,30].

    Most BITs for prevention and treatment of behavioral health problems in youth have included adolescents as the primary or sole users, and promising BITs exist for a range of presenting concerns, including anxiety, depression, and chronic health conditions [31-34]. BITs designed for parents may expand access and use of behavioral health further because of the potential to engage families who may not seek FTF behavioral health care because of fear of stigma or barriers of perception and those families who may be more willing to engage in BITs that are often self-directed and relatively more private [35,36]. Indeed, looking to the internet for parenting support and behavior change strategies is an emerging trend among parents [37-40].

    BITs for parents have predominantly focused on translating evidence-based parent training interventions originally developed and tested through FTF implementation [41]. There are examples of BITs for parents of children with disruptive behavior concerns that have successfully been adapted from FTF implementation for web-based platforms and have shown positive outcomes [29,35,37]. Overall, parents report a high rate of interest in and satisfaction with available BITs [40,42], yet the scope and availability of existing BITs need more development to realize this potential. One notable line of research has been conducted on the ezParent Program, which is a tablet-based preventive behavioral parent training intervention adapted from the Chicago Parent Program [43] tailored for youth aged 2 to 5 years in primary care settings. An advantage of the development strategy for ezParent is that many of its aspects, including implementation factors, adherence, and parental perceptions of engaging with the program, have been studied [44-46]. Nevertheless, when tested in a randomized controlled trial, ezParent was not more effective on child outcomes than enhanced usual care [47]. These findings suggest that BITs such as ezParent may work best in primary care settings when offered along with a range of more intensive interventions tailored to salient family characteristics that influence interest and engagement.

    Realizing the Potential for BITs to Improve Targeted Prevention in Primary Care

    There is a strong potential to expand the use of BITs across a wide range of developmental, behavioral, and emotional needs beyond parenting guidance for challenging behaviors [29,35,37,40,42]. Targeted prevention in the primary care setting may help to address an important care gap because PCCs routinely engage in anticipatory guidance as part of well-child visits, but it is impractical and potentially unhelpful for PCCs to discuss every relevant domain. For example, it is estimated that if PCCs addressed every relevant prevention target with every patient according to evidence-based guidelines, then it would comprise 7.4 hours of their workday [48]; however, only 52 of 2161 recommended topics for well-child visits are considered actionable [49].

    The importance of targeted prevention becomes even more salient when considering that PCCs are routinely asked to increase their roles and responsibilities (eg, developmental and behavioral health screening), yet visit lengths have not changed [50]. BITs that help PCCs do more with less must also consider parents’ preferences for guidance to be maximally effective. Parents often want more and different types of guidance and information than are typically provided by their child’s PCC [51,52]. Schuster et al [51] found that most surveyed parents endorsed having unmet needs regarding subjects that PCCs routinely discuss, such as crying, learning, discipline, and toilet training, and many endorsed needing more information. Combs-Orme et al [53] found that even though discipline was one of the most frequently discussed topics with PCCs, this was the area in which parents had the most questions. Therefore, research to develop BITs must also carefully examine the determinants of maladaptive parenting behaviors, such as lack of information regarding typical development and behavior or lack of parenting skills that promote healthy behavioral and emotional growth for children [54-56].

    Intentionally developing BITs from the outset to meet the range of needs of families and PCCs working to address behavioral health problems may help to address the limitations of extant BITs. Research on elements of effective implementation and scaling of FTF behavioral health services in primary care has robustly shown that effectiveness is influenced by contextual factors such as provider knowledge and skills about and attitudes toward behavioral health topics, motivation to change, management and leadership practices, and financial resources [57]. This has also been shown to be relevant to BITs, as clinic personnel implementing the ezParent Program reported that despite supporting the program, substantial contextual barriers impeded referrals to the program because of time, workflow, and organizational factors [58].

    This Study

    This paper reports the initial stages of development for a targeted prevention BIT to empower parents to take charge of their child’s behavioral health care in pediatric primary care clinics within a predominantly rural health system in the Northeast United States. Developing targeted prevention BITs is part of an overarching approach to extend the continuum of primary care behavioral health services, including integrating behavioral health–health care stakeholders (HCS) into pediatric primary care locations and improving the scope and quality of training for PCCs in behavioral health topics.

    Our approach to developing these BITs is informed by the approach described by Lyon et al [59] for adapting evidence-based psychosocial interventions for implementation in naturalistic settings. We describe the findings of the discover phase of development to identify the needs and perspectives of stakeholders and potential barriers to usability and implementation in the targeted intervention context. The goal is for the findings of this study to identify modification targets in extant evidence-based interventions and then apply this knowledge to iterative design and build cycles used to redesign interventions using prototypes and stakeholder feedback in preparation for developing a polished prototype to rigorously test for effectiveness in naturalistic setting. This approach is compatible with recommendations to improve BIT implementation measurement in part by distinguishing between BIT development and implementation, enhancing responsiveness to stakeholder outcomes, and integrating the BIT into existing services in the implementation context [60].

    Therefore, the primary objective of this study is to identify the needs and preferences of parents and HCS within the health system that the BITs will ultimately implement, as these are the 2 key stakeholder groups which the BITs are intended to serve. We obtained input from parents and HCS using different methods to maximize the depth of information from each stakeholder group. For parents, we developed and administered a survey of parent preferences to be addressed in BITs. In addition, we developed a survey of current needs and practices for handling behavioral health concerns and administered it to a market research panel of parents within the health system. We chose to conduct a series of focus group interviews with a range of HCS to allow for more flexibility and depth of explanation of the intervention context and any associated barriers and facilitators to the implementation of BITs.


    Design and Data Analysis Plan

    We used the Pragmatic Robust Implementation and Sustainability Model (PRISM) [61] framework to inform our development activities, as it is an implementation science framework that encompasses the diverse priorities of the design phase by expanding the conceptualization and measurement of RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) [62] implementation outcomes by explicitly including contextual factors, overarching issues, and interdependency among components of the model. Additionally, PRISM has been shown to be compatible with qualitative methods throughout the intervention development and implementation continuum [63].

    We integrated quantitative survey data obtained from parents regarding their views and experiences on a variety of behavioral health topics with qualitative focus group interview data of HCS on their perceptions of unmet needs and current practices of parents regarding managing their child’s behavioral health care. To accomplish this, we employed a convergent mixed-parallel design [64] as depicted in Figure 1.

    ‎
    Figure 1. The data collection and analysis process in this study using a convergent mixed-parallel design.

    Setting and Participants

    Focus groups were conducted between April 23, 2019 and June 24, 2019 in 9 child-serving clinics within a large, predominantly rural health system. A total of 83% (48/58) of HCS participated. Participants comprised HCS from 5 primary care sites and 1 developmental medicine clinic; 2 primary care sites invited to participate declined. The primary care site focus groups were each completed in a single session in the clinic over lunch. Of these, 2 focus groups were conducted with the developmental medicine clinic stakeholders during monthly administrative meetings to accommodate the availability of participants. The focus group participants comprised a range of roles and professional backgrounds, including 16 pediatricians, 2 pediatric psychologists, 2 genetic counselors, 1 speech pathologist, 2 behavior analysts, 5 licensed nurse practitioners, 6 registered nurses, 7 physician or medical assistants, 4 patient access representatives, 1 family liaison, 1 operations manager, and 1 pediatric technician.

    An electronic parent survey was conducted using Qualtrics in Spring 2019 from a geographically representative patient panel within a rural health system in the Northeast United States who had previously opted in a program to be contacted to complete web-based surveys regarding their perspectives on health care services offered within the health system in which the study was conducted. To be eligible to complete the survey, the respondent had to endorse screening items indicating that they were the parent or guardian of at least one minor child (0-18 years of age) at the time of completing the survey and were somewhat or very interested (as opposed to not at all interested) in using web-based resources to research issues and concerns they may have about their children and parenting. Invitations were distributed twice with the goal of acquiring 400 completed surveys. Of the 2240 respondents who initiated the survey, 411 met the inclusion criteria and proceeded to the rest of the survey. However, 6.3% (26/411) of these respondents abandoned the survey before completing the initial content questions. Table 1 shows the demographics of the remaining 385 respondents.

    Table 1. Demographic characteristics of parent survey respondents (N=385).
    CharacteristicRespondents, n (%)
    Age (years)

    		18-246 (1.6)

    		25-3490 (23.4)

    		35-44142 (36.9)

    		45-5497 (51.2)

    		55-6438 (9.9)

    		65-747 (1.8)

    		75+1 (0.3)

    		Prefer not to say2 (0.5)

    		Missing data2 (0.5)
    Number of childrena

    		1184 (47.8)

    		2139 (36.1)

    		340 (10.4)

    		412 (3.1)

    		55 (1.3)

    		Missing data5 (1.3)
    Education level

    		High school graduate (high school diploma or equivalent including GEDb)37 (9.6)

    		Some college44 (11.4)

    		Associate\'s degree in college (2-year program)19 (4.9)

    		Bachelor\'s degree in college (4-year program)61 (15.8)

    		Master\'s degree44 (11.4)

    		Doctoral degree7 (1.8)

    		Professional degree (JDc, MDd)3 (0.8)

    		Prefer not to say4 (1)

    		Missing data116 (30.1)
    Sex

    		Female245 (63.6)

    		Male66 (17.1)

    		Missing data74 (19.2)
    Annual income (US $)

    		Less than 10,00012 (3.1)

    		10,000-29,99927 (7)

    		30,000-49,99930 (7.8)

    		50,000-79,99937 (9.6)

    		80,000-99,99926 (6.8)

    		100,000 or more58 (15.1)

    		Prefer not to say29 (7.5)

    		Missing data166 (43.1)
    Race or ethnicity

    		American Indian or Native Alaskan0 (0)

    		Asian4 (1)

    		Black or African American4 (1)

    		Hispanic, Latino, or Spanish4 (1)

    		White347 (90.1)

    		Other2 (0.5)

    		Prefer not to say6 (1.6)

    		Missing data18 (4.7)
    Has children in each age range (years)

    		0 to 5143 (37.1)

    		6 to 12143 (37.1)

    		13 to 18175 (45.5)
    Insurance type

    		Private44 (11.4)

    		Public (Medicaid, Medicare)35 (9.1)

    		Missing data306 (79.5)

    aEighteen years of age or younger.

    bGED: General Education Development.

    cJD: Juris Doctor.

    dMD: Doctor of Medicine.

    Measures

    Health Care Provider Focus Groups

    The focus group moderator guide (Multimedia Appendix 1) was developed by the study authors using PRISM [61], which aims to identify and leverage multiple dimensions of internal and external contextual factors that contribute to stakeholder influence and implementation outcomes. Prompts were designed to evoke discussion among participants about the topic of unmet parental needs, including healthy development and social, emotional, and behavioral functioning of their children. The moderator introduced the study and its objectives, read prompts, and encouraged discussion among the focus group participants. Prompts also included uncovering what the HCS perceived that parents were doing to address unmet needs, and how the BIT platform website might help. In line with PRISM, participants were also asked about institutional leadership and what barriers health care HCS foresee the study team encountering in developing a mobile responsive website as a behavioral health intervention directed toward parents.

    Parent Quantitative Survey

    Questions were developed by the study authors and additional study personnel who were engaged as content experts in relevant disciplines. Given our emphasis on evidence-based content and aim to complement and expand upon the behavioral health care support provided by PCCs, content topics were selected from the American Academy of Pediatrics (AAP) anticipatory guidance recommendations described in Bright Futures: Guidelines for Health Supervision of Infants, Children, and Adolescents [65]. The anticipatory guidance described in Bright Futures covers a wide range of health, developmental, and behavioral topics across infancy, childhood, and adolescence. The study authors adopted relevant behavioral health survey topics from the Bright Futures topics based on their strong potential for delivery using a BIT. For example, the anxiety in children, behavioral challenges, and mood or depression in children survey topics were selected from the broader Promoting Mental Health anticipatory guidance topics from Bright Futures. Table 2 depicts the Bright Futures content domains and the resulting parent survey topics.

    Table 2. Parent survey topics.
    Bright Futures health promotion topicsParent or caregiver survey topics
    Promoting lifelong health for families and communities—a
    Promoting family supportParenting stress; family communication
    Promoting health for children and youth with special health care needs—
    Promoting healthy developmentSpeech or language skills, independence and activities of daily living; academic skills and intelligence; social skills; motor skills; toileting
    Promoting mental healthAnxiety in children; behavioral challenges; mood or depression in children
    Promoting healthy weightNutrition and eating
    Promoting healthy nutrition—
    Promoting physical activity—
    Promoting oral health—
    Promoting healthy sexual development and sexualitySex and sexual development
    Promoting the healthy and safe use of social mediaThe internet and social media
    Promoting safety and injury preventionChild safety; drugs and substance abuse

    aBright Futures health promotion topic not covered in parent or caregiver survey topics.

    Procedures

    Health Care Stakeholder Focus Groups

    The project manager and research assistant traveled to each clinic to conduct in person focus groups. The project manager was trained in interviewing techniques and led the focus group discussions based on the guide included in Multimedia Appendix 1. Focus groups were audio recorded and transcribed by a skilled research assistant using Start-Stop Universal software and then deidentified for analysis. Transcripts were coded by the project manager, research assistant, and 2 psychology postdoctoral fellows using Microsoft Word. The order in which the focus group interviews were coded by the study team was randomly selected using a web-based randomizing service to remove bias from the coding. An a priori codebook based on the interview guide was created to identify and code common topics within each transcript; emergent codes and themes were also identified during the course of coding. The coders individually analyzed each interview and met every week to review and establish interrater agreement. The final coded transcripts were then uploaded to Atlas.ti 8.4.15 (ATLAS.ti Scientific Software Development GmbH) for Windows, where thematic quotes could be exported into spreadsheets based on individual codes for further analysis.

    Parent Survey

    Survey respondents who met the inclusion criteria for the study rated up to 3 of the 17 topics as their top choices for content that they would be interested in learning more about through a BIT. To understand if the topic was a current challenge or if the respondent wanted more information for future reference, respondents then identified whether the topic of interest had or had not been a challenge that they had encountered so far. Next, for each of the top 3 topics, respondents were provided a list of subtopics and were asked to identify which subtopics were a problem or concern. Next, respondents were provided with a list of common strategies for addressing the broad topic (eg, anxiety in children) and asked to rate if they had used the strategies and the extent to which they perceived each strategy to be helpful. Common strategies included those with an empirical evidence base as well as those without one in the interest of learning about the prevalence and preference of a range of strategies. The survey is included in Multimedia Appendix 2.

    Data Analysis Plan

    Parent quantitative survey data and HCS qualitative interview data were analyzed in parallel. For survey data, summaries were created using descriptive statistics for the most frequently endorsed content topics in the total sample. Descriptive statistics summarized the prevalence of endorsements for topics representing parental concerns and engagement with and perceived helpfulness of the strategies listed. Three qualitative topics from focus groups were selected for use in data integration: unmet needs, current practice providers, and current practice-parents because of their relevance to the quantitative data collected from parents. Each of these topics was analyzed and summarized by the study team based on the codes and topics identified. Each quote was then subcoded to expand on popular topics within each main code. The subcodes were utilized as framework for the overall summaries of all 3 topics. Once the parallel analyses were completed, the results were merged using a joint display to identify areas of confirmation, expansion, and discordance (Figure 2). We randomly selected 2 team members to integrate data for each survey topic, and the results were based on the comparing and relating these findings. To supplement these analyses, including those for parent survey topics that were not selected for data integration by joint display, we queried the qualitative interviews for mentions of parent survey topics and related keywords to make additional interpretations.

    ‎
    Figure 2. Joint display.

    Parallel Analyses of Qualitative and Quantitative Data

    Health Care Stakeholder Focus Groups

    Focus group participants reported that the most common unmet behavioral health needs of the parents they work with related to common parenting challenges such as disruptive behaviors, sleep, toileting, and nutrition. Participants commonly reported a perception that lack of foundational knowledge in promoting healthy development across behavioral health topics represented vulnerability. Other contextual factors, such as lack of easy access to credible information, were commonly reported to compound the barriers to accessing local behavioral health resources. There was also a common theme noted among participants that social networks within the family (eg, grandparents) are often resources to help with common parenting challenges.

    Focus group participants noted that parents frequently turn first to web-based resources (eg, web searches and social media) to find ideas and strategies to address behavioral health needs, which often led to unproven techniques being tried first (eg, weighted vests and cannabidiol products) and, in turn, maintaining or exacerbating behavioral health problems over time. Parents’ desire for a quick fix was posited as an underlying reason for these choices, whereas focus group participants also noted that another subset of parents seem to follow a wait and see approach, in which they may wait several months to seek advice at the next routine visit, which was reported to unintentionally contribute to problems becoming ingrained and intractable. Ultimately, participants reported that this contributed to increased frustration for parents and more challenge for HCS implementing a more comprehensive and effortful course of treatment. For their part, HCS in focus group interviews reported that they made concerted efforts to spend extra time in their visits to provide guidance and psychoeducation on foundational parenting strategies. They also reported making specialty referrals when appropriate and acknowledged that they do not always have the time or resources to be responsive to parent concerns.

    Parent Survey

    Table 3 provides the frequency of each topic area selected by respondents across child age ranges of 1-5, 6-12, and 13-18 years. Across all respondents, the most frequently endorsed topics included anxiety in children (111/380, 29.2%), behavioral challenges (106/380, 27/9%), nutrition or eating (105/380, 27.6%), mood or depression in children (100/380, 26.3%), and the internet and social media (99/380, 26.1%). Responses were further examined based on whether respondents endorsed having a child of 1-5, 6-12, or 13-18 years. Anxiety in children, behavioral challenges, and nutrition and eating continued to be highly endorsed topics, regardless of child age. Respondents who reported having a child in the 6-12-year age range and the 13-18-year age range also frequently endorsed mood or depression in children and the internet and social media. Respondents who indicated they had a child aged 1-5 years also showed interest in speech or language skills, academic skills and intelligence, parenting stress, and sleep or bedtime routine.

    Multimedia Appendix 3 provides descriptive statistics for the responses to each of the top 6 content topics. Most respondents endorsed each topic because of a past or current parenting challenge, as opposed to interest related to general guidance. With a few exceptions, the challenging topics listed within each topic were also endorsed by a substantial proportion of respondents, indicating that the issues parents face within each topic are often multifaceted. Similarly, respondents endorsed a variety of common strategies to help with the identified topics within each topic. Few strategies were endorsed as tried and was helpful by more than half of the respondents, suggesting that respondents are likely to try several strategies and find that few of them make a positive difference. Strategies rated as potentially problematic by the investigators are noted in the supplementary tables, and, overall, were some of the least likely strategies to be endorsed as helpful by respondents.

    At the end of the survey, parents were asked if there were any additional topics they would like to see in a BIT. Of the 36 free-text responses, 7 pertained to topics that fit within the scope of the survey topics (eg, language and speech and behavior). Of the 385 respondents, 4 indicated that additional information about puberty would be helpful. Moreover, 9 responses were highly specific concerns that were outside the scope of a BIT for targeted behavioral health prevention (eg, caring for a child with a chronic illness). Several parents responded that they would have selected more or all the survey topics. Other responses pertained to coparenting, dealing with divorce, dealing with death and grief, and attachment.

    Table 3. Parent endorsement of survey topics by age ranges of children.
    TopicAge range (years), n (%)

    		All (n=380)1-5 (n=143)6-12 (n=176)13-18 (n=175)
    Anxiety in children111 (29.2)33 (23.1)66 (37.5)55 (31.4)
    Behavioral challenges106 (27.9)43 (30.1)57 (32.4)40 (22.9)
    Nutrition and eating105 (27.6)51 (35.7)44 (25)38 (21.7)
    Mood or depression in children100 (26.3)16 (11.2)49 (27.8)67 (38.3)
    The internet and social media99 (26.1)24 (16.8)47 (26.7)50 (28.6)
    Parenting stress79 (20.8)36 (25.2)33 (18.8)31 (17.7)
    Academic skills and/or intelligence76 (20)28 (19.6)40 (22.7)35 (20)
    Social skills68 (17.9)21 (14.7)31 (17.6)31 (17.7)
    Family communication63 (16.6)19 (13.3)28 (15.9)34 (19.4)
    Speech or language skills45 (11.8)29 (20.3)14 (8)10 (5.7)
    Independence and activities of daily living42 (11.1)12 (8.4)21 (11.9)24 (13.7)
    Sex and sexual development41 (10.8)6 (4.2)19 (10.8)29 (16.6)
    Sleep or bedtime routine38 (10)28 (19.6)13 (7.4)3 (1.7)
    Drugs and substance abuse35 (9.2)3 (2.1)11 (6.3)29 (16.6)
    Child safety28 (7.4)22 (15.4)13 (7.4)3 (1.7)
    Toileting27 (7.1)24 (16.8)8 (4.5)1 (0.6)
    Motor skills13 (3.4)10 (7)0 (0) 3 (1.7)

    Interpretations Based on Comparing and Relating Health Care Stakeholder Focus Groups and Parent Surveys

    Overview

    After parallel analyses of the qualitative health care stakeholder focus group data and the quantitative parent survey data, we compared and related features of these data to integrate and make interpretations to guide further development efforts for the BIT. The results of the data integration phase are described next according to each of the parent survey topics. Multimedia Appendix 4 presents a summary of the areas of confirmation, expansion, and discordance for selected behavioral health topics with substantial data from both sources.

    Anxiety

    Anxiety was the most prevalent concern endorsed by parents; although HCS did not identify anxiety as a pressing unmet need, there were 6 mentions of anxious or anxiety in the qualitative data. There was agreement between parents and HCS in the demand for more web-based resources. HCS reported that parents lack resources for behavioral health concerns, yet parents rated psychotherapy as the most helpful strategy for anxiety management. From these data, it is unclear how many families access these services. Additionally, other commonly reported parental strategies (eg, comforting the child) are not typically effective in the long term or when used as a standalone strategy, which may relate to health care stakeholder observations that parents are seeking a quick fix and need more support in long-term behavior change.

    Behavioral Challenges

    HCS and parents reported that disruptive behaviors are a common concern, but parents tend to use ineffective behavior management strategies. HCS also lacked some awareness of key parental challenges within the disruptive behavior topic. It appears that primary care provider strategies alone are not enough to be beneficial for parents. This highlights the potential benefits of a BIT to address parental needs in this area more effectively. Demonstration of specific behavior management techniques may be helpful to include in a BIT to help parents put strategies into action.

    Nutrition and Eating

    Nutrition and eating concerns were commonly reported by both HCS and parents. HCS reported that parents tended to use unhelpful strategies to manage eating concerns, but parents reported a mix of helpful and unhelpful strategies. HCS were also unaware of several common strategies parents endorsed to manage nutrition and eating concerns, and many parents indicated that they did not discuss nutrition and eating concerns with their primary care HCS. Of the queries of qualitative data returned, 3 mentions of nutrition, 2 mentions of food, and 9 mentions of eat were made.

    Parenting Stress

    HCS and parents both reported that parenting stress is a common concern, although parents and HCS had differing perspectives on the factors contributing to parental stress. HCS tended to discuss parental stress in terms of parental frustration with child behavior as opposed to parent-specific factors (eg, coping with emotions). The strategies parents use to manage stress may impact their use of primary care resources. Addressing parental stress is beyond the traditional scope of pediatric primary care, and HCS are likely to lack the knowledge of how to deal with more complex cases. Therefore, a BIT addressing parental stress may help HCS direct parents to useful resources.

    Family Communication

    Family communication was a commonly endorsed topic for parents, and HCS mentioned this as a concern. Furthermore, it also related to collaborative communication with HCS about child behavioral health needs. HCS perceptions of parental communication strategies were discordant with the parent-reported strategies. HCS did discuss how family structure may impact parental communication in the qualitative topics reviewed, and further queries of the qualitative data returned several mentions of divorce, mixed households, and nontraditional families. Interestingly, HCS expressed concern about both a lack of parental communication and excessive parental communication, whereas parents were most concerned about a lack of communication among family members. It was observed the HCS found it challenging to find common ground with parents. Similarly, parents also faced difficulty in finding common ground with other caregivers.

    Additional Interpretations of Parent Survey Topics Not Selected for Joint Display

    For parent survey topics without substantial HCS qualitative data in the codes that we analyzed in the parallel phase, we searched for key terms in the qualitative data to determine if we could further compare and relate these data to make interpretations.

    Sleep or Bedtime Routine

    The parent survey topic of sleep or bedtime routine was noteworthy, in that although it was not commonly endorsed overall (38/380, 10%), it was more prevalent (28/243, 19.6%) for respondents with a child in the age range of 1-5 years. We also found 13 mentions of sleep, which co-occurred with our unmet needs code 7 times. More specifically, sleep patterns and sleep hygiene at different ages were brought up during at least 2 focus groups as something with which parents discussed struggling or not understanding what is normal, whether it be newborn sleep or even sleep patterns throughout childhood and adolescence. This is confirmed through coding, in that mentions of sleep co-occurred with the lack of knowledge code 4 times throughout the 7 focus groups.

    This suggests multiple opportunities to target BIT content for young children on this topic to be most efficient with resources.

    Mood and Depression in Children

    Results regarding mood were discordant between parents and HCS. Parents commonly identified depression or mood as a top concern, but HCS did not discuss mood and depression concerns as unmet needs or in terms of strategies parents use to manage mood concerns. The qualitative data included 4 mentions of words beginning with depress. These mentions often co-occurred with mentions of anxiety and may suggest that HCS tend to conceptualize these as related (eg, internalizing problems) or find them frequently co-occurring in their patients. These results were somewhat surprising and may indicate a domain which improved clinical training for HCS in clinical interviewing and behavioral health screening may be helpful.

    Drugs and Substance Abuse

    Similarly, the parent survey topic of drugs and substance abuse was rated more commonly by parents with a child in the 13-18- year age range (29/175, 16.6%) than the overall prevalence (35/380, 9.2%). In reviewing the 3 mentions we found in the qualitative data of drug, there was poignant discussion among participants in one of the focus groups highlighting the complexity of addressing this topic with parents who are suspicious or concerned about drug abuse, how they might rely on AAP guidelines, and publications that discuss how HCS can help parents (Textbox 1).

    Respondent 1 is most likely referring to the AAP clinical report by Levy et al [66]. This resource provides guidance on how pediatricians can navigate this complex and important topic for which there is presently minimal empirical literature available. Further BIT development efforts may help to design a BIT module that can provide high-quality information and resources to parents in need of guidance on this topic that they commonly reach out to their pediatrician to address.

    Textbox 1. Exemplary quotes related to drugs and substance abuse.
    The others--the teenager who is non-compliant either at school and outperforms in other areas where they like things and how do we manage that behavior because they don't want to take away the good activities; what do I do? Or you have a parent who's suspicious of particular drug use; what do I do in this particular situation? Can we drug-test them, which is almost universal: No. However, what do we do in these situations?
    [Respondent 1]
    Why can't you do that?
    [Respondent 4]
    We can talk about that, but just ethically, no we don't do that. Ask me later...or, getting back to the drug-testing, having an explanation of, here's how to handle if you're concerned about your child's drug use, here's what you can do at home...
    [Respondent 1]
    You don’t want to know what I do at home.
    [Respondent 2]
    I guarantee he's not coming to see you.
    [Respondent 4]
    Here's the formal policy of the national organization called the American Academy of Pediatrics on how to address this with your child and our stance on drug-testing teens. It is understood that it is not just a clinic, but also nationally what is done. It would be cool to see what that does for parents.
    [Respondent 1]
    Parent Survey Topics With Substantive Additional Qualitative Data

    Among other parent survey topics not selected for joint display, we found some useful additional information within the qualitative data that may inform future BIT development. Regarding the topic of child safety, we found that this was commonly endorsed by parents with a child in the 1-5-year age range (22/143, 15.4%); however, only 1 mention of this topic was found in the qualitative data. Speech and language skills were also commonly endorsed by parents of children in the 1-5-year age range (29/143, 20.3%), and the only mentions within the qualitative data related to accessibility of the website for parents for those who are speakers of languages other than English or may have lower educational attainment. Independence and activities of daily living were more commonly endorsed by parents of youth aged 6-12 years (21/176, 11.9%) and 13-18 years (24/175, 13.7%). The qualitative data included mentions self-help, hygiene, daily routines, and chores, which may indicate the topics that HCS most commonly discuss with parents. Finally, toileting was another topic commonly endorsed by parents of children aged 1-5 years (27/380, 7.1%); although no related mentions were found within the codes we analyzed in joint display, other qualitative data did include mentions of toileting (3 mentions) and potty (2 mentions).

    Parent Survey Topics Without Substantive Additional Qualitative Data

    The topic of academic skills and/or intelligence was commonly endorsed across age ranges (range 20%-23%), but was not selected for joint display because of a lack of discussion in the qualitative topics we included. The qualitative data also did not include terms related to child intelligence, so no more details for interpretation are available. In the focus group data, the same was true for social skills (0 mentions), the internet and social media (1 mention), sex and sexual development (0 mentions), and motor skills (0 mentions).


    Principal Findings

    This paper on the mixed-methods study reports the initial development of a targeted prevention BIT focused on behavioral health topics for parents to be implemented in pediatric primary care within a large, predominantly rural health system. We used the discover, design and build, and test framework [59] to inform our development efforts. In this manuscript, we report the outcomes of the discover phase to gather information on the implementation context and current issues facing parents and HCS navigating behavioral health topics in pediatric primary care that a BIT can address.

    Overall, the approach we selected shows promise that taking both parent and HCS input into consideration at the outset of BIT development in the discover phase provides unique insights that may help to address the limitations of the extant literature on BITs for parents of children with behavioral health problems. For example, research on the ezParent Program, a parent-focused BIT adaptation of the Chicago Parenting Program [43], stands out among the research on BITs for parents for having carefully studied implementation and sustainability factors from the parent perspective [45,46], yet, when tested in a randomized controlled trial, it did not demonstrate superiority to enhanced usual care [47]. Findings from other research on ezParent suggest that inconsistent referrals to the program were discovered only after rolling out the program in primary care and were attributed to operational workflow issues for primary care staff, and these issues were unforeseen [67]. By first studying the unmet needs of parents and HCS that a BIT might address, in the implementation context that the BIT is being developed and for the expressed purpose of extending the continuum of primary care behavioral health services already available, we may be able to obviate comparable setbacks through work in our design and build and test phases.

    The analysis of parent and HCS data in this study provided unique insights that will help in focusing the resources on developing and conducting preliminary testing on prototypes of BITs to better meet the behavioral health needs of parents using pediatric primary care within the health system. While the extant BIT literature in this area has primarily focused on engaging adolescents with a range of behavioral health problems, including anxiety, depression, and chronic pain, in adaptations of empirically supported treatments delivered in a BIT [31-34], our results indicate that BITs for parents also have the potential to greatly expand the reach and impact of evidence-based behavioral health care. Parents reported interest in BITs across several behavioral health topics, and we learned that parent interest sometimes varied across the pediatric age range. Owing to space constraints, we highlight 1 example next. Although only 10% (38/380) of parents endorsed the sleep or bedtime routine among the top 3 concerns, twice as many parents with a child aged 1-5 years endorsed this topic (28/143, 19.6%) and relatively fewer parents of children in the 6-12-year age range (13/176, 7.4%) or 13-18-year age range (3/175, 1.7%) endorsed the topic (Table 3). The implications of such findings for resource allocation for subsequent BIT development and clinical uptake are substantial. If guided solely by the overall prevalence of endorsement, we may not have selected sleep or bedtime routine as a topic for further BIT development. By extension, knowing that 19.6% (28/143) of parents with a child between 1-5 years are interested in this topic helps us to focus our BIT development efforts on topics most relevant for this age range even though research supports the effectiveness of behavioral sleep interventions for school-age youth [68]. Insights like these deepen our understanding of more detailed feedback from parents within each behavioral health topic and help the development efforts in the design and build phase, and these may increase the likelihood of BIT uptake in clinical settings for those found to be efficacious in the test phase [69].

    Limitations

    Our findings should be interpreted with recognition of the methodological limitations inherent to our approach, which focused on the initial development of a BIT to fit a specific implementation context. Therefore, surveys based primarily on selected Bright Futures topics that the research team felt would be a good fit for a BIT may not comprehensively represent the needs and preferences of parents related to empowerment to guide child development and behavioral well-being. A related limitation is that the study population is representative of the population in the region; survey respondents are mostly White and middle class; therefore, these findings may not be generalized to the needs and preferences of parents from other demographic and socioeconomic backgrounds. The sample of focus group interviewees was also recruited from the health system in which the BIT is being developed, which also introduces the possibility of limited generalizability. Finally, some caution in interpreting the findings of the data integration is warranted, given that we have not yet conducted any empirical studies to triangulate our interpretations with parent and provider interactions with BIT prototypes. Awareness of these potential limitations is also important to address in our future BIT development research because of the potential of unintentionally driving disproportionality in access to behavioral health care by developing a BIT that may not be engaging to historically excluded groups, who already face difficulties in accessing behavioral health care in rural areas [69,70]. Oversampling in the design andbuild and test phases may help in guarding against this unwanted outcome.

    Suggestions for Future Research

    Our approach to the discover phase for the development of a BIT to empower parents to take charge of their child’s behavioral health care was shaped by our perspectives on contributing factors to the longstanding issue of limited access to high-quality behavioral health care in primary care settings. This approach may also be useful for future research developing BITs with different goals in mind. Although evidence-based treatments are often conceptualized and developed as packaged intervention products, there is usually an observed voltage drop when taking efficacious psychosocial treatments out of the laboratory into community practice settings [71]. This undermines the conceptualization of psychosocial treatments as a product per se, whereas conceptualization as a cocreated service between parents and HCS suggests that reduced effectiveness is not inevitable [72]. High-value behavioral health care designed with input from transdisciplinary researchers, clinicians, and patient stakeholders in the setting intended for use may provide a better chance at comparable efficacy and effectiveness [73]. The findings from our discover phase support the notion that usual care is a cocreated service between parents and HCS within the health system, although one which often leads to unmet needs for both stakeholder groups in the health system in which the study was conducted. Therefore, the value of a BIT can be measured against the degree to which the implementation of BITs contributed to these needs being met. Research has demonstrated that parental comfort in discussing behavioral health concerns is shaped by the quality of the PCC response; that is, when PCCs dismiss these concerns, parents report that they are less comfortable discussing these topics [74]. BITs may help in this regard, as these conversations have been shown to be brief and work well when combined with videos to illustrate effective interventions for child discipline [75].

    At this juncture, we have entered the designandbuild phase to triangulate our mixed-methods findings with parent and provider feedback on the prototypes of the BIT [59]. We are currently collecting data for 2 mixed-methods user testing studies to triangulate these findings for the content topic of behavioral challenges. In 1 study, we recruited a group of parents who completed the survey and endorsed this topic in their top 3 (n=9) and another group of parents who completed the survey but did not endorse this topic in their top 3 (n=9). We chose to recruit from the parents who completed the quantitative survey to aid in triangulating findings from this study and from the behavioral challenges topic because there is substantial extant BIT literature for parents on this topic [41,47]. Another study was conducted with PCCs within the health system (n=16) to determine the usability and acceptability of provider-facing BIT to address behavioral challenges and how this can be incorporated into the electronic health record and clinic workflow.

    Conclusions

    This mixed-methods study provided some unique insights into the needs and preferences of parents and HCS. These results appear useful for designing a BIT platform to enhance access to effective self-help to empower parents to take charge of their child’s behavioral health care. Future research will triangulate these mixed-methods findings with parent and health care provider reactions to BIT prototypes in preparation for an effectiveness trial on a fully functional BIT prototype.

    Acknowledgments

    This study was funded by the Geisinger Quality Pilot Fund (principal investigators: SMOD and HRF). The authors would like to thank their colleagues who contributed to the development of the parent survey described in this paper: Samuel Faulkner, Julie Heier, Tawnya Meadows, and Maribeth Wicoff. A special thanks to Aaron Lyon for consultation on the study.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Focus group moderator guide.

    DOCX File , 27 KB
    Multimedia Appendix 2

    Parent survey.

    PDF File (Adobe PDF File), 361 KB
    Multimedia Appendix 3

    Parent survey descriptive statistics.

    PDF File (Adobe PDF File), 106 KB
    Multimedia Appendix 4

    Data integration summary.

    DOCX File , 24 KB
    1. Polanczyk GV, Salum GA, Sugaya LS, Caye A, Rohde LA. Annual research review: a meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J Child Psychol Psychiatry. Mar 2015;56(3):345-365. [CrossRef] [Medline]
    2. Blanchard LT, Gurka MJ, Blackman JA. Emotional, developmental, and behavioral health of American children and their families: a report from the 2003 National Survey of Children's Health. Pediatrics. Jun 2006;117(6):e1202-e1212. [CrossRef] [Medline]
    3. Lavigne JV, Lebailly SA, Hopkins J, Gouze K, Binns H. The prevalence of ADHD, ODD, depression, and anxiety in a community sample of 4-year-olds. J Clin Child Adolesc Psychol. May 2009;38(3):315-328. [CrossRef] [Medline]
    4. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. Jun 2005;62(6):593-602. [CrossRef] [Medline]
    5. Donenberg G, Baker BL. The impact of young children with externalizing behaviors on their families. J Abnorm Child Psychol. Apr 1993;21(2):179-198. [CrossRef] [Medline]
    6. Palermo T, Childs G, Burgess E, Kaugars A, Comer D, Kelleher K. Functional limitations of school-aged children seen in primary care. Child Care Health Dev. Sep 2002;28(5):379-389. [CrossRef] [Medline]
    7. Valdez C, Lambert S, Ialongo N. Identifying patterns of early risk for mental health and academic problems in adolescence: a longitudinal study of urban youth. Child Psychiatry Hum Dev. Oct 2011;42(5):521-538. [FREE Full text] [CrossRef] [Medline]
    8. Shaw D, Gilliom M, Ingoldsby E, Nagin D. Trajectories leading to school-age conduct problems. Dev Psychol. Mar 2003;39(2):189-200. [CrossRef] [Medline]
    9. Bradshaw CP, Schaeffer CM, Petras H, Ialongo N. Predicting negative life outcomes from early aggressive-disruptive behavior trajectories: gender differences in maladaptation across life domains. J Youth Adolesc. Aug 2010;39(8):953-966. [CrossRef] [Medline]
    10. Costello EJ, Shugart MA. Above and below the threshold: severity of psychiatric symptoms and functional impairment in a pediatric sample. Pediatrics. Sep 1992;90(3):359-368. [Medline]
    11. Angold A, Costello EJ, Farmer EM, Burns BJ, Erkanli A. Impaired but undiagnosed. J Am Acad Child Adolesc Psychiatry. Feb 1999;38(2):129-137. [CrossRef] [Medline]
    12. Costello EJ, He J, Sampson NA, Kessler RC, Merikangas KR. Services for adolescents with psychiatric disorders: 12-month data from the National Comorbidity Survey-Adolescent. Psychiatr Serv. Mar 01, 2014;65(3):359-366. [FREE Full text] [CrossRef] [Medline]
    13. Kataoka SH, Zhang L, Wells KB. Unmet need for mental health care among U.S. children: variation by ethnicity and insurance status. Am J Psychiatry. Sep 2002;159(9):1548-1555. [CrossRef] [Medline]
    14. Owens P, Hoagwood K, Horwitz S, Leaf PJ, Poduska JM, Kellam SG, et al. Barriers to children's mental health services. J Am Acad Child Adolesc Psychiatry. Jun 2002;41(6):731-738. [CrossRef] [Medline]
    15. Larson J, dosReis S, Stewart M, Kushner R, Frosch E, Solomon B. Barriers to mental health care for urban, lower income families referred from pediatric primary care. Adm Policy Ment Health. May 2013;40(3):159-167. [CrossRef] [Medline]
    16. Walders N, Childs GE, Comer D, Kelleher KJ, Drotar D. Barriers to mental health referral from pediatric primary care settings. Am J Manag Care. Oct 2003;9(10):677-683. [FREE Full text] [Medline]
    17. Rushton J, Bruckman D, Kelleher K. Primary care referral of children with psychosocial problems. Arch Pediatr Adolesc Med. Jun 2002;156(6):592-598. [CrossRef] [Medline]
    18. Sanchez A, Cornacchio D, Poznanski B, Golik A, Chou T, Comer J. The effectiveness of school-based mental health services for elementary-aged children: a meta-analysis. J Am Acad Child Adolesc Psychiatry. Mar 2018;57(3):153-165. [FREE Full text] [CrossRef] [Medline]
    19. Wilson SJ, Lipsey MW, Derzon JH. The effects of school-based intervention programs on aggressive behavior: a meta-analysis. J Consult Clin Psychol. Feb 2003;71(1):136-149. [Medline]
    20. Langley A, Nadeem E, Kataoka S, Stein B, Jaycox L. Evidence-based mental health programs in schools: barriers and facilitators of successful implementation. School Ment Health. Sep 2010;2(3):105-113. [FREE Full text] [CrossRef] [Medline]
    21. Cook C, Lyon A, Kubergovic D, Browning Wright D, Zhang Y. A supportive beliefs intervention to facilitate the implementation of evidence-based practices within a multi-tiered system of supports. School Ment Health. Jan 21, 2015;7(1):49-60. [FREE Full text] [CrossRef]
    22. Lyon AR, Ludwig K, Romano E, Leonard S, Stoep AV, McCauley E. "If it's worth my time, i will make the time": school-based providers' decision-making about participating in an evidence-based psychotherapy consultation program. Adm Policy Ment Health. Nov 2013;40(6):467-481. [FREE Full text] [CrossRef] [Medline]
    23. Polaha J, Dalton W, Allen S. The prevalence of emotional and behavior problems in pediatric primary care serving rural children. J Pediatr Psychol. Jul 2011;36(6):652-660. [CrossRef] [Medline]
    24. Cooper S, Valleley RJ, Polaha J, Begeny J, Evans JH. Running out of time: physician management of behavioral health concerns in rural pediatric primary care. Pediatrics. Jul 2006;118(1):e132-e138. [CrossRef] [Medline]
    25. Costello EJ, Angold A, Burns BJ, Stangl DK, Tweed DL, Erkanli A, et al. The great smoky mountains study of youth. Goals, design, methods, and the prevalence of DSM-III-R disorders. Arch Gen Psychiatry. Dec 1996;53(12):1129-1136. [CrossRef] [Medline]
    26. Goldsmith HF, Wagenfeld MO, Manderscheid RW, Stiles D. Specialty mental health services in metropolitan and nonmetropolitan areas: 1983 and 1990. Adm Policy Ment Health. Jul 1997;24(6):475-488. [CrossRef] [Medline]
    27. Jones DJ, Forehand R, Cuellar J, Kincaid C, Parent J, Fenton N, et al. Harnessing innovative technologies to advance children's mental health: behavioral parent training as an example. Clin Psychol Rev. Mar 2013;33(2):241-252. [FREE Full text] [CrossRef] [Medline]
    28. Mobile Fact Sheet. Pew Research Center. 2021. URL: http://www.pewinternet.org/fact-sheet/mobile/ [accessed 2021-09-19]
    29. Metzler CW, Sanders MR, Rusby JC, Crowley RN. Using consumer preference information to increase the reach and impact of media-based parenting interventions in a public health approach to parenting support. Behav Ther. Jun 2012;43(2):257-270. [FREE Full text] [CrossRef] [Medline]
    30. Poznanski B, Silva K, Conroy K, Georgiadis C, Comer J. Expanding the reach of evidence-based psychotherapy through remote technologies. In: Steele RG, Roberts MC, editors. Handbook of Evidence Based Therapies for Children and Adolescents. New York. Springer; 2020.
    31. Dickter B, Bunge EL, Brown LM, Leykin Y, Soares EE, Van Voorhees BW, et al. Impact of an online depression prevention intervention on suicide risk factors for adolescents and young adults. Mhealth. 2019;5:11. [FREE Full text] [CrossRef] [Medline]
    32. Hill C, Creswell C, Vigerland S, Nauta MH, March S, Donovan C, et al. Navigating the development and dissemination of internet cognitive behavioral therapy (iCBT) for anxiety disorders in children and young people: a consensus statement with recommendations from the #iCBTLorentz Workshop Group. Internet Interv. Feb 19, 2018;12:1-10. [FREE Full text] [CrossRef] [Medline]
    33. Stiles-Shields C, Crowe AN, Driscoll CF, Ohanian DM, Stern A, Wartman E, et al. A systematic review of behavioral intervention technologies for youth with chronic health conditions and physical and intellectual disabilities: implications for adolescents and young adults with Spina Bifida. J Pediatr Psychol. Apr 01, 2019;44(3):349-362. [FREE Full text] [CrossRef] [Medline]
    34. Van Voorhees BW, Gladstone T, Cordel S, Marko-Holguin M, Beardslee W, Kuwabara S, et al. Development of a technology-based behavioral vaccine to prevent adolescent depression: a health system integration model. Internet Interv. Sep 2015;2(3):303-313. [FREE Full text] [CrossRef] [Medline]
    35. McGoron L, Hvizdos E, Bocknek EL, Montgomery E, Ondersma SJ. Feasibility of internet-based parent training for low-income parents of young children. Child Youth Serv Rev. Jan 2018;84:198-205. [FREE Full text] [CrossRef] [Medline]
    36. Cairns K, Potter S, Nicholas M, Buhagiar K. Development of ReachOut Parents: a multi-component online program targeting parents to improve youth mental health outcomes. Adv Ment Health. May 25, 2018;17(1):55-71. [CrossRef]
    37. Baker S, Sanders MR, Morawska A. Who uses online parenting support? A cross-sectional survey exploring Australian parents’ internet use for parenting. J Child Fam Stud. Nov 14, 2016;26(3):916-927. [CrossRef]
    38. Enebrink P, Högström J, Forster M, Ghaderi A. Internet-based parent management training: a randomized controlled study. Behav Res Ther. Apr 2012;50(4):240-249. [CrossRef] [Medline]
    39. Sanders MR, Baker S, Turner KM. A randomized controlled trial evaluating the efficacy of Triple P Online with parents of children with early-onset conduct problems. Behav Res Ther. Nov 2012;50(11):675-684. [CrossRef] [Medline]
    40. Cotter KL, Bacallao M, Smokowski PR, Robertson CI. Parenting interventions implementation science: how delivery format impacts the parenting wisely program. Res Soc Work Pract. May 31, 2013;23(6):639-650. [CrossRef]
    41. Breitenstein SM, Gross D, Christophersen R. Digital delivery methods of parenting training interventions: a systematic review. Worldviews Evid Based Nurs. Jun 2014;11(3):168-176. [CrossRef] [Medline]
    42. Callejas E, Byrne S, Rodrigo MJ. ‘Gaining health and wellbeing from birth to three’: a web-based positive parenting programme for primary care settings. Early Child Dev Care. Jul 03, 2018;188(11):1553-1566. [CrossRef]
    43. Breitenstein SM, Gross D, Bettencourt AF. The Chicago parent program. In: Gershoff ET, Lee SJ, editors. Ending the physical punishment of children: A guide for clinicians and practitioners. Washington, DC. American Psychological Association; 2020:109-119.
    44. Breitenstein SM, Laurent S, Pabalan L, Risser HJ, Roper P, Saba MT, et al. Implementation findings from an effectiveness-implementation trial of tablet-based parent training in pediatric primary care. Fam Syst Health. Dec 2019;37(4):282-290. [FREE Full text] [CrossRef] [Medline]
    45. Brager J, Breitenstein SM, Miller H, Gross D. Low-income parents' perceptions of and engagement with a digital behavioral parent training program: a mixed-methods study. J Am Psychiatr Nurses Assoc. 2021;27(1):33-43. [CrossRef] [Medline]
    46. Breitenstein SM, Brager J, Ocampo EV, Fogg L. Engagement and adherence with ezPARENT, an mHealth parent-training program promoting child well-being. Child Maltreat. Nov 2017;22(4):295-304. [CrossRef] [Medline]
    47. Breitenstein SM, Fehrenbacher C, Holod AF, Schoeny ME. A randomized trial of digitally delivered, self-administered parent training in primary care: effects on parenting and child behavior. J Pediatr. Apr 2021;231:207-14.e4. [CrossRef] [Medline]
    48. Yarnall KS, Pollak KI, Østbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention? Am J Public Health. Apr 2003;93(4):635-641. [CrossRef] [Medline]
    49. Finnell S, Stanton J, Downs S. Actionable recommendations in the Bright Futures child health supervision guidelines. Appl Clin Inform. Jul 23, 2014;5(3):651-659. [FREE Full text] [CrossRef] [Medline]
    50. Schonwald A, Horan K, Huntington N. Developmental screening: is there enough time? Clin Pediatr (Phila). Jul 2009;48(6):648-655. [CrossRef] [Medline]
    51. Schuster MA, Duan N, Regalado M, Klein DJ. Anticipatory guidance: what information do parents receive? What information do they want? Arch Pediatr Adolesc Med. Dec 2000;154(12):1191-1198. [CrossRef] [Medline]
    52. Young KT, Davis K, Schoen C, Parker S. Listening to parents. A national survey of parents with young children. Arch Pediatr Adolesc Med. Mar 1998;152(3):255-262. [Medline]
    53. Combs-Orme T, Holden Nixon B, Herrod HG. Anticipatory guidance and early child development: pediatrician advice, parent behaviors, and unmet needs as reported by parents from different backgrounds. Clin Pediatr (Phila). Aug 2011;50(8):729-737. [CrossRef] [Medline]
    54. Huebner C, Riedy C. Behavioral determinants of brushing young children's teeth: implications for anticipatory guidance. Pediatr Dent. 2010;32(1):48-55. [FREE Full text] [Medline]
    55. Perrin EC, Sheldrick RC, McMenamy JM, Henson BS, Carter AS. Improving parenting skills for families of young children in pediatric settings: a randomized clinical trial. JAMA Pediatr. Jan 01, 2014;168(1):16-24. [CrossRef] [Medline]
    56. Johnson JG, Cohen P, Kasen S, Smailes E, Brook JS. Association of maladaptive parental behavior with psychiatric disorder among parents and their offspring. Arch Gen Psychiatry. May 01, 2001;58(5):453-460. [CrossRef] [Medline]
    57. Wakida EK, Talib ZM, Akena D, Okello ES, Kinengyere A, Mindra A, et al. Barriers and facilitators to the integration of mental health services into primary health care: a systematic review. Syst Rev. Nov 28, 2018;7(1):211. [FREE Full text] [CrossRef] [Medline]
    58. Fehrenbacher C, Schoeny ME, Reed M, Shattell M, Breitenstein SM. Referral to digital parent training in primary care: facilitators and barriers. Clin Pract Pediatr Psychol. Sep 2020;8(3):268-277. [CrossRef]
    59. Lyon AR, Munson SA, Renn BN, Atkins DC, Pullmann MD, Friedman E, et al. Use of human-centered design to improve implementation of evidence-based psychotherapies in low-resource communities: protocol for studies applying a framework to assess usability
. JMIR Res Protoc. Oct 09, 2019;8(10):e14990. [FREE Full text] [CrossRef] [Medline]
    60. Hermes ED, Lyon AR, Schueller SM, Glass JE. Measuring the implementation of behavioral intervention technologies: recharacterization of established outcomes. J Med Internet Res. Jan 25, 2019;21(1):e11752. [FREE Full text] [CrossRef] [Medline]
    61. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019;7:64. [FREE Full text] [CrossRef] [Medline]
    62. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. Sep 1999;89(9):1322-1327. [CrossRef] [Medline]
    63. McCreight M, Rabin B, Glasgow R, Ayele RA, Leonard CA, Gilmartin HM, et al. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. Nov 25, 2019;9(6):1002-1011. [CrossRef] [Medline]
    64. Creswell J, Plano CV. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA. Sage Publications Inc; 2017.
    65. Hagan JF, Shaw JS, Duncan PM, editors. Bright Futures: Guidelines for Health Supervision of Infants, Children and Adolescents, 4th Edition. Elk Grove Village, IL. American Academy of Pediatrics; 2017.
    66. Levy S, Siqueira LM, Committee on Substance Abuse, Ammerman SD, Gonzalez PK, Ryan SA, et al. Testing for drugs of abuse in children and adolescents. Pediatrics. Jun 2014;133(6):e1798-e1807. [CrossRef] [Medline]
    67. Breitenstein SM, Schoeny M, Risser H, Johnson T. A study protocol testing the implementation, efficacy, and cost effectiveness of the ezParent program in pediatric primary care. Contemp Clin Trials. Sep 2016;50:229-237. [FREE Full text] [CrossRef] [Medline]
    68. Bourchtein E, Langberg JM, Eadeh H. A review of pediatric nonpharmacological sleep interventions: effects on sleep, secondary outcomes, and populations with co-occurring mental health conditions. Behav Ther. Jan 2020;51(1):27-41. [CrossRef] [Medline]
    69. Gonzalez GE, Brossart DF. Telehealth videoconferencing psychotherapy in rural primary care. J Rural Ment Health. Jul 2015;39(3-4):137-152. [CrossRef]
    70. Hughes MC, Gorman JM, Ren Y, Khalid S, Clayton C. Increasing access to rural mental health care using hybrid care that includes telepsychiatry. J Rural Ment Health. Jan 2019;43(1):30-37. [CrossRef]
    71. Biglan A, Flay BR, Embry DD, Sandler IN. The critical role of nurturing environments for promoting human well-being. Am Psychol. 2012;67(4):257-271. [CrossRef]
    72. Stein BD, Adams AS, Chambers DA. A Learning Behavioral Health Care System: Opportunities to Enhance Research. Psychiatr Serv. Sep 01, 2016;67(9):1019-1022. [CrossRef] [Medline]
    73. Embry DD. Behavioral vaccines and evidence-based kernels: nonpharmaceutical approaches for the prevention of mental, emotional, and behavioral disorders. Psychiatr Clin North Am. Mar 2011;34(1):1-34. [FREE Full text] [CrossRef] [Medline]
    74. Pidano AE, Segool NK, Delgado N, Forness K, Hagen K, Gurganus EA, et al. Parent perceptions of pediatric primary care providers' mental health-related communication and practices. J Pediatr Health Care. 2020;34(5):e49-e58. [CrossRef] [Medline]
    75. Scholer SJ, Hudnut-Beumler J, Mukherjee A, Dietrich MS. A brief intervention facilitates discussions about discipline in pediatric primary care. Clin Pediatr. Jul 14, 2015;54(8):732-737. [CrossRef] [Medline]

    ‎
    AAP: American Academy of Pediatrics
    BIT: behavioral intervention technology
    FTF: face-to-face
    HCS: health care stakeholders
    PCC: primary care clinician
    PRISM: Pragmatic Robust Implementation and Sustainability Model
    RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance

    Edited by G Eysenbach; submitted 29.01.21; peer-reviewed by A Riley, B Chaudhry; comments to author 22.02.21; revised version received 01.07.21; accepted 28.07.21; published 05.10.21.

    Copyright

    ©Sean M O'Dell, Heidi R Fisher, Victoria Schlieder, Tracey Klinger, Rachel L Kininger, McKenna Cosottile, Stacey Cummings, Kathy DeHart. Originally published in JMIR Pediatrics and Parenting (https://pediatrics.jmir.org), 05.10.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.