This paper is in the following e-collection/theme issue:

Substance and Alcohol Abuse in Adolescence (96) Substance Abuse (382) Prevention and Health Promotion (891) Alcohol Abuse, Alcoholism Prevention and Recovery (284) Electronic/Mobile Data Capture, Internet-based Survey & Research Methodology (788) Design and Usability of Websites for Special User Groups (132) Human Factors and Usability Case Studies (746) Web-based and Mobile Health Interventions (3113)

Published on in Vol 5, No 3 (2022): Jul-Sep

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/34721, first published .
Evaluation of Positive Choices, a National Initiative to Disseminate Evidence-Based Alcohol and Other Drug Prevention Strategies: Web-Based Survey Study

Evaluation of Positive Choices, a National Initiative to Disseminate Evidence-Based Alcohol and Other Drug Prevention Strategies: Web-Based Survey Study

Evaluation of Positive Choices, a National Initiative to Disseminate Evidence-Based Alcohol and Other Drug Prevention Strategies: Web-Based Survey Study

Authors of this article:

Lexine Ann Stapinski1 Author Orcid Image ;   Smriti Nepal1, 2 Author Orcid Image ;   Tara Guckel1 Author Orcid Image ;   Lucinda Rachel Grummitt1 Author Orcid Image ;   Cath Chapman1 Author Orcid Image ;   Samantha Jane Lynch1 Author Orcid Image ;   Siobhan Maree Lawler1 Author Orcid Image ;   Maree Teesson1 Author Orcid Image ;   Nicola Clare Newton1 Author Orcid Image
Article Authors Cited by (1) Tweetations (29) Metrics

    Original Paper

    1The Matilda Centre for Research in Mental Health and Substance Use, The University of Sydney, Camperdown, Australia

    2SAX Institute, Glebe, Australia

    *these authors contributed equally

    Corresponding Author:

    Tara Guckel, BHlthMedSci (Hons)

    The Matilda Centre for Research in Mental Health and Substance Use

    The University of Sydney

    Level 6 Jane Foss Russell Building, G02

    Camperdown, 2006

    Australia

    Phone: 61 0426200221

    Email: tara.guckel@sydney.edu.au


    Background: To prevent adolescents from initiating alcohol and other drug use and reduce the associated harms, effective strategies need to be implemented. Despite their availability, effective school-based programs and evidence-informed parental guidelines are not consistently implemented. The Positive Choices alcohol and other drug prevention initiative and website was launched to address this research and practice gap. The intended end users were school staff, parents, and school students. An 8-month postlaunch evaluation of the website showed that end users generally had positive feedback on the website’s usability, and following its use, most of them would consider the evidence base and effectiveness of drug education resources. This study extends this initial evaluation by examining the effectiveness and impact of the Positive Choices initiative over a 3-year period.

    Objective: Guided by the five dimensions of the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework, the study assessed the impact of the Positive Choices initiative in increasing awareness and implementation of evidence-based drug prevention.

    Methods: Data were collected between 2017 and 2019, using web-based evaluation and community awareness surveys. Data from the surveys were merged to examine reach, effectiveness, adoption, implementation, and maintenance using descriptive statistics. Google Analytics was used to further understand the reach of the website. The System Usability Scale was used to measure website usability. In addition, inductive analysis was used to assess the participants’ feedback about Positive Choices.

    Results: A total of 5 years after launching, the Positive Choices website has reached 1.7 million users. A national Australian campaign increased awareness from 8% to 14% among school staff and from 15% to 22% among parents after the campaign. Following a brief interaction with the website, most participants, who were not already following the recommended strategies, reported an intention to shift toward evidence-based practices. The System Usability Scale score for the website was good for both user groups. The participants intended to maintain their use of the Positive Choices website in the future. Both user groups reported high level of confidence in communicating about topics related to alcohol and other drugs. Participants’ suggestions for improvement informed a recent website update.

    Conclusions: The Positive Choices website has the capacity to be an effective strategy for disseminating evidence-based drug prevention information and resources widely. The findings highlight the importance of investing in ongoing maintenance and promotion to enhance awareness of health websites. With the increased use and acceptability of health education websites, teams should ensure that websites are easy to navigate, are engaging, use simple language, contain evidence-informed resources, and are supported by ongoing promotional activities.

    JMIR Pediatr Parent 2022;5(3):e34721

    doi:10.2196/34721

    Keywords

    alcohol and other drugs; prevention; adolescence; schools; drug prevention; drug prevention website 


    Background

    Adolescence is marked by considerable emotional, social, and physical changes, including increasing autonomy from parents, greater influence from peers, and increased likelihood to engage in risk-taking behaviors [1]. Corresponding to this period of experimentation, harms associated with risky behaviors such as alcohol and other drug (AOD) use peak during adolescence and early adulthood [2-4]. Globally, alcohol, tobacco, and cannabis are the most commonly used drugs among adolescents [4]. This also holds true for Australia, where a 2017 national survey among secondary school students aged 12 to 17 years showed that 46% had consumed a full serve of alcohol, 14% had used cannabis, and 13% had smoked cigarettes in the past year [5]. Early initiation of AOD use is associated with a range of negative outcomes including poor school performance, truancy, school dropout, juvenile offending, and increased risk of drug dependence and mental illness during adulthood [6-9]. To interrupt this trajectory and reduce the harms associated with AOD use, effective strategies are needed to prevent the onset and escalation of their use.

    Studies have identified a number of modifiable individual risk factors that are consistently associated with AOD use among adolescents [10]. They include peer AOD use, low self-efficacy to refuse alcohol or other drugs, poor school engagement and connectedness, and mental health disorders such as depression and attention-deficit/hyperactivity disorder [10-12]. Studies also highlight the importance of parents in influencing and preventing adolescents’ AOD use. Recent evidence shows an association between parental supply of alcohol and increased risk of alcohol-related harm during adolescence and early adulthood [13,14]. This has challenged the commonly held perception that allowing teenagers to drink alcohol under parental supervision protects them against alcohol-related harms. Similarly, less restrictive parental attitudes toward alcohol use are likely to lead to earlier and more frequent alcohol use and increased drunkenness among adolescents [15].

    Prevention programs targeting individual and parental risk factors are effective in reducing AOD use among adolescents [16-22]. A number of prevention programs implemented during secondary school have consistently demonstrated effectiveness in reducing AOD use [18,21], and school-based delivery offers a number of advantages including tailoring of messages to students’ developmental level and universal delivery to achieve wide reach [22,23]. Internet- and computer-based AOD prevention programs offer additional benefits in terms of engaging youth and increasing implementation fidelity and have been found to be effective in reducing adolescent AOD use [17]. Despite the growing evidence base, only a small proportion of schools implement effective AOD prevention strategies. A 2003 review of 3 decades of AOD education studies concluded that worldwide, effective AOD prevention is not widely implemented [24]. More recently, our 2017 survey of Australian schoolteachers found that <25% of teachers had implemented evidence-based AOD education programs. The study identified lack of confidence, resources, time, and support from school; attitude of parents and students; and difficulty in communicating as the main barriers to implementing programs [25].

    Parenting strategies that are consistently associated with delayed initiation of alcohol use include parental monitoring, limited access to alcohol, parent-child relationship quality, parental involvement, and communication [20,26]. However, studies show that parents do not always act in accordance with evidence-based parenting strategies, which may be related to lack of clear guidance or confidence [27,28]. Although parents actively seek information about illicit drugs and parenting practices to prevent AOD use, primarily from friends or the internet, they often report low-to-moderate confidence in communicating and influencing their children’s choices regarding AOD use [29]. Programs aimed at modifying parenting practices and promoting the use of effective prevention strategies have shown promise in reducing adolescent AOD use [30-32]. Honest communication between parents and adolescents, established on the basis of positive parent-adolescent relationship, has been associated with reduced alcohol use among adolescents [32-34]. This highlights the importance of engaging parents in efforts to reduce harms associated with adolescent AOD use.

    It is critical that effective AOD prevention strategies are implemented consistently and widely to alleviate the considerable burden associated with AOD use. Therefore, teachers, school staff, and parents need to have easy access to evidence-based information, strategies, and programs that equip them to respond most effectively. The Positive Choices national AOD prevention initiative was funded by the Australian Government Department of Health to enhance access to and implementation of evidence-based AOD prevention strategies within school communities. The initiative responds to a call from school principals for support to implement evidence-based AOD prevention resources (Australian National Council on Drugs; 2013) and was developed iteratively with experts and end users. School staff, parents, and students provided input and feedback across two phases (for full details refer to the study by Stapinski et al [25]): (1) formative consultation to clarify scope and identify user needs and (2) review and feedback on a beta version of the website [35]. The final website was launched in December 2015 and provides web-based training, support, and access to a centralized database of evidence-based AOD prevention programs, recommendations, and resources [25]. The website emphasizes the importance of implementing resources that are supported by research evidence—only evidence-based resources are listed, and each resource page provides a rating to indicate the strength of the supporting evidence according to the Australian National Health and Medical Research Council’s evidence hierarchy [36]. To facilitate implementation, resources are categorized based on purpose and intended audience in a searchable database, with brief factsheets to guide users about when and how to implement evidence-based prevention strategies. A survey was conducted 8 months after the launch to determine the initial impact of Positive Choices. Among teachers who accessed the website, most found it useful and reported that they would continue using it, would recommend it to others, felt more comfortable discussing AOD use prevention following website access, and felt that their students were more engaged with AOD education since using the website [25]. When compared with a general teacher sample, teachers who used Positive Choices were more likely to consider the evidence base and effectiveness of AOD education resources before using them in class [25]. Despite highlighting these benefits associated with the use of Positive Choices, this initial evaluation did not reveal the specific factors or features that facilitated the dissemination of evidence-based AOD prevention strategies.

    Objectives

    In this study, we extended this initial evaluation by conducting a more comprehensive examination of the effectiveness and impact of the Positive Choices initiative over a 3-year period, between 2017 and 2019. This is the first evaluation of a web-based health initiative that specifically promotes translation of evidence-based AOD prevention resources to school communities, parents, and youth. To guide this evaluation, we applied the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework, which was developed to facilitate comprehensive and rigorous evaluation of health promotion initiatives, spanning these 5 key dimensions. It has been widely applied to evaluate the implementation and real-world impact across a variety of settings, including educational settings [37-40]. Guided by this framework, this study assessed the impact of the Positive Choices initiative in increasing awareness and implementation of evidence-based AOD prevention strategies. The study aimed to evaluate the reach of the Positive Choices website, its effectiveness in improving access to and uptake of effective AOD prevention strategies, and its adoption in accessing evidence-based prevention strategies. Furthermore, implementation will be evaluated by users’ capacity to interact with the website, and maintenance will be evaluated by users’ intention to continue using the website for evidence-based strategies. The study also obtained feedback from end users to improve and optimize the Positive Choices website.


    Data were collected between 2017 and 2019, from several sources to evaluate the dimensions aligned with the RE-AIM framework, as described in the following sections.

    Reach

    Overview

    Reach was assessed via the measurement of access and awareness, using 2 data sources. Access was measured using site use analytics and operationalized as follows: How many unique users have accessed Positive Choices? Awareness was assessed via community awareness surveys and operationalized as follows: Have you heard of Positive Choices before?

    Site Use Analytics

    Google Analytics was used to obtain a detailed analysis of website traffic between January 2017 and March 2021. This included information on the number of unique website users and page views, pages viewed per session, and average time users spent on each page.

    Community Awareness Survey: School Staff and Parents

    A web-based survey was administered in July 2018 to assess the Australian community’s awareness and use of Positive Choices resources. Using voluntary response sampling, school staff and parents were recruited via targeted advertisements on Positive Choices social media channels or mailing lists. Participants who completed the survey were offered the chance to enter a prize draw to win a laptop. Within this general sample, the proportion of the community that was aware of the website was identified by using a single question: Have you heard of Positive Choices before? Information was also collected to ascertain the number of participants who were using sources other than Positive Choices and how participants were accessing AOD information. The same survey was administered after 6 months (November 2018) and 12 months (May 2019) to determine whether awareness increased following a national social media campaign promoting Positive Choices.

    Effectiveness

    Overview

    Positive Choices aims to improve access to and uptake of effective AOD prevention strategies. Accordingly, effectiveness was assessed via an evaluation survey completed by Positive Choices end users (school staff and parents). It was measured through examination of whether engagement with Positive Choices was associated with increased intention to implement evidence-based AOD education and was operationalized as follows: Has the use of Positive Choices changed users’ intentions in implementing evidence-based teaching and parenting practices?

    Evaluation Survey: School Staff and Parents

    To capture the effectiveness of Positive Choices, two anonymous evaluation surveys were administered: the first between August 2017 and September 2017 and the second between May 2019 and June 2019. Using voluntary random sampling, participants were recruited via the Positive Choices mailing list and social media campaigns. Eligible participants were school staff or parents or guardians of children or adolescents, were Australian residents, and had access to the internet and a device to complete the survey. Participants were reimbursed for their time with an Aus $40 (US $28) gift voucher (2017) or the opportunity to enter a prize draw to win a laptop (2019).

    Participants were asked to spend time reading and interacting with the Positive Choices resources, after which they reported on their intentions to implement evidence-based prevention strategies. Data were collected via a web-based survey platform, Survey Monkey, and responses from the 2 evaluations were collated to provide an overview of the effectiveness spanning from 2017 to 2019.

    For school staff, questions assessed whether they intended to (1) implement teaching resources that have been tested in schools and proven to prevent AOD use, (2) communicate with students about the risks and effects associated with AOD use, and (3) correct the misperception that AOD use is the norm. For parents, questions assessed whether they intended to (1) encourage open communication with their children about AOD, (2) have explicit conversation with their children about AOD use, (3) correct the misperception that AOD use is common, (4) clearly communicate about their expectations to their children, (5) change their own AOD use to model appropriate behavior, (6) avoid parental supply of alcohol, and (7) closely monitor their children’s whereabouts. School staff and parents responded to these behavioral intention items by selecting whether they (1) already do or have done this, (2) plan to do this after viewing Positive Choices, or (3) do not plan to do this in the future.

    Adoption and Implementation

    Similar to the effectiveness dimension, the adoption and implementation dimensions were assessed via the web-based evaluation surveys. Adoption was assessed by identifying barriers to and enablers of access and uptake of evidence-based prevention strategies by school staff and parents. A single question assessed which of the following characteristics of web-based AOD prevention resources or information were most valued by school staff: (1) evidence-based information, (2) resources that had been tested in schools and proven to prevent AOD use, (3) engaging website, (4) interactive website, (5) website that is easy to navigate and use, and (6) simple and easy-to-understand language. Similarly, enablers for parents and guardians were identified through the same items with the addition of the following two items that pertained specifically to parents: (1) parental strategies that have been proven to be effective and (2) website with advice from other parents.

    Implementation was evaluated using the System Usability Scale (SUS) [41]. It is a standardized instrument used to measure the usability of products, software, apps, and websites. It provides participants with 10 usability-related statements on a 5-point Likert scale, ranging from “strongly disagree” to “strongly agree.” SUS scores range from 0 to 100, with score ≥85 representing exceptional usability, score between 50 and 70 representing good usability, and score <50 representing unacceptable usability [41,42]. As such, participants’ capacity to interact with the website to access and subsequently deliver evidence-based strategies was assessed using the SUS scores.

    Maintenance

    This dimension was assessed using the website evaluation surveys, as described previously. For the participating school staff and parents, the question assessed whether users intended to access Positive Choices in the future. It was conceptualized as follows: Did participants intend to maintain their use of the Positive Choices resources?

    General Impression

    To assess school staff and parents’ general feedback about the Positive Choices website, the evaluation survey included the following open-text items: (1) Do you have any suggestions for improving the website? and (2) Do you have any final comments about the website?

    Data Analysis

    The evaluation surveys were administered via Survey Monkey, and the data were exported to Stata (version 14; StataCorp), which was used to generate descriptive data, including mean scores and response frequencies. Data from the 2017 and 2019 evaluation surveys were merged to examine reach, effectiveness, adoption, implementation, and maintenance, using descriptive statistics, separately for school staff and parents. The SUS score was calculated by converting raw individual scores for each question to a number; these scores were summed to obtain the total score, 5 was subtracted from the total score of all odd-numbered questions, and 25 was subtracted from the total score of all even-numbered questions. Then, the total score of the new values was multiplied by 2.5 [43].

    Data on community awareness were also collected via Survey Monkey and exported to Stata to generate descriptive data. The data were examined separately for school staff and parents at baseline, 6 months, and 12 months.

    Inductive analysis was used to assess participants’ feedback about Positive Choices [44], using the qualitative data analysis software, NVivo (version 12; QSR International). Feedback from participants was examined by SN, who then developed a coding framework. Using the framework, common themes were identified independently by two coders (SN and TG).

    Ethics Approval

    The 2017 survey was approved by the Human Research Ethics Committee, University of New South Wales (project number HC12548), and the 2019 survey was approved by the Human Research Ethics Committee, University of Sydney (project number 2018/873).


    Sample Characteristics: Evaluation Survey

    From 2017 to 2019, a total of 200 participants completed the evaluation surveys, of which 73 (36.5%) participants were school staff and 127 (63.5%) were parents.

    School Staff

    Table 1 shows the demographic characteristics of the school staff who completed the evaluation surveys. The average age of the participants was 40 (SD 10.71) years and ranged between 24 and 63 years. Most participants were women (56/73, 77%), resided in New South Wales (NSW; 29/73, 40%), and were based in major cities (36/73, 49%). Most school staff worked in coeducational, secondary schools (ie, years 7-12; 51/73, 70%), and were employed as teachers (48/73, 66%).

    Table 1. Demographic characteristics of school staff and parents (evaluation survey).
    CharacteristicsSchool staff (n=73)Parents (n=127)
    Age (years), range24-6326-63
    Gender, n (%)

    		Women56 (77)108 (85)

    		Men16 (22)19 (14.9)

    		Nonbinary1 (1)0 (0)
    State or territory, n (%)

    		Australian Capital Territory5 (7)4 (3.1)

    		New South Wales29 (40)45 (35.4)

    		Queensland10 (14)23 (18.1)

    		South Australia6 (8)6 (4.7)

    		Tasmania4 (5)2 (1.6)

    		Victoria16 (22)28 (22)

    		Western Australia3 (4)15 (11.8)

    		Northern Territory0 (0)4 (3.1)
    Location of school or residence, n (%)

    		Major city36 (49)78 (61.4)

    		Regional33 (45)48 (37.8)

    		Remote4 (5)1 (0.8)
    Year levels taught or children’s year levelsa, n (%)

    		Foundation5 (7)10 (7.9)

    		Year 18 (11)6 (4.7)

    		Year 28 (11)5 (3.9)

    		Year 38 (11)10 (7.9)

    		Year 47 (10)13 (10.2)

    		Year 59 (12)11 (8.7)

    		Year 68 (11)15 (11.8)

    		Year 738 (52)26 (20.5)

    		Year 837 (51)29 (22.8)

    		Year 938 (52)23 (18.1)

    		Year 1040 (55)24 (18.9)

    		Year 1139 (53)35 (27.6)

    		Year 1239 (53)33 (25.9)

    		N/Ab18 (25)N/A
    School type, n (%)

    		Coeducational63 (86)N/A

    		Single sex10 (14)N/A
    Profession, n (%)

    		Teacher48 (66)N/A

    		School counselor chaplain10 (14)N/A

    		Youth worker2 (3)N/A

    		Researcher2 (3)N/A

    		Other11 (15)N/A
    Employment status, n (%)

    		Full timeN/A52 (40.9)

    		Part time or casualN/A42 (33.1)

    		Home duties (including carer)N/A19 (14.9)

    		UnemployedN/A6 (4.7)

    		Unable to workN/A2 (1.6)

    		StudentN/A5 (3.9)

    		OtherN/A1 (0.8)

    aRespondents were able to select multiple responses; thus, the column total for this item does not add to 100%.

    bN/A: not applicable.

    Parents

    Table 1 also shows the demographic characteristics of the parents who completed the surveys. The average age of the parents was 46 (SD 6.63) years, and they were aged between 26 and 63 years. Most parent respondents were women (108/127, 85%), resided in NSW (45/127, 35.4%), and were based in major cities (78/127, 61.4%). Most parents were employed full time (52/127, 40.9%), with most of them having children who attended secondary school (ie, years 7-12; 122/127, 96.1%).

    Sample Characteristics: Community Awareness Survey

    Overview

    A total of 1435 participants completed the community awareness surveys across baseline, 6-month follow-up, and 12-month follow-up. At baseline, participants were 48.55% (201/414) school staff and 51.45% (213/414) parents. At the 6-month follow-up, participants were 51.45% (249/484) school staff and 48.55% (235/484) parents. Finally, at the 12-month follow-up, participants were 50.09% (269/537) school staff and 49.91 (268/537) parents.

    School Staff

    All school staff who completed the community awareness surveys were high school staff (719/719, 100%), most resided in NSW (259/719, 36%), and were based in major cities (381/719, 52.9%; Table 2). Most school staff (432/719, 60.1%) worked in public schools in a number of different roles (eg, head teacher, curriculum coordinator, language teacher, mathematics or science teacher, advanced skills teacher, student support or school counselor, and teacher assistant) and worked with Aboriginal or Torres Strait Islander students.

    Table 2. Demographic characteristics of school staff and parents (community awareness survey).
    CharacteristicsSchool staff (n=719), n (%)Parents (n=716), n (%)
    State or territory

    		Australian Capital Territory5 (0.7)15 (2.1)

    		New South Wales259 (36)207 (28.9)

    		Queensland147 (20.4)136 (18.9)

    		South Australia46 (6.4)75 (10.5)

    		Tasmania32 (4.5)19 (2.7)

    		Victoria134 (18.6)168 (23.5)

    		Western Australia85 (11.8)88 (12.3)

    		Northern Territory11 (1.5)8 (1.1)
    Location of school or residence

    		Major city381 (53)379 (52.9)

    		Regional185 (25.7)192 (26.8)

    		Rural93 (12.9)93 (12.9)

    		Remote34 (4.7)29 (4.1)

    		Very remote26 (3.6)23 (3.2)
    Works with Aboriginal and/or Torres Strait Islander student (school staff) or identified as Aboriginal and/or Torres Strait Islander (parents)

    		Yes568 (78.9)21 (2.9)

    		No122 (16.9)687 (95.9)

    		Prefer not to answer29 (4)8 (1.1)
    School

    		Public432 (60.1)N/Aa

    		Faith-based145 (20.2)N/A

    		Independent95 (13.2)N/A

    		Coeducational21 (2.9)N/A

    		Single sex (female)12 (1.7)N/A

    		Single sex (male)9 (1.3)N/A

    		Selective5 (0.7)N/A
    Family structureb

    		Single-parent householdN/A192 (26.8)

    		2-parent householdN/A500 (69.8)

    		1 childN/A69 (9.6)

    		2-4 childrenN/A397 (55.4)

    		≥5 childrenN/A39 (5.4)

    		Step siblings (cohabiting)N/A27 (3.8)

    		Prefer not to answerN/A2 (0.3)

    aN/A: not applicable.

    bRespondents were able to select multiple responses; thus, the column total for this item does not add to 100%.

    Parents

    All parents who completed the community awareness surveys (716/716, 100%) were parents of high school students. Most of them resided in NSW (207/716, 28.9%), were based in major cities (379/716, 52.9%), and did not identify as Aboriginal and/or Torres Strait Islander (687/716, 95.9%; Table 2). Most parents were part of 2-parent households (500/716, 69.8%) and had between 2 and 4 children (397/716, 55.4%).

    Evaluation of the Positive Choices Website (RE-AIM Framework)

    Reach: How Many Users Have Accessed Positive Choices?
    Overview

    Between January 1, 2017, and March 31, 2021, Positive Choices had been visited by 1.7 million unique users, of which 1.53 million (90%) were first-time users. Figure 1 shows that the number of monthly users has consistently increased since January 2017. Similarly, the number of monthly page views have increased from 8367 in January 2017 to 123,699 in March 2021. In terms of geographical reach, most of the website’s visitors are Australian residents (41%); however, the website is also frequently accessed by international users from countries such as the United States, India, the Philippines, and the United Kingdom.

    ‎
    Figure 1. Monthly Positive Choices website visitors, between January 2017 and March 2021.
    View this figure
    Community Awareness

    The spike in website visitors seen between July 2018 and November 2018 (Figure 1) coincided with the community awareness campaign run from June 2018 to October 2018. During this period, there was an 84% increase in the average monthly users of the Positive Choices website from 20,115 to 37,047. There was also an increase in users’ engagement on the Positive Choices Facebook page, with the number of likes increasing by 86%, from 670 to 1243. Results from the surveys showed that awareness about Positive Choices increased among school staff from 8% at baseline to 14% at 6 months (immediately after the awareness campaign) and 15% at 12 months. Among parents, awareness increased from 15% at baseline to 22% at 6 months (immediately after the awareness campaign) before returning to 15% at 12 months.

    According to the 12-month survey, of the school staff who were not using Positive Choices for AOD education resources and information (191/269,71%), 45.5% (87/191) were using Alcohol and Drug Foundation, 18.3% (35/191) were using Drug Free Australia, 14.1% (27/191) were using Australian Drug Information Network, 18.3% (35/191) were using Drug Help, 4.7% (9/191) were using Prevention Smart, and 6.3% (12/191) were using other sources (headspace, general practitioners, guidance officer, etc). Similarly, among the same group that was not using Positive Choices, approximately 69.6% (133/191) of school staff used websites (government, Google, or other websites) to access AOD information; 43.5% (83/191) obtained information through school administrators (eg, Department of Education and Catholic Education Office); and 13.6% (26/191) used other sources including AOD professionals, books, police, and so on. Similarly, among parents who were not using Positive Choices (201/268, 75%), 60.7% (122/201) parents reported that they accessed information through Google or other websites; 26.4% (53/201) from other parents; 30.3% (61/201) through government websites; 8.5% (17/201) through their general practitioners; 4.9% (10/201) through their children’s classroom teachers; and 7.9% (16/201) of parents were using other sources such as headspace, other health professionals, presenters at local AOD prevention events, and so on.

    Effectiveness: Has the Use of Positive Choices Changed Users’ Teaching and Parenting Practices?
    School Staff

    Table 3 shows that approximately half of the school staff sample (35/73, 48%) were already implementing AOD education resources that were tested and found to be effective in schools; among those who were not implementing those resources, 89% (34/38) intended to after using Positive Choices. Most respondents were already communicating to their students about the risks and effects of AOD use (52/73, 71%), and of the remaining respondents, 90% (19/21) intended to do so after using Positive Choices. When it came to correcting students’ misperceptions about AOD use, most respondents were already doing this (52/73, 71%) and of the remaining respondents, 81% (17/21) intended to commence this after viewing the Positive Choices website.

    Table 3. Effectiveness in changing intentions to use evidence-based strategies (school staff).
    Respondents who are currently implementing evidence-based strategy (n=73), n (%)Among respondents who were not implementing evidence-based strategies, those who intend to after viewing Positive Choices, n (%)Respondents who will not implement evidence-based strategies in the future, n (%)
    Implement teaching resources that were tested in schools and proven to prevent alcohol and drug use35 (48)34 (89)a4 (11)a
    Communicate with students about the risks and effects of alcohol and drug use53 (73)18 (90)b2 (10)b
    Correct the misperception that alcohol and other drug use is common or “the norm”52 (71)17 (81)c4 (19)c

    aSample size, n=38.

    bSample size, n=20.

    cSample size, n=21.

    Parents

    Table 4 shows that most parents reported that they were already having open (100/127, 78.7%) and clear conversations (95/127, 74.8%) with their children about AOD use. Among those parents who were not already doing this, 85% (23/27) planned to have open and 78% (25/32) planned to have clear conversations with their children after viewing resources on Positive Choices. Most parents reported that they were already having explicit conversations about AOD with their children (88/127, 69.3%), and most of the remaining parents (36/39, 92%) planned to do so after using Positive Choices. Most parents in the survey were already avoiding supplying alcohol to their children (103/127, 81.1%). Of those who were not doing this, 38% (9/24) reported that they would avoid supplying alcohol after viewing resources on Positive Choices. Similar to the school staff, most parents reported that they were already correcting misperceptions about AOD use being the norm (80/127, 62.9%), and of the remaining parents, 83% (39/47) planned to implement this behavior after using Positive Choices. Most parents reported that they have already adapted their own AOD use to model appropriate behavior for their children (87/127, 68.5%), and of the remaining parents, 45% (18/40) planned to modify their behavior after using Positive Choices. After viewing resources on Positive Choices, most parents reported that they will monitor their children’s whereabouts more closely (64/74, 86%).

    Table 4. Effectiveness in changing intentions (parents).
    Respondents who are currently implementing evidence-based strategy (n=127), n (%)Among respondents who were not implementing evidence-based strategies, those who intend to after viewing Positive Choices, n (%)Respondents who will not implement evidence-based strategies in the future, n (%)
    Encourage open communication with my child about alcohol and other drugs100 (78.7)23 (85)a4 (15)a
    Have an explicit conversation with my child about alcohol and other drugs88 (69.3)36 (92)b3 (8)b
    Correct the misperception that alcohol and other drug use is common or “the norm”81 (63.8)38 (83)c8 (17)c
    Clearly communicate my expectations about drug and alcohol use to my child95 (74.8)25 (78)d7 (22)d
    Change my own drug or alcohol use to model appropriate behavior87 (68.5)18 (45)e22 (55)e
    Avoid supplying my child with alcohol103 (81.1)9 (38)f15 (63)f
    Monitor my child’s whereabouts more closely54 (42.5)63 (86)g10 (14)g

    aSample size, n=27.

    bSample size, n=39.

    cSample size, n=46.

    dSample size, n=32.

    eSample size, n=40.

    fSample size, n=24.

    gSample size, n=73.

    Adoption and Implementation
    School Staff
    Overview

    School staff reported that they spent between 5 and 10 hours per semester on AOD education. Most school staff rated the following factors highly (either “very important” or “important”) when selecting web-based AOD prevention resources: evidence-based information (69/73, 95%), easy-to-navigate and easy-to-use website (69/73, 95%), engaging website (70/73, 96%), simple and easy-to-use language (68/73, 93%), and resources tested in school and found to be effective (62/73, 85%). Although school staff also valued interactive features of prevention websites (55/73, 75%), the proportion of participants who rated this factor highly was lower than those for the other factors facilitating effective AOD prevention. When discussing AOD topics, school staff displayed high levels of confidence, with 41% (30/73) feeling “very confident,” 38% (28/73) feeling “confident,” and 16% (12/73) feeling “somewhat confident”; only 4% (3/73) reported feeling “not very confident.”

    Implementation

    Regarding usability of the Positive Choices website, the mean SUS score was 75 (SD 15.1; range 35-100), indicating good website usability.

    Parents
    Overview

    Most parents rated the following factors highly (either “very important” or “important”) when selecting web-based AOD prevention resources: evidence-based information (119/127, 93.7%); strategies that were tested and proven to be effective in AOD use prevention (119/127, 93.7%); simple and easy-to-use language (109/127, 85.8%); and engaging (115/127, 90.6%), interactive (72/127, 56.7%), and easy-to-navigate and easy-to-use (123/127, 96.9%) website. In addition, 57.5% (73/127) of the parents valued prevention advice from other parents.

    Parents also reported high confidence in discussing AOD topics: 48.8% (62/127) were “very confident,” 31.5% (40/127) were “confident,” 17.3% (22/127) felt “somewhat confident,” and 2.4% (3/127) felt “not very confident.” These ratings suggest that confidence was not a significant barrier to evidence-based prevention in this sample. This is in contrast to the general community sample, from the community awareness survey, where lack of confidence was reported as the greatest barrier for parents to having conversations with their children about AOD use.

    Implementation

    The mean SUS score for the parent group was 74 (SD 12.9; range 40-100), indicating good website usability, similar to that found in the school staff group.

    Maintenance: Are Users Likely to Continue Accessing Positive Choices?
    School Staff

    Most school staff reported that they would use Positive Choices in the future (66/73, 90%), use the website frequently (53/73, 73%), and recommend the website to friends or colleagues (68/73, 93%).

    Parents

    Most parents also reported that they would use Positive Choices in the future (116/127, 91.3%), use the website frequently (78/127, 61.4%), and recommend the website to friends or colleagues (113/127, 88.9%).

    User Feedback on the Website

    The main themes that emerged from the analysis of users’ feedback on the website centered around the website’s content and features and promotion and increasing the usability and diversity of the website. Most responses from participants were suggestions to improve the Positive Choices website’s layout to make it less cluttered and more visually appealing and engaging (especially to the student user group) and to improve website navigation. The other subthemes that were identified included feedback about making the website more accessible to multicultural communities and students with additional learning needs, suggestions for new content and features, and the need to increase promotion. In response to this feedback, redesign of the Positive Choices website was conducted from December 2020 to January 2021, in consultation with a specialist user experience company. The themes and subthemes from the participants’ feedback and actions taken in response to user suggestions are described in Multimedia Appendix 1.


    Principal Findings

    Adolescence is a time of increased susceptibility to engaging in risk-taking behaviors such as AOD use. Prevention strategies designed to target modifiable risk factors have been demonstrated to be effective in reducing AOD use and related harms among adolescents [16,18]. Despite this evidence, effective AOD prevention strategies are not widely implemented in schools or by parents. Positive Choices was developed to help overcome some of the barriers faced by school communities and parents and promote widespread implementation of evidence-based prevention. This study used the RE-AIM framework to evaluate the success of Positive Choices in increasing awareness and implementation of evidence-based AOD prevention practices among school staff and parents across the five domains: reach, efficacy, adoption, implementation, and maintenance.

    A total of 5 years following its launch, this Australian AOD prevention website has reached >1.7 million users, and the page views have continued to grow. The user reach of Positive Choices has expanded beyond Australia, with 59% of users currently being international. Promotion via a national social media campaign proved to be an effective method for increasing awareness within the Australian community, with awareness increasing from 8% to 14% among school staff and from 15% to 22% among parents after the campaign. The findings suggest that sustained promotion efforts may be required to maintain high levels of awareness about Positive Choices. Even among participants who were not using Positive Choices, most reported using other web-based sources to obtain AOD education information and resources, thus supporting the value of the internet as a tool for disseminating health information and promoting evidence-based prevention strategies [45,46].

    Although evidence shows that effective AOD prevention strategies are not commonly implemented by schools [24] or parents [27], most school staff and parents in this study reported that they were already implementing evidence-based AOD prevention strategies. Moreover, following a brief interaction with the website, most school staff and parents reported an intention to shift toward evidence-based practices in cases where they were not already following the recommended prevention strategies for young people. The exception was the recommendation that parents modify their own alcohol use to model appropriate behavior for their children, which may indicate that changing their own behavior is a significant challenge for parents, which requires additional attention and support. The sample intended to maintain their use of the Positive Choices website in the future to obtain evidence-based information and resources. The enablers that facilitated adoption and implementation were ease of availability and good website usability. Both user groups reported high level of confidence in communicating about AOD-related topics. Study participants’ suggestions for improvement pertained to some of these enablers (eg, improving navigation and making the website more engaging), and thus, have informed a recent website update.

    The widespread use of web-based health promotion tools and websites by school staff and parents to access information and resources highlights the need for quality control measures. This will ensure that they are evidence-based, up to date, and engaging and use simple language. In addition, the websites themselves should be accessible and easy to use and navigate. Findings from robust evaluations will allow critical assessment of the benefits of such tools and websites, inform content, and inform website updates and developments to optimize their usability. Although there have been evaluations of mental health information websites [47,48], to the best of our knowledge, this is the first evaluation of a web-based health initiative that specifically promotes the translation of evidence-based AOD prevention resources for school communities, parents, and youth.

    Limitations

    A limitation of the study is that the findings rely on users’ self-reported intentions to implement evidence-based strategies, rather than actual assessment of their subsequent behavior. Furthermore, the study design assessed participants’ feedback and behavioral intentions after they interacted with the website for a relatively short period. Future studies with a pre-post study design will enable a more comprehensive evaluation, including assessment of whether the website affected subsequent behaviors and implementation of evidence-based strategies by school staff and parents. Another limitation is that the targeted campaign used to recruit participants for the evaluation study may have resulted in a sample selection bias. Most of the surveyed school staff and parents in the current sample reported that they were already implementing evidence-based prevention strategies and were confident about discussing alcohol and drug use. These results contrast with previous evidence suggesting low confidence and implementation of evidence-based AOD prevention strategies among parents and in schools [24,27], and thus, may reflect that our recruitment strategies attracted a sample who were already interested in and aware of evidence-based AOD prevention approaches.

    Conclusions

    The findings from the evaluation of Positive Choices demonstrate that the website reached 1.7 million users, and it has the capacity to be an effective strategy for disseminating evidence-based AOD prevention information and resources. Furthermore, this evaluation highlights the importance of investing in ongoing promotion to maintain or enhance awareness of health websites. As the use and acceptability of health education websites increase, developers and health care and research teams should ensure that health websites developed in the future are easy to navigate, are engaging, use simple language, contain evidence-informed resources, and are supported by ongoing promotional activities. This study provides methodology and recommendations to guide future evaluations of web-based health tools to determine their effect on behavioral and health outcomes.

    Acknowledgments

    Positive Choices is supported by funding from the Australian Government Department of Health. This study was also supported by the Australian National Health and Medical Research Council via fellowships (NCN: GNT1166377, LAS: GNT1132853, and MT: GNT1078407 and GNT1195284) and the Centre of Research Excellence in the Prevention and Early Intervention in Mental Illness and Substance Use (GNT11349009). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. Positive Choices was developed by researchers at the Matilda Centre for Research in Mental Health and Substance Use at the University of Sydney (LAS, SN, TG, LRG, CC, SJL, SML, MT, and NCN). The authors receive no financial or other benefits from the Positive Choices website.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Themes and subthemes from participants’ feedback on the website.

    DOCX File , 30 KB
    1. Vijayakumar N, Op de Macks Z, Shirtcliff EA, Pfeifer JH. Puberty and the human brain: insights into adolescent development. Neurosci Biobehav Rev 2018 Sep;92:417-436 [FREE Full text] [CrossRef] [Medline]
    2. Degenhardt L, O'Loughlin C, Swift W, Romaniuk H, Carlin J, Coffey C, et al. The persistence of adolescent binge drinking into adulthood: findings from a 15-year prospective cohort study. BMJ Open 2013 Aug 19;3(8):e003015 [FREE Full text] [CrossRef] [Medline]
    3. Palmer RH, Young SE, Hopfer CJ, Corley RP, Stallings MC, Crowley TJ, et al. Developmental epidemiology of drug use and abuse in adolescence and young adulthood: evidence of generalized risk. Drug Alcohol Depend 2009 Jun 01;102(1-3):78-87 [FREE Full text] [CrossRef] [Medline]
    4. Degenhardt L, Stockings E, Patton G, Hall WD, Lynskey M. The increasing global health priority of substance use in young people. Lancet Psychiatry 2016 Mar;3(3):251-264. [CrossRef] [Medline]
    5. Guerin N, White V. Cancer Council Victoria. 2020.   URL: https:/​/www.​health.gov.au/​sites/​default/​files/​documents/​2020/​07/​trends-in-substance-use-among-australian-secondary-school-students-1996-2017.​pdf [accessed 2020-10-01]
    6. Townsend L, Flisher AJ, King G. A systematic review of the relationship between high school dropout and substance use. Clin Child Fam Psychol Rev 2007 Dec;10(4):295-317. [CrossRef] [Medline]
    7. Kelly AB, Evans-Whipp TJ, Smith R, Chan GC, Toumbourou JW, Patton GC, et al. A longitudinal study of the association of adolescent polydrug use, alcohol use and high school non-completion. Addiction 2015 Apr;110(4):627-635 [FREE Full text] [CrossRef] [Medline]
    8. Zapolski TC, Clifton RL, Banks DE, Hershberger A, Aalsma M. Family and peer influences on substance attitudes and use among juvenile justice-involved youth. J Child Fam Stud 2019 Mar;28(2):447-456 [FREE Full text] [CrossRef] [Medline]
    9. Heradstveit O, Skogen JC, Hetland J, Hysing M. Alcohol and illicit drug use are important factors for school-related problems among adolescents. Front Psychol 2017 Jun 20;8:1023 [FREE Full text] [CrossRef] [Medline]
    10. Whitesell M, Bachand A, Peel J, Brown M. Familial, social, and individual factors contributing to risk for adolescent substance use. J Addict 2013;2013:579310 [FREE Full text] [CrossRef] [Medline]
    11. Valdebenito S, Ttofi M, Eisner M. Prevalence rates of drug use among school bullies and victims: a systematic review and meta-analysis of cross-sectional studies. Aggress Violent Behav 2015 Jul;23:137-146. [CrossRef]
    12. Alhyas L, Al Ozaibi N, Elarabi H, El-Kashef A, Wanigaratne S, Almarzouqi A, et al. Adolescents' perception of substance use and factors influencing its use: a qualitative study in Abu Dhabi. JRSM Open 2015 Mar;6(2):2054270414567167 [FREE Full text] [CrossRef] [Medline]
    13. Clare PJ, Dobbins T, Bruno R, Peacock A, Boland V, Yuen WS, et al. The overall effect of parental supply of alcohol across adolescence on alcohol-related harms in early adulthood-a prospective cohort study. Addiction 2020 Oct;115(10):1833-1843. [CrossRef] [Medline]
    14. Sharmin S, Kypri K, Khanam M, Wadolowski M, Bruno R, Mattick RP. Parental supply of alcohol in childhood and risky drinking in adolescence: systematic review and meta-analysis. Int J Environ Res Public Health 2017 Mar 09;14(3):287 [FREE Full text] [CrossRef] [Medline]
    15. Tael-Öeren M, Naughton F, Sutton S. The relationship between parental attitudes and children's alcohol use: a systematic review and meta-analysis. Addiction 2019 Sep;114(9):1527-1546. [CrossRef] [Medline]
    16. Faggiano F, Minozzi S, Versino E, Buscemi D. Universal school-based prevention for illicit drug use. Cochrane Database Syst Rev 2014;2014(12):CD003020 [FREE Full text] [CrossRef] [Medline]
    17. Champion KE, Newton NC, Barrett EL, Teesson M. A systematic review of school-based alcohol and other drug prevention programs facilitated by computers or the Internet. Drug Alcohol Rev 2013 Mar;32(2):115-123. [CrossRef] [Medline]
    18. Foxcroft DR, Tsertsvadze A. Cochrane Review: universal school-based prevention programs for alcohol misuse in young people. Evid Based Child Health 2012 Mar 01;7(2):450-575. [CrossRef]
    19. Hodder RK, Freund M, Wolfenden L, Bowman J, Nepal S, Dray J, et al. Systematic review of universal school-based 'resilience' interventions targeting adolescent tobacco, alcohol or illicit substance use: a meta-analysis. Prev Med 2017 Jul;100:248-268. [CrossRef] [Medline]
    20. Carver H, Elliott L, Kennedy C, Hanley J. Parent–child connectedness and communication in relation to alcohol, tobacco and drug use in adolescence: an integrative review of the literature. Drugs (Abingdon Engl) 2016 Sep 26;24(2):119-133. [CrossRef]
    21. Teesson M, Newton NC, Barrett EL. Australian school-based prevention programs for alcohol and other drugs: a systematic review. Drug Alcohol Rev 2012 Sep;31(6):731-736. [CrossRef] [Medline]
    22. Midford R, Cahill H, Lester L, Foxcroft DR, Ramsden R, Venning L. Smoking prevention for students: findings from a three-year program of integrated harm minimization school drug education. Subst Use Misuse 2016;51(3):395-407. [CrossRef] [Medline]
    23. Lee NK, Cameron J, Battams S, Roche A. What works in school-based alcohol education: a systematic review. Health Educ J 2016 Jan 28;75(7):780-798. [CrossRef]
    24. Cuijpers P. Three decades of drug prevention research. Drugs (Abingdon Engl) 2009 Jul 10;10(1):7-20. [CrossRef]
    25. Stapinski L, Lawler S, Newton N, Reda B, Chapman C, Teesson M. Empowering young people to make Positive Choices: evidence-based resources for the prevention of alcohol and other drug use in Australian schools. Learn Communities J 2017 Nov;21:152-167. [CrossRef]
    26. Ryan SM, Jorm AF, Lubman DI. Parenting factors associated with reduced adolescent alcohol use: a systematic review of longitudinal studies. Aust N Z J Psychiatry 2010 Sep;44(9):774-783. [CrossRef] [Medline]
    27. Yap MB, Jorm AF, Lubman DI. What are parents doing to reduce adolescent alcohol misuse? Evaluating concordance with parenting guidelines for adolescent alcohol use. BMC Public Health 2015 Mar 10;15:114 [FREE Full text] [CrossRef] [Medline]
    28. Gilligan C, Kypri K. Parent attitudes, family dynamics and adolescent drinking: qualitative study of the Australian Parenting Guidelines for Adolescent Alcohol Use. BMC Public Health 2012 Jun 29;12:491 [FREE Full text] [CrossRef] [Medline]
    29. Stapinski L, Reda B, Newton NC, Chapman C, Kay-Lambkin F, McBride N. Development and evaluation of the Positive Choices portal for evidence-based drug and alcohol prevention. Report prepared for: Drug Strategy Branch. Canberra, Australia: Australian Government Department of Health; 2015.
    30. Kuntsche S, Kuntsche E. Parent-based interventions for preventing or reducing adolescent substance use - a systematic literature review. Clin Psychol Rev 2016 Apr;45:89-101. [CrossRef] [Medline]
    31. Hurley E, Dietrich T, Rundle-Thiele S. A systematic review of parent based programs to prevent or reduce alcohol consumption in adolescents. BMC Public Health 2019 Nov 04;19(1):1451 [FREE Full text] [CrossRef] [Medline]
    32. Ryan SM, Jorm AF, Kelly CM, Hart LM, Morgan AJ, Lubman DI. Parenting strategies for reducing adolescent alcohol use: a Delphi consensus study. BMC Public Health 2011 Jan 06;11:13 [FREE Full text] [CrossRef] [Medline]
    33. Miller-Day M, Kam JA. More than just openness: developing and validating a measure of targeted parent-child communication about alcohol. Health Commun 2010 Jun;25(4):293-302 [FREE Full text] [CrossRef] [Medline]
    34. Choi HJ, Miller-Day M, Shin Y, Hecht ML, Pettigrew J, Krieger JL, et al. Parent prevention communication profiles and adolescent substance use: a latent profile analysis and growth curve model. J Fam Commun 2017;17:15-32 [FREE Full text] [CrossRef] [Medline]
    35. Positive Choices: Drug and Alcohol Information - Get informed, stay smart, stay safe. Positive Choices, Department of Health, Australian Government.   URL: https://positivechoices.org.au/ [accessed 2022-08-16]
    36. NHMRC levels of evidence and grades for recommendations for developers of guidelines. National Health and Medical Research Council. 2009 Dec.   URL: https://www.nhmrc.gov.au/sites/default/files/images/NHMRC%20Levels%20and%20Grades%20(2009).pdf [accessed 2022-06-01]
    37. Wagner NM. Assessing the Value of the Vaccine Social Media Intervention Through the Re-aim Framework Implementation Dimension. University of Colorado at Denver. 2019.   URL: https:/​/www.​proquest.com/​openview/​c1bbf3c206581e16720ea78af839a9d1/​1?pq-origsite=gscholar&cbl=18750&diss=y [accessed 2020-10-01]
    38. Wagner B, Cross D, Adams E, Symons M, Mazzucchelli TG, Watkins R, et al. RE-AIM evaluation of a teacher-delivered programme to improve the self-regulation of children attending Australian Aboriginal community primary schools. Emot Behav Difficult 2019 Oct 07;25(1):42-58. [CrossRef]
    39. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health 2013 Jun;103(6):e38-e46. [CrossRef] [Medline]
    40. Jenkinson KA, Naughton G, Benson AC. The GLAMA (Girls! Lead! Achieve! Mentor! Activate!) physical activity and peer leadership intervention pilot project: a process evaluation using the RE-AIM framework. BMC Public Health 2012 Jan 19;12:55 [FREE Full text] [CrossRef] [Medline]
    41. Lewis JR. The system usability scale: past, present, and future. Int J Human Comput Interact 2018 Mar 30;34(7):577-590. [CrossRef]
    42. Mol M, van Schaik A, Dozeman E, Ruwaard J, Vis C, Ebert DD, et al. Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis. BMC Psychiatry 2020 May 12;20(1):218 [FREE Full text] [CrossRef] [Medline]
    43. McLellan S, Muddimer A, Peres SC. The effect of experience on System Usability Scale ratings. J Usability Stud 2012 Feb;7(2):56-67.
    44. Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval 2006 Jun 1;27(2):237-246. [CrossRef]
    45. Wangberg S, Andreassen H, Kummervold P, Wynn R, Sørensen T. Use of the Internet for health purposes: trends in Norway 2000-2010. Scand J Caring Sci 2009 Dec;23(4):691-696. [CrossRef] [Medline]
    46. Kummervold PE, Wynn R. Health information accessed on the Internet: the development in 5 European countries. Int J Telemed Appl 2012;2012:297416 [FREE Full text] [CrossRef] [Medline]
    47. Jeong D, Cheng M, St-Jean M, Jalali A. Evaluation of eMentalHealth.ca, a Canadian mental health website portal: mixed methods assessment. JMIR Ment Health 2019 Sep 06;6(9):e13639 [FREE Full text] [CrossRef] [Medline]
    48. Young M, Richards C, Gunning M. Online mental health resources for teenagers: an evaluation of two websites developed for adolescents. Adv School Mental Health Promot 2012 Oct;5(4):277-289. [CrossRef]

    ‎
    AOD: alcohol and other drug
    NSW: New South Wales
    RE-AIM: reach, effectiveness, adoption, implementation, and maintenance
    SUS: System Usability Scale

    Edited by S Badawy; submitted 07.11.21; peer-reviewed by A Abubakar, L Kremer; comments to author 01.05.22; revised version received 26.06.22; accepted 16.07.22; published 26.08.22

    Copyright

    ©Lexine Ann Stapinski, Smriti Nepal, Tara Guckel, Lucinda Rachel Grummitt, Cath Chapman, Samantha Jane Lynch, Siobhan Maree Lawler, Maree Teesson, Nicola Clare Newton. Originally published in JMIR Pediatrics and Parenting (https://pediatrics.jmir.org), 26.08.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.