This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.
To prevent adolescents from initiating alcohol and other drug use and reduce the associated harms, effective strategies need to be implemented. Despite their availability, effective school-based programs and evidence-informed parental guidelines are not consistently implemented. The
Guided by the five dimensions of the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework, the study assessed the impact of the
Data were collected between 2017 and 2019, using web-based evaluation and community awareness surveys. Data from the surveys were merged to examine reach, effectiveness, adoption, implementation, and maintenance using descriptive statistics. Google Analytics was used to further understand the reach of the website. The System Usability Scale was used to measure website usability. In addition, inductive analysis was used to assess the participants’ feedback about
A total of 5 years after launching, the
The
Adolescence is marked by considerable emotional, social, and physical changes, including increasing autonomy from parents, greater influence from peers, and increased likelihood to engage in risk-taking behaviors [
Studies have identified a number of modifiable individual risk factors that are consistently associated with AOD use among adolescents [
Prevention programs targeting individual and parental risk factors are effective in reducing AOD use among adolescents [
Parenting strategies that are consistently associated with delayed initiation of alcohol use include parental monitoring, limited access to alcohol, parent-child relationship quality, parental involvement, and communication [
It is critical that effective AOD prevention strategies are implemented consistently and widely to alleviate the considerable burden associated with AOD use. Therefore, teachers, school staff, and parents need to have easy access to evidence-based information, strategies, and programs that equip them to respond most effectively. The
In this study, we extended this initial evaluation by conducting a more comprehensive examination of the effectiveness and impact of the
Data were collected between 2017 and 2019, from several sources to evaluate the dimensions aligned with the RE-AIM framework, as described in the following sections.
Reach was assessed via the measurement of access and awareness, using 2 data sources. Access was measured using site use analytics and operationalized as follows: How many unique users have accessed
Google Analytics was used to obtain a detailed analysis of website traffic between January 2017 and March 2021. This included information on the number of unique website users and page views, pages viewed per session, and average time users spent on each page.
A web-based survey was administered in July 2018 to assess the Australian community’s awareness and use of
To capture the effectiveness of
Participants were asked to spend time reading and interacting with the
For school staff, questions assessed whether they intended to (1) implement teaching resources that have been tested in schools and proven to prevent AOD use, (2) communicate with students about the risks and effects associated with AOD use, and (3) correct the misperception that AOD use is
Similar to the effectiveness dimension, the adoption and implementation dimensions were assessed via the web-based evaluation surveys. Adoption was assessed by identifying barriers to and enablers of access and uptake of evidence-based prevention strategies by school staff and parents. A single question assessed which of the following characteristics of web-based AOD prevention resources or information were most valued by school staff: (1) evidence-based information, (2) resources that had been tested in schools and proven to prevent AOD use, (3) engaging website, (4) interactive website, (5) website that is easy to navigate and use, and (6) simple and easy-to-understand language. Similarly, enablers for parents and guardians were identified through the same items with the addition of the following two items that pertained specifically to parents: (1) parental strategies that have been proven to be effective and (2) website with advice from other parents.
Implementation was evaluated using the System Usability Scale (SUS) [
This dimension was assessed using the website evaluation surveys, as described previously. For the participating school staff and parents, the question assessed whether users intended to access
To assess school staff and parents’ general feedback about the
The evaluation surveys were administered via
Data on community awareness were also collected via
Inductive analysis was used to assess participants’ feedback about
The 2017 survey was approved by the Human Research Ethics Committee, University of New South Wales (project number HC12548), and the 2019 survey was approved by the Human Research Ethics Committee, University of Sydney (project number 2018/873).
From 2017 to 2019, a total of 200 participants completed the evaluation surveys, of which 73 (36.5%) participants were school staff and 127 (63.5%) were parents.
Demographic characteristics of school staff and parents (evaluation survey).
Characteristics | School staff (n=73) | Parents (n=127) | |
Age (years), range | 24-63 | 26-63 | |
|
|||
|
Women | 56 (77) | 108 (85) |
|
Men | 16 (22) | 19 (14.9) |
|
Nonbinary | 1 (1) | 0 (0) |
|
|||
|
Australian Capital Territory | 5 (7) | 4 (3.1) |
|
New South Wales | 29 (40) | 45 (35.4) |
|
Queensland | 10 (14) | 23 (18.1) |
|
South Australia | 6 (8) | 6 (4.7) |
|
Tasmania | 4 (5) | 2 (1.6) |
|
Victoria | 16 (22) | 28 (22) |
|
Western Australia | 3 (4) | 15 (11.8) |
|
Northern Territory | 0 (0) | 4 (3.1) |
|
|||
|
Major city | 36 (49) | 78 (61.4) |
|
Regional | 33 (45) | 48 (37.8) |
|
Remote | 4 (5) | 1 (0.8) |
|
|||
|
Foundation | 5 (7) | 10 (7.9) |
|
Year 1 | 8 (11) | 6 (4.7) |
|
Year 2 | 8 (11) | 5 (3.9) |
|
Year 3 | 8 (11) | 10 (7.9) |
|
Year 4 | 7 (10) | 13 (10.2) |
|
Year 5 | 9 (12) | 11 (8.7) |
|
Year 6 | 8 (11) | 15 (11.8) |
|
Year 7 | 38 (52) | 26 (20.5) |
|
Year 8 | 37 (51) | 29 (22.8) |
|
Year 9 | 38 (52) | 23 (18.1) |
|
Year 10 | 40 (55) | 24 (18.9) |
|
Year 11 | 39 (53) | 35 (27.6) |
|
Year 12 | 39 (53) | 33 (25.9) |
|
N/Ab | 18 (25) | N/A |
|
|||
|
Coeducational | 63 (86) | N/A |
|
Single sex | 10 (14) | N/A |
|
|||
|
Teacher | 48 (66) | N/A |
|
School counselor chaplain | 10 (14) | N/A |
|
Youth worker | 2 (3) | N/A |
|
Researcher | 2 (3) | N/A |
|
Other | 11 (15) | N/A |
|
|||
|
Full time | N/A | 52 (40.9) |
|
Part time or casual | N/A | 42 (33.1) |
|
Home duties (including carer) | N/A | 19 (14.9) |
|
Unemployed | N/A | 6 (4.7) |
|
Unable to work | N/A | 2 (1.6) |
|
Student | N/A | 5 (3.9) |
|
Other | N/A | 1 (0.8) |
aRespondents were able to select multiple responses; thus, the column total for this item does not add to 100%.
bN/A: not applicable.
A total of 1435 participants completed the community awareness surveys across baseline, 6-month follow-up, and 12-month follow-up. At baseline, participants were 48.55% (201/414) school staff and 51.45% (213/414) parents. At the 6-month follow-up, participants were 51.45% (249/484) school staff and 48.55% (235/484) parents. Finally, at the 12-month follow-up, participants were 50.09% (269/537) school staff and 49.91 (268/537) parents.
All school staff who completed the community awareness surveys were high school staff (719/719, 100%), most resided in NSW (259/719, 36%), and were based in major cities (381/719, 52.9%;
Demographic characteristics of school staff and parents (community awareness survey).
Characteristics | School staff (n=719), n (%) | Parents (n=716), n (%) | |
|
|||
|
Australian Capital Territory | 5 (0.7) | 15 (2.1) |
|
New South Wales | 259 (36) | 207 (28.9) |
|
Queensland | 147 (20.4) | 136 (18.9) |
|
South Australia | 46 (6.4) | 75 (10.5) |
|
Tasmania | 32 (4.5) | 19 (2.7) |
|
Victoria | 134 (18.6) | 168 (23.5) |
|
Western Australia | 85 (11.8) | 88 (12.3) |
|
Northern Territory | 11 (1.5) | 8 (1.1) |
|
|||
|
Major city | 381 (53) | 379 (52.9) |
|
Regional | 185 (25.7) | 192 (26.8) |
|
Rural | 93 (12.9) | 93 (12.9) |
|
Remote | 34 (4.7) | 29 (4.1) |
|
Very remote | 26 (3.6) | 23 (3.2) |
|
|||
|
Yes | 568 (78.9) | 21 (2.9) |
|
No | 122 (16.9) | 687 (95.9) |
|
Prefer not to answer | 29 (4) | 8 (1.1) |
|
|||
|
Public | 432 (60.1) | N/Aa |
|
Faith-based | 145 (20.2) | N/A |
|
Independent | 95 (13.2) | N/A |
|
Coeducational | 21 (2.9) | N/A |
|
Single sex (female) | 12 (1.7) | N/A |
|
Single sex (male) | 9 (1.3) | N/A |
|
Selective | 5 (0.7) | N/A |
|
|||
|
Single-parent household | N/A | 192 (26.8) |
|
2-parent household | N/A | 500 (69.8) |
|
1 child | N/A | 69 (9.6) |
|
2-4 children | N/A | 397 (55.4) |
|
≥5 children | N/A | 39 (5.4) |
|
Step siblings (cohabiting) | N/A | 27 (3.8) |
|
Prefer not to answer | N/A | 2 (0.3) |
aN/A: not applicable.
bRespondents were able to select multiple responses; thus, the column total for this item does not add to 100%.
All parents who completed the community awareness surveys (716/716, 100%) were parents of high school students. Most of them resided in NSW (207/716, 28.9%), were based in major cities (379/716, 52.9%), and did not identify as Aboriginal and/or Torres Strait Islander (687/716, 95.9%;
Between January 1, 2017, and March 31, 2021,
Monthly Positive Choices website visitors, between January 2017 and March 2021.
The spike in website visitors seen between July 2018 and November 2018 (
According to the 12-month survey, of the school staff who were not using
Effectiveness in changing intentions to use evidence-based strategies (school staff).
Respondents who are currently implementing evidence-based strategy (n=73), n (%) | Among respondents who were not implementing evidence-based strategies, those who intend to after viewing |
Respondents who will not implement evidence-based strategies in the future, n (%) | |
Implement teaching resources that were tested in schools and proven to prevent alcohol and drug use | 35 (48) | 34 (89)a | 4 (11)a |
Communicate with students about the risks and effects of alcohol and drug use | 53 (73) | 18 (90)b | 2 (10)b |
Correct the misperception that alcohol and other drug use is common or “the norm” | 52 (71) | 17 (81)c | 4 (19)c |
aSample size, n=38.
bSample size, n=20.
cSample size, n=21.
Effectiveness in changing intentions (parents).
Respondents who are currently implementing evidence-based strategy (n=127), n (%) | Among respondents who were not implementing evidence-based strategies, those who intend to after viewing |
Respondents who will not implement evidence-based strategies in the future, n (%) | |
Encourage open communication with my child about alcohol and other drugs | 100 (78.7) | 23 (85)a | 4 (15)a |
Have an explicit conversation with my child about alcohol and other drugs | 88 (69.3) | 36 (92)b | 3 (8)b |
Correct the misperception that alcohol and other drug use is common or “the norm” | 81 (63.8) | 38 (83)c | 8 (17)c |
Clearly communicate my expectations about drug and alcohol use to my child | 95 (74.8) | 25 (78)d | 7 (22)d |
Change my own drug or alcohol use to model appropriate behavior | 87 (68.5) | 18 (45)e | 22 (55)e |
Avoid supplying my child with alcohol | 103 (81.1) | 9 (38)f | 15 (63)f |
Monitor my child’s whereabouts more closely | 54 (42.5) | 63 (86)g | 10 (14)g |
aSample size, n=27.
bSample size, n=39.
cSample size, n=46.
dSample size, n=32.
eSample size, n=40.
fSample size, n=24.
gSample size, n=73.
School staff reported that they spent between 5 and 10 hours per semester on AOD education. Most school staff rated the following factors highly (either “very important” or “important”) when selecting web-based AOD prevention resources: evidence-based information (69/73, 95%), easy-to-navigate and easy-to-use website (69/73, 95%), engaging website (70/73, 96%), simple and easy-to-use language (68/73, 93%), and resources tested in school and found to be effective (62/73, 85%). Although school staff also valued interactive features of prevention websites (55/73, 75%), the proportion of participants who rated this factor highly was lower than those for the other factors facilitating effective AOD prevention. When discussing AOD topics, school staff displayed high levels of confidence, with 41% (30/73) feeling “very confident,” 38% (28/73) feeling “confident,” and 16% (12/73) feeling “somewhat confident”; only 4% (3/73) reported feeling “not very confident.”
Regarding usability of the
Most parents rated the following factors highly (either “very important” or “important”) when selecting web-based AOD prevention resources: evidence-based information (119/127, 93.7%); strategies that were tested and proven to be effective in AOD use prevention (119/127, 93.7%); simple and easy-to-use language (109/127, 85.8%); and engaging (115/127, 90.6%), interactive (72/127, 56.7%), and easy-to-navigate and easy-to-use (123/127, 96.9%) website. In addition, 57.5% (73/127) of the parents valued prevention advice from other parents.
Parents also reported high confidence in discussing AOD topics: 48.8% (62/127) were “very confident,” 31.5% (40/127) were “confident,” 17.3% (22/127) felt “somewhat confident,” and 2.4% (3/127) felt “not very confident.” These ratings suggest that confidence was not a significant barrier to evidence-based prevention in this sample. This is in contrast to the general community sample, from the community awareness survey, where lack of confidence was reported as the greatest barrier for parents to having conversations with their children about AOD use.
The mean SUS score for the parent group was 74 (SD 12.9; range 40-100), indicating
Most school staff reported that they would use
Most parents also reported that they would use
The main themes that emerged from the analysis of users’ feedback on the website centered around the website’s content and features and promotion and increasing the usability and diversity of the website. Most responses from participants were suggestions to improve the
Adolescence is a time of increased susceptibility to engaging in risk-taking behaviors such as AOD use. Prevention strategies designed to target modifiable risk factors have been demonstrated to be effective in reducing AOD use and related harms among adolescents [
A total of 5 years following its launch, this Australian AOD prevention website has reached >1.7 million users, and the page views have continued to grow. The user reach of
Although evidence shows that effective AOD prevention strategies are not commonly implemented by schools [
The widespread use of web-based health promotion tools and websites by school staff and parents to access information and resources highlights the need for quality control measures. This will ensure that they are evidence-based, up to date, and engaging and use simple language. In addition, the websites themselves should be accessible and easy to use and navigate. Findings from robust evaluations will allow critical assessment of the benefits of such tools and websites, inform content, and inform website updates and developments to optimize their usability. Although there have been evaluations of mental health information websites [
A limitation of the study is that the findings rely on users’ self-reported intentions to implement evidence-based strategies, rather than actual assessment of their subsequent behavior. Furthermore, the study design assessed participants’ feedback and behavioral intentions after they interacted with the website for a relatively short period. Future studies with a pre-post study design will enable a more comprehensive evaluation, including assessment of whether the website affected subsequent behaviors and implementation of evidence-based strategies by school staff and parents. Another limitation is that the targeted campaign used to recruit participants for the evaluation study may have resulted in a sample selection bias. Most of the surveyed school staff and parents in the current sample reported that they were already implementing evidence-based prevention strategies and were confident about discussing alcohol and drug use. These results contrast with previous evidence suggesting low confidence and implementation of evidence-based AOD prevention strategies among parents and in schools [
The findings from the evaluation of
Themes and subthemes from participants’ feedback on the website.
alcohol and other drug
New South Wales
reach, effectiveness, adoption, implementation, and maintenance
System Usability Scale
Positive Choices is supported by funding from the Australian Government Department of Health. This study was also supported by the Australian National Health and Medical Research Council via fellowships (NCN: GNT1166377, LAS: GNT1132853, and MT: GNT1078407 and GNT1195284) and the Centre of Research Excellence in the Prevention and Early Intervention in Mental Illness and Substance Use (GNT11349009). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. Positive Choices was developed by researchers at the Matilda Centre for Research in Mental Health and Substance Use at the University of Sydney (LAS, SN, TG, LRG, CC, SJL, SML, MT, and NCN). The authors receive no financial or other benefits from the Positive Choices website.
None declared.