Evaluating Digital Program Support for the Physical Activity 4 Everyone (PA4E1) School Program: Mixed Methods Study

Background Effectively scaled-up physical activity interventions are urgently needed to address the high prevalence of physical inactivity. To facilitate scale-up of an efficacious school-based physical activity program (Physical Activity 4 Everyone [PA4E1]), provision of implementation support to physical education (PE) teachers was adapted from face-to-face and paper-based delivery modes to partial delivery via a website. A lack of engagement (usage and subjective experience) with digital delivery modes, including websites, may in part explain the typical reduction in effectiveness of scaled-up interventions that use digital delivery modes. A process evaluation focused on the PA4E1 website was undertaken. Objective The 2 objectives were to (1) describe the usage of the PA4E1 program website by in-school champions (PE teachers leading the program within their schools) and PE teachers using quantitative methods; (2) examine the usage, subjective experience, and usability of the PA4E1 program website from the perspective of in-school champions using mixed methods. Methods The first objective used website usage data collected across all users (n=273) throughout the 9 school terms of the PA4E1 implementation support. The 4 usage measures were sessions, page views, average session duration, and downloads. Descriptive statistics were calculated and explored across the duration of the 26-month program. The second objective used mixed methods, triangulating data from the first objective with data from a think-aloud survey and usability test completed by in-school champions (n=13) at 12 months. Qualitative data were analyzed thematically alongside descriptive statistics from the quantitative data in a triangulation matrix, generating cross-cutting themes using the “following a thread” approach. Results For the first objective, in-school champions averaged 48.0 sessions per user, PE teachers 5.8 sessions. PE teacher sessions were of longer duration (10.5 vs 7.6 minutes) and included more page views (5.4 vs 3.4). The results from the mixed methods analysis for the second objective found 9 themes and 2 meta-themes. The first meta-theme indicated that the website was an acceptable and appropriate delivery mode, and usability of the website was high. The second meta-theme found that the website content was acceptable and appropriate, and identified specific suggestions for improvement. Conclusions Digital health interventions targeting physical activity often experience issues of lack of user engagement. By contrast, the findings from both the quantitative and mixed methods analyses indicate high usage and overall acceptability and appropriateness of the PA4E1 website to school teachers. The findings support the value of the website within a multidelivery mode implementation intervention to support schools to implement physical activity promoting practices. The analysis identified suggested intervention refinements, which may be adopted for future iterations and further scale-up of the PA4E1 program. Trial Registration Australian New Zealand Clinical Trials Registry ACTRN12617000681358; https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=372870


Background
As much as 1 in 4 (25%) adults and 4 in 5 (80%) adolescents do not meet the global recommendations for aerobic exercise and are therefore at increased risk of noncommunicable diseases and premature mortality [1,2]. Scalable programs with proven effectiveness are urgently needed to increase population physical activity, including those in the school setting, as outlined in the Global Action Plan on Physical Activity 2018-2030 [3].
Many efficacious health interventions exist [4], but scaling-up efficacious interventions with proven health benefits is important to ensure the population health benefit of such interventions can be realized [5]. Digital delivery modes are often used to facilitate scale-up of efficacious interventions, as they have the potential to achieve considerable reach at relatively low cost compared with traditional modes of program delivery [4]. A recent systematic review of scaled-up trials of obesity-prevention interventions found that the main adaptations to interventions to facilitate their scale-up were changes in the modes of delivery [4]. Specifically, websites or other digital delivery modes were commonly added or replaced face-to-face components, for example, to provide program resources and training for teachers or clinicians in program delivery [4]. Research examining digital delivery modes such as websites, however, often report poor usage and engagement with such technologies [6][7][8].
A lack of engagement with digital technologies [4] may, in part, explain the modest impact of physical activity interventions that have relied on such technologies to support population-wide scale-up [4,6,9]. A recent meta-analysis of digital health interventions targeting physical activity found that higher usage engagement is associated with targeted behavior changes [10]. Perski et al [9] have conceptualized engagement with digital health interventions to include both amount, duration and depth of usage, and user subjective experience, characterized by attention, interest, and affect [9]. Systematic reviews of the human-computer interaction literature suggest that good usability (ie, functionality and efficiency of the digital application) and subjective experience engagement (ie, users attention, interest, and affect) are important drivers of reducing attrition and increasing usage of digital applications [9,11].
Process evaluations are recommended as part of comprehensive evaluations of all randomized trials to aid the interpretation of trial findings and to better understand and explain why an intervention has or has not worked [12]. Process evaluations may be particularly important as part of trials of interventions that have been scaled-up, given the limited research and insight scientists currently have regarding scale-up processes. However, detailed process evaluations are seldom undertaken. For example, a recent systematic review of school-based physical activity interventions found that just 4 of the 17 trials included in the review had conducted a process evaluation [13]. Further, of those undertaken in the secondary school setting, few have used mixed methods [14][15][16][17] and many have focused on a narrow range of quantitative process outcomes (eg, fidelity, reach, dose) [18][19][20][21]. Perhaps most importantly for those interested in the application of digital technologies to support scale-up, none of the trials identified in the review [4], or in reviews of the scale-up literature generally, have undertaken a process evaluation examining the key dimensions of subjective experience and usage of digital components employed to deliver implementation support strategies.
Given the lack of process evaluations focused on the digital delivery mode of interventions utilizing multiple delivery modes, this study sought to address this evidence gap by conducting a process evaluation of the digital delivery mode used to support scale-up of an effective health intervention. This study addresses the scale-up of Physical Activity 4 Everyone (PA4E1), a secondary school-based physical activity program that included a website to support the delivery of implementation support strategies to school teachers (physical education [PE] teachers and in-school champions [PE teachers leading the program within their schools]). The objectives of this study are below, followed by an overview of the PA4E1 program and evaluation trials, which provide more detailed context for this study. 1. To describe the usage of the PA4E1 program website by in-school champions and PE teachers using quantitative methods. 2. Examine the usage, subjective experience, and usability of the PA4E1 program website from the perspective of in-school champions using mixed methods.

Physical Activity 4 Everyone (PA4E1)
School practices, programs, and policies can support adolescents to be physically active and are recommended by the World Health Organization and governments internationally through whole-school approaches [3,[22][23][24]. One school program that has been shown to assist schools (PE teachers, principals) to support their students to become more physically active is PA4E1 [25][26][27].
The PA4E1 program consisted of 7 physical activity practices, and 6 implementation support strategies.

PA4E1 Postscale (This Study): Type III Hybrid Implementation-Effectiveness Trial
The program was scaled-up for delivery in more schools across a larger geographic area, utilizing a website optimized for desktop, mobile, and tablet devices, to support the delivery of implementation support strategies to schools. The PA4E1 scale-up trial was a type III hybrid implementation-effectiveness cluster randomized controlled trial [29]. The trial involved 49 schools, 24 allocated to the program (intervention) group. An outline of the logic for this trial is shown in Figure 1. Program schools were offered 7 implementation support strategies to support their adoption of 7 physical activity practices, supporting school students to become more physically active. Details of the physical activity practices and implementation support strategies (as well as additional information on the timing of the implementation support strategies) are provided in Multimedia Appendix 1. Adaptations for scale-up were made to both the physical activity practices and the implementation support strategies, as detailed elsewhere [29]. The main change relevant to this study was the introduction of a digital delivery mode, a website, for the provision of program implementation support. Multimedia Appendix 1 (Table S1) shows how the  website was used within the 7 implementation support strategies  (n=23 substrategies). To summarize, a password-protected program website replaced face-to-face and paper-based delivery modes to part-deliver teacher professional learning to all PE teachers in participating schools; provide program resources for in-school champions and PE teachers; prompt in-school champions and PE teachers to implement PA4E1; monitor schools' performance on meeting practice implementation milestones in each school term via an in-school champion-completed termly survey; and provide feedback to school stakeholders (in-school champions, principals) based on their termly survey results [25,29].
The primary trial outcome was uptake of physical activity practices by schools at study midpoint (12 months) and 24 months. At 12 months the trial's primary outcome (proportion of schools adopting at least four of the seven physical activity practices) was significant, with more schools implementing 4 of the 7 practices in the program group (16/24, 67%) than in the control group (1/25, 4%; P<.001) [32]. Further process evaluation outcomes will be reported elsewhere, in line with the process evaluation protocol [28].

Methods
The methods are reported by objective.

Sampling
Usage data were tracked for the entire study period (October 2017 to December 2019; 9 school terms) for all users of the PA4E1 website who had a registered account with the program website, including in-school champions, PE teachers, principals, school administration staff, and nonschool staff (support officers and the PA4E1 research team). At the start of the program, all users were provided a registration link via email to register for the website. During the program, new school staff were provided a link to register for the website.

Data Collection
Usage data were collected via Google Analytics throughout the 9 terms of the program (26 months, October 9, 2017, to December 20, 2019). Prior to March 29, 2018 (term 3), data could not be analyzed by website user type; only overall usage could be captured. In order to further understand the different types of users, a "custom view" was applied in Google Analytics from this date to prospectively segregate different school user types (ie, in-school champions, PE teachers).

Measures
The post hoc analysis method described by the Analyzing and Measuring Usage and Engagement Data (AMUsED) framework [33] was used to guide the selection of usage variables to report (Multimedia Appendix 2). Google Analytics was used to track 4 variables for the entire study period (Textbox 1).

Sessions
Similar to a "login," sessions are defined as a group of user interactions (eg, page views, downloads) that occur within a single period, specifically a session ends after 30 minutes of inactivity or if users leave and revisit the website. Sessions were chosen as a better measure than "logins" for the PA4E1 website as users remained logged in for a rolling period of 30 days, whereby users were only logged out following 30 days of inactivity.

Page Views
Page view is a count of total visits to each page of the website. If a user clicks reload after reaching the page, this was counted as an additional page view. If a user navigated to a different page and then returned to the original page, a second page view was recorded.

Average Session Duration
The length of time of a session from the first click to the last, excluding the inactive period of a session immediately following the last click.

Downloads
A count of the total number of downloads of resources from the resources page of the website. Users had to click onto the particular resource to be counted as a download.

Data Analysis
Data were first downloaded from Google Analytics for the entire study period. Descriptive statistics were produced in SAS software [34] for all users for the entire study period and for each type of user from March 29, 2018. Mean and SD were calculated for each measure.

Sampling
All in-school champions (n=24) were invited to complete a "think-aloud" survey and usability test via email after 4 school terms of website use (November 2018) [35]. Participants could complete the survey anytime through until May 2019 (Term 6). All in-school champions were emailed a study information letter and informed that completion of the survey acted as consent to the study. Completion of the survey and usability test was expected to take 25-30 minutes based on piloting. Participants were provided with an AUD $30 (US $23) e-gift card reimbursement for completing the survey.

Data Collection
The think-aloud survey and usability test was conducted remotely via in-school champions' own digital devices (ie, in-school champions' own mobile, laptop, or computers). Loop11 user-testing software was used to conduct the think-aloud survey and usability test [36]. As Figure 2 shows, Loop11 displays a set of questions and tasks imposed upon the website [36]. Throughout the think-aloud survey and usability test, participants were prompted by the Loop 11 platform to verbalize their thoughts while responding online to a series of questions (survey) and tasks (usability test) (Multimedia Appendix 3). The researchers were not present during the study, and participants were invited to complete the study in their own time using a link provided to them via email. This method was chosen to increase the real-world relevance of the findings, as the participants complete the activities in their own setting [9].
As well as the quantitative responses from the survey questions and usability tasks, a video screen capture and microphone audio data were also collected concurrently, and participants were frequently encouraged to explain their responses by "thinking aloud." A practice question was designed to give participants the opportunity to familiarize themselves with the format of the Loop11 user testing platform and to practice thinking aloud.  Figure 2 shows a screenshot of the think-aloud survey and usability test which is imposed upon the program website. The survey contained 4 validated tools [37][38][39]. All tools were on a 5-point Likert-scale (1=strongly disagree, 2=disagree, 3=neutral, 4=agree, and 5=strongly agree). The Systems Usability Scale is a 10-item scale to assess usability of the website (scores range 0-100) [37] and comparable data exist to interpret the score derived from the Systems Usability Scale [40]. The second and third tools were the Acceptability of Intervention [39] and Intervention Appropriateness Measure [39], to assess the acceptability and appropriateness of the website, respectively (scores range 0-5). The fourth tool was the long form of the User Engagement Scale [38], which is a 31-item tool split into 4 subscales (Aesthetic appeal; Focused attention; Perceived usability; and Reward; scores range 0-5 for each subscale and overall).

Measures
The usability test contained 3 tasks. The average time to task completion and success rate of task completion were recorded within Loop11. Tasks asked participants to navigate to certain sections of the website. The first task asked participants to navigate to the discussion forum, the second to find information on forming community links (see Practice 7 in Table S2 of Multimedia Appendix 1) and the final task asked participants where they would post a lesson observation (see Practice 1 in Table S2 of Multimedia Appendix 1). Participants were encouraged to verbalize their thoughts (think aloud) throughout the 3 tasks.
The survey also contained 12 prompting questions which asked participants to think aloud to verbalize their responses (see Multimedia Appendix 3 for a full list of these questions). The questions asked participants to explain their thoughts toward a particular hypothetical scenario or comment on a particular aspect of the website (eg, usefulness of the website if PA4E1 were to be rolled out state-wide). The questions were used to elicit additional responses from participants. Concurrently, audio data and a video of the participants screen (screen capture) were recorded.
Additional administration data were used to identify usage of other delivery modes for implementation support strategies ( Figure 1), including attendance at face-to-face events and contact logs between in-school champions and support officers (data sources have been described in the process evaluation protocol [28]).

Mixed Method Data Analysis
A mixed methods data analysis was informed by Perski et al's [9] conceptualization of digital behavior change intervention engagement framework, whereby delivery and content were considered separately. Delivery focuses on themes that emerge relating to the aesthetics/design, challenge, complexity, control features, credibility features, ease of use, familiarity, guidance, interactivity, message tone, mode of delivery, novelty, narrative, personalization, and professional support features of the digital intervention. Content focuses on themes that emerge relating to the behavior change techniques (eg, feedback, goal setting, reminders, rewards, self-monitoring, social support features). Themes relating to delivery and content were extracted separately.
All quantitative data were downloaded and analyzed in MS Excel. The quantitative data from the validated tools were scored according to their instructions [37][38][39] and summary descriptive statistics were produced (mean score [SD]). Task completions were calculated as percentage of successful completions compared with total attempts. Time taken to complete tasks were recorded from the initial click to the final successful click by the participant. Mean time and SD were calculated across all in-school champions for each task.
Audio data were transcribed verbatim and analyzed using inductive thematic analysis [41]. Screen recording data were also downloaded (video) and analyzed concurrently with the audio data, within Nvivo 12 (QSR International) [42], to provide additional context to the audio data (eg, a participant's cursor location within website).
The quantitative and qualitative data were mixed during analysis using a triangulation matrix. Both data sets were given equal emphasis to address the objective. The descriptive statistics were produced at the same time as the initial codes of the qualitative data. As per the process evaluation protocol [28], the "following the thread" approach was employed to generate hypothesis and questions of the qualitative data from the quantitative data, and vice versa. The findings were combined and compared (triangulated) using a triangulation matrix to assess where findings from one method agreed or partially agreed (convergence), appear to contradict each other (discrepancy or dissonance), or are silent (ie, a theme arises in 1 data set but not in another). The matrix allowed the production of themes and meta-themes (higher-level themes) that combined and compared the qualitative and quantitative data sets [41]. Throughout the entire analysis, MM kept a research journal.
The iterative phases of the mixed method analysis [41] included: • Familiarizing with the data (both audio and screen capture as well as quantitative data).
• Generating initial codes and following the thread (MM produced descriptive statistics, followed a thread, and developed initial codes of the transcriptions using Nvivo 12 [42]).
• Searching for themes (MM reviewed each code and ordered them under headings, before discussing the ordered codes with JD to produce an initial triangulation matrix).
• Reviewing themes (EC, TM, and RS reviewed the themes, participant quotes, and matrix labels-resolving disagreements through discussion with MM and JD).
• Defining and naming themes (all authors discussed and agreed upon final themes).
• Producing the report.

Ethical Approval
The trial was prospectively registered (ACTRN12617000681358) and approved by the Hunter New

Availability of Data and Materials
The full data set supporting the conclusions of this article containing data not already included within the article or its additional files is available from the corresponding author on reasonable request.

Results
The results are reported below by objective.

Objective 1: Describe the Usage of the PA4E1 Program Website by In-School Champions and PE Teachers Using Quantitative Methods
There were a total of 273 users of the PA4E1 website during the whole study period. School users of the website were in-school champions (n=30) and PE teachers (n=198). Few principals (n=7) or school administration staff (n=2) registered for the program. Additionally, there were also nonschool users (n=20; ie, support officers and the PA4E1 research team) and unidentified users (n=16) of the program website. Table 1 provides a breakdown of website usage for in-school champions and PE teachers from March 28, 2018 (term 3) to December 20, 2019 (term 9) (when the data were available by user) and for all program users from October 9, 2017, to December 20, 2019 (entire study period). In-school champions were the most frequent users of the program, with a mean of 48.0 sessions per user compared with 5.8 sessions per user by PE teachers. They also had higher total page views (276.6) and downloads (33.6) per user than PE teachers. PE teachers had fewer sessions (5.8), but these were of longer duration (10.5 vs 7.6 minutes) and included more page views on average (5.4 vs 3.4).  The most frequently viewed pages by in-school champions were the home page (18.4% of all views), surveys and progress reports (15%), professional learning (10.3%), and resources (7.3%). PE teachers most frequently viewed professional learning (19.7% of all views) and the home page (9.6%).
A total of 90 different resources were downloaded from the website at least once by either an in-school champion or a PE teacher, equating to a total of 1007 downloads. The top 20 most downloaded resources accounted for more than half of all total downloads (n=559). Among the top downloaded resources were those directly assisting a particular implementation support strategy ( Figure 1; see Table S1 in Multimedia Appendix 1) or physical activity practice (Figure 1; see Table S1 in Multimedia Appendix 1), which included a template for lesson observation (Practice 1), a student physical activity plan template (Practice 2), newsletter snippets (Practice 6), partnership agreement (Practice 7), and a school physical activity policy template (Practice 5).
The usage over time was explored by looking at the termly number of sessions, page views per session, average session duration, and number of downloads. Figure 3 shows these data per month for both in-school champions and PE teachers.

Objective 2: Examine the Usage, Subjective Experience, and Usability of the PA4E1 Program Website From the Perspective of In-School Champions Using Mixed Methods
Of the 24 in-school champions invited to complete the think-aloud survey and usability test, 13 participated (54%). The average time to complete the survey and usability test was 20 minutes. In-school champions (9 females and 4 males) represented schools located in major cities (n=6), inner regional (n=6), and outer regional (n=1) regions [43]. Seven of the in-school champions also had responsibility for leading the health and PE programs in their school. All schools (9 government and 4 Catholic) were in the lower 50% of the Socio-Economic Indexes for Areas (SEIFA) of Relative Socioeconomic Disadvantage (suburb in lower 50% of NSW) [44]. School enrollments ranged from 348 to 1316 (mean 900 [SD 246]). Six schools had 10% or more Indigenous student enrollment [45]. Table 2 summarizes the meta-themes and themes for delivery and content, respectively. Multimedia Appendix 4 includes the full triangulation matrix table and convergence labels. Table 2. Summary of delivery and content meta-themes and themes.

Delivery
The website was an acceptable and appropriate delivery mode. Usability of the website was high.

Meta-theme
The website complemented other delivery modes, but in-school champions preferred the other delivery modes (support officer and face-to-face).

Theme 1
The website was used as a "utility" delivery mode (ie, used only when required, not as the first delivery mode of choice). Theme 2 If the program were delivered statewide, the website would be useful, though embedding within other systems that teachers already used may be helpful.

Theme 3
Mixed reactions toward the possible addition of a website chat feature with a support officer available to video call during business hours.

Theme 4
The discussion forum was not used, but in-school champions reported that it could be potentially useful, perhaps if delivered differently.

Content
The website content was generally acceptable and appropriate, with a few specific suggestions for improvement.

Meta-theme
A lack of notifications (or prompts) to highlight new things in the website reduced return traffic to the website. Theme 1 The lesson observation form (a resource for Practice 1) was difficult to find on the website, it was difficult to track lesson observation completion, and some usability issues with completing out of internet range.

Theme 2
Termly surveys (Support Strategy 7) were generally acceptable and completed. Theme 3 The resources were acceptable and downloaded frequently. Theme 4

Meta-Theme: The Website Was an Acceptable and Appropriate Delivery Mode. Usability of the Website Was High (Triangulation Convergence Label: Agree)
Quantitative data indicated in-school champions agreed or strongly agreed that the PA4E1 website was acceptable and appropriate. For in-school champions (n=13), the mean acceptability of the intervention measure was 4.52 (SD 0.04) out of 5, and the mean Intervention Appropriateness Measure was 4.46 (SD 0.0) out of 5, both indicating agreement that the website is acceptable and appropriate. Additionally, the mean in-school champions' (n=13) overall User Engagement Scale score was 3.4 (SD 0.7) out of 5 [32]. The mean individual dimension scores from the User Engagement Scale were 4.2 (SD 0.1), 3.7 (SD 0.5), 3.4 (SD 0.3), and 2.4 (SD 0.3) for aesthetic appeal, reward factor, perceived usability, and focused attention, respectively.
In-school champions found the usability of the website to be "good." The overall in-school champion (n=13) Systems Usability Scale score was 77.7, which corresponds to a "good" website within the "acceptable" range of the Systems Usability Scale [31,37]. Additionally, the majority of the 3 navigation tasks were completed successfully by in-school champions (74% [29/39] of all tasks completed successfully). The average time to task completion was 71 seconds per task. The discussion forum and community links tasks had higher success rates (11/13 [85%] and 12/13 [92%], respectively) than the lesson observation task (6/13, 46%). Time to complete each task was 91, 66, and 55 seconds for discussion forum, community links, and lesson observation tasks, respectively. No tasks were abandoned.
Qualitative data also indicated good acceptability, appropriateness, and usability of the website delivery mode. Some example quotes are as follows: ...it's set out very clearly, easy to use, easy to access, and is updated relatively frequently.
So, we all know that teachers are the time poor people of the world. So definitely having an online mode is very, very suitable.

Theme 1: The Website Complemented Other Delivery Modes, But In-School Champions Preferred the Other Delivery Modes (Support Officer and Face-to-Face; Triangulation Convergence Label: Agree)
Quantitative data showed schools utilized the website ( Figure  1). In-school champions also relied on the other delivery modes, for example, contact logs between support officers and in-school champions indicated that 18/24 schools received a face-to-face contact with their support officer at least once a term in the first 12 months. In the second 12 months, this dropped to 8/24 schools, though the criteria for this implementation support strategy were different for the second 12 months (Multimedia Appendix 1). However, weekly emails or phone calls between support officers and in-school champions occurred in 16/24 schools during the first 12 months, but this increased to 22/24 schools in the second 12 months. Additionally, the 2 face-to-face workshops were well attended; 23/24 in-school champions attended the first workshop at the start of the program while 22/24 attended the second workshop at the midpoint of the program.
Qualitative data indicated that in-school champions used the website to complement interactions with their support officer. The website often was not the first point of call, and it may not be able to fully replace face-to-face and direct contact methods from the support officers. The website may be particularly useful for after-hours support (eg, on weekends and evenings).

Theme 2: The Website Was Used as a "Utility" Delivery Mode (ie, Used Only When Required, Not as the First Delivery Mode of Choice; Triangulation Convergence Label: Agree)
Quantitative data from objective 1 indicated that in-school champions used the website on average every fortnight, corresponding to 6.3 sessions per term (SD 2.0). Qualitative data identified that the website was perhaps not used to its full potential, but instead it was used for its core functions (eg, termly surveys, professional learning), rather than to peruse additional ideas and resources. Example quotes:

Theme 3: If the Program Were Delivered Statewide, the Website Would Be Useful, Though Embedding Within Other Systems That Teachers Already Used May Be Helpful (Triangulation Convergence Label: Slightly Agree)
All in-school champions (13/13) agreed that online delivery of the PA4E1 program should remain. The majority of in-school champions (12/13) suggested the PA4E1 website would be a suitable mode of delivery in its current form should the PA4E1 program be rolled out to schools statewide.
There were conflicting suggestions emerging from the qualitative data. In-school champions indicated that while a website would be useful for statewide delivery of the program, it could become another platform that teachers are asked to use, so it would be better embedding any roll out within existing platforms. Example quotes: ...it is another platform that we need to access in order to implement our teaching. I do think it is quite suitable, to roll out in its current form if there was to be no changes.
The more people that are on it [the website] as well, I think the more discussion that would kind of, be generated....

Theme 4: Mixed Reactions Toward the Possible Addition of a Website Chat Feature With a Support Officer Available to Video Call During Business Hours (Triangulation Convergence Label: Slightly Agree)
Quantitative data were generally supportive; the majority of in-school champions (11/13) liked the idea that support officers would be available during working hours for support within the PA4E1 website via video, audio, or text chat features. Qualitative data were mixed, with some supporting the feature, while some preferring traditional contact methods such as phone call and face-to-face. Some example quotes:

Theme 5: The Discussion Forum Wasn't Used, But In-School Champions Reported It Could Be Potentially Useful, Perhaps If Delivered Differently (Triangulation Convergence Label: Agree)
Quantitative usage data showed that use of the discussion forum was very low. There were 3 discussion forum posts made by support officers to generate content; however, there were 0 discussion forum posts by in-school champions or PE teachers during the entire program. In response, the support officers created a Facebook group for in-school champions which was established on November 7, 2018. From the launch of the Facebook group until the end of the program, there were 10 posts by the PA4E1 team (support officers and staff) and 12 posts by in-school champions in the Facebook group.
Qualitative data suggested the discussion forum was not very useful, and may be better delivered within the website, or could be better delivered using Facebook. Example quotes:

Meta-Theme: The Website Content Was Generally Acceptable and Appropriate, With a Few Specific Suggestions for Improvement (Triangulation Convergence Label: Agree)
As per quantitative data from the previous meta-theme relating to delivery, the majority of in-school champions agreed or strongly agreed that the website was acceptable, appropriate, and had good usability.
Qualitative data also indicated the website content was appropriate. Example quotes: Yeah, I think the online portal in its current form is kind of suitable, I think a few minor changes and adjustments would make it better.
...the online portal is essential with implementation of the program. You couldn't do it without it. It's your go-to resource.

Theme 1: A Lack of Notifications (or Prompts) to Highlight New Things in the Website Reduced Return Traffic to the Website (Triangulation Convergence Label: Silence)
No quantitative data were available for this theme. Qualitative data highlighted that notifications would have been useful to highlight updates to the website, which may subsequently increase usage.

Theme 2: The Lesson Observation Form (a Resource for Practice 1) Was Difficult to Find on the Website, It Was Difficult to Track Lesson Observation Completion, and Some Usability Issues With Completing Out Of Internet Range (Triangulation Convergence Label: Agree)
More than half of in-school champions (7/13) were unable to find the lesson observation section on the website. Qualitative data also indicated that in-school champions had difficulty finding the lesson observations, tracking PE teachers' completion of these observations, and also some issues with signal to the website while out of the internet range.

Theme 3: Termly Surveys (Support Strategy 7) Were Generally Acceptable and Completed (Triangulation Convergence Label: Slightly Agree)
From 0 to 12 months, 24/24 in-school champions completed all termly surveys and received the feedback reports from these surveys. From 12 to 24 months, 21/24 in-school champions completed all termly surveys and received feedback reports (3 schools missed 1 survey each). Few principals received the feedback reports, 7/24 principals had registered accounts with the program website and therefore received the feedback reports via email. The remaining principals did not directly receive the feedback reports, but may have been shown to them by their in-school champions. Qualitative data were scarce, but some support for the acceptability of the termly surveys was provided. Example quote: ...the termly surveys, I mean this was handy to see where we were at but I suppose before I did the survey I kind of had an idea of what we did and didn't do well.

Theme 4: The Resources Were Acceptable and Downloaded Frequently (Triangulation Convergence Label: Agree)
A total of 90 different resources were downloaded from the website at least once by either an in-school champion or a PE teacher, equating to a total of 1007 downloads. Example quotes:

Principal Findings
We have previously reported that the multimode implementation support strategies used in the PA4E1 scale-up trial increased the implementation of physical activity-promoting practices in lower socioeconomic secondary schools in New South Wales, Australia [32]. The results presented in this paper expand on these findings with quantitative and mixed methods process evaluation data on the role of the PA4E1 website in delivery of the implementation support strategies of the PA4E1 program. To our knowledge, this is the first comprehensive process evaluation of a school-based physical activity program to focus exclusively on the digital delivery mode of a multidelivery mode implementation support strategy. The primary findings indicate that usage of the website by both the in-school champions and PE teachers was high, which aligns with the high fidelity and reach of the implementation support found at 12 months reported elsewhere [32]. As intended, in-school champions had more frequent website use than PE teachers. The results of the mixed methods analysis indicated that both the delivery and content of the website were acceptable and appropriate. A number of usability issues were identified and are included as suggested modifications for future iterations of the website.
The findings support the value of the website within a multidelivery mode implementation intervention to support schools to implement physical activity promoting practices. Although other digital health interventions targeting physical activity have often suffered with issues of engagement [6,10,46], the website delivering implementation support for the PA4E1 program does not appear to have impaired the potential of the program to have a positive impact. As discussed by Sebire et al [47], embracing technology in school physical activity interventions may be an effective way to efficiently deliver content, for example, by reducing challenges related to limited time for training.
Completing this study about website engagement as a component of our broader process evaluation has provided information not typically collected within trials of school-based physical activity programs. Previous studies have either omitted evaluation of the website delivery mode [48,49], or largely relied on quantitative website usage data to assess usage engagement [9,[50][51][52]. These studies have reported varying levels of website usage, with studies' usage engagement varying between pilot and full trial [52,53]. Focusing only on usage data ignores the other component of engagement, subjective experience [9]. This study includes both quantitative and qualitative data exploring both usage and subjective experience engagement [9], with both data sets being triangulated to report high acceptability and appropriateness of the delivery and content of the website. Usage data showed that in-school champions and PE teachers accessed the website frequently, though this appeared to decline over time ( Figure 3). As intended by the design of the PA4E1 implementation support strategies (Multimedia Appendix 1), in-school champions accessed the website more frequently than PE teachers. Additionally, compared with other websites, the PA4E1 website has a rating of "good" usability [37,40].
The study was novel in its use of "think-aloud" methodology to explore teachers' experiences of using a website to support the delivery of a health-based intervention. Think-aloud methodologies have largely been used to inform the development of websites and apps for use directly with the target user of the health intervention [54][55][56][57][58] (eg, an app to support weight management among adults with diabetes [54]). By contrast, we used a think-aloud methodology with teachers (in-school champions) who have been accessing a website to support the delivery of the PA4E1 program to adolescent students. Previous studies using the think-aloud methodology [54][55][56][57][58] have found it to be useful for identifying usability and subjective experience issues. However, the procedure used in these studies involved the presence of a researcher, which may have affected participants' reactions. Our think-aloud procedure allowed in-school champions to respond to the think-aloud survey and usability test remotely, using their own devices, in their own time. Such a method is likely to be more ecologically valid than those involving the presence of researchers, at research sites, using research devices [59].

Intervention Refinements
This process evaluation revealed several refinements and suggestions for the PA4E1 website. In-school champions suggested the website was used in conjunction with other delivery modes (face-to-face, phone, email) and that these delivery modes of the implementation support strategies were also highly valued, which was additionally supported by usage data of multiple delivery modes. In addition, while the program website was acceptable and appropriate in its current format, it was often not the "first-choice" for in-school champions; often, in-school champions would use other delivery modes first. In-school champions also suggested the website could be embedded within other systems that teachers already access, such as state education portals. Therefore, we suggest future iterations of the PA4E1 program should carefully consider the balance of delivery modes used to deliver the implementation support strategies.
In-school champions suggested that both a website chat feature (to access support officers) and a discussion forum (to chat with other in-school champions) may be useful, although this was not a unanimous suggestion. Strategies to increase engagement with such features may be required, which was further highlighted by in-school champions who suggested the addition of notifications within the website and via email to promote usage of the website and highlight new content. Other features that may improve engagement include social networking platforms [60], social support, and behavioral prompts [61]. Finally, the lesson observation form (see Practice 1 outlined in Multimedia Appendix 1, Table S2) was difficult to find on the website (task completion = 46% [6/13]) and in-school champions reported that this could be made more prominent within the website.
Future iterations of a website supporting PA4E1 delivery may benefit from user testing and refinement prior to rolling out the website for delivery [62]. Such formative evaluations are common to inform e-commerce websites and from the human-computer interaction literature [62].

Strengths and Limitations
A key strength of this study is the detailed mixed methods design, triangulating multiple data sources to provide a more coherent and actionable set of themes from the data. Another strength is the use of an ecologically valid remote think-aloud survey and usability test procedure, to collect in-school champions experiences of using the PA4E1 website.
Limitations of this study include the low response rate to the think-aloud survey and usability test. Despite the remote nature of the survey, some respondents found it difficult to use the Loop 11 think-aloud survey and usability test software. Just over half of in-school champions (13/24, 54%) responded to the think-aloud survey and usability test. This may have introduced response bias, whereby those who responded were more likely to report high acceptability and appropriateness of the website. However, with regard to sample size, 5 participants are deemed sufficient for detecting most usability problems [63]. Another limitation was the inability to segregate user types prior to term 2, 2018. This was due to a technical difficulty in collecting the data in Google Analytics that could only be fixed prospectively from school term 2, 2018.

Conclusion
The results of this study provide context to support the primary outcome of the trial, highlighting that both in-school champions and PE teachers used the program website, and acceptability and appropriateness of the website were high among in-school champions. Our study has focused on the website delivery mode of a multidelivery mode implementation support strategy to support schools to implement physical activity promoting practices. We will publish further process evaluation results elsewhere, focusing on the remaining components of our process evaluation protocol [28].