A brief prestudy questionnaire, administered via REDCap, collected participants’ demographic information. A total of 2 research team members with expertise in qualitative methods conducted 5 web-based co-design and 12 usability testing sessions between December 2021 and August 2022 using Zoom (Zoom Video Communications); 2 researchers cofacilitated each session (MDW along with CF or YC). The researchers briefly introduced the session, including an overview of the research program, described the co-design or usability evaluation process to illustrate the intended purpose of each activity, and conducted an icebreaker activity to increase participant comfort. Panels of approximately 3-5 family participants or 3-5 clinicians were targeted for each co-design session; usability evaluation sessions had 1 participant each.

All sessions were conducted web-based, so participants were required to have an internet connection, access to an electronic device with a camera, and proficiency in English. Co-design sessions lasted approximately 60 minutes each, were audio-recorded, and digitally transcribed; usability testing sessions lasted approximately 45 minutes each. Participant names were replaced by sequential identifiers, and transcripts were verified by a research team member (CF or YC).

Participants were given 4 minutes to rapidly sketch 4 distinct pain risk scores to evaluate potential design approaches and visualization strategies. Next, each participant completed a wireframing exercise in turn, in which each participant was presented with empty rectangular text boxes indicating the prototype sections (demographics, risk factors, and mitigation strategies) and risk score examples (bar chart, pie chart, line plot, people count arrays, thermometer, or fuel gauge). They were then instructed to place the 3 prototype sections and 1 risk score into an empty canvas to indicate their preferences for placement, positioning, and sizing of each section. Finally, participants were shown our initial prototype [26] (Figure 1A) and prompted to critically evaluate the design. Data from the first round of co-design sessions were used to generate redesign themes and requirements.

Participants were invited to return for a second co-design session to elicit additional design requirements and visualization preferences to inform iterative prototype development. They were also asked to provide feedback on low- and high-risk clinical scenarios (including the corresponding risk score visualization) to be used during the usability evaluation sessions, specifically to comment on the scenario’s accuracy, understandability, realistic representation of patient risk, and whether they had any final suggestions.

Pilot usability testing sessions were conducted to confirm the applicability and feasibility of using the proposed low- and high-risk scenarios and to ensure the appropriateness of the interview questions before formal testing sessions. Then, an additional cohort of family participants and clinicians, who did not participate in any previous session, was recruited for usability evaluation.

Each participant was asked to identify information in the various prototype sections, for which we recorded time and accuracy. Specifically, they were asked to complete four tasks: (1) identify key demographic information (patient’s visit number and diagnosis date), (2) report the patient’s likelihood (and uncertainty) of developing postoperative pain if they did not adopt any team strategies to reduce its risk, (3) identify the patient’s risk factor that has the highest contribution to the patient’s overall chance of developing postoperative pain, and (4) indicate what the patient’s chance of developing postoperative pain is if the patient adopts the identified team risk-reduction strategies. This list of tasks was informed by the design team and our patient and clinical partners and was confirmed with study participants during the second co-design session.

Participants were then asked to role-play and “think aloud” through both the low-risk (inguinal hernia) and high-risk (scoliosis correction) scenarios (see Multimedia Appendix 1). Clinicians were asked to state how they would use the tool to communicate risk scores and team risk-reduction strategies to the patients’ families; family participants were asked to state how they would explain their child’s condition to their spouse or child. The order in which the low- and high-risk scenarios were presented to participants was alternated to minimize order effects. Task completion time, the accuracy of prototype interpretation, and the participants’ decision-making processes were recorded and analyzed. At the end of each session, we measured participants’ mental workload with the National Aeronautics and Space Administration (NASA) Task Load Index (TLX) [29] and user satisfaction with the Post-Study System Usability Questionnaire (PSSUQ) [30]. Both were administered via REDCap [28].

Through 2 rounds of co-design sessions, we identified requirements for (1) presentation and usability of the visualization tool, (2) identifying and categorizing risks and mitigation strategies, and (3) supporting collaboration and communication. We present them in detail below.

The key themes from the first co-design sessions were that the tool should (A) present risk severity descriptively and visually and (B) ensure that the appearance and navigation are user-friendly (Table 1). Most participants strongly preferred including a descriptive representation of risk severity (from “low” to “high”; design requirement R1.1, Table 1). To represent risk graphically, family participants typically drew child-friendly symbols and real-world objects, such as traffic lights or a race car on a track (R1.2). At the same time, clinicians preferred more traditional graphical representations, such as bar or pie charts (R1.3). This conflict was resolved by incorporating both views in the final set of requirements (R1.2 and R1.3). We were encouraged to change our original circular charts for risk factors and mitigation strategies (Figure 1A) to “fishbowls” for a more child-friendly tone, with the water level representing risk percentage (R1.2; Figure 1B).

Participants suggested emphasizing the risk score visualization as it represented essential information. Similarly, the elements for successful risk reduction and the change in risk score “after team strategies” (see also R2.4) were deemed essential and should be emphasized with color (R1.4). The wireframing exercise established an expected ordering of sections (Figure 1B). To accentuate the risk score, most participants enlarged the visualization, placed it centrally in the frame, and placed risk factors and mitigation strategies adjacently (R1.5).

Further themes derived from a critical examination of the existing prototype were that the tool should (C) frame risk identification and mitigation strategies positively and (D) clearly categorize and describe risks (Table 1).

Most participants approved of using a risk score’s range with uncertainty, as “it shows that there is always a margin of error” (family participant 2), and showing risk factors with percentages. However, some clinicians insisted that risk information be presented in a way that reduces anxiety, for example, displaying “non-modifiable [risk factors], such as biological sex, might distress the patient” (clinician 7), and “presenting families with a risk assessment of the potential pain after surgery could cause unintended adverse effects…you may be inducing pain by showing patients how much pain they might feel” [Clinician 4].

By “focusing on the possibility of mitigation, clinicians can focus the conversation on discussing pre-habilitation” (clinician 5) to reduce their risk score (R1.6). Separating risk factors that are “modifiable from non-modifiable” (clinician 1) and prioritizing modifiable factors may emphasize positive aspects of improving outcomes (R1.7).

Following a critical review of the revised prototype in the second co-design session, participants suggested we clarify some textual statements, such as changing “improving comfort” to “optimizing comfort” (R2.1; Figure 2A; for a higher-resolution version, see Multimedia Appendix 2).

Clinicians remained concerned that a patient’s level of pain “is highly influenced by consultation dialogue” (clinician 2) and thus favored using positive framing of risk around the concept of comfort; on the other hand, some family participants insisted that “using comfort/discomfort in a pain management tool confuses the goal of the tool for patients and families” (family participant 4). A clinician noted that “there is a lot of evidence that suggests the language we use can change people’s perception of pain” but acknowledged that some “pain language must be there” (clinician 5). This conflict was resolved by balancing both viewpoints equally in our design requirement; that is, we determined that we should maintain the positive framing of “optimizing comfort” but include pain terminology, such as “significant postoperative pain,” where relevant (R2.3).

Both clinicians and family participants needed clarification about the percentage changes associated with risk reduction (Figure 1B). They suggested we separate our risk score into 2 components: “Before team strategies” and “After team strategies” (R2.4 and see also R1.4). Most participants did not identify the asterisk footnote about the statistical uncertainty of the risk score (Figure 1B), which suggested we move this detail to the textual risk statement (R2.1).

Clinicians and family participants recommended “focusing families on the mitigation strategies” (clinician 2) and their ability to reduce the risk of pain following surgery; that is, we should emphasize how much risk reduction the cross-functional teams can achieve in the “After Team Strategies” risk score (R2.5; Figure 2A). Finally, it was suggested that we reorder the modifiable contributors to read from high to low, reflecting how users would process the information (reinforcing R1.5).

Some participants were confused about whether the risk factor percentages (Figure 1) were population-based or personalized and recommended “stressing that the percentages are individualized” (clinician 4; Figure 2A). Similarly, separating “Other contributors to score” (eg, medication) from “Nonmodifiable contributors to score” (eg, age and biological sex) would improve clarity (reinforcing R1.7).

We were asked to update “Diet” to “Nutrition” to emphasize nutrient intake rather than food quantity (Figure 2A), and a family participant noted that diet and exercise “might be the hardest to implement in your daily life if you have disabilities, live in poverty, and/or a pre-existing condition(s)” (family participant 4). It was suggested that we change the section title to “Potentially Modifiable Contributors” and list the strategies in point form to increase ease of readability (R2.1).

The final theme from the co-design sessions was that the tool should (E) emphasize collaboration and effective communication (Table 1). Following a critical review of the revised prototype in the second co-design session, participants suggested it should emphasize the collaboration of cross-functional care teams (patient’s family, nurses, and physicians) by changing “your physician” to “your health care team” (R2.2).

Our original notes section was too small; it needed sufficient space for participants to write and draw essential information from the consultation (R2.6) and thus was separated and placed on a separate numbered page (see also R2.7; Figure 2B). Participants approved the checklist questions as reminders of topics to discuss during the consultation (Figure 1B). However, several participants stated the importance of knowing who to contact to assist with their child’s care (see also R2.1). Some insisted that the questions should be connected to the information shown and not require complex answers (see also R2.1; Figure 2B).

During low- and high-risk clinical role-play, participants typically said they would use the tool by comparing the risk scores before and after risk reduction strategies to demonstrate the proposed benefits to motivate positive change. Participants typically said they would describe the top 3 modifiable contributors and ensure the family could implement the targeted strategies. Most participants thought that the comprehension checklist was a helpful reminder and that the notes section was beneficial to document clinical information, such as targeted strategies, referral contacts, aftercare instructions, and questions arising during the consultation.

Most participants agreed that the child would benefit from seeing the tool, at least for older children (>12 years of age), though some felt it would benefit children as young as 5 years of age. There were conflicting views on the use of the tool in practice: some clinicians would not focus on discussing low-risk pain with the patient, or if only a small risk mitigation was anticipated, as it may increase patient anxiety; however, family participants all agreed that the risk score should be used in both low- and high-risk scenarios as any decrease in postoperative pain would be meaningful for a child’s surgical experience. All participants recognized that nonmodifiable contributors, such as biological sex and age, provided context but would not discuss these with the child.

Completion time across tasks varied, but tasks were largely completed quickly (median time to complete each task ranged from median 5.0, IQR 2.9-6.8 seconds to median 16.6, IQR 10.9-34.3 seconds) and accuracy ranged from 11/12 (92%) correct to 12/12 (100%) correct, with only 2 requests for assistance and no evidence for substantial differences between clinicians and family participants (Table 2).

Median (IQR) PSSUQ scores [30] (on a 1-7 scale) across all participants were 2.0 (IQR 1.5-2.2) for system usefulness, 2.5 (IQR 1.9-3.0) for information quality, 2.0 (IQR 1.8-2.7) for interface quality, and 2.1 (IQR 1.5-2.7) for overall score, which are within typical ranges of acceptability [36]. The scores were similar between clinicians and family participants, but there was some evidence for differences in their levels of satisfaction with information quality and interface quality (Table 3), which may reflect conflicting views on the use of this tool in all scenarios.

Median (IQR) NASA TLX subcomponent scores [29] suggested that participants generally felt they were able to perform the tasks required of them, though clinicians scored this subcomponent lower than family participants. Mental demand and effort to achieve their performance contributed the most to the overall workload; scores for temporal demand, physical demand, and frustration were low (Table 4).

Personalized risk communication tools have yet to be widely implemented in clinical practice and have not yet been studied in the context of pediatric pain risk following surgery. However, in preliminary studies previously conducted with adult patients, participants have largely rated personalized risk communication tools as easy to use [37,38], helpful [39], and beneficial to patients [40]. Participants believed that personalized risk communication might result in increased patient engagement [41], increased awareness and understanding of potential surgical complications, and deeper discussion with providers [42]. Participants who were presented with a personalized risk score while consenting for surgery, in particular, agreed that they had received adequate time discussing surgical risks, felt more comfortable with their procedure, had decreased anxiety [43], were significantly more satisfied with the consent process [40,43], and had increased knowledge about the risks associated with their surgery [40]. These other studies explicitly support the motivation behind our research program and the development of a perioperative pediatric risk communication tool. Our usability data mirror some of these findings.

In previous studies, patients also preferred the visual consent tool to text-based documents [42], found personalized surgical risk communication tools helpful for informed decision-making [39], indicated that they would use a personalized risk tool again before a future procedure [40], and believed sharing personalized risk information should be a universal requirement during surgical consultations [44]. Clinicians also highlighted that identifying modifiable risk factors was more important than nonmodifiable ones; thus, separating their contributions was critical [38]. On viewing their risk of postsurgical complications, most patients said they would consider participating in a structured prehabilitation program to decrease their risk and improve postsurgical outcomes [44]. These results are promising and broadly consistent with our findings. However, these previous studies were conducted only with adult participants. While they helped inspire our design decisions, their detailed recommendations can only be fully translated by reevaluating how this approach should be applied in the pediatric surgical setting, as we have done in this study.

Other studies have shown that data-driven personalized risk calculations can perform significantly better than physicians in predicting patient outcomes in the perioperative domain, and exposure to the calculated risk scores can improve physicians’ prediction accuracy [37]. Patients with low-risk may be more likely to overestimate their surgical risk [39,44], and patients with high-risk may be more likely to underestimate their risk [44], highlighting the importance of improving patient understanding of potential postsurgical complications through supported communication. As we discovered during our usability testing sessions, a point of conflict between patients and health care professionals is that patients typically prefer to be shown their personalized risk scores even when they are low-risk, while some health care professionals felt that this might result in a misuse of clinic time (as was also found in other previous studies [39,41]) or unnecessarily increase anxiety; our family participants universally agreed the risk tool should be used in low- as well as high-risk situations, explaining that any opportunity to decrease postoperative pain, no matter how small, would be beneficial to their child. This further highlights the benefit of involving family participants in the design process to ensure effective use by expected end users.

Where patients and health care professionals have provided input to the design and feature considerations of personalized risk score tools, their requirements echo the findings from this study. For example, other studies also identified simplicity and clarity as essential characteristics to facilitate shared decision-making [45,46], and to ensure that risk information does not overwhelm patients [45-47]. Other focus group studies have also suggested including less complex language for risk severity, such as “low,” “medium,” and “high” [44], and that simplistic visualizations and language are crucial to understanding—patients have been confused by highly interactive visualizations and better comprehended static charts [45]. These previous findings are similar to the design requirements we established for this prototype.

Patients and care providers have previously indicated that a multimodal risk score supports the learning styles and preferences of various users [40,46]. Furthermore, continuing to engage stakeholders, educating staff, and allowing smooth integration into workflows have previously been noted as important implementation success factors [40]. The findings from these other studies are broadly reflected in our participants’ comments during our co-design sessions and usability evaluation.

While addressing user needs overall is critical, we must acknowledge some differences between clinicians’ and family participants’ perceptions of the risk communication tool in our usability testing. While clinicians were satisfied with the tool overall, their lower levels of satisfaction with information and interface quality (as evidenced by their lower PSSUQ scores for these components) may have reflected concerns about what information is presented; clinicians also felt less able to perform the tasks required of them (as evidenced by their lower NASA TLX performance scores). These differences may reflect clinicians’ concerns about the potential adverse effects of communicating risk to patients via the “nocebo effect” [48], which should be recognized and addressed, particularly in the pediatric setting. We incorporated this clinical guidance into the final iteration of our prototype to provide the risk information through a “comfort” lens; that is, we aimed to focus the efforts of the clinician and family team on “maintaining comfort” rather than “reducing pain,” while still acknowledging the risk of pain inherent in their surgical procedure. This approach is consistent with international initiatives aimed at prioritizing compassionate approaches to the recognition, prevention, and treatment of children’s pain (eg, ChildKind International [49]) as well as our own institution’s recommended approach [50]. It seems these changes may not have been sufficient to address the concerns of our clinical participants in usability testing. Importantly, however, other research has shown that, while increased risk has been correlated with patients wanting to discuss the procedure in more detail with their providers [37], presenting patients with their personalized risk has not been associated with canceling procedures or changes in the decision to undergo surgery [39,43,44].