Background: Risk identification and communication tools have the potential to improve health care by supporting clinician-patient or family discussion of treatment risks and benefits and helping patients make more informed decisions; however, they have yet to be tailored to pediatric surgery. User-centered design principles can help to ensure the successful development and uptake of health care tools.
Objective: We aimed to develop and evaluate the usability of an easy-to-use tool to communicate a child’s risk of postoperative pain to improve informed and collaborative preoperative decision-making between clinicians and families.
Methods: With research ethics board approval, we conducted web-based co-design sessions with clinicians and family participants (people with lived surgical experience and parents of children who had recently undergone a surgical or medical procedure) at a tertiary pediatric hospital. Qualitative data from these sessions were analyzed thematically using NVivo (Lumivero) to identify design requirements to inform the iterative redesign of an existing prototype. We then evaluated the usability of our final prototype in one-to-one sessions with a new group of participants, in which we measured mental workload with the National Aeronautics and Space Administration (NASA) Task Load Index (TLX) and user satisfaction with the Post-Study System Usability Questionnaire (PSSUQ).
Results: A total of 12 participants (8 clinicians and 4 family participants) attended 5 co-design sessions. The 5 requirements were identified: (A) present risk severity descriptively and visually; (B) ensure appearance and navigation are user-friendly; (C) frame risk identification and mitigation strategies in positive terms; (D) categorize and describe risks clearly; and (E) emphasize collaboration and effective communication. A total of 12 new participants (7 clinicians and 5 family participants) completed a usability evaluation. Tasks were completed quickly (range 5-17 s) and accurately (range 11/12, 92% to 12/12, 100%), needing only 2 requests for assistance. The median (IQR) NASA TLX performance score of 78 (66-89) indicated that participants felt able to perform the required tasks, and an overall PSSUQ score of 2.1 (IQR 1.5-2.7) suggested acceptable user satisfaction with the tool.
Conclusions: The key design requirements were identified, and that guided the prototype redesign, which was positively evaluated during usability testing. Implementing a personalized risk communication tool into pediatric surgery can enhance the care process and improve informed and collaborative presurgical preparation and decision-making between clinicians and families of pediatric patients.
Surgery poses a substantial risk for postoperative pain, with roughly 1 in 5 children experiencing pain 12 months following surgery , which can have detrimental consequences on their long-term well-being and future care-seeking behaviors [ ]. The discovery of factors that increase the risk of postoperative pain (eg, anxiety, poor pain coping skills, and pain catastrophizing) [ - ] and the development of prehabilitation plans (eg, improved nutrition and exercise) [ - ] presents an opportunity to improve postoperative outcomes (eg, reduced length of stay [ ] and reduced pain [ ]). However, these risk factors and potential mitigation are not always communicated clearly and consistently in line with the information needs of the patients and their families [ ]. Consequently, researchers are developing a risk identification and communication tools to improve information-sharing with patients and to improve care [ ], with some initial success (eg, improved comprehension of procedure-associated risks) [ ]. This work has all been focused on adult patients, but there is a clear need to extend this approach to pediatric surgery [ ].
Furthermore, the development of health risk communication tools has typically not applied user-centered design principles [, ]. Participatory design techniques, such as co-design focus groups, can directly incorporate stakeholders throughout development and ensure end user needs are addressed in the design process [ ]. Engaging end users has several benefits: access to tacit knowledge and improved knowledge generation [ ]; more profound understanding of user needs through examining visual overviews and written communication between users [ - ]; improved design of research materials; and improved quality of service, including a better fit between user needs and the service provided, and increased trust during participation in clinical trials [ , ]. These benefits can all potentially increase the generalizability of results [ ]. Hence, incorporating primary end users’ feedback into the development process is imperative to creating easy-to-use and effective risk communication tools and their successful uptake in clinical practice.
Aligned with the BC Children’s Hospital’s (BCCH) continued priority of improving pediatric pain management  and patient-centered care [ ], our research program aims to (1) contribute to the efforts to minimize postoperative pain and to reduce pain medication requirements, specifically opioids and (2) generalize from the specific use case of pediatric postsurgical pain to develop a range of risk prediction communication tools pertinent to other clinical scenarios. In a previous study, we established the design requirements for a pediatric postoperative pain risk visualization tool, which guided the development of the initial prototype for an easy-to-use tool targeted at clinicians and family users [ ]. Our preliminary prototype design was guided by our expected end users’ requirements and included the nonthreatening and multimodal presentation of risk, an estimation of risk factors’ contribution, and mitigation strategies to decrease the patient’s level of risk [ ].
The purpose of this study was to further develop and evaluate our preliminary pediatric postoperative pain risk communication prototype before implementing it into clinical practice. Hence, we aimed to (1) acquire additional design requirements from both sets of expected end users (clinicians and family members) through a critique of our preliminary design and a series of co-design activities and (2) evaluate the usability of the redesigned prototype through role-play based on low- and high-risk clinical scenarios and follow-up with standardized usability questionnaires.
Study Design and Approval
We conducted a two-stage study: (1) small group co-design over 2 sessions, followed by (2) individual usability evaluation sessions. Both stages were conducted with BCCH clinicians and family participants, including parents whose children had previously undergone surgery and adults with lived pediatric surgical experience.
We recruited BCCH attending physicians and nurse practitioners via departmental email distribution lists, parents via BCCH patient experience email lists and in-person invitations in the anesthetic care unit, and adults with previous childhood surgery via a provincial research network platform (REACH BC). A trained research team member described the study in detail and acquired written informed consent or web-based informed consent using Research Electronic Data Capture (REDCap, Vanderbilt University) [, ]. In our report, parents and adults with pediatric surgical experience are collectively referred to as “family participant(s)” to protect their privacy. All participants were remunerated CAD $25 (US $19.50) per hour for their expertise and time. The co-design sessions included a mixture of family participants and clinicians. Usability evaluation sessions were individual, and participants in these sessions had not previously participated in co-design.
A brief prestudy questionnaire, administered via REDCap, collected participants’ demographic information. A total of 2 research team members with expertise in qualitative methods conducted 5 web-based co-design and 12 usability testing sessions between December 2021 and August 2022 using Zoom (Zoom Video Communications); 2 researchers cofacilitated each session (MDW along with CF or YC). The researchers briefly introduced the session, including an overview of the research program, described the co-design or usability evaluation process to illustrate the intended purpose of each activity, and conducted an icebreaker activity to increase participant comfort. Panels of approximately 3-5 family participants or 3-5 clinicians were targeted for each co-design session; usability evaluation sessions had 1 participant each.
All sessions were conducted web-based, so participants were required to have an internet connection, access to an electronic device with a camera, and proficiency in English. Co-design sessions lasted approximately 60 minutes each, were audio-recorded, and digitally transcribed; usability testing sessions lasted approximately 45 minutes each. Participant names were replaced by sequential identifiers, and transcripts were verified by a research team member (CF or YC).
Co-Design Iteration 1
Participants were given 4 minutes to rapidly sketch 4 distinct pain risk scores to evaluate potential design approaches and visualization strategies. Next, each participant completed a wireframing exercise in turn, in which each participant was presented with empty rectangular text boxes indicating the prototype sections (demographics, risk factors, and mitigation strategies) and risk score examples (bar chart, pie chart, line plot, people count arrays, thermometer, or fuel gauge). They were then instructed to place the 3 prototype sections and 1 risk score into an empty canvas to indicate their preferences for placement, positioning, and sizing of each section. Finally, participants were shown our initial prototype  ( A) and prompted to critically evaluate the design. Data from the first round of co-design sessions were used to generate redesign themes and requirements.
Co-Design Iteration 2
Participants were invited to return for a second co-design session to elicit additional design requirements and visualization preferences to inform iterative prototype development. They were also asked to provide feedback on low- and high-risk clinical scenarios (including the corresponding risk score visualization) to be used during the usability evaluation sessions, specifically to comment on the scenario’s accuracy, understandability, realistic representation of patient risk, and whether they had any final suggestions.
Pilot usability testing sessions were conducted to confirm the applicability and feasibility of using the proposed low- and high-risk scenarios and to ensure the appropriateness of the interview questions before formal testing sessions. Then, an additional cohort of family participants and clinicians, who did not participate in any previous session, was recruited for usability evaluation.
Each participant was asked to identify information in the various prototype sections, for which we recorded time and accuracy. Specifically, they were asked to complete four tasks: (1) identify key demographic information (patient’s visit number and diagnosis date), (2) report the patient’s likelihood (and uncertainty) of developing postoperative pain if they did not adopt any team strategies to reduce its risk, (3) identify the patient’s risk factor that has the highest contribution to the patient’s overall chance of developing postoperative pain, and (4) indicate what the patient’s chance of developing postoperative pain is if the patient adopts the identified team risk-reduction strategies. This list of tasks was informed by the design team and our patient and clinical partners and was confirmed with study participants during the second co-design session.
Participants were then asked to role-play and “think aloud” through both the low-risk (inguinal hernia) and high-risk (scoliosis correction) scenarios (see). Clinicians were asked to state how they would use the tool to communicate risk scores and team risk-reduction strategies to the patients’ families; family participants were asked to state how they would explain their child’s condition to their spouse or child. The order in which the low- and high-risk scenarios were presented to participants was alternated to minimize order effects. Task completion time, the accuracy of prototype interpretation, and the participants’ decision-making processes were recorded and analyzed. At the end of each session, we measured participants’ mental workload with the National Aeronautics and Space Administration (NASA) Task Load Index (TLX) [ ] and user satisfaction with the Post-Study System Usability Questionnaire (PSSUQ) [ ]. Both were administered via REDCap [ ].
Session transcripts were analyzed with NVivo, and results were summarized using a thematic analysis . The 2 research team members (MDW along with CF or YC) independently reviewed the 2 transcripts. They used inductive coding [ ] to develop a preliminary list of thematic codes organized by theme, subtheme, and participant type [ ]. These researchers then compared interpretations and developed consistent codes, which were applied to the remaining 3 transcripts using deductive coding [ ]. The 3 researchers discussed additional themes that emerged after coding these remaining transcripts, resolved any discrepancies, and modified the coding framework to ensure that the key concepts were captured. For co-design, a saturation criterion was implemented [ ]; specifically, the 2 research team members (MDW along with CF or YC) determined that similar comments and concerns were repeatedly discussed and that data saturation had occurred.
Our prototype was developed using an iterative process in which the research team created, discussed, and revised the prototype using Figma (Figma Inc). Prominent themes emerging from co-design sessions were then used to generate design themes and requirements for prototype redesign.
For the usability evaluation, quantitative data for task completion times, workload, and user perceptions were summarized as median (IQR) values using R (R Core Team). Task accuracies and requests for assistance were summarized as counts and percentages.
Ethical approval was obtained from the University of British Columbia or Children’s & Women’s Health Centre of British Columbia Research Ethics Board (H20-00613; date of approval 2020-10-20; Principal Investigator: MG). Findings are reported following the Consolidated Criteria for Reporting Qualitative Research checklist .
A total of 12 participants, including 8 clinicians (3 nurse practitioners and 5 anesthesiologists) and 4 family participants attended 5 co-design sessions with a mix of 3-5 clinical and family participants in each session; 12 participants, including 7 clinicians (1 registered nurse, 5 anesthesiologists, and 1 surgeon) and 5 family participants evaluated the usability of our prototype. Family participants were: 6/9 (67%) female; 3/9 (33%) over the age of 49; and 6/9 (67%) with either a certificate (university or nonuniversity) or university degree, and 3/9 (33%) with a high school diploma (or equivalent). Clinicians were female (8/15, 53%), older than 49 years (7/15, 47%), in practice for more than 5 years (10/15, 67%), and clinical fellows (2/15, 13%).
Through 2 rounds of co-design sessions, we identified requirements for (1) presentation and usability of the visualization tool, (2) identifying and categorizing risks and mitigation strategies, and (3) supporting collaboration and communication. We present them in detail below.
Presentation and Usability of the Visualization Tool
The key themes from the first co-design sessions were that the tool should (A) present risk severity descriptively and visually and (B) ensure that the appearance and navigation are user-friendly (). Most participants strongly preferred including a descriptive representation of risk severity (from “low” to “high”; design requirement R1.1, ). To represent risk graphically, family participants typically drew child-friendly symbols and real-world objects, such as traffic lights or a race car on a track (R1.2). At the same time, clinicians preferred more traditional graphical representations, such as bar or pie charts (R1.3). This conflict was resolved by incorporating both views in the final set of requirements (R1.2 and R1.3). We were encouraged to change our original circular charts for risk factors and mitigation strategies ( A) to “fishbowls” for a more child-friendly tone, with the water level representing risk percentage (R1.2; B).
|Design requirement theme||Requirements elicited during the first co-design session||Requirements elicited during the second co-design session|
|ID||Specific design requirement||ID||Specific design requirement|
|(A) Present risk severity descriptively and visually|
|R1.1||Communicate risk severity descriptively (eg, “low” or “high”)||N/Aa||N/A|
|R1.3||Visualize continuum of risk level|
|(B) Ensure appearance and navigation are user-friendly|
|R1.2||Use real-world symbols to increase user-friendliness||R2.7||Add page numbers to encourage users to review both pages|
|R1.4||Use color to highlight essential information|
|R1.5||Order sections to match users’ intuitive information processing expectations|
|(C) Frame risk identification and mitigation strategies in positive terms|
|R1.6||Reduce patient and family anxiety by emphasizing positive impact risk mitigation strategies can provide||R2.3||Maintain positive framing of optimizing comfort; include pain terminology in relevant sections|
|R2.4||Separate risk scale into before and after to highlight the effect of mitigation strategies|
|R2.5||Emphasize mitigation strategies and subsequent effects on reducing the risk of pain|
|(D) Clearly categorize and describe risks|
|R1.7||Increase clarity by improving categorization of risk factors||R2.1||Reword textual risk statements to increase readability and reduce anxiety|
|(E) Emphasize collaboration and effective communication|
|N/A||N/A||R2.2||Emphasize collaboration of cross-functional care teams|
|R2.6||Create a separate page for the checklist and notes section|
aN/A: not applicable.
Participants suggested emphasizing the risk score visualization as it represented essential information. Similarly, the elements for successful risk reduction and the change in risk score “after team strategies” (see also R2.4) were deemed essential and should be emphasized with color (R1.4). The wireframing exercise established an expected ordering of sections (B). To accentuate the risk score, most participants enlarged the visualization, placed it centrally in the frame, and placed risk factors and mitigation strategies adjacently (R1.5).
Identifying and Categorizing Risks and Mitigation Strategies
Further themes derived from a critical examination of the existing prototype were that the tool should (C) frame risk identification and mitigation strategies positively and (D) clearly categorize and describe risks ().
Most participants approved of using a risk score’s range with uncertainty, as “it shows that there is always a margin of error” (family participant 2), and showing risk factors with percentages. However, some clinicians insisted that risk information be presented in a way that reduces anxiety, for example, displaying “non-modifiable [risk factors], such as biological sex, might distress the patient” (clinician 7), and “presenting families with a risk assessment of the potential pain after surgery could cause unintended adverse effects…you may be inducing pain by showing patients how much pain they might feel” [Clinician 4].
By “focusing on the possibility of mitigation, clinicians can focus the conversation on discussing pre-habilitation” (clinician 5) to reduce their risk score (R1.6). Separating risk factors that are “modifiable from non-modifiable” (clinician 1) and prioritizing modifiable factors may emphasize positive aspects of improving outcomes (R1.7).
Following a critical review of the revised prototype in the second co-design session, participants suggested we clarify some textual statements, such as changing “improving comfort” to “optimizing comfort” (R2.1;A; for a higher-resolution version, see ).
Clinicians remained concerned that a patient’s level of pain “is highly influenced by consultation dialogue” (clinician 2) and thus favored using positive framing of risk around the concept of comfort; on the other hand, some family participants insisted that “using comfort/discomfort in a pain management tool confuses the goal of the tool for patients and families” (family participant 4). A clinician noted that “there is a lot of evidence that suggests the language we use can change people’s perception of pain” but acknowledged that some “pain language must be there” (clinician 5). This conflict was resolved by balancing both viewpoints equally in our design requirement; that is, we determined that we should maintain the positive framing of “optimizing comfort” but include pain terminology, such as “significant postoperative pain,” where relevant (R2.3).
Both clinicians and family participants needed clarification about the percentage changes associated with risk reduction (B). They suggested we separate our risk score into 2 components: “Before team strategies” and “After team strategies” (R2.4 and see also R1.4). Most participants did not identify the asterisk footnote about the statistical uncertainty of the risk score ( B), which suggested we move this detail to the textual risk statement (R2.1).
Clinicians and family participants recommended “focusing families on the mitigation strategies” (clinician 2) and their ability to reduce the risk of pain following surgery; that is, we should emphasize how much risk reduction the cross-functional teams can achieve in the “After Team Strategies” risk score (R2.5;A). Finally, it was suggested that we reorder the modifiable contributors to read from high to low, reflecting how users would process the information (reinforcing R1.5).
Some participants were confused about whether the risk factor percentages () were population-based or personalized and recommended “stressing that the percentages are individualized” (clinician 4; A). Similarly, separating “Other contributors to score” (eg, medication) from “Nonmodifiable contributors to score” (eg, age and biological sex) would improve clarity (reinforcing R1.7).
We were asked to update “Diet” to “Nutrition” to emphasize nutrient intake rather than food quantity (A), and a family participant noted that diet and exercise “might be the hardest to implement in your daily life if you have disabilities, live in poverty, and/or a pre-existing condition(s)” (family participant 4). It was suggested that we change the section title to “Potentially Modifiable Contributors” and list the strategies in point form to increase ease of readability (R2.1).
Supporting Collaboration and Communication
The final theme from the co-design sessions was that the tool should (E) emphasize collaboration and effective communication (). Following a critical review of the revised prototype in the second co-design session, participants suggested it should emphasize the collaboration of cross-functional care teams (patient’s family, nurses, and physicians) by changing “your physician” to “your health care team” (R2.2).
Our original notes section was too small; it needed sufficient space for participants to write and draw essential information from the consultation (R2.6) and thus was separated and placed on a separate numbered page (see also R2.7;B). Participants approved the checklist questions as reminders of topics to discuss during the consultation ( B). However, several participants stated the importance of knowing who to contact to assist with their child’s care (see also R2.1). Some insisted that the questions should be connected to the information shown and not require complex answers (see also R2.1; B).
Potential Use of the Tool in Practice
During low- and high-risk clinical role-play, participants typically said they would use the tool by comparing the risk scores before and after risk reduction strategies to demonstrate the proposed benefits to motivate positive change. Participants typically said they would describe the top 3 modifiable contributors and ensure the family could implement the targeted strategies. Most participants thought that the comprehension checklist was a helpful reminder and that the notes section was beneficial to document clinical information, such as targeted strategies, referral contacts, aftercare instructions, and questions arising during the consultation.
Most participants agreed that the child would benefit from seeing the tool, at least for older children (>12 years of age), though some felt it would benefit children as young as 5 years of age. There were conflicting views on the use of the tool in practice: some clinicians would not focus on discussing low-risk pain with the patient, or if only a small risk mitigation was anticipated, as it may increase patient anxiety; however, family participants all agreed that the risk score should be used in both low- and high-risk scenarios as any decrease in postoperative pain would be meaningful for a child’s surgical experience. All participants recognized that nonmodifiable contributors, such as biological sex and age, provided context but would not discuss these with the child.
Task Completion, Usability Ratings, and Workload
Completion time across tasks varied, but tasks were largely completed quickly (median time to complete each task ranged from median 5.0, IQR 2.9-6.8 seconds to median 16.6, IQR 10.9-34.3 seconds) and accuracy ranged from 11/12 (92%) correct to 12/12 (100%) correct, with only 2 requests for assistance and no evidence for substantial differences between clinicians and family participants ().
|Task||Time (seconds), median (IQR)||Accuracy (n=12), n (%)||Requests for assistance|
|Clinicians (n=7)||Family participants (n=5)|
|Task 1: demographics||11.8 (7.5-20.9)||11.4 (9.2-17.4)||11 (92)||0|
|Task 2: risk of pain||16.5 (7.2-25.3)||16.6 (15.6-34.3)||11 (92)||1 Family participant|
|Task 3: main risk factor||7.0 (3.4-10.7)||6.9 (4.0-9.3)||11 (92)||1 Clinician|
|Task 4: effect of team strategies||6.1 (3.0-7.3)||4.3 (2.9-6.1)||12 (100)||0|
Median (IQR) PSSUQ scores  (on a 1-7 scale) across all participants were 2.0 (IQR 1.5-2.2) for system usefulness, 2.5 (IQR 1.9-3.0) for information quality, 2.0 (IQR 1.8-2.7) for interface quality, and 2.1 (IQR 1.5-2.7) for overall score, which are within typical ranges of acceptability [ ]. The scores were similar between clinicians and family participants, but there was some evidence for differences in their levels of satisfaction with information quality and interface quality ( ), which may reflect conflicting views on the use of this tool in all scenarios.
|PSSUQb factor||Composite score, median (IQR)||Composite score of ≤3 equating to strongly agree, agree, or somewhat agree, n (%)|
|Clinicians (n=7)||Family participants (n=5)||Clinicians (n=7)||Family participants (n=5)|
|System usefulness||2.0 (2.0-2.4)||1.3 (1.0-1.5)||7 (100)||5 (100)|
|Information quality||3.0 (2.7-3.4)||1.7 (1.0-2.0)||4 (57)||5 (100)|
|Interface quality||2.7 (2.3-3.2)||1.5 (1.0-2.0)||5 (71)||4 (100)c|
|Overall||2.6 (2.2-2.9)||1.4 (1.4-1.5)||5 (71)||5 (100)|
aComposite scores are the mean of subcomponent scores on a 1-7 scale, with lower numbers indicating greater agreement.
bPSSUQ: Post-Study System Usability Questionnaire.
cOne family participant answered not applicable for all “interface quality” questions.
Median (IQR) NASA TLX subcomponent scores  suggested that participants generally felt they were able to perform the tasks required of them, though clinicians scored this subcomponent lower than family participants. Mental demand and effort to achieve their performance contributed the most to the overall workload; scores for temporal demand, physical demand, and frustration were low ( ).
|TLXb subcomponent||Score, median (IQR)|
|Clinicians (n=7)||Family participants (n=5)|
|1. Mental demand||27 (19-34)||16 (15-23)|
|2. Physical demand||12 (0-12)||0 (0-2)|
|3. Temporal demand||15 (14-18)||3 (1-6)|
|4. Performance||67 (64-79)||96 (82-97)|
|5. Effort||50 (17-56)||48 (21-50)|
|6. Frustration||13 (8-23)||2 (1-3)|
aScores are given on a sliding scale from 0=very low to 100=very high.
bTLX: Task Load Index.
In this user-centered study, 5 outline design requirements for a pediatric postoperative pain risk visualization tool were identified during co-design focus group sessions, which built on previously published work : (A) present risk severity descriptively and visually, (B) ensure appearance and navigation are user-friendly, (C) frame risk identification and mitigation strategies in positive terms, (D) categorize and describe risks clearly, and (E) emphasize collaboration and effective communication. A revised risk communication prototype based on these requirements was used by both clinical and family participants quickly and accurately. Only minor frustration was reported during the usability evaluation, and user perception of the tool was acceptable.
Comparison With Prior Work
Development and Use of Risk Communication Tools
Personalized risk communication tools have yet to be widely implemented in clinical practice and have not yet been studied in the context of pediatric pain risk following surgery. However, in preliminary studies previously conducted with adult patients, participants have largely rated personalized risk communication tools as easy to use [, ], helpful [ ], and beneficial to patients [ ]. Participants believed that personalized risk communication might result in increased patient engagement [ ], increased awareness and understanding of potential surgical complications, and deeper discussion with providers [ ]. Participants who were presented with a personalized risk score while consenting for surgery, in particular, agreed that they had received adequate time discussing surgical risks, felt more comfortable with their procedure, had decreased anxiety [ ], were significantly more satisfied with the consent process [ , ], and had increased knowledge about the risks associated with their surgery [ ]. These other studies explicitly support the motivation behind our research program and the development of a perioperative pediatric risk communication tool. Our usability data mirror some of these findings.
In previous studies, patients also preferred the visual consent tool to text-based documents , found personalized surgical risk communication tools helpful for informed decision-making [ ], indicated that they would use a personalized risk tool again before a future procedure [ ], and believed sharing personalized risk information should be a universal requirement during surgical consultations [ ]. Clinicians also highlighted that identifying modifiable risk factors was more important than nonmodifiable ones; thus, separating their contributions was critical [ ]. On viewing their risk of postsurgical complications, most patients said they would consider participating in a structured prehabilitation program to decrease their risk and improve postsurgical outcomes [ ]. These results are promising and broadly consistent with our findings. However, these previous studies were conducted only with adult participants. While they helped inspire our design decisions, their detailed recommendations can only be fully translated by reevaluating how this approach should be applied in the pediatric surgical setting, as we have done in this study.
Personalized and Contextualized Risk Scores
Other studies have shown that data-driven personalized risk calculations can perform significantly better than physicians in predicting patient outcomes in the perioperative domain, and exposure to the calculated risk scores can improve physicians’ prediction accuracy . Patients with low-risk may be more likely to overestimate their surgical risk [ , ], and patients with high-risk may be more likely to underestimate their risk [ ], highlighting the importance of improving patient understanding of potential postsurgical complications through supported communication. As we discovered during our usability testing sessions, a point of conflict between patients and health care professionals is that patients typically prefer to be shown their personalized risk scores even when they are low-risk, while some health care professionals felt that this might result in a misuse of clinic time (as was also found in other previous studies [ , ]) or unnecessarily increase anxiety; our family participants universally agreed the risk tool should be used in low- as well as high-risk situations, explaining that any opportunity to decrease postoperative pain, no matter how small, would be beneficial to their child. This further highlights the benefit of involving family participants in the design process to ensure effective use by expected end users.
Essential Design and Feature Considerations
Where patients and health care professionals have provided input to the design and feature considerations of personalized risk score tools, their requirements echo the findings from this study. For example, other studies also identified simplicity and clarity as essential characteristics to facilitate shared decision-making [, ], and to ensure that risk information does not overwhelm patients [ - ]. Other focus group studies have also suggested including less complex language for risk severity, such as “low,” “medium,” and “high” [ ], and that simplistic visualizations and language are crucial to understanding—patients have been confused by highly interactive visualizations and better comprehended static charts [ ]. These previous findings are similar to the design requirements we established for this prototype.
Patients and care providers have previously indicated that a multimodal risk score supports the learning styles and preferences of various users [, ]. Furthermore, continuing to engage stakeholders, educating staff, and allowing smooth integration into workflows have previously been noted as important implementation success factors [ ]. The findings from these other studies are broadly reflected in our participants’ comments during our co-design sessions and usability evaluation.
Differences Between Clinical and Family Participants’ Design Requirements
While addressing user needs overall is critical, we must acknowledge some differences between clinicians’ and family participants’ perceptions of the risk communication tool in our usability testing. While clinicians were satisfied with the tool overall, their lower levels of satisfaction with information and interface quality (as evidenced by their lower PSSUQ scores for these components) may have reflected concerns about what information is presented; clinicians also felt less able to perform the tasks required of them (as evidenced by their lower NASA TLX performance scores). These differences may reflect clinicians’ concerns about the potential adverse effects of communicating risk to patients via the “nocebo effect” , which should be recognized and addressed, particularly in the pediatric setting. We incorporated this clinical guidance into the final iteration of our prototype to provide the risk information through a “comfort” lens; that is, we aimed to focus the efforts of the clinician and family team on “maintaining comfort” rather than “reducing pain,” while still acknowledging the risk of pain inherent in their surgical procedure. This approach is consistent with international initiatives aimed at prioritizing compassionate approaches to the recognition, prevention, and treatment of children’s pain (eg, ChildKind International [ ]) as well as our own institution’s recommended approach [ ]. It seems these changes may not have been sufficient to address the concerns of our clinical participants in usability testing. Importantly, however, other research has shown that, while increased risk has been correlated with patients wanting to discuss the procedure in more detail with their providers [ ], presenting patients with their personalized risk has not been associated with canceling procedures or changes in the decision to undergo surgery [ , , ].
Limitations and Future Work
We must acknowledge several limitations in our study. First, our study participants included no surgeons during the requirements gathering phase and only 1 surgeon in the usability testing phase; however, all clinician participants were anesthesiologists or nurses with expertise and regular involvement in predicting, diagnosing, treating, and offering families guidance on managing postoperative pain. Second, we did not include children or adolescents. Future work may involve evaluating the benefit of developing a child-centered version of the tool. At this stage, we adopted the strategy of developing a family-centered tool, but we aim to expand this idea to a tool targeted at adolescents. We have begun this process with a perioperative survey of our adolescent surgical population, which evaluates potential preoperative risk factors and postoperative recovery indicators. If successful, we can expand this work to younger children. Participants in this study echoed this aim: most agreed that children older than 12 years of age would benefit, and some felt we could consider including children between the ages of 5 and 11 years.
Third, our focus group sizes were relatively small (3-5 participants per session), typically due to the challenges of scheduling time with clinicians and family participants. While this could have avoided “group think,” it may have hindered collaborative idea generation. Furthermore, we did not use a dynamic prototyping tool in our co-design sessions with participants working collaboratively on a shared resource, which may have enhanced the design process; in pilot-testing, this had been found to be a barrier to participation. However, our patient partners suggested that screen sharing the exercises would increase accessibility as it may be challenging to teach participants of various education and language proficiencies to use a prototyping tool in real time or if they were using a small screen (eg, cell phone) and could not easily complete each exercise, or had a disability that might limit motor function and participation. Fourth, conducting co-design sessions over Zoom rather than in-person may have biased our sample. On the one hand, the requirement for an internet connection may have prevented participation by some families, which may have impacted the social equity of our findings and on the other hand, it may have facilitated participation for some people who would otherwise have been able to contribute due to travel or time constraints.
Finally, our usability evaluation provides only limited evidence of the tool’s readiness for implementation. Although its intended real-world use is as a shared clinician-family resource, we evaluated with individuals (either a clinician or family participant but not both at the same time), using a static prototype, with tasks that may have been too simple to evaluate the usability effectively. That said, the tool is not designed to be a complex, dynamic tool, but rather to communicate a fixed set of information personalized to each clinician-family presurgical encounter.
Future work will involve implementing the design in digital form, which will first be evaluated with a range of clinical scenarios and take account of the need for joint clinician-family evaluation. We are also collecting data in a separate study to generate a risk prediction model, which will supply the personalized risk predictions and proposed prehabilitation strategies that the tool is designed to present . A key point that should be addressed ideally before implementation is how widely the tool should be used in practice: in all cases, as our family participants suggested; only in cases with a high risk of pain; or where team mitigation strategies are expected to make a significant difference.
This user-centered co-design study identified essential requirements for a pediatric postoperative pain risk visualization tool to present risk severity descriptively and visually, to ensure that appearance and navigation are user-friendly, to frame risk identification and mitigation strategies in positive terms, to categorize and describe risks clearly, and to emphasize collaboration and effective communication. The usability of the resulting paper prototype was positively evaluated by both clinical and family participants suggesting that it is ready to be implemented as a digital prototype that can be tested in a clinical setting to establish its efficacy in supporting communication about postoperative pain risk.
This work was supported, in part, by a University of British Columbia Canada’s Digital Technology Supercluster expansion research grant and a BC Children’s Hospital Research Institute Evidence to Innovation Research Theme seed grant. MG holds a Michael Smith Health Research BC scholar award; MDW holds a Mitacs postdoctoral fellowship. The authors thank the participating families and BC Children’s Hospital staff for their time and insights. The authors would also like to thank Kim Correa for her help in conducting the initial co-design sessions and verifying transcripts and Rama Sreepada for assisting with recruitment and providing content expertise to study implementation. Finally, they would like to thank Amanda Ross-White (Queen’s University) for lending her librarian expertise to assist in searching and identifying literature relevant to study preparation. The authors thank the BC Children’s Hospital Foundation for its support.
MDW, CF, and YC participated in study design, data collection, qualitative analysis, prototype design, and paper drafting and editing. EPC, KCL, and SDW participated in study design, designing of the prototype, and paper drafting and editing. NCW and KC participated in the study design, paper drafting, and editing. MG participated in study design, data collection, designing of the prototype, and paper drafting and editing.
Conflicts of Interest
We disclosed our redesigned prototype for Intellectual Property protection to the University of British Columbia University-Industry Liaison Office (2022-07-29).
Low- and high-risk scenarios for role-play and "think aloud."PDF File (Adobe PDF File), 101 KB
Higher-resolution version of the risk communication tool prototype following the second co-design session.PDF File (Adobe PDF File), 31596 KB
- Rabbitts JA, Fisher E, Rosenbloom BN, Palermo TM. Prevalence and predictors of chronic postsurgical pain in children: a systematic review and meta-analysis. J Pain 2017;18(6):605-614 [https://linkinghub.elsevier.com/retrieve/pii/S1526-5900(17)30522-9] [CrossRef] [Medline]
- Williams G, Howard RF, Liossi C. Persistent postsurgical pain in children and young people: prediction, prevention, and management. Pain Rep 2017;2(5):e616 [https://europepmc.org/abstract/MED/29392231] [CrossRef] [Medline]
- Robillard JM, Bourne SC, Tam MT, Page PM, Lamb EA, Gogal C, et al. Improving the paediatric surgery patient experience: an 8-year analysis of narrative quality data. BMJ Open Qual 2020;9(2):e000924 [https://bmjopenquality.bmj.com/lookup/pmidlookup?view=long&pmid=32381597] [CrossRef] [Medline]
- Tam MT, Wu JM, Page PM, Lamb EA, Jordan I, Chambers CT, et al. Barriers and facilitators to effective pain management by parents after pediatric outpatient surgery. J Pediatr Health Care 2020;34(6):560-567 [CrossRef] [Medline]
- Feinstein AB, Sturgeon JA, Bhandari RP, Yoon IA, Ross AC, Huestis SE, et al. Risk and resilience in pediatric pain: the roles of parent and adolescent catastrophizing and acceptance. Clin J Pain 2018;34(12):1096-1105 [https://europepmc.org/abstract/MED/30028367] [CrossRef] [Medline]
- Busse JW, Heels-Ansdell D, Makosso-Kallyth S, Petrisor B, Jeray K, Tufescu T, et al. Fluid Lavage of Open Wounds Investigators. Patient coping and expectations predict recovery after major orthopaedic trauma. Br J Anaesth 2019 Jan;122(1):51-59 [https://linkinghub.elsevier.com/retrieve/pii/S0007-0912(18)30556-7] [CrossRef] [Medline]
- Pillinger NL, Robson JL, Kam P. Nutritional prehabilitation: physiological basis and clinical evidence. Anaesth Intensive Care 2018;46(5):453-462 [https://aaic.net.au/PMID/30189818] [CrossRef] [Medline]
- Minnella EM, Carli F. Prehabilitation and functional recovery for colorectal cancer patients. Eur J Surg Oncol 2018;44(7):919-926 [CrossRef] [Medline]
- Jensen BT, Laustsen S, Jensen JB, Borre M, Petersen AK. Exercise-based pre-habilitation is feasible and effective in radical cystectomy pathways-secondary results from a randomized controlled trial. Support Care Cancer 2016;24(8):3325-3331 [CrossRef] [Medline]
- Corr AM, Liu W, Bishop M, Pappo A, Srivastava DK, Neel M, et al. Feasibility and functional outcomes of children and adolescents undergoing preoperative chemotherapy prior to a limb-sparing procedure or amputation. Rehabil Oncol 2017;35(1):38-45 [https://europepmc.org/abstract/MED/28948112] [Medline]
- Ferreira V, Agnihotram RV, Bergdahl A, van Rooijen SJ, Awasthi R, Carli F, et al. Maximizing patient adherence to prehabilitation: what do the patients say? Support Care Cancer 2018 Aug;26(8):2717-2723 [CrossRef] [Medline]
- Brady MA, Carrington H, Theologis T. Prehabilitation in paediatric orthopaedic surgery reduces hospital length of stay. Clin Audit 2021;13:1-7 [https://www.dovepress.com/prehabilitation-in-paediatric-orthopaedic-surgery-reduces-hospital-len-peer-reviewed-fulltext-article-CA] [CrossRef]
- Fortier MA, Kain ZN. Treating perioperative anxiety and pain in children: a tailored and innovative approach. Paediatr Anaesth 2015;25(1):27-35 [https://europepmc.org/abstract/MED/25266082] [CrossRef] [Medline]
- Arulanandam B, Selvarajan A, Piche N, Sheldon S, Bloom R, Emil S, et al. Use of a risk communication survey to prioritize family-valued outcomes and communication preferences for children undergoing outpatient surgical procedures. J Pediatr Surg 2022;57(5):788-797 [CrossRef] [Medline]
- Wijeysundera DN. Predicting outcomes: is there utility in risk scores? Can J Anaesth 2016;63(2):148-158 [https://link.springer.com/article/10.1007/s12630-015-0537-2] [CrossRef] [Medline]
- Harris EP, MacDonald DB, Boland L, Boet S, Lalu MM, McIsaac DI. Personalized perioperative medicine: a scoping review of personalized assessment and communication of risk before surgery. Can J Anaesth 2019;66(9):1026-1037 [https://link.springer.com/article/10.1007/s12630-019-01432-6] [CrossRef] [Medline]
- Ancker JS, Senathirajah Y, Kukafka R, Starren JB. Design features of graphs in health risk communication: a systematic review. J Am Med Inform Assoc 2006;13(6):608-618 [https://europepmc.org/abstract/MED/16929039] [CrossRef] [Medline]
- Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q 2016;94(2):392-429 [https://europepmc.org/abstract/MED/27265562] [CrossRef] [Medline]
- Langley J, Wolstenholme D, Cooke J. 'Collective making' as knowledge mobilisation: the contribution of participatory design in the co-creation of knowledge in healthcare. BMC Health Serv Res 2018;18(1):585 [https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-018-3397-y] [CrossRef] [Medline]
- Carr EC, Babione JN, Marshall D. Translating research into practice through user-centered design: an application for osteoarthritis healthcare planning. Int J Med Inform 2017;104:31-37 [CrossRef] [Medline]
- Xu X, Griva K, Koh M, Lum E, Tan WS, Thng S, et al. Creating a smartphone app for caregivers of children with atopic dermatitis with caregivers, health care professionals, and digital health experts: participatory co-design. JMIR Mhealth Uhealth 2020;8(10):e16898 [https://mhealth.jmir.org/2020/10/e16898/] [CrossRef] [Medline]
- Garne K, Brødsgaard A, Zachariassen G, Clemensen J. Telemedicine in neonatal home care: identifying parental needs through participatory design. JMIR Res Protoc 2016;5(3):e100 [https://www.researchprotocols.org/2016/3/e100/] [CrossRef] [Medline]
- Steen M, Manschot M, De Koning N. Benefits of co-design in service design projects. Int J Des 2011;5(2):53-60 [http://www.ijdesign.org/index.php/IJDesign/article/view/890/346]
- Jagosh J, Macaulay AC, Pluye P, Salsberg J, Bush PL, Henderson J, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q 2012;90(2):311-346 [https://europepmc.org/abstract/MED/22709390] [CrossRef] [Medline]
- Slattery P, Saeri AK, Bragge P. Research co-design in health: a rapid overview of reviews. Health Res Policy Syst 2020;18(1):17 [https://health-policy-systems.biomedcentral.com/articles/10.1186/s12961-020-0528-9] [CrossRef] [Medline]
- Wood MD, Correa K, Ding P, Sreepada R, Loftsgard KC, Jordan I, et al. Identification of requirements for a postoperative pediatric pain risk communication tool: focus group study with clinicians and family members. JMIR Pediatr Parent 2022;5(3):e37353 [https://pediatrics.jmir.org/2022/3/e37353/] [CrossRef] [Medline]
- Lawrence CE, Dunkel L, McEver M, Israel T, Taylor R, Chiriboga G, et al. A REDCap-based model for electronic Consent (eConsent): moving toward a more personalized consent. J Clin Transl Sci 2020;4(4):345-353 [https://europepmc.org/abstract/MED/33244416] [CrossRef] [Medline]
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42(2):377-381 [https://linkinghub.elsevier.com/retrieve/pii/S1532-0464(08)00122-6] [CrossRef] [Medline]
- Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Meshkati N, Hancock PA, editors. Human Mental Workload. Amsterdam: Elsevier Science; 1988:139-183
- Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact 1995;7(1):57-78 [CrossRef]
- King N. Using templates in the thematic analysis of text. In: Cassell C, Symon G, editors. Essential Guide to Qualitative Methods in Organizational Research. London, United Kingdom: Sage Publications Ltd; 2004:256-270
- Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval 2016;27(2):237-246 [CrossRef]
- Vollstedt M, Rezat S. An introduction to grounded theory with a special focus on axial coding and the coding paradigm. In: Kaiser G, Presmeg N, editors. Compendium for Early Career Researchers in Mathematics Education. Cham: Springer International Publishing; 2019:81-100
- Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant 2018;52(4):1893-1907 [https://europepmc.org/abstract/MED/29937585] [CrossRef] [Medline]
- Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007;19(6):349-357 [https://academic.oup.com/intqhc/article/19/6/349/1791966?login=false] [CrossRef] [Medline]
- Sauro J, Lewis J. Standardized usability questionnaires. In: Sauro J, Lewis J, editors. Quantifying the User Experience, 2nd Edition. Burlington, MA: Morgan Kaufmann; 2016:185-248
- Brennan M, Puri S, Ozrazgat-Baslanti T, Feng Z, Ruppert M, Hashemighouchani H, et al. Comparing clinical judgment with the MySurgeryRisk algorithm for preoperative risk assessment: a pilot usability study. Surgery 2019;165(5):1035-1045 [https://europepmc.org/abstract/MED/30792011] [CrossRef] [Medline]
- Fritz BA, Pugazenthi S, Budelier TP, Pennington BRT, King CR, Avidan MS, et al. User-centered design of a machine learning dashboard for prediction of postoperative complications. Anesth Analg 2023 [CrossRef] [Medline]
- Bronsert MR, Lambert-Kerzner A, Henderson WG, Hammermeister KE, Atuanya C, Aasen DM, et al. The value of the "Surgical Risk Preoperative Assessment System" (SURPAS) in preoperative consultation for elective surgery: a pilot study. Patient Saf Surg 2020;14:31 [https://pssjournal.biomedcentral.com/articles/10.1186/s13037-020-00256-4] [CrossRef] [Medline]
- Hladkowicz E, Yachnin D, Boland L, Wilson K, McKinnon A, Hawrysh K, et al. Evaluation of a preoperative personalized risk communication tool: a prospective before-and-after study. Can J Anaesth 2020;67(12):1749-1760 [https://link.springer.com/article/10.1007/s12630-020-01809-y] [CrossRef] [Medline]
- Lambert-Kerzner AC, Aasen DM, Overbey DM, Damschroder LJ, Henderson WG, Hammermeister KE, et al. Use of the consolidated framework for implementation research to guide dissemination and implementation of new technologies in surgery. J Thorac Dis 2019;11(Suppl 4):S487-S499 [https://europepmc.org/abstract/MED/31032067] [CrossRef] [Medline]
- Gisladottir U, Nakikj D, Jhunjhunwala R, Panton J, Brat G, Gehlenborg N. Effective communication of personalized risks and patient preferences during surgical informed consent using data visualization: qualitative semistructured interview study with patients after surgery. JMIR Hum Factors 2022;9(2):e29118 [https://humanfactors.jmir.org/2022/2/e29118/] [CrossRef] [Medline]
- Wiesen BM, Bronsert MR, Aasen DM, Singh AB, Lambert-Kerzner A, Henderson WG, et al. Use of Surgical Risk Preoperative Assessment System (SURPAS) and patient satisfaction during informed consent for surgery. J Am Coll Surg 2020;230(6):1025-1033.e1 [CrossRef] [Medline]
- Raymond BL, Wanderer JP, Hawkins AT, Geiger TM, Ehrenfeld JM, Stokes JW, et al. Use of the American College of Surgeons National Surgical Quality Improvement Program surgical risk calculator during preoperative risk discussion: the patient perspective. Anesth Analg 2019;128(4):643-650 [CrossRef] [Medline]
- Hakone A, Harrison L, Ottley A, Winters N, Gutheil C, Han PKJ, et al. PROACT: iterative design of a patient-centered visualization for effective prostate cancer health risk communication. IEEE Trans Vis Comput Graph 2017;23(1):601-610 [CrossRef] [Medline]
- Gainer RA, Curran J, Buth KJ, David JG, Légaré JF, Hirsch GM. Toward optimal decision making among vulnerable patients referred for cardiac surgery: a qualitative analysis of patient and provider perspectives. Med Decis Making 2017;37(5):600-610 [CrossRef] [Medline]
- Lambert-Kerzner A, Ford KL, Hammermeister KE, Henderson WG, Bronsert MR, Meguid RA. Assessment of attitudes towards future implementation of the "Surgical Risk Preoperative Assessment System" (SURPAS) tool: a pilot survey among patients, surgeons, and hospital administrators. Patient Saf Surg 2018;12:12 [https://pssjournal.biomedcentral.com/articles/10.1186/s13037-018-0159-z] [CrossRef] [Medline]
- Arrow K, Burgoyne LL, Cyna AM. Implications of nocebo in anaesthesia care. Anaesthesia 2022;77(Suppl 1):11-20 [https://onlinelibrary.wiley.com/doi/10.1111/anae.15601] [CrossRef] [Medline]
- Childkind.: ChildKind International; 2022. URL: https://childkindinternational.org/ [accessed 2023-01-06]
- Pediatric pain guide for caregivers: comfort approaches. BC Children's Hospital and BC Women's Hospital & Health Centre. 2022. URL: https://cw-bc.libguides.com/c.php?g=718190&p=5202851 [accessed 2023-01-06]
- Wood MD, West NC, Sreepada RS, Loftsgard KC, Petersen L, Robillard JM, et al. Identifying risk factors, patient-reported experience and outcome measures, and data capture tools for an individualized pain prediction tool in pediatrics: focus group study. JMIR Perioper Med 2022;5(1):e42341 [https://periop.jmir.org/2022/1/e42341/] [CrossRef] [Medline]
|BCCH: BC Children’s Hospital|
|NASA: National Aeronautics and Space Administration|
|PSSUQ: Post-Study System Usability Questionnaire|
|REDCap: Research Electronic Data Capture|
|TLX: Task Load Index|
Edited by S Badawy; submitted 24.02.23; peer-reviewed by H Yang, D Nakikj; comments to author 06.06.23; revised version received 07.07.23; accepted 23.10.23; published 17.11.23Copyright
©Michael D Wood, Nicholas C West, Christina Fokkens, Ying Chen, Kent C Loftsgard, Krystal Cardinal, Simon D Whyte, Elodie Portales-Casamar, Matthias Görges. Originally published in JMIR Pediatrics and Parenting (https://pediatrics.jmir.org), 17.11.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.