An emergency department (ED) visit is often the first point of contact for patients with a health care institution and thus a showcase of its efficiency. Providing patients with a positive experience should take high priority [1] and is one of the fundamental determinants of health care quality [2]. In a recent meta-synthesis, a study by Graham et al [1] conceptualized a model to understand the most commonly identified drivers of the ED patient experience. These included interpersonal and informational communication, patients’ expectations and empowerment, recognition of emotional needs, actual and perceived waiting times, competent care, and physical and environmental needs [1]. A similar conceptual framework was also developed by Sonis et al [3,4]. The same drivers have been observed in other studies focusing on identifying the determinants of patient and family experience in the pediatric EDs [5-11]. This highlights the essential nature of these drivers and the attention that should be paid to them when implementing an intervention to improve the adult or pediatric ED patient experience and ED efficiency. Several recent reviews have demonstrated a strong correlation between a positive ED patient experience and a range of benefits at the individual and institutional levels. These include increased therapeutic compliance [12]; improved health clinical outcomes [1,13,14]; outpatient [15], inpatient [16], and staff satisfaction [12]; reduced complaints and medicolegal risks [17]; institutional profitability and reputation in the community [12,18,19]; and other health care system goals [13].

Unfortunately, the hectic, unpredictable, crowded, demanding, and time-pressured environment of the ED may adversely affect patient experience [13]. In particular, there is strong pressure from public and institutional leaders to alleviate overcrowding and long waiting times experienced in the ED [20]. Overcrowding because of nonurgent visits negatively impacts the quality of care and patient safety (prolonged waiting times, delays in diagnosis and treatment, delays in treating seriously ill patients, and medication errors). It also affects the costs of care and patient experience. For hospitals, crowding results in loss of revenue because of patients leaving the ED without being seen, diversion of EDs secondary to patient dissatisfaction, and shifting of the market share to competitors [21]. Moreover, overcrowding exposes ED staff to stressful and unpredictable work-related events, resulting in decreased productivity and increased turnover [22,23].

The body of literature assessing conventional intervention strategies aimed at improving these specific ED issues is highly heterogeneous [24-34]. Proposed interventions vary widely and often require major structural or organizational changes that are not necessarily easily scalable to all hospitals. Importantly, a few address the aforementioned drivers of the ED experience in a scoping and integrative manner along the entire patient journey. Successfully addressing these dimensions requires enlisting patients and families as allies in designing, implementing, and evaluating care systems through patient-centered care approaches [35]. One solution to the serious challenges facing the ED today may be found in information technologies, which have the potential to both reduce institutional burdens and improve patients’ experience [36]. Supported by the rapid spread of mobile devices in the community and their innovative features (eg, versatile connectivity, on-board computing and communication capabilities, privacy, and small size), mobile apps may provide such a solution within the easy reach of end users. However, to date, there is a lack of studies on the potential use of mobile apps to individually support the entire emergency care journey. On the basis of this finding and guided by the principles of patient- and family-centered care [5,35], we developed InfoKids [37], an integrated eHealth solution composed of 3 modules connecting patients, caregivers, and administrative clerks through a web and mobile app, with the aim of supporting the entire emergency care process, thus facilitating caregiving and administrative work and streamlining the arrival of patients in the ED [38]. This system is freely available at Geneva University Hospitals, Geneva, Switzerland, for pediatric patients. It is expected to be soon redesigned to cover the entire population seeking ED care (ie, adult, geriatric, and gynecologic) in a service area of more than 1 million individuals. Before scaling up this app to such a large population, an essential step in determining the potential for the success of this patient-centered eHealth intervention was to assess its capacity to meet end users’ needs and improve health care at our institution before clinical effectiveness testing [39-42].

This study aims to evaluate the usability of the InfoKids mobile app to support the entire patient’s ED journey through quantitative and qualitative usability metrics in a laboratory setting. We then aim to identify potential problems related to its use and formulate mitigation measures to inform both the development of its upcoming version as a universal app for all ED outpatient consultations in our hospitals and future mobile app development in this medical field by other research groups.

The usability of the app was assessed through a scenario-based, summative evaluation of human-computer interactions using a mixed methods approach [43]. Multitask quantitative and qualitative usability metrics were used and are described in detail in subsequent sections.

Usability is defined as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [44]. Usability of a mobile app can be measured by the completeness and successfulness whereby users solve specified tasks centered around the main features of the app. Conversely, systems with poor usability can lead to low goal achievement efficiency or technology not being used [45].

The study was conducted in a medical informatics usability laboratory room at Geneva University Hospitals to standardize the intervention and technically facilitate measurements. The evaluation framework was a user-task-system interaction, deliberately omitting the user’s real environments [43]. Tasks were performed on an LG G5 mobile phone with a 5.3-inch screen size at a resolution of 2560×1440 pixels and an Android operating system V7. According to recommendations on the minimum sample size required to conduct a summative evaluation, at least 15 participants were planned to be recruited [46]. Participants were recruited through advertisements posted on Facebook groups and displayed at the Geneva University Medical Center. Participation was open to adults with children of pediatric age (0-16 years). Exclusion criteria were non–French-speaking persons and those who had previously used the app.

Participants were invited by emails to individual sessions. The study procedure was explained to the participants upon arrival at the evaluation laboratory. Written informed consent was obtained from all the participants. After completing a baseline questionnaire on demographics and user experience with smartphones, the participants were asked to imagine themselves in a situation where they had heard about the InfoKids app and to follow a scripted and timed standardized scenario. The scenario was developed to sequentially guide the user toward the completion of 9 goal-oriented tasks covering the main functionalities of the app (Textbox 1; Multimedia Appendix 1). The sequence of tasks reflected the sequence of actions that parents seeking medical advice for their sick child with worrying symptoms at home would have to perform. For reasons related to the study design and use of the app, the possibility of patients being able to temporarily leave the ED while waiting for a scheduled consultation was not evaluated but will be the subject of further research. For greater realism, the dates and times were adapted to the time of the experiment. No training on the app was offered before the evaluation began to avoid preparation bias. The participants were not given any assistance to complete the tasks. Study investigators only intervened to encourage participants to keep talking during the intervention, thus avoiding bias of results and minimizing any disruption of participants’ thoughts. The participants were informed that their interaction with the app and their verbal exchanges would be video recorded.

To understand participants’ thoughts, the concurrent think-aloud method was applied by asking them to verbalize during task completion [55]. Upon completion of the scenario, the moderator had a debriefing with each participant following a semistructured grid interview, with the aim of assessing overall experience with the tool and usability improvements and perform a retrospective think-aloud method to analyze difficulties encountered and understand their causes [56]. Finally, to assess user satisfaction, the participants were asked to complete the System Usability Scale (SUS) questionnaire [57,58].

A pediatric emergency physician (JNS) wrote a credible and standardized scenario based on these tasks, which was then screened and approved by two ergonomists (JR and AR) at the evaluation laboratory. In the scenario, the participant decides to install the app in the eventuality that an ED visit might be necessary. Shortly after, the participant (ie, the parent) needs to use the app for the first time following the onset of cough in their child. A week later, when the cough and the child’s health had deteriorated, the parent had to use the app again to be guided and supported to go to the ED with the child.

Participants’ task performance was video-recorded and audio-recorded to retrospectively analyze the usability of the app. Video and audio captures were acquired with an Elmo L-l2iD camera document placed above the phone. Morae software (TechSmith Corporation) was used to analyze the video and audio recordings of participants’ interactions with the app. Subsequently, the recordings and usability metrics were transcribed onto Microsoft Excel spreadsheets. The SUS paper questionnaires were collected immediately after the intervention and subsequently transcribed onto Microsoft Excel spreadsheets. Two researchers (JR and VGR) analyzed the success rates of each task and their duration independently of each other. In case of disagreement, both researchers discussed together to reach a consensus. All data collected were anonymized.

Descriptive statistics were used to summarize continuous measures at a significance level of .05. Frequency counts were used for summarizing categorical measures. Age categories and SUS mean scores were compared to make comparisons between user characteristics and satisfaction. Data were analyzed, and graph figures were created with GraphPad Prism 9 and Microsoft Excel.

The study was submitted to the Regional Research Ethics Committee (Req-2021-00505), which waived the need for further evaluation by issuing a no objection statement, as such projects did not fall within the scope of the Swiss federal law on human research [66]. Only data from a fictitious patient were used in this study. Written informed consent was obtained from all participants before the intervention. No participants’ medical information was used. Participants were not identifiable on video and audio recordings. Participants’ data and results obtained through the intervention were deidentified and assigned an individual identifying code that did not contain identifying information. Data were secured by protected access passwords at Geneva University Hospitals on secured hard disks. This study was conducted in accordance with the Declaration of Helsinki [50] and principles of Good Clinical Practice [51].

Between June and September 2017, a total of 17 participants participated in the study. Baseline demographic characteristics are shown in Table 2.

The mean overall time on task for all tasks was 101.26 (SD 44.07) seconds. Tasks 1, 2, and 4a had a higher time on task than the other tasks (Table 3). These findings showed that the most complicated task (ie, task 4a) was the third most time-consuming task, although it did not require much action compared with tasks 1 and 2, which were the longest and required several pieces of data to be entered into the app, thus explaining their duration.

Multimedia Appendices 8 and 9 show the TBE and ORE for every single performed task, respectively. Multimedia Appendix 8 shows that tasks 1, 2, and 4a had the shortest TBE. Therefore, creating the parental account and the child’s profile was not the most efficient task. Task 4a showed the lowest efficiency, with the shortest TBE (0.0065 tasks per second) and lowest ORE (50.2%).

The mean overall SUS score was 80.88 (SD 8.57; Table 4). This shows that the usability of the InfoKids app was perceived as good to excellent [68] (Figure 7). The detailed scores indicate that of the 17 participants, 4 (24%) assessed the app as fair, 5 (29%) as good, and 8 (47%) as excellent. Mean SUS scores were similar when analyzed by two age categories, ≤40 years (mean 82.14, SD 9.94 years) and >40 years (mean 80, SD 8.42 years; Mann–Whitney U test=29.5; P=.60).

In this study, we report an overall good-to-excellent perceived usability of a patient-centered mHealth app aimed at covering the entire emergency care process by supporting patients before, during, and after an ED visit. Given the high percentage of patient-centered assigned tasks that participants successfully completed, we observed a good overall rate of understanding of how the app worked. Participants found most of the features useful, particularly the recommendations provided according to their child’s symptoms, access to information related to waiting times and the diagnosis made in the ED, and ability to inform the ED upon their arrival. However, the ergonomic evaluation identified 81 occurrences of 14 usability problems, of which 50% (7/14) were serious, as their severity ratings were either major or catastrophic. These results indicated areas for app improvements. From participants’ and ergonomists’ suggested usability improvements, mitigation measures were listed to further improve the app and avoid barriers to its adoption (Table 8).

Apps’ attrition has emerged as an area of particular concern in recent literature on new technological innovations [69,70]. Even when apps are evidence based, this does not guarantee that they will be used consistently over time. Similar to other health information technologies, the benefits of apps can only be achieved if end users intend to adopt them [71]. Poor usability and a lack of user-centered design have been described as 2 drivers for low adoption rates of mobile apps [45]. Although usability has been identified as a key component of good practice in the development of digital apps [72], only a small fraction of medical apps publish their usability evaluation results, despite their growing number [42]. The main concerns of these apps are health conditions or diseases such as mental health [45,73], cancer [74,75], nutrition [76], diabetes [77,78], chronic disease self-management [79,80], and child health [81-86], among others [42].

However, there is no app that addresses more broadly patients’ accompaniment throughout their entire ED care journey (ie, before, during, and after their visits), as well as providing personalized health information and support to manage illness or trauma. We found only 2 studies describing the usability evaluation of prototype app versions providing a personalized treatment schedule and an indoor navigation service for outpatients [87,88]. Moreover, both apps seem to be limited to this sole in-hospital purpose, without patient-centered information regarding their disease, and restricted to Android operating software systems. A study by Westphal et al [89] described a very promising web-based system for providing real-time information to ED patients regarding the procedures that they may encounter during their journey. However, similar to the previous 2 studies, this system focused only on the patient’s journey within the hospital and did not address the patient’s experience over the entire course of care.

The InfoKids app aims to bridge these gaps. Importantly, it is intended for wider use within our institution. Through the current iterative processes of development and evaluation, it is intended to soon become a more universal tool to connect the whole population seeking ED care (ie, adults, geriatric, and gynecologic) in a service area of more than 1 million people. In this sense, this study contributes to this iterative development process. Given its interconnection with the hospital’s computerized system, this app has the potential to ensure better coordination, continuity, and transition of care, thus improving both the patient experience and hospital efficiency.

This study had several strengths. To our knowledge, this was the first report of findings of the usability evaluation of an app supporting the longitudinal patient care transition from home up to ED discharge. Second, the mixed methods approach used in combination with different types of usability methods was another strength already identified by studies recognizing the utility of using qualitative and quantitative approaches for app usability testing [72,90]. Third, the 9 goal-oriented tasks assessed were centered around the main features of the app. The fact that users can perform a set of tasks centered around these features that are representative of those that users would normally use in clinical care was identified as a good way to determine the usability of the app and its features and workflow [91]. Fourth, this study added to the literature that recommends more usability studies focused on patient-centered apps [72,91-97]. It also contributed to the effort to publish usability studies based on academic development and patient-centered care, rather than a purely commercial development approach [42].

Our study had some limitations. First, we used an artificial laboratory environment, which has a low degree of fidelity. As a result, the generalizability and transferability of the results may be limited in real-life settings. Furthermore, the results obtained were based on an assessment of usability with participants who were naïve to its use. Therefore, it can be assumed that in-depth use of the app beforehand could have improved the perception of its usability among people who had used it before and avoided certain problems of comprehension and navigation. Interestingly, the use of a tutorial that was supposed to correct these problems seems to have been a source of difficulties for users in itself, if only to find it in the app. Therefore, it might be judicious in future versions to replace it with an interface offering contextual help on each page, rather than a long tutorial to memorize or search for. These assumptions should be addressed in future studies. Another limitation is that the small sample of 17 users might not have been sufficiently large to reveal all usability issues. However, it was assumed that 5 users are already a sufficient sample to reveal 85% of usability problems, whereas 15 users were sufficient to uncover almost 97% of problems [98]. In contrast, the fact that only one scenario was proposed to users in an arbitrary order set by the investigators raises concerns about the applicability of the results to any other clinical situations or navigational pattern in the app. This scenario was chosen to test most of the functions of the app according to a logical workflow model that parents wishing to consult with their child in the emergency room would follow. However, it cannot be excluded that other scenarios could have generated other navigation schemes and usability problems or facilitation. For example, if task 4a (evaluated as the most complicated task) had not been interposed between the choice of symptoms (task 3) and the announcement of departure to the ED (task 5) in this scenario, it is possible that no navigational problems would have occurred. It might be interesting in a future study to test the usability of the app based only on several standardized scenarios without predefined tasks. Instead, tasks and navigation would be left to users’ discretion, as in real life. Finally, as the InfoKids app is intended to be used in case of emergency (or at least perceived as such by parents), the quiet and nonstressful laboratory environment used in the study may appear to be a limitation. Guidelines for conducting usability testing recommend establishing a calm and relaxed atmosphere in which users can work without feeling stressed [99-101], although stress in usability testing has rarely been studied so far. One of the few existing studies by Janneck and Dogan [99] compared a usability test performed in a laboratory under calm and relaxed conditions with a test situation in which several stressors (time pressure, noise, and social pressure) were applied. They observed that participants under stressful conditions demonstrated poorer performance in the execution and accuracy of tasks and rated the usability and user experience of the software much more negatively. However, it should be noted that although various situations tend to elicit different patterns of stress responses, there are also individual differences in perceived and behavioral stress responses to the same situation [102]. Indeed, future research assessing the impact of stressors on the usability of InfoKids would provide valuable input for future development in the adult setting.

The usability of mHealth apps is an important factor for their adoption and use. This study addresses a gap in the literature by reporting findings from a usability evaluation relevant to a patient-centered mobile app designed to support the entire emergency care process by assisting patients before, during, and after an ED visit. Our results show that the usability of the current version of InfoKids is rated as good to excellent by users. However, areas for app improvement are identified and mitigation measures are proposed. These usability problems will be addressed in updated releases of InfoKids and will be used to inform the development of its next version as a universal app for all patients seeking ED care. The next step would be to determine whether this mobile app benefits ED patient experience and ED efficiency in a real-life patient environment and clinical conditions. Given the paucity of research in this area, we conclude that our findings could also be useful in paving the way for further research on mobile apps aimed at supporting and accompanying patients in their care episodes.