During May and June 2020, all internet-based PEMs were downloaded from the pediatric associations’ websites. A total of 10 national associations were identified and are listed in Table 1 along with the number of unique PEMs obtained from each association. The downloaded PEMs included materials describing any topic with intended use by parents, guardians, or children on the pediatric websites. Therefore, this excluded any material intended for health care providers. PDF files were manually converted to plain text for further analysis. Text sections containing nonmedical information such as page numbers, disclaimers, tables, diagrams, phone numbers, emails, and webpage navigation were removed from each of the PEMs before analysis.

A readability assessment was performed on the PEMs using the software package Readability Studio Professional Edition (version 2019.3; Oleander Software Ltd). The readability scales used to determine the reading level, which was reported as a grade level, of the PEMs included 8 numerical scales and 2 graphical scales. The 8 readability scales included the Degrees of Reading Power–grade equivalent test (DRP-GE); Flesch-Kincaid Grade Level (FK); Simple Measure of Gobbledygook index (SMOG); Coleman-Liau index (CLI); Gunning Fog index (GF); New Fog Count (NFC); New Dale-Chall readability formula (NDC); and Ford, Caylor, Sticht scale (FORCAST). The 2 graphical scales included the Raygor Readability Estimate Graph (RREG) and the Fry Readability Graph (FRG). These 10 scales are often used when assessing medical text and offer externally validated measures of readability [31-33].

PEMs often contain text that must be modified before the analysis to appropriately apply the readability scales. This includes the removal of charts as well as the modification of bullet points to form complete sentences for analysis. To address the limitation of narrative-based readability scales, PEMs were individually edited to create high- and low-sentence documents, as performed by Perni et al [32]. For example, in high-sentence documents, each individual bullet point was treated as an independent sentence and resulted in a lower grade level estimate. On the other hand, low-sentence documents had each bullet point separated with a comma, with the final bullet point ending the sentence; this resulted in a higher grade level estimate [31,32]. The high- and low-level estimates were then averaged for further analysis. The associations’ readability level using the 8 numerical scales can be seen in Figure 1, and the 2 graphical scales can be seen in Figure 2.

A difficult word analysis was performed to identify the number and percentage of complex words (composed of 3 or more syllables), long words (composed of 6 or more letters), and unfamiliar words in each PEM according to the NDC criteria [34,35]. Once all the words were extracted from the PEMs, they were compared to the NDC word list as well as the New General Service List (NGSL). Words that appeared in either of the lists were removed and considered to be nonjargon words. All words that appeared in less than 3 PEMs were excluded from the analysis. Words with 3 or more syllables were then extracted, and the various tenses of the 10 most frequently identified words, where applicable, were combined. Alternative words were then proposed for any 3-syllable word that appeared in 3 or more PEMs, either using the Readability Studio Software, the Merriam-Webster thesaurus, or in consultation with a physician, to identify synonyms that can decrease the difficulty of the word.

Graphical data in Figure 1 were reported as the arithmetic means with the error bars representing the standard deviations. Data sets had their normality tested using a Shapiro-Wilk test when central limit theorem conditions were not met. Equal variance was tested using a Brown-Forsythe test to determine if the data would need to be transformed before analysis. Normally distributed data with equal variance then underwent a one-way analysis of variance (ANOVA). If the data were not normally distributed, then a nonparametric Kruskal-Wallis test was employed. Multiple comparisons tests, such as Tukey tests, were used to identify differences between sample means in the ANOVA analysis [36]. The data were analyzed using Graph Pad Prism (version 9; GraphPad Software Inc).

Following conversion to plain text, high- and low-sentence PEMs were subjected to 8 readability tests, including the DRP-GE, FK, SMOG, CLI, GF, NFC, NDC, and FORCAST. Figure 1 illustrates a summary of the results for the pediatric associations. The reading levels (reported as grade levels) measured by the 8 readability scales were averaged for each pediatric association, where the mean and standard deviations are reported as follows: Caring for Kids (8.8, SD 1.8), Pediatric Endocrine Group (9.5, SD 2.2), and the Atlantic Pediatric Society (13.1, SD 2.1). The overall mean was 9.1 (SD 2.4), with a grade-level range of 5 to 17. Note that Caring for Kids is a website developed by the Canadian Pediatric Society.

When the 8 readability scores of the individual PEMs were averaged, only 18 (7%) and 144 (55.8%) of the 258 PEMs, were below a seventh-grade and ninth-grade level, respectively. The RREG score of the high-sentence PEMs (Figure 2) ranges from a third-grade reading level to a grade level equivalent to that in university, with 26 (10.1%) and 127 (49.2%) of the 258 PEMs written at a grade level below 7 and 9, respectively. The FRG score of the high-sentence estimate, as seen in Figure 2, ranges from a third-grade to a 17th-grade (university-educated) reading level, with 14 (5.4%) and 118 (45.7%) of the 258 PEMs written at a grade level below 7 and 9, respectively.

The grade levels calculated by all 8 scales from the Atlantic Pediatric Society’s PEMs were also compared to those from Caring for Kids and the Pediatric Endocrine Group. Comparisons with 7 out of 8 reading tests were found to be statistically significant for both pediatric associations, where the NFC test was the only test to show no statistical significance (Table 2).

From the difficult word analysis, it was determined that of all the words found in the PEMs, on average, 13.3% (SD 5.3%) were complex words which contained 3 or more syllables, 31.9% (SD 6.1%) contained 6 or more letters, and 19.9% (SD 6.6%) were unfamiliar words. All PEMs collected had a target audience of caregivers or pediatric patients (described as patients between the ages of 0 and 19). The most frequent terms included cannabis, marijuana, medication(-s), calcium, cortisol, and hepatitis. Table 3 describes the most frequent difficult words in compliance with the criteria described in the methods section.

PEMs found on pediatric associations’ websites serve as an important link between health care professionals and caregivers. Through these web-based resources, parents can access reputable information endorsed by health care professionals to inform childcare practices on a day-to-day basis [1-3]. Although these resources are readily available with internet access, they are not always functionally accessible to all caregivers and patients. Pediatric PEMs have consistently been shown globally to be written at higher reading levels than recommended for a public audience, which is consistent with this study’s findings [9,15-20].

Based on the analyses using the DRP-GE, FK, SMOG, CLI, GF, NFC, NDC, and FORCAST scales, PEMs available on Canadian pediatric association websites were found to be written at a ninth-grade reading level (mean 9.1, SD 2.4) on average, wherein only 7% (18/258) of PEMs were written below the recommended seventh-grade reading level. Similar results were shown by the RREG and FRG (Figure 2), wherein only 5.4% (14/258) to 10.1% (26/258) of PEMs were found to be written below a seventh-grade reading level. This suggests that the PEMs cannot be easily understood by most Canadians, and even less so by pediatric patients. This is particularly true for the Atlantic Pediatric Society’s PEMs, which are written at a university reading level (mean 13.1, SD 2.1). In Table 2, it can be seen that the Atlantic Pediatric Society’s PEMs were statistically significantly different from other pediatric associations’ PEMs in a majority of the readability tests employed. This suggests that the Atlantic Pediatric Society should consider all parameters used in each readability test, such as word and sentence length, the number of syllables in each word, and the familiarity of the words used, to improve readability. In addition to reducing the reading level of the text directed to caregivers, pediatric associations should consider stratifying their websites to include simpler educational materials directed to children.

Although the study itself uses 8 numerical and 2 graphical readability indices to better represent the many parameters that are factored into readability, special emphasis should be given toward readability formulas designed for health care materials. This includes the SMOG and FRG, with the SMOG considered as the gold standard by many large institutes such as the National Cancer Institute [37]. When factoring this into account, the average reading level of the PEMs would be closer to an 11th-grade rather than a ninth-grade reading level. Although this study focused on PEMs derived from only Canadian pediatric hospitals and associations, patients retrieve information from a variety of other sources. Health content obtained from common sources, such as Wikipedia and other popular websites containing health information, has also been shown to be written at a reading level far above the seventh grade [38-40]. These findings suggest that even if pediatric patients use education materials outside of those analyzed in this study, they may still face challenges in identifying information that is at an appropriate reading level.

The difficult word analysis revealed long words were the most common type of difficult word in the PEMs. This is in line with another recent Canadian publication on cancer-related PEMs [34]. Difficult terminology should be replaced with more familiar alternatives whenever possible to improve the ease of understanding. For example, “beneficial” could be replaced with “helpful,” and “clinician” could be replaced with “doctor” for simplicity. Although the substitution of terminology is ideal, it may not be applicable to cases in which information must be fully and accurately communicated. In these situations, a clear definition should be included when the word is introduced. Although the content of the PEMs was analyzed for reading level and word difficulty in this study, additional factors such as organization, layout, and design can impact PEMs’ readability. Therefore, further studies should be undertaken once additional instruments, such as the Suitability Assessment of Materials and PMOSE/IKIRSCH document readability formula, are validated for medical literature [41,42].

The readability tests used in this study consider parameters such as word and sentence length, number of syllables per word, words per sentence, and the difficulty level of words [31-33]. Although the use of multiple tests allows for more dimensions of readability to be considered, there are still limitations to using readability tests overall. Tests that assess syllable count may overestimate the readability of the text. Monosyllabic medical terminology, such as the word “stent,” contributes to a lower readability score, but that may not necessarily reflect a person’s ability to understand the terminology [15,31]. Conversely, tests that assess word familiarity may underestimate the readability of the text. Well-known medical terminology, such as the word “pediatrician,” acts to increase readability scores but may not contribute to increased difficulty in understanding the text. Furthermore, the act of defining difficult words within the text, which would greatly improve comprehension, is also not considered to impact readability. Additionally, the readability tests in this study do not account for the formatting of the text or the inclusion of diagrams. Although bullet points were analyzed as both sentences and comma-separated phrases, this does not fully capture the improvement to readability that lists provide. The evaluation of communication tools such as tables and images should be considered for future studies, as they can also serve to improve the ease of understanding [12]. Lastly, interpretations of the results must be taken into context as only Canadian pediatric associations were assessed, with just 3 of the 10 associations having PEMs on their websites. The results are therefore not representative of the totality of the information that caregivers and pediatric patients would be exposed to.

Overall, web-based PEMs developed by Canadian pediatric associations exceed the recommended seventh-grade reading level. Difficult words should be replaced when possible or defined, and educational content directed specifically toward pediatric patients should be included. Additional consideration should be placed on the incorporation of multimedia PEMs [43]. Qualitative studies should be conducted in the future to better understand caregiver and provider information needs, as well as the barriers toward implementing more functionally accessible PEMs on pediatric association websites [44-46]. Additionally, the quality of the PEMs should be evaluated to determine if the information provided is accurate [47-49]. Once collected, this data can be used to inform changes that improve the usefulness, quality, and accessibility of pediatric PEMs. As the role of technology in health care increases, it is important that all individuals are able to understand and use reputable resources on the internet.