This paper is in the following e-collection/theme issue:

Developmental Problems (39) Research Instruments, Questionnaires, and Tools (540) Autism Spectrum Disorder (ASD) (96) New Diagnostic Tools and Instruments for Pediatrics (19)

Published on in Vol 4, No 1 (2021): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/20172, first published .
Interformat Reliability of Web-Based Parent-Rated Questionnaires for Assessing Neurodevelopmental Disorders Among Preschoolers: Cross-sectional Community Study

Interformat Reliability of Web-Based Parent-Rated Questionnaires for Assessing Neurodevelopmental Disorders Among Preschoolers: Cross-sectional Community Study

Interformat Reliability of Web-Based Parent-Rated Questionnaires for Assessing Neurodevelopmental Disorders Among Preschoolers: Cross-sectional Community Study

Authors of this article:

Masanori Tanaka1 Author Orcid Image ;   Manabu Saito2 Author Orcid Image ;   Michio Takahashi3, 4 Author Orcid Image ;   Masaki Adachi3, 4 Author Orcid Image ;   Kazuhiko Nakamura2, 4 Author Orcid Image
Article Authors Cited by (1) Tweetations (5) Metrics

    Original Paper

    1Faculty of Business Administration, Hokkai-Gakuen University, Sapporo, Hokkaido, Japan

    2Department of Neuropsychiatry, Graduate School of Medicine, Hirosaki University, Hirosaki, Aomori, Japan

    3Graduate School of Health Sciences, Hirosaki University, Hirosaki, Aomori, Japan

    4Research Center for Child Mental Development, Graduate School of Medicine, Hirosaki University, Hirosaki, Aomori, Japan

    Corresponding Author:

    Masanori Tanaka, PhD

    Faculty of Business Administration

    Hokkai-Gakuen University

    4-1-40, Asahi-machi, Toyohira

    Sapporo, Hokkaido, 062-8605

    Japan

    Phone: 81 11 841 1161 ext 2302

    Email: m-tnk@hgu.jp


    Background: Early detection and intervention for neurodevelopmental disorders are effective. Several types of paper questionnaires have been developed to assess these conditions in early childhood; however, the psychometric equivalence between the web-based and the paper versions of these questionnaires is unknown.

    Objective: This study examined the interformat reliability of the web-based parent-rated version of the Autism Spectrum Screening Questionnaire (ASSQ), Attention-Deficit/Hyperactivity Disorder Rating Scale (ADHD-RS), Developmental Coordination Disorder Questionnaire 2007 (DCDQ), and Strengths and Difficulties Questionnaire (SDQ) among Japanese preschoolers in a community developmental health check-up setting.

    Methods: A set of paper-based questionnaires were distributed for voluntary completion to parents of children aged 5 years. The package of the paper format questionnaires included the ASSQ, ADHD-RS, DCDQ, parent-reported SDQ (P-SDQ), and several additional demographic questions. Responses were received from 508 parents of children who agreed to participate in the study. After 3 months, 300 parents, who were among the initial responders, were randomly selected and asked to complete the web-based versions of these questionnaires. A total of 140 parents replied to the web-based format and were included as a final sample in this study.

    Results: We obtained the McDonald ω coefficients for both the web-based and paper formats of the ASSQ (web-based: ω=.90; paper: ω=.86), ADHD-RS total and subscales (web-based: ω=.88-.94; paper: ω=.87-.93), DCDQ total and subscales (web-based: ω=.82-.94; paper: ω=.74-.92), and P-SDQ total and subscales (web-based: ω=.55-.81; paper: ω=.52-.80). The intraclass correlation coefficients between the web-based and paper formats were all significant at the 99.9% confidence level: ASSQ (r=0.66, P<.001); ADHD-RS total and subscales (r=0.66-0.74, P<.001); DCDQ total and subscales (r=0.66-0.71, P<.001); P-SDQ Total Difficulties and subscales (r=0.55-0.73, P<.001). There were no significant differences between the web-based and paper formats for total mean score of the ASSQ (P=.76), total (P=.12) and subscale (P=.11-.47) mean scores of DCDQ, and the P-SDQ Total Difficulties mean score (P=.20) and mean subscale scores (P=.28-.79). Although significant differences were found between the web-based and paper formats for mean ADHD-RS scores (total: t132=2.83, P=.005; Inattention subscale: t133=2.15, P=.03; Hyperactivity/Impulsivity subscale: t133=3.21, P=.002), the effect sizes were small (Cohen d=0.18-0.22).

    Conclusions: These results suggest that the web-based versions of the ASSQ, ADHD-RS, DCDQ, and P-SDQ were equivalent, with the same level of internal consistency and intrarater reliability as the paper versions, indicating the applicability of the web-based versions of these questionnaires for assessing neurodevelopmental disorders.

    JMIR Pediatr Parent 2021;4(1):e20172

    doi:10.2196/20172

    Keywords

    neurodevelopmental disorders; web-based questionnaire; preschoolers; parents; interformat reliability 


    In the American Psychiatric Association's Diagnostic and Statistical Manual for Mental Disorders, Fifth Edition (DSM-5), neurodevelopmental disorders are identified in the early developmental stages and are characterized by developmental deficits that lead to impairments in personal, social, academic, and vocational functioning [1]. Representative examples of such disorders include autism spectrum disorder, attention-deficit/hyperactivity disorder, and developmental coordination disorder. The core characteristics of autism spectrum disorder include 2 main dimensions—social communication and restricted, repetitive sensory–motor behaviors—that are irrespective of culture, race, ethnicity, or socioeconomic group [2]. Estimates of the total-population prevalence of autism spectrum disorder range from 2.2% to 3.2% [3,4]. The hallmarks of attention-deficit/hyperactivity disorder are developmentally impaired attention, motor hyperactivity, impulsivity, and the difficulties associated with them [5]. A recent meta-analysis [6] revealed that the estimated prevalence of children with attention-deficit/hyperactivity disorder is 3.4% in the general population. Developmental coordination disorder is characterized by marked impairment in the acquisition and execution of motor skills. This impairment significantly and sustainably interferes with activities of daily living, including academic achievement [1]. A recent review [7] reported that prevalence estimates for developmental coordination disorder among children range from 2% to 20%, with 5% to 6% being the most commonly reported prevalence rate.

    It is known that children with these conditions not only have various secondary mental health problems [8-10] but also experience maladjustment in adulthood [11-13]. Since a number of studies [7,14,15] have reported that early detection and intervention for neurodevelopmental disorders is effective, it is necessary to develop useful screening tools for assessing these conditions in early childhood.

    Several questionnaires have been developed to assess a variety of neurodevelopmental disorders. The Autism Spectrum Screening Questionnaire (ASSQ) was developed to screen for autism spectrum disorder in school-age children based on their parents’ or primary caregivers’ ratings [16] and has been shown to be highly accurate in screening for autism spectrum disorder [17]. Additional research has confirmed that the ASSQ has good reliability and validity and can be applied to preschool-age children, as well [18,19]. The ADHD-Rating Scale (ADHD-RS) is one of the most widely used questionnaires developed to assess ADHD symptoms in children age 5 to 18 years [20]. There are 2 versions of the ADHD-RS: the home form is rated by parents or primary caregivers of the children, and the school form is rated by teachers. It has been demonstrated that the ADHD-RS has good reliability and validity in preschool children [21]. Moreover, previous research has confirmed that the ADHD-RS shows higher sensitivity and specificity in parent ratings than those in teacher ratings among preschoolers [22]. The Developmental Coordination Disorder Questionnaire 2007 (DCDQ) was developed to identify children age 5 to 15 years who are at risk for developmental coordination disorder, based on parents’ ratings [23]. It has been found that the DCDQ has good psychometric properties and has been recommended for use in clinical practice as supplemental information for the diagnosis of children with developmental coordination disorder [7]. It is known that children with neurodevelopmental disorders have behavioral and emotional difficulties. The Strengths and Difficulties Questionnaire (SDQ) is a brief behavioral screening questionnaire about externalizing and internalizing problems in children [24]. It has been reported that using parent ratings for the SDQ has satisfactory reliability and validity in a community sample of 5- to 15-year-old children [25].

    These questionnaires have been developed as paper-and-pencil type questionnaires and can be useful in individual clinical settings; however, to screen large populations in a local community for early detection of neurodevelopmental disorders, web-based versions are more efficient than the paper-and-pencil version because of the significant time and effort required to distribute and collect paper-and-pencil questionnaires. Not only can costs be saved, but also there are additional advantages; for example, we can automate the process of manual data entry after completing data collection [26]. In addition, using web-based questionnaires to collect data generally improves the quality of the data because the validation checks can incorporate prompts that alert respondents if they enter incorrect or incomplete answers [27]. Furthermore, web-based instruments expand the reach of assessments, which is particularly important under pandemic conditions when it is difficult or impossible to administer in-person assessments [28,29].

    On one hand, several studies [30-32] have reported psychometric equivalence between web-based and paper-and-pencil versions of the questionnaires used to assess various psychological disorders; on the other hand, some studies have revealed psychometrically significant differences between the 2 formats [26,33,34]. Therefore, it is necessary to evaluate the comparability of web-based and paper-and-pencil versions of the questionnaires [35]. In particular, high interformat reliability, meaning the level of equality between different delivery formats, indicates that the psychometric properties of the instrument are independent of the delivery format [36]. However, to our knowledge, no studies have assessed the interformat reliability of web-based questionnaires that aim to assess neurodevelopmental disorders.

    We aimed to examine the interformat reliability of the web-based versions of the ASSQ, ADHD-RS, DCDQ, and SDQ. Based on previous work [36], we confirmed interformat reliability from the following 3 perspectives. First, we verified the internal consistency of the web-based and paper-and-pencil formats of each questionnaire. Second, we examined the intraclass correlations between the 2 formats of each questionnaire to test intrarater reliability. Third, we investigated the mean score differences between the 2 formats of each questionnaire to confirm equivalence in quality.


    Participants

    This study was conducted as part of the Hirosaki Five-Year-Old Children Developmental Health Check-up Study (HFC Study), a large community-based cohort study initiated in 2013 that examined the impact of children's neurodevelopmental disorders and lifestyle habits on their adaptation and emotional and behavioral problems at age 5 years. Located in Aomori Prefecture in the northeastern part of Japan, Hirosaki City has approximately 175,000 residents, 1 university, and several colleges, and its main industry is agriculture.

    Participants in this study were recruited in July 2018. The local government of Hirosaki City distributed a set of paper-based questionnaires for voluntary completion to the parents of 620 5-year-old children in the city via the municipal health center. The package included the ASSQ, ADHD-RS, DCDQ, parent-reported SDQ (P-SDQ), and demographic questions. Responses were received from 508 parents who agreed to participate in the study. After 3 months, 300 of the 508 respondents were randomly selected and informed of the objective of this study. The individuals who gave their written consent to participate were asked to complete web-based versions of these questionnaires. Participants were given an ID and password to complete the web-based survey on their own computers. There were no restrictions on the type of computer (eg, personal computer, tablet, or smartphone) that they could use to complete the survey. A total of 140 parents replied to the web-based format and were included in the final sample in the present study. Table 1 shows the demographic characteristics of this sample; Multimedia Appendix 1 contains the characteristics of the 368 people who did not respond to the web-based survey.

    Table 1. Participants’ demographic characteristics.
    CharacteristicsValue (N=140), n (%)
    Children’s gender

    		Boy78 (55.7)

    		Girl62 (44.3)
    Children’s age (months)

    		603 (2.1)

    		6125 (17.9)

    		6221 (15.0)

    		6325 (17.9)

    		6419 (13.6)

    		6532 (22.9)

    		6615 (10.7)
    Respondent

    		Mother127 (90.7)

    		Father13 (9.3)
    Childcare during daytime

    		Nursery school115 (82.1)

    		Kindergarten24 (17.1)

    		Mother1 (0.7)
    Household income (JPYa)

    		<2 million10 (7.1)

    		2-4 million44 (31.4)

    		4-7 million56 (40.0)

    		7-10 million18 (12.9)

    		>10 million7 (5.0)

    		Don’t know5 (3.6)

    aJPY: Japanese Yen; an approximate exchange rate of US $1= 103.80 JPY.

    Measures

    ASSQ

    The ASSQ has 27 items that assess autistic features such as social interaction and communication problems, behaviors that are restrictive and repetitive, motor clumsiness, and other associated symptoms, including motor and vocal tics [16,17]. The items are rated on a 3-point scale ranging from 0 (not true) to 2 (true). A higher ASSQ score indicates more severe autistic problems. The total possible score of the ASSQ ranges from 0 to 54. In this study, we used the Japanese version of the ASSQ. A previous study [19] revealed that the ASSQ had good reliability (autism spectrum disorder clinical group: Cronbach α=.88; community group: Cronbach α=.87) and validity as a screening instrument for use with preschoolers in Japanese community settings.

    ADHD-RS

    The ADHD-RS includes 18 items to measure 2 features of attention-deficit/hyperactivity disorder: Inattention (9-item subscale) and Hyperactivity/Impulsivity (9-item subscale) [20]. It is evaluated on a 4-point scale ranging from 0 (not at all or rarely) to 3 (very often). Higher scores on the ADHD-RS indicate more severe attention-deficit/hyperactivity disorder problems, with total scores ranging from 0 to 54. This study used the Japanese version of the ADHD-RS home form [37]. A previous study [22] revealed that this version of the ADHD-RS had sufficient reliability (Inattention subscale: Cronbach α=.88, Hyperactivity/Impulsivity subscale: Cronbach α=.85) and validity to screen for children potentially living with attention-deficit/hyperactivity disorder in a community setting.

    DCDQ

    The DCDQ consists of 15 items organized into 3 subscales—Control During Movements (6 items), Fine Motor and Handwriting (4 items), and General Coordination (5 items) [23]. Parents or primary caregivers were asked to evaluate the degree of motor coordination in their children compared to that of other children of the same age on a 5-point scale ranging from 1 (not at all like your child) to 5 (extremely like your child). Lower scores indicate severe developmental coordination disorder symptoms. The total possible score ranges from 15 to 75. This study used the Japanese version of the DCDQ, which has sufficient criterion validity, fit indices, and internal consistency (Total: Cronbach α=.93; Control During Movements subscale: Cronbach α=.91; Fine Motor and Handwriting: Cronbach α=.91; General Coordination: Cronbach α=.81) when used with preschool- and school-age children [38].

    P-SDQ

    The P-SDQ includes 25 items that assess children’s strengths and difficulties on 5 different subscales (each comprising 5 items): Emotional Symptoms, Conduct Problems, Hyperactivity/Inattention, Peer Relationship Problems, and Prosocial Behavior [24,25]. Parents or principal caregivers rated the items on a 3-point scale ranging from 0 (not true) to 2 (certainly true). The score for each subscale is calculated by summing the scores of 5 items, ranging from 0 to 10. The Total Difficulties score is calculated by summing the 4 difficulty subscale scores, ranging from 0 to 40. Higher scores on the 4 difficulty subscales as well as the Total Difficulties score indicate more severe emotional and behavioral deficits. Meanwhile, a higher score on the Prosocial Behavior subscale represents a more positive aspect of prosocial behavior. In this study, we used the P-SDQ, which showed favorable psychometric properties in Japanese community-based samples (Total Difficulties: Cronbach α =.77; Emotional Symptoms: Cronbach α=.61; Conduct Problems: Cronbach α=.52; Hyperactivity/Inattention: Cronbach α=.75; Peer Relationship Problems: Cronbach α=.52; Prosocial Behavior: Cronbach α=.69) [39].

    Statistical Analysis

    To test internal consistency, we calculated McDonald ω coefficients for the total and subscale scores of each measure, based on a previous study’s recommendation [40], for both web-based and paper formats. We also calculated intraclass correlation coefficients between the web-based and paper formats to evaluate the interformat reliability. Paired 2-tailed t tests were performed to evaluate mean score differences between the web-based and paper formats to examine equivalence in quality. A P value <.05 was statistically significant. Analyses were performed using SPSS software (version 25.0; IBM Corp) and R (version 4.0.3; R Foundation for Statistical Computing).

    Ethics

    The research was performed in accordance with the ethical guidelines of the Declaration of Helsinki. The protocol of this study was approved by the Committee on Medical Ethics of Hirosaki University (IRB 2018-168). To protect personal data, we adhered to the city’s and the committee’s information security policies.


    Internal Consistency

    Table 2 shows the McDonald ω coefficients for both formats of the questionnaires.

    Table 2. McDonald ω coefficients for the web-based and paper versions of the ASSQ, ADHD-RS, DCDQ, and P-SDQ (N=140).
    Scale and subscalesMcDonald ω


    		Web-basedPaper
    Autism Spectrum Screening Questionnaire.90.86a
    Attention-Deficit/Hyperactivity Disorder Rating Scale


    		Total.94.93b

    		Inattention.90.88c

    		Hyperactivity/Impulsivity.88.87c
    Developmental Coordination Disorder Questionnaire


    		Total.94.92a

    		Control During Movement.87.86a

    		Fine Motor/Handwriting.88.91d

    		General Coordination.82.74
    Parent-rated Strength and Difficulties Questionnaire


    		Total Difficulties.81.78

    		Emotional Symptoms.64.70

    		Conduct Problems.55.50

    		Hyperactivity/Inattention.78.79

    		Peer Relationship Problems.57.52

    		Prosocial Behavior.76.80

    aCalculated for 137 participants because of missing data.

    bCalculated for 133 participants because of missing data.

    cCalculated for 134 participants because of missing data.

    dCalculated for 139 participants because of missing data.

    Based on a previous study [41], an internal consistency coefficient below .70 is considered unacceptable, a coefficient from .70 to .79 is considered fair, a coefficient from .80-.89 is considered good, and a coefficient of .90 or above is considered excellent. The McDonald ω coefficients for both the web-based and paper formats of the ASSQ ranged from .86 to .90, indicating good to excellent internal consistency. The McDonald ω coefficients for both the web-based and paper formats of the overall ADHD-RS and its subscales ranged from .87 to .94, also indicating good to excellent internal consistency. Meanwhile, those for both the web-based and paper formats of the overall DCDQ and its subscales ranged from .74 to .94, indicating fair to excellent internal consistency. The McDonald ω coefficients for the web-based and paper formats of the Total Difficulties subscale and the subscales of the P-SDQ ranged from .52 to .81. Notably, the McDonald ω coefficients for both the web-based and paper versions of the Peer Relationship Problems and Conduct Problems subscales and the web-based version of the Emotional Symptoms subscale were all unacceptable [41], with coefficients ranging from .51 to .66.

    Intraclass Correlation Coefficients

    Table 3 presents the intraclass correlation coefficients between each format for ASSQ, ADHD-RS, DCDQ, and P-SDQ.

    Table 3. Intraclass correlation coefficients between the web-based and paper formats of the ASSQ, ADHD-RS, DCDQ, and P-SDQ (N=140).
    Scale and subscalesIntraclass correlationa (95% CI)
    Autism Spectrum Screening Questionnaire0.66b (0.56-0.75)
    Attention-Deficit/Hyperactivity Disorder Rating Scale

    		Total0.72c (0.62-0.79)

    		Inattention0.66d (0.55-0.74)

    		Hyperactivity/Impulsivity0.74d (0.65-0.80)
    Developmental Coordination Disorder Questionnaire

    		Total0.71b (0.61-0.78)

    		Control During Movement0.71b (0.62-0.79)

    		Fine Motor/Handwriting0.66e (0.55-0.74)

    		General Coordination0.66 (0.56-0.75)
    Parent-rated Strength and Difficulties Questionnaire

    		Total Difficulties0.73 (0.65-0.80)

    		Emotional Symptoms0.59 (0.47-0.69)

    		Conduct Problems0.66 (0.55-0.74)

    		Hyperactivity/Inattention0.68 (0.58-0.76)

    		Peer Relationship Problems0.58 (0.45-0.68)

    		Prosocial Behavior0.55 (0.43-0.66)

    aAll correlations were significant at the P<.001 level.

    bCalculated for 137 participants because of missing data.

    cCalculated for 133 participants because of missing data.

    dCalculated for 134 participants because of missing data.

    eCalculated for 139 participants because of missing data.

    Intraclass correlation coefficients between 0.50 and 0.75 are considered moderate, whereas values above 0.75 are considered high [42]. The intraclass correlation coefficient between the web-based and paper formats of ASSQ was moderate and significant (P<.001). There were also moderate significant (P<.001) intraclass correlations found between the web-based and paper formats of the overall scale and subscales of the ADHD-RS and the DCDQ, and the P-SDQ subscales.

    Mean Differences Between the Web-Based and Paper-and-Pencil Formats

    Table 4 shows the mean scores and standard deviations of both formats of the ASSQ, ADHD-RS, DCDQ, and P-SDQ.

    Table 4. Mean scores for the web-based and paper formats of ASSQ, ADHD-RS, DCDQ, and P-SDQ (N=140).
    Scale and subscalesWeb-based, mean (SD)Paper, mean (SD)t test (df)P valueCohen d
    Autism Spectrum Screening Questionnaire4.10 (5.46)3.99 (4.67)0.31 (136).760.02
    Attention-Deficit/Hyperactivity Disorder Rating Scale




    		Total4.88 (7.08)6.14 (6.94)2.83 (132).0050.18

    		Inattention2.64 (3.84)3.22 (3.69)2.15 (133).030.22

    		Hyperactivity/Impulsivity2.22 (3.46)2.92 (3.62)3.22 (136).0020.20
    Developmental Coordination Disorder Questionnaire





    		Total58.85 (11.53)57.74 (10.52)1.55 (136).120.09

    		Control During Movement22.69 (4.89)22.21 (4.48)1.57 (136).120.06

    		Fine Motor/Handwriting16.54 (3.41)16.36 (3.66)0.73 (138).470.05

    		General Coordination19.72 (4.35)19.26 (3.95)1.61 (139).110.11
    Parent-rated Strength and Difficulties Questionnaire





    		Total Difficulties7.42 (4.83)7.80 (4.72)1.29 (139).200.08

    		Emotional Symptoms1.74 (1.69)1.88 (1.76)1.08 (139).280.07

    		Conduct Problems1.82 (1.50)1.94 (1.51)1.08 (139).280.08

    		Hyperactivity/Inattention2.67 (2.16)2.76 (2.17)0.63 (139).530.05

    		Peer Relationship Problems1.19 (1.37)1.22 (1.36)0.27 (139).790.02

    		Prosocial Behavior7.86 (1.92)7.77 (2.12)0.53 (139).600.05

    There was no significant difference between the web-based and paper formats for the total mean score of the ASSQ (P=.76). Web-based scores were significantly lower than those of the paper format for the total mean scores of the ADHD-RS (t132=2.83, P=.005, Cohen d=0.18) and for its subscales (Inattention: t133=2.15, P=.03, Cohen d=0.22; Hyperactivity/Impulsivity: t133=3.21, P=.002, Cohen d=0.20). We found no significant differences between the web-based and paper formats for the mean scores on the DCDQ total (P=.12) and subscales (Control during Movement: P=.12; Fine Motor/Handwriting: P=.47; General Coordination: P=.11). Similarly, there were no significant differences between the web-based and paper formats for total P-SDQ scores (P=.20) and mean subscale scores (Emotional Symptoms: P=.28; Conduct Problems: P=.28; Hyperactivity/Inattention: P=.53; Peer Relationship Problems: P=.79; Prosocial Behavior: P=.60).


    Principal Findings

    The purpose of this study was to examine the interformat reliability of the web-based versions of the ASSQ, ADHD-RS, DCDQ, and SDQ by comparing the internal consistency, intraclass correlation, and mean score differences of their web-based and paper formats.

    For the ASSQ, ADHD-RS, and DCDQ, the McDonald ω coefficients were sufficient for both the web-based and paper versions, similar to findings in previous studies [19,22,38] that calculated Cronbach α. These results indicate that the web-based format of these questionnaires has good internal consistency. The McDonald ω coefficients for the Total Difficulties, Hyperactivity/Inattention, and Prosocial Behavior subscales of both the web-based and paper versions of the P-SDQ were also good. The McDonald ω coefficient for the paper version of the Emotional Symptoms subscale in this study was also good; however, it was unsatisfactory for the Conduct Problems and Peer Relationship Problems subscales of both the web-based and paper versions. The McDonald ω coefficient for the web-based version of the Emotional Symptoms subscale was also relatively low. These results are similar to those found in studies [25,39,43] that calculated Cronbach α for the paper format versions. Therefore, our findings suggest that, similar to the paper format, there are a few difficulties in using the web-based format of the P-SDQ for assessing externalizing and internalizing problems in children.

    We found that there were significant (P<.001) moderate positive intraclass correlations between the web-based and paper formats of the ASSQ, ADHD-RS, DCDQ total scores, and Total Difficulties score of the P-SDQ. Similar to this study, several earlier studies [44-46] on the equivalence between web-based and paper formats of self-report questionnaires meant to assess psychiatric symptoms have reported moderate significant correlations, suggesting that web-based questionnaire administration was a reliable alternative to using the paper format. Hence, it seems that the questionnaires assessing neurodevelopmental disorders, such as the ASSQ, ADHD-RS, DCDQ, and P-SDQ, can be made available and administered in web-based situations as well. However, another study [27] had earlier pointed out that the agreement rate between the web-based and paper formats was higher for objective factual questions than for questions based on personal subjective evaluation. The questionnaires used in this study were subjective evaluations of children's developmental status by parents, which may have impacted the correlation values.

    Furthermore, the analyses revealed that there were no significant differences in the ASSQ total mean scores between the web-based and paper formats (P=.76). This result suggests that the web-based version of the ASSQ is equal in quality to that of the paper version. However, there were significant differences in the ADHD-RS total and subscale mean scores between the web-based and paper formats (total: P=.005; Inattention: P=.03; Hyperactivity/Impulsivity: P=.002). Previous studies [33,36,47] have reported a significant difference in mean scores between web-based and paper formats; however, due to the small effect size (Cohen d=0.14-0.27), it was determined that the statistically significant difference in mean scores was not clinically meaningful in practice. In light of this finding, web-based questionnaires are a potential substitute for paper-based questionnaires. The effect sizes obtained in this study were also small (Cohen d=0.18-0.22). Furthermore, the web-based version McDonald ω values in this study were slightly higher than those of the paper format. These results suggest that the ADHD-RS is applicable for web-based utilization. We found that there were no significant differences in the DCDQ total and subscale mean scores between the web-based and paper formats (total: P=.12; Control during Movement: P=.12; Fine Motor/Handwriting: P=.47; General Coordination: P=.11). Additionally, we also confirmed that there were no significant differences in the P-SDQ total and subscale mean scores between the web-based and paper formats (total: P=.20; Emotional Symptoms: P=.28; Conduct Problems: P=.28; Hyperactivity/Inattention: P=.53; Peer Relationship Problems: P=.79; Prosocial Behavior: P=.60). These results suggest that both the DCDQ and P-SDQ web-based formats are equivalent in quality to those of the paper format.

    Strengths and Limitations

    The evidence found in this study supports the applicability of the web-based versions of the ASSQ, ADHD-RS, DCDQ, and P-SDQ. It has been pointed out that children with possible neurodevelopmental disorders may be overlooked during developmental health check-ups in Japan [48], and there is a lack of specialized organizations capable of assessing neurodevelopmental disorders. Furthermore, in the current COVID-19 pandemic, it is also difficult to conduct in-person evaluations. This study, which shows the applicability of web-based questionnaires assessing neurodevelopmental disorders, has the potential to improve early detection and intervention for these disorders in regions where specialized services are lacking and under the present pandemic conditions.

    However, there are some limitations to this research. First, the discriminant validity of the web-based version of each questionnaire was not confirmed in this study. This is because this study used a cross-sectional research design as part of a community developmental health check-up. Therefore, it is unclear whether the children included in this study had been diagnosed with neurodevelopmental disorders or experienced other emotional or behavioral problems or deficits that do not meet the criteria for a clinical diagnosis. It is necessary to verify the discriminant ability of the web-based version of each questionnaire for both clinical and nonclinical groups. Second, the raters who evaluated the children’s condition in this study were mostly parents. Previous research [17,49,50] conducted with paper-based questionnaires has examined the psychometric properties of the teacher-rated version of each questionnaire that was used in this study. In the future, we also need to clarify the psychometric properties of the teacher-rated web-based version of the questionnaires used in this study compared to the parent-rated web-based version. Third, we were not able to examine the impact of the order in which the questionnaires were administered. Previous studies [26,33] have confirmed that the order in which web- and paper-based questionnaires are administered has an effect on the scores of those questionnaires. In the future, it will be necessary to investigate the effect of the order of administration of the questionnaires in the research design. Fourth, the age of the children in this study was limited to 5 years. Previous studies [17,38,49,50] on the paper-based ASSQ, ADHD-RS, and SDQ have been conducted with school-age children. It is necessary to investigate the psychometric properties of the web-based version of the ASSQ, ADHD-RS, DCDQ, and P-SDQ among school-age children. Fifth, we did not control for the computer used by the participants in this study. We need to test whether the type of computer affects their responses in the future. Finally, this study was conducted in one medium-size city in Japan, thereby limiting the generalizability of its findings to other regions.

    Conclusions

    This study examined the interformat reliability of the web-based versions of questionnaires for assessing neurodevelopmental disorders. Our findings showed that the web-based versions of the ASSQ, ADHD-RS, DCDQ, and P-SDQ had the same level of internal consistency, intrarater reliability, and equality as their paper versions. These results indicate the web applicability of these questionnaires for assessing neurodevelopmental disorders.

    Acknowledgments

    M Tanaka and MS conceptualized and designed the study. Data collection was conducted by M Tanaka and MS. M Tanaka conducted data analysis and drafted the initial manuscript. MS, M Takahashi, and MA reviewed the manuscript and approved the final manuscript as submitted. KN conceptualized and designed the study, critically reviewed the manuscript, and approved the final manuscript as submitted. MS, M Takahashi, MA, and KN administered developmental check-ups for children who participated in the study. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work. This study was conducted by the Graduate School of Medicine at Hirosaki University, in close collaboration with the municipal health center and the city. We express gratitude to all the participants and their families. The authors gratefully acknowledge the contributions of local practitioners, public servants, and students. This study was financially supported by the Japan Society for the Promotion of Science KAKENHI (grant numbers: JP18K03106, JP16K10239), Hirosaki Institute of Neuroscience in Japan, Hirosaki City and the Survey Research Center Co Ltd.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Participants who took part in the paper-and-pencil survey only (N=368).

    DOCX File , 16 KB
    1. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. Arlington, VA. American Psychiatric Association; 2013.
    2. Khan NZ, Gallo LA, Arghir A, Budisteanu B, Budisteanu M, Dobrescu I, et al. Autism and the grand challenges in global mental health. Autism Res. Jun 2012;5(3):156-159. [CrossRef] [Medline]
    3. Kim YS, Fombonne E, Koh Y, Kim S, Cheon K, Leventhal BL. A comparison of DSM-IV pervasive developmental disorder and DSM-5 autism spectrum disorder prevalence in an epidemiologic sample. J Am Acad Child Adolesc Psychiatry. May 2014;53(5):500-508. [FREE Full text] [CrossRef] [Medline]
    4. Saito M, Hirota T, Sakamoto Y, Adachi M, Takahashi M, Osato-Kaneda A, et al. Prevalence and cumulative incidence of autism spectrum disorders and the patterns of co-occurring neurodevelopmental disorders in a total population sample of 5-year-old children. Mol Autism. May 14, 2020;11(1):35. [FREE Full text] [CrossRef] [Medline]
    5. Thapar A, Cooper M. Attention deficit hyperactivity disorder. Lancet. Mar 19, 2016;387(10024):1240-1250. [CrossRef] [Medline]
    6. Polanczyk GV, Salum GA, Sugaya LS, Caye A, Rohde LA. Annual Research Review: a meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J Child Psychol Psychiatry. Mar 2015;56(3):345-365. [CrossRef] [Medline]
    7. Blank R, Barnett AL, Cairney J, Green D, Kirby A, Polatajko H, et al. International clinical practice recommendations on the definition, diagnosis, assessment, intervention, and psychosocial aspects of developmental coordination disorder. Dev Med Child Neurol. Mar 2019;61(3):242-285. [FREE Full text] [CrossRef] [Medline]
    8. Piek JP, Rigoli D, Pearsall-Jones JG, Martin NC, Hay DA, Bennett KS, et al. Depressive symptomatology in child and adolescent twins with attention-deficit hyperactivity disorder and/or developmental coordination disorder. Twin Res Hum Genet. Aug 2007;10(4):587-596. [CrossRef] [Medline]
    9. Spencer T, Biederman J, Wilens T. Attention-deficit/hyperactivity disorder and comorbidity. Pediatr Clin North Am. Oct 1999;46(5):915-927. [CrossRef] [Medline]
    10. Strang JF, Kenworthy L, Daniolos P, Case L, Wills MC, Martin A, et al. Depression and anxiety symptoms in children and adolescents with autism spectrum disorders without intellectual disability. Res Autism Spectr Disord. Jan 2012;6(1):406-412. [FREE Full text] [CrossRef] [Medline]
    11. Cousins M, Smyth MM. Developmental coordination impairments in adulthood. Hum Mov Sci. Nov 2003;22(4-5):433-459. [CrossRef] [Medline]
    12. Howlin P, Magiati I. Autism spectrum disorder: outcomes in adulthood. Curr Opin Psychiatry. Mar 2017;30(2):69-76. [CrossRef] [Medline]
    13. Schoenfelder EN, Kollins SH. Topical review: ADHD and health-risk behaviors: toward prevention and health promotion. J Pediatr Psychol. Aug 30, 2016;41(7):735-740. [FREE Full text] [CrossRef] [Medline]
    14. Daley D, van der Oord S, Ferrin M, Danckaerts M, Doepfner M, Cortese S, et al. European ADHD Guidelines Group. Behavioral interventions in attention-deficit/hyperactivity disorder: a meta-analysis of randomized controlled trials across multiple outcome domains. J Am Acad Child Adolesc Psychiatry. Aug 2014;53(8):835-47, 847.e1-5. [CrossRef] [Medline]
    15. Landa RJ. Efficacy of early interventions for infants and young children with, and at risk for, autism spectrum disorders. Int Rev Psychiatry. Feb 2018;30(1):25-39. [FREE Full text] [CrossRef] [Medline]
    16. Ehlers S, Gillberg C, Wing L. A screening questionnaire for Asperger syndrome and other high-functioning autism spectrum disorders in school age children. J Autism Dev Disord. Apr 1999;29(2):129-141. [CrossRef] [Medline]
    17. Posserud M, Lundervold AJ, Gillberg C. Validation of the Autism Spectrum Screening Questionnaire in a total population sample. J Autism Dev Disord. Jan 1, 2009;39(1):126-134. [CrossRef] [Medline]
    18. Posserud M, Lundervold AJ, Steijnen MC, Verhoeven S, Stormark KM, Gillberg C. Factor analysis of the Autism Spectrum Screening Questionnaire. Autism. Jan 2008;12(1):99-112. [CrossRef] [Medline]
    19. Adachi M, Takahashi M, Takayanagi N, Yoshida S, Yasuda S, Tanaka M, et al. Adaptation of the Autism Spectrum Screening Questionnaire (ASSQ) to preschool children. PLoS One. 2018;13(7):e0199590. [FREE Full text] [CrossRef] [Medline]
    20. DuPaul GJ, Power TJ, Anastopoulos AD, Reid R. ADHD Rating Scale-IV: Checklists, Norms, and Clinical Interpretation. New York. Guilford Press; 1998.
    21. McGoey KE, DuPaul GJ, Haley E, Shelton TL. arent and teacher ratings of attention-deficit/hyperactivity disorder in preschool: the ADHD rating scale-IV preschool version. J Psychopathol Behav Assess. Jun 28, 2007;29(4):269-276. [CrossRef]
    22. Takayanagi N, Yoshida S, Yasuda S, Adachi M, Kaneda-Osato A, Tanaka M, et al. Psychometric properties of the Japanese ADHD-RS in preschool children. Res Dev Disabil. Aug 2016;55:268-278. [CrossRef] [Medline]
    23. Wilson BN, Crawford SG, Green D, Roberts G, Aylott A, Kaplan BJ. Psychometric properties of the revised Developmental Coordination Disorder Questionnaire. Phys Occup Ther Pediatr. Jul 10, 2009;29(2):182-202. [CrossRef] [Medline]
    24. Goodman R. The Strengths and Difficulties Questionnaire: a research note. J Child Psychol Psychiatry. Jul 1997;38(5):581-586. [CrossRef] [Medline]
    25. Goodman R. Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatry. Nov 2001;40(11):1337-1345. [CrossRef] [Medline]
    26. Carlbring P, Brunt S, Bohman S, Austin D, Richards J, Öst L, et al. Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Comput Human Behav. May 2007;23(3):1421-1434. [CrossRef]
    27. Braekman E, Berete F, Charafeddine R, Demarest S, Drieskens S, Gisle L, et al. Measurement agreement of the self-administered questionnaire of the Belgian Health Interview Survey: Paper-and-pencil versus web-based mode. PLoS One. May 21, 2018;13(5):e0197434. [FREE Full text] [CrossRef] [Medline]
    28. Moreno C, Wykes T, Galderisi S, Nordentoft M, Crossley N, Jones N, et al. How mental health care should change as a consequence of the COVID-19 pandemic. Lancet Psychiatry. Sep 2020;7(9):813-824. [FREE Full text] [CrossRef] [Medline]
    29. Ong CH, Ragen ES, Aishworiya R. Ensuring continuity of pediatric psychological assessment services during the COVID-19 pandemic: challenges and long-term implications. Eur J Psychol Assess. Jul 2020;36(4):516-524. [CrossRef]
    30. Hedman E, Ljótsson B, Rück C, Furmark T, Carlbring P, Lindefors N, et al. Internet administration of self-report measures commonly used in research on social anxiety disorder: a psychometric evaluation. Comput Human Behav. Jul 2010;26(4):736-740. [CrossRef]
    31. Ritter P, Lorig K, Laurent D, Matthews K. Internet versus mailed questionnaires: a randomized comparison. J Med Internet Res. Sep 15, 2004;6(3):e29. [FREE Full text] [CrossRef] [Medline]
    32. Van De Looij-Jansen PM, De Wilde EJ. Comparison of web-based versus paper-and-pencil self-administered questionnaire: effects on health indicators in Dutch adolescents. Health Serv Res. Oct 2008;43(5 Pt 1):1708-1721. [FREE Full text] [CrossRef] [Medline]
    33. Holländare F, Andersson G, Engström I. A comparison of psychometric properties between internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. J Med Internet Res. Dec 19, 2010;12(5):e49. [FREE Full text] [CrossRef] [Medline]
    34. Whitehead L. Methodological issues in internet-mediated research: a randomized comparison of internet versus mailed questionnaires. J Med Internet Res. Dec 04, 2011;13(4):e109. [FREE Full text] [CrossRef] [Medline]
    35. Buchanan T. Internet-based questionnaire assessment: appropriate use in clinical contexts. Cogn Behav Ther. 2003;32(3):100-109. [CrossRef] [Medline]
    36. Alfonsson S, Maathz P, Hursti T. Interformat reliability of digital psychiatric self-report questionnaires: a systematic review. J Med Internet Res. Dec 03, 2014;16(12):e268. [FREE Full text] [CrossRef] [Medline]
    37. Tani I, Okada R, Ohnishi M, Nakajima S, Tsujii M. Japanese version of home form of the ADHD-RS: an evaluation of its reliability and validity. Res Dev Disabil. 2010;31(6):1426-1433. [CrossRef] [Medline]
    38. Nakai A, Miyachi T, Okada R, Tani I, Nakajima S, Onishi M, et al. Evaluation of the Japanese version of the Developmental Coordination Disorder Questionnaire as a screening tool for clumsiness of Japanese children. Res Dev Disabil. 2011;32(5):1615-1622. [CrossRef] [Medline]
    39. Matsuishi T, Nagano M, Araki Y, Tanaka Y, Iwasaki M, Yamashita Y, et al. Scale properties of the Japanese version of the Strengths and Difficulties Questionnaire (SDQ): a study of infant and school children in community samples. Brain Dev. Jun 2008;30(6):410-415. [CrossRef] [Medline]
    40. McNeish D. Thanks coefficient alpha, we'll take it from here. Psychol Methods. Sep 2018;23(3):412-433. [CrossRef] [Medline]
    41. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. Dec 1994;6(4):284-290. [CrossRef]
    42. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. Jun 2016;15(2):155-163. [FREE Full text] [CrossRef] [Medline]
    43. Doi Y, Ishihara K, Uchiyama M. Reliability of the Strengths and Difficulties Questionnaire in Japanese preschool children aged 4-6 Years. J Epidemiol. Aug 23, 2014;24(6):514-518. [FREE Full text] [CrossRef] [Medline]
    44. Austin DW, Carlbring P, Richards JC, Andersson G. Internet administration of three commonly used questionnaires in panic research: equivalence to paper administration in Australian and Swedish samples of people with panic disorder. Int J Test. Mar 2006;6(1):25-39. [CrossRef]
    45. Read JP, Farrow SM, Jaanimägi U, Ouimette P. Assessing trauma and traumatic stress via the internet: measurement equivalence and participant reactions. Traumatology. Mar 2009;15(1):94-102. [CrossRef]
    46. Vallejo MA, Jordán CM, Díaz MI, Comeche MI, Ortega J. Psychological assessment via the internet: a reliability and validity study of online (vs paper-and-pencil) versions of the General Health Questionnaire-28 (GHQ-28) and the Symptoms Check-List-90-Revised (SCL-90-R). J Med Internet Res. Jan 31, 2007;9(1):e2. [FREE Full text] [CrossRef] [Medline]
    47. Yu S, Yu M. Comparison of internet-based and paper-based questionnaires in Taiwan using multisample invariance approach. Cyberpsychol Behav. Aug 2007;10(4):501-507. [CrossRef] [Medline]
    48. Ministry of Internal Affairs and Communications, Japan. Administrative evaluation and monitoring about support for persons with developmental disabilities - evidence based recommendation. 2017. URL: https://www.soumu.go.jp/main_content/000458760.pdf
    49. Ohnishi M, Okada R, Tani I, Nakajima S, Tsujii M. Japanese version of school form of the ADHD-RS: an evaluation of its reliability and validity. Res Dev Disabil. 2010;31(6):1305-1312. [CrossRef] [Medline]
    50. Moriwaki A, Kamio Y. Normative data and psychometric properties of the Strengths and Difficulties Questionnaire among Japanese school-aged children. Child Adolesc Psychiatry Ment Health. Jan 21, 2014;8(1):1. [FREE Full text] [CrossRef] [Medline]

    ‎
    ADHD-RS: Attention-Deficit/Hyperactivity Disorder Rating Scale
    ASSQ: Autism Spectrum Screening Questionnaire
    DCDQ: Developmental Coordination Disorder Questionnaire 2007
    DSM-5: Diagnostic and Statistical Manual for Mental Disorders (Fifth Edition)
    P-SDQ: Parent-reported Strengths and Difficulties Questionnaire
    SDQ: Strengths and Difficulties Questionnaire

    Edited by S Badawy; submitted 12.11.20; peer-reviewed by J Piqueras; comments to author 04.12.20; revised version received 12.12.20; accepted 16.01.21; published 04.02.21.

    Copyright

    ©Masanori Tanaka, Manabu Saito, Michio Takahashi, Masaki Adachi, Kazuhiko Nakamura. Originally published in JMIR Pediatrics and Parenting (http://pediatrics.jmir.org), 04.02.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on http://pediatrics.jmir.org, as well as this copyright and license information must be included.