This paper is in the following e-collection/theme issue:

Chronic Disease Self-Management in Childhood and Adolescence (139) Electronic/Mobile Data Capture, Internet-based Survey & Research Methodology (832) Autism Spectrum Disorder (ASD) (109) Diagnostic Tools in Mental Health (242) Memory (43)

Published on in Vol 6 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/39720, first published .
Long-term Memory Testing in Children With Typical Development and Neurodevelopmental Disorders: Remote Web-based Image Task Feasibility Study

Long-term Memory Testing in Children With Typical Development and Neurodevelopmental Disorders: Remote Web-based Image Task Feasibility Study

Long-term Memory Testing in Children With Typical Development and Neurodevelopmental Disorders: Remote Web-based Image Task Feasibility Study

Authors of this article:

Truong An Bui1 Author Orcid Image ;   Cory Scott Rosenfelt1 Author Orcid Image ;   Kerri Hope Whitlock1 Author Orcid Image ;   Mickael Leclercq2 Author Orcid Image ;   Savannah Weber1 Author Orcid Image ;   Arnaud Droit2 Author Orcid Image ;   Sandra A Wiebe3, 4 Author Orcid Image ;   Jacqueline Pei5 Author Orcid Image ;   Francois V Bolduc1, 4, 6 Author Orcid Image
Article Authors Cited by Tweetations (3) Metrics

    Original Paper

    1Department of Pediatrics, University of Alberta, Edmonton, AB, Canada

    2Computational Biology Laboratory, Centre de Recherche du CHU de Québec, Université Laval Research Centre, Québec City, QC, Canada

    3Department of Psychology, University of Alberta, Edmonton, AB, Canada

    4Neuroscience and Mental Health Institute, University of Alberta, Edmonton, AB, Canada

    5Department of Educational Psychology, University of Alberta, Edmonton, AB, Canada

    6Women and Children Health Research Institute, University of Alberta, Edmonton, AB, Canada

    Corresponding Author:

    Francois V Bolduc, MD, PhD

    Department of Pediatrics

    University of Alberta

    3020 Katz Group Centre

    11315 87 Ave

    Edmonton, AB, T6G 2E1

    Canada

    Phone: 1 780 492 9616

    Email: fbolduc@ualberta.ca


    Background: Neurodevelopmental disorders (NDD) cause individuals to have difficulty in learning facts, procedures, or social skills. NDD has been linked to several genes, and several animal models have been used to identify potential therapeutic candidates based on specific learning paradigms for long-term and associative memory. In individuals with NDD, however, such testing has not been used so far, resulting in a gap in translating preclinical results to clinical practice.

    Objective: We aim to assess if individuals with NDD could be tested for paired association learning and long-term memory deficit, as shown in previous animal models.

    Methods: We developed an image-based paired association task, which can be performed at different time points using remote web-based testing, and evaluated its feasibility in children with typical development (TD), as well as NDD. We included 2 tasks: object recognition as a simpler task and paired association. Learning was tested immediately after training and also the next day for long-term memory.

    Results: We found that children aged 5-14 years with TD (n=128) and with NDD of different types (n=57) could complete testing using the Memory Game. Children with NDD showed deficits in both recognition and paired association tasks on the first day of learning, in both 5-9–year old (P<.001 and P=.01, respectively) and 10-14–year old groups (P=.001 and P<.001, respectively). The reaction times to stimuli showed no significant difference between individuals with TD or NDD. Children with NDD exhibited a faster 24-hour memory decay for the recognition task than those with TD in the 5-9–year old group. This trend is reversed for the paired association task. Interestingly, we found that children with NDD had their retention for recognition improved and matched with typically developing individuals by 10-14 years of age. The NDD group also showed improved retention deficits in the paired association task at 10-14 years of age compared to the TD group.

    Conclusions: We showed that web-based learning testing using simple picture association is feasible for children with TD, as well as with NDD. We showed how web-based testing allows us to train children to learn the association between pictures, as shown in immediate test results and those completed 1 day after. This is important as many models for learning deficits in NDD target both short- and long-term memory for therapeutic intervention. We also demonstrated that despite potential confounding factors, such as self-reported diagnosis bias, technical issues, and varied participation, the Memory Game shows significant differences between typically developing children and those with NDD. Future experiments will leverage this potential of web-based testing for larger cohorts and cross-validation with other clinical or preclinical cognitive tasks.

    JMIR Pediatr Parent 2023;6:e39720

    doi:10.2196/39720

    Keywords

    memory; neurodevelopmental disorder; autism spectrum disorder; intellectual disability; developmental delay; hippocampus; recognition; paired association learning; remote testing; autism; disorder; genetics; developmental; developmental disorder; game; remote; testing; diagnose; diagnosis 


    Neurodevelopmental Disorders Are Common

    Learning new information by forming associations is at the core of development and daily functioning. Yet, our understanding of how such associations may differ between typically developing individuals and those with neurodevelopmental disorders (NDD) remains insufficient in some important aspects. NDD includes a group of diagnoses, in which the development of typical brain functions, such as attention, cognition, or social functioning, is altered [1]. Examples of common NDDs include attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder (ASD), and intellectual disability [2]. In the United States alone, approximately 1 in 6 (17%) children between the ages of 3 and 17 years had an NDD, as reported by the parents [3]. This mirrors the rate identified in population studies worldwide [4-10]. Not only individuals with NDD but also their families will experience significant financial and psychological burdens [11-13], necessitating more effort in developing targeted interventions, as most NDD will have lifelong effects [14].

    Current Discrepancies in the Memory Type Tested Between Preclinical and Clinical NDD Models

    In the last decade, there has been a vast increase in studies identifying genes associated with each neurodevelopmental condition and the development of multiple animal models to study the disorders [15-17]. Nevertheless, there is a disparity between the cognitive testing used in humans and the cognitive measures used in animal models, which may hamper the translation of candidate treatments [18,19].

    As many of the genes discovered in NDD belong to signaling pathways that were already investigated in animal models of learning and memory [20,21], many well-established “memory assays” previously used in NDD animal models do not necessarily align with NDD clinical testing. Animal testing has been focusing on studying nonassociative (sensitization and habituation) versus associative memory (fear conditioning, spatial navigation, and olfactory conditioning) [22,23], and short-term memory (STM; training and recall within minutes) versus long-term memory (LTM; 24 hours) [24]. Mutants for NDD gene orthologues and candidate pharmacological interventions have been characterized for their effects on those behaviors [25-27]. The tools to detect memory problems of NDD individuals in clinical settings, however, remain inadequate or rely heavily on self-reports [28,29].

    While some of the pioneering work on memory and recall in typical individuals, such as one by Ebinhaus, investigated STM and LTM [30], most of the clinical memory testing has been focusing on STM types, such as working memory [31]. The majority of individuals with NDD, including those with common genetically defined conditions such as Fragile X syndrome [32-35], Down syndrome [36,37], Williams syndrome [38], as well as clinically defined disorders, such as specific language impairment [39,40], ADHD [41-43], ASD [44], and intellectual disability [45-48], have been found to have STM defects. LTM, however, has not been reported, except in 1 study done in the 1990s, showing LTM defects in Down syndrome [49]. We hypothesize that this is partly due to technical and financial limitations that limit the feasibility of developing novel and highly accurate memory testing, and repeated testing in a lab setting. This suggests that web-based or remote testing might be the solution.

    Similarly, limitations in the current test design have limited consistent use of memory testing in NDD. Paired association is often employed clinically with the use of picture and name or word-word association [50]. This type of test relies on hippocampal functioning [51-54]. Typically, an individual will learn an association between a person’s picture and their name. At testing time, they will be presented with the photo and 3 names (1 correct and 2 other names of different individuals seen in the training session). Those with NDD can have challenges in reading and have, therefore, not been exposed to paired association testing extensively [55,56]. Nevertheless, picture-based association tasks have been used for children with dyslexia [22,57] and NDD [58] to circumvent issues with language delay [59,60].

    A potential issue in remote testing would be to infer the degree of attention of the participants. Control tasks [61], such as recognition of a pair, a less difficult, hippocampal-independent task, have been used to test the ability of an individual to participate and their attention to a task [54,62,63]. In this situation, using flanker images, which have never been seen before, makes recalling the association easier.

    Emergence of Web-based and Remote Testing as a Method for Cognitive Assessment in NDD

    Touch screen-based testing has emerged as an accurate and engaging tool for testing children and individuals with NDD. Evidently, subtasks of the NIH Toolbox for the Assessment of Neurological and Behavioral Function, such as episodic memory [64] and working memory [65], have been used recently in a clinical trial for Fragile X syndrome [66]. However, those tasks do not assess associative or LTM skills. Additionally, the NIH toolbox currently needs to be administered locally on tablets, with guidance from trained researchers in a laboratory. Another test that is used for individuals with NDD is the Cambridge Neuropsychological Test Automated Battery, which evaluates working memory, episodic memory, attention, and decision-making [67]. This test is licensed and has been administered in individuals with ADHD using researchers’ tablets [68,69]. While being well constructed, these tests currently do not allow for the testing of associative memory at different time points. They are also limited to in situ testing in the laboratory or on researchers’ devices, making it expensive and inaccessible to a large number of participants. Testing in traditional laboratory environments can also be challenging for children, especially those with NDD, considering the high prevalence of anxiety [70-76].

    Thus, we aim to develop a remote web-based and accessible cognitive test, which could better mirror the type of memory (associative) and the different time points (STM and LTM) used in preclinical models, in hope that this will make preclinical therapies more easily translated into clinical trials. Here, we assessed the feasibility of such testing administered web-based in individuals with typical development (TD) and NDD.


    Ethics Approval

    The project was approved by the ethics committee at the University of Alberta (Pro00033138).

    Informed Consent

    Informed consent to participate in the study was sought from the participant’s parent or guardian (Multimedia Appendix 1). Next, participants, or parents and caregivers, were asked to provide information about the participant’s age, neurological and medical conditions, sex, and current medications (Figure 1).

    ‎
    Figure 1. Registration. Participants or caregivers enter their email addresses (to receive the participant code) and then provide demographic information as well as the diagnosis they have (if applicable).

    Participants

    In total, 128 individuals with no known neurological conditions and 57 individuals with NDD, including developmental delay, ASD, and intellectual disability participated in the study. Eligible participants’ caregivers self-reported their diagnoses or the absence thereof. Table 1 includes the demographic information for the participants. The participants were contacted by their School Board via the Healthy Infants and Children Clinical Research Program (HICCUP) or via family support groups. The access was free of charge to participants. The participants received the URL for the Memory Game.

    Table 1. Demographic characteristics of participants.
    Condition and age groupDay 1, nDay 2, n
    Typical development


    		5-9 years97 (female: 4, male: 4, not available: 89)56 (female: 4, male: 2, not available: 50)

    		10-14 years31 (female: 3, male: 3, not available: 25)17 (female: 0, male: 2, not available: 15)
    Neurodevelopmental disorders


    		5-9 years30 (female: 7, male: 6, not available: 17)16 (female: 5, male: 4, not available: 7)

    		10-14 years27 (female: 3, male: 6, not available: 18)19 (female: 3, male: 6, not available: 10)

    The Memory Game Interface

    All participation in the Memory Game was done web-based using a touch screen tablet device (eg, iPad). The procedure has five main components: (1) consent, (2) registration, (3) tutorial video, (4) training phase, and (5) testing phase.

    Having completed the consent and registration, participants received a code via email, which allows them to move on to the training and testing components. These occur over 2 days. On day 1, the participants participate in 3 phases of the game: a tutorial phase, a practice phase, and a testing phase. The phases had to be completed to proceed through the test. The tutorial phase consisted of verbal and visual explanations of completing the test by properly matching the pairs. This is in the form of an instructional video to explain how the game works. A cartoon frog demonstrates with examples, while a voiceover explains what to do (Figure 2). The video is followed by the practice phase, in which participants are first shown 6 sets of pairs, followed by 6 questions. In this section, they were given feedback as to whether their response was correct or incorrect. Finally, participants went through the testing phase where they were shown 20 sets of pairs followed by 20 questions, this time without any feedback on their responses. On day 2, the participants were directed to the testing phase (as on day 1) and were not given feedback on their responses.

    ‎
    Figure 2. Tutorial video demonstration. In order to explain the task as easily as possible (without direct supervision from the researchers), a tutorial video explaining the procedure to the participants was developed. This includes a voiceover reinforcing the information provided visually about the pictures that go together.

    Task Description

    The Memory Game includes questions testing both paired association and recognition memory, depending on the distractor photos present (Figure 3). In a recognition question, the 2 distractor photos are not part of any pairs seen in training and have not been previously viewed. In a paired association question, 1 of the distractors will be the matching photo for a prompt not currently being displayed and so will have been previously viewed. The images used have been selected based on their shape and appearance from the Bank of Standardized stimuli (BOSS) database created by Dr Mathieu Brodeur.

    ‎
    Figure 3. Example of stimuli and paradigms used in the Memory Game. (A-D) Representative examples of pairs of images used in the training phase. The testing phase consists of 2 different types of tasks: paired association (PA) and recognition (R). (E) In the PA task, the participant must distinguish the correct association (the bee goes with the dice as seen in training 3C) but with flanker images present that were part of other associations presented in the training phase (hat from training 3D), making the task more difficult. (F) For the R task, the target image from the pair (the camera which goes with the bear) is flanked by 2 pictures, which were not seen in the training phase.

    Task Development

    We initially developed the Memory Game for testing using E-Prime (Psychology Software Tools) and administered it to children with NDD in our developmental neurology clinic, using a touch screen laptop under the supervision of trained undergraduate neuroscience students. This allowed us to optimize the instruction video and the choice of images, and refine the task. For instance, we found that having a voiceover during the instruction video made it easier for the participants and caregivers to follow. The choice of the pictures used for the paired association and recognition testing, which were standardized in the BOSS database, was reviewed for general understandability and quality assurance with caregivers and for formatting (orientation, color, and shapes) with 2 psychologists from our team (SAW and JP). Similarly, we changed the position of the “correct response” from left, middle, and right to prevent a “false success” if, for instance, an individual always chose a given position. We also sought feedback from caregivers present in the room during the testing about the flow used and challenges in understandability. In addition to general comments, we ensured that the participants understood the instruction video and the training and testing phases of the Memory Game. Once established in E-Prime software, a widely used program for stimulus presentation [77], the final version of the task was discussed with the programmers (Jeffrey Van Alstine, Arthur Schuiltz, Department of Education from University of Alberta) and developed for web-based testing on tablets. The Memory Game underwent several revisions to optimize responsiveness. While diagnosis and age were included in all versions, sex was added in the second version only.

    Coding Approach

    The Memory Game consists of 3 parts: the front-end game interface, the researcher interface, and the application programming interface (API) that both communicate with. The front-end game interface is static HTML running a custom web application built with jQuery 1.11 (OpenJS Foundation) and reliant on the YouTube iFrame API. The researcher interface is a small custom PHP 7 application with no dependencies. The API itself is constructed on the PHP Slim framework v3.4, uses PHPMailer v5.3 to send confirmation or reminder emails through a local university SMTP server, and in turn saves the data in a MySQL (Oracle Corporation) database with any identifying information (email addresses) encrypted such that it can only be decoded and read by researchers with the correct key. The Memory Game is web based and can be accessed in [78].

    Data Analysis

    From the game component, the percentage of correct responses was recorded, as well as the time required to answer each question as reaction time (RT).

    The performance of participants with NDD was compared against the corresponding age group of the TD participants. The groups were broken down into 2 age groups: 5-9 years and 10-14 years. The percent correct score is given as a percentage of questions answered correctly. The performance on day 1 and day 2 was plotted. A line joining those 2 was drawn and the slope was used to estimate memory decay. RT is the average time participants took to answer the questions and is given in ms. Unpaired t tests with assumed Gaussian distribution (using parametric test), no assumption about consistent SDs (the Welch t test), and no correction for multiple comparisons were performed (P value * <.05, ** <.01, *** <.001; Cronbach α=.05). Effect sizes between groups were reported as Cohen d values. All statistical analyses were run using Prism software (GraphPad).

    Outlier Analysis

    For RT, any outliers for a participant were detected using a modified Z-score [79], adapted for small data sets, and then removed. This was done on a participant-by-participant basis, with any absolute value greater than 3.5 labeled as an outlier.


    Paired Association and Recognition Performance on Day 1

    We recruited individuals with TD and individuals with different types of NDD. These were self-reported by the users’ parents and included developmental delay, intellectual disability, and ASD (Table 2).

    We started by assessing learning performance (day 1) for individuals with NDD compared to individuals with TD. We evaluated both performance and RT for each type of paradigm (paired association and recognition) in the function of age group (5-9 years old and 10-14 years old).

    The typical session length ranged from 3 to 6 minutes. Not all participants completed the test on day 1 after initiating it. For day 1, 69 TD and 32 NDD individuals started but did not finish the practice portion, while 32 TD and 14 NDD individuals finished the practice but not the testing portion. For day 2, among participants that completed day 1, 5 TD and 12 NDD started but did not finish the testing.

    NDD performance was significantly lower for recognition memory in both age groups (ages 5-9 years, P<.001, d=0.89; ages 10-14 years, P=.001, d=0.90) (Figure 4A). We found that RT was not significantly different between TD and NDD individuals (ages 5-9 years, P=.90; ages 10-14 years, P=.09) (Figure 4B). We observed a trend toward decreased RT from 5-9 to 10-14 years in TD, but we did not observe such a reduction in NDD individuals.

    Table 2. Distribution of diagnosis by age and day of memory test performance.
    ConditionDay 1Day 2

    		Ages 5-9 years (n=30), nAges 10-14 years (n=27), nAges 5-9 years (n=16), nAges 10-14 years (n=19), n
    Autism7342
    Developmental delay151299
    Intellectual disability0101
    Multiple conditions81137
    ‎
    Figure 4. Performance in recognition (R) and paired association (PA) tasks. (A) Performance of individuals with typical development (TD) and neurodevelopmental disorders (NDD) in the recognition task by age group. (B) Reaction time (RT) in the R task in individuals with TD and NDD by age group. (C) Performance for the PA task in individuals with TD and NDD by age group. (D) RT for the PA task in individuals with TD and NDD by age group. TD ages 5-9 years, day 1: n=97; TD ages 10-14 years, day 1: n=31; NDD ages 5-9 years, day 1: n=30; NDD ages 10-14 years, day 1: n=27. t tests were performed to assess differences. *P<.05, **P<.01, ***P<.001.

    For paired association memory, we found significant deficits in 5-9 years old children (P=.01, d=0.53) and 10-14 years old (P<.001, d=0.94) with NDD compared to TD (Figure 4C). RT did not show significant differences (ages 5-9 years, P=.91; ages 10-14 years, P=.23) but showed a trend for decreased RT in TD (Figure 4D) between the 2 age groups. RT in NDD trended toward increasing when comparing 5-9 years old to 10-14 years old individuals, suggesting that NDD performance potentially requires more cognitive processing, and thus the increased reaction time.

    We also found that recognition scores were higher than the paired association. This difference was observed for both groups; however, it was only significant for the TD groups (ages 5-9 years, P<.001, d=0.60; ages 10-14 years, P=.03, d=0.57) and not the NDD groups (ages 5-9 years, P=.53; ages 10-14 years, P=.12).

    Memory Decay From Day 1 to Day 2

    We found that for the recognition task, 5-9 years old individuals with NDD presented with a faster decay in performance on day 2 (24-hour decay) compared to individuals with TD, as shown by the difference in slope (Y=5.637 in TD vs Y=9.333 in NDD) (Figure 5A). Nevertheless, this was not observed in individuals aged 10-14 years (Figure 5B). Strikingly, the opposite trend was observed for the paired association. Individuals with NDD showed similar decay in the 5-9 years old group (Figure 5C) but had higher decay in 10-14 years old (Figure 5D). No differences were observed in any of the reaction time comparisons (Multimedia Appendix 2). There was similar attrition in participation from day 1 to day 2 at all ages for both TD and NDD groups (41 out of 97 participants [42%] for individuals with TD vs 14 out of 30 [47%] for those with NDD in the 5-9 years old group, and 14 out of 31 participants [45%] in TD vs 12 out of 27 (44%) in individuals with NDD in the 10-14 years old group).

    ‎
    Figure 5. Comparative performance right after training (day 1) compared to performance the day after (day 2). (A) Performance of individuals with typical development (TD) and neurodevelopmental disorders (NDD) in the recognition task by age group. (B) Performance in the recognition (R) task in individuals with TD and NDD by age group. (C) Performance for the paired association (PA) task in individuals with TD and NDD by age group. (D) Performance for the PA task in individuals with TD and NDD by age group. TD ages 5-9 years, day 1: n=97; TD ages 5-9 years, day 2: n=56; TD ages 10-14 years, day 1: n=31; TD ages 10-14 years, day 2: n=17; NDD ages 5-9 years, day 1: n=30; NDD ages 5-9 years, day 2: n=16; NDD ages 10-14 years, day 1: n=27; NDD ages 10-14 years, day 2: n=19.

    Performance Distribution Homogeneity Between Groups

    We compared individuals from TD and NDD by plotting their performance and RT in both paired association and recognition to demonstrate homogeneity. Several individuals with NDD had mixed diagnoses; hence, we could not plot them by diagnosis individually. We found that those with NDD appeared to have a different group distribution in performance, especially in the younger age group (5-9 years old), with a bimodal distribution for recognition and paired association (Figure 6). On the other hand, RT had a more uniform distribution (Multimedia Appendix 3).

    ‎
    Figure 6. Distribution of performance on day 1 and day 2 for individuals with typical development (TD) and neurodevelopmental disorders (NDD). (A) Memory performance for recognition (R) in individuals with TD of the 5-9 years old group. (B) Memory performance for R in individuals with NDD of the 5-9 years old group. (C) Memory performance for R for individuals with TD of the 10-14 years old group. (D) Memory performance for R for those with NDD of the 10-14 years old group. (E) Memory performance for paired association (PA) for individuals with TD of the 5-9 years old group. (F) Memory performance for PA for individuals with NDD of the 5-9 years old group. (G) Memory performance for PA for individuals with TD of the 10-14 years old group. (H) Memory performance for PA for individuals with NDD of the 10-14 years old group. TD ages 5-9 years, day 1: n=97; TD ages 5-9 years, day 2: n=56; TD ages 10-14 years, day 1: n=31, TD ages 10-14 years, day 2: n=17; NDD ages 5-9 years, day 1: n=30; NDD ages 5-9 years, day 2: n=16; NDD ages 10-14 years, day 1: n=7; NDD ages 10-14 years, day 2: n=19.

    Feasibility of Remote Web-based Cognitive Testing

    Our work shows that remote web-based memory testing is feasible for children with TD (n=128 individuals) and NDD (n=57 individuals), as early as 5 years old. Indeed, using picture matching allowed for the testing of children with NDD. The results are also consistent with a previous report of a progressive increase in performance for recognition tasks in children with TD [80]. Our task also allowed us to measure paired association memory and find significant differences between participants with TD versus NDD.

    It also found that children with TD or NDD were able to perform LTM (day 2) testing in the same proportion, supporting further the feasibility of the Memory Game as a tool to probe both STM and LTM. In addition, a significantly decreased performance in individuals with NDD compared to TD was found.

    Benefits of Web-based Testing

    We believe that the tablet approach makes cognitive testing more engaging for children. Evidently, we observed during our initial pilot testing (done in person) that most children enjoyed the game. In addition, by using pictures, we could help test individuals with limited literacy and language development, who could not perform traditional word-matching tasks. The task could be extended also to individuals using other languages; however, this will require updating the instruction video. Considering the prevalence of anxiety in individuals with NDD [67,81], we propose that remote testing with the Memory Game, being complete in the individual’s familiar environment, may better capture the full potential of individuals with NDD.

    Web-based testing on participants’ devices is important as remote testing has been increasingly considered by researchers to resolve problems with participant compensation for travel, geographical issues in rare disorders, and recruiting large numbers of participants.

    While this work focuses on clinical diagnosis (NDD), having access to a larger number of participants will be key in future work focused on specific genes or syndromes. In addition, the closer proximity of the cognitive measures tested by the Memory Game (LTM and paired association) could facilitate the translation of findings from animal models.

    Current Limitations

    One of the trade-offs of remote web-based testing is in ascertaining the behavior of the participant as one would do with in-person testing. For instance, an individual may be distracted and perform suboptimally. The participant and caregiver may also not understand the instruction video, and thus may not be able to complete the task optimally. While we aimed to develop the instruction video with the feedback of caregivers during the early pilot testing in the clinic, it is possible that some users may not have understood it well. It is also challenging to assess how much and what type of guidance was provided by caregivers. Another important technical improvement to consider would be the flow of the test so that the next step is dependent and determined by the performance of the participant in the practice trial. At the moment, even though the participant receives feedback, they can still proceed to the testing phase regardless of their previous performance. It may be important to redemonstrate to the participant the tutorial or to train them further before they could proceed to the testing phase.

    Some aspects of NDD such as repetitive behavior, or motor delay, may also impact the performance and thus, not reflect their memory capacity accurately. In addition, visual impairment could also influence the ability of participants to recognize the pictures and should be included in the future version as an item that would be reported by caregivers. Web-based testing also relies on self-reported diagnosis by caregivers, which might be incorrect. Therefore, future testing will include a correlation with in-person psychometric testing. While testing in larger cohorts may allow limiting the proportion of such outliers, smaller cohorts may be more impacted by such types of limitations. We also proposed an outlier analysis to identify individuals who may have experienced technical difficulties or stopped the test due to lack of attention or compliance, so as not to bias the results. Future research will be needed to develop adjustments for technical aspects, such as internet speed, device types, and browsers, which could also affect the results. Finally, one could also record the individual while performing the task. This has been implemented widely, for instance, during COVID-19 for web-based test administration in universities, but this presents many privacy issues and therefore was not used in the Memory Game.

    In addition, despite the Memory Game allowing extensive testing by removing potential physical or geographical barriers, it remains challenging to achieve diversity of age, sex, gender, and diagnosis. Moreover, despite being potentially easier by being image-based rather than text-based paired association tasks, we recognize that many individuals with NDD may still not be able to perform the testing. In order to be of extensive use, the tutorial video will need to be translated, as well as some of the buttons, but the pictures used will be readily reusable.

    This study also measured performance across different age groups but does not provide longitudinal data about memory performance in function of previous performance. Therefore, longitudinal follow-up will be important. We also recognize the importance of including individuals with diverse NDD etiology or a larger sample size of individuals, with a given specific NDD diagnosis to further assess diversity in the future and correlate the Memory Game findings with other cognitive tests such as for attention, working memory, intelligence quotient, or simpler form of plasticity (sensitization and habituation).

    Future research will be needed to correlate the Memory Game performance with specific diagnoses and identify if the genes involved more specifically in LTM in animal models would affect LTM more than STM. In addition, repeated training has been shown to be important for LTM in typically developing individuals, for instance, when learning a language [82,83]. It will be interesting to evaluate if individuals with NDD would perform similarly on such tasks; however, we did not include those paradigms at this time based on the feedback of families in our initial development phase mentioning the importance of keeping the task short.

    Acknowledgments

    We would like to thank Dr Achim and Beaulieu for their help with the BOSS database of pictures. We would also like to thank Drs Tim Tully and Mary Pyc for providing constructive feedback for the design of the Memory Game.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Online consent. The participants (or caregivers) first see the landing page with the Memory Game logo and are then directed to a consent form (approved by the University of Alberta). The participant (or caregiver) can then decide whether to give consent or not.

    PNG File , 186 KB
    ‎
    Multimedia Appendix 2

    Comparative reaction time right after training (day 1) compared to performance the day after (day 2). (A) Reaction time (RT) of individuals with typical development (TD) and neurodevelopmental disorders (NDD) in the recognition (R) task by age group. (B) RT in the R task in individuals with TD and NDD by age group. (C) RT for the paired association (PA) task in individuals with TD and NDD by age group. (D) RT for the PA task in individuals with TD and NDD by age group. TD ages 5-9 years, day 1: n=97; TD ages 5-9 years, day 2: n=56; TD ages 10-14 years, day 1: n=31; TD ages 10-14 years, day 2: n=17; NDD ages 5-9 years, day 1: n=30; NDD ages 5-9 years, day 2: n=16; NDD ages 10-14 years, day 1: n=27; NDD ages 10-14 years, day 2: n=19.

    PNG File , 125 KB
    ‎
    Multimedia Appendix 3

    Distribution of reaction time on day 1 and day 2 for individuals with typical development (TD) and neurodevelopmental disorders (NDD). (A) Reaction time (RT) for recognition (R) of individuals with TD of the 5-9 years old group. (B) RT for recognition of individuals with NDD of the 5-9 years old group. (C) RT for R for individuals with typical development of the 10-14 years old group. (D) RT for R for individuals with NDD of the 10-14 years old group. (E) RT for paired association (PA) for individuals with TD of the 5-9 years old group. (F) RT for PA for individuals with NDD of the 5-9 years old group. G) RT for PA for individuals with TD of the 10-14 years old group. H) RT for PA for individuals with NDD of the 10-14 years old group. TD ages 5-9 years, day 1: n=97; TD ages 5-9 years, day 2: n=56; TD ages 10-14 years, day 1: n=31; TD ages 10-14 years, day 2: n=17; NDD ages 5-9 years, day 1: n=30; NDD ages 5-9 years, day 2: n=16; NDD ages 10-14 years, day 1: n=27; NDD ages 10-14 years, day 2: n=19.

    PNG File , 165 KB
    1. Thapar A, Cooper M, Rutter M. Neurodevelopmental disorders. Lancet Psychiatry 2017;4(4):339-346. [CrossRef] [Medline]
    2. Jeste SS. Neurodevelopmental behavioral and cognitive disorders. Continuum (Minneap Minn) 2015;21(3):690-714. [CrossRef] [Medline]
    3. Zablotsky B, Black LI, Maenner MJ, Schieve LA, Danielson ML, Bitsko RH, et al. Prevalence and trends of developmental disabilities among children in the United States: 2009-2017. Pediatrics 2019;144(4):e20190811 [FREE Full text] [CrossRef] [Medline]
    4. Arora NK, Nair MKC, Gulati S, Deshmukh V, Mohapatra A, Mishra D, et al. Neurodevelopmental disorders in children aged 2-9 years: population-based burden estimates across five regions in India. PLoS Med 2018;15(7):e1002615 [FREE Full text] [CrossRef] [Medline]
    5. Emerson E. Deprivation, ethnicity and the prevalence of intellectual and developmental disabilities. J Epidemiol Community Health 2012;66(3):218-224. [CrossRef] [Medline]
    6. Taylor E. Developing ADHD. J Child Psychol Psychiatry 2009;50(1-2):126-132. [CrossRef] [Medline]
    7. Johnson S, Fawke J, Hennessy E, Rowell V, Thomas S, Wolke D, et al. Neurodevelopmental disability through 11 years of age in children born before 26 weeks of gestation. Pediatrics 2009;124(2):e249-e257. [CrossRef] [Medline]
    8. Zauche LH, Darcy Mahoney AE, Higgins MK. Predictors of co-occurring neurodevelopmental disabilities in children with autism spectrum disorders. J Pediatr Nurs 2017;35:113-119. [CrossRef] [Medline]
    9. Hansen BH, Oerbeck B, Skirbekk B, Petrovski BÉ, Kristensen H. Neurodevelopmental disorders: prevalence and comorbidity in children referred to mental health services. Nord J Psychiatry 2018;72(4):285-291. [CrossRef] [Medline]
    10. Tatishvili N, Gabunia M, Laliani N, Tatishvili S. Epidemiology of neurodevelopmental disorders in 2 years old Georgian children. Pilot study - population based prospective study in a randomly chosen sample. Eur J Paediatr Neurol 2010;14(3):247-252. [CrossRef] [Medline]
    11. Knüppel A, Telléus GK, Jakobsen H, Lauritsen MB. Quality of life in adolescents and adults with autism spectrum disorder: results from a nationwide Danish survey using self-reports and parental proxy-reports. Res Dev Disabil 2018;83:247-259 [FREE Full text] [CrossRef] [Medline]
    12. Alves NS, Gavina VP, Cortellazzi KL, Antunes LAA, Silveira FM, Assaf AV. Analysis of clinical, demographic, socioeconomic, and psychosocial determinants of quality of life of persons with intellectual disability: a cross-sectional study. Spec Care Dentist 2016;36(6):307-314. [CrossRef] [Medline]
    13. Bastiaansen D, Koot HM, Ferdinand RF, Verhulst FC. Quality of life in children with psychiatric disorders: self-, parent, and clinician report. J Am Acad Child Adolesc Psychiatry 2004;43(2):221-230. [CrossRef] [Medline]
    14. Kripke C. Adults with developmental disabilities: a comprehensive approach to medical care. Am Fam Physician 2018;97(10):649-656 [FREE Full text] [Medline]
    15. Tello JA, Williams HE, Eppler RM, Steinhilb ML, Khanna M. Animal models of neurodegenerative disease: recent advances in fly highlight innovative approaches to drug discovery. Front Mol Neurosci 2022;15:883358 [FREE Full text] [CrossRef] [Medline]
    16. Molenhuis RT, Bruining H, Brandt MJV, van Soldt PE, Abu-Toamih Atamni HJ, Burbach JPH, et al. Modeling the quantitative nature of neurodevelopmental disorders using collaborative cross mice. Mol Autism 2018;9:63 [FREE Full text] [CrossRef] [Medline]
    17. Doi M, Li M, Usui N, Shimada S. Genomic strategies for understanding the pathophysiology of autism spectrum disorder. Front Mol Neurosci 2022;15:930941 [FREE Full text] [CrossRef] [Medline]
    18. Stephan M, Volkmann P, Rossner MJ. Assessing behavior and cognition in rodents, nonhuman primates, and humans: where are the limits of translation? Dialogues Clin Neurosci 2019;21(3):249-259 [FREE Full text] [CrossRef] [Medline]
    19. Zhao X, Bhattacharyya A. Human models are needed for studying human neurodevelopmental disorders. Am J Hum Genet 2018;103(6):829-857 [FREE Full text] [CrossRef] [Medline]
    20. Parenti I, Rabaneda LG, Schoen H, Novarino G. Neurodevelopmental disorders: from genetics to functional pathways. Trends Neurosci 2020;43(8):608-621 [FREE Full text] [CrossRef] [Medline]
    21. Kumar S, Reynolds K, Ji Y, Gu R, Rai S, Zhou CJ. Impaired neurodevelopmental pathways in autism spectrum disorder: a review of signaling mechanisms and crosstalk. J Neurodev Disord 2019;11(1):10 [FREE Full text] [CrossRef] [Medline]
    22. Messbauer VCS, de Jong PF. Word, nonword, and visual paired associate learning in Dutch dyslexic children. J Exp Child Psychol 2003;84(2):77-96. [CrossRef] [Medline]
    23. Liu C, Chung KKH. The relationships between paired associate learning and Chinese word writing in kindergarten children. Read Writ 2021;34:2127-2148 [FREE Full text] [CrossRef]
    24. Atkinson RC, Shiffrin RM. Human memory: a proposed system and its control processes. Psychol Learn Motiv 1968;2:89-195. [CrossRef]
    25. Androschuk A, Al-Jabri B, Bolduc FV. From learning to memory: what flies can tell us about intellectual disability treatment. Front Psychiatry 2015;6:85 [FREE Full text] [CrossRef] [Medline]
    26. Bolduc FV, Tully T. Fruit flies and intellectual disability. Fly 2009;3(1):91-104 [FREE Full text] [CrossRef] [Medline]
    27. Androschuk A, Bolduc FV. Modeling intellectual disability in drosophila. In: Yager J, editor. Animal Models of Neurodevelopmental Disorders. New York: Humana Press; 2015:215-237.
    28. Seifan A, Shih C, Hackett K, Pensack MJ, Schelke MW, Lin M, et al. Detection of neurodevelopmental diversity in memory clinics-validation of a self-report measure. Res Dev Disabil 2018;77:60-67. [CrossRef] [Medline]
    29. Gathercole SE, Alloway TP. Practitioner review: short-term and working memory impairments in neurodevelopmental disorders: diagnosis and remedial support. J Child Psychol Psychiatry 2006;47(1):4-15. [CrossRef] [Medline]
    30. Ebbinghaus H. Memory: A Contribution to Experimental Psychology. New York: Teachers College Press; 1913.
    31. Cowan N. What are the differences between long-term, short-term, and working memory? Prog Brain Res 2008;169:323-338 [FREE Full text] [CrossRef] [Medline]
    32. Munir F, Cornish KM, Wilding J. Nature of the working memory deficit in fragile-X syndrome. Brain Cogn 2000;44(3):387-401. [CrossRef] [Medline]
    33. Munir F, Cornish KM, Wilding J. A neuropsychological profile of attention deficits in young males with fragile X syndrome. Neuropsychologia 2000;38(9):1261-1270. [CrossRef] [Medline]
    34. Baker S, Hooper S, Skinner M, Hatton D, Schaaf J, Ornstein P, et al. Working memory subsystems and task complexity in young boys with fragile X syndrome. J Intellect Disabil Res 2011;55(1):19-29 [FREE Full text] [CrossRef] [Medline]
    35. Menon V, Leroux J, White CD, Reiss AL. Frontostriatal deficits in fragile X syndrome: relation to FMR1 gene expression. Proc Natl Acad Sci U S A 2004;101(10):3615-3620 [FREE Full text] [CrossRef] [Medline]
    36. Buckley S, Broadley I, MacDonald J. Working memory in children with Down Syndrome. Down Syndr Res Pract 1995;3(1):3-8 [FREE Full text] [CrossRef]
    37. Lukowski AF, Milojevich HM, Eales L. Cognitive functioning in children with Down Syndrome: current knowledge and future directions. Adv Child Dev Behav 2019;56:257-289. [CrossRef] [Medline]
    38. Farran EK, Jarrold C, Gathercole SE. Block design performance in the Williams syndrome phenotype: a problem with mental imagery? J Child Psychol Psychiatry 2001;42(6):719-728. [CrossRef] [Medline]
    39. Ellis Weismer S, Evans J, Hesketh LJ. An examination of verbal working memory capacity in children with specific language impairment. J Speech Lang Hear Res 1999;42(5):1249-1260. [CrossRef] [Medline]
    40. Ellis Weismer S, Tomblin JB, Zhang X, Buckwalter P, Chynoweth JG, Jones M. Nonword repetition performance in school-age children with and without language impairment. J Speech Lang Hear Res 2000;43(4):865-878. [CrossRef] [Medline]
    41. Geurts HM, Verté S, Oosterlaan J, Roeyers H, Sergeant JA. How specific are executive functioning deficits in attention deficit hyperactivity disorder and autism? J Child Psychol Psychiatry 2004;45(4):836-854. [CrossRef] [Medline]
    42. Fahie CM, Symons DK. Executive functioning and theory of mind in children clinically referred for attention and behavior problems. J Appl Dev Psychol 2003;24(1):51-73 [FREE Full text] [CrossRef]
    43. Kerns KA, McInerney RJ, Wilde NJ. Time reproduction, working memory, and behavioral inhibition in children with ADHD. Child Neuropsychol 2001;7(1):21-31. [CrossRef] [Medline]
    44. Rabiee A, Vasaghi-Gharamaleki B, Samadi SA, Amiri-Shavaki Y, Alaghband-Rad J. Working memory deficits and its relationship to autism spectrum disorders. Iran J Med Sci 2020;45(2):100-109 [FREE Full text] [CrossRef] [Medline]
    45. Van der Molen MJ, Van Luit JE, Van der Molen MW, Jongmans MJ. Everyday memory and working memory in adolescents with mild intellectual disability. Am J Intellect Dev Disabil 2010;115(3):207-217. [CrossRef] [Medline]
    46. Palomino E, López-Frutos JM, Botella J, Sotillo M. Impairment of cognitive memory inhibition in individuals with intellectual disability: a meta-analysis. Psicothema 2019;31(4):384-392. [CrossRef] [Medline]
    47. Carretti B, Belacchi C, Cornoldi C. Difficulties in working memory updating in individuals with intellectual disability. J Intellect Disabil Res 2010;54(4):337-345. [CrossRef] [Medline]
    48. Schuchardt K, Gebhardt M, Mäehler C. Working memory functions in children with different degrees of intellectual disability. J Intellect Disabil Res 2010;54(4):346-353. [CrossRef] [Medline]
    49. Carlesimo GA, Marotta L, Vicari S. Long-term memory in mental retardation: evidence for a specific impairment in subjects with Down's syndrome. Neuropsychologia 1997;35(1):71-79. [CrossRef] [Medline]
    50. Byrne JH, editor. Learning and Memory: A Comprehensive Reference. Cambridge, Massachusetts: Academic Press; 2008.
    51. Manns JR, Hopkins RO, Squire LR. Semantic memory and the human hippocampus. Neuron 2003;38(1):127-133.
    52. Squire LR, Zola SM. Episodic memory, semantic memory, and amnesia. Hippocampus 1998;8(3):205-211. [CrossRef]
    53. Lee I, Solivan F. The roles of the medial prefrontal cortex and hippocampus in a spatial paired-association task. Learn Mem 2008;15(5):357-367 [FREE Full text] [CrossRef] [Medline]
    54. Yoon J, Seo Y, Kim J, Lee I. Hippocampus is required for paired associate memory with neither delay nor trial uniqueness. Learn Mem 2011;19(1):1-8 [FREE Full text] [CrossRef] [Medline]
    55. Brimo K, Dinkler L, Gillberg C, Lichtenstein P, Lundström S, Åsberg Johnels J. The co-occurrence of neurodevelopmental problems in dyslexia. Dyslexia 2021;27(3):277-293. [CrossRef] [Medline]
    56. Rodríguez C, Areces D, García T, Cueli M, Gonzalez-Castro P. Neurodevelopmental disorders: an innovative perspective the response to intervention model. World J Psychiatry 2021;11(11):1017-1026 [FREE Full text] [CrossRef] [Medline]
    57. Romani C, Di Betta AM, Tsouknida E, Olson A. Lexical and nonlexical processing in developmental dyslexia: a case for different resources and different impairments. Cogn Neuropsychol 2008;25(6):798-830. [CrossRef] [Medline]
    58. Eden GF, Olulade OA, Evans TM, Krafnick AJ, Alkire DR. Developmental dyslexia. In: Neurobiology of Language. Cambridge, Massachusetts: Academic Press; 2016:815-826.
    59. Millward LJ, Bryan K, Everatt J, Collins R. Clinicians and dyslexia--a computer-based assessment of one of the key cognitive skills involved in drug administration. Int J Nurs Stud 2005;42(3):341-353 [FREE Full text] [CrossRef] [Medline]
    60. FragaGonzález G, Karipidis II, Tijms J. Dyslexia as a neurodevelopmental disorder and what makes it different from a chess disorder. Brain Sci 2018;8(10):189 [FREE Full text] [CrossRef] [Medline]
    61. Eriksen BA, Eriksen CW. Effects of noise letters upon the identification of a target letter in a nonsearch task. Percept psychophys 1974;16:143-149. [CrossRef]
    62. Yoon T, Graham LK, Kim JJ. Hippocampal lesion effects on occasion setting by contextual and discrete stimuli. Neurobiol Learn Mem 2011;95(2):176-184 [FREE Full text] [CrossRef] [Medline]
    63. Düzel E, Vargha-Khadem F, Heinze HJ, Mishkin M. Brain activity evidence for recognition without recollection after early hippocampal damage. Proc Natl Acad Sci U S A 2001;98(14):8101-8106 [FREE Full text] [CrossRef] [Medline]
    64. Weintraub S, Dikmen SS, Heaton RK, Tulsky DS, Zelazo PD, Bauer PJ, et al. Cognition assessment using the NIH toolbox. Neurology 2013;80(11 suppl 3):S54-S64 [FREE Full text] [CrossRef] [Medline]
    65. Tulsky DS, Carlozzi N, Chiaravalloti ND, Beaumont JL, Kisala PA, Mungas D, et al. NIH toolbox cognition battery (NIHTB-CB): list sorting test to measure working memory. J Int Neuropsychol Soc 2014;20(6):599-610 [FREE Full text] [CrossRef] [Medline]
    66. Berry-Kravis EM, Harnett MD, Reines SA, Reese MA, Ethridge LE, Outterson AH, et al. Inhibition of phosphodiesterase-4D in adults with fragile X syndrome: a randomized, placebo-controlled, phase 2 clinical trial. Nat Med 2021;27(5):862-870. [CrossRef] [Medline]
    67. Goldberg MC. CANTAB. In: Volkmar FR, editor. Encyclopedia of Autism Spectrum Disorders. New York: Springer; 2013:519-522.
    68. Gau SS, Shang CY. Executive functions as endophenotypes in ADHD: evidence from the Cambridge Neuropsychological Test Battery (CANTAB). J Child Psychol Psychiatry 2010;51(7):838-849. [CrossRef] [Medline]
    69. Fried R, Hirshfeld-Becker D, Petty C, Batchelder H, Biederman J. How informative is the CANTAB to assess executive functioning in children with ADHD? A controlled study. J Atten Disord 2015;19(6):468-475. [CrossRef] [Medline]
    70. Lerwick JL. The Impact of Child-Centered Play Therapy on Anxiety Levels in Pre-Neurosurgical Pediatric Patients. Ann Arbor, Michigan: ProQuest LLC; 2012.
    71. Smith ML. Interventions to minimize distress during pediatric primary care visits: a systematic literature review. BYU: Brigham Young University. 2014.   URL: https://scholarsarchive.byu.edu/studentpub/4/ [accessed 2022-11-28]
    72. Erikson EH. Childhood and Society. 2nd ed. New York: Norton; 1963.
    73. Brown GW. Measurement and the epidemiology of childhood trauma. Semin Clin Neuropsychiatry 2002;7(2):66-79. [CrossRef] [Medline]
    74. Pine DS, Cohen JA. Trauma in children and adolescents: risk and treatment of psychiatric sequelae. Biol Psychiatry 2002;51(7):519-531. [CrossRef] [Medline]
    75. Rodriguez CM, Clough V, Gowda AS, Tucker MC. Multimethod assessment of children's distress during noninvasive outpatient medical procedures: child and parent attitudes and factors. J Pediatr Psychol 2012;37(5):557-566. [CrossRef] [Medline]
    76. Morse BA. Handbook of Play Therapy. Edited by Charles E. Schaefer and Kevin J. O'Connor. New York: John Wiley and Sons, 1983. 489 pp. $34.95. Child Sch 1985;7(4):266-267. [CrossRef]
    77. E-Prime®. Psychology Software Tools: Solutions for Research, Assessment, and Education. 2017.   URL: https://pstnet.com/products/e-prime/ [accessed 2023-01-11]
    78. The Memory Game. Dart NeuroScience, LLC.   URL: https://memory-game.med.ualberta.ca [accessed 2023-02-27]
    79. Iglewicz B, Hoaglin DC. How to Detect and Handle Outliers. Milwaukee, Wisconsin: ASQC Quality Press; 1993.
    80. Riggins T, Rollins L. Developmental differences in memory during early childhood: insights from event-related potentials. Child Dev 2015;86(3):889-902 [FREE Full text] [CrossRef] [Medline]
    81. Kerns CM, Renno P, Storch EA, Kendall PC, Wood JJ, editors. Anxiety in Children and Adolescents with Autism Spectrum Disorder: Evidence-Based Assessment and Treatment. Cambridge, Massachusetts: Academic Press; 2017.
    82. Kelley P, Whatson T. Making long-term memories in minutes: a spaced learning pattern from memory research in education. Front Hum Neurosci 2013;7:589 [FREE Full text] [CrossRef] [Medline]
    83. Smith CD, Scarf D. Spacing repetitions over long timescales: a review and a reconsolidation explanation. Front Psychol 2017;8:962 [FREE Full text] [CrossRef] [Medline]

    ‎
    ADHD: attention-deficit/hyperactivity disorder
    API: application programming interface
    ASD: autism spectrum disorder
    BOSS: Bank of Standardized stimuli
    LTM: long-term memory
    NDD: neurodevelopmental disorders
    RT: reaction time
    STM: short-term memory
    TD: typical development

    Edited by S Badawy; submitted 20.05.22; peer-reviewed by K Shamlian, J Louw; comments to author 24.06.22; revised version received 13.12.22; accepted 20.01.23; published 08.05.23

    Copyright

    ©Truong An Bui, Cory Scott Rosenfelt, Kerri Hope Whitlock, Mickael Leclercq, Savannah Weber, Arnaud Droit, Sandra A Wiebe, Jacqueline Pei, Francois V Bolduc. Originally published in JMIR Pediatrics and Parenting (https://pediatrics.jmir.org), 08.05.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.