Routine childhood immunization is the cornerstone of an efficient public health system and childhood disease prevention. Despite progress in improving routine immunization coverage, in 2022, 17% and 7% of children aged 0 to 23 months in Pakistan and Bangladesh, respectively, did not receive the third dose of the diphtheria-tetanus-pertussis vaccine [1]. The problem is further exacerbated for children who do visit health facilities in low- and middle-income countries (LMICs), but the immunization system fails to provide all age-appropriate vaccines to one in every 3 of these children [2]. This creates missed opportunities for vaccination (MOVs), where despite being eligible for vaccination, a child is not administered one or more of the vaccine doses [3]. MOVs remain a rampant problem in many LMICs [4,5], leading to underimmunization and delayed vaccination of children.

A systematic review of MOVs across LMICs showed a pooled MOV prevalence of 32.2% (95% CI 26.8%-37.7%) among children and 46.9% (95% CI 29.7%-64.0%) among women of childbearing age [2]. MOV prevalence estimates are even higher for some countries, ranging from 42% to 89% [6,7]. A study from Pakistan reported that despite being eligible for the pentavalent vaccine (Penta) at the measles vaccine visit, 34% of children did not receive the required dose of Penta even though they visited the immunization clinic and were vaccinated for measles [8]. Lack of awareness among parents and providers is one of the primary causes of MOVs, with vaccinator confusion about contraindications and the immunization schedule being the main factors [2,9]. In the absence of frequent refresher training, improper health care worker practices lead to repeated MOVs in LMICs and high-income countries [10,11]. However, regardless of the amount of training, scheduling age-appropriate vaccinations, especially catch-up schedules (when the child has missed or delayed one or more vaccines), is a complex task [12,13]. A study from Illinois showed that 33% of all health workers constructed incorrect hypothetical catch-up schedules in a survey designed to determine health workers’ knowledge of catch-up immunizations [10]. In LMICs, where vaccinators are more likely to be overburdened and pressed for time [14], the likelihood of making errors in scheduling is even higher.

Decision support systems have proven to be powerful tools for improving clinical care and patient outcomes across various domains [15,16]. They have contributed to improving evidence-based medical practices, reducing medical errors, and improving adherence to standard care practices [17,18]. At the basic level, an immunization decision support system (iDSS) provides patient-specific vaccination recommendations at each immunization visit, considering the child’s date of birth and vaccination history and ensuring the appropriate scheduling of feasible doses.

Currently, examples of iDSS deployment are limited and almost exclusively constitute complex systems from high-income country settings. For example, in the United States, clinical decision support systems based on national standard guidelines for immunization are embedded in regional immunization information systems [19] or web-based immunization forecasting services accessible by various electronic health records [20]. There is a dearth of literature on contextually appropriate, efficient, and practical mobile-based iDSS that is easily implementable within LMICs.

We aimed to analyze the diagnostic accuracy of an innovative mobile-based iDSS in an LMIC setting to construct age-appropriate vaccination schedules for children aged 0 to 23 months and compared with the gold standard of evaluation (World Health Organization [WHO]–recommended Expanded Programme on Immunization [EPI] schedule constructed by a vaccines expert). We also aimed to generate preliminary evidence for MOVs among visiting children (caused by inaccurate vaccination scheduling by vaccinators) that could be reduced through iDSS by comparing the manual schedules constructed by vaccinators with the gold standard. Finally, we reported the findings on the vaccinators’ understanding, perceived usability, and acceptability of the iDSS.

We implemented a sequential exploratory mixed methods study design with a quantitative component preceding the qualitative interviews. In the quantitative component, the children’s demographic details and immunization history were recorded by trained study staff using iDSS. The iDSS used this information to formulate an age-appropriate immunization schedule for the children, and it was recorded on the back end and was not visible to the study staff (or vaccinators). Simultaneously, the study staff also captured the manually constructed immunization schedules determined by the vaccinators as indicated on the child’s government-issued immunization cards. Through this process, we were able to capture both the iDSS and vaccinator schedules simultaneously for the same child (antigen doses). We used this information to assess the diagnostic accuracy of the iDSS algorithm by comparing the age-appropriate immunization schedules constructed by the iDSS for children aged 0 to 23 months with the gold standard of evaluation (WHO-recommended EPI schedule constructed by a vaccine expert). We also independently compared the vaccine schedules constructed manually by the vaccinators for the same children with the gold standard. This allowed us to generate preliminary evidence of MOVs resulting from inaccurate vaccination schedules constructed by vaccinators. This phase was followed by a qualitative phase in which the vaccinators were provided with iDSS-enabled study phones. After vaccinators had a chance to use the iDSS, we conducted in-depth interviews with vaccinators at the participating immunization centers regarding their experience of using the iDSS, its perceived utility, and acceptability.

We conducted a multicountry study at 6 immunization centers, 3 each in Pakistan and Bangladesh. Both countries vary in terms of full immunization coverage rate (Pakistan 66%; Bangladesh 84%) and infant mortality rates (81/1000 live births in Pakistan; 43/1000 live births in Bangladesh) [21,22]. Immunization centers were selected based on the influx of children, availability of vaccines and vaccinators, and absence of any kind of decision support system at these sites. The selected study sites had high penetration of mobile phones (>90%) and the presence of cellular networks (data connectivity) among the population [23,24].

Two of the selected immunization centers in Pakistan were located in Gilgit district in Gilgit Baltistan territory, which had a full immunization coverage rate of 57% among children aged 12 to 23 months in 2018 [21]. The third selected immunization center in Pakistan was a private center located in the Rahim Yar Khan district in Punjab province, with a full immunization coverage rate of 65% in 2019 [23].

In Bangladesh, all 3 immunization centers were located in Dhaka city in Dhaka district. The district had a full immunization coverage rate of 85% among children aged 12 to 23 months as of 2019 [25].

The inclusion criteria for the study were as follows: children must be aged <2 years, visiting any of the 6 selected immunization centers for routine vaccination, and presenting with an immunization card. Exclusion criteria were as follows: children visiting for immunization campaigns and those who did not receive vaccination during their visit. We obtained verbal consent from the caregivers of the eligible children before enrollment.

For the qualitative component, our study population included all vaccinators from the participating immunization centers who used iDSS app during the study. We obtained written consent from the study participants before the interviews.

Ethics approval for the study was obtained from the Institutional Review Boards of the Interactive Research and Development (Pakistan; IRD_IRB_2019_01_010) and Building Resources Across Communities James P. Grant School of Public Health (Bangladesh).

Pakistan’s routine EPI schedule in 2019 included BCG (bacille Calmette-Guérin) vaccine and oral polio vaccine (OPV) at birth; 3 doses of Penta (containing diphtheria-tetanus-pertussis, hepatitis B, Haemophilus influenzae type b vaccine), pneumococcal vaccine (PCV10), and OPV at 6, 10, and 14 weeks; 2 doses of rotavirus vaccine (Rota) at 6 and 10 weeks; a single dose of inactivated polio vaccine (IPV) at 14 weeks; and 2 doses of measles vaccine at 9 and 15 months.

For Bangladesh, the EPI schedule included BCG vaccine at birth; Penta, PCV, and OPV at 6, 10, and 14 weeks; IPV at 6 and 14 weeks; and measles-rubella (MR) vaccine at 9 and 15 months.

We developed an android-based iDSS designed for mobile-based deployment, packaged in the form of an application programming interface (API) to function both as a stand-alone module and interoperable with other digital applications or platforms, such as web-based or mobile-based electronic immunization registries (EIRs).

The iDSS was formulated as a 2-step process comprising a data entry form and display interface showing the proposed vaccination schedule. In the first step, the details of the child, including the date of birth and immunization history, are entered in the data entry form of the iDSS. The algorithm uses this information to construct an age-appropriate immunization schedule (including the vaccines due at the current visit and those to be scheduled) tailored to the child and the respective country’s EPI schedule.

iDSS automatically reformulates the child’s immunization schedule after every visit, adjusting for missed appointments and delayed vaccinations and considering the birth date, previous vaccines, standard interval based on live or inactivated vaccines and interdosing schedule, without the need for any manual calculation by the vaccinators. Changes in the EPI schedule and new vaccines are also incorporated into the iDSS algorithm. The results (current and scheduled vaccines) are instantly displayed on the iDSS interface through a color-coded system that allows for straightforward interpretation, especially by low literacy and overburdened vaccinators (Figure 1). The iDSS proposes the dates for scheduled vaccines (accounting for public holidays and vaccine-specific days at centers) and prompts the vaccinator through warning messages if vaccines are being administered out of schedule to ensure that interdose gap guidelines are followed.

In addition, the iDSS algorithm can be customized to the needs of the country, according to the respective EPI schedule. The iDSS is also multilingual and is developed as open-source software for easy integration and interoperability across a variety of settings and systems. As part of the iDSS development, the module was pretested in-house for quality assurance and with randomly selected vaccinators (outside the study site) for limited field deployment. A detailed overview of the iDSS is provided in Multimedia Appendix 1.

We enrolled eligible children whose accompanying caregivers provided verbal consent for participation. At enrollment, a unique study identifier was allocated to each child, and each follow-up visit by the child to the participating immunization center was recorded as a unique visit. Trained study staff used the iDSS installed on study mobile phones to record the demographics and immunization history of the enrolled children to enable iDSS to schedule their current and future vaccination visits. This information was recorded at the back end of the iDSS, and both the study staff and vaccinators were blinded to it. Simultaneously, the study staff also recorded the vaccination schedules determined by the vaccinators as per routine, which were obtained from the child’s immunization cards. Vaccinators did not use the iDSS during the quantitative phase of the study. The iDSS app was linked to a web-based dashboard that allowed the real-time downloading of data for further analysis.

For the qualitative component of the study, we interviewed 16 vaccinators (11 from Pakistan and 5 from Bangladesh) after obtaining written consent. These vaccinators were provided with mobile phones with iDSS that they could use to schedule current and future vaccinations. Before using the iDSS, vaccinators were given a 2-day training by the study staff on using the iDSS module as part of their daily immunization activities. At the end of the period when the vaccinators had used the iDSS for at least 4 weeks, the study team interviewed the vaccinators. In-person interviews were conducted by Project Managers at both sites, who were qualified public health researchers with a medical background. The interviews were facilitated by Research Associates, who helped to take notes. We used a 16-item semistructured interview guide covering user experience, acceptability, and feasibility of the iDSS feature, and suggestions for improvement. Demographic indicators, including age, education, gender, and years of experience, were also collected from the participants. Each interview lasted between 25 and 40 minutes and was audio-recorded and transcribed for further analysis.

Our primary study outcome was the diagnostic accuracy of the iDSS in constructing age-appropriate vaccination schedules as per the WHO-recommended EPI guidelines. In addition, we examined the vaccinators’ perceived usability and acceptability of the iDSS. A secondary outcome included preliminary estimates of the number of MOVs among children aged 0 to 23 months visiting study immunization centers for routine immunization visits.

The phones used for data collection by the field staff and vaccinators had password locks with an additional level of protection through software sign-in passwords. Data collected through iDSS were uploaded to a secure database server. All the data sent from the phones to the server and back were encrypted in transit using the PBKDF2 algorithm (an industry standard). Access to data on the server was through a password-protected web dashboard interface, and only those involved with the project had access to the data for research purposes.

From July 5, 2019, to April 11, 2020, a total of 6241 immunization visits were recorded from 4613 eligible children. A total of 73% (4557/6241) of visits were recorded from 3 immunization centers in Pakistan, and 27% (1684/6241) of visits were recorded from 3 immunization centers in Bangladesh (Figure 2).

The proportion of male children (2435/4613, 52.8%) enrolled in the study was slightly higher than that of female children (2178/4613, 47.2%) across both the sites (Pakistan: 1720/3197, 53.8%) and Bangladesh (715/1416, 50.5%; Table 1). Most (3348/4613, 72.6%) children enrolled in the study were aged ≤6 months at the time of enrollment, both in Pakistan (2423/3197, 75.8%) and Bangladesh (925/1416, 65.3%).

Table 2 shows the age-appropriate immunization doses scheduled by both the iDSS and vaccinators compared with the gold standard. In Pakistan, we collected data on 13,039 immunization doses for all antigens. The iDSS correctly scheduled 99.8% (13,015/13,039) of all age-appropriate immunization doses that should have been administered at the current visit compared with the gold standard, ranging from 99.2% to 100% for BCG vaccine, OPV-0 to 3, Penta-1 to 3, PCV-1 to 3, Rota-1 to 2, measles vaccine 1, and IPV vaccine doses. Of all the antigens, the proportion of correctly scheduled doses by iDSS was the lowest at 96.7% (202/209) for the measles-2 vaccine. In comparison, the vaccinator correctly scheduled 96.2% (12,545/13,039) of all the immunization doses that should have been administered on the current visit compared with the gold standard, ranging from 97% to 100% for BCG vaccine, OPV-0 to 3, Penta-1 to 3, PCV-1 to 3, Rota-1 to 2, and measles-2. However, the proportion of correctly scheduled doses by vaccinators due on the current visit decreased to 94.5% (362/383) and 66.3% (653/985) for measles-1 and IPV vaccine doses, respectively. Owing to errors in the vaccination schedules constructed by vaccinators, 3.8% (494/13,039) of all due immunization doses were missed by the vaccinators. Among the age-eligible children, the highest proportion of MOVs was for the polio and measles vaccines, where 33.7% (332/985) and 5.5% (21/383) of all due immunization doses, respectively, were missed by vaccinators.

In Bangladesh, we recorded data on 4922 immunization doses. The iDSS correctly scheduled 99.9% (4917/4922) of the age-appropriate immunization doses that should have been administered on the current visit compared with the gold standard, with the lowest proportion of correctly scheduled doses for the measles-2 vaccine (196/201, 97.5%). In comparison, the vaccinator correctly scheduled 98.2% (4833/4922) of all immunization doses that should have been administered on the current visit, ranging from 97.5% to 100% for BCG vaccine, OPV-1 to 3, Penta-1 to 3, PCV-1 to 3, IPV-1, and measles-rubella-2, but dropped to 95.1% (137/144) for measles-rubella-1 and 87.4% (355/406) for IPV-2 vaccine doses. Owing to errors in the vaccination schedules constructed by vaccinators, 1.8% (89/4922) of all due immunization doses were missed by the vaccinators. Similar to the findings in Pakistan, the highest proportion of MOVs was for polio and measles-1 vaccines, where 12.6% (51/406) and 4.9% (7/144) of all due immunization doses, respectively, were missed by vaccinators.

Compared with the gold standard, in Pakistan, the accuracy of the iDSS varied between 94.5% and 100% across vaccines versus 88.5% to 99.4% for the immunization doses scheduled by the vaccinator. The iDSS demonstrated a sensitivity of 97.1% to 100%, whereas the sensitivity of the vaccinator’s scheduling was between 66.3% and 100%. The estimated specificity for iDSS compared with the gold standard was 92.6% to 100%, whereas the specificity for vaccinator scheduling was 84.1% to 100%. The results were similar for the Bangladesh site, with iDSS demonstrating a higher accuracy (97.9%-100%) and a higher sensitivity of 100% compared with the gold standard, whereas estimates of accuracy and sensitivity for vaccinator scheduling were 89.4% to 99.8% and 87.4% to 100%, respectively. The iDSS had a specificity of 97.6% to 100% compared with the gold standard, whereas the specificity for the vaccinator’s scheduling was 83.2% to 100% (Figures 3 and 4).

Across both the sites, receiver operating curve analysis showed that the iDSS had high overall accuracy of scheduling age-appropriate immunization doses (area under the curve ranging from 99%-100% across antigens), whereas for vaccinator schedules, estimates for area under the curve ranged between 88% and 100%.

On the basis of the findings of the diagnostic accuracy of the iDSS in which the sensitivity and specificity were <100% for selected antigens, we updated the iDSS algorithm by fixing the errors related to the vaccine intervals and recommended age for administering the vaccines based on the WHO guidelines.

Our study has demonstrated the diagnostic accuracy and feasibility of iDSS to accurately schedule age-appropriate vaccination doses across a multicountry LMIC setting. We also demonstrated the potential of iDSS to reduce MOVs caused by poor vaccinator adherence to standard immunization schedules and the interpretation of complex immunization schedules. Our study reports positive feedback received from a diverse set of vaccinators with varying levels of education and experience on the usability and acceptability of the iDSS.

Overall, the iDSS had higher accuracy in constructing age-appropriate immunization schedules as per the WHO-recommended EPI guidelines compared with vaccinators. Our study highlights the issue of missed vaccine doses despite children making contact with vaccinators. Reducing MOVs caused by complexity and changes in EPI schedules can improve immunization coverage, timeliness, and equity [14]. One study from the United States estimated that the potentially achievable vaccination coverage for 4+Diphtheria, tetanus, and acellular pertussis vaccine, 4+PCV, and full series of Haemophilus influenzae type b for children aged 19 to 35 months would have been 90% if missed opportunities in the administration of these vaccines were eliminated [26]. From an equity perspective, iDSS can enable children from disadvantaged backgrounds, who are already delayed on their vaccination, to catch-up with the rest through accurate administration of all due vaccine doses when they encounter a vaccinator.

Our findings show that the highest MOVs across Pakistan and Bangladesh sites were for polio and measles vaccines, as 27.5% (383/1391) and 5.3% (28/527) of all due immunization doses, respectively, were missed by vaccinators. A systematic review of MOVs across LMICs also reported the prevalence rate of MOVs with regard to polio vaccines to range from 13.4% to 46.7% [2]. According to the WHO-recommended EPI schedule, Pakistan and Bangladesh administer the IPV vaccine to all children at 14 weeks. However, feedback from vaccinators across both sites revealed that vaccinators considered it to be given alongside vaccines administered with Penta-3 (irrespective of age). This practice delayed the IPV vaccination, as the Penta, OPV, and PCV vaccines depend on previous doses and are often administered beyond 14 weeks of age [27]. These MOVs, especially for polio vaccines, are alarming, as they impede global efforts to eradicate polio. Pakistan remains one of the last 2 polio-endemic countries with a substantial surge in the number of polio cases since 2017 (147 cases in 2019 vs 8 in 2017) [28], putting the country on a failing trajectory and the broader Global Polio Eradication Initiative at risk. Similarly, the measles incidence in the country increased from 24.6 cases per million to 80.4 per million between 2000 and 2018 [29].

Although previous studies have reported examples of EIR in LMIC settings [30,31], not all of them have an in-built immunization scheduling or decision support system [32-34]. Our study adds to this important area by providing evidence of the diagnostic accuracy of a stand-alone mobile-based iDSS implemented across a multicountry LMIC setting. Our results are in line with findings reported from high-income countries where iDSS has improved the scheduling of vaccinations. A hospital-based study in the United States that implemented a computer-based clinical decision support algorithm demonstrated an increase in the tetanus, diphtheria, pertussis vaccination in postpartum women [35]. Our results corroborate this finding, making a strong case for high diagnostic accuracy of iDSS technology in LMIC settings. In addition, the end-user feedback from a diverse set of vaccinators, who varied in terms of gender, education, age, and experience, also confirms the utility and functionality of iDSS, its acceptance, and user satisfaction; however, interviews with vaccinators in our study did reveal that only about half (9/16, 56%) of them found the iDSS easy to use, presumably because of maintaining paper-based and electronic records simultaneously in the study. Switching to electronic records completely is likely to address this concern [14].

In contrast to most web-based iDSS developed and used in high-income countries, the iDSS in this study is packaged in the form of an API that is interoperable with health information systems and allows flexibility of deployment. As EIRs continue to be adapted, incorporating iDSS through APIs should be considered, in line with established standards such as the Pan American Health Organization EIR guidelines [36]. API-based iDSS such as the one used in this study also allows flexibility to adjust to multicountry EPI schedules that change frequently and can also adapt to cosmetic user interface changes such as varying languages and displays.

Integration of an iDSS module in LMIC Immunization Programs can, therefore, yield enormous benefits, not only in terms of reducing MOVs but also in designing accurate catch-up regimens to ensure universal immunization. This is particularly relevant in the context of the huge numbers of children who have missed their vaccine doses during the COVID-19 pandemic and associated lockdowns [18]. Equipping vaccinators with iDSS technology can help ensure that these children are effectively immunized for all due vaccines at subsequent visits, thereby maintaining optimal immunity levels and reducing the likelihood of secondary vaccine-preventable disease outbreaks.

A major strength of our study was the demonstration of the feasibility of the iDSS across 2 different LMIC settings, with varying EPI schedules, infrastructure, coverage levels, vaccinator education, experience, demographics, and immunization-related challenges. One limitation of our study was that the evidence of MOVs generated in this study only accounted for MOVs because of inaccurate vaccination schedules constructed by vaccinators and hence would be an underestimation of the overall MOV prevalence in our study sites.

The iDSS has high diagnostic accuracy for scheduling age-appropriate vaccinations and reducing MOVs; and high acceptability among vaccinators regardless of gender, education, and experience. The iDSS boasts a variety of features for easy adaptability and replicability across LMIC settings. The evidence generated from this study demonstrates the prevalence of MOVs, especially for measles and polio vaccines. iDSS can increase immunization coverage, timeliness, and equity by eliminating these MOVs and help design accurate catch-up regimens to ensure universal immunization, especially in the aftermath of the COVID-19 pandemic. The findings from this study provide the impetus for rigorously evaluating the impact of iDSS through a randomized controlled trial and paving the way for a scaled implementation of this tool across LMICs.