Dear Xavier, apologize for the delay in answering your question. Please check out our article at this link for the methodology: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5547172/

Dear Rustam;
Thank you for your special attention and constructive response. It showed great interest. I do apologize for the delay in answering these questions.
The NIIS has been scaled up nationwide since Mar 2017, all facilities across the country provide birth delivery and/or vaccination services have to use the system to update data (immunization and stock data). As you said, staff training is not a successful story, training is though a necessary activity for successful scale-up.
1. In Vietnam, we have to maintain a paper-system because they have to comply with policies from different Ministries, they request printed reports with leaders’ signature and stamp that is reason to keep paper-system now. In some of our project provinces, we advocated for local government to accept paperless reports, but some a hard-copied ledger is still needed because they need approval from ministries.
I also agree with you that there is still room for data quality improvement of EIR system and now we are working on this issue by different approaches like developing and introducing standard operation procedures, request all facilities have to provide barcode labels ID for children; introducing e-immunization card to help parents track their children vaccination history and also enable interaction with service providers, parents can report wrong information entered and request correction, etc.
With a paper-system, staff cannot track all vaccination history if children get vaccinated at different facilities, and there is no evidence show that data quality of the paper-system better than digital one. Tentatively, end of 2020, we will do an evaluation to compare data quality between paper- system with digital system.
2. When we piloted in 1 district, with technical support from only the project team, we found that we could not support end-users in time if there was no technical support at local. Thus, we provided TOT training to build capacity for district and provincial level health workers so they could support grass-root level health workers, we also introduced application (Teamviewers, chrome remote desktop) help to support end-users remotely during the implementation when we scaled up to entire province in 2014. This method we continued to use for nationwide scale-up, so cascade training is very important to sustain as well as to scale up the system. Nowadays, social media network like Facebook and Zalo are more popular so we use these channels to connect end-users together so they can share experience, peer-training, report bugs, provide feedback for improvement. These channels are very effective and helpful in Vietnam.
3. Computers are available at all facilities across countries to use for many health programs not only the NIIS and local governments do allocate annual small budget for replacing broken devices and maintenance fees for computers, printers, etc. at facilities. For some remote facilities, local governments mobilize budget from local and international organizations to equip computers for facilities. We plan to conduct a cost analysis for the implementation of the NIIS, to estimate the co-share cost for the NIIS, including the cost for devices, electricity, internet connection, labor cost, training cost.
4. We developed and introduced the implementation guideline, in that supportive supervision activity is recommended. Provincial staff usually combine monitoring trips to supervise/inspect several health programs, they list checking the implementation and use of the NIIS as one iterm for the trips and they use a checklist for the NIIS use, and checking data quality is one of the items that supervision is conducted every week.
Besides on-site supervision, on the NIIS, we have a module to support managers at higher-level to monitor and evaluate health facility performance of the implementation, to check on health workers' data entry for timeliness and duplication, etc.
5. And certainly, data quality is very important so all provinces have strengthened supervision onsite and virtually to increase data quality. Low up-take of the NIIS at big fee-based facilities is our challenge since they are very busy, and they use their own system to enter data. To encourage their involvement in the implementation, we worked with MNO to develop API to support exchange data between the NIIS and their system, we also have been working with local government to issue regulations with a penalty if facilities don’t enter data into the system as required.
We find it a continuing process of improvement. We are still working on the transition to paperless nationwide, engaging the increasing private sector, ensuring the data quality, upgrading different features and functions of the systems.
Hope this addresses some of your concerns. Happy to provide more details as needed.
Again, Thanks for your comments.

Dear Xavier,
I support your suggestion, as it is important to understand better how to measure success of EIR solutions, and have evidence on what’s working and not working.

There are 3 categories of measurements that are proposed for EIR:
1) To measure direct benefits of EIR, such as data quality, reduced admin time for health workers, cost-effectiveness of the tool, and effective operations of the tool by existing health system structure.
2) To measure how data is used from IER for planning and improvement actions
3) To measure the health outcomes, such as coverage and drop-out rates

When EIR is being implemented and has started being used by health workers, we should measure the 1st category of indicators, related to data quality improvements, reduction in admin time and evaluating whether existing health system structures are operating the solution independently from the partners

If the project or the initiative additionally implements some data use processes, such as defaulter tracing processes or improved supply chain processes, then it makes sense to measure data use.

And when the data is used to do improvement actions, we can then evaluate the 3rd category, which is health outcomes, or in immunisation space, improved timeliness, coverage and equity.

Some people assume and believe that EIRs will miraculously solve problems in all 3 categories, improving data quality, data use and health outcomes. What’s important to understand is that EIR is just a tool and when choosing an EIR, one should look at the evidence from the 1st category of indicators. For instance, defaulter tracing won’t happen, until there is a process and resources in place to do defaulter tracing. This process won’t come as part of EIR tool and has to be established as well. Or, if there are no resources to strengthen supply chains, EIR as a tool can’t improve availability of vaccines. We should be able to derive causality, and we all understand that EIR as a tool cannot minimise number of defaulters, unless there are additional interventions put in place, such as defaulter tracing mechanisms.

Therefore, I think we need to agree on how to measure success of EIRs and based on that see what’s working and not working.

External evaluations show that fully digital EIRs that require a mobile device at the facility level have low data quality (accuracy, completeness), due to the issues we saw in Vietnam and other countries. Those issues are cause not because of EIR itself, but due to external factors, such as lack of technical support, duplication of efforts by health workers, lack of resources to operate the solution and timely replacement of mobile devices. Additionally, the costs to operate range from 3 USD to 5 USD per child/year, which means that most Governments in low-income countries won’t be able to afford those costs.

The other alternative is to have a hybrid (paper-to-digital) EIRs. There are several external evaluations and reports done from different countries and the results are promising. But there is no peer-reviewed publication yet.

In summary, EIR is a tool and we should separate different categories of indicators when measuring the success of EIRs. If we bundle all 3 categories of indicators together, then we are asking from EIR things that it is not able to deliver by itself.