Electronic community health information system usability evaluation in Ethiopia | BMC Medical Informatics and Decision Making

The study’s findings, grounded in usability contextual factors such as user, task, environment, and technology, uncover intriguing insights and present opportunities for enhancing the usability of the next generation of eCHIS applications. On the one hand, we observed a high percentage of agreement scores among participants. The eCHIS usability evaluation agreement score indicates “how well the users learn and use eCHIS to achieve their goals and how satisfied they are with that process [27]. On the other hand, we observed neutrality or disagreement among respondents during the eCHIS evaluation. The disagreement score helps determine the features and functionality of the next generation of eCHIS in Ethiopia, including improving user experience and satisfaction [28,29,30], enhancing eCHIS functionality and performance [29,30,31], and increasing eCHIS adoption (intention to use and continue to use as well as recommend the app to others) [28, 30, 31] and retention, such as reducing user frustration, dissatisfaction, and withdrawal of eCHIS app usage [29, 31]. In this study, the agreement score was calculated based on ‘strongly agree’ and ‘agree’ responses, while the disagreement score was calculated based on ‘undecided,’ ‘disagree,’ and ‘strongly disagree’ responses. Overall, the HEW eCHIS app usability agreement ranges from 41% on the help menu provided by eCHIS to 99% on the home page of HEW eCHIS. HCR eCHIS app agreement ranges from action provided by eCHIS to 26% to 99% of the home page, easy-to-learn language, and available contents of HCR eCHIS. Focal Person eCHIS app usability agreement ranges from 25% session time to 100% agreement on the home page, easy to use, consistent navigation structure, error prevention (the application stops making errors), logical presentation of tasks, and UI style and format.

On the learnability of eCHIS based on nine parameters in the HEW eCHIS app, the highest agreement scores of greater than 90% were observed in the home page, menu names, icons, or buttons of eCHIS, the overall interface consistency of eCHIS, the ease of use of eCHIS, and the available content of eCHIS. In the HCR eCHIS app, except for the graphics of the eCHIS app (82%), all the other parameters scored an agreement of greater than 90%. Whereas, in the FP eCHIS app, all the parameters scored greater than 90%, and two parameters scored 100%, such as the home page and ease of use of the FP eCHIS app. However, the available content of the eCHIS FP app got an agreement score of 83%. Overall, except for the home page and easy-to-use eCHIS FP app, there was a disagreeable response. Specifically, a 15% disagreement score on the ‘easy-to-learn’ eCHIS HEW app, a 14% disagreement score on the graphics of the eCHIS HEW app, an 18% disagreement score on the graphics of the eCHIS HCR app, a 9% disagreement score on the graphics of the eCHIS FP app, and so on. To address this, the study findings emphasize that improving the learnability of eCHIS requires standardizing the user interface (UI), rules, and guidelines based on user experience, such as color, number of clicks, steps, etc., in the next generation of eCHIS applications. Addressing language translation consistency and refinement is also crucial. It is also good to include quick tips and assistance without affecting performance to improve usability. Overall, a well-designed eCHIS should be easy to learn and remember [23, 30].

Efficiency was based on one parameter: the loading performance of the eCHIS app agreement score was 74%, 66%, and 58% on the HEW, HCR, and FP of the eCHIS application, respectively. However, we observed a disagreement score of 26%, 44%, and 42% for HEW, HCR, and FP, respectively. This disagreement may be related to internet connectivity, but it requires further exploration and investigation. Resolving the disagreement regarding the efficiency of eCHIS will enhance the overall user experience. Ultimately, should allow HEW, midwives, and FP to complete activities swiftly while not interfering with their workflow, and it should become a central tool for Ethiopian health information systems [25, 32].

Effectiveness was based on three parameters, all of which scored higher than 90% agreement in the effectiveness of the eCHIS HCR app. The information organization of the eCHIS HEW app and the tasks performed scored an agreement of 85%, while the FP eCHIS task performed scored 82%. However, the 15% disagreement regarding task performance in eCHIS and the information organization in HEW eCHIS indicate that the next version of eCHIS will consider these parameters to enhance its effectiveness. Additionally, the 18% disagreement score in tasks performed by eCHIS on the FP eCHIS app will impact the overall performance of eCHIS. Moreover, minimizing the number of steps (clicks) will be crucial for improving the efficiency of eCHIS. A well-designed eCHIS facilitates the flow of clinical information from Kebele to the national level, resulting in better patient care and documentation. It should enable HEWs, Midwives, and FPs to carry out their duties appropriately and thoroughly [25].

Trustfulness is an important aspect of usability in eCHIS, and four parameters were used to evaluate the trustfulness of eCHIS. An agreement scores of 65%, 66%, and 75% was obtained in access detail support provided by eCHIS in HEW, HCR, and FP, respectively. HEW, HCR, and FP session time provided by the eCHIS agreement score were 55%, 60%, and 25%, respectively. The user ID or password style and format of the eCHIS agreement score were 88%, 74%, and 50% in HEW, HCR, and FP, respectively. Authentication techniques provided by eCHIS were 72%, 91%, and 91% in HEW, HCR, and FP, respectively. However, the disagreement in trustfulness is significantly higher in comparison with the other categories. Actions such as access support and password requirements (e.g., password age) were identified as crucial factors for enhancing the trustworthiness of eCHIS. Overall, enhancing data quality and integrity (including providing accurate, complete, and consistent data) [30], system performance and reliability (ensuring quick response times and minimizing errors and failures) [25], user interface design and feedback (ensuring a clear, intuitive, and user-friendly design) [33], and security and privacy measures (protecting data from unauthorized access and use) [25, 33] will improve the confidence and satisfaction of HEWs, midwives, and FPs with the system.

Finally, to evaluate the satisfaction of eCHIS, nine parameters were used. Among the nine satisfaction parameters, the agreement score was higher than 80% on seven parameters. Action history not provided by eCHIS and the help menu provided by eCHIS were scored at 78% and 41%, respectively. Similarly, action histories not provided by eCHIS scored an agreement of 26% and 74% on the HCR and FP apps, respectively. To enhance eCHIS user satisfaction, this study identifies automatic data synchronization (via push notifications) as crucial, dependent on internet connectivity or a set of predefined criteria. Conversely, for evaluating a user’s competency in using eCHIS, baseline standards must be established and introduced. In the end, users such as HEWs, midwives, and FPs should find the eCHIS easy to learn and remember, which often comes from intuitive design and a positive user experience [25].

This study also examined the usability of eCHIS based on participant type, total experience in years, eCHIS experiences, and internet connectivity. We observed that the following groups of health workers were satisfied with the learnability, efficiency, effectiveness, trustfulness, and satisfaction of eCHIS. First, compared to novice users, casual users agreed. This finding is evident, as a casual user refers to someone who has received eCHIS training but only uses it occasionally, while a novice is a user who has undergone eCHIS training but has not yet started using the system. Next, health workers who had service experience greater than 10 years and were in areas with good internet connectivity expressed their agreement in comparison with those who had less than 10 years of experience and were in areas with poor internet connectivity. Nevertheless, health workers in areas with poor internet connectivity express agreement with the effectiveness of eCHIS compared to those with good connectivity. This may need further investigation, as we are uncertain about the exact reasons. Finally, health workers with less than 2 years of eCHIS experience show a higher level of agreement, while the level of agreement among those with more than 2 years of experience is lower. While the exact reason is unknown, this may need further investigation and the formulation of hypotheses. Typically, under normal circumstances, higher experience with eCHIS correlates with greater agreement on its learnability, efficiency, effectiveness, trustworthiness, and satisfaction. Possible reasons may include the sample size of our study and the likelihood that, as health workers become familiar with eCHIS, they may demand new features and user experiences, among other factors.

Furthermore, comparison among novice, casual, and expert users based on twenty-six parameters revealed that, despite their different levels of experience, these user groups demonstrated comparable levels of usability when using the eCHIS application. For instance, a maximum of 0.45 mean difference was observed between expert and causal users, while a value of 0.38 was observed between expert and novice users on the learnability of eCHIS. More information on the learnability, efficiency, effectiveness, trustworthiness, and satisfaction of the eCHIS application is shown in Fig. 1.

However, this study has certain limitations. The study has a limited scale, and a large-scale usability study is required. For the future development and enhancement of eCHIS, follow user experience best practices and address the gaps identified by this study in the next generation of eCHIS to improve learnability, efficiency, effectiveness, trustworthiness, and satisfaction.