Implementation of an information system for tuberculosis in healthcare facilities in Indonesia: evaluation of its effectiveness and challenges | Archives of Public Health

The evaluation of the System for Information on TB (SITB) in Indonesia, including the Evaluation of User Convenience and Satisfaction (EUCS), has shed light on several critical areas that warrant attention for enhancing the system’s effectiveness and user satisfaction. This section discusses the key findings related to user convenience, timeliness, system-real world alignment, error prevention, and challenges in data entry, as identified through heuristic evaluation and interviews, including the EUCS. Additionally, the discussion explores potential strategies to address these issues, such as the development of user manuals, server maintenance, and the design of information and forms tailored for TB management in field settings. By addressing these aspects, it is possible to improve the overall usability and functionality of the SITB, ultimately contributing to more effective TB control efforts in Indonesia.

Heuristic evaluation

Heuristic evaluations are crucial in implementing health information systems because they can identify several challenges, including those with major and catastrophic severity categories. It is crucial for system designers and developers to address these issues [26, 27].

Research has shown that heuristic methods are more effective than cognitive tracing methods in identifying problems in health information systems [28]. Therefore, heuristic evaluation is a critical and effective technique for diagnosing existing issues and improving patient safety and treatment quality. This evaluation provides more accurate results and helps identify the actual problems in the system [28]. Moreover, numerous studies have successfully employed heuristic evaluation to assess the utility of health service information systems [28].

Importance of creating a system that matches the real health system and services

The study revealed an incompatibility between the TB management process and system, including the absence of a menu to change drug regimens in patients with DS-TB. Thus, officers had to assume that therapy had failed, and the patient’s status was changed to DR-TB. Another finding was the lack of a menu for contact history with DR-TB in patients receiving TB prevention therapy, which is a critical component of the tracing process.

The radiology system interface did not align with real world medical practices [28]. For example, the radiographic implementation protocol listed on the system did not match the actual implementation standard operating procedure, which can cause confusion for users and hinder task completion timeliness. Similarly, it was found that 57% of errors in a heuristic evaluation involved a mismatch between the system and the real world [28]. To enhance this compatibility, health information systems should use language that is easy for users to understand, employing familiar words, phrases, and concepts instead of system-oriented terms [28].

The match between the system and the real world is a crucial indicator [28], which stressed that systems should employ language that is understandable for users. This includes using words, phrases, and everyday language concepts, rather than relying solely on systematic terms that may only be understood by system designers.

Preventing errors in the health system to ensure patient safety and effective treatment

SITB application includes a help menu to assist users with problems or errors; however, an open access book version of the user manual is not yet available. Currently, the user guide is only available as a video and can only be accessed using an SITB account. Therefore, when health facility users experience problems or errors, they should rely on colleagues or TB managers at the district office level for assistance. Furthermore, the SITB application does not have a reset menu or return-to-default feature, meaning that users should save and then edit data to make changes. Although a warning pop-up appears when saving incomplete data, this can still be improved to prevent adverse events during TB patient treatment.

Although electronic health records (EHRs) have been recognized for their potential to reduce health costs and improve care quality in hospitals, improper design and use can lead to serious, unwanted impacts. Poor EHR system design and improper use can lead to EHR-related errors that compromise the integrity of the information, endangering patient safety, or reducing care quality [28]. Furthermore, user errors and bad interfaces can interfere with receiving information, leading to bookkeeping errors during decision-making [29].

Therefore, ensuring the safety of technology and its use in clinical settings is a crucial challenge in the development of health information technology. To achieve this, we highlight nine key points, including the methods and technology used in information systems, user and function standards, security software and network-enabled clinical environments, decision support, real time methods for automated surveillance and monitoring, information on hazards and adverse events, and models to improve patient safety [30].

EUCS evaluation

According to the findings of the present study, a significant proportion of users of the SITB application were female nurses. This observation aligns with previous research that reported that 70% of female nurses are involved in the management of TB cases [31]. Notably, other studies on TB have shown that most healthcare workers involved in managing such cases are female [32,33,34]. These findings reveal that the SITB application can reach and benefit a group of healthcare workers who play a vital role in the management of TB. Future research should explore the reasons behind the high usage of SITB among female nurses and the way it can be further tailored to their specific needs.

The management of TB patients is the responsibility of TB program implementers, namely, program managers, doctors, nurses, pharmacists, and laboratory and pharmacy staff. This is consistent with the findings of recent studies by [34], which highlight the role of radiographers in TB treatment. In the province of DIY, TB program implementers have over 10 years of work experience, are of productive age, and have a diploma-level education background. These findings underscore the importance of having experienced and educated healthcare professionals to manage TB, a complex and challenging disease that requires specialized knowledge and skills [32, 35].

Studies have shown that knowledge of TB infection control among healthcare workers is a concern, with several studies focusing on the demographic characteristics of health workers responsible for TB management. Research on the level of knowledge of TB infection control found that 48.5% of health workers are diploma graduates, and the rest are either undergraduate or postgraduate [34]. Similarly, studies have found that healthcare workers responsible for TB management have a working duration of 10 years and over, with the highest age range being between 31 and 40 years [32, 36,37,38]. A study involving nine countries, including Indonesia, has revealed that most electronic TB (eTB) manager users fall within the age range of 30–49 years [39]. These findings suggest that healthcare workers with significant work experience and diploma-level education play a critical role in TB management, and interventions to improve knowledge of TB infection control should target this group of healthcare workers. Further research should explore the most effective methods for improving TB infection control knowledge among healthcare workers, particularly those in the 31–40 age range, and the impact of such interventions on patient outcomes.

The EUCS evaluation model is a suitable tool for assessing the satisfaction of users of information systems in the healthcare sector, such as hospitals and health centers. Five EUCS dimensions affect user satisfaction with information systems in the healthcare sector [40].

The EUCS model has been shown to be both valid and reliable in measuring end-user satisfaction with IT systems. Content, accuracy, format, and timeliness play important roles in determining end-user satisfaction and the quality of the information provided [41]. Moreover, the EUCS method has been successfully applied to different contexts as a valid measurement of computer user satisfaction, including in the healthcare sector, while considering the policy implementation process and the context of the health information system [42].

Our findings revealed varying levels of satisfaction across different rural areas in Yogyakarta. Staff satisfaction was generally positive for content and accuracy dimensions, with scores above 3.5 on a 5-point scale. Most indicators on the content dimension were rated “very satisfied” by users, although one indicator was disputed by the assessor due to a discrepancy between the system and the manual register. However, satisfaction levels were lower for ease of use (3.2) and timeliness (3.0), often hindered by limited computer literacy and frequent server downtime of the SITB. Previous studies have highlighted the significant influence of the content dimension on user satisfaction and identified it as one of the dominant factors alongside the accuracy dimension [43, 44]. Recent research has confirmed that all indicators on the content dimension were classified as “satisfied,” with one indicator rated “very satisfied” because the system provides sufficient information.

Although most indicators in the accuracy dimension were rated “very satisfied,” the system failed to meet user expectations, particularly about frequent errors, which are often caused by server downtime. This forces officers to manually input data outside of working hours to prevent system errors. The accuracy dimension is crucial in maintaining user satisfaction, as shown by a survey of users [45, 46]. The findings are consistent with that of a research [9], which emphasizes that the suitability of information and the continuity of the system without error constraints are critical factors in ensuring patient satisfaction.

The user interface’s clarity is a dominant factor contributing to the ease of use of the SITB application. Two groups of factors affect the user interface, consisting of positive and significant variables, such as content, format, and accuracy, with content being the most positive and significant influence [47]. The format dimension, which relates to website design appearance, has the most significant impact on results [48]. A recent study has confirmed that the format dimension had the most dominant results, with a mean value of 3.59, falling into the “very satisfied” category [49]. These reveals that the clarity of the user interface is critical to the success of the SITB application and that improving the content, accuracy, and format dimensions can enhance user satisfaction. Further research is required to explore strategies for addressing server downtime issues and improving the accuracy dimension to enhance user satisfaction.

The use of SITB in these health facilities can have a significant impact on the management and control of TB cases in the region. This is evidenced by the enhanced detection and reporting capabilities of SITB, which have contributed to record-high case notification achievements in 2022 and 2023. Specifically, over 724,000 new TB cases were identified in 2022, with this number increasing to 809,000 cases in 2023 [3]. Thus, the use of digital information systems strongly supports addressing health issues [50]. However, Indonesia consists of urban and rural areas with varying facilities across hospitals, which may lead to variations in the acceptance of such information systems. Therefore, it is crucial to ensure that the SITB system is designed and implemented in a way that accommodates the needs of different types of health facilities and their users, regardless of their background. Therefore, it is crucial to ensure that the SITB system is designed and implemented in a way that accommodates the needs of different types of health facilities and their users, regardless of their background.

The SITB is mandatory to be implemented across all healthcare facilities in Indonesia, of which consisted of 10.180 primary health care (Puskesmas) and more than 1000 hospitals covering rural, urban and remote areas [51]. This comprehensive implementation highlights the importance of examining the system’s effectiveness and challenges in diverse contexts. Yogyakarta, with its mix of urban and rural areas, serves as a crucial microcosm for understanding these aspects on a national scale [52]. We measured user satisfaction using the End-User Computing Satisfaction (EUCS) model, adapted to the rural context. While we did not conduct a formal infrastructure readiness assessment specifically evaluate SITB users’ computer proficiency in this research.

Attributes that should be improved from the user’s viewpoint.

1. 1.

   High volume of applications overwhelms processing officers.

   Health facilities are currently required to input disease data into ten different applications, including SITB for TB; SIARVI for DHF; SISMAL for malaria; SILANTOR for chikungunya; E-FIlCA for filariasis; SKDR for worms, pneumococci, yellow fever, influenza, rotavirus, Mers-COv, typhoid, cholera, and gastrointestinal and respiratory tract infections; E-zoonosis for rabies, anthrax, plague, leptospirosis, avian influenza, and toxoplasma; SIHA for HIV and sexual infections; SI PD3I for diphtheria, pertussis, tetanus, meningitis, and polio; PWS for measles; and SITASIA for leprosy and yaws. However, the World Health Organization has acknowledged that the current health information system subsystems lack coordination, leading to an excessive burden on health workers who must collect and report data.

1. 2.

   The SITB application is designed to enhance the monitoring of TB patient care processes, which has been shown to improve the cure rate.

   SITB facilitates the identification of gaps between TB suspects and the diagnostic examinations that are performed, resulting in improved monitoring of TB patient care processes and increased cure rates. However, the electronic implementation of TB (ETR.Net) in South Africa often generates a high rate of error reporting, which negatively impacts the TB control program. Owing to the suboptimal function of e-TB, no significant effect was found on the cure rate of TB patients [43]. This highlights the need to optimize the use of electronic TB management systems to reduce error rates and improve TB control programs.

1. 3.

   SITB facilitates reporting and can improve the process of determining policies.

   SITB provides health facilities and city or provincial health offices with critical information for developing policies, such as trends in TB diagnoses made by health facilities, trends in TB cases (DS-TB, DR-TB, HIV-TB, and DM-TB), patient treatment monitoring processes, involvement of cadres/community in contact investigations, and availability of TB logistics in each health facility. Information technology can save healthcare professionals time in tasks such as recording, maintaining, and preparing reports [44]. These reports are crucial for decision-making in both administration and patient care.

This study has substantial implications for research and practice in health information systems. We introduce a novel, comprehensive approach for evaluation by combining heuristic evaluation, the EUCS model, and in-depth interview method, which can be applied to other health information systems. Our findings underscore the importance of user-centered design in system development, potentially influencing future designs. The study’s results can inform policy decisions regarding the implementation and improvement of health information systems in Indonesia, particularly for tuberculosis management. Additionally, this work provides a foundation for future studies on integrating SITB with Electronic Medical Records systems, contributing to long-term improvements in tuberculosis management. Our comprehensive evaluation offers actionable insights for enhancing SITB’s usability and user satisfaction, potentially leading to more effective tuberculosis management across Indonesia.

Limitation of this study include the absence of formal assessment of SITB users’ readiness in terms of technology use and computer skills, nor did we perform a comprehensive infrastructure evaluation. While our study has successfully evaluated SITB using a combined evaluation method, these gaps present opportunities for future research. We recommend that future studies conduct thorough infrastructure assessments to identify potential barriers to SITB adoption across diverse healthcare settings in Indonesia. Such studies will provide a more complete understanding of SITB’s effectiveness and user satisfaction, ultimately informing improved implementation strategies.

Ethical clearance

Ethical approval for this study was obtained from the Institutional Review Board of the Medical Faculty of Gadjah Mada University (ref. no. KE/FK/1404/EC/2022). Participation in the study was voluntary, and informed consent was obtained from all participants.