This study employed a qualitative approach, utilizing a two-phase methodology that combined semi-structured interviews with the Delphi technique to ensure methodological rigor and depth in the development of Key Benchmarking Indicators (KBIs) for Hospital Information Systems (HIS). The research was conducted in 2023 at Ahvaz Jundishapur University of Medical Sciences’ academic hospitals in Iran.
Phase 1: Semi-structured interviews
Sampling and participants
Purposive sampling was used to select participants, ensuring that the interviewees had extensive expertise in health information technology, medical informatics, and health information management. The inclusion criteria for participants were: (1) a minimum of five years of experience in HIS (2), current or previous involvement in HIS implementation or management, and (3) affiliation with academic hospitals. A total of 14 experts were recruited, including six faculty members and eight senior IT professionals. The sample included a diverse range of expertise, with participants specializing in software development, system architecture, IT support, and health information management. The gender distribution was balanced, with seven male and seven female participants, and their work experience ranged from 5 to 20 years in the field.
Interview guide development
The initial interview guide was developed based on a thorough review of existing literature on HIS benchmarking and KBIs. The guide included open-ended questions organized around eight key areas: software quality, hardware quality, architecture and user interface, vendor quality, IT support quality, workflow quality, output quality, and cost [10] (Attachment 1). To ensure the clarity and relevance of the questions, the guide underwent pilot testing with five participants who were not part of the final sample. Feedback from the pilot phase indicated the need for additional probing questions to explore specific technical and organizational challenges in greater depth. Based on this feedback, the guide was revised to include more detailed questions on interoperability, user training, and cost-effectiveness.
Data collection
Before each interview, the interviewer introduced themselves, explained the study’s objectives, and obtained verbal consent from participants. Confidentiality was assured, and participants were informed that their names and specific opinions would not be disclosed. Each interview lasted between 45 and 60 min and was audio-recorded. The interviews were transcribed verbatim by the first author, and transcripts were returned to participants for verification to ensure accuracy. Data collection continued until thematic saturation was achieved after 14 interviews. All procedures performed in this study were in accordance with the ethical standards of the institutional and/or national research committee.
Bias mitigation
To mitigate potential biases associated with purposive sampling, the research team employed several strategies. First, the sample included a diverse range of experts from different departments and with varying levels of experience. Second, the interview guide was pilot-tested and refined to reduce leading questions. Third, the research team conducted regular debriefing sessions to discuss emerging themes and ensure that interpretations were grounded in the data. Finally, participant feedback on the transcripts was incorporated to validate the findings.
Data analysis
The interviews were analyzed using conventional content analysis. Three researchers independently coded the transcripts, identifying sub-themes and themes [10]. The initial themes were reviewed by the research team, and five group discussions were held to resolve disagreements and reach consensus. MAXQDA Analytics Pro 2018 was used to manage and analyze the qualitative data. To ensure trustworthiness, Lincoln and Guba’s criteria were followed, including credibility (through peer debriefing and participant feedback), transferability (through detailed descriptions of the study context), dependability (through clear documentation of procedures), and confirmability (through external checking by an independent researcher not involved in the study) [11].
Phase 2: Delphi technique
Questionnaire development
In the second phase, a researcher-designed questionnaire containing 76 indicators derived from the interview findings was used to prioritize the KBIs (Attachment 2). The questionnaire was validated by faculty experts in health information management and medical informatics at Isfahan University of Medical Sciences (IUMS). Reliability was established through a test-retest on a 20% sample, yielding a Cronbach’s alpha of 0.967.
Delphi process
The Delphi technique involved two rounds of feedback from the same 14 participants who participated in the interviews. In the first round, indicators scoring 50% or below were excluded, while those scoring 75% or higher were accepted. Indicators falling between these thresholds were re-evaluated in the second round. In the second round, participants were asked to prioritize the indicators, and final approval of the KBIs was obtained. If any indicators still fell within the 50–75% range after the second round, further discussion sessions were conducted to reach a consensus. Ultimately, 76 indicators were finalized as KBIs for HIS.
Data analysis
SPSS was used for statistical analysis, including measures of central tendency (median and mean) and dispersion (frequency and percentage).
Confirmability
To ensure confirmability, an external researcher with expertise in health information systems reviewed the final themes and indicators. This external validation process helped to ensure that the findings were not influenced by the researchers’ biases and were grounded in the data.
By employing this rigorous two-phase approach, the study ensured the credibility, transferability, dependability, and confirmability of its findings, providing a robust framework for benchmarking HIS in-healthcare organizations.
link