Clinicians’ Experiences with Explainable AI in Clinical Decision-Making
Main Article Content
Abstract
Artificial intelligence (AI) and machine learning have become integral to contemporary healthcare, particularly through AI-based clinical decision support systems designed to assist clinicians in diagnostic and treatment-related decisions. Within this field, explainable AI has gained prominence as a response to concerns about transparency, trust, and accountability, yet understanding of how clinicians actually experience explainability in practice remains limited. Existing research has largely focused on technical performance and quantitative measures of acceptance, leaving unanswered questions about how clinicians interpret, trust, and negotiate AI explainability in real-world clinical decision-making. Here, an interpretative phenomenological approach is used to explore clinicians’ lived experiences with explainable AI and to illuminate how explainability is understood and made meaningful in clinical practice. Data were generated through in-depth interviews with clinicians who regularly use AI-based decision support systems and were analyzed using interpretative phenomenological analysis to identify essential experiential themes. The analysis reveals that explainability is experienced as partial understanding rather than full transparency and that trust in AI is dynamically negotiated through alignment with clinical intuition and contextual judgment. The findings further show that moral and professional responsibility remains firmly with clinicians, and that explainable AI can both reassure and intensify emotional and ethical burdens during decision-making. Importantly, the study demonstrates that explainability does not simply enhance trust in a linear manner; instead, it reshapes how clinicians construct accountability, manage uncertainty, and justify decisions in complex clinical contexts. These findings indicate that the effectiveness of explainable AI depends not only on technical clarity but also on its alignment with clinicians’ experiential reasoning and professional norms. These results advance a human-centered understanding of explainable AI in healthcare by demonstrating that explainability functions as an experiential and ethical phenomenon. By clarifying the practical and ethical implications of explainable AI use, this study provides evidence-based insights for developers, healthcare institutions, and policymakers seeking to design and implement AI systems that genuinely support clinical judgment rather than merely increase algorithmic transparency.
Article Details
Section
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Abgrall, G., Holder, A. L., Chelly-Dagdia, Z., Zeitouni, K., & Monnet, X. (2024). Should AI models be explainable to clinicians? Critical Care, 28(1). Scopus. https://doi.org/10.1186/s13054-024-05005-y
Aristidou, A., Jena, R., & Topol, E. J. (2022). Bridging the chasm between AI and clinical implementation. The Lancet, 399(10325), 620. Scopus. https://doi.org/10.1016/S0140-6736(22)00235-5
Bragazzi, N. L., Dai, H., Damiani, G., Behzadifar, M., Martini, M., & Wu, J. (2020). How big data and artificial intelligence can help better manage the covid-19 pandemic. International Journal of Environmental Research and Public Health, 17(9). Scopus. https://doi.org/10.3390/ijerph17093176
Braun, M., Hummel, P., Beck, S., & Dabrock, P. (2021). Primer on an ethics of AI-based decision support systems in the clinic. Journal of Medical Ethics, 47(12), E3. Scopus. https://doi.org/10.1136/medethics-2019-105860
Chen, M., Zhang, B., Cai, Z., Seery, S., Méndez, M. J., Ali, N. M., Ren, R., Qiao, Y., Xue, P., & Jiang, Y. (2022). Acceptance of clinical artificial intelligence among physicians and medical students: A systematic review with cross-sectional survey. Frontiers in Medicine, 9. Scopus. https://doi.org/10.3389/fmed.2022.990604
Debnath, S., Barnaby, D. P., Coppa, K., Makhnevich, A., Kim, E. J., Chatterjee, S., Tóth, V., Levy, T. J., Paradis, M., Cohen, S. L., Hirsch, J. S., & Zanos, T. P. (2020). Machine learning to assist clinical decision-making during the COVID-19 pandemic. Bioelectronic Medicine, 6(1). Scopus. https://doi.org/10.1186/s42234-020-00050-8
Giannakopoulos, K., Kavadella, A., Salim, A. A., Stamatopoulos, V., & Kaklamanos, E. G. (2023). Evaluation of the Performance of Generative AI Large Language Models ChatGPT, Google Bard, and Microsoft Bing Chat in Supporting Evidence-Based Dentistry: Comparative Mixed Methods Study. Journal of Medical Internet Research, 25(1). Scopus. https://doi.org/10.2196/51580
Guerra, G. A., Hofmann, H., Sobhani, S., Hofmann, G., Gomez, D., Soroudi, D., Hopkins, B. S., Dallas, J., Pangal, D. J., Cheok, S., Nguyen, V. N., MacK, W. J., & Zada, G. (2023). GPT-4 Artificial Intelligence Model Outperforms ChatGPT, Medical Students, and Neurosurgery Residents on Neurosurgery Written Board-Like Questions. World Neurosurgery, 179, e160–e165. Scopus. https://doi.org/10.1016/j.wneu.2023.08.042
Henry, K. E., Kornfield, R., Sridharan, A., Linton, R. C., Groh, C., Wang, T., Wu, A., Mutlu, B., & Saria, S. (2022). Human–machine teaming is key to AI adoption: Clinicians’ experiences with a deployed machine learning system. Npj Digital Medicine, 5(1). Scopus. https://doi.org/10.1038/s41746-022-00597-7
Hopkins, B. S., Mazmudar, A., Driscoll, C., Svet, M., Goergen, J., Kelsten, M., Shlobin, N. A., Kesavabhotla, K., Smith, Z. A., & Dahdaleh, N. S. (2020). Using artificial intelligence (AI) to predict postoperative surgical site infection: A retrospective cohort of 4046 posterior spinal fusions. Clinical Neurology and Neurosurgery, 192. Scopus. https://doi.org/10.1016/j.clineuro.2020.105718
Huang, Y., Gomaa, A., Semrau, S., Haderlein, M., Lettmaier, S., Weissmann, T., Grigo, J., Tkhayat, H. B., Frey, B., Gaipl, U., Distel, L., Maier, A., Fietkau, R., Bert, C., & Putz, F. (2023). Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: Potentials and challenges for ai-assisted medical education and decision making in radiation oncology. Frontiers in Oncology, 13. Scopus. https://doi.org/10.3389/fonc.2023.1265024
Jha, A. K., Bradshaw, T. J., Buvat, I., Hatt, M., Kc, P., Liu, C., Obuchowski, N. F., Saboury, B., Slomka, P. J., Sunderland, J. J., Wahl, R. L., Yu, Z., Zuehlsdorff, S., Rahmim, A., & Boellaard, R. (2022). Nuclear Medicine and Artificial Intelligence: Best Practices for Evaluation (the RELAINCE Guidelines). Journal of Nuclear Medicine, 63(9), 1288–1299. Scopus. https://doi.org/10.2967/jnumed.121.263239
Jutzi, T. B., Krieghoff-Henning, E. I., Holland-Letz, T., Utikal, J. S., Hauschild, A., Schadendorf, D., Sondermann, W., Fröhling, S., Hekler, A., Schmitt, M., Maron, R. C., & Brinker, T. J. (2020). Artificial Intelligence in Skin Cancer Diagnostics: The Patients’ Perspective. Frontiers in Medicine, 7. Scopus. https://doi.org/10.3389/fmed.2020.00233
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., de Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2 February). Scopus. https://doi.org/10.1371/journal.pdig.0000198
Kwon, J.-M., Jeon, K.-H., Kim, H. M., Kim, M. J., Lim, S. M., Kim, K.-H., Song, P. S., Park, J., Choi, R. K., & Oh, B.-H. (2020). Comparing the performance of artificial intelligence and conventional diagnosis criteria for detecting left ventricular hypertrophy using electrocardiography. Europace, 22(3), 412–419. Scopus. https://doi.org/10.1093/europace/euz324
Kwon, J.-M., Kim, K.-H., Medina-Inojosa, J., Jeon, K.-H., Park, J., & Oh, B.-H. (2020). Artificial intelligence for early prediction of pulmonary hypertension using electrocardiography. Journal of Heart and Lung Transplantation, 39(8), 805–814. Scopus. https://doi.org/10.1016/j.healun.2020.04.009
Labkoff, S., Oladimeji, B., Kannry, J., Solomonides, A., Leftwich, R., Koski, E., Joseph, A. L., López-González, M., Fleisher, L. A., Nolen, K., Dutta, S., Levy, D. R., Price, A., Barr, P. J., Hron, J. D., Lin, B., Srivastava, G., Pastor, N., Luque, U. S., … Quintana, Y. (2024). Toward a responsible future: Recommendations for AI-enabled clinical decision support. Journal of the American Medical Informatics Association, 31(11), 2730–2739. Scopus. https://doi.org/10.1093/jamia/ocae209
Li, J., Cairns, B. J., Li, J., & Zhu, T. (2023). Generating synthetic mixed-type longitudinal electronic health records for artificial intelligent applications. Npj Digital Medicine, 6(1). Scopus. https://doi.org/10.1038/s41746-023-00834-7
Lin, W.-C., Chen, J. S., Chiang, M. F., & Hribar, M. R. (2020). Applications of artificial intelligence to electronic health record data in ophthalmology. Translational Vision Science and Technology, 9(2). Scopus. https://doi.org/10.1167/tvst.9.2.13
Maaßen, O., Fritsch, S., Palm, J., Deffge, S., Kunze, J., Marx, G., Riedel, M., Schuppert, A., & Bickenbach, J. (2021). Future medical artificial intelligence application requirements and expectations of physicians in german university hospitals: Web-based survey. Journal of Medical Internet Research, 23(3). Scopus. https://doi.org/10.2196/26646
Mukhlis, L. (2025). A Phenomenological Study of Personal Spiritual Experiences in Navigating Religious Pluralism within Interfaith Communities. Irfana: Journal of Religious Studies, 1(6), 212–220.
Mukhlis, L., Arifin, T., Ridwan, A. H., & Zulbaidah. (2024). Integrating Artificial Intelligenceand Maqāṣid al-Syarī‘ah: Revolutionizing Indonesia’s Sharia Online Trading System. Computer Fraud and Security, 2024(11), 301–309. https://doi.org/10.52710/cfs.238
Mukhlis, L., Arifin, T., Ridwan, A. H., Zulbaidah, Rosadi, A., & Solehudin, E. (2025). Reformulation of Islamic Stock Law: The Application of Taṣarrufāt al-Rasūl and Maqāṣid al-Syarī‘ahto Develop a Dynamic and Sustainable Islamic Capital Market in Indonesia. Journal of Posthumanism, 5(3), 1–13. https://doi.org/10.63332/joph.v5i3.913
Mukhlis, L., Maryam, S., & Sormin, S. A. (2023). Model Pembelajaran Living History Berbasis PjBL Untuk Meningkatkan Keterampilan Histografi Mahasiswa. Jurnal Educatio FKIP UNMA, 9(4), 1800–1809. https://doi.org/10.31949/educatio.v9i4.5595
Mukhlis, L., & Saidah, Y. (2025). Dynamics of Nature-Based learning in Developing Children’s Motoricic Skills: Teacher and Parent Perspectives. HUMANISMA: Journal of Gender Studies, 9(1), 64–79. http://dx.doi.org/10.30983/humanisme.v4i2.9366
Mukhlis, L., Suradi, Janwari, Y., & Syafe`i, R. (2023). Sosialisasi Saham Syariah sebagai Instrumen Pengembangan Ekonomi Masyarakat di Badan Kontak Majelis Taklim (BKMT) Kabupaten Mandailing Natal. Jurnal Pengabdian Multidisiplin, 3(2), 2–9. https://doi.org/10.51214/japamul.v3i2.604
Mursch-Edlmayr, A. S., Ng, W. S., Diniz-Filho, A., Cordeiro Sousa, D. C., Arnould, L., Schlenker, M. B., Duenas-Angeles, K., Keane, P. A., Crowston, J. G., & Jayaram, H. (2020). Artificial intelligence algorithms to diagnose glaucoma and detect glaucoma progression: Translation to clinical practice. Translational Vision Science and Technology, 9(2), 1–21. Scopus. https://doi.org/10.1167/tvst.9.2.55
Nelson, C. A., Perez-Chada, L. M., Creadore, A., Li, S. J., Lo, K., Manjaly, P., Pournamdari, A. B., Tkachenko, E., Barbieri, J. S., Ko, J. M., Menon, A. V., Hartman, R. I., & Mostaghimi, A. (2020). Patient Perspectives on the Use of Artificial Intelligence for Skin Cancer Screening: A Qualitative Study. JAMA Dermatology, 156(5), 501–512. Scopus. https://doi.org/10.1001/jamadermatol.2019.5014
Plana, D., Shung, D. L., Grimshaw, A. A., Saraf, A., Sung, J. J. Y., & Kann, B. H. (2022). Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review. JAMA Network Open, 5(9), E2233946. Scopus. https://doi.org/10.1001/jamanetworkopen.2022.33946
Reddy, S. (2024). Generative AI in healthcare: An implementation science informed translational path on application, integration and governance. Implementation Science, 19(1). Scopus. https://doi.org/10.1186/s13012-024-01357-9
Röösli, E., Rice, B., & Hernandez-Boussard, T. (2021). Bias at warp speed: How AI may contribute to the disparities gap in the time of COVID-19. Journal of the American Medical Informatics Association, 28(1), 190–192. Scopus. https://doi.org/10.1093/jamia/ocaa210
Rubinger, L., Gazendam, A., Ekhtiari, S., & Bhandari, M. (2023). Machine learning and artificial intelligence in research and healthcare. Injury, 54, S69–S73. Scopus. https://doi.org/10.1016/j.injury.2022.01.046
Shamout, F. E., Shen, Y., Wu, N., Kaku, A., Park, J., Makino, T., Jastrzębski, S., Witowski, J., Wang, D., Zhang, B., Dogra, S., Cao, M., Razavian, N., Kudlowitz, D., Azour, L., Moore, W., Lui, Y. W., Aphinyanaphongs, Y., Fernandez-Granda, C., & Geras, K. J. (2021). An artificial intelligence system for predicting the deterioration of COVID-19 patients in the emergency department. Npj Digital Medicine, 4(1). Scopus. https://doi.org/10.1038/s41746-021-00453-0
Shinners, L., Aggar, C., Grace, S., & Smith, S. (2020). Exploring healthcare professionals’ understanding and experiences of artificial intelligence technology use in the delivery of healthcare: An integrative review. Health Informatics Journal, 26(2), 1225–1236. Scopus. https://doi.org/10.1177/1460458219874641
Tsopra, R., Fernández, X., Luchinat, C., Alberghina, L., Lehrach, H., Vanoni, M., Dreher, F., Sezerman, O. U., Cuggia, M., de Tayrac, M., Miklaševičs, E., Itu, L. M., Geantă, M., Ogilvie, L., Godey, F., Boldisor, C. N., Campillo-Gimenez, B., Cioroboiu, C., Ciuşdel, C. F., … Burgun, A. (2021). A framework for validating AI in precision medicine: Considerations from the European ITFoC consortium. BMC Medical Informatics and Decision Making, 21(1). Scopus. https://doi.org/10.1186/s12911-021-01634-3
Tveit, J., Aurlien, H., Plis, S., Calhoun, V. D., Tatum, W. O., Schomer, D. L., Arntsen, V., Cox, F., Fahoum, F., Gallentine, W. B., Gardella, E., Hahn, C. D., Husain, A. M., Kessler, S., Kural, M. A., Nascimento, F. A., Tankisi, H., Ulvin, L. B., Wennberg, R., & Beniczky, S. (2023). Automated Interpretation of Clinical Electroencephalograms Using Artificial Intelligence. JAMA Neurology, 80(8), 805–812. Scopus. https://doi.org/10.1001/jamaneurol.2023.1645