Exploring Meaning and Trust in Data Analysts’ Experiences with Opaque Machine Learning Models _within Enterprise Decision-Making Contexts Using a Phenomenological Approach

Abstract

Data science and analytics have become central to modern decision-making, transforming how individuals and organizations interpret information in algorithm-driven environments. Within this domain, machine learning systems especially opaque or “black-box” models present significant challenges to human understanding and interpretive trust. Despite advancements in explainable artificial intelligence (XAI), little is known about how data analysts experience and make sense of the opacity inherent in these systems. This study addresses that gap by asking: How do data analysts interpret and find meaning in the process of working with non-transparent machine learning models? Using an interpretative phenomenological approach (IPA), the study reveals that analysts experience an ongoing tension between cognitive reasoning, emotional ambivalence, and ethical reflection when engaging with AI systems. Data were collected through semi-structured interviews with twelve professional analysts and analyzed thematically using hermeneutic interpretation to uncover the essence of their lived experiences. The analysis identified three concrete experiential patterns: (1) analysts frequently rely on workaround strategies—such as proxy validation, intuition-based pattern checking, and collaborative sense-making—to compensate for missing model transparency; (2) opacity often produces practical constraints, including slower diagnostic processes and increased uncertainty in high-stakes decision contexts; and (3) trust is negotiated not only through technical explanations but through repeated model performance consistency and peer affirmation. The findings show that explainability is perceived not merely as a technical attribute but as a deeply human process of meaning-making and trust negotiation. These results highlight actionable implications, including the need for XAI tools that support step-by-step diagnostic reasoning, enable collaborative interpretation, and reduce ambiguity in operational workflows. The study offers important implications for developing human-centered AI systems that prioritize psychological intelligibility and epistemic responsibility alongside computational transparency.