Persuasive but Unfaithful. Conversational Diagnostic Artificial Intelligence and the Trust-Alignment Divide

Silvano Zipoli Caiani (Università degli Studi di Firenze, Università degli Studi di Milano)

Abstract:

Conversational Diagnostic Artificial Intelligence promises to reduce opacity in medical AI by producing explanations that resemble human clinical reasoning. This presentation argues, however, that this promise gives rise to a trust–alignment divide. AI-generated chains of thought can be highly persuasive, making recommendations appear coherent and worthy of trust, while remaining only weakly faithful to the processes that produced them. This divergence matters because, as evidence shows, users are more likely to rely on systems that can articulate reasons, even when those reasons do not reliably disclose the basis of the recommendation. The presentation argues that the persuasiveness–faithfulness gap is not merely a contingent failure of current systems, but partly reflects the autoregressive and probabilistic architecture of large language models, which generates reasons through the same sequence-based mechanisms that generate outputs. In clinical settings, this creates the distinctive risk that persuasive but unfaithful explanations may reinforce automation bias by making reliance on AI appear not only convenient, but epistemically justified. The challenge for explainable medical AI is therefore not simply to produce more fluent and human-like explanations, but to distinguish explanations that merely elicit trust from explanations that can genuinely justify it.