In the past decades Machine Learning tools have been successfully used in several medical diagnostic problems. While they often significantly outperform expert physicians (in terms of diagnostic accuracy, sensitivity, and specificity), they are mostly not used in practice. One reason for this is that it is dificult to obtain an unbiased estimation of diagnose’s reliability. We propose a general framework for reliability estimation, based on transductive inference. We show that our reliability estimation is closely connected with a general notion of significance tests. We compare our approach with classical stepwise diagnostic process where reliability of diagnose is presented as its post-test probability. The presented approach is evaluated in practice in the problem of clinical diagnosis of coronary artery disease, where significant improvements over existing techniques are achieved.