In this paper we propose a sparse representation for the Bayes Machine based on the approach followed by the Informative Vector Machine (IVM). However, some extra modifications are included to guarantee a better approximation to the posterior distribution. That is, we introduce additional refining stages over the set of active patterns included in the model. These refining stages can be thought as a backfitting algorithm that tries to fix some of the mistakes that result from the greedy approach followed by the IVM. Experimental comparison of the proposed method with a full Bayes Machine and a Support Vector Machine seems to confirm that the method is competitive with these two techniques. Statistical tests are also carried out to support these results.