Linear discriminant analysis that takes spatial smoothness into account has been developed and widely used in image processing society. However, two questions remain unanswered. First, which is the best way to incorporate the smoothness property of images with linear discriminant analysis? Second, which is the best representation for the smoothness property of images? To answer the first question, we propose a Bayesian framework of Gaussian process in order to extend Fisher’s discriminant for image data. The probability structure for our extended Fisher’s discriminant is explicitly formulated, and the smoothness properties of images are utilized as prior probabilities. For the second question, we suggest a family of prior probabilities derived from natural image statistics. The unknown parameters in our model are estimated via the maximum a posteriori probability (MAP) estimation. We will show that existing methods imposing smoothness assumption of images are rough approximations to the proposed MAP estimates in this framework. Experimental results on the Yale face database and the ETH-80 object categorization dataset show that the proposed method significantly outperforms the other Fisher’s discriminant methods for various image data.