We discuss a few selected hypotheses on how the visual system judges differences of color images. We then derive five image-difference features from these hypotheses and address their relation to the visual processing. Three models are proposed to combine these features for the prediction of perceived image differences. The parameters of the image-difference features are optimized on human image-difference assessments. For each model, we investigate the impact of individual features on the overall prediction performance. If chromatic features are combined with lightness-based features, the prediction accuracy on a test dataset is significantly higher than that of the SSIM index, which only operates on the achromatic component.