Color constancy involves correctly attributing a bias in the color of the light reaching your eyes to the illumination, and therefore compensating for it when judging surface reflectance. But not all biases are caused by the illumination, and surface colors will be misjudged if a bias is incorrectly attributed to the illumination. Evidence from within a scene (highlights, shadows, gradients, mutual reflections, etc.) could help determine whether a bias is likely to be due to the illumination. To examine whether the human visual system considers such evidence we asked subjects to match two surfaces on differently colored textured backgrounds. When the backgrounds were visibly rendered on screens in an otherwise dark room, the influence of the difference in background color was modest, indicating that subjects did not attribute much of the difference in color to the illumination. When the simulation of a change in illumination was more realistic, the results were very similar. We conclude that the visual system does not seem to use a sophisticated analysis of the possible illumination in order to obtain color constancy.