The rapid development of 3D display technologies allows consumers to enjoy the 3D world visually through different display systems such as the stereoscopic, multiview, and light field displays. On the other hand, the conventional multiview synthesis-based 3D rendering technique demands more memory usage and computation time as the number of views is increased in multiview or light field displays. Conventional 3D rendering processing for 3D display becomes complex in order to generated real like 3D display image. This paper proposes a novel method to generate light rays of 3D display not by the conventional algorithm such as multiview rendering method technique but by 3D direct light field rendering. Our algorithm interprets light rays from 3D display and input color and disparity value in the light field domain, and it significantly reduces the computational complexity and memory usage. Since direct light field rendering algorithm is different from general multiview image processing algorithms, we propose new 3D image generation algorithm for hole filling, boundary matting, and view filtering from common stereo input images. © 2016, Society for Imaging Science and Technology (IS&T).