We propose and analyze an optical watermarking system based on a digital holographic technique. This system constructs a digital holographic watermark by the modified Mach-Zehnder interferometric architecture, and watermark detection is implemented using an optically VanderLugt 4-f correlator with a weighting matched filter mask. The correlation signal at the output plane can be used to recognize whether there is an authorized mark pattern in the watermarked image. In addition, we examine the nonlinear effect of the optical device on the correlation output. The output of the nonlinear device is considered as a mask of matched filter in the Fourier transform domain of an optically VanderLugt 4-f correlator. Through theoretical results, we provide an analytical solution for the correlation output of the nonlinear watermarking system.