In optical implementations of the phase-encoded BB84 protocol, the bit information is usually encoded in the phase of two consecutive photon pulses generated in a Mach-Zehnder interferometer. In the actual experimental realization, the loss in the arms of the Mach-Zehnder interferometer is not balanced, for example because only one arm contains a lossy phase modulator. Therefore, the amplitudes of the pulses is not balanced, and the structure of the signals and measurements no longer corresponds to the (balanced) ideal BB84 protocol. Hence, the BB84 security analysis no longer applies in this scenario. We provide a security proof of the unbalanced phase-encoded BB84. The resulting key rate turns out to be lower than the key rate of the ideal BB84 protocol. Therefore, in order to guarantee security, the loss due to the phase modulator cannot be ignored.