By means of first-principles density functional calculations, we study the structural, magnetic and electronic properties of YMnO3/MnO3 heterostructures. Although in the bulk the ground state of YMnO3 is an antiferromagnet, the YMnO3/MnO3 heterostructure stabilizes the ferromagnetic (FM) phase in YMnO3 in the interface region over a wide range of Coulomb repulsion parameters. The hypothetical FM phase of bulk YMnO3 is dielectric and due to substantial differences between the lattice constants in the ab plane, a strong magnetocrystalline anisotropy is present. This anisotropy produces a high coercivity of the unusual FM YMnO3 that can explain the large vertical shift in the hysteresis loops observed in recent experiments (Paul et al 2014 J. Appl. Crystallogr. 47 1054). The correlation between weak exchange bias and the vertical shift is proposed, which calls for reinvestigation of various systems showing vertical shifts.