Living systems evolve one mutation at a time, but a single mutation can alter the effect of subsequent mutations. The underlying mechanistic determinants of such epistasis are unclear. Here, we argue that the physical properties of a biological system can generically and easily constrain its epistasis. We analyze the interaction between mutations in generic models of proteins and biochemical networks. In each case, a slow, collective physical mode is actuated upon mutation, reducing the dimensionality of mutational effects and thus the rank of the epistatic matrix. This, in turn, reduces the ruggedness of the sequence-to-function map. By providing a mechanistic basis for experimentally observed global epistasis, these results suggest that slow collective physical modes can make biological systems more evolvable.