The minimal degree of a permutation group G is the minimum number of points not fixed by non-identity elements of G. Lower bounds on the minimal degree have strong structural consequences on G. In 2014 Babai proved that the automorphism group of a strongly regular graph with n vertices has minimal degree at least cn, with known exceptions. Strongly regular graphs correspond to primitive coherent configurations of rank 3. We extend Babai's result to primitive coherent configurations of rank 4. We also show that the result extends to non-geometric primitive distance-regular graphs of bounded diameter. The proofs combine structural and spectral methods. The results have consequences to primitive permutation groups that were previsouly known using the classification of finite simple groups (Cameron, Liebeck).