Let G be a group and H be a subgroup of G. We say that H is left relatively convex in G if the left G-set G/H has at least one G-invariant order. When G is left orderable, this holds if and only if H is convex in G under some left ordering of G. We give a criterion for H to be left relatively convex in G that generalizes a well known criterion of Burns and Hale. We then use this criterion to show that all maximal cyclic subgroups are left relatively convex in free groups, in right-angled Artin groups, and in surface groups that are not the Klein-bottle group. The free-group case extends a result of Duncan and Howie. We show that if G is left orderable, then each free factor of G is left relatively convex in G. More generally, for any graph of groups, if each edge group is left relatively convex in each of its vertex groups, then each vertex group is left relatively convex in the fundamental group; this generalizes a result of Chiswell. Finally, we show that all maximal cyclic subgroups in locally residually torsion-free nilpotent groups are left relatively convex. This is a joint work with Yago Antolin and Warren Dicks.