Ambiguity set is a key element in distributionally robust optimization models. Here we investigate the impact of perturbation of ambiguity set on the optimal value and the optimal solutions. We consider the case where the ambiguity set is defined through generalized prior moment conditions and the perturbation is caused by (a) increasing sample data to be used in the moment system and (b) discretization of the moment system. We quantify the perturbation against change of sample data or refinement of discretization and its impact on the underlying optimization problem. We also consider the case where the ambiguity set is constructed through zeta-ball and extend the analysis to a two-stage distributionally robust risk minimization problem and a distributionally robust chance constrained optimization problem.