Subspace iteration for computing eigenvalues and eigenvectors, a natural generalization of the power method, is among the most straight-forward methods to analyze and implement. A variant of subpace iteration, called FEAST, that uses rational filters to accelerate convergence toward a targeted invariant subspace (typically selected by enclosing the eigenvalues of interest by a simple contour), has gained significant interest in recent years, having been adopted as part of Intel's Math Kernel Library for matrix eigenvalue problems. Such rational filters are often derived from quadrature approximations of contour integrals---think of approximating Cauchy's integral formula for the indicator function of the region enclosed by the contour. The rational filter governs the iterative convergence of the method, so it is perhaps surprising that quadrature error plays little role in the analysis of the algorithm, and we will explain why this is so. Inspired by this approach, we consider filtered subspace iteration for (possibly) unbounded selfadjoint operators, having in mind differential operators as motivational examples. In this broader context, it is natural to consider errors in approximating the invariant subspace with respect to different norms, and we provide a fairly general framework for analyzing both iteration and discretization errors for eigenvalues and invariant subspaces. The bulk of the computational effort in the algorithm involves approximating the action of the resolvent at a few points along a contour enclosing the eigenvalues of interest. In our numerical examples, such finite-rank approximations of the resolvent are obtained by finite element discretizations.