Overlap Violations in Clustered Observational Studies of Educational Interventions

In education settings, treatments are often non-randomly assigned to clusters, such as schools or classrooms, while outcomes are measured for students. This research design is called the clustered observational study (COS). We examine the consequences of common support violations in the COS context. Common support violations occur when the covariate distributions of treated and control units have substantial areas of non-overlap. Such violations are likely to occur in a COS, especially with a small number of treated clusters. We provide a comprehensive review of methods for overlap violations in the context of COS designs. We provide an overview of diagnostic tests and trimming methods to ensure overlap holds for the distributions of treated and control covariates. We then outline how trimming changes the estimand and how profiling can be used to understand the causal quantity for which overlap holds. Finally, we demonstrate how steps to achieve adequate overlap can result in very narrowly defined causal effects that may have little policy relevance. We use data on Catholic schools to illustrate concepts throughout.