Bridging Finite and Super Population Causal Inference

1 Introduction

Neyman [1, 2] defined causal effects in terms of potential outcomes, and proposed an inferential framework viewing all potential outcomes of a finite population as fixed and the treatment assignment as the only source of randomness. This finite population view allows for easy interpretation free of any hypothetical data generating process of the outcomes, and is used in a variety of contexts e.g., [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]. This approach is considered desirable because, in particular, it does not assume the data are somehow a representative sample of some larger (usually infinite) population.

Alternative approaches, also using the potential outcomes framework, assume that the potential outcomes are independent and identical draws from a hypothetical infinite population. Mathematical derivations under this approach are generally simpler, but the approach itself can be criticized because of this typically untenable sampling assumption. Furthermore, this approach appears to ignore the treatment assignment mechanism.

That being said, it is well known that the final variance formulae from either approach tend to be quite similar. For example, while the variance of the difference-in-means estimator for a treatment-control experiment under an infinite population model is different from the one under Neyman’s [1] finite population formulation, this difference is easily represented as a function of the variance of the individual causal effects. Furthermore, this difference term is unidentifiable and is often assumed away under a constant causal effect model ([1, 2, 23, 24, 25, 26]), or by appeals to the final estimators being “conservative.”

For the difference in means, the infinite population variance estimate gives a conservative (overly large) estimate of the finite population variance. As deriving infinite population variance expressions, relative to finite population variance expressions, tends to be more mathematically straightforward, we might naturally wonder if we could use infinite population expressions as conservative forms of finite population expressions more generally. In this work we show that in fact we can assume an infinite population model as an assumption of convenience, and derive formula from this perspective. This shows that we can thus consider the resulting formula as focused on the treatment assignment mechanism and not on a hypothetical sampling mechanism, i.e., we show variance derivations under the infinite population framework can be used as conservative estimators in a finite context.

Mathematically, this result comes from a variance decomposition and a completeness-style argument characterizing the connection and the difference between these two views. The variance decomposition we use has previously appeared in Imai [7], Imbens and Rubin [18], and Balzer et al. [27]. The completeness-style argument, which we believe is novel in this domain, then sharpens the variance decomposition by moving from an expression on an overall average relationship to one that holds for any specific sample.

Our overall goal is simple: we wish to demonstrate that if one uses variance formula derived from assuming an infinite population sampling model, then the resulting inference one obtains will be correct with regards to the analogous sample-specific treatment effects (although it could be potentially conservative in that the standard errors may be overly large) regardless of the existence of any sampling mechanism.

2 Super population, finite population, and samples

Assume that random variables (Y(1),Y(0)) represent the pair of potential outcomes of an infinite super population, from which we take an independent and identically distributed (IID) finite population of size n:

The individual causal effect for unit i is τi=Yi(1)−Yi(0). We first discuss completely randomized experiments, and comment on other experiments in Section 4. For a completely randomized experiment, we randomly assign n1 units to receive treatment, leaving the remaining n0=n−n1 units to receive control. Let Z=(Z1,…,Zn)∈{0,1}n be the treatment assignment vector, which takes a particular value (z1,…,zn)∈{0,1}n with probability n1!n0!/n!, for any (z1,…,zn)∈{0,1}n with ∑i=1nzi=n1. The observed outcome for unit i is then Yiobs=ZiYi(1)+(1−Zi)Yi(0).

At the super population level, the average potential outcomes are E{Y(1)} and E{Y(0)}, and the average causal effect is

The population variances of the potential outcomes and individual causal effect are

At the finite population level, i.e., for a fixed sample S, the average potential outcomes and average causal effect are

The corresponding finite population variances of the potential outcomes and individual causal effects are

where, following the tradition of survey sampling [28], we use the divisor n−1. \;All these quantities are fully dependent on S. In classical casual inference [1], the potential outcomes of these n experimental units, S, are treated as fixed numbers. Equivalently, we can consider such causal inference to be conducted conditional on S e.g., [5, 6, 18, 19, 20].

Regardless, we have two parameters – the population average treatment effect τ, and the sample average treatment effect τS. After collecting the data, we would want to draw inference about τ or τS.

Our primary statistics are the averages of the observed outcomes and the difference-in-means estimator:

We also observe the sample variances of the outcomes under treatment and control using

We do not have the sample analogue of Sτ2 or Vτ because Yi(1) and Yi(0) are never jointly observed for any unit i in the sample.

We summarize the infinite population, finite population and sample quantities in Table 1.

3 Deriving complete randomization results with an independent sampling model

The three levels of quantities in Table 1 are connected via independent sampling and complete randomization. Neyman [1], without reference to any infinite population and by using the assignment mechanism as the only source of randomness, represented the assignment mechanism via an urn model, and found

He then observed that the final term was unidentifiable but nonnegative, and thus if we dropped it we would obtain an upper bound of the estimator’s uncertainty.

We next derive this result by assuming a hypothetical sampling mechanism from some assumed infinite super-population model of convenience. This alternative derivation of the above result, which can be extended to other assignment mechanisms, shows how we can interpret formulae based on super-population derivations as conservative formulae for finite-sample inference.

4 Discussion

Equation (8) relies on the assumption that the hypothetical infinite population exists, but eq. (1) does not. However, the completeness-style argument allowed us to make our sampling assumption only for convenience in order to prove eq. (1) by, in effect, dropping the expectation on both sides of eq. (8). Similar argument exists in the classical statistics literature; see Efron and Morris [30] for the empirical Bayes view of Stein’s estimator. While the final result is, of course, not new, we offer it as it gives an alternative derivation that does not rely on asymptotics such as a growing super population or a focus on the properties of the treatment assignment mechanism.

Using Freedman’s [8] results, Aronow et al. [12] considered a super population with N units, with the finite population being a simple random sample of size n. Letting N→∞, we can obtain similar results. We go in the other direction: we use the variance decomposition of eq. (6) to derive the finite population variance from the super population one.

This decomposition approach also holds for other types of experiments. First, for a stratified experiment, each stratum is essentially a completely randomized experiment. Apply the result to each stratum, and then average over all strata to obtain results for a stratified experiment. Second, because a matched-pair experiment is a special case of a stratified experiment with two units within each stratum, we can derive the Neyman-type variance cf. ([7, 18]) directly from that of a stratified experiment. Third, a cluster-randomized experiment is a completely randomized experiment on the clusters. If the causal parameters can be expressed as cluster-level outcomes, then the result can be straightforwardly applied cf.([11, 17]). Fourth, for general experimental designs, the variance decomposition in eq. (6) still holds, and therefore we can modify the derivation of the finite population variance according to different forms of eqs (4) and (5).

In a completely randomized experiment, the finite population sampling variance of τˆ in eq. (1) depends on three terms: the first two can be unbiasedly estimated by s12/n1 and s02/n0, but the third term Sτ2/n is unidentifiable from the data. Assuming a constant causal effect model, Sτ2=0 and the variance estimators under both finite and super population inference coincide. However, Aronow et al. [12], Robins [31] and Ding and Dasgupta [32] demonstrated that the treatment variation term Sτ2 has a sharp lower bound that may be larger than 0, which allows for more precise variance estimators under the finite population view. This demonstrates that we can indeed make better inference conditional on the sample we have. On the other hand, in this work, we showed that assuming an infinite population, while not necessarily giving the tightest variance expressions, nonetheless gives valid (conservative) variance expressions from a finite-population perspective. We offer this approach as a possible method of proof that could ease derivations for more complex designs. More broadly, it is a step towards establishing that infinite population derivations for randomized experiments can be generally thought of as pertaining to their finite population analogs. Also see Lin [16] and Samii and Aronow [33], who provide alternative discussions of super population regression-based variance estimators under the finite population framework.

Our discussion is based on the frequentists’ repeated sampling evaluations of the difference-in-means estimator for the average causal effect. In contrast, Fisher [34] proposed the randomization test against the sharp null hypothesis that Yi(1)=Yi(0) for all units, which is numerically the same as the permutation test for exchangeable units sampled from an infinite population [29]. This connection becomes more apparent when only the ranks of the outcomes are used to construct the test statistics, as discussed extensively by Lehmann [35].