More Potential Outcomes

The Fundamental Problem, ATE, and ATT

Professor Benjamin Noble

More Potential Outcomes (yay?)

Animated scene of villagers cheering, captioned 'And There Was Much Rejoicing.'

via Monty Python and the Holy Grail

Am I a Good Teacher?

  • Good: students’ score on a 100-point data analysis test.
  • Suppose we have a binary treatment variable:

D_i = \begin{cases} 1 & \text{if } i \text{ takes POLI 170A} \\ 0 & \text{if } i \text{ does not take POLI 170A} \end{cases}

  • Suppose we have an outcome variable, Y_i, which is person i’s score on the test.

Two Potential Outcomes

  • No POLI 170A (D_i = 0)
  • Take POLI 170A (D_i = 1)

Y_i = \begin{cases} Y_i^1 & \text{if } D_i = 1 \ \text{(take POLI 170A)} \\ Y_i^0 & \text{if } D_i = 0 \ \text{(no POLI 170A)} \end{cases}

A student stands before two closed doors labeled No POLI 170 and POLI 170.

The same student faces the two doors now open: a dark starry path behind the No POLI 170 door labeled Y sub i sup 0, and a warm sunlit path behind the POLI 170 door labeled Y sub i sup 1.

You Got an 88…So What?

[Y_i \mid D_i = 1] = 88 Students taking an exam in a sunlit classroom — the treated world.

[Y_i \mid D_i = 1] = 88 Treated world — score 88.

[Y_i \mid D_i = 0] = 14 Counterfactual world — score 14.

[Y_i \mid D_i = 1] - [Y_i \mid D_i = 0] = 74

[Y_i \mid D_i = 1] = 88 Treated world — score 88.

[Y_i \mid D_i = 0] = 88 Counterfactual world — score 88.

[Y_i \mid D_i = 1] - [Y_i \mid D_i = 0] = 0

[Y_i \mid D_i = 1] = 88 Treated world — score 88.

[Y_i \mid D_i = 0] = 97 Counterfactual world — score 97.

[Y_i \mid D_i = 1] - [Y_i \mid D_i = 0] = -9

[Y_i \mid D_i = 1] = 88 Treated world — score 88.

[Y_i \mid D_i = 0] = \;??? Unobservable counterfactual world.

[Y_i \mid D_i = 1] - [Y_i \mid D_i = 0] = \;???

Observed Data

Suppose four students take the test:

  • Two are in treatment (D_i = 1), they took the class.
  • Two are in control (D_i = 0), they did not.
D_i Y_i
1 88
1 96
0 46
0 68

What We Want

Each row also has two potential outcomes, but we only see one of them.

ATE = \mathbb{E}[Y_i^1] - \mathbb{E}[Y_i^0] = \;???

D_i Y_i Y_i^1 Y_i^0
1 88 88 ???
1 96 96 ???
0 46 ??? 46
0 68 ??? 68

A Tempting Substitution

Plug in the observed group means for the missing potential outcomes.

\widehat{ATE} = \mathbb{E}[Y_i \mid D_i = 1] - \mathbb{E}[Y_i \mid D_i = 0]

= \frac{88 + 96}{2} - \frac{46 + 68}{2} = 35

D_i Y_i Y_i^1 Y_i^0
1 88 88 ???
1 96 96 ???
0 46 ??? 46
0 68 ??? 68

Learning the Truth

If we could see the true potential outcomes, the real average treatment effect would be much smaller.

ATE = \mathbb{E}[Y_i^1] - \mathbb{E}[Y_i^0]

= \frac{(88-80) + (96-88) + (54-46) + (76-68)}{4}

= 8

D_i Y_i Y_i^1 Y_i^0 Y_i^1 - Y_i^0
1 88 88 counterfactual: 80 8
1 96 96 counterfactual: 88 8
0 46 counterfactual: 54 46 8
0 68 counterfactual: 76 68 8

Why the Substitution Failed

The control group is a bad counterfactual. Note that \mathbb{E}[Y_i^0 \mid D_i = 1] \gg \mathbb{E}[Y_i^0 \mid D_i = 0]

  • \mathbb{E}[Y_i^1 \mid D_i = 1] = \{88, 96\}
  • \mathbb{E}[Y_i^0 \mid D_i = 0] = \{46, 68\}
  • counterfactual: \color{#b4321f}{\mathbb{E}[Y_i^0 \mid D_i = 1] = \{80, 88\}}
  • counterfactual: \color{#b4321f}{\mathbb{E}[Y_i^1 \mid D_i = 0] = \{54, 76\}}
D_i Y_i^1 Y_i^0
1 88 counterfactual: 80
1 96 counterfactual: 88
0 counterfactual: 54 46
0 counterfactual: 76 68

The Fundamental Question of Causal Inference (as invented by Prof. Noble)

  • When can we assume

\mathbb{E}[Y_i \mid D_i = 1] - \mathbb{E}[Y_i \mid D_i = 0] \;\approx\; \mathbb{E}[Y_i^1] - \mathbb{E}[Y_i^0]\;?

  • When the potential outcomes are independent of treatment assignment:

(Y_i^1, Y_i^0) \perp D_i

Better Counterfactuals

When the groups are comparable, the naive difference recovers the truth.

ATE = \frac{8 + 8 + 10 + 6}{4} = 8

\widehat{ATE} = \frac{88 + 96}{2} - \frac{84 + 84}{2} = 8

D_i Y_i Y_i^1 Y_i^0 Y_i^1 - Y_i^0
1 88 88 counterfactual: 80 8
1 96 96 counterfactual: 88 8
0 84 counterfactual: 94 84 10
0 84 counterfactual: 90 84 6

What About This ATT We Keep Talking About?

The ATE is the average effect across all units in the study. The ATT is the average effect across the treatment group only.

  • ATE = \mathbb{E}[Y_i^1] - \mathbb{E}[Y_i^0]
  • ATT = \mathbb{E}[Y_i^1 \mid D_i = 1] - \mathbb{E}[Y_i^0 \mid D_i = 1]
D_i Y_i Y_i^1 Y_i^0 Y_i^1 - Y_i^0
1 88 88 counterfactual: 80 8
1 96 96 counterfactual: 88 8
0 84 counterfactual: 94 84 10
0 84 counterfactual: 90 84 6

When ATE and ATT Diverge

Replace unit four with a first year who barely benefits from the class.

  • \widehat{ATE} = 6.75 across all four units.
  • \widehat{ATT} = 8 across the treated only.

The ATT is the effect for the kind of student who actually takes the class.

D_i Y_i Y_i^1 Y_i^0 Y_i^1 - Y_i^0
1 88 88 counterfactual: 80 8
1 96 96 counterfactual: 88 8
0 84 counterfactual: 94 84 10
0 43 counterfactual: 44 43 1

Estimating ATE and ATT in Practice

  • ATE = \mathbb{E}[Y_i^1] - \mathbb{E}[Y_i^0]
    • \widehat{ATE} = \mathbb{E}[Y_i \mid D_i = 1] - \mathbb{E}[Y_i \mid D_i = 0]
  • ATT = \mathbb{E}[Y_i^1 \mid D_i = 1] - \mathbb{E}[Y_i^0 \mid D_i = 1]
    • \widehat{ATT} = \mathbb{E}[Y_i \mid D_i = 1] - \mathbb{E}[Y_i \mid D_i = 0]
  • The formula is the same. The population we’re claiming to describe changes.
D_i Y_i Y_i^1 Y_i^0 Y_i^1 - Y_i^0
1 88 88 ?? ??
1 96 96 ?? ??
0 84 ?? 84 ??
0 43 ?? 43 ??

Every Causal Method Is Designed to Create a Good Counterfactual

Every method we cover is a different strategy to eliminate selection bias:

  • OLS: control for variables confounding treatment and control.
  • IV: find exogenous variation that shifts treatment assignment.
  • RDD: compare units just above and below a threshold.
  • DiD: subtract out unit-specific baseline levels.