Difference-in-Differences

John Snow, Parallel Trends, and the DiD Estimator

Professor Benjamin Noble

Let’s Talk About John Snow…

Jon Snow from Game of Thrones, looking thoughtful in winter furs.

via HBO

No, the Other John Snow…

Black and white portrait of the epidemiologist John Snow, seated.

Illustration of Victorian London along the Thames, with people strolling on an embankment, boats on the river, and landmarks including Big Ben and St. Paul's Cathedral in the background.

Snow’s Cholera Numbers

Illustration contrasting two Victorian water sources: on the left, people drawing water from a polluted Thames River in an industrial city; on the right, people drawing clean water from an upstream rural pump.

Company name Cholera, 1849 Cholera, 1854
Southwark and Vauxhall 135 147
Lambeth 85 19

Source: Cunningham (2021)

Am I a Good Teacher (Redux, Redux, For the Last Time…)

  • Good: score on a 100-point data analysis test.
  • Suppose we have a binary treatment variable:

D_i = \begin{cases} 1 & \text{if } i \text{ takes POLI 170} \\ 0 & \text{if } i \text{ does not take POLI 170} \end{cases}

  • Suppose we have an outcome variable, Y_i, which is person i’s score on the test.

Bad Counterfactuals

Illustration of a professor talking with students on the UC San Diego campus.

Di Yi Yi(1) Yi(0)
1 90 90 ???
1 98 98 ???
0 48 ??? 48
0 70 ??? 70

If We Could See the Counterfactuals

Illustration of a professor talking with students on the UC San Diego campus.

Di Yi Yi(1) Yi(0)
1 90 90 82
1 98 98 90
0 48 56 48
0 70 78 70

A Pre-Test at the Start of the Quarter

Di Yi (T=0) Yi (T=1) Δ
1 80
1 88
0 46
0 68

A Post-Test at the End of the Quarter

Di Yi (T=0) Yi (T=1) Δ
1 80 90 +10
1 88 98 +10
0 46 48 +2
0 68 70 +2

The Difference-in-Differences Estimator and ATT

Di Yi (T=0) Yi (T=1) Δ
1 80 90 +10
1 88 98 +10
0 46 48 +2
0 68 70 +2

\begin{align*} ATT_{\text{DiD}} &= \Big(\mathbb{E}[Y_{i, D_i = 1} \mid T = 1] - \mathbb{E}[Y_{i, D_i = 1} \mid T = 0]\Big) \\ &\quad - \Big(\mathbb{E}[Y_{i, D_i = 0} \mid T = 1] - \mathbb{E}[Y_{i, D_i = 0} \mid T = 0]\Big) \end{align*}

\begin{align*} ATT_{\text{DiD}} &= \left(\frac{90 + 98}{2} - \frac{80 + 88}{2}\right) - \left(\frac{70 + 48}{2} - \frac{68 + 46}{2}\right) \\ &= (94 - 84) - (59 - 57) = 10 - 2 = 8 \end{align*}

In Pictures: The Naive ATE

Average test scores before and after the course. POLI 170 is shown with blue circles and a solid line; control is shown with coral triangles and a long-dashed line. Both groups increase over time, and a vertical bracket at the post period marks the large gap between groups as the naive ATE.

In Tables: Naive Differences

Took 170 Time Test Score Time Difference Treatment Difference
Yes Y = S + D
No Y = F

Based on Cunningham (2021)

In Tables: Naive Differences

Took 170 Time Test Score Time Difference Treatment Difference
Yes Y = S + D
No Y = F
D + (S − F)

Based on Cunningham (2021)

In Tables: Adding Time

Took 170 Time Test Score Time Difference Treatment Difference
Yes Before Y = S
After Y = S + (D + T) D + T
No Before
After

Based on Cunningham (2021)

In Tables: Difference of Differences

Took 170 Time Test Score Time Difference Treatment Difference
Yes Before Y = S
After Y = S + (D + T) D + T
No Before Y = F
After Y = F + T T
D

Based on Cunningham (2021)

In Pictures: The DiD Estimate

  • The solid blue line is the actual treatment group.
  • The dashed blue line is the counterfactual: what the treated would have looked like under the control group’s trend.
  • The DiD estimate is the gap between those two endpoints.

Average test scores before and after the course. POLI 170 is shown with blue circles and a solid line; control is shown with coral triangles and a long-dashed line. A dashed blue counterfactual starts at the POLI 170 pre-period mean and follows the control group's change. A bracket at the post period marks the gap between the observed POLI 170 mean and its counterfactual as the DiD ATT.

Using Regression

  • Regress the outcome on the treatment, the time variable, and their interaction: y ~ D + T + D * T
  • The DiD effect is the coefficient on D x T (in R, written as D:T).
  • Remember our interactions?
Term Estimate Std. Error p-value
(Intercept) 49.90 0.27 < .001
D 30.48 0.39 < .001
T 2.10 0.39 < .001
D:T 7.92 0.55 < .001