Extraordinary Least Squares

Best-Fit Lines, Statistical Control, and Uncertainty

Professor Benjamin Noble

Regression Is About Relationships

We want to know if X causes Y. But first, we need to describe the relationship between them.

  • Does Y change as X changes?
  • Is the relationship positive or negative?
  • How big is the relationship?
  • Is the relationship real or just noise?

Do Taller People Earn More Money?

Scatterplot of earnings against height showing a cloud of points with a slight upward drift.

Do Taller People Earn More Money?

Scatterplot of earnings against height with the best-fit regression line through the cloud of points.

Do Taller People Earn More Money?

Scatterplot of earnings against height with a deliberately too-steep line that misses the cloud of points.

Linear Regression

y = mx + b y_i = \beta_0 + \beta_1 x_i + \epsilon_i

Where…

  • y_i is the dependent/outcome variable.
  • x_i is the independent/predictor/explanatory variable.
  • \beta_0 is the intercept.
  • \beta_1 is the slope.
  • \epsilon_i is the error (everything about y we don’t capture with x).

Modeling the Gender Gap

Quantity Value
$0K $20K $40K $60K $80K +14.3KFemaleMaleEarnings ($1K)

Reading Regression Output

summary(lm(earnk ~ male, data = earnings))
Term Estimate Std. Error t value p-value
(Intercept) 15.85 0.64 24.9 < .001
male 14.26 1.04 13.7 < .001

R-squared 0.094

Why We Need Regression

With a binary variable, computing averages is easy. With a continuous variable, it isn’t.

  • Sex has two values → two group means, one difference.
  • Height has many values → averaging at every height gets unwieldy.
  • Regression uses one fitted line to approximate the averages.
  • Scales to many variables at once.

Modeling Height and Earnings

lm(earnk ~ height, data = earnings)
Term Estimate Std. Error
(Intercept) -85.03 8.86
height 1.59 0.13
R-squared 0.074

Scatterplot of earnings against height with the fitted regression line.

Regression as Prediction

Plug a height into the equation to predict earnings.

  • Equation: earnings = -85.03 + 1.59 × height
  • Plug in 60: -85.03 + 1.59 × 60 = 10.37
  • A 60-inch person is predicted to earn about $10K.
Height (in) Predicted Earnings ($1K)
58 7.2
60 10.4
65 18.3
70 26.3
72 29.5

Where Does the Line Come From?

Some lines miss the points by less than others. Which line is best?

  • A residual is the vertical miss: y_i - \hat{y}_i
  • Bad lines have big residuals, good lines have small ones.
  • We need one number that summarizes how “bad” a line is. Choose the least bad.
  • Choose \beta_0 and \beta_1 to minimize the sum of squared residuals:

\sum (y_i - \hat{y}_i)^2 = \sum (y_i - \beta_0 - \beta_1 x_i)^2

Five points with a teal-blue best-fit line through them, red-orange vertical residual segments, and three worse-fitting dashed gray lines.

The Conditional Expectation Function

E[Y \mid X = x], the average y at a chosen value of x.

  • With a binary predictor, there are two conditional means.
  • They match the regression line.
  • That isn’t an accident: for a single binary predictor, regression is the conditional mean.
Condition Mean earnings ($1K)
male = 0 15.8
male = 1 30.1

Jittered earnings by sex with diamond markers for the two group means sitting exactly on the regression line.

The CEF and Regression for Height

  • The dashed line (with triangle markers) is the the CEF: average earnings at each height.
  • The solid line is the regression of earnings on height.
  • Regression is the best linear approximation of the CEF.

Average earnings at each height shown as red-orange triangles joined by a dashed line, with the straight teal-blue regression line running through them.

But Wait: Does Height Really Matter?

  • Men are taller than women on average.
  • Men also earn more, on average.
  • So the height coefficient may be partly a disguised gender difference.
  • Regression lets us ask whether height still matters after holding sex constant.

Controlling for Sex

Adding sex to the model shrinks the height coefficient from 1.60 to 0.65.

lm(earnk ~ male + height, data = earnings)
Term Estimate Std. Error t value p-value
(Intercept) -25.87 11.96 -2.2 0.031
male 10.63 1.47 7.2 < .001
height 0.65 0.19 3.5 < .001
R-squared 0.1

Earnings against height with separate parallel fitted lines for women and men, labeled directly on the lines, plus a black dashed line showing the original single regression line that ignores sex.

Another Visualization: A Plane Through the Data

Three-dimensional scatterplot of earnings against sex and height with the fitted regression plane tilted through the points.

Danger: Extrapolating Beyond the Data

Regression will make predictions even where there’s no data.

Earnings against height with parallel fitted lines for women and men and a yellow band above 75 inches, where the data contain no women.

Interactions

An interaction lets the height slope differ for women and men.

lm(earnk ~ male * height, data = earnings)
Term Estimate Std. Error
(Intercept) -8.49 15.87
male -31.51 25.33
height 0.38 0.25
male:height 0.62 0.37

Earnings against height with separately fitted, non-parallel lines for women and men; the line for men is steeper.

Adding Additional Predictors

  • Each coefficient compares people who match on every other variable in the model.
    • The height coef now holds sex, age, education, and race constant.
  • More controls means fewer real matches in the data backing each comparison.
Term Estimate Std. Error p-value
(Intercept) -60.74 11.88 < .001
male 11.40 1.39 < .001
height 0.48 0.18 0.007
education 2.70 0.19 < .001
age 0.19 0.03 < .001
ethnicityHispanic -0.93 2.50 0.71
ethnicityOther 1.22 3.61 0.736
ethnicityWhite 2.22 1.60 0.166

How Certain Are We About These Estimates?

  • We estimated people earn about $1.6K per inch.
  • But…another sample of 1,800 people could give a slightly different slope.
  • That sample-to-sample movement is sampling uncertainty.
  • Standard errors tell us how much the estimate would move across samples.

Sampling Variation (The Bootstrap)

  • Draw 1,800 people from the data, with replacement.
  • Refit regression model on new data and save the slope.
  • Repeat 1,000 times.
  • The standard deviation of the slopes approximates the standard error.
A single bootstrap slope estimate near 1.3, with a vertical line marking our actual estimate of 1.59. With only one sample there is no distribution yet, just one number.

Where Standard Errors Come From

The standard error captures how much the slope shifts across repeated samples.

se_{\hat{\beta}_1} = \sqrt{\frac{\sum \hat \epsilon_i^2}{(n - k)\sum (x_i - \bar{x})^2}}

  • Numerator (squared residuals): a worse-fitting line means a bigger SE.
  • Denominator: bigger n or more spread in x means a smaller SE.

What About p-values and Statistical Significance?

  • Probability of observing a test statistic as extreme as observed under the assumption that the true coefficient is 0.
  • Divide the coefficient by the SE. If the coefficient is at least twice its standard error, the estimate is “hard to explain” as noise.
  • The “2” comes from the normal distribution, where two standard deviations capture 95% of the data.
Histogram of 1,000 bootstrap height-coefficient estimates forming a bell shape centered near 1.6. A vertical line marks our estimated slope of 1.6, and two more lines mark two standard errors above and below it, from about 1.3 to 1.9. That band sits entirely above zero, so the estimate is statistically significant.

Why Regression?

Regression turns messy data into interpretable numbers.

  • A cloud of points becomes a line.
  • A line becomes a coefficient.
  • A coefficient becomes an interpretation.
  • Uncertainty tells us how much to trust it.