What is β? On the Statistical vs. Structural Interpretation of Linear Regression
The Statistical Interpretation
In his 1922 seminal paper “On the Mathematical Foundations of Theoretical Statistics,” Ronald Fisher – arguably the father of modern mathematical statistics – succinctly summarized what he believed was the objective of statistics:
Briefly, and in its most concrete form, the object of statistical methods is the reduction of data.
To motivate the goal of data reduction, Fisher adopted a mathematical model of the world. First, assume there exists a hypothetical infinite population of only observable random variables \(X_1, \ldots, X_K\) with a fixed but unknown joint distribution \(f\). Second, assume the observed data \(\{X_{i1}, \ldots, X_{iK}\}_{i=1}^n\) are a finite number of independent and identically distributed (iid) random variables drawn from \(f\). In other words, the underlying mechanism that generates the observed data – called the data generating process (DGP) – is fully characterized by \(f\). These two assumptions together allow statisticians to estimate features of the joint distribution \(f\) – such as its moments – using their sample analog in the observed data. Thus, by framing data reduction as the process of learning about some fixed, unknown feature of a hypothetical infinite population, Fisher laid the epistemological foundations for statistical methods.
The Linear Regression of Output on Labor and Capital Inputs
Let’s make the above ideas concrete using the linear regression of output \(Q_t\) on labor inputs \(L_t\) and capital inputs \(K_t\) presented in Reiss and Wolak (2007). For brevity in notation, define the vector of regressors \(X_t \equiv (1, L_t, K_t)'\). A moment of the unknown distribution \(f\) that we are often interested in learning about is the conditional expectation function (CEF) of \(Q_t\) given \(X_t\), \[ m(X_t) \equiv \mathbb{E}[Q_t \mid X_t]. \] The best linear predictor (BLP) of the CEF is defined as \[ BLP(Q_t \mid X_t) \equiv X_t'\beta_{BLP}, \] where the coefficients are the minimizer of the mean squared error (MSE) between the CEF and all linear functions of \(X_t\) \[ \beta_{BLP}= {\arg\min}_b \, \mathbb{E}[(m(X_t)-X'_tb)^2]. \] Solving the first order condition of the above optimization problem, we get the following formula for the coefficient vector \[ \beta_{BLP} = \mathbb{E}[X_tX_t']^{-1}\mathbb{E}[X_tQ_t]. \] Defining the population residual as \[ U_t \equiv Q_t - X_t'\beta_{BLP}, \] we can decompose the quantity \(Q_t\) as \[ Q_t = X'_t\beta_{BLP} + U_t. \] Observe the property that the residual \(U_t\) is orthogonal to the regressors \(X_t\) follows immediately from its definition: \[ \begin{align} \mathbb{E}[X_tU_t]&=\mathbb{E}\bigg[X_t\big(Q_t-X'_t \,\underbrace{\mathbb{E}[X_tX'_t]^{-1}\mathbb{E}[X_tQ_t]}_{=\beta_{BLP}}\big)\bigg] \\ &= \mathbb{E}[X_tQ_t] - \mathbb{E}[X_tX'_t]\mathbb{E}[X_tX'_t]^{-1}\mathbb{E}[X_tQ_t]= 0. \end{align} \] It can be shown that the iid assumption combined with the orthogonal property of the residuals implies that a consistent estimator of \(\beta_{BLP}\) is given by replacing the population quantities with their sample analogs: \[ B \equiv \bigg(\frac{1}{n}\sum_{t=1}^n X_tX_t'\bigg)^{-1} \bigg(\frac{1}{n}\sum_{t=1}^n X_tQ_t\bigg). \] Realizations of the estimator \(B\) can thus be interpreted as giving us insight into a feature of the unknown distribution \(f\) governing the observable random variables.