price nbath nbed sqft city pets
1 1199 2 3 1200 Columbia yes
2 1199 2 3 1450 Columbia yes
3 1299 2 3 1950 Columbia yes
4 1175 2 3 1312 Columbia yes
5 1349 2 4 2015 Columbia yes
6 1225 2 3 1336 Columbia yesSTAT 516 Lec 03
Multiple linear regression (part 1/2)
Karl Gregory
2026-10-26
SC apartments example
These data are a subset of “Apartment for Rent Classified” (2019).
There are \(n = 214\) data points.
SC apartments
SC apartments
SC apartments
Setup
Consider data \((\mathbf{x}_1,Y_1),\dots,(\mathbf{x}_1,Y_n)\), with each \(\mathbf{x}_i = (x_{i1},\dots,x_{ip})^T\).
The multiple linear regression model is
\[ Y_i = \beta_0 + x_{i1}\beta_1 + \dots + x_{ip}\beta_p + \varepsilon_i, \quad i = 1,\dots,n, \] where
- \(\mathbf{x}_1,\dots,\mathbf{x}_n\) are vectors in \(\mathbb{R}^p\) of covariate or predictor values.
- \(Y_1,\dots,Y_n\) are the response values
- \(\beta_0, \beta_1, \dots, \beta_p\) are the regression coefficients.
- \(\varepsilon_1,\dots,\varepsilon_n\) are iid \(\text{Normal}(0,\sigma^2)\) error terms.
- \(\sigma^2\) is the error term variance.
Goals in multiple linear regression
As in simple linear regression (SLR), will learn how to
- Estimate the regression coefficients \(\beta_0\) and \(\beta_1,\dots,\beta_p\).
- Estimate the error term variance \(\sigma^2\).
- Perform inference on \(\beta_1,\dots,\beta_p\).
- Build a CI for \(\beta_0 + \beta_1 x_{\operatorname{new},1} + \dots + \beta_p x_{\operatorname{new},p}\) at any \(\mathbf{x}_{\operatorname{new}}\).
- Build a prediction interval for \(Y\) at any \(\mathbf{x}_{\operatorname{new}}\).
- Decompose the variation in \(Y\) into (sums of) sums of squares.
- Check whether the model assumptions are satisfied.
- Identify outliers and understand their effects.
Beyond the above, in multiple linear regression we wish to
- Include interaction effects
- Test for significance of a subset of covariates.
- Understand how correlations among the covariates affect inferences.
- Do variable selection.
Latter goals considered in part 2/2.
Least-squares estimation of regression coefficients
Define the squared error criterion as \[ Q(b_0,b_1,\dots,b_p) = \sum_{i=1}^n(Y_i - (b_0 + b_1 x_{i1} + \dots + b_p x_{ip}))^2. \] Suppose \(Q(b_0,b_1,\dots,b_p)\) is uniquely minimized at \((\hat \beta_0,\hat \beta_1,\dots,\hat \beta_p)\).
Then we call \(\hat \beta_0,\hat \beta_1,\dots,\hat \beta_p\) the least-squares estimators of \(\beta_0,\beta_1,\dots,\beta_p\).
The best way to compute \(\hat \beta_0,\hat \beta_1,\dots,\hat \beta_p\) is with matrix calculations
Linear regression model in matrix form
Write equations \(Y_i = \beta_0 + x_{i1}\beta_1 + \dots + x_{ip}\beta_p + \varepsilon_i\), for \(i = 1,\dots,n\), as
\[\begin{align*} Y_1 &= \beta_0 + \beta_1 x_{11} + \dots + \beta_p x_{1p} + \varepsilon_1\\ Y_2 &= \beta_0 + \beta_1 x_{21} + \dots + \beta_p x_{2p} + \varepsilon_2 \\ & \vdots \\ Y_n &= \beta_0 + \beta_1 x_{n1} + \dots + \beta_p x_{np} + \varepsilon_n \end{align*}\]
Now set \[ \mathbf{Y}= \left[\begin{matrix}Y_1 \\ Y_2 \\ \vdots \\ Y_n \end{matrix}\right], \quad \mathbf{X}= \left[ \begin{matrix} 1 &x_{11}& \dots & x_{1p} \\ 1 &x_{21}& \dots & x_{2p} \\ \vdots &\vdots& \ddots & \vdots\\ 1 &x_{n1}& \dots & x_{np} \end{matrix} \right],\quad \mathbf{b}= \left[\begin{matrix}\beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{matrix}\right], \quad \mathbf{e}= \left[\begin{matrix}\varepsilon_1 \\ \varepsilon_2 \\ \vdots \\ \varepsilon_n \end{matrix}\right] \] Then the above equations can be written in matrix form as \(\mathbf{Y}= \mathbf{X}\mathbf{b}+ \mathbf{e}\).
Least-squares estimators in matrix form
Provided \(\mathbf{X}^T\mathbf{X}\) is invertible, the entries of the vector \[ \hat{\mathbf{b}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y} \] give the least-squares estimators \(\hat \beta_0,\hat \beta_1,\dots,\hat \beta_p\).
Important: Can only compute \(\hat{\mathbf{b}}\) if no column of \(\mathbf{X}\) can be constructed as a linear combination of other columns (equivalent to \(\mathbf{X}^T\mathbf{X}\) invertible).
SC apartments
Make indicator or dummy variables to encode the four cities:
\[\begin{align*} \texttt{Columbia} &= \left\{\begin{array}{ll} 1& \text{ if \texttt{city} is Columbia}\\ 0& \text{otherwise }\end{array}\right. \\ \texttt{Greenville} &= \left\{\begin{array}{ll} 1& \text{ if \texttt{city} is Greenville}\\ 0& \text{otherwise }\end{array}\right. \\ \texttt{Rock Hill} &= \left\{\begin{array}{ll} 1& \text{ if \texttt{city} is Rock Hill}\\ 0& \text{otherwise }\end{array}\right. \end{align*}\]
Then Charleston is indicated by \((\texttt{Columbia},\texttt{Greenville},\texttt{Rock Hill}) = (0,0,0)\).
Note: If we included in \(\mathbf{X}\) an indicator for all four levels, \(\mathbf{X}^T\mathbf{X}\) would not be invertible due to the presence of the intercept column (this is linear algebra knowledge)!
SC apartments
The \(\texttt{pets}\) column contains the values “yes” and “no”.
Convert it to a dummy variable column: \[ \texttt{pets} = \left\{\begin{array}{ll} 1& \text{ if yes}\\ 0& \text{ if no} \end{array}\right. \]
SC apartments
We wish to estimate the coefficients in the model
\[\begin{align*} \log(\texttt{price}) = \beta_0 &+ \beta_1 \cdot \texttt{Columbia} + \beta_2 \cdot \texttt{Greenville} + \beta_3 \cdot \texttt{Rock Hill} \\ &+ \beta_4 \cdot \texttt{nbath} + \beta_5 \cdot \texttt{nbed} + \beta_6 \log(\texttt{sqft}) + \beta_7 \cdot \texttt{pets} + \varepsilon, \end{align*}\]
where now all the covariates take numeric values.
SC apartments
The code below demonstrates the construction of the design matrix \(\mathbf{X}\).
Note how the values in \(\texttt{city}\) and \(\texttt{pets}\) are encoded with dummy variables.
Y <- log(scapts$price)
X <- cbind("Intercept" = rep(1,n),
"Columbia" = ifelse(scapts$city == "Columbia",1,0),
"Greenville" = ifelse(scapts$city == "Greenville",1,0),
"Rock Hill" = ifelse(scapts$city == "Rock Hill",1,0),
"nbath" = scapts$nbath,
"nbed" = scapts$nbed,
"log_sqft" = log(scapts$sqft),
"pets" = ifelse(scapts$pets == "yes",1,0))SC apartments
This code computes \(\hat{\mathbf{b}}\) from \(\mathbf{X}\) and \(\mathbf{Y}\).
Interpretation of the slope parameters
Consider what stories \(\hat \beta_0,\hat \beta_1,\dots,\hat \beta_p\) tell in the fitted MLR model \[ \hat Y_i = \hat \beta_0 + x_{i1}\hat \beta_1 + \dots + x_{ip}\hat \beta_p , \quad i =1,\dots,n. \]
- \(\hat \beta_0\), as in SLR, just gives the function the right “height”.
- \(\hat \beta_j\) is the amount by which the mean of \(Y\) is estimated to change due to a 1-unit increase in covariate \(j\), with all other variables held fixed.
- Take note of how interpretations change when variables are transformed!
SC apartments
We can represent the fitted model as:
\[\begin{align*} \widehat{\log(\texttt{price})} = 3.933 &+ (-0.241) \cdot \texttt{Columbia} + (-0.12) \cdot \texttt{Greenville} \\ & + (-0.145) \cdot \texttt{Rock Hill} + (-0.107) \cdot \texttt{nbath} \\ &+ (0.065) \cdot \texttt{nbed} + (0.473) \cdot \log(\texttt{sqft}) \\ &+ (-0.019)\cdot \texttt{pets}. \end{align*}\]
SC apartments
The effect of an additional bedroom (all else being equal) is to increase the expected log price by \(0.065\), or to increase the expected price by a factor of \(1.067\), i.e. by \(6.666\) percent.
An apartment with \(1\%\) more square feet (all else being equal) is expected to cost approximately \(0.473\%\) more.
An apartment which allows pets (all else being equal) is expected to have a log price \(0.019\) lower, that is, to be cheaper by a factor of \(0.982\), i.e. to be \(1.84\%\) cheaper.
In which city are apartments most expensive (all else being equal)?
Do not omit all else being equal (or ceteris paribus in the Latin ;-))!
Estimating the error term variance
After obtaining \(\hat \beta_0,\hat \beta_1,\dots,\hat \beta_p\), define the
- fitted values as \(\hat Y_i = \hat \beta_0 + \hat\beta_1 x_{i1} + \dots + \hat \beta_{p}x_{ip}\)
- residuals as \(\hat \varepsilon_i = Y_i - \hat Y_i\)
for \(i = 1,\dots,n\).
Then an unbiased estimator of \(\sigma^2\) is given by \[ \hat \sigma^2 = \frac{1}{n-(p+1)} \sum_{i=1}^n \hat \varepsilon_i^2. \]
SC apartments
The code below computes the fitted values, the residuals, and the estimator of the error term variance.
For the SC apartments regression we obtain \(\hat \sigma^2 = 0.043\).
Confidence intervals for the slope parameters
Let \(\displaystyle \Omega = (\mathbf{X}^T\mathbf{X})^{-1}\) with \(\Omega_{jj}\) the diagonal entry corresponding to \(\beta_j\).
Then the estimator \(\hat \beta_j\) is distributed as \[ \hat \beta_j \sim \text{Normal}\left(\beta_j,\sigma^2 \Omega_{jj}\right). \]
“Studentizing” the above gives \[ \frac{\hat \beta_j - \beta_j}{\hat \sigma\sqrt{ \Omega_{jj}}} \sim t_{n-(p+1)}. \]
So a \((1-\alpha)100\%\) confidence interval for \(\beta_j\) is \[ \hat \beta_j \pm t_{n-(p+1),\alpha/2} \hat \sigma\sqrt{\Omega_{jj}}. \]
We often write \(\widehat{\text{s.e.}}(\hat \beta_1) = \hat \sigma \sqrt{\Omega_{jj}}\), s.e. for standard error.
SC apartments
This code constructs 95% CIs for the slope coefficients.
estimate lower upper
Intercept 3.9327 2.9839 4.8816
Columbia -0.2409 -0.3207 -0.1611
Greenville -0.1195 -0.1909 -0.0481
Rock Hill -0.1454 -0.2397 -0.0510
nbath -0.1073 -0.1927 -0.0219
nbed 0.0645 -0.0026 0.1317
log_sqft 0.4728 0.3188 0.6268
pets -0.0186 -0.0783 0.0411Tests of hypotheses about the slope coeffcients
We most often test hypotheses about the \(\beta_j\) of the form \[ \begin{array}{lclcl} H_0\text{: } \beta_j \geq 0 &\text{ or } & H_0\text{: } \beta_j = 0 &\text{ or } & H_0\text{: } \beta_j \leq 0\\ H_1\text{: } \beta_j < 0 && H_1\text{: } \beta_j \neq 0 && H_1\text{: } \beta_j > 0. \end{array} \] Reject or fail to reject \(H_0\) based on the value of the test statistic \[ T_{\operatorname{test}} = \frac{\hat \beta_j }{\hat\sigma \sqrt{\Omega_{jj}}}. \] Rejection rules for the above at significance level \(\alpha\) are \[ \begin{array}{lclcl} T_{\operatorname{test}} < -t_{n-(p+1),\alpha} &\text{ or } & |T_{\operatorname{test}}| > t_{n-(p+1),\alpha/2}&\text{ or } & T_{\operatorname{test}} > t_{n-(p+1),\alpha}. \end{array} \]
The corresponding p-values are, with \(T \sim t_{n-(p+1)}\), the probabilities \[ \begin{array}{lclcl} P(T < T_{\operatorname{test}}) &\text{ or } & 2 \times P(T > |T_{\operatorname{test}}|) & \text{ or } & P(T > T_{\operatorname{test}}). \end{array} \]
SC apartments
Obtain p-values for testing \(H_0\): \(\beta_j = 0\) vs \(H_1\): \(\beta_j \neq 0\) for each \(j\).
estimate sehat Tstat pval
Intercept 3.9327 0.4812 8.172 0.0000
Columbia -0.2409 0.0405 -5.951 0.0000
Greenville -0.1195 0.0362 -3.299 0.0011
Rock Hill -0.1454 0.0479 -3.037 0.0027
nbath -0.1073 0.0433 -2.476 0.0141
nbed 0.0645 0.0341 1.895 0.0595
log_sqft 0.4728 0.0781 6.053 0.0000
pets -0.0186 0.0303 -0.614 0.5402SC apartment
Demonstrate the lm(), summary(), and confint() functions in R:
Call:
lm(formula = log(price) ~ city + nbath + nbed + log(sqft) + pets,
data = scapts)
Residuals:
Min 1Q Median 3Q Max
-0.6528 -0.1243 -0.0150 0.0876 0.7166
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 3.9327 0.4812 8.17 3.0e-14 ***
cityColumbia -0.2409 0.0405 -5.95 1.1e-08 ***
cityGreenville -0.1195 0.0362 -3.30 0.0011 **
cityRock Hill -0.1454 0.0479 -3.04 0.0027 **
nbath -0.1073 0.0433 -2.48 0.0141 *
nbed 0.0645 0.0341 1.89 0.0595 .
log(sqft) 0.4728 0.0781 6.05 6.6e-09 ***
petsyes -0.0186 0.0303 -0.61 0.5402
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.208 on 206 degrees of freedom
Multiple R-squared: 0.414, Adjusted R-squared: 0.394
F-statistic: 20.8 on 7 and 206 DF, p-value: <2e-16SC apartments
2.5 % 97.5 %
(Intercept) 2.98394 4.8816
cityColumbia -0.32071 -0.1611
cityGreenville -0.19093 -0.0481
cityRock Hill -0.23972 -0.0510
nbath -0.19270 -0.0219
nbed -0.00261 0.1317
log(sqft) 0.31884 0.6268
petsyes -0.07826 0.0411CI for the mean and PI for \(Y_{\operatorname{new}}\) at \(\mathbf{x}_{\operatorname{new}}\)
For a new vector of covariate values \(\mathbf{x}_{\operatorname{new}}\), let \[ \hat Y_{\operatorname{new}} = \hat \beta_0 + \hat \beta_1 x_{\operatorname{new},1} + \dots + \hat \beta_p x_{\operatorname{new},p} \]
A \((1-\alpha)\times 100\) CI for \(\beta_0 + \beta_1 x_{\operatorname{new},1} + \dots + \beta_p x_{\operatorname{new},p}\) is given by \[ \hat Y_{\operatorname{new}} \pm t_{n-(p+1),\alpha/2} \hat \sigma \sqrt{ \Omega_{\operatorname{new}}}, \]
A \((1-\alpha)\times 100\) PI for \(Y_{\operatorname{new}}\) corresponding to \(\mathbf{x}_{\operatorname{new}}\) is given by \[ \hat Y_{\operatorname{new}} \pm t_{n-(p+1),\alpha/2} \hat \sigma \sqrt{1 + \Omega_{\operatorname{new}}}, \] where \(\Omega_{\operatorname{new}} = \tilde{\mathbf{x}}_{\text{new}}^T \Omega ~\tilde{\mathbf{x}}_{\text{new}}^T\) with \(\tilde{\mathbf{x}}_{\text{new}}^T = (1 ~ x_{\operatorname{new},1} ~ \cdots ~ x_{\operatorname{new},p})^T\).
SC apartments
Build 95% CI for the average rent of apartments in Greenville with one bathroom, two bedrooms, nine-hundred square feet, and which allow pets.
The confidence interval is \([1040.545,1234.266]\).
SC apartments
Now build a 95% PI for the rent of a single such a property.
The prediction interval is \([745.099,1723.675]\).
The predict() function in R
Sums of squares in multiple linear regression
We decompose the variation in \(Y_1,\dots,Y_n\) by defining the:
- Total sum of squares: \(\operatorname{SS}_{\operatorname{Tot}} = \sum_{i=1}^n(Y_i - \bar Y_n)^2\)
- Regression sum of squares: \(\operatorname{SS}_{\operatorname{Reg}} = \sum_{i=1}^n(\hat Y_i - \bar Y_n)^2\)
- Error sum of squares: \(\operatorname{SS}_{\operatorname{Error}} = \sum_{i=1}^n(Y_i - \hat Y_i)^2\)
We have \(\operatorname{SS}_{\operatorname{Tot}} = \operatorname{SS}_{\operatorname{Reg}} + \operatorname{SS}_{\operatorname{Error}}\).
The coefficient of determination is defined as \(\displaystyle R^2 = \frac{\operatorname{SS}_{\operatorname{Reg}}}{\operatorname{SS}_{\operatorname{Tot}}}\).
- \(R^2 \in [0,1]\)
- Proportion of variation in \(Y\) “explained” by the covariates \(x_1,\dots,x_p\).
The mean squares in multiple linear regression
The SS, appropriately scaled, follow chi-square distributions:
- \(\displaystyle \operatorname{SS}_{\operatorname{Tot}}/\sigma^2 \sim \chi^2_{n-1}(\phi_{\operatorname{Tot}})\)
- \(\displaystyle \operatorname{SS}_{\operatorname{Reg}}/\sigma^2 \sim \chi^2_{p}(\phi_{\operatorname{Reg}})\)
- \(\displaystyle \operatorname{SS}_{\operatorname{Error}}/\sigma^2 \sim \chi^2_{n-(p+1)}\),
where \(\phi_{\operatorname{Tot}}\) and \(\phi_{\operatorname{Reg}}\) are noncentrality parameters.
Dividing \(\operatorname{SS}_{\operatorname{Reg}}\) and \(\operatorname{SS}_{\operatorname{Error}}\) by their dfs, we define:
- Regression mean square: \(\displaystyle \operatorname{MS}_{\operatorname{Reg}} = \frac{\operatorname{SS}_{\operatorname{Reg}}}{p}\)
- Error mean square: \(\displaystyle \operatorname{MS}_{\operatorname{Error}} = \frac{\operatorname{SS}_{\operatorname{Error}}}{n-(p+1)}\)
Moreover, define the adjusted R squared as \(\displaystyle \bar R^2 = 1 - \frac{\operatorname{MS}_{\operatorname{Error}} }{\operatorname{SS}_{\operatorname{Tot}}/(n-1)}\).
Adjustment “penalizes” the inclusion of additional covariates.
The Analysis of Variance (ANOVA) table
We often present the SS, df, and MS values in a table like this:
| Source | Df | SS | MS | F value | p-value |
|---|---|---|---|---|---|
| Reg | \(p\) | \(\operatorname{SS}_{\operatorname{Reg}}\) | \(\operatorname{MS}_{\operatorname{Reg}}\) | \(F_{\operatorname{test}}\) | \(P(F > F_{\operatorname{test}})\) |
| Error | \(n-(p+1)\) | \(\operatorname{SS}_{\operatorname{Error}}\) | \(\operatorname{MS}_{\operatorname{Error}}\) | ||
| Total | \(n-1\) | \(\operatorname{SS}_{\operatorname{Tot}}\) |
This is an example of an ANOVA table.
The F-value and the p-value we will discuss later in these slides.
Building the ANOVA table
| Source | Df | SS | MS | F value | p-value |
|---|---|---|---|---|---|
| Regression | 7 | 6.31 | 0.9 | 20.79 | 0 |
| Error | 206 | 8.93 | 0.04 | ||
| Total | 213 | 15.24 |
Moreover \(R^2 = 0.414\) and \(\bar R^2 = 0.394\).
ANOVA quantities in output from lm() with summary()
Call:
lm(formula = log(price) ~ city + nbath + nbed + log(sqft) + pets,
data = scapts)
Residuals:
Min 1Q Median 3Q Max
-0.6528 -0.1243 -0.0150 0.0876 0.7166
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 3.9327 0.4812 8.17 3.0e-14 ***
cityColumbia -0.2409 0.0405 -5.95 1.1e-08 ***
cityGreenville -0.1195 0.0362 -3.30 0.0011 **
cityRock Hill -0.1454 0.0479 -3.04 0.0027 **
nbath -0.1073 0.0433 -2.48 0.0141 *
nbed 0.0645 0.0341 1.89 0.0595 .
log(sqft) 0.4728 0.0781 6.05 6.6e-09 ***
petsyes -0.0186 0.0303 -0.61 0.5402
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.208 on 206 degrees of freedom
Multiple R-squared: 0.414, Adjusted R-squared: 0.394
F-statistic: 20.8 on 7 and 206 DF, p-value: <2e-16Sequential SS with anova() function (seldom use)
Sequential SS report the changes in \(\operatorname{SS}_{\operatorname{Reg}}\) from adding new variables.
SC apartments
The sequential SS depend on the order in which variables are added:
Analysis of Variance Table
Response: log(price)
Df Sum Sq Mean Sq F value Pr(>F)
city 3 1.03 0.345 7.95 4.9e-05 ***
nbath 1 1.79 1.792 41.32 8.8e-10 ***
nbed 1 1.87 1.867 43.04 4.2e-10 ***
log(sqft) 1 1.60 1.602 36.94 5.8e-09 ***
pets 1 0.02 0.016 0.38 0.54
Residuals 206 8.93 0.043
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Analysis of Variance Table
Response: log(price)
Df Sum Sq Mean Sq F value Pr(>F)
nbath 1 1.31 1.309 30.18 1.2e-07 ***
city 3 1.52 0.506 11.66 4.3e-07 ***
nbed 1 1.87 1.867 43.04 4.2e-10 ***
log(sqft) 1 1.60 1.602 36.94 5.8e-09 ***
pets 1 0.02 0.016 0.38 0.54
Residuals 206 8.93 0.043
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Sequential model fits to obtain sequential SS
lm1 <- lm(log(price) ~ nbath, data = scapts)
lm2 <- lm(log(price) ~ nbath + city, data = scapts)
lm3 <- lm(log(price) ~ nbath + city + nbed, data = scapts)
lm4 <- lm(log(price) ~ nbath + city + nbed + log(sqft), data = scapts)
lm5 <- lm(log(price) ~ nbath + city + nbed + log(sqft) + pets, data = scapts)
SSR1 <- SST - sum(lm1$residuals^2)
SSR2 <- SST - sum(lm2$residuals^2)
SSR3 <- SST - sum(lm3$residuals^2)
SSR4 <- SST - sum(lm4$residuals^2)
SSR5 <- SST - sum(lm5$residuals^2)
seqSS <- c(SSR1,SSR2 - SSR1,SSR3 - SSR2, SSR4 - SSR3, SSR5 - SSR4)
names(seqSS) <- c("city","nbath","nbed","log(sqft)","pets")
round(seqSS,3) city nbath nbed log(sqft) pets
1.309 1.517 1.867 1.602 0.016 Checking model assumptions
Validity of the foregoing analyses depends on these assumptions:
The responses are normally distributed around the regression line (Check QQ plot of residuals). If \(n\) is large this only matters for prediction intervals.
The response has the same variance for all covariate values (Check residuals vs fitted values plot).
The covariates and the response are linearly related (Check residuals vs fitted values plot).
The response values are independent of each other (No way to check; must trust experimental design).
Generating diagnostic plots from lm() with plot()
Leverage and Cook’s distance in MLR
The leverage of a point \((\mathbf{x}_i,Y_i)\) among \((\mathbf{x}_1,Y_1),\dots,(\mathbf{x}_n,Y_n)\) is \[ \text{lev}_i = \text{ entry $i$ on the diagonal of the matrix } \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}. \] Leverage only shows outlying-ness in the covariate space.
Cook’s Distance measures how much each data point changes the fit: \[ D_i = \frac{1}{(p+1) \hat \sigma^2}\sum_{j = 1 }^n(\hat Y_j - \hat Y_{j(i)})^2 = \frac{\hat e_i^2}{(p+1)\hat \sigma^2}\frac{\text{lev}_i}{(1 - \text{lev}_i)^2} \quad \text{ for } i = 1,\dots,n, \] where \(\hat Y_{j(i)}\) is the \(j\)th fitted value from the model fitted without obs \(i\).
Interactions
When the effect of one predictor depends on the value of another it is called an interaction.
E.g., does the effect of square feet differ between Charleston and Rock Hill, Charleston and Columbia?
We can include interactions by including in \(\mathbf{X}\) products of pairs of predictors.
Most often we consider interactions between dummy variables and numeric variables.
Such interactions act as “slope modifiers”, allowing different slope coefficients for different subsets of the data.
SC apartments
Add an interaction between square feet and each city indicator.
Then the model becomes:
\[\begin{align*} \log(\texttt{price}) = \beta_0 &+ \beta_1 \cdot \texttt{Columbia} + \beta_2 \cdot \texttt{Greenville} + \beta_3 \cdot \texttt{Rock Hill} \\ &+ \beta_4 \cdot \texttt{nbath} + \beta_5 \cdot \texttt{nbed} + \beta_6 \log(\texttt{sqft}) + \beta_7 \cdot \texttt{pets} \\ &+ \beta_8 \cdot \texttt{Columbia} \times \log(\texttt{sqft}) \\ &+ \beta_9 \cdot \texttt{Greenville} \times \log(\texttt{sqft}) \\ &+ \beta_{10} \cdot \texttt{Rock Hill} \times \log(\texttt{sqft}) + \varepsilon \end{align*}\]
SC apartments
This code shows the construction of the design matrix \(\mathbf{X}\) with the interaction terms.
Y <- log(scapts$price)
X <- cbind("Intercept" = rep(1,n),
"Columbia" = ifelse(scapts$city == "Columbia",1,0),
"Greenville" = ifelse(scapts$city == "Greenville",1,0),
"Rock Hill" = ifelse(scapts$city == "Rock Hill",1,0),
"nbath" = scapts$nbath,
"nbed" = scapts$nbed,
"log_sqft" = log(scapts$sqft),
"pets" = ifelse(scapts$pets == "yes",1,0),
"log_sqft:Columbia" = log(scapts$sqft)*ifelse(scapts$city == "Columbia",1,0),
"log_sqft:Greenville" = log(scapts$sqft)*ifelse(scapts$city == "Greenville",1,0),
"log_sqrt:Rock Hill" = log(scapts$sqft)*ifelse(scapts$city == "Rock Hill",1,0))SC apartments
SC apartments
We can represent the fitted model as:
\[\begin{align*} \widehat{\log(\texttt{price})} = 2.886 &+ (1.525) \cdot \texttt{Columbia} + (0.873) \cdot \texttt{Greenville} \\ &+ (0.335) \cdot \texttt{Rock Hill} + (-0.135) \cdot \texttt{nbath} \\ &+ (0.08) \cdot \texttt{nbed} + (0.629) \log(\texttt{sqft}) + (-0.008) \cdot \texttt{pets} \\ &+ (-0.259) \cdot \texttt{Columbia} \times \log(\texttt{sqft}) \\ &+ (-0.146) \cdot \texttt{Greenville} \times \log(\texttt{sqft}) \\ &+ (-0.072) \cdot \texttt{Rock Hill} \times \log(\texttt{sqft}). \end{align*}\]
This gives a different fitted model for each city
SC apartments
- Charleston: \[\begin{align*} \widehat{\log(\texttt{price})} = 2.886 &+ (-0.135) \cdot \texttt{nbath} + (0.08) \cdot \texttt{nbed} \\ &+ (0.629) \log(\texttt{sqft}) + (-0.008) \cdot \texttt{pets} \end{align*}\]
- Columbia: \[\begin{align*} \widehat{\log(\texttt{price})} = 4.411 &+ (-0.135) \cdot \texttt{nbath} + (0.08) \cdot \texttt{nbed} \\ &+ (0.37 ) \log(\texttt{sqft}) + (-0.008) \cdot \texttt{pets} \end{align*}\]
- Greenville: \[\begin{align*} \widehat{\log(\texttt{price})} = 3.759 &+ (-0.135) \cdot \texttt{nbath} + (0.08) \cdot \texttt{nbed} \\ &+ (0.483 ) \log(\texttt{sqft}) + (-0.008) \cdot \texttt{pets} \end{align*}\]
- Rock Hill: \[\begin{align*} \widehat{\log(\texttt{price})} = 3.221 &+ (-0.135) \cdot \texttt{nbath} + (0.08) \cdot \texttt{nbed} \\ &+ (0.556 ) \log(\texttt{sqft}) + (-0.008) \cdot \texttt{pets} \end{align*}\]
SC apartments
Include interactions in the lm() function as below:
Call:
lm(formula = log(price) ~ city + nbath + nbed + log(sqft) + pets +
log(sqft):city, data = scapts)
Residuals:
Min 1Q Median 3Q Max
-0.582 -0.133 -0.004 0.093 0.684
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.88590 0.68527 4.21 3.8e-05 ***
cityColumbia 1.52500 0.75905 2.01 0.0459 *
cityGreenville 0.87340 0.88640 0.99 0.3256
cityRock Hill 0.33480 1.36863 0.24 0.8070
nbath -0.13511 0.04508 -3.00 0.0031 **
nbed 0.07974 0.03470 2.30 0.0226 *
log(sqft) 0.62871 0.10617 5.92 1.3e-08 ***
petsyes -0.00804 0.03089 -0.26 0.7949
cityColumbia:log(sqft) -0.25861 0.11112 -2.33 0.0209 *
cityGreenville:log(sqft) -0.14616 0.13059 -1.12 0.2644
cityRock Hill:log(sqft) -0.07234 0.19835 -0.36 0.7157
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.207 on 203 degrees of freedom
Multiple R-squared: 0.43, Adjusted R-squared: 0.402
F-statistic: 15.3 on 10 and 203 DF, p-value: <2e-16Check for significance of any interaction effects and give interpretations.
References
“Apartment for Rent Classified.” 2019. UCI Machine Learning Repository.