STAT 516 Lec 12
Logistic regression
2026-10-26
Programming task data from Kutner et al. (2005)
Twenty-five people succeeded or failed at a programming task.
Months of programming experience was recorded for each person.
Can we predict probability of success based on experience?
Programming task
Logistic regression model
Assume \[ Y_i \sim \text{Bernoulli}(\pi_i), \quad \log\left(\frac{\pi_i}{1-\pi_i}\right) = \beta_0 + \beta_1 x_i, \] for \(i =1,\dots,n\), where
- \(Y_i\) is the response for observation \(i\).
- \(x_i\) is the value of a predictor/covariate/explanatory variable for obs \(i\).
- \(\pi_i\) is the probability of “success” for observation \(i\).
- \(\beta_0\) and \(\beta_1\) are slope and intercept parameters.
- \(\pi_i/ (1-\pi_i)\) is the odds of “success” for obs \(i\).
- \(\log(\pi_i/(1-\pi_i))\) is the log-odds for obs \(i\).
Logistic regression assumes the log-odds are linear in the predictor.
Odds
Let \(\pi\) be the probability of success.
Then \(\pi/(1-\pi)\) is called the odds in favor of success.
- If \(\pi = 1/2\) then \(\pi/(1-\pi) = 1\). “One-to-one” odds of success.
- If \(\pi = 2/3\) then \(\pi/(1-\pi) = 2\). Success 2x more likely than failure.
- If \(\pi = 1/4\) then \(\pi/(1-\pi) = 1/3\). Failure 3x more likely than success.
The logit and logistic transformations
- The transformation \(y = \frac{e^x}{1 +e^x}\) is called the logistic transformation.
- Its inverse \(x = \log(\frac{y}{1-y})\) is called the logit transformation.
- We have \[ \log\left(\frac{\pi_i}{1-\pi_i}\right) = \beta_0 + \beta_1 x_i \iff \pi_i = \frac{e^{\beta_0 + \beta_1 x_i}}{1 + e^{\beta_0 + \beta_1 x_i}} \]
The logit and logistic transformations
Goals in logistic regression
- Estimate \(\beta_0\) and \(\beta_1\).
- Obtain fitted probabilities \(\hat \pi_1,\dots,\hat \pi_n\).
- Build CI for \(\beta_1\) and test \(H_0\): \(\beta_1 = 0\).
- Give interpretations of the estimated regression coefficients.
- Check goodness of fit of the logistic regression model.
- Add additional covariates
Maximum likelihood estimation in logistic regression
- We do not use least-squares to estimate \(\beta_0\) and \(\beta_1\).
- Instead we use maximum likelihood estimators (MLEs).
- The MLEs are the parameter values giving the observed data the highest possible probability.
- Intercept \(b_0\) and slope \(b_1\) give to the observed data the probability \[ \mathcal{L}_n(b_0,b_1) = \prod_{i=1}^n [\pi_i(b_0,b_1)]^{Y_i}[1-\pi_i(b_0,b_1)]^{1-Y_i} \] with \(\displaystyle\pi_i(b_0,b_1) = \frac{e^{b_0 + b_1 x_i}}{1 + e^{b_0 + b_1 x_i}}\) for \(i = 1,\dots,n\).
- The MLEs \(\hat \beta_0\), \(\hat \beta_1\) are the values of \(b_0\), \(b_1\) that maximize \(\mathcal{L}_n(b_0,b_1)\).
- \(\mathcal{L}_n(b_0,b_1)\) is called the likelihood function.
Computing the MLEs in logistic regression
- There is no “closed-form” expression for \(\hat \beta_0\) and \(\hat \beta_1\).
- One must find their values numerically, that is with an algorithm.
- More convenient to work with \(\log \mathcal{L}_n(b_0,b_1)\), which is given by \[ \ell_n(b_0,b_1) = \sum_{i=1}^n[Y_i(b_0 + b_1 x_i) - \log(1 + e^{b_0 + b_1 x_i})]. \]
- Newton’s method is one way to find the maximizers of \(\ell_n(b_0,b_1)\).
Programming task
Newton-Raphson algorithm for computing \(\hat \beta_0\) and \(\hat \beta_1\).
Generalized linear models
- The logistic regression model is in a class of models called GLMs.
- GLM stands for generalized linear model.
- Poisson regression, binomial response regression, i.a. are GLMs too.
- Use
glm()function in R to obtain \(\hat \beta_0\) and \(\hat \beta_1\).
Programming task
Use glm() function with the option family = "binomial".
Call:
glm(formula = success ~ experience, family = "binomial")
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -3.060 1.259 -2.43 0.015 *
experience 0.162 0.065 2.49 0.013 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 34.296 on 24 degrees of freedom
Residual deviance: 25.425 on 23 degrees of freedom
AIC: 29.42
Number of Fisher Scoring iterations: 4Programming task
Fitted probabilities
- Define the fitted probabilities as \[ \hat \pi_i = \dfrac{e^{\hat \beta_0 + \hat \beta_1 x_i}}{1 + e^{\hat \beta_0 + \hat\beta_1 x_i}} \quad \text{ for } \quad i = 1,\dots,n. \]
- For any value \(x_{\operatorname{new}}\), we estimate the probability of “success” as \[ \hat\pi_{\operatorname{new}} =\dfrac{e^{\hat \beta_0 + \hat \beta_1 x_{\operatorname{new}}}}{1 + e^{\hat \beta_0 + \hat\beta_1 x_{\operatorname{new}}}}. \]
Programming task
Asymptotic distribution of slope estimator and CI
- For large enough \(n\), \(\hat \beta_1\) is approximately Normal, such that \[ \frac{\hat \beta_1 - \beta_1}{\widehat{\operatorname{se}}\{\hat\beta_1\}} \overset{\operatorname{approx}}{\sim} \text{Normal}\left(0,1\right), \] where, setting \(\hat w_i = \hat \pi_i(1-\hat\pi_i)\) for \(i = 1,\dots,n\), we may write \[ \widehat{\operatorname{se}}\{\hat\beta_1\} = \textstyle \left[\sum_{i=1}^n \hat w_ix_i^2 - (\sum_{i=1}^n \hat w_i)^{-1}(\sum_{i=1}^n \hat w_i x_i)^2\right]^{-\frac{1}{2}}. \]
- We can make an approximate \((1-\alpha)100\%\) CI for \(\beta_1\) as \[ \hat \beta_1 \pm z_{\alpha/2} ~ \widehat{\operatorname{se}}\{\hat\beta_1\}. \]
Programming task
experience experience
0.0341 0.2888 2.5 % 97.5 %
(Intercept) -5.5280 -0.591
experience 0.0341 0.289Testing whether the slope coefficient is zero
To test \(H_0\): \(\beta_1 = 0\) versus \(H_1\): \(\beta_1 \neq 0\), do:
Compute \(Z_{\operatorname{test}} = \dfrac{\hat \beta_1}{\widehat{\operatorname{se}}\{\hat\beta_1\}}\).
Reject \(H_0\) at \(\alpha\) if \(|Z_{\operatorname{test}}| > z_{\alpha/2}\).
Obtain p value as \(2(1- P(Z > |Z_{\operatorname{test}}|))\), \(Z \sim \text{Normal}(0,1)\).
The summary() function on the glm() output prints this p value.
Odds ratios
Let \(\pi_0\) and \(\pi_1\) be success probs under an initial and an altered condition, respectively.
Then we call the ratio \[ \displaystyle \frac{\pi_1/(1-\pi_1)}{\pi_0/(1-\pi_0)} \] the odds ratio associated with the change from the initial to the altered condition.
The odds ratio is the factor by which the odds are multiplied when the initial condition is changed to the altered condition.
Interpreting the logistic regression parameters
Let \(\pi_0\) and \(\pi_1\) be the “success” probabilities at \(x_0\) and \(x_0 + 1\).
Then we have the two equations
- \(\log\left(\frac{\pi_0}{1-\pi_0}\right) = \beta_0 + \beta_1 x_0\)
- \(\log\left(\frac{\pi_1}{1-\pi_1}\right) = \beta_0 + \beta_1 (x_0 + 1)\)
Subtracting the first equation from the second gives \[ \beta_1 = \log\left(\frac{\pi_1}{1-\pi_1}\right) - \log\left(\frac{\pi_0}{1-\pi_0}\right) = \log\left(\frac{\pi_1/(1-\pi_1)}{\pi_0/(1-\pi_0)}\right). \]
The quantity \(\dfrac{\pi_1/(1-\pi_1)}{\pi_0/(1-\pi_0)}\) is called an odds ratio.
So \(\beta_1\) is log of the odds ratio associated with a unit increase in \(x\).
Odds ratio from a unit increase in x
From the previous slide, we have \[ e^{\beta_1} = \frac{\pi_1/(1-\pi_1)}{\pi_0/(1-\pi_0)}. \]
Can build a CI for \(e^{\beta_1}\) by exponentiating the CI for \(\beta_1\).
Gives CI for \(e^{\beta_1}\) as \([e^{\hat \beta_1 - z_{\alpha/2}\widehat{\operatorname{se}}\{\hat\beta_1\}},e^{\hat \beta_1 + z_{\alpha/2}\widehat{\operatorname{se}}\{\hat\beta_1\}}]\).
A unit increase in \(x\) multiplies the odds of success by the factor \(e^{\beta_1}\).
What if the CI for \(e^{\beta_1}\) contains \(1\)?
Programming task
Each additional month of experience increases the odds of completing the programming task by a factor of 1.035 to 1.335, with \(95\%\) confidence.
Residuals for logistic regression
- Ordinary residuals \(Y_i - \hat \pi_i\) cannot be Normally distributed.
- In GLMs, one looks at special residuals called deviance residuals.
- In logistic regression, the deviance residuals are defined as \[ \hat d_i = \operatorname{sign}(Y_i - \hat \pi_i)\sqrt{-2[Y_i\log \hat \pi_i + (1- Y_i)\log(1 - \hat \pi_i)]} \] for \(i=1,\dots,n\).
- These are not Normal either, but are useful for assessing model fit.
Programming task
Checking model fit with a simulated envelope
The simulated envelope method is described in Kutner et al. (2005):
- Fit the logistic regression model and obtain \(\hat \pi_1,\dots,\hat \pi_n\).
- Obtain the deviance residuals; sort them as \(\hat d_{(1)} < \hat d_{(2)} < \dots <\hat d_{(n)}\).
- Generate many new data sets \(Y_i^* \sim \text{Bernoulli}(\hat \pi_i)\), \(i=1,\dots,n\).
- For each new data set, obtain sorted \(\hat d_{(1)}^* < \hat d_{(2)}^* < \dots <\hat d_{(n)}^*\).
- Plot \(\hat d_{(i)}\) as well as the 0.025 and 0.975 quantiles and the mean of the \(\hat d_{(i)}^*\) \(\forall ~i\) (it doesn’t matter what is chosen as the \(x\)-axis).
- The quantiles of the \(\hat d_{(i)}^*\) make a band. If the model fits, then the \(\hat d_{(i)}\) should lie within the band and close to the mean.
Asks: If the model is correct, how would the deviance residuals behave?
Programming task
Programming task
German credit score data from Hofmann (1994)
Response is credit rating (good/bad), various predictors.
[1] "checking_status" "duration" "credit_history"
[4] "purpose" "credit_amount" "savings_status"
[7] "employment" "installment_commitment" "personal_status"
[10] "other_parties" "residence_since" "property_magnitude"
[13] "age" "other_payment_plans" "housing"
[16] "existing_credits" "job" "num_dependents"
[19] "own_telephone" "foreign_worker" "class" German credit score
Logistic multiple regression model
Observe \((\mathbf{x}_1,Y_1),\dots,(\mathbf{x}_n,Y_n)\), \(\mathbf{x}_i = (x_{i1},\dots,x_{ip})^T\) and assume \[ Y_i \sim \text{Bernoulli}(\pi_i), \quad \log\left(\frac{\pi_i}{1-\pi_i}\right) = \beta_0 + \beta_1 x_{i1} + \dots + \beta_p x_{ip}, \] for \(i =1,\dots,n\), where
- \(Y_i\) is the response for observation \(i\).
- \(x_{i1},\dots,x_{ip}\) are the values of the predictors for obs \(i\).
- \(\pi_i\) is the probability of “success” for observation \(i\).
- \(\beta_0\) is an intercept and \(\beta_1,\dots,\beta_p\) are slope parameters.
- \(\pi_i/ (1-\pi_i)\) is the odds of “success” for obs \(i\).
- \(\log(\pi_i/(1-\pi_i))\) is the log-odds for obs \(i\).
So we assume the log-odds are a linear function of the predictors.
Interpreting multiple logistic regression parameters
- Let \(\pi_{0j}\) and \(\pi_{1j}\) be the “success” probabilities at \(x_{0j}\) and \(x_{0j}+1\) but with \(x_{0k}\) fixed for all \(k \neq j\).
- Then we have the two equations
- \(\log\left(\frac{\pi_{0j}}{1-\pi_{0j}}\right) = \beta_0 + \sum_{k \neq j}\beta_k x_{0k} + \beta_j x_{0j}\)
- \(\log\left(\frac{\pi_{1j}}{1-\pi_{1j}}\right) = \beta_0 + \sum_{k\neq j}\beta_k x_{0k} + \beta_j(x_{0j} + 1)\)
- Subtracting the first equation from the second gives \[ \beta_j = \log\left(\frac{\pi_{1j}/(1-\pi_{1j})}{\pi_{0j}/(1-\pi_{0j})}\right) \quad \text{ and } \quad e^{\beta_j} = \frac{\pi_{1j}/(1-\pi_{1j})}{\pi_{0j}/(1-\pi_{0j})}. \]
- So \(\beta_1\) is log of the odds ratio associated with a unit increase in \(x_j\) with all other predictors held fixed.
German credit score
Call:
glm(formula = class ~ ., family = "binomial", data = credg)
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 1.50e+00 1.25e+00 1.21 0.22780
checking_status>=200 9.66e-01 3.69e-01 2.62 0.00890 **
checking_status0<=X<200 3.75e-01 2.18e-01 1.72 0.08540 .
checking_statusno checking 1.71e+00 2.32e-01 7.37 1.7e-13 ***
duration -2.79e-02 9.30e-03 -3.00 0.00272 **
credit_historycritical/other existing credit 1.58e+00 4.38e-01 3.60 0.00031 ***
credit_historydelayed previously 9.97e-01 4.70e-01 2.12 0.03410 *
credit_historyexisting paid 7.29e-01 3.85e-01 1.89 0.05824 .
credit_historyno credits/all paid 1.43e-01 5.49e-01 0.26 0.79392
purposedomestic appliance -2.17e-01 8.04e-01 -0.27 0.78698
purposeeducation -7.76e-01 4.66e-01 -1.67 0.09572 .
purposefurniture/equipment 5.15e-02 3.54e-01 0.15 0.88439
purposenew car -7.40e-01 3.34e-01 -2.22 0.02667 *
purposeother 7.49e-01 8.00e-01 0.94 0.34920
purposeradio/tv 1.51e-01 3.37e-01 0.45 0.65300
purposerepairs -5.24e-01 5.93e-01 -0.88 0.37743
purposeretraining 1.32e+00 1.23e+00 1.07 0.28462
purposeused car 9.26e-01 4.41e-01 2.10 0.03564 *
credit_amount -1.28e-04 4.44e-05 -2.89 0.00389 **
savings_status>=1000 1.34e+00 5.25e-01 2.55 0.01073 *
savings_status100<=X<500 3.58e-01 2.86e-01 1.25 0.21113
savings_status500<=X<1000 3.76e-01 4.01e-01 0.94 0.34848
savings_statusno known savings 9.47e-01 2.62e-01 3.61 0.00031 ***
employment>=7 2.10e-01 2.95e-01 0.71 0.47672
employment1<=X<4 1.16e-01 2.42e-01 0.48 0.63242
employment4<=X<7 7.64e-01 3.05e-01 2.50 0.01227 *
employmentunemployed -6.69e-02 4.27e-01 -0.16 0.87548
installment_commitment -3.30e-01 8.83e-02 -3.74 0.00018 ***
personal_statusmale div/sep -2.75e-01 3.87e-01 -0.71 0.47604
personal_statusmale mar/wid 9.16e-02 3.12e-01 0.29 0.76891
personal_statusmale single 5.41e-01 2.10e-01 2.57 0.01011 *
other_partiesguarantor 1.41e+00 5.69e-01 2.49 0.01283 *
other_partiesnone 4.36e-01 4.10e-01 1.06 0.28770
residence_since -4.78e-03 8.64e-02 -0.06 0.95592
property_magnitudelife insurance -8.69e-02 2.31e-01 -0.38 0.70711
property_magnitudeno known property -5.36e-01 4.02e-01 -1.33 0.18221
property_magnitudereal estate 1.95e-01 2.36e-01 0.82 0.40974
age 1.45e-02 9.22e-03 1.58 0.11498
other_payment_plansnone 6.46e-01 2.39e-01 2.70 0.00687 **
other_payment_plansstores 1.23e-01 4.12e-01 0.30 0.76488
housingown -2.40e-01 4.50e-01 -0.53 0.59369
housingrent -6.84e-01 4.77e-01 -1.43 0.15166
existing_credits -2.72e-01 1.90e-01 -1.44 0.15111
jobskilled -7.52e-02 2.85e-01 -0.26 0.79142
jobunemp/unskilled non res 4.79e-01 6.62e-01 0.72 0.46909
jobunskilled resident -5.67e-02 3.50e-01 -0.16 0.87145
num_dependents -2.65e-01 2.49e-01 -1.06 0.28825
own_telephoneyes 3.00e-01 2.01e-01 1.49 0.13606
foreign_workeryes -1.39e+00 6.26e-01 -2.22 0.02609 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 1221.73 on 999 degrees of freedom
Residual deviance: 895.82 on 951 degrees of freedom
AIC: 993.8
Number of Fisher Scoring iterations: 5German credit score
Note that glm() estimates three coefficients for checking_status.
Numeric predictors to encode the levels of the categorical predictor: \[\begin{align*} x_{i1} &= \left\{\begin{array}{ll} 1& 200 \leq \texttt{checking}\\ 0& \text{otherwise}\end{array}\right. \\ x_{i2} &= \left\{\begin{array}{ll} 1& 0 \leq \texttt{checking} <200 \\ 0& \text{otherwise}\end{array}\right.\\ x_{i3} &= \left\{\begin{array}{ll} 1& \texttt{no checking}\\ 0& \text{otherwise}\end{array}\right. \end{align*}\]
Likewise for other categorical predictors.
Deviances replace error sums of squares in GLMs
- The deviance is the sum of squared deviance residuals \(\sum_{i=1}^n \hat d_i^2\) .
- In logistic regression the deviance can be computed as \[ \operatorname{Dev} = -2\sum_{i=1}^n[Y_i \log \hat \pi_i + (1 - Y_i) \log(1 - \hat \pi_i)]. \]
- Full-reduced model test: Reject \(H_0\): \(\beta_j = 0\) for all \(j \in D\) if \[ \operatorname{Dev}(\operatorname{Reduced}) - \operatorname{Dev}(\operatorname{Full}) > \chi_{s,\alpha}^2, \] where \(s\) is the number of predictors in \(D\) (need large \(n\)).
German credit score
Test whether any level of checking status is important to the credit score.
credg_red <- credg[,-1] # remove checking_status column
glm_full <- glm(class ~ ., family = "binomial", data = credg)
glm_red <- glm(class ~ ., family = "binomial", data = credg_red)
p <- length(coef(glm_full)) - 1
s <- nlevels(credg$checking_status) - 1
1 - pchisq(glm_red$deviance - glm_full$deviance,s)[1] 2.73e-14Variable selection in logistic regression
- We may want to discard some of our predictors.
- One way is to add/remove variables stepwise according to AIC.
- Can do this just as we did in multiple linear regression.
- Be cautious about making inferences after selecting a model.
German credit score
Call:
glm(formula = class ~ checking_status + duration + credit_history +
purpose + credit_amount + savings_status + installment_commitment +
personal_status + other_parties + age + other_payment_plans +
housing + own_telephone + foreign_worker, family = "binomial",
data = credg)
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 4.84e-01 1.02e+00 0.48 0.63436
checking_status>=200 1.02e+00 3.63e-01 2.82 0.00474 **
checking_status0<=X<200 3.90e-01 2.12e-01 1.84 0.06593 .
checking_statusno checking 1.72e+00 2.28e-01 7.53 5e-14 ***
duration -2.57e-02 8.94e-03 -2.87 0.00407 **
credit_historycritical/other existing credit 1.37e+00 4.04e-01 3.40 0.00068 ***
credit_historydelayed previously 7.91e-01 4.49e-01 1.76 0.07799 .
credit_historyexisting paid 7.12e-01 3.79e-01 1.88 0.06030 .
credit_historyno credits/all paid -1.19e-01 5.27e-01 -0.23 0.82161
purposedomestic appliance -2.58e-01 7.76e-01 -0.33 0.74004
purposeeducation -9.26e-01 4.57e-01 -2.03 0.04263 *
purposefurniture/equipment -4.22e-02 3.42e-01 -0.12 0.90175
purposenew car -7.83e-01 3.27e-01 -2.39 0.01675 *
purposeother 6.52e-01 7.83e-01 0.83 0.40495
purposeradio/tv 1.37e-01 3.29e-01 0.42 0.67733
purposerepairs -6.40e-01 5.81e-01 -1.10 0.27036
purposeretraining 1.38e+00 1.24e+00 1.11 0.26523
purposeused car 8.25e-01 4.29e-01 1.92 0.05449 .
credit_amount -1.29e-04 4.22e-05 -3.07 0.00217 **
savings_status>=1000 1.29e+00 5.07e-01 2.54 0.01101 *
savings_status100<=X<500 3.28e-01 2.77e-01 1.19 0.23548
savings_status500<=X<1000 4.30e-01 3.93e-01 1.09 0.27390
savings_statusno known savings 9.63e-01 2.57e-01 3.75 0.00018 ***
installment_commitment -3.30e-01 8.55e-02 -3.86 0.00011 ***
personal_statusmale div/sep -2.87e-01 3.76e-01 -0.76 0.44533
personal_statusmale mar/wid 1.30e-01 3.06e-01 0.42 0.67180
personal_statusmale single 5.36e-01 1.97e-01 2.71 0.00669 **
other_partiesguarantor 1.53e+00 5.58e-01 2.74 0.00619 **
other_partiesnone 4.87e-01 4.00e-01 1.22 0.22265
age 1.31e-02 8.40e-03 1.56 0.11897
other_payment_plansnone 6.99e-01 2.35e-01 2.98 0.00292 **
other_payment_plansstores 7.86e-02 4.03e-01 0.19 0.84539
housingown 2.92e-01 2.89e-01 1.01 0.31213
housingrent -1.50e-01 3.41e-01 -0.44 0.66079
own_telephoneyes 2.79e-01 1.84e-01 1.52 0.12939
foreign_workeryes -1.38e+00 6.21e-01 -2.23 0.02593 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 1221.7 on 999 degrees of freedom
Residual deviance: 910.5 on 964 degrees of freedom
AIC: 982.5
Number of Fisher Scoring iterations: 5German credit score
Classification with the logistic regression model
Consider classifying the observations as 1 or 0 according to \(\hat \pi_i\):
Choose a threshold \(c \in [0,1]\) and make the classification \[ \hat Y_i = \left\{ \begin{array}{ll} 1,& \hat \pi_i \geq c\\ 0,& \hat \pi_i < c. \end{array}\right. \]
Can compute misclassification rate \(\#\{\hat Y_i \neq Y_i\}/n\).
Can compute observed true positive and false positive rates \[ \text{TP} = \frac{\#\{\hat Y_i = 1 \cap Y_i = 1\}}{\#\{Y_i = 1\}} \quad \text{ and } \quad \text{FP} = \frac{\#\{\hat Y_i = 1 \cap Y_i = 0\}}{\#\{Y_i = 0\}}. \]
Plotting TP against FP over all \(c \in [0,1]\) creates the receiver operating characteristic (ROC) curve.
German credit score
Compute TP and FP over a range of thresholds \(c\). Plot ROC curve.
German credit score
German credit score
Can use ROC curves to compare models.
Traning and testing data sets
Overfitting is fitting a model such that it performs well on the data used to fit it, but will perform poorly when it is given new data.
To avoid, split data into training and testing data sets \[ \{(\mathbf{x}_i^{\text{train}},Y_i^{\text{train}})\}_{i \in \mathcal{I}_{\text{train}}} \quad \text{ and } \quad \{(\mathbf{x}_i^{\text{test}},Y_i^{\text{test}})\}_{i \in \mathcal{I}_{\text{test}}}, \] where \(\mathcal{I}_{\text{train}} \cap \mathcal{I}_{\text{test}} = \emptyset\) and \(\mathcal{I}_{\text{train}} \cup \mathcal{I}_{\text{test}} = \{1,\dots,n\}\).
Fit models on training data. Compare performance on testing data.
For example, compute misclassification rate or draw ROC curve for testing data.
German credit score
set.seed(1)
n <- nrow(credg)
n_train <- ceiling(.5*n)
n_test <- n - n_train
ind_train <- sample(n,n_train) # randomly select a training set
credg_train <- credg[ind_train,]
credg_test <- credg[-ind_train,]
glm_train <- glm(class ~ ., family = "binomial", data = credg_train)
ypred_train <- glm_train$fitted.values >= 0.5
y_train <- as.numeric(credg_train$class == "good")
misclass_train <- mean(ypred_train != y_train)
misclass_train[1] 0.204[1] 0.258