Overlearning, Training, Testing, and Validation
Academic Year 1447
Term 1
Dr. Jomana Bashatah
Slides adapted from Casten Lange
You will learn about:
Overlearning in detail.
Circumstances that make overlearning more likely to occur.
Consequences of overlearning when predicting new data.
Hyper-parameter tuning to avoid overlearning.
If a model performs well when approximating the training data but does not perform well when it faces new data to predict outcomes.
Overlearning is one of the most pressing and still not fully solved problems in machine learning.
If the training dataset does not have a sufficient number of observations.
If the model considers many variables and thus contains many parameters to calibrate.
If the underlying machine learning model is highly non-linear.
In what follows we use the Kings County Real Estate dataset.
We want to demonstrate overlearning. Therefore, we ceate conditions that likely trigger overlearning. Consequently, we work only with a very small training dataset (20 observations=0.1% of total observations. All other observations become testing data:
There seems to be a non-liner trend:
Price Sqft
1 221900 1180
2 538000 2570
3 180000 770
4 604000 1960
5 510000 1680
6 1230000 5420Regular univariate prediction equation: \[ \widehat{Price}=\beta_1 Sqft+\beta_2 \]
Polynomial univariate prediction equation (degree 5):
\[\begin{eqnarray*} \widehat{Price}&=&\beta_1 Sqft+\beta_2 Sqft^2+\beta_3 Sqft^3 \\ && +\beta_4 Sqft^4+\beta_5 Sqft^5+\beta_6 \end{eqnarray*}\]Polynomial univariate prediction equation (degree 5):
\[\begin{eqnarray*} \widehat{Price}&=&\beta_1 Sqft+\beta_2 Sqft^2+\beta_3 Sqft^3\\ && +\beta_4 Sqft^4+\beta_5 Sqft^5+\beta_6 \end{eqnarray*}\]We create \(Sqft^2\), \(Sqft^3\), \(Sqft^4\), and \(Sqft^5\) as new variables in the data and treat them as they were separate variables in a multivariate regression.
Consequently we can OLS to find the optimal \(\beta s\).
Price Sqft Sqft2 Sqft3 Sqft4 Sqft5
1 221900 1180 1392400 1643032000 1.938778e+12 2.287758e+15
2 538000 2570 6604900 16974593000 4.362470e+13 1.121155e+17
3 180000 770 592900 456533000 3.515304e+11 2.706784e+14
4 604000 1960 3841600 7529536000 1.475789e+13 2.892547e+16
5 510000 1680 2822400 4741632000 7.965942e+12 1.338278e+16
6 1230000 5420 29376400 159220088000 8.629729e+14 4.677313e+18
\[\widehat{Price}=\beta_1 Sqft+\beta_2 Sqft^2+\beta_3 Sqft^3 + +\beta_4 Sqft^4 +\beta_5 Sqft^5 +\beta_{6}\]
\[\widehat{Price}=\beta_1 Sqft+\beta_2 Sqft^2+\beta_3 Sqft^3 + \cdots +\beta_{10} Sqft^{10}+\beta_{11}\]
# A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 rmse standard 170208.
2 rsq standard 0.557
3 mae standard 139352. # A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 rmse standard 136432.
2 rsq standard 0.715
3 mae standard 104047. # A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 rmse standard 123463.
2 rsq standard 0.767
3 mae standard 87585. Overfitting occurs when a highly non-linear (flexible) model adjusts almost perfectly to a relatively small number of training observations, but the adjustment is so specific that the model fails when predicting the testing data.
If we do not have enough data, a polynomial regression with a high degree might lead to overlearning
What is the right degree?
We could try different degrees (e.g., 2, 3, 4, … 10) and see which model performs best.
Which data are we using to measure performance? Training data (overlearning) and testing data (cannot be used for model optimization) are out.
We could split off data from the training dataset (validation data). These validation data are not used to calculate the Bs. Instead, they are used to find the best setting for the degree of polynomial regression (aka hyper-parameter of polynomial regression).
Hyper-Parameters are parameters other than the \(\beta\) parameters, because they can not be optimized by the optimizer.
Hyper-Parameters are like settings for a machine learning model such as the number of polynomials (e.g., \(Sqft^N\)) to be considered for polynomial regression. Another example are the number of \(k\) Nearest Neighbors.
Hyper parameters often make a model more or less complex and thus influence the quality of predicting but also the chance of overlearning.
Hyper-parameters cannot be determined with an Optimizer based on training data like a model???s beta parameters. Therefore,we must utilize a trial and-error process to find suitable hyper-parameters.
We cannot utilize the testing data for hyper-parameter tuning because this can extend overfitting into the testing data
library(tidymodels); library(rio); library(janitor)
DataHousing=
import("https://raw.githubusercontent.com/jomanab-cloud/ISE351-25-Presentations/80f7609c6bcb4887517dcaff62908459cee33e33/HousingData.csv") |>
clean_names("upper_camel") |>
select(Price, Sqft=SqftLiving)
set.seed(987)
Split80=DataHousing |>
initial_split(prop=0.8, strata=Price, breaks=5)
DataTrain=training(Split80)
DataTest=testing(Split80)Building the workflow follows the same steps as before. We first create the recipe then the model design, then we combine both in a workflow
Note:
1. In step_poly() where the argument degree= determines the highest power of sqft in the prediction equation, we do not assign a number for the degree.
- This makes sense because the aim of hyper-parameter tuning is to find this number (the degree of the Polynomial Model).
- Since the argument degree= needs to be determined, we use tune() as a placeholder
2. The workflow does not contain a fit() command to fit the model parameters to the training data.
- This also makes sense because the degree for the polynomial function is not determined, fitting the model is not possible.
DataValidate includes the complete training dataset, but observations are internally earmarked with analysis to indicate that an observation will be used for training, and with assessment to indicate that the observation will be used as validation data to assess hyper-parameter performance.
Using a validation dataset is appropriate for lareg datasets. For smaller datasets it comes with 2 disadvantages
For each hyper-parameter setting:
1. Splits off validation data from training data (e.g. last quarter)
2. Runs the model and calculates metrics based on validation data.
3. Splits off validation data from training data (next quarter)
4. Repeats steps 2 – 3 four times.
We end up with four results for each hyper-parameter setting. We calculate the average of the four results as an result for that specific hyper parameter.
The advantage of Cross-Validation is that different sets of observations are used for assessment (the mean prediction error is used to assess overall performance) and all observations of the training data at some stage of model assessment are used for validation. Therefore, the risk of an unusual assessment dataset is mitigated.
The disadvantage of Cross-Validation is that each hyper-parameter setup needs to be trained and assessed separately for each of the folds. Computation time increases exponentially with the number of hyper-parameters tried out and proportionally with the number of folds used.
The 10 general steps are:
1. Generating training and testing data with initial_split(), training(), testing().
set.seed(987)
Split80=MyData |>
initial_split(prop=0.8, strata=OUTCOME_VARIABLE, breaks=5)
DataTrain=training(Split80)
DataTest=testing(Split80)2. Create recipe to determine predictor and outcome variables. Optionally add one or more step_X() commands.
Recipe=recipe(<OUTCOME VARIABLE>~<PREDICTOR VARIABLE(S)>,
data=DataTrain) |>
step_<NAME OFSTEP>(<ARGUMENT(S) OFSTEP>)3. Create model design and mark parameters to be tune() ed. without fit()
ModelDesign=<NAME OFML-COMMAND>(<ARGUMENT(S) OFCOMMAND>) |>
set_engine("<PACKAGE NAME>") |>
set_mode("<MODE>")4. Create workflow by add_recipe() and add_model()
TuneWFModel=workflow() |>
add_recipe(Recipe) |>
add_model(ModelDesign)5. Create a hyper-parameter grid containing the hyper-parameter combinations to be validated.
ParGrid=data.frame(<HYPER-PAR1>=c(<LIST OFVALUES>),
<HYPER-PAR2>=c(<LIST OFVALUES>), <ETC>)ParGrid=data.frame(neighbors=c(1,3,6))6. Create cross validation datasets (aka resamples) containing the folds (use commands vfold()).
FoldsForTuning=vfold_cv(DataTrain, v=10, strata=<OUTCOME VARIABLE>)7. Tune the machine learning model with tune_grid() and track specific metrics defined by metric_set(). Runs all hyper-parameter combinations for all folds.
TuneResults=tune_grid(TuneWFModel, resamples=FoldsForTuning,
grid=ParGrid, metrics=metric_set(<LIST OFMETRICS>)),
control_grid(verbose=TRUE))8. Extract the best hyper-parameter combination from the tuning results based on selected metrics (use select_best())
BestHyperPar=select_best(TuneResults, "<METRIC>")9. Finalize the model by training it with the full set of training data with the best hyper-parameter combination (see finalize_workflow() %>% fit()).
BestWFModel=TuneWFModel |>
finalize_workflow(BestHyperPar) |>
fit(DataTrain)10. Assessing predictive quality of the final model by using the testing dataset to predict (see augment() %>% metrics()).
DataTestWithPredBestModel=augment(BestWFModel, DataTest)
metrics(DataTestWithPredBestModel, truth=<OUTCOME VARIABLE>,
estimate=.pred)══ Workflow [trained] ══════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: linear_reg()
── Preprocessor ────────────────────────────────────────────────────────────────
1 Recipe Step
• step_poly()
── Model ───────────────────────────────────────────────────────────────────────
Call:
stats::lm(formula = ..y ~ ., data = data)
Coefficients:
(Intercept) Sqft_poly_1 Sqft_poly_2 Sqft_poly_3 Sqft_poly_4 Sqft_poly_5
-1.297e+04 6.569e+02 -4.902e-01 2.051e-04 -3.758e-08 3.184e-12
Sqft_poly_6
-9.933e-17 In Lecture 4, you developed a k-Nearest Neighbor model to predict the species of a penguin. We arbitrarily set k=4 to consider the four nearest neighbors.
Start by loading the required libraries and read the data
# A tibble: 10 × 3
Species BillLengthMm BodyMassG
<fct> <dbl> <int>
1 Adelie 39.1 3750
2 Adelie 39.5 3800
3 Adelie 40.3 3250
4 Adelie 36.7 3450
5 Adelie 39.3 3650
6 Adelie 38.9 3625
7 Adelie 39.2 4675
8 Adelie 34.1 3475
9 Adelie 42 4250
10 Adelie 37.8 3300As before, we will use the penguin dataset and split the data into training (DataTrain) and testing (DataTest).
We will use the same seed (876) so the random split will be identical to the one we had with the k=4 model.
# A tibble: 6 × 3
Species BillLengthMm BodyMassG
<fct> <dbl> <int>
1 Adelie 40.3 3250
2 Adelie 36.7 3450
3 Adelie 39.3 3650
4 Adelie 38.9 3625
5 Adelie 39.2 4675
6 Adelie 42 4250This is the same recipe as before. We will be dropping NAs and normalizing all predictor variables.
Use the appropriate ML model “knn”. Remember that we will not specify the number of neighbors, but we will use tune() placeholder
K-Nearest Neighbor Model Specification (classification)
Main Arguments:
neighbors = tune()
weight_func = rectangular
Computational engine: kknn ══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: nearest_neighbor()
── Preprocessor ────────────────────────────────────────────────────────────────
2 Recipe Steps
• step_naomit()
• step_normalize()
── Model ───────────────────────────────────────────────────────────────────────
K-Nearest Neighbor Model Specification (classification)
Main Arguments:
neighbors = tune()
weight_func = rectangular
Computational engine: kknn Later, when tuning is executed in Step 7, values reaching from 1 ??? 15 for the hyper-parameter neighbors shall be tried out.
You need to provide these values in a data frame column that is named the same as the hyper-parameter.
neighbors
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 10
11 11
12 12
13 13
14 14
15 15The values you have created above for k (hyper-parameter neighbors) will be evaluated later using five folds (resamples). Each fold contains the complete training data, but different sections are used for training and assessment in each fold.
# 5-fold cross-validation using stratification
# A tibble: 5 × 2
splits id
<list> <chr>
1 <split [189/49]> Fold1
2 <split [190/48]> Fold2
3 <split [191/47]> Fold3
4 <split [191/47]> Fold4
5 <split [191/47]> Fold5Now it is time to run the tuning procedure using the tune_grid() command. Be patient because it will take some time to fully execute. Since we have to try out 15 parameters and use five folds for each model, the tune_grid() command has to fit 75 models (15x 5=75)
All assessment results for the specified metrics are stored in the tuning object TuneResults. Use the select_best() command to extract the best hyper-parameter (value for neighbors) for the metric you specified:
# A tibble: 1 × 2
neighbors .config
<int> <chr>
1 6 Preprocessor1_Model06The finalize_workflow() command will use the value from BestHyperPar to substitute the tune() placeholder in the R object TuneWFModel (created in Steps 2 ??? 4). The hyper-parameter is now set to neighbors=6 completing the workflow.
use the command fit(DataTrain) to calibrate the workflow to the training data and save the result into WFModelBest.
══ Workflow [trained] ══════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: nearest_neighbor()
── Preprocessor ────────────────────────────────────────────────────────────────
2 Recipe Steps
• step_naomit()
• step_normalize()
── Model ───────────────────────────────────────────────────────────────────────
Call:
kknn::train.kknn(formula = ..y ~ ., data = data, ks = min_rows(6L, data, 5), kernel = ~"rectangular")
Type of response variable: nominal
Minimal misclassification: 0.06302521
Best kernel: rectangular
Best k: 6Since WFModelBest is a fitted model, you can use it for predictions. In this last step, you will use the augment() command to predict Species. The augment() command will then add the prediction results as column .pred to the testing data.
Use the conf_mat() command to compare the predictions in column .pred to the true values to create a confusion matrix.
Truth
Prediction Adelie Chinstrap Gentoo
Adelie 41 1 1
Chinstrap 2 19 0
Gentoo 3 1 36Dr. Jomana Bashatah | ISE 423