library(tidyverse)
library(janitor)
library(palmerpenguins)
DataPenguins <- penguins
print(DataPenguins[1:10,])Lecture 4
Projects: 1) Identifying Penguin Species and 2) Optical Character Recognition
Academic Year 1447
Term 1
Dr. Jomana Bashatah
Slides adapted from Casten Lange
Before we Begin Let Us Do A Thought Experiment
I want to find somebody to spend a Saturday afternoon with and I am looking for somebody most similar to me (nearest neighbor) in terms of:
- Gender (coded as 0 for female, and 1 for male)
- Age (coded in years)
- Outdoor sports interest (coded from 0 (no interest) to 10 (enthusiast))
(all categories matter the same to me)
Let us do the Calculation for a Similarity Score
(average absolute differences)
Sake of argument: I am male (1), 50 years, outdoor sports score 7:
- first candidate a male student (21 years old) (score = 5)
- second candidate an athletic outdoor (score=9) women (0) 51 years old
- third candidate an athletic outdoor (score=9), man (1)
53 years
Average Absolute Difference
\[\text{Similarity Score} = \frac{\sum |x_i - y_i|}{n}\]
Where lower scores indicate higher similarity.
Absolute Differences for Candidate 1:
- Gender: |1 - 1| = 0
- Age: |50 - 21| = 29
- Sports: |7 - 5| = 2
Average Absolute Difference: \[\frac{0 + 29 + 2}{3} = \frac{31}{3} = \boxed{10.33}\]
Absolute Differences for Candidate 2:
- Gender: |1 - 0| = 1
- Age: |50 - 51| = 1
- Sports: |7 - 9| = 2
Average Absolute Difference: \[\frac{1 + 1 + 2}{3} = \frac{4}{3} = \boxed{1.33}\]
Absolute Differences for Candidate 3:
- Gender: |1 - 1| = 0
- Age: |50 - 53| = 3
- Sports: |7 - 9| = 2
Average Absolute Difference: \[\frac{0 + 3 + 2}{3} = \frac{5}{3} = \boxed{1.67}\]
Normalized Calculations (0-10 Scale)
\[\text{Normalized Diff} = \frac{|x_i - y_i|}{\text{Range}_i}\] Attribute Ranges: Gender: 0-1 (Range = 1), Age: 21-53 (Range = 32), Sports: 0-10 (Range = 10)
Normalized Differences for Candidate 1:
- Gender: \(\frac{|1-1|}{1} = \frac{0}{1} = \boxed{0.00}\)
- Age: \(\frac{|50-21|}{32} = \frac{29}{32} = \boxed{0.91}\)
- Sports: \(\frac{|7-5|}{10} = \frac{2}{10} = \boxed{0.20}\)
Average Normalized Difference: \[\frac{0.00 + 0.91 + 0.20}{3} = \boxed{0.37}\]
Normalized Differences for Candidate 2:
- Gender: \(\frac{|1-0|}{1} = \frac{1}{1} = \boxed{1.00}\)
- Age: \(\frac{|50-51|}{32} = \frac{1}{32} = \boxed{0.03}\)
- Sports: \(\frac{|7-9|}{10} = \frac{2}{10} = \boxed{0.20}\)
Average Normalized Difference: \[\frac{1.00 + 0.03 + 0.20}{3} = \boxed{0.41}\]
Normalized Differences for Candidtae 3:
- Gender: \(\frac{|1-1|}{1} = \frac{0}{1} = \boxed{0.00}\)
- Age: \(\frac{|50-53|}{32} = \frac{3}{32} = \boxed{0.09}\)
- Sports: \(\frac{|7-9|}{10} = \frac{2}{10} = \boxed{0.20}\)
Average Normalized Difference: \[\frac{0.00 + 0.09 + 0.20}{3} = \boxed{0.10}\]
Overview
In this session you will learn:
What is the underlying idea of k-Nearest Neighbors
How similarity can be measured with Euclidean distance
Why scaling predictor variables is important for some machine learning models
Why the tidymodels package makes it easy to work with machine learning models
How you can define a recipe to pre-process data with the
tidymodelspackageHow you can define a model-design with the
tidymodelspackageHow you can create a machine learning workflow with the
tidymodelspackageHow metrics derived from a confusion matrix can be used to asses prediction quality
Why you have to be careful when interpreting accuracy, when you work with unbalanced observations
How a machine learning model can process images and how OCR (Optical Character Recognition) works
About the Penguin Dataset
We will work with the Palmer Penguins dataset containing 344 observations about different penguin species and their morphological measurements.
Our goal is to develop a k-Nearest Neighbors model that can predict the species of a penguin (Adelie, Chinstrap, or Gentoo) based on the penguin’s bill dimensions, flipper length, and body mass
Raw Observations from Penguin Dataset
# A tibble: 10 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
4 Adelie Torgersen NA NA NA NA
5 Adelie Torgersen 36.7 19.3 193 3450
6 Adelie Torgersen 39.3 20.6 190 3650
7 Adelie Torgersen 38.9 17.8 181 3625
8 Adelie Torgersen 39.2 19.6 195 4675
9 Adelie Torgersen 34.1 18.1 193 3475
10 Adelie Torgersen 42 20.2 190 4250
# ℹ 2 more variables: sex <fct>, year <int>Observations from Penguin Dataset for Selected Variables
Bill Length,
Note we use clean_names("upper_camel") from the janitor package to change all column (variable) names to UpperCamel.
# A tibble: 10 × 3
Species BillLengthMm BodyMassG
<fct> <dbl> <int>
1 Adelie 39.1 3750
2 Adelie 39.5 3800
3 Adelie 40.3 3250
4 Adelie 36.7 3450
5 Adelie 39.3 3650
6 Adelie 38.9 3625
7 Adelie 39.2 4675
8 Adelie 34.1 3475
9 Adelie 42 4250
10 Adelie 37.8 3300Before Starting with k Nearest Neighbors
Let us find some eyeballing techniques that are related to various machine learning models
Eye Balling Techniques to Identify Penguin Species
try eyeballing the data
Eye Balling Techniques to Identify Penguin Species
Horizonal Boundary
Code
ggplot(DataTrain |>
add_row(Species="unknown", BillLengthMm=42, BodyMassG=4200),
aes(y=BodyMassG, x=BillLengthMm, color=Species)) +
labs(x="Bill Length (mm)", y="Body Mass (g)", color="Species",
alt="A point plot of penguin bill length and body mass by species") +
geom_point(size=3, alpha=0.7) +
#geom_point(aes(x=42, y=4200), size=5, color="black") +
scale_x_continuous(breaks=seq(30, 60, 2)) +
scale_y_continuous(breaks=seq(2500, 6500, 250)) +
scale_colour_manual(values = c("orange", "purple", "darkgreen", "black")) +
geom_hline(yintercept = 3800, linetype = "dashed", color = "red") +
geom_hline(yintercept = 4800, linetype = "dashed", color = "green") +
theme(legend.position = c(0.2, 0.8))
Confusion Matrix
Truth
Prediction Adelie Chinstrap Gentoo
Adelie 64 41 0
Chinstrap 31 16 0
Gentoo 0 30 56 Accuracy:
Overall Accuracy: 57.1 %Can we improve the accuracy?
Eyeballing Techniques to Identify Penguin Species
Creating Subspaces Like Similar to a Decision Tree
Confusion Matrix
Predicted
Actual Adelie Chinstrap Gentoo
Adelie 96 6 3
Chinstrap 1 42 4
Gentoo 2 5 79
Tree-like Boundaries Accuracy: 91.2 %Eyeballing Techniques to Identify Penguin Species
Using a non-linear Decision Boundary Like a Neural Network
Confusion Matrix
Truth
Prediction Adelie Chinstrap Gentoo
Adelie 13 68 24
Chinstrap 32 14 1
Gentoo 0 5 81
Non-linear Boundary Accuracy: 45.4 %So, how does k Nearest Neighbors Work?
- Predicts the class (species) for a new observation (penguin with unkown Species) by finding the observation closest to it - the nearest neighbor
- If k-Nearest Neighbors considers more than one neighboring point (k > 1 ), e.g., the four nearest neighbors (k = 4), the class of the majority of these neighbor points is the predicted class. In case of a tie, the prediction is chosen randomly
- the hyper-parameter k determines the number of neighbors to be considered.
- k is called a hyper-parameter because it has to be chosen at the design stage of the model.
- In contrast to parameters that are determined based on data
k Nearest Neighbors k=1
- Most basic K-Nearest Neighbors model (k=1)
k Nearest Neighbors k=1
[1] "Nearest neighbor at: 42.3 4150"
How to calculate Euclidean Distance for Two Variables
Assume our observations have two predictor variables \(x\) and \(y\). We compare the unknown point \(p\) to one of the points from the training data (e,g., point \(i\)): \[Dist_i=\sqrt{(x_p-x_i)^2+(y_p-y_i)^2}\] ??
How to calculate Euclidean Distance for Three Variables
Assume our observations have three predictor variables \(x\), \(y\), and \(z\). We compare the unknown point \(p\) to one of the points from the training data (e,g., point \(i\)): \[Dist_i=\sqrt{(x_p-x_i)^2+(y_p-y_i)^2+(z_p-z_i)^2}\] ??
How to calculate Euclidean Distance for N Variables
Assume our observations have \(N\) predictor variables \(v_j\) with \(j=1 ... N\). We compare the unknown point \(p\) to one of the points from the training data (e,g., point \(i\)): \[Dist_i=\sqrt{\sum_{j=1}^N(v_{p,j}-v_{i,j})^2}\] ??
Process of Prediction using K-Nearest Neighbors
- Take the first observation from the testing dataset and calculate the distance from this record to all observations in the training dataset.
- Find the observation that has the smallest distance to the testing observation.
- The predicted class (e.g., species) for the observation from the testing dataset is the same as the class from its nearest neighbor.
- For the observation from the testing dataset, we actually know the true class ??? although we never showed it to the model. Therefore, we can compare the true class of the testing observation with the prediction to find out if the prediction was true or false.
- Depending on the prediction (red or white) and whether it was correct, we update one of the four cells in the confusion matrix.
We repeat Steps 1 ??? 5 for all observations from the testing dataset.
Note, when values for the outcome class (e.g., species) are unknown, Steps 4 and 5 are omitted.
k Nearest Neighbors k=4 (for a different unknown species)
k Nearest Neighbors k=4 (for a different unknown species)
4 nearest neighbors vote on species classification
Find the Optimal K hyperparameters
In a real-world application, you have to choose the value for the hyper-parameter
k in the model design stage. The chosen
k is then valid for all model predictions.
This raises the question: How do we find an appropriate value for k ?
The answer is: We use a systematic trial-and-error process called ???tuning???.
Tuning to be covered in later chapters.
Right now, you might be tempted to run the model for different values of k and then use the testing dataset to see which k delivers the best prediction performance.
- This is not an appropriate way to optimize hyper-parameters. Using the testing dataset to optimize hyper-parameters can lead to overfitting
Overfitting occurs when a prediction model performs well on the training data, but when it is used for preditions based on new data that the model has ???never seen before???, it performs poorly.
In general, a k that is too low is prone to be influenced by isolated outliers, although the surrounding neighborhood would suggest otherwise.
On the other hand, a k that is too high would consider a neighborhood so large that it does not represent the neighborhood surrounding the prediction point anymore
k Nearest Neighbors k=4 (for a different unknown species)
Watch the scale: mm vs. g. Need better scaling!
The Visual vs. The Reality
What We See:
N2 (purple point) looks very far away horizontally.
N4 (green point) looks much closer overall.
Visually, N4 should be the closer neighbor.
What The Math Says:
# Distance to N2 (approx at 53, 4500)
bill_diff_N2 <- 53 - 45 # = 8 mm
mass_diff_N2 <- 4500 - 4500 # = 0 g
sqrt(bill_diff_N2^2 + mass_diff_N2^2) # ??? 8[1] 8# Distance to N4 (approx at 43, 4450)
bill_diff_N4 <- 43 - 45 # = -2 mm
mass_diff_N4 <- 4450 - 4500 # = -50 g
sqrt(bill_diff_N4^2 + mass_diff_N4^2) # ??? 50[1] 50.03998Why Does This Happen?
The Scaling Problem
Different Measurement Scales:
| Variable | Units | Typical Range | Example Difference |
|---|---|---|---|
| Bill Length | millimeters (mm) | 30-60 | 0.1 - 2 mm |
| Body Mass | grams (g) | 2500-6500 | 20 - 500 g |
The Problem:
Body Mass values are naturally 100-1000?? larger than Bill Length values
When we square these differences for Euclidean distance, the gap becomes 10,000-1,000,000?? larger
Body Mass completely dominates the distance calculation
This is unfair! Both measurements should contribute appropriately to finding the nearest neighbor.
We Need to Scale Our Variables
Goal: Transform both variables to comparable ranges so neither dominates the distance calculation.
Before Scaling:
Bill Length: 42.1 - 42.0 = 0.1 ??? contributes 0.01 to distance??Body Mass: 4220 - 4200 = 20 ??? contributes 400 to distance??
Body Mass dominates (99.997% of total distance)A Few Common Scaling Options
Same units
Divide or multiply to get the same units. This is often not possible (e.g.,
BillLengthandBodyMass). Or it is not feasible (e.g.BillLengthin mm andBodyMassin grams are in vastly different ranges)Rescaling
Generates a variable \(y\) that is scaled to a range between 0 and 1 based on the original variable’s value \(x\), its minimum \(x_{min}\) and its maximum \(x_{max}\): \[ y= \frac{x-x_{min}}{x_{max} - x_{min}}\]
Z-Score Normalization
Z-score normalization uses the mean (\(\overline x\)) and the standard deviation (\(s\)) of a variable to scale the variable \(x\) to the variable \(z\):
\[z=\frac{x-\overline x}{s}\]??
Comparison: Before and After Scaling
Raw Values (Problem)
Standardized Values (Solution)
Time to Run k-Nearest Neighbors
Loading Data and Selecting Variables
# A tibble: 10 × 3
Species BillLengthMm BodyMassG
<fct> <dbl> <int>
1 Adelie 39.1 3750
2 Adelie 39.5 3800
3 Adelie 40.3 3250
4 Adelie 36.7 3450
5 Adelie 39.3 3650
6 Adelie 38.9 3625
7 Adelie 39.2 4675
8 Adelie 34.1 3475
9 Adelie 42 4250
10 Adelie 37.8 3300Time to Run k-Nearest Neighbors
The tidymodels package provides a standardized workflow with standardized commands for the following tasks:
- Data splitting (training and testing)
- Pre-processing data with
recipes
- Creating machine learning model-design with only three standardized commands
- Tuning hyper-parameters
- assessing prediciton quality using a set of predefined metrics
Generate Training and Testing Data (Splitting):
# A tibble: 6 × 3
Species BillLengthMm BodyMassG
<fct> <dbl> <int>
1 Adelie 40.3 3250
2 Adelie 36.7 3450
3 Adelie 39.3 3650
4 Adelie 38.9 3625
5 Adelie 39.2 4675
6 Adelie 42 4250# A tibble: 6 × 3
Species BillLengthMm BodyMassG
<fct> <dbl> <int>
1 Adelie 39.1 3750
2 Adelie 39.5 3800
3 Adelie 34.1 3475
4 Adelie 41.1 3200
5 Adelie 36.6 3700
6 Adelie 38.7 3450Time to Run k-Nearest Neighbors
Click here to find a reference list for various Step_ commands
- Recipes simplify data pre-processing
- You can compare a
tidymodelsrecipe to a recipe in a cookbook.- First, the ingredients are listed, and then you find the steps to use these ingredients to cook the meal.
Recipe: Prepare Data for Analysis:
The recipe() command is followed by instructions on how to process the data step by step. Each step starts with step_, indicating that the instruction (command) is part of a recipe.
Or:
When to Use Recipes and When to Use Select() and Mutate()
Generally, using a recipe is advisable because we can reuse a recipe on other data frames.
It is good practice to use select() before a recipe to reduce the columns of an original data frame to only those columns (variables) that are required for the analysis.
- This allows us to use the .-notation in the formula argument of the recipe() command.
- When transforming outcome variables, it is advised to always do this outside of a recipe. For example, in the R code above, we used mutate() to convert the outcome variable Species from a
characterdata type to afactordata type outside the recipe.
- If a recipe is later used on another dataset for prediction, this dataset might not contain a column for the outcome variable.
Time to Run k-Nearest Neighbors
Click here to find a reference list for various ML algorithm commands
a model-design determines which machine learning model from which R package should be used.
To define a model design within the tidymodels environment, only three commands (connected with |>) are required.
Creating a Model Design:
K-Nearest Neighbor Model Specification (classification)
Main Arguments:
neighbors = 4
weight_func = rectangular
Computational engine: kknn Time to Run k-Nearest Neighbors
So far, we’ve defined a recipe and a model-design
We put it all together in a workflow
Then, the workflow is fiited (calibrated) to the training data
Putting it all together in a fitted workflow:
══ Workflow [trained] ══════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: nearest_neighbor()
── Preprocessor ────────────────────────────────────────────────────────────────
2 Recipe Steps
• step_naomit()
• step_normalize()
── Model ───────────────────────────────────────────────────────────────────────
Call:
kknn::train.kknn(formula = ..y ~ ., data = data, ks = min_rows(4, data, 5), kernel = ~"rectangular")
Type of response variable: nominal
Minimal misclassification: 0.05882353
Best kernel: rectangular
Best k: 4Time to Run k-Nearest Neighbors
How to use the fitted workflow to predict the penguin species for the penguins in the testing dataset:
- Start with observation \(i=1\) from
DataTest(the first observation). - Take observation \(i\) from
DataTestand useBillLengthandBodyMassGto calculate the Euclidean distance to each of the observations ofDataTrain. - Isolate the 4 observations with the smallest Euclidean distance and use the majority of their species as a prediction for observation \(i\) from
DataTest(in case of a tie, decide randomly). - Increase \(i\) by one (i.e., take the next observation from
DataTest) and go to step 2 (until allDataTestobservations are processed).
Time to Run k-Nearest Neighbors
Predicting with the fitted workflow using predict() (not exactly helpful!):
# A tibble: 6 × 1
.pred_class
<fct>
1 Adelie
2 Adelie
3 Adelie
4 Adelie
5 Adelie
6 Adelie Note: we cannot see if the predicitons are correct because we cannot easily compare
Time to Run k-Nearest Neighbors
Predicting with the fitted workflow using augment() which augments DataTest with the predictions:
# A tibble: 6 × 7
.pred_class .pred_Adelie .pred_Chinstrap .pred_Gentoo Species BillLengthMm
<fct> <dbl> <dbl> <dbl> <fct> <dbl>
1 Adelie 1 0 0 Adelie 39.1
2 Adelie 1 0 0 Adelie 39.5
3 Adelie 1 0 0 Adelie 34.1
4 Adelie 0.75 0.25 0 Adelie 41.1
5 Adelie 1 0 0 Adelie 36.6
6 Adelie 1 0 0 Adelie 38.7
# ℹ 1 more variable: BodyMassG <int>Having a Data Frame with truth and estimate we can calculate performance metrics
The tidymodels package provides several commands to calculate metrics that reflect predictive performance.
Most of these commands compare the estimate with the truth and then calculate the related metrics.
We can use the conmat() command to create the confusion matrix
Confusion Matrix:
Truth
Prediction Adelie Chinstrap Gentoo
Adelie 42 2 1
Chinstrap 1 19 0
Gentoo 3 0 36Reading the Confusion Matrix
Truth Prediction Adelie Chinstrap Gentoo Adelie 42 2 1 Chinstrap 1 19 0 Gentoo 3 0 36
- Key Insight: Calculate TP, FP, FN, TN for each class separately
- Think “One-vs-Rest” for each class
Understanding the Metrics
TP: Correctly predicted as THIS class
FP: Incorrectly predicted as THIS class
FN: Actually WAS this class, but we predicted it as something else
TN: Correctly identified as NOT this class
Understanding the Metrics
To make sure the accuracy rate is not misleading, we look at accuracy(), sensitivity(), precision() and specificity() for the penguin data.
- Sensitivity: the rate of correctly predicted positives
- “Of all the actual positives, how many did we catch?”
- Precision: the rate of correct positive predicitons
- “Of all our positive predictions, how many were correct?”
- Specificity: the rate of correctly predicted negatives
- “Of all the actual negatives, how many did we correctly”
Note: In our case (3 classes), sensitivity and specificity need to be calculated for each class
For each penguin species, calculate separately
Example: Adelie Penguins
“Is this penguin an Adelie?”
Truth Prediction Adelie Chinstrap Gentoo Adelie TP:42 FP: 2 FP: 1 Chinstrap FN: 1 TN:19 TN: 0 Gentoo FN: 3 TN: 0 TN:36
- TP = 42: Correctly identified as Adelie
- FP = 3: Said “Adelie” but wrong (2+1)
- FN = 4: Missed Adelie penguins (1+3)
- TN = 55: Correctly NOT Adelie (19+0+0+36)
Precision = TP/(TP+FP) = 42/(42+3) = 42/45 = 93.3%
Recall (Sensitivity) = TP/(TP+FN) = 42/(42+4) = 42/46 = 91.3%
Specificity = TN/(TN+FP) = 55/(55+3) = 55/58 = 94.8%
Example: Chinstrap Penguins
“Is this penguin a Chinstrap?”
Truth Prediction Adelie Chinstrap Gentoo Adelie TN:42 FN: 2 TN: 1 Chinstrap FP: 1 TP:19 FP: 0 Gentoo TN: 3 FN: 0 TN:36
- TP = 19: Correctly identified as Chinstrap
- FP = 1: Said “Chinstrap” but wrong (1+0)
- FN = 2: Missed Chinstrap penguins (2+0)
- TN = 82: Correctly NOT Chinstrap (42+1+3+36)
Precision = TP/(TP+FP) = 19/(19+1) = 19/20 = 95.0%
Recall (Sensitivity) = TP/(TP+FN) = 19/(19+2) = 19/21 = 90.5%
Specificity = TN/(TN+FP) = 82/(82+1) = 82/83 = 98.8%
Example: Gentoo Penguins
“Is this penguin a Gentoo?”
Truth Prediction Adelie Chinstrap Gentoo Adelie TN:42 TN: 2 FN: 1 Chinstrap TN: 1 TN:19 FN: 0 Gentoo FP: 3 FP: 0 TP:36
- TP = 36: Correctly identified as Gentoo
- FP = 3: Said “Gentoo” but wrong (3+0)
- FN = 1: Missed Gentoo penguins (1+0)
- TN = 64: Correctly NOT Gentoo (42+2+1+19)
Precision = TP/(TP+FP) = 36/(36+3) = 36/39 = 92.3%
Recall (Sensitivity) = TP/(TP+FN) = 36/(36+1) = 36/37 = 97.3%
Specificity = TN/(TN+FP) = 64/(64+3) = 64/67 = 95.5%
Summary: All Three Classes
| Class | TP | FP | FN | TN | Precision | Recall |
|---|---|---|---|---|---|---|
| Adelie | 42 | 3 | 4 | 55 | 93.3% | 91.3% |
| Chinstrap | 19 | 1 | 2 | 82 | 95.0% | 90.5% |
| Gentoo | 36 | 3 | 1 | 64 | 92.3% | 97.3% |
Overall Accuracy: (42+19+36)/104 = 93.3%
Let’s Look at Metrics in R
# A tibble: 1 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy multiclass 0.933# A tibble: 1 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 sensitivity macro 0.930# A tibble: 1 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 specificity macro 0.964# A tibble: 1 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 precision macro 0.935Get all the metrics at Once
# A tibble: 4 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 sensitivity macro 0.930
2 specificity macro 0.964
3 precision macro 0.935
4 recall macro 0.930When to Use Each Metric: Examples
Medical Diagnosis (Cancer Detection)
- Prioritize: Recall (Sensitivity)
- Don’t miss any cancer cases
- Missing a positive case (FN) is very costly
Spam Email Filter
- Prioritize: Precision
- Don’t mark important emails as spam
- False positives (legitimate email marked as spam) are costly
COVID-19 Screening Test
- Prioritize: Specificity
- Correctly identify healthy people
- False positives cause unnecessary quarantine and anxiety
Balanced Dataset (Equal class sizes)
- Use: Accuracy
- Simple and interpretable
- All error types have similar costs
Imbalanced Dataset (e.g., fraud detection: 99% normal, 1% fraud)
- Avoid: Accuracy (can get 99% by predicting “normal” every time!)
- Use: Precision, Recall, and F1-Score
The Precision-Recall Trade-off
There’s often a trade-off:
- Increasing Recall (catch more positives) ??? often decreases Precision (more false alarms)
- Increasing Precision (fewer false alarms) ??? often decreases Recall (miss some positives)
F1-Score balances both:
F1-Score = 2 ?? (Precision ?? Recall) / (Precision + Recall) - Harmonic mean of Precision and Recall
- Good when you need a single metric that balances both
Our Penguin Example: - Adelie: F1 = 2 ?? (0.933 ?? 0.913) / (0.933 + 0.913) = 0.923
- Chinstrap: F1 = 2 ?? (0.905 ?? 0.950) / (0.905 + 0.950) = 0.927
- Gentoo: F1 = 2 ?? (0.973 ?? 0.923) / (0.973 + 0.923) = 0.947
Project: Design a Machine Learning Workflow for Optical Character Recognition ??
MNIST Data Set
You will develop a machine learning model based on k-Nearest Neighbors to recognize handwritten digits from images.
You will use the MNIST dataset, a standard dataset for image recognition in machine learning (60,000 images for training and 10,000 images for testing). Developed by LeCun, Cortes, and Burges (2010) based on two datasets from handwritten digits obtained from Census workers and high school students.
We will use only the first 500 images of the original MNIST dataset to speed up the k-Nearest Neighbors model’s training time.
Visualization of the First Six Images from the MNIST Data Set
How a Image is Stored in the Mnist Dataset
Image of a Handwritten Nine
The image has 28 rows and 28 columns. Each of the 784 cells (pixels) holds a value between 0 (black) and 255 (white)
How a Image is Stored in the Mnist Dataset
- Pixel values for a single image are not stored in a table. Ohterwise we would end-up with a table containing tables.
- Pixel values are stored as one row for each image.
- Concatenating the 28 rows of an image into one row with 28*28=784 cells (pixels)
Three Rows from the Data Frame of the MNIST Dataset
Label Pix1 Pix2 Pix3 Pix4 Pix5 Pix6 Pix7 Pix8 Pix9 Pix10 Pix11 Pix12 Pix13
1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2 5 0 0 0 0 0 0 0 0 0 0 0 0 0
3 3 0 0 0 0 0 0 0 0 0 0 0 0 0
Pix14 Pix15 Pix16 Pix17 Pix18 Pix19 Pix20 Pix21 Pix22 Pix23 Pix24 Pix25 Pix26
1 0 0 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0 0 0
Pix27 Pix28 Pix29 Pix30 Pix31 Pix32 Pix33 Pix34 Pix35 Pix36 Pix37 Pix38 Pix39
1 0 0 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0 0 0
Pix40 Pix41 Pix42 Pix43 Pix44 Pix45 Pix46 Pix47 Pix48 Pix49 Pix50 Pix51 Pix52
1 0 0 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0 0 0
Pix53 Pix54 Pix55 Pix56 Pix57 Pix58 Pix59 Pix60 Pix61 Pix62 Pix63 Pix64 Pix65
1 0 0 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0 0 0
Pix66 Pix67 Pix68 Pix69 Pix70 Pix71 Pix72 Pix73 Pix74 Pix75 Pix76 Pix77 Pix78
1 0 0 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0 0 0
Pix79 Pix80 Pix81 Pix82 Pix83 Pix84 Pix85 Pix86 Pix87 Pix88 Pix89 Pix90 Pix91
1 0 0 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0 0 0
Pix92 Pix93 Pix94 Pix95 Pix96 Pix97 Pix98 Pix99 Pix100 Pix101 Pix102 Pix103
1 0 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0 0
Pix104 Pix105 Pix106 Pix107 Pix108 Pix109 Pix110 Pix111 Pix112 Pix113 Pix114
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix115 Pix116 Pix117 Pix118 Pix119 Pix120 Pix121 Pix122 Pix123 Pix124 Pix125
1 0 0 0 0 0 0 0 0 0 5 138
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix126 Pix127 Pix128 Pix129 Pix130 Pix131 Pix132 Pix133 Pix134 Pix135 Pix136
1 253 148 22 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix137 Pix138 Pix139 Pix140 Pix141 Pix142 Pix143 Pix144 Pix145 Pix146 Pix147
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix148 Pix149 Pix150 Pix151 Pix152 Pix153 Pix154 Pix155 Pix156 Pix157 Pix158
1 0 0 0 0 120 252 252 231 245 59 0
2 0 13 191 255 253 253 253 253 192 113 191
3 0 0 149 253 253 253 96 11 0 0 0
Pix159 Pix160 Pix161 Pix162 Pix163 Pix164 Pix165 Pix166 Pix167 Pix168 Pix169
1 0 0 0 0 0 0 0 0 0 0 0
2 113 191 255 90 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix170 Pix171 Pix172 Pix173 Pix174 Pix175 Pix176 Pix177 Pix178 Pix179 Pix180
1 0 0 0 0 0 0 0 0 0 0 161
2 0 0 0 0 0 0 0 29 252 253 252
3 0 0 0 0 0 0 0 147 253 252 252
Pix181 Pix182 Pix183 Pix184 Pix185 Pix186 Pix187 Pix188 Pix189 Pix190 Pix191
1 252 185 122 253 156 101 44 0 0 0 0
2 252 252 252 253 252 252 252 252 253 243 50
3 252 252 189 0 0 0 0 0 0 0 0
Pix192 Pix193 Pix194 Pix195 Pix196 Pix197 Pix198 Pix199 Pix200 Pix201 Pix202
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix203 Pix204 Pix205 Pix206 Pix207 Pix208 Pix209 Pix210 Pix211 Pix212 Pix213
1 0 0 0 0 19 236 252 119 21 169 252
2 0 0 60 252 253 201 195 195 195 222 201
3 0 26 236 253 252 252 252 252 247 99 0
Pix214 Pix215 Pix216 Pix217 Pix218 Pix219 Pix220 Pix221 Pix222 Pix223 Pix224
1 252 236 155 0 0 0 0 0 0 0 0
2 208 252 252 196 195 43 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix225 Pix226 Pix227 Pix228 Pix229 Pix230 Pix231 Pix232 Pix233 Pix234 Pix235
1 0 0 0 0 0 0 0 0 0 0 181
2 0 0 0 0 0 0 0 0 169 252 253
3 0 0 0 0 0 0 57 224 252 253 235
Pix236 Pix237 Pix238 Pix239 Pix240 Pix241 Pix242 Pix243 Pix244 Pix245 Pix246
1 252 221 25 0 3 169 252 252 252 106 0
2 27 0 0 0 38 9 19 84 84 0 0
3 160 160 202 253 244 56 0 0 0 0 0
Pix247 Pix248 Pix249 Pix250 Pix251 Pix252 Pix253 Pix254 Pix255 Pix256 Pix257
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix258 Pix259 Pix260 Pix261 Pix262 Pix263 Pix264 Pix265 Pix266 Pix267 Pix268
1 0 0 0 0 11 255 253 173 0 0 0
2 0 0 0 169 252 253 27 0 0 0 0
3 0 122 252 252 243 60 0 0 63 253 252
Pix269 Pix270 Pix271 Pix272 Pix273 Pix274 Pix275 Pix276 Pix277 Pix278 Pix279
1 0 32 229 253 231 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 121 0 0 0 0 0 0 0 0 0 0
Pix280 Pix281 Pix282 Pix283 Pix284 Pix285 Pix286 Pix287 Pix288 Pix289 Pix290
1 0 0 0 0 0 0 0 0 0 0 136
2 0 0 0 0 0 0 0 0 0 170 253
3 0 0 0 0 0 0 0 185 253 253 168
Pix291 Pix292 Pix293 Pix294 Pix295 Pix296 Pix297 Pix298 Pix299 Pix300 Pix301
1 253 244 56 0 0 0 0 0 186 252 245
2 141 0 0 0 0 0 0 0 0 0 0
3 0 0 19 128 255 253 190 5 0 0 0
Pix302 Pix303 Pix304 Pix305 Pix306 Pix307 Pix308 Pix309 Pix310 Pix311 Pix312
1 80 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix313 Pix314 Pix315 Pix316 Pix317 Pix318 Pix319 Pix320 Pix321 Pix322 Pix323
1 0 0 0 0 68 246 253 174 0 0 0
2 0 0 0 51 243 252 140 0 19 85 38
3 0 0 163 252 231 42 0 0 207 252 253
Pix324 Pix325 Pix326 Pix327 Pix328 Pix329 Pix330 Pix331 Pix332 Pix333 Pix334
1 0 0 0 68 246 253 206 0 0 0 0
2 38 85 66 0 0 0 0 0 0 0 0
3 252 252 67 0 0 0 0 0 0 0 0
Pix335 Pix336 Pix337 Pix338 Pix339 Pix340 Pix341 Pix342 Pix343 Pix344 Pix345
1 0 0 0 0 0 0 0 0 0 0 93
2 0 0 0 0 0 0 0 0 0 166 252
3 0 0 0 0 0 0 0 0 51 183 48
Pix346 Pix347 Pix348 Pix349 Pix350 Pix351 Pix352 Pix353 Pix354 Pix355 Pix356
1 252 253 92 0 0 0 0 0 0 0 188
2 252 229 197 209 252 221 222 252 239 197 119
3 0 0 0 207 252 253 252 252 227 131 0
Pix357 Pix358 Pix359 Pix360 Pix361 Pix362 Pix363 Pix364 Pix365 Pix366 Pix367
1 253 244 56 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix368 Pix369 Pix370 Pix371 Pix372 Pix373 Pix374 Pix375 Pix376 Pix377 Pix378
1 0 0 0 0 0 93 252 243 50 0 0
2 0 0 0 57 234 252 252 253 252 252 252
3 0 0 0 0 0 0 0 0 0 207 252
Pix379 Pix380 Pix381 Pix382 Pix383 Pix384 Pix385 Pix386 Pix387 Pix388 Pix389
1 0 0 0 0 0 116 253 252 69 0 0
2 252 253 252 252 252 252 16 0 0 0 0
3 253 252 252 252 252 0 0 0 0 0 0
Pix390 Pix391 Pix392 Pix393 Pix394 Pix395 Pix396 Pix397 Pix398 Pix399 Pix400
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 85 252
3 0 0 0 0 0 0 0 0 0 0 0
Pix401 Pix402 Pix403 Pix404 Pix405 Pix406 Pix407 Pix408 Pix409 Pix410 Pix411
1 208 253 221 0 0 0 0 0 0 0 0
2 252 252 253 252 252 252 252 253 173 252 252
3 0 0 0 0 113 242 243 137 168 252 252
Pix412 Pix413 Pix414 Pix415 Pix416 Pix417 Pix418 Pix419 Pix420 Pix421 Pix422
1 0 255 253 69 0 0 0 0 0 0 0
2 252 203 94 0 0 0 0 0 0 0 0
3 210 0 0 0 0 0 0 0 0 0 0
Pix423 Pix424 Pix425 Pix426 Pix427 Pix428 Pix429 Pix430 Pix431 Pix432 Pix433
1 0 0 0 0 0 13 215 252 95 0 0
2 0 0 0 0 0 32 140 140 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix434 Pix435 Pix436 Pix437 Pix438 Pix439 Pix440 Pix441 Pix442 Pix443 Pix444
1 0 0 0 0 0 0 0 253 252 69 0
2 0 0 0 0 32 140 203 255 206 25 0
3 0 0 0 0 136 241 255 92 0 0 0
Pix445 Pix446 Pix447 Pix448 Pix449 Pix450 Pix451 Pix452 Pix453 Pix454 Pix455
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix456 Pix457 Pix458 Pix459 Pix460 Pix461 Pix462 Pix463 Pix464 Pix465 Pix466
1 70 252 252 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix467 Pix468 Pix469 Pix470 Pix471 Pix472 Pix473 Pix474 Pix475 Pix476 Pix477
1 0 0 253 252 69 0 0 0 0 0 0
2 0 140 253 252 55 0 0 0 0 0 0
3 95 253 113 0 0 0 0 0 0 0 0
Pix478 Pix479 Pix480 Pix481 Pix482 Pix483 Pix484 Pix485 Pix486 Pix487 Pix488
1 0 0 0 0 0 0 70 252 252 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix489 Pix490 Pix491 Pix492 Pix493 Pix494 Pix495 Pix496 Pix497 Pix498 Pix499
1 0 0 0 0 0 0 0 43 253 252 69
2 0 0 0 0 0 0 0 110 253 252 55
3 0 0 0 0 0 0 0 253 219 19 0
Pix500 Pix501 Pix502 Pix503 Pix504 Pix505 Pix506 Pix507 Pix508 Pix509 Pix510
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix511 Pix512 Pix513 Pix514 Pix515 Pix516 Pix517 Pix518 Pix519 Pix520 Pix521
1 0 70 252 252 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix522 Pix523 Pix524 Pix525 Pix526 Pix527 Pix528 Pix529 Pix530 Pix531 Pix532
1 0 95 230 243 117 6 0 0 0 0 0
2 0 0 0 253 252 149 0 0 0 0 0
3 0 0 211 252 69 0 0 0 0 0 0
Pix533 Pix534 Pix535 Pix536 Pix537 Pix538 Pix539 Pix540 Pix541 Pix542 Pix543
1 0 0 0 0 0 0 0 32 229 253 11
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 22
Pix544 Pix545 Pix546 Pix547 Pix548 Pix549 Pix550 Pix551 Pix552 Pix553 Pix554
1 0 0 0 0 0 5 55 233 253 221 0
2 0 0 0 0 0 0 0 0 79 253 252
3 32 0 0 0 0 0 0 0 191 252 69
Pix555 Pix556 Pix557 Pix558 Pix559 Pix560 Pix561 Pix562 Pix563 Pix564 Pix565
1 0 0 0 0 0 0 0 0 0 0 0
2 195 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix566 Pix567 Pix568 Pix569 Pix570 Pix571 Pix572 Pix573 Pix574 Pix575 Pix576
1 0 0 0 186 252 193 17 0 0 0 26
2 0 38 113 38 0 0 0 0 0 0 0
3 0 0 0 0 0 162 222 97 24 0 0
Pix577 Pix578 Pix579 Pix580 Pix581 Pix582 Pix583 Pix584 Pix585 Pix586 Pix587
1 136 252 252 231 42 0 0 0 0 0 0
2 38 144 253 253 255 253 133 0 0 0 0
3 0 9 128 255 253 122 0 0 0 0 0
Pix588 Pix589 Pix590 Pix591 Pix592 Pix593 Pix594 Pix595 Pix596 Pix597 Pix598
1 0 0 0 0 0 0 0 0 0 93 252
2 0 0 0 0 0 0 0 85 252 234 146
3 0 0 0 0 0 0 0 0 0 0 0
Pix599 Pix600 Pix601 Pix602 Pix603 Pix604 Pix605 Pix606 Pix607 Pix608 Pix609
1 253 209 184 184 184 222 252 252 227 100 0
2 85 85 66 57 85 226 234 252 252 252 253
3 88 252 252 252 162 161 161 194 252 253 244
Pix610 Pix611 Pix612 Pix613 Pix614 Pix615 Pix616 Pix617 Pix618 Pix619 Pix620
1 0 0 0 0 0 0 0 0 0 0 0
2 223 37 0 0 0 0 0 0 0 0 0
3 56 0 0 0 0 0 0 0 0 0 0
Pix621 Pix622 Pix623 Pix624 Pix625 Pix626 Pix627 Pix628 Pix629 Pix630 Pix631
1 0 0 0 0 17 98 253 252 252 252 252
2 0 0 19 209 252 252 253 252 239 234 252
3 0 0 0 0 0 0 47 252 252 252 253
Pix632 Pix633 Pix634 Pix635 Pix636 Pix637 Pix638 Pix639 Pix640 Pix641 Pix642
1 253 235 160 50 0 0 0 0 0 0 0
2 253 252 252 252 252 196 52 0 0 0 0
3 252 252 252 252 247 98 0 0 0 0 0
Pix643 Pix644 Pix645 Pix646 Pix647 Pix648 Pix649 Pix650 Pix651 Pix652 Pix653
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 97 227
3 0 0 0 0 0 0 0 0 0 0 0
Pix654 Pix655 Pix656 Pix657 Pix658 Pix659 Pix660 Pix661 Pix662 Pix663 Pix664
1 0 33 137 221 252 147 75 18 0 0 0
2 252 253 252 252 252 252 253 252 245 129 84
3 0 9 45 173 252 253 252 252 252 252 146
Pix665 Pix666 Pix667 Pix668 Pix669 Pix670 Pix671 Pix672 Pix673 Pix674 Pix675
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix676 Pix677 Pix678 Pix679 Pix680 Pix681 Pix682 Pix683 Pix684 Pix685 Pix686
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 13 189 253 252 252 252
3 0 0 0 0 0 0 0 0 0 9 75
Pix687 Pix688 Pix689 Pix690 Pix691 Pix692 Pix693 Pix694 Pix695 Pix696 Pix697
1 0 0 0 0 0 0 0 0 0 0 0
2 252 190 112 87 0 0 0 0 0 0 0
3 201 252 221 137 137 0 0 0 0 0 0
Pix698 Pix699 Pix700 Pix701 Pix702 Pix703 Pix704 Pix705 Pix706 Pix707 Pix708
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix709 Pix710 Pix711 Pix712 Pix713 Pix714 Pix715 Pix716 Pix717 Pix718 Pix719
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix720 Pix721 Pix722 Pix723 Pix724 Pix725 Pix726 Pix727 Pix728 Pix729 Pix730
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix731 Pix732 Pix733 Pix734 Pix735 Pix736 Pix737 Pix738 Pix739 Pix740 Pix741
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix742 Pix743 Pix744 Pix745 Pix746 Pix747 Pix748 Pix749 Pix750 Pix751 Pix752
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix753 Pix754 Pix755 Pix756 Pix757 Pix758 Pix759 Pix760 Pix761 Pix762 Pix763
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix764 Pix765 Pix766 Pix767 Pix768 Pix769 Pix770 Pix771 Pix772 Pix773 Pix774
1 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0
Pix775 Pix776 Pix777 Pix778 Pix779 Pix780 Pix781 Pix782 Pix783
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0Go to Project in Book
Build your own OCR system.??
Dr. Jomana Bashatah | ISE 423