CertSanctuary

AIF-C01 · MLA-C02

Accuracy, precision, recall and F1 score explained

How to read a confusion matrix and calculate accuracy, precision, recall and F1 score, with a worked fraud example and tips for choosing the right metric.

Start with the confusion matrix

Classification metrics all come from four counts. Imagine a model that flags fraudulent transactions. For each transaction it predicts fraud or not fraud, and each prediction lands in one of four boxes:

Accuracy, with a worked example

Say there are 1,000 transactions and 20 are actually fraud. The model flags 25 transactions, and 15 of those are real fraud. That gives TP = 15, FP = 10, FN = 5 and TN = 970. Every metric below uses these numbers.

Accuracy is the share of all predictions that were right: (TP + TN) / total. Here that is (15 + 970) / 1,000 = 98.5%.

That sounds great, but look closer. A model that always predicts "not fraud" would score 98% accuracy on this data while catching zero fraud. Accuracy is misleading when one class is much rarer than the other, which is called class imbalance. Fraud, disease screening and equipment failure data are almost always imbalanced.

Precision

Precision answers: when the model says positive, how often is it right? The formula is TP / (TP + FP). Here, 15 / 25 = 60%.

Optimize for precision when false positives are expensive. A spam filter that sends a customer's invoice to the junk folder causes real harm, so a spam model needs high precision.

Recall

Recall answers: of all the real positives, how many did the model find? The formula is TP / (TP + FN). Here, 15 / 20 = 75%. Recall is also called sensitivity or the true positive rate.

Optimize for recall when false negatives are expensive. Missing a cancer case or a fraudulent wire transfer costs far more than a second look at a false alarm, so those models need high recall.

F1 score and the trade-off

Precision and recall pull against each other. Lowering the decision threshold flags more cases, which raises recall and usually lowers precision. Raising it does the opposite.

The F1 score combines both into one number using the harmonic mean: 2 x (precision x recall) / (precision + recall). Here, 2 x (0.60 x 0.75) / (0.60 + 0.75) = 0.67. The harmonic mean punishes imbalance, so a model with 100% recall and 5% precision gets a low F1. Use F1 when you care about both kinds of error and the classes are imbalanced.

Two related terms show up too. Specificity is TN / (TN + FP), the share of real negatives correctly identified. AUC-ROC measures how well the model separates classes across all thresholds, where 1.0 is perfect and 0.5 is no better than random guessing.

Choosing a metric on the exam

Read the scenario for which mistake hurts more.

Not for regression

These metrics apply to classification, where the output is a category. If the model predicts a number, such as a house price or next week's sales, use regression metrics such as mean absolute error (MAE), root mean squared error (RMSE) or R squared instead. A question asking for the F1 score of a price prediction model has a wrong premise.

Practice questions

  1. A hospital is piloting a large language model (LLM) that writes discharge summaries from clinical notes. Clinicians have written reference summaries for 500 varied past cases. The hospital wants to measure how closely the model's summaries match the clinicians' versions. Which metric should the hospital use?

    1. Value of the loss function
    2. Recall- Oriented Understudy for Gisting Evaluation (ROUGE) score
    3. Latency of the text generation
    4. Semantic robustness
    Answer

    B. Comparing the model's summaries with the clinicians' versions calls for ROUGE, which measures overlap between generated text and human-written reference text. Semantic robustness measures how much output changes when the input is slightly altered. Latency and loss don't compare output with a reference at all.

    AIF-C01 question 106

  2. A metal parts manufacturer uses a foundation model (FM) to classify photos of castings as pass, crack or porosity. Before relying on the FM on the production line, the quality team wants to know how accurate it is at this image classification task. What should the team do?

    1. Assess how accurately the model reproduces the colors in the processed images.
    2. Measure the model's accuracy against a predefined benchmark dataset of labeled images.
    3. Count the number of layers in the model's neural network.
    4. Calculate the total cost of the compute resources the model uses.
    Answer

    B. Accuracy can only be measured against known correct answers, so the team should run the model on a benchmark set of images that already carry correct labels and compare its predictions with those labels. Compute cost and the number of network layers describe expense and architecture, and neither says whether the model's classifications are right.

    AIF-C01 question 70

  3. A recycling plant trains a deep learning model that sorts photos of items on a conveyor belt into glass, plastic, metal or paper. The engineers want to measure how well the model performs, including which materials it mistakes for each other. Which option should the engineers use to evaluate the model?

    1. Mean squared error (MSE)
    2. Correlation matrix
    3. R2 score
    4. Confusion matrix
    Answer

    D. A confusion matrix is a table of actual classes against predicted classes, so it shows how many items of each material were sorted correctly and exactly which materials are confused. Mean squared error and R2 score are for regression models that predict numbers, and a correlation matrix describes relationships between variables in a dataset, not model performance.

    AIF-C01 question 59

  4. A ride-share company uses an ML model to predict arrival times. Since a new bridge opened and several streets became one-way, travel patterns in the city keep changing, and the company needs the model's accuracy to keep improving instead of slipping. Which approach fits this situation?

    1. Use Amazon CloudWatch to analyze customer orders.
    2. Use Amazon Personalize and iterate with historical data.
    3. Use Amazon Rekognition to optimize the model.
    4. Use Amazon SageMaker and iterate with newer data.
    Answer

    D. When real-world patterns shift, the model has to learn from data that reflects the change, so retrain it in Amazon SageMaker AI on newer trip data. Amazon Personalize builds recommendations, and iterating on historical data would only reinforce patterns that no longer hold. Rekognition analyzes images and CloudWatch monitors metrics and logs, and neither retrains a prediction model.

    AIF-C01 question 118

Practice AIF-C01 Practice MLA-C02 All topics