Easy ReviewAIF-C01
Fundamentals of AI and ML23

ML lifecycle and MLOps

The phases of a machine learning project, how a model is trained and judged, and how MLOps keeps it working in production.

A machine learning project runs through repeating phases, from the business goal to monitoring in production. These sections follow the AWS Well-Architected Machine Learning Lens.

Phases of the ML lifecycle

An ML project isn’t a single training run. It’s a loop: agree on the business goal, frame it as a prediction, prepare data, build a model, deploy it, and watch it, feeding what you learn back into earlier phases.

How it works

  1. Business goal: define the problem, the value and how you’ll measure it. Ask whether ML is needed at all.
  2. Problem framing: decide what to predict (the target) from which inputs, and the metric to optimize.
  3. Data processing: collect data, clean it, explore it (EDA) and engineer features.
  4. Model development: pick an algorithm, train, tune hyperparameters and evaluate.
  5. Deployment: serve the model so applications get predictions (inference).
  6. Monitoring: watch data and quality in production and retrain when they drift.

Exam cues

  • If the question says a fixed rule gives the exact answer, or there’s too little data, think “ML isn’t appropriate”, because the lens says rules can beat ML.
  • If the question says understand the data with charts and summary statistics, think exploratory data analysis.
  • If the question says create, transform or select input variables, think feature engineering.

Remember

  • Six phases, in a loop: goal, framing, data, development, deployment, monitoring.
  • Data processing = collection + preparation (preprocessing and feature engineering).

Trap: Training is not the first step. Business goal identification comes first and is the most important phase.

From the AWS docs (8)
  • The Well-Architected ML lifecycle has six phases: business goal identification, ML problem framing, data processing, model development, deployment and monitoring. The phases loop back with feedback instead of running only once.
  • Business goal identification is the most important phase; it starts by deciding whether ML is the right approach at all and whether enough relevant, high-quality training data exists.
  • Some business problems don't need ML because simple business rules do a better job, and others lack enough data for ML to work.
  • Problem framing turns the business problem into an ML problem by deciding what is observed, what to predict (the label or target) and which metric to optimize.
  • Data processing covers data collection and data preparation; preparation includes preprocessing and feature engineering, with exploratory data analysis (EDA) to understand the data and check its quality.
  • The same data processing steps applied to the training data must also be applied to inference requests.
  • Feature engineering selects and transforms variables (features) for a model, through feature creation, transformation, extraction (dimensionality reduction such as PCA) and selection.
  • Model development consists of building, training, tuning and evaluating the model; deployment then serves predictions (inference), and monitoring watches the model in production.

Training, overfitting and underfitting

Training fits a model to data; the real question is whether it works on data it hasn’t seen. A model that memorizes its training set overfits, and one that is too simple underfits.

How it works

  1. Split the data: a training set to learn from, a validation set to tune, and a test (holdout) set kept aside for the final check.
  2. Train with chosen hyperparameters; the model learns its parameters (weights).
  3. Compare training and test scores. Close and high: a good fit. Training high, test low: overfitting. Both low: underfitting.
  4. Fix overfitting with more or augmented data, regularization, early stopping or fewer features. Fix underfitting with a more complex model or more training.
Fit Training score Test score Also called
Underfit Low Low High bias
Good fit High High Balanced
Overfit High Low High variance

Exam cues

  • If the question says great on training data, poor on new data, think overfitting, because the model learned noise.
  • If the question says poor everywhere, think underfitting.
  • If the question asks for settings chosen before training, such as learning rate or epochs, think hyperparameters.

Remember

  • Overfit = high variance; underfit = high bias.
  • The holdout set is never used for training or validation.

Trap: Training longer doesn’t fix overfitting; it usually makes it worse. Early stopping exists to halt training before the model learns noise.

From the AWS docs (9)
  • Training gives an algorithm the training data, an objective metric to optimize and hyperparameters that control the training process.
  • Hyperparameters are settings that control how an algorithm learns, such as learning rate, number of epochs and hidden layers; hyperparameter tuning searches for the best values.
  • Model artifacts are the output of training, typically the trained parameters, a model definition and metadata.
  • In offline evaluation, a holdout set that was never used for training or validation measures the errors of the final model.
  • An overfit model predicts the training data well but performs poorly on new data; it shows high variance. An underfit model is inaccurate on both training and test data; it shows high bias.
  • A large gap between accuracy on the training data and on test data indicates overfitting.
  • Ways to reduce overfitting include early stopping, pruning (feature selection), regularization, ensembling and data augmentation; underfitting comes from a model that is too simple or trained too little.
  • K-fold cross-validation splits the training data into k equal folds, trains on k-1 folds and validates on the remaining one, and repeats k times with a different validation fold.
  • SageMaker AI incremental training starts from an existing model's artifacts and trains a new model on an expanded dataset, saving time and resources when model quality degrades.

Model performance metrics

Model metrics say how well a model predicts; business metrics say whether that matters. For classification, the confusion matrix counts true and false positives and negatives, and precision, recall and F1 come from it.

How it works

  1. Compare predictions against the true labels of a test set.
  2. Count true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN): the confusion matrix.
  3. Compute metrics from those counts.
Metric Formula Use when
Accuracy Correct / all Classes are balanced
Precision TP / (TP + FP) False positives are costly
Recall TP / (TP + FN) False negatives are costly
F1 Harmonic mean of precision and recall You need both
AUC Area under the ROC curve Comparing classifiers across thresholds
RMSE Error size for numbers Regression

Exam cues

  • If the question says missing a positive case is dangerous (fraud, disease), think recall.
  • If the question says false alarms are costly, think precision.
  • If the question says balance both or imbalanced classes, think F1.
  • If the question says cost per user, ROI or customer feedback, think business metrics, not model metrics.
  • If the question says evaluate a model that predicts a number, think MAE, RMSE or R2, not precision or recall.

Remember

  • All of these scores range from 0 to 1; AUC 0.5 is random.
  • Confusion matrix for classification; RMSE for regression.

Trap: High accuracy on imbalanced data proves little; always check precision and recall for the rare class.

From the AWS docs (10)
  • Accuracy is the share of all items that a classification model labels correctly, from 0 to 1.
  • Precision is true positives divided by all positive predictions; it measures how often a positive prediction is right.
  • Recall (sensitivity) is true positives divided by all actual positives; it measures how many of the real positives the model finds.
  • The F1 score is the harmonic mean of precision and recall, from 0 (worst) to 1 (best).
  • AUC is the area under the ROC curve, which plots the true positive rate against the false positive rate across thresholds; 1 is perfect and 0.5 is no better than random.
  • A confusion matrix is a common metric for classification models, and root mean squared error (RMSE) is a common metric for regression models.
  • Business metrics such as fraud detection rate, new business acquisition or customer satisfaction should be tied to the technical metric, so model performance maps to a business outcome.
  • Mean absolute error (MAE) is the average absolute difference between predicted and actual values across all values; it measures a regression model's prediction error, and lower is better.
  • Root mean squared error (RMSE) is the square root of the averaged squared differences between predicted and actual values; it highlights large errors and outliers.
  • R2, the coefficient of determination, measures how much of the variance in the target a regression model explains; higher values mean more explained variability, and values near zero mean the model explains very little.

Flashcards

tap to flip
Answer

·

All 23 cards as a list

Put the Well-Architected ML lifecycle phases in order - deployment, data processing, business goal identification, monitoring, ML problem framing, model development.

Business goal identification, ML problem framing, data processing, model development, deployment, monitoring.

Which phase of the ML lifecycle should decide whether ML is the right approach at all?

Business goal identification, together with problem framing. Some problems are better solved with simple business rules.

A company must always produce the same exact result from a fixed formula, such as a tax calculation. Is ML a good fit?

No. Rules give an exact, repeatable answer; ML gives a prediction that can be wrong.

Use ML when patterns must be learned from data.

A data scientist plots distributions and checks for missing values before training. Which activity is this?

Exploratory data analysis (EDA), part of data preparation in the data processing phase.

A team derives "building age" from a construction date column before training a model. What is this called?

Feature engineering (feature creation), which builds better input variables from raw data.

A model scores well in testing but badly in production, and the team finds the training data was normalized while live requests are not. Which rule was broken?

The same data processing steps used for training must also run on inference requests.

In which ML lifecycle phase do you choose the label to predict and the metric to optimize?

ML problem framing.

A model reaches 99% accuracy on its training data but 70% on new data. What is happening?

Overfitting. The model memorized the training data (high variance) and doesn't generalize.

A model is inaccurate on both the training data and the test data. What is the likely problem?

Underfitting (high bias). The model is too simple or wasn't trained enough to capture the relationship.

Name three techniques that reduce overfitting.

Any three of: more or augmented training data, regularization, early stopping, feature selection (pruning), ensembling.

What is the difference between a model's parameters and its hyperparameters?

Parameters (weights) are learned from the data during training.

Hyperparameters, such as learning rate and epochs, are set before training to control how it learns.

Why should the test (holdout) set never be used during training?

It must measure how the final model does on data it hasn't seen; data used in training would make the score look better than real-world performance.

What does the bias-variance trade-off describe?

Balancing a model that is too simple (high bias, underfits) against one that is too complex (high variance, overfits).

A team has little data and wants a reliable estimate of how its model generalizes. Which validation technique fits?

K-fold cross-validation, which trains and validates k times, each time holding out a different fold.

A fraud model must catch as many real fraud cases as possible, even if some legitimate payments get flagged. Which metric should it maximize?

Recall, the share of actual positives the model finds. Missing fraud (false negatives) is the costly error.

A spam filter must almost never put a real email in the spam folder. Which metric matters most?

Precision, the share of positive (spam) predictions that are correct. False positives are the costly error.

Which metric balances precision and recall in a single number?

The F1 score, their harmonic mean.

Why can accuracy mislead on a dataset where only 1% of transactions are fraud?

A model that always predicts "not fraud" is 99% accurate but useless.

Use precision, recall or F1 for imbalanced classes.

Which metric suits a model that predicts house prices, and which suits one that classifies images?

Regression (prices): RMSE or another error measure.

Classification (images): accuracy, precision, recall, F1, confusion matrix.

Which model metric measures how often positive predictions are correct?

Precision measures the share of positive predictions that are correct.

Besides model metrics such as accuracy, what should an ML project measure to prove value?

Business metrics tied to the goal, such as fraud detection rate, customer satisfaction, cost savings or return on investment.

A model predicts house prices. Which metrics evaluate it: precision and recall, or MAE, RMSE and R2?

MAE, RMSE and R2, because predicting a price is regression. Precision, recall, F1 and the confusion matrix are for classification.

Why should an ML team consider business metrics as well as model performance metrics?

Business metrics connect technical model performance to outcomes such as customer satisfaction.

You finished every topic of AIF-C01!

Your badge turned gold. Next step: book the exam on AWS. Once you pass, add yourself to Who passed.

Share

Share that you finished this certification

I finished every topic of AWS Certified AI Practitioner (AIF-C01) on Easy Review. Come study too!