Build a ML Workflow for Scones Unlimited on Amazon SageMaker

Project Summary

This project implements a complete SageMaker image classification workflow for a sample application. It covers dataset staging, SageMaker model training and deployment, Lambda authoring, Step Function orchestration, inference validation, and cleanup of AWS resources.

What was executed

1. Data staging and ETL using the CIFAR-100 dataset.
2. Model training and deployment to a SageMaker real-time inference endpoint.
3. Three AWS Lambda functions to serialize input, invoke the endpoint, and validate inference confidence.
4. An AWS Step Function that orchestrates the lambdas sequentially.
5. Testing with Step Function execution and inference verification.
6. Cleanup of cloud resources after validation.

Data Staging

• Extracted the CIFAR-100 dataset from the public hosting URL.
• Transformed the extracted dataset into usable image tensors and saved intermediate artifacts.
• Loaded the dataset into the SageMaker training workflow.

Model Training and Deployment

• Trained an image classification model using Amazon SageMaker.
• Deployed the trained model to a SageMaker real-time inference endpoint.
• The endpoint name used in the Lambda implementation is image-classification-2025-09-25-11-10-04-957.

Lambda Functions

Three Lambda functions were authored and connected via a Step Function:

1. SerializedImage Lambda

• Downloads an image from S3 using s3_bucket and s3_key values from the Step Function event.
• Reads the image file and converts it to a base64-encoded string.
• Returns a payload containing image_data, s3_bucket, s3_key, and an empty inferences list.

2. ImageClassification Lambda

• Accepts the event payload and extracts the base64 image data.
• Decodes the image and invokes the SageMaker runtime endpoint image-classification-2025-09-25-11-10-04-957.
• Parses the inference response and attaches the results to the payload under inferences.

3. Inference Lambda

• Reads the inferences array from the incoming event payload.
• Applies a confidence threshold of 0.93.
• Continues the workflow when the highest inference score meets or exceeds the threshold.
• Raises a THRESHOLD_CONFIDENCE_NOT_MET error if no inference surpasses the threshold.

Step Function Workflow

• Orchestrates the three Lambda functions in order:
  1. SerializedImage
  2. ImageClassification
  3. Inference
• Ensures the image is fetched, classified, and validated.
• Supports error handling for low-confidence predictions.

Testing and Evaluation

• Executed the Step Function end to end and validated the state machine execution graph.
• Verified that the workflow successfully passes image data through all three Lambda steps.
• Confirmed the final output contains the inference array and the original event metadata.

Repository Files

• README.md - Project overview and workflow summary.
• starter.ipynb - Notebook containing the data staging, model training, and deployment process.
• lambda_function.py - Lambda implementation for serialization, SageMaker invocation, and confidence filtering.
• execution-detail.json - Execution metadata for the workflow.

Cleanup

• After validation, delete the deployed SageMaker endpoint and workflow resources to avoid ongoing AWS charges.
• Clean up S3 artifacts, Lambda functions, and Step Function definitions if they are no longer needed.