YouTube Data Pipeline

A cloud-native AWS data engineering project that ingests YouTube trending video data, stores raw JSON in S3, cleans it in Glue, and builds a curated analytics layer for reporting and exploration.

Project Goal

This project demonstrates a complete data lake pipeline using:

• AWS Lambda for ingestion
• Amazon S3 for raw and processed storage
• AWS Glue for ETL and transformation
• Glue Data Catalog for metadata management
• Athena / SQL-friendly data access for analytics
• AWS IAM and Step Functions for orchestration and access control

The pipeline is designed to collect trending statistics from the YouTube Data API and convert them into structured, cleaned, and aggregated datasets across Bronze, Silver, and Gold layers.

Architecture

YouTube Data API
        │
        ▼
Lambda: youtube-api-ingestion
        │
        ▼
S3 Bronze Layer
        │
        ▼
AWS Glue: Bronze → Silver Job
        │
        ├── cleaned statistics data
        ├── category/reference data
        └── data quality checks
        │
        ▼
AWS Glue: Silver → Gold Job
        │
        ├── region summary
        ├── daily trending analytics
        └── channel analytics

Repository Structure

.
├── README.md
├── data_quality/
│   └── dq_lambda.py
├── Glue_jobs(ETL job)/
│   ├── yt-data-pipeline-bronze-to-silver-nikhil/
│   │   └── script
│   └── yt-data-pipeline-silver-to-gold-nikhil/
│       └── script
├── iam_permission/
│   ├── yt-data-pipeline-glue-role-nikhil
│   ├── yt-data-pipeline-lambda-role-nikhil
│   └── yt-data-pipeline-stepfunction-role-nikhil
├── lambda/
│   ├── json-to-parquet/
│   │   └── lambda_function.py
│   └── youtube-api-ingestion/
│       └── lambda_function.py
├── scripts/
│   ├── iam-role.sh
│   └── information.sh
├── step_function/
│   └── yt_data_pipeline_orcherstration
└── .gitignore

Layers

1. Bronze Layer

Raw JSON is stored in S3 after ingestion from the YouTube API. Files are organized by region and time partition, for example:

youtube/raw_statistics/region=US/date=2026-08-26/hour=14/<timestamp>.json

2. Silver Layer

The Bronze-to-Silver Glue job reads the raw records, explodes the API response items, cleans and validates the data, and writes Parquet data to the Silver layer.

The Silver dataset includes:

• video metadata
• channel information
• region and date
• engagement metrics
• derived columns such as like ratio and engagement rate

3. Gold Layer

The Silver-to-Gold Glue job reads the cleaned statistics and builds analytical aggregates such as:

• region-wise summaries
• trending-date reports
• channel performance summaries

Key Components

Lambda: YouTube API Ingestion

Location:

• lambda/youtube-api-ingestion/lambda_function.py

This Lambda:

• calls the YouTube Data API v3
• retrieves trending videos for configured regions
• pulls category reference data
• stores JSON payloads in the Bronze bucket
• sends an SNS alert on partial failures

Lambda: JSON to Parquet

Location:

• lambda/json-to-parquet/lambda_function.py

This Lambda:

• reads JSON reference files from S3
• normalizes nested JSON structures
• cleans category data
• writes transformed Parquet data to the Silver layer

Glue Job: Bronze → Silver

Location:

• Glue_jobs(ETL job)/yt-data-pipeline-bronze-to-silver-nikhil/script

This script:

• reads raw JSON from Bronze S3
• validates required fields
• filters bad records
• standardizes region values
• removes duplicate video records
• calculates derived metrics
• writes Parquet to the Silver bucket and Glue Catalog

Glue Job: Silver → Gold

Location:

• Glue_jobs(ETL job)/yt-data-pipeline-silver-to-gold-nikhil/script

This script performs aggregate transformations for reporting and analytical consumption.

Data Quality Layer

Location:

• data_quality/dq_lambda.py

This handles data quality checks and alerting to support pipeline reliability.

AWS Resources Used

• Amazon S3
• AWS Lambda
• AWS Glue
• AWS Step Functions
• Amazon SNS
• AWS IAM
• AWS Athena (for querying curated data)

Environment Variables

Ingestion Lambda

• YOUTUBE_API_KEY
• S3_BUCKET_BRONZE
• YOUTUBE_REGIONS
• SNS_ALERT_TOPIC_ARN

Reference JSON Lambda

• S3_BUCKET_SILVER
• GLUE_DB_SILVER
• GLUE_TABLE_REFERENCE
• SNS_ALERT_TOPIC_ARN

Glue Job Parameters

The Bronze-to-Silver job uses parameters such as:

• --JOB_NAME
• --bronze_bucket
• --bronze_database
• --bronze_table
• --silver_bucket
• --silver_database
• --silver_table

The Silver-to-Gold job uses parameters such as:

• --JOB_NAME
• --silver_database
• --gold_bucket
• --gold_database

Deployment Workflow

1. Create IAM roles from the files in iam_permission/.
2. Deploy the Lambda functions from the lambda/ directory.
3. Configure the Bronze and Silver S3 buckets.
4. Run the Bronze-to-Silver Glue job.
5. Validate the Silver table and schema.
6. Run the Silver-to-Gold Glue job.
7. Use Athena or downstream reporting tools for analytics.

Notes

• This project is built around a practical AWS data engineering workflow using live YouTube data.
• Several fixes were applied in the Glue scripts for issues such as missing args, bucket hardcoding, incorrect columns, and partition mismatch.
• The pipeline uses clean partitioning and catalog metadata to make data easier to query and maintain.

Future Improvements

• Add Step Functions scheduling and orchestration
• Add robust error handling and retry logic
• Add automated quality metric reporting
• Integrate with QuickSight or dashboarding tools
• Add CI/CD deployment automation for AWS resources

License

This project is intended for learning, portfolio, and internal engineering use. Update the license if you plan to share it publicly under different terms.