High-Performance Batch Image Processor

A production-grade CLI tool built in Go to batch-convert 500,000+ SVG images to optimized WebP format at multiple sizes. Originally developed for VectorIcons.com, a multi-vendor marketplace for vector illustrations and icons.

A Note on Scope

This is a working application, but it was built for one specific production task: backfilling WebP renditions for an existing marketplace. It is not a general-purpose conversion tool, and some pieces will not work outside that environment — the PostgreSQL contributor validation and the AWS role/S3 integration are tied to VectorIcons infrastructure. I left them in deliberately so the code reflects a real production workload rather than a sanitized demo.

Treat this repo as a reference for building high-throughput, concurrent image-conversion pipelines in Go — worker pools, buffered channels, and a multi-stage processing pipeline — rather than as a drop-in tool. If you want to reuse it for your own conversions, see Adapting This Code for Your Own Use below for how to strip out the environment-specific parts.

The Problem

When VectorIcons launched, only PNG conversions were performed at upload time to save on development time. Later, we needed to generate multiple WebP versions retroactively for CDN delivery, previews, and browser compatibility across 500,000+ existing images.

Initial estimates using Node.js suggested 11.5 days of processing time. Go's concurrency model reduced this to 45 minutes - a 16x performance improvement.

---

Technology Stack

• Language: Go 1.22 with goroutines and channels
• Cloud: AWS S3, AWS STS (IAM role assumption)
• Database: PostgreSQL with GORM
• Image Processing: rsvg-convert (SVG→PNG), ffmpeg (PNG→WebP, watermarking)
• Configuration: YAML-based with runtime validation

---

Performance Results

Single-Threaded Baseline

I first built a single-threaded version to establish a baseline, processing 4,500 test files:

Metric	Value
Files processed	4,500 files
Total time	7 minutes 17 seconds
Throughput	10.4 files/sec
Extrapolated for 500K files	13.4 hours

Concurrent Implementation (10 Workers)

After adding Go's worker pool pattern with 10 concurrent goroutines:

Metric	Value
Files processed	4,500 files
Total time	27 seconds
Throughput	166.67 files/sec
Extrapolated for 500K files	50 minutes

Language Comparison

Language	Est. Time per Image	Total Runtime (500K files)	vs Go
Go	~1ms	~45 minutes	1x
Python	~50ms	~41.6 hours	55x slower
Node.js	~200ms	~11.5 days	368x slower

Result: Go's lightweight goroutines and native parallelism delivered a 16x improvement over single-threaded execution.

---

Architecture

The application follows a producer-consumer pattern with dual worker pools, using Go's concurrency primitives (goroutines, channels, and WaitGroups) to achieve high throughput.

---

How It Works

Dual Worker Pool Architecture

The processor uses two independent worker pools to parallelize I/O-bound (downloading) and CPU-bound (processing) operations:

1. Download Workers fetch files from source (S3 or local filesystem) and add them to the Process Queue
2. Process Workers pull from the queue and execute the image transformation pipeline
3. Buffered Channels connect the pools, enabling continuous processing without blocking

Processing Pipeline

For each image, the processor generates multiple sizes (thumbnail: 128px, preview: 512px, watermark: 512px):

1. SVG → PNG Conversion using rsvg-convert at target dimensions
2. Optional Watermarking using ffmpeg's overlay filter for the watermark variant
3. PNG → WebP Conversion using ffmpeg with quality optimization (-q:v 75)
4. Delivery: every WebP is written to the run's output directory (<work_dir>/<uuid>/output; run.sh collects results into ./test/output). S3 upload is optional, controlled by the local flag: when is_local: false, each WebP is also uploaded to the target bucket

Concurrency Model

// Buffered channels for work distribution
DownloadQueue := make(chan ImageFile, len(files))
ProcessQueue  := make(chan ImageFile, len(files))

// Configurable worker pools
for i := 0; i < downloadWorkers; i++ {
    go downloadWorker(DownloadQueue, ProcessQueue)
}
for i := 0; i < processWorkers; i++ {
    go processWorker(ProcessQueue)
}

// Synchronization with WaitGroups
downloadWG.Wait() // Wait for all downloads
close(ProcessQueue)
processWG.Wait()  // Wait for all processing

---

Key Features

• Highly Concurrent: Configurable worker pools optimize for I/O and CPU workloads
• Storage Flexibility: Supports both local filesystem and AWS S3 (via strategy pattern)
• Production-Ready: Database validation, comprehensive logging, automatic cleanup
• Config-Driven: YAML configuration with sensible defaults
• Hardware Acceleration: Optional VideoToolbox support for ffmpeg on macOS
• Modular Design: Interface-based architecture with dependency injection

---

Design Patterns

• Producer-Consumer: Decouples file discovery from processing via buffered channels
• Strategy Pattern: Abstract file service enables runtime switching between local/S3 backends
• Worker Pool: Limits concurrency to prevent resource exhaustion
• Pipeline: Sequential transformation stages (SVG → PNG → WebP) with conditional watermarking

---

Installation & Usage

Prerequisites

• Go 1.22+
• rsvg-convert and ffmpeg on your PATH:

# macOS
brew install librsvg ffmpeg

# Ubuntu
sudo apt-get install librsvg2-bin ffmpeg

• AWS credentials (via ~/.aws or environment) that can assume the IAM role named in config.yml. The setup.sh script creates a suitable role with read/write access to your S3 buckets.
• PostgreSQL reachable with a users table containing the contributor you process. The contributor name passed on the command line is validated against this table at startup.

Environment

The database connection is configured through a .env file in the project root (never commit this file — it is gitignored):

POSTGRES_HOST=your-database-host
POSTGRES_PORT=5432
POSTGRES_USER=your-database-user
POSTGRES_PASS=your-database-password
POSTGRES_DB=your-database-name

If no .env file exists, the same variables are read from the process environment.

Configuration

Copy config-example.yml to config.yml and adjust it. The values that matter most for a local run:

aws_region: us-east-1
role_arn: arn:aws:iam::111111111111:role/svg-webp-app-role  # role your AWS user can assume

# Local mode: read SVGs from local_source instead of S3
is_local: true
upload_to_s3: false
local_source: /path/to/svg/input        # tree of {contributor}/{icons|illustrations}/{familyID}/{setID}/*.svg

# Worker pool configuration
download_worker_pool_size: 5
process_worker_pool_size: 10

# Output sizes
webp_sizes:
  thumbnail: 128
  preview: 512
  watermark: 512

ffmpegPath: /opt/homebrew/bin/ffmpeg    # output of `which ffmpeg`
watermark_path: ./assets/watermark.svg
work_dir: ./tmp/work
use_hardware_acceleration: true          # VideoToolbox on macOS

In local mode, output WebP files are written under <work_dir>/<run-uuid>/output/; when you run through run.sh, the results are copied to ./test/output before the RAM disk is destroyed. The local_target config value is managed internally (it is overwritten at startup) and does not control the output location.

For S3 mode, set is_local: false and configure source_bucket / target_bucket instead of the local paths.

Build

./build.sh            # runs: go build -o image-processor main.go

Run

The recommended entry point is the wrapper script, which builds the binary, mounts a 4GB RAM disk for intermediate files, points work_dir at it, runs the processor, and restores your config afterward:

./run.sh

To run the binary directly:

./image-processor -f config.yml -c contributor-name

Test

go test ./...

The suite is self-contained: database tests verify generated SQL against a dry-run ORM session (no database needed), S3 tests run against an injected mock client (no AWS needed), and the two conversion tests skip automatically when rsvg-convert/ffmpeg are not installed.

---

Project Structure

go-batch-svg-to-webp/
├── main.go                 # CLI entry point
├── src/
│   ├── image-processor/    # Main orchestrator, config, pipeline
│   ├── file-service/       # Storage abstraction (Local/S3)
│   ├── image-file/         # Image metadata parser
│   ├── database/           # PostgreSQL integration
│   ├── models/             # GORM data models
│   └── common/             # Shared utilities
├── docs/                   # Code overview, spec, roadmap
├── build.sh / run.sh       # Build and RAM-disk run wrappers
└── config-example.yml      # Configuration template

---

Adapting This Code for Your Own Use

The concurrency machinery — the worker pools, the buffered channels, and the SVG → PNG → WebP pipeline — is fully generic. What ties this app to its original environment are two integrations: PostgreSQL (used only to validate the contributor name at startup) and AWS (an STS role assumption plus the S3 storage backend). Both can be removed cleanly.

Removing the PostgreSQL dependency

The database is consulted exactly once, in IsValidContributor(), which NewImageProcessor() calls during startup (src/image-processor/main.go). To remove it:

1. Delete the IsValidContributor() method and the call to it in NewImageProcessor(). The contributor name then acts purely as a path prefix with no validation.
2. Delete the src/database/ and src/models/ packages, and remove their imports from src/image-processor/main.go.
3. Delete your .env file — nothing else reads the POSTGRES_* variables.

Removing the AWS dependency

AWS appears in two places: session setup and the S3 storage backend.

1. In NewImageProcessor(), delete the call to SessionWithRole() (and the function itself). This removes the STS role assumption, so role_arn in config.yml becomes unnecessary.
2. In src/file-service/IFileService.go, simplify the NewFileService() factory to always return a LocalFileService. Then delete S3FileService.go, S3FileService_test.go, IS3FileService.go, and S3FileServiceConfig.go.
3. Remove the now-unused AWS imports and run go mod tidy.

What you keep

After both removals, the app reads SVGs from local_source, fans them out across the download and process worker pools, and writes sized WebP files locally — the interesting part of the codebase, with no external services required:

• Run() in src/image-processor/main.go — worker pool orchestration with buffered channels and WaitGroups
• ProcessFile(), ConvertSVGToPNG(), and RunFFmpeg() in src/image-processor/funcs.go — the conversion pipeline
• src/file-service/LocalFileService.go — filesystem walking and transfer
• src/image-file/ — path parsing into structured image metadata

You still need rsvg-convert and ffmpeg installed, since the pipeline shells out to them for the actual rasterization and encoding.

---

License

MIT License. See LICENSE for details.

Disclaimer

This software is provided "as is" without warranty of any kind. You are responsible for testing it in your environment and ensuring it meets your needs.