feat: add workflow benchmarks and cloud orchestration support for: aws, azure, gcp - #303

Open

userlaurin wants to merge 267 commits into

spcl:masterfrom

userlaurin:feature/workflows

This pull request is big! We're only showing the most recent 250 commits

Commits on Mar 18, 2026

wrappers - run_sync for storage.py

ldzgch (MacOS)
authored and
laurin
committed
214c947
nosql wrapper uses run_sync

ldzgch
authored and
laurin
committed
b8f7c5c
cloudflare nodejs wrapper without r2 as fs polyfill, just node_compat modules
laurin
committed
dba2992
cleanup nodejs deployment cloudflare, no uploading files necessary anymore
laurin
committed
5390021
add folder structure to python code package

ldzgch (MacOS)
authored and
laurin
committed
24497a2
nosql wrapper - duarble object - may work

ldzgch (MacOS)
authored and
laurin
committed
8812708
fix python. 110 runs for me.

ldzgch (MacOS)
authored and
laurin
committed
5b3d784
make it read the requirements.txt when it has a number

ldzgch (MacOS)
authored and
laurin
committed
cd183b8
durable objects compatibility for nodejs
laurin
committed
9379f39
asyncified the function calls...
laurin
committed
5f9ad9c
fix python vendored modules

ldzgch
authored and
laurin
committed
92db5ae
added request polyfill for benchmark 120
laurin
committed
51892b0
fixed r2 usage for 120, 311
laurin
committed
3235d3f
support for cloudflare containers (python and nodejs), container worker orchestrator, worker orchestrator proxy for r2 and durable objects to be used in container
laurin
committed
416b67b
bigger container for python containers
laurin
committed
5284880
sleep delay longer
laurin
committed
b6de39b
request_id has to be string
laurin
committed
812f592
update container fixed
laurin
committed
9229f9f
fixed benchmark wrapper request ids for experiment results as well as begin/end

=Laurin
authored and
laurin
committed
5899d87
extract memory correctly

=Laurin
authored and
laurin
committed
6e0cd2b
pyiodide does not support resource module for memory measurement

=Laurin
authored and
laurin
committed
3cd741f
timing fix for cloudflare handler

=Laurin
authored and
laurin
committed
2615a36
fixed python timing issue

=Laurin
authored and
laurin
committed
e69243a
removed faulty nodejs implementations of 000 bmks
laurin
committed
f39aad0
removed unnecessary logging
laurin
committed
e76f846
removed experiments.json and package*.json
laurin
committed
dc2f6ed
updated cloudflare readme to reflect final changes
laurin
committed
437cc97
has platform check according to convention, durable object items removal suggestion, function deserialize suggestion
laurin
committed
1eb375c
updated readme to document the correct return object structure by the bmk wrappers
laurin
committed
6c0768e
documented cold start tracking limitation
laurin
committed
0dfcfa8
removed unreachable return statement in cloudflare.py
laurin
committed
b2465f9
small fix to use public property
laurin
committed
0eb4d0b
small fix for public field in durable objects
laurin
committed
db84f2d
converted nosql client calls to async and removed the corresponding post processing in the build script
laurin
committed
7e2d8ac
Fix instance variable naming in nosql_do class

ldzgch
authored and
laurin
committed
35a556d
Rename class instance reference from nosql to nosql_kv

ldzgch
authored and
laurin
committed
92c5dea
Apply suggestions from code review - storage.py

authored and
laurin
committed
bcd5ecb
Apply suggestions from code review

authored and
laurin
committed
96ac2c1
config placeholder for api tokens, r2 etc
laurin
committed
b028151
variable base image in docker file... have to replace the right image varibale at compile time, as wrangler does not provide docker build args
laurin
committed
03e274e
copy and execute init.sh file from benchmark directory, and execute in dockerfile to download and link static binary dependencies
laurin
committed
a11236a
add to existing markdown for cloudflare specific documentation instead of readme
laurin
committed
35755d6
more detail about download_metrics()
laurin
committed
b427c5b
Docker build container for build orchestration locally
laurin
committed
734eadf
using docker client to build local image
laurin
committed
5ffcb06
do not create library trigger
laurin
committed
4da0c31
removed some deprecated logging, throw exception if cold start is used
laurin
committed
4874794
split the cloudflare.py into containers.py + workers.py... each handling their own package & deployment process orchestrated by main entry point cloudflare.py
laurin
committed
6b8e695
use toml writer, cleaner approach to write wrangler.toml
laurin
committed
865ca06
accidental capitalization of the table name resulting in errors in 130
laurin
committed
20eb8db
warning that locationHint is not supported
laurin
committed
ebe0794
s3 client for r2 storage code duplication removed. and also removed librarytrigger
laurin
committed
9dd0a6e

Commits on Mar 26, 2026

feat(cloudflare): Introduce KVStore for NoSQL storage and remove Durable Objects
laurin
committed
b4f08a1
refactor(nosql_kv): simplify index check by removing redundant conditions
laurin
committed
fa7b2f5
feat(cloudflare): Enhance HTTP invocation with User-Agent header and error handling for Cloudflare http invocation protection
laurin
committed
1311f20
feat(cloudflare): Add support for benchmark validation based on language and deployment type
laurin
committed
02cb35a
fix(cloudflare): Update deployment logic for Python workers and ensure package directory creation
laurin
committed
60aa631

Commits on Mar 29, 2026

refactor(cloudflare): abstract nodejs worker build into Dockerfile.worker
laurin
committed
cf9d333

Commits on Apr 13, 2026

fix(cloudflare): Implement health check pings to keep container warm during provisioning
laurin
committed
cf5a547
feat(cloudflare): Enhance language configuration with variant support for Cloudflare deployments
laurin
committed
91bb9a1
feat(cloudflare): Implement multipart upload support for R2 storage in Python and Node.js workers
laurin
committed
8ea37c5
feat(cloudflare): Enable parallel downloads in download_directory method for improved performance
laurin
committed
a60e5d4
fix(regression): Update benchmark filtering to use test_benchmark for accuracy
laurin
committed
18070f0
fix(cloudflare): Enhance error handling for container provisioning in HTTPTrigger
laurin
committed
044b9ef

Commits on Apr 16, 2026

fix(regression): Update benchmark filtering logic to correctly extract benchmark IDs
laurin
committed
7ac2b8c
fix(config): Correct cloudflare worker variant to use default configuration
laurin
committed
78b2979

Commits on Apr 17, 2026

fix(cloudflare): Remove hardcoded language-variant from Cloudflare test configurations and enhance deployment type handling
laurin
committed
6d7e4d0
fix(cloudflare): Improve handling of Cloudflare error code 1042 for worker readiness and CPU time limit
laurin
committed
2cc8f93
fix(cloudflare): Enhance content type inference and improve upload key handling in storage modules
laurin
committed
3f8e69c
fix(cloudflare): Update storage documentation to clarify container upload behavior and KVStore integration
laurin
committed
c8ca384
fix(cli): Add deployment type option to regression command for better test targeting
laurin
committed
b71e2c8
fix(cloudflare): Pin workers-py to version 1.8.0 to avoid broken import issues
laurin
committed
53a1cd2

Commits on Apr 20, 2026

fix(cloudflare): Correct typo in typename and implement download method for R2 with error handling
laurin
committed
32debbf
fix(docs): Update comments to clarify Cloudflare Workers differences for Node.js and Python implementations
laurin
committed
e22bb62
fix(cloudflare): Improve request handling and error responses in container server
laurin
committed
acf2e33
fix(cloudflare): Enhance debugging by utilizing Node.js debuglog for response logging
laurin
committed
e4b2abf
fix(nosql): Update documentation and streamline query method by removing debug logs
laurin
committed
28d90d7
refactor(docker): Moved Dockerfiles for Node.js and Python functions
laurin
committed
7517eaa
feat(docker): Add Dockerfile.build for Python worker validation and update references in deployment logic
laurin
committed
6eae47c
docs(storage): Clarify proxy usage for Cloudflare R2 and explain storage wrapper limitations
laurin
committed
6a3a8db
fix(storage): Enhance debugging by replacing console logs with debuglog for R2 operations
laurin
committed
95fdaba
docs(cloudflare): Add deployment architecture and detailed flow for script-based and container-based Workers
laurin
committed
6b9434e
refactor(handler): Move all imports to the top.
laurin
committed
8966909
refactor(storage): Remove content type inference from upload functions and update documentation
laurin
committed
3665df8

Commits on Apr 22, 2026

docs(build): Enhance documentation for build process and clarify Workers compatibility
laurin
committed
2dafdf9
feat(handler): Implement advanceWorkersClock function to manage timing in Cloudflare Workers
laurin
committed
eb21ce5
docs(nosql): Add clarification on resource access in Cloudflare Workers
laurin
committed
0be8706
docs(nosql): Update module description to clarify HTTP POST operations and server-side implementation
laurin
committed
5908cef
refactor(storage): Simplify list URL request by removing unnecessary User-Agent header
laurin
committed
2040294
refactor(handler): Remove unused functions and clean up code in handler.py
laurin
committed
6d8a1c6
docs(worker): Update comments to clarify the purpose and usage of worker.js
laurin
committed
6a5ef6b
docs(benchmark): Enhance comments to clarify handling of worker.js and variant types
laurin
committed
9ed8f9c
refactor(cli): Update deployment type options in regression command
laurin
committed
26ea601
feat(dependencies): Add patch-ng and tomli dependencies for compatibility
laurin
committed
f14bce9
feat(cloudflare): changes in workers.py including moving the templates into the cloudflare directory, defining the pyodide packages in a seperate file and better handling the requirements in python…
laurin
committed
5a1fdcc
refactor(regression): streamline Cloudflare benchmark configurations and update input size handling
laurin
committed
bcd3dae
refactor(nosql): simplify constructor by removing unnecessary comment
laurin
committed
4f7a279
fix(docs): update Wrangler template paths to reflect new directory structure
laurin
committed
8acec87
refactor(cli): streamline Dockerfile path resolution using get_resource_path
laurin
committed
812fe72
refactor(containers): simplify Dockerfile path resolution using get_resource_path + template path directory update
laurin
committed
d8f08db
docs(cloudflare): enhance authentication section with detailed API token and legacy key usage instructions
laurin
committed
97bb674
refactor(containers): update Cloudflare CLI initialization and improve R2 bucket binding error handling
laurin
committed
e69c836
docs(kvstore): enhance KVStore class documentation with detailed namespace and key mapping explanations
laurin
committed
42eef0e
refactor(r2): improve error handling and logging in list_bucket method
laurin
committed
4a7f036
refactor(containers): update Cloudflare CLI initialization to use constructor directly
laurin
committed
cd1def7
feat(cloudflare): enhance variant selection logic for Cloudflare deployments
laurin
committed
9b9894a

Commits on Apr 23, 2026

refactor(cloudflare): restructure Dockerfile.build for Node.js and Python, add installer scripts
laurin
committed
6d94293
refactor(cloudflare): update base images for Python and Node.js to use slim variants
laurin
committed
31624ec
refactor(cloudflare): enhance Dockerfile.build for Node.js and Python with gosu installation and user privileges
laurin
committed
0253844
refactor(cloudflare): simplify Docker image handling and improve singleton pattern in CloudflareCLI
laurin
committed
b84c330
refactor(cloudflare): update Docker socket mount comment for clarity and remove unused docker_build method
laurin
committed
b7ee302
refactor(cloudflare): enhance container image handling and add worker readiness check
laurin
committed
8bc3595

Commits on Apr 24, 2026

Commits on Apr 25, 2026

refactor(cli): enhance Docker image handling and add containers_push method for Cloudflare registry
laurin
committed
360b2a2
refactor(cloudflare): adjust wrangler.toml generation order to utilize container registry URI
laurin
committed
51f1c3b
refactor(cloudflare): simplify worker URL generation and update logging level for configuration updates
laurin
committed
76d5d17
fix(cloudflare): raise error for missing workers.dev subdomain instead of logging warning
laurin
committed
4af9a0f

Commits on Apr 26, 2026

fix(cli): ensure @cloudflare/container npm dependency is installed before deploying with wrangler
laurin
committed
d6baba3
fix(containers): update image tag generation to use timestamp for versioning
laurin
committed
cec0f9d
refactor(cloudflare): enhance container deployment flow with local image build and registry push details
laurin
committed
89270cf
merged new commits into fork
laurin
committed
cfb935c
Merge branch 'spcl-master'
laurin
committed
ff45c7a
refactor: formatted using black
laurin
committed
fad8b40

Commits on Apr 27, 2026

fix(handler): update return structure to include result object
laurin
committed
fbe8185
fix(handler): update log_data structure to use 'result' key instead of 'output'
laurin
committed
bf1b0ac
fix(build): extend node built-ins filter to include 'constants' and handle 'graceful-fs' redirection
laurin
committed
584a3f8
fix(workers): streamline package code logic and ensure proper handling of requirements
laurin
committed
bd2111d
fix(storage): simplify data handling in aupload_stream by removing base64 conversion
laurin
committed
c476ac8
fix(benchmarks): update subproject commit reference to latest version
laurin
committed
b9eb11f
fix(cli): add 'cloudflare' option to deployment platforms in CLI commands
laurin
committed
e34c67d
fix(cloudflare): enhance CLI container management for thread safety and simplify shutdown process
laurin
committed
cd5992b
fix(cloudflare): add manage image to systems.json
laurin
committed
6500fd2
fix(docs): clarify terminology for Cloudflare Workers in platforms.md
laurin
committed
16e7454
fix(docs): add wall-clock timing explanation for Cloudflare Workers in platforms.md
laurin
committed
d3e6568
refactor(cloudflare): remove container warm-up logic from Cloudflare class
laurin
committed
9681b6e
refactor: linting with flake8
laurin
committed
4d2db1f
refactor(cloudflare): update type hints and improve error handling in various modules
laurin
committed
eda4d0f
feat(storage): add downloadDirectory method to facilitate directory downloads
laurin
committed
305c4fb
black reformat
laurin
committed
1e402e4
refactor(cloudflare): enhance function cache handling and redeployment logic
laurin
committed
9340696
refactor(cloudflare): streamline logging message for worker redeployment
laurin
committed
867f6b7
refactor(mypy): add missing imports configuration for docker module
laurin
committed
c076d79
refactor(cloudflare): update docker client type hints for consistency
laurin
committed
7c8867b
refactor(triggers): increase provisioning retries and wait time for HTTP trigger
laurin
committed
617e7d2
refactor(cli): update Docker image tag handling to include versioning
laurin
committed
88e232b
black
laurin
committed
606e1ac
refactor(cli): add assertion for output type in command execution
laurin
committed
c123da6
refactor(cloudflare): enhance docstrings for clarity and consistency across modules
laurin
committed
c9c5372

Commits on Apr 29, 2026

refactor(health-check): enhance comments for clarity on health check endpoint and provisioning logic
laurin
committed
2053171

Commits on Apr 30, 2026

Merge branch 'master' of upstream — integrate GCP gen2 support
laurin
committed
50ca4d6
refactor(cloudflare): replace container_deployment with system_variant for clarity
laurin
committed
a083ef3

Commits on May 6, 2026

refactor(cloudflare): update deployment types to reflect correct options for Cloudflare
laurin
committed
e1c937a

Commits on May 7, 2026

refactor(cloudflare): enhance container deployment logic and readiness checks
laurin
committed
e9f95b7

Commits on May 8, 2026

feat(cloudflare): add compression benchmark using fflate for Cloudflare Workers
laurin
committed
960d27f
refactor(cloudflare): black formatting
laurin
committed
f6c5d2a
refactor(cloudflare): shorten too long lines
laurin
committed
1c66b92
Merge branch 'spcl:master' into master
userlaurin
authored
078e25f

Commits on May 11, 2026

feat(cloudflare): add CI support for cloudflare. unified the config to all the other platforms. removed specific cloudflare config.
laurin
committed
ed3f23b
fix(cloudflare): update Python version to 3.12 in regression tests
laurin
committed
c60557e
feat(cloudflare): add system_variant configuration for Cloudflare test sequences
laurin
committed
3fcfb19
feat(cloudflare): add max_instances configuration for Cloudflare container deployments
laurin
committed
6d27ff1

Commits on May 25, 2026

[workflows] Add FSM state machine abstraction for workflow definitions

committed
0cabc2f
[workflows] Add Workflow base class and get_workflow to FaaS abstractions

committed
c4400c6
[workflows] Add workflow build support, Redis helpers, and deployment config

laurin
and
mcopik
committed
859e71f
[workflows] Add AWS Step Functions workflow implementation

committed
a87ab07
[workflows] Add GCP Workflows workflow implementation

committed
5525b1d
[workflows] Add AWS and GCP workflow handler wrappers

committed
cc117ac
[workflows] Add Azure Durable Functions workflow handler wrappers

laurin
and
larissaschmid
committed
82991ea
[workflows] Add workflow CLI command and documentation

laurin
and
mcopik
committed
1314695
[workflows] Add workflow benchmark definitions (610.gen, 620-631, 640, 650, 660, 680, 690, 1000-genome, trip-booking)

committed
8fb0ce7
[workflows] Update configuration and enhance AWS workflow handlers
laurin
committed
94adcde
fix: Ensure newline at end of file in requirements.txt for auth and excamera workflows
laurin
committed
d3dd732
[storage] Add list_directory method to various storage wrappers
laurin
committed
af03f53
fix: Move workflow retrieval to correct position in workflow function
laurin
committed
9a72eef
fix: Enhance benchmark data loading and ensure newline in package outputs
laurin
committed
340baf5
fix: Ensure directory creation for data path in generate_input function
=Laurin
committed
0ce3240
feat: Add workflow benchmarks and regression testing framework for workflows
laurin
committed
3a0c994
Merge branch 'workflows-upstream-pr' of github.com:userlaurin/serverless-benchmarks-cloudflare into workflows-upstream-pr
laurin
committed
a2907e8
fix: Update requirements to specify versions for numpy and scikit-learn
=Laurin
committed
51c0cd5
fix: Add missing newline for numpy version specification in requirements.txt
=Laurin
committed
68271f2

Commits on Jun 1, 2026

feat: Replace fsm.py symlink with full implementation

committed
28e8ac9
feat: Add Azure workflow benchmarks and testing suite
laurin
committed
9a6d2a8
feat: Add selfish detour benchmark implementation in C and corresponding packaging script
=Laurin
committed
33968e9
feat: Update benchmark file handling to include additional file types and remove package script check
=Laurin
committed
edfc59e

Commits on Jun 3, 2026

refactor: Simplify function import and Redis connection handling in handler_workflow.py
laurin
committed
1ebea93
refactor: Update import handling and streamline request ID assignment in Azure workflow files
laurin
committed
0dd17a3
refactor: Improve Redis connection handling and key generation in workflow handler
laurin
committed
9b8b3e4
feat: Add initialization for NoSQL and storage instances based on environment variables
laurin
committed
7e7ce46
refactor: Enhance upload functionality to support optional unique naming for files
laurin
committed
0f872f8
refactor: Update deployment configuration to include additional handler files and package dependencies
laurin
committed
eb7da3c
feat: Implement Azure Durable Functions workflow packaging and deployment
laurin
committed
bcccf45
feat: Add Redis configuration properties for workflow measurements
laurin
committed
56d534e
fix: Update resource group initialization to handle missing key gracefully
laurin
committed
a2cb740
feat: Add AzureWorkflow class for Durable Functions implementation
laurin
committed
5e33a6b

Commits on Jun 7, 2026

feat: Update requirements and enhance AWS storage functionality with new download method
=Laurin
committed
53a7323
feat: Update workflow definitions and enhance state management with common parameters
=Laurin
committed
17eec1e
feat: Enhance GCP workflow support with new functions and timeout configuration
laurin
committed
272a044
feat: Add download_within_range method to storage classes for partial downloads
=Laurin
committed
36740aa

Commits on Jun 8, 2026

feat: Enhance GCP workflow handling with YAML serialization and execution timeout
laurin
committed
14a9670
feat: Enhance GCP workflow encoding with improved handling for Map and Parallel states
laurin
committed
d1e618e
feat: Update timeout configuration in workflow to 1800 seconds
laurin
committed
5e1fe49

Commits on Jun 10, 2026

feat: Add validation functions for output across multiple workflows
laurin
committed
e1044a5
feat: Update postprocess method to return a single string for final result and enhance async_invoke to parse execution result
laurin
committed
7e7c643

Commits on Jun 18, 2026

Fix Azure workflow regression execution
laurin
committed
4766315
feat: Add google-cloud-workflows dependency to GCP requirements
=Laurin
committed
bc40b40
fix: Revert timeout configuration in workflow to 900 seconds. Set benchmarks-data to the appropriate branch "workflows"
=Laurin
committed
fa0c4e6

Commits on Jun 20, 2026

fix(workflows): harden measurement collection
=Laurin
committed
53decc9

Commits on Jun 21, 2026

workflow: include definitions in package hash
=Laurin
committed
7576828
workflow: persist refreshed cached workflows
=Laurin
committed
000f533
azure: make Durable workflows replay-safe
=Laurin
committed
ce0a36a
gcp: isolate workflow child requests per instance
=Laurin
committed
2000f1f
regression: fail workflow tests on cleanup failure
=Laurin
committed
7954f70
cli: default workflow trigger to library
=Laurin
committed
bacc08f
Fix workflow lint and typing issues
=Laurin
committed
2a16a17

Commits on Jul 13, 2026

Remove Azure Premium workflow support
=Laurin
committed
ae31622

Commits on Jul 17, 2026

Route container storage through Worker bindings
=Laurin
committed
e4a231d
Merge commit 'e4a231d7' into HEAD
=Laurin
committed
5564a5f
feat(cloudflare): route container storage through outbound bindings
=Laurin
committed
fb56847
Merge branch 'master' into feature/workflows
=Laurin
committed
d483aa5
docs(cloudflare): document outbound storage bindings
=Laurin
committed
64c8591
Merge branch 'master' into feature/workflows
=Laurin
committed
2bf8a32

Commits on Aug 16, 2026

[cf] Make sure to raise errorts instead of silent failing
mcopik
committed
e6393da
[cf] Wait for container with timeout
mcopik
committed
c3b574e
[cf] Ensure that 120.uploader fails when data cannot be fetched
mcopik
committed
796b54c
[cf] Remove dead code
mcopik
committed
c2bca2c
[cf] Extra docs
mcopik
committed
68396c5
[cf] Remove R2 access keys entirely - rely for now on REST API
mcopik
committed
e650de6
[cf] Simplify R2 handling and correctly parse output of worker availability
mcopik
committed
e23a26e
Merge remote-tracking branch 'origin/master' into laurin_master
mcopik
committed
ce3345e
[cf] Fix token verification
mcopik
committed
e582e93
[benchmarks] Unify 411 implementation for Cloudflare
mcopik
committed
3f1117f
[system] Add a switch to the default system variant
mcopik
committed
487493e
[cf] Activate Node.js version of 501 benchmark
mcopik
committed
777e79b
[cf] Remove no longer needed example config file
mcopik
committed
676022f
[cf] Update docs
mcopik
committed
f4de61b

Commits on Aug 19, 2026

[cf] Add Cloudflare container instance_type variable to global config with error for too small instance type compatibility
laurin
committed
f9bc6f1
[cf] Add configurable Cloudflare container sleepAfter parameter
laurin
committed
1d34fbc
[cf] Add placement and container config options
laurin
committed
0623b23
Merge branch 'spcl:master' into master
userlaurin
authored
4b7138f
Merge remote-tracking branch 'origin/master' into feature/workflows
laurin
committed
9367718

Commits on Aug 20, 2026

[aws] Fix 690.ml SciPy pin for Python 3.12 AWS builds
laurin
committed
1bd6b3a

Commits on Aug 21, 2026

[lint] black formatted cloudflare.py
laurin
committed
ad783c4
Merge branch 'master' into feature/workflows
laurin
committed
faf6059