diff --git a/storage/caios-to-caios/.gitignore b/storage/caios-to-caios/.gitignore new file mode 100644 index 0000000..6b2e0bc --- /dev/null +++ b/storage/caios-to-caios/.gitignore @@ -0,0 +1,3 @@ +# Credentials and rendered manifests — never commit these +caios-keys.env +job.yaml diff --git a/storage/caios-to-caios/README.md b/storage/caios-to-caios/README.md new file mode 100644 index 0000000..866b691 --- /dev/null +++ b/storage/caios-to-caios/README.md @@ -0,0 +1,266 @@ +# CAIOS to CAIOS Cross-Region Transfer + +This solution copies data between two [CoreWeave AI Object Storage (CAIOS)](https://docs.coreweave.com/products/storage/object-storage/about) +buckets that live in **different regions / Availability Zones**, using the +[CoreWeave fork of `s5cmd`](https://github.com/coreweave/s5cmd) run as a +Kubernetes Job. + +Unlike [`caios-to-dfs`](../caios-to-dfs/) (object storage → filesystem) or +[`gcs-caios-copy`](../gcs-caios-copy/) (Google Cloud Storage → CAIOS), both the +source and destination here are the *same* service — CAIOS. The "cross-region" +dimension lives entirely in each bucket's **`LocationConstraint`** (its +Availability Zone), not in the credentials or the endpoint. + +## How CAIOS addressing works (read this first) + +A few facts about CAIOS drive the whole design: + +- **Buckets are AZ-scoped.** A bucket is pinned to one Availability Zone at + creation time via `LocationConstraint` (for example `US-EAST-04A`). You cannot + move a bucket between AZs — you create a second bucket in the target AZ and + copy into it. +- **There is one global endpoint, not one-per-region.** + - `https://cwobject.com` — the primary endpoint, used from **outside** a + CoreWeave cluster (your laptop, CI, etc.). Requires TLS v1.3. + - `http://cwlota.com` — the [LOTA](https://docs.coreweave.com/products/storage/object-storage/improving-performance/about-lota) + endpoint, used from **inside** a CoreWeave cluster. LOTA caches **reads**. + - CAIOS uses **virtual-hosted-style** addressing (`.cwobject.com`), so + the bucket name in the request routes to the correct AZ automatically. +- **One org access key reaches every bucket in the org, in every AZ.** Because + the endpoint is global and access keys are organization-scoped, a single + Access Key ID / Secret Key pair can read the source bucket and write the + destination bucket regardless of which AZs they live in. You do **not** need a + separate key per region. +- **Cross-region writes are supported but slower.** Per the + [object storage best practices](https://docs.coreweave.com/products/storage/object-storage/improving-performance/best-practices#handle-cross-region-writes), + CAIOS supports cross-region writes; LOTA only caches reads, so writing into a + remote-AZ bucket incurs higher latency. +- **Bucket-to-bucket copy is server-side.** `s5cmd cp`/`sync` between two S3 + URLs issues S3 `CopyObject` calls, so the object bytes move **on the CAIOS + backend between AZs** — they do not stream down to the pod and back up. This + is exactly the method CoreWeave documents for + [moving data across regions](https://docs.coreweave.com/support/storage/articles/how-do-i-move-data-across-regions). + +Practical consequences for the design: + +- The transfer pod is a lightweight **control plane** — it drives many + concurrent `CopyObject` requests, but doesn't carry the data. Pod CPU/memory + and node network are not the bottleneck, so there's **no need to shard across + many nodes**; a single pod with a high worker count handles the whole bucket. +- Running it as a Job inside one of your CoreWeave clusters keeps the transfer + in-network and easy to monitor and re-run — but because the copy is + server-side, you could run the same `s5cmd` command from anywhere that has the + credentials (including your laptop). +- `s5cmd` is the CoreWeave-recommended tool here: high default concurrency (256 + workers) and server-side copy make it faster than a stream-through approach. + +## Prerequisites + +- A CoreWeave organization with CAIOS enabled. +- CAIOS available in both the source and destination Availability Zones — see + the [supported AZ list](https://docs.coreweave.com/products/storage/object-storage/buckets/create-bucket). +- A CoreWeave **API access token** *or* a kubeconfig for your org with + `cwobject:createaccesskey` permission (used to mint the CAIOS access key). +- A CoreWeave Kubernetes (CKS) cluster in your org to run the transfer Job. +- Local tooling: `kubectl`, `curl`, `jq`, and the [AWS CLI](https://aws.amazon.com/cli/) + (used to mint the key and create the AZ-pinned buckets). Optionally the + [CoreWeave `s5cmd` fork](https://github.com/coreweave/s5cmd/releases) locally + if you want to test the copy off-cluster before running the Job. + +## Files + +| File | Purpose | +|------|---------| +| `setup-credentials.sh` | Mints a CAIOS access key for your org (from a kubeconfig token) and writes `caios-keys.env`. | +| `create-buckets.sh` | Creates the source and destination buckets, each in its own AZ via `LocationConstraint`. Idempotent. | +| `caios-keys.env.template` | Template for the CAIOS Access Key ID / Secret Key. | +| `job.yaml.template` | Kubernetes Job that downloads the pinned CoreWeave `s5cmd` fork and runs the server-side `sync` from `src` → `dst`. | + +## Setup + +Every step is parameterized with environment variables so you can point it at +your own org, AZs, and bucket names. Set these once in your shell and the rest +of the commands pick them up. + +### 1. Choose your parameters + +```bash +# Source bucket and the AZ it lives in. +# Tip: if a cluster will run the Job, put the source in that cluster's AZ so +# reads are served from the local LOTA cache. +export SRC_BUCKET="" +export SRC_AZ="" # e.g. US-EAST-04A + +# Destination bucket and a different region's AZ. +export DST_BUCKET="" +export DST_AZ="" # e.g. US-WEST-01A +``` + +Pick AZs from the [list of zones that support AI Object Storage](https://docs.coreweave.com/products/storage/object-storage/buckets/create-bucket). + +> **Bucket naming rules:** 3–63 chars, lowercase letters / numbers / hyphens +> only, globally unique, must start and end with a letter or number, and must +> **not** begin with `cw-`, `vip-`, or `log-stitcher-ch-` (these are reserved +> for CoreWeave). + +### 2. Mint a CAIOS access key + +The CAIOS access key is organization-scoped and works for both buckets in both +AZs. You can create one two ways: + +**Option A — Cloud Console (no CLI):** Create an access key in the +[Cloud Console](https://console.coreweave.com/object-storage/access-keys), then +copy `caios-keys.env.template` to `caios-keys.env` and paste in the values. + +**Option B — Script (from a kubeconfig):** If you have a kubeconfig for your org +with `cwobject:createaccesskey` permission: + +```bash +# Point kubectl at your org's kubeconfig first, then: +./setup-credentials.sh +``` + +This calls the CoreWeave API, retrieves a fresh Access Key ID / Secret Key, and +writes them to `caios-keys.env` (mode `600`). **Treat this file like a password +— it is git-ignored by default.** + +Load the keys into your shell for the next step: + +```bash +set -a; source caios-keys.env; set +a +``` + +### 3. Create the two buckets + +```bash +./create-buckets.sh +``` + +This creates `$SRC_BUCKET` in `$SRC_AZ` and `$DST_BUCKET` in `$DST_AZ` using the +AWS CLI against `https://cwobject.com` with virtual-hosted addressing. It is +idempotent — if a bucket already exists and you own it, it is left untouched. + +> **Bucket creation delay:** When you create a bucket with an S3 client, it can +> take ~1 minute to become available due to DNS caching. If a follow-up command +> returns `InvalidRegion ... Region does not match`, wait a minute and retry. + +### 4. (Optional) Verify access locally with s5cmd + +If you installed the [CoreWeave `s5cmd` fork](https://github.com/coreweave/s5cmd/releases) +locally, confirm the key reaches both buckets before deploying the Job: + +```bash +# Lists should succeed (empty is fine) +s5cmd --endpoint-url https://cwobject.com ls "s3://$SRC_BUCKET/" +s5cmd --endpoint-url https://cwobject.com ls "s3://$DST_BUCKET/" +``` + +You can also run the full copy directly from your laptop (it's server-side, so +it works from anywhere with the credentials): + +```bash +s5cmd --endpoint-url https://cwobject.com \ + --numworkers 256 \ + sync "s3://$SRC_BUCKET/*" "s3://$DST_BUCKET/" +``` + +The Job below does the same thing inside your cluster. + +## The transfer workload (Kubernetes Job) + +With the key and buckets in place, the transfer is a single `s5cmd sync` running +as a Kubernetes Job in your CKS cluster. Because S3→S3 copy is **server-side**, +one pod with a high worker count copies the whole bucket — there's no sharding, +manifest, or multi-node fan-out. + +The Job uses a stock `alpine` image and downloads the **pinned CoreWeave `s5cmd` +fork** release at startup (arch auto-detected for x86_64 / aarch64). The fork is +required: it defaults to virtual-host addressing for `cwobject.com` / +`cwlota.com`, which AI Object Storage needs — stock upstream `s5cmd` uses +path-style and won't work. + +### 1. Create the namespace and credentials secret + +```bash +kubectl create namespace data-migration + +# Build the secret straight from caios-keys.env (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) +kubectl -n data-migration create secret generic caios-credentials \ + --from-env-file=caios-keys.env +``` + +### 2. Configure and deploy the Job + +```bash +cp job.yaml.template job.yaml +``` + +Edit `job.yaml` and set: +- `` → your source bucket (`$SRC_BUCKET`) +- `` → your destination bucket (`$DST_BUCKET`) +- (optional) `NUMWORKERS` / `CONCURRENCY` for your object-size mix +- (optional) `S5CMD_TAG` / `S5CMD_VER` to pin a different fork release + +Then deploy: + +```bash +kubectl -n data-migration apply -f job.yaml +``` + +### 3. Monitor + +```bash +kubectl -n data-migration get job caios-xregion-copy +kubectl -n data-migration get pods +kubectl -n data-migration logs -l job-name=caios-xregion-copy -f +``` + +### 4. Verify completion + +`s5cmd sync` only copies objects that are missing or changed at the destination, +so the Job is safely re-runnable — re-applying it resumes where it left off. +Compare the two buckets when it finishes: + +```bash +# Object count + total size on each side (run locally, or with kubectl exec) +s5cmd --endpoint-url https://cwobject.com du "s3://$SRC_BUCKET/*" +s5cmd --endpoint-url https://cwobject.com du "s3://$DST_BUCKET/*" +``` + +### 5. Clean up + +```bash +kubectl -n data-migration delete job caios-xregion-copy +kubectl -n data-migration delete secret caios-credentials +# Optionally: kubectl delete namespace data-migration +``` + +## Performance & tuning notes + +- **`--numworkers`** (global worker pool, default 256) controls how many files + copy concurrently. Raise it for **many small files**. +- **`--concurrency`** (parts per file, default 5) controls multipart parallelism + for a **single large file**. Raise it (and lower `--numworkers`) for a few + large objects. +- **Mixed workloads** — balance both (e.g. `--numworkers 64 --concurrency 8`). +- **Pod resources** — server-side copy keeps the pod light; the defaults + (2 CPU / 4–8 Gi) are generous for the control-plane role. Increase only if you + push `--numworkers` very high. +- See [Optimize s5cmd performance](https://docs.coreweave.com/products/storage/object-storage/migrate-data#optimize-s5cmd-performance). + +> **Fallback — if server-side cross-AZ copy ever errors:** s5cmd S3→S3 copy is +> always server-side. In the unlikely event CAIOS rejects a cross-AZ +> `CopyObject`, switch to rclone with `--s3-no-server-side-copy`, which streams +> the data through the pod instead (see the [`caios-to-dfs`](../caios-to-dfs/) +> sibling for the rclone pattern). + +## Security considerations + +- `caios-keys.env` and any rendered `job.yaml` contain credentials and are + git-ignored. Never commit them. +- `caios-keys.env` is written with `600` permissions; the Job reads credentials + from a Kubernetes Secret, not from the image or args. +- The access key is organization-scoped — anyone holding it can read/write every + bucket in the org. Rotate or delete it from the + [Cloud Console](https://console.coreweave.com/object-storage/access-keys) when + the transfer is complete. diff --git a/storage/caios-to-caios/caios-keys.env.template b/storage/caios-to-caios/caios-keys.env.template new file mode 100644 index 0000000..430d184 --- /dev/null +++ b/storage/caios-to-caios/caios-keys.env.template @@ -0,0 +1,6 @@ +# CAIOS organization access key. +# Copy to caios-keys.env and fill in, OR generate it with ./setup-credentials.sh. +# Organization-scoped: one key works for every bucket in this org, in every AZ. +# Treat the real caios-keys.env like a password — it is git-ignored. +AWS_ACCESS_KEY_ID=KEY_ID_HERE +AWS_SECRET_ACCESS_KEY=KEY_SECRET_HERE diff --git a/storage/caios-to-caios/create-buckets.sh b/storage/caios-to-caios/create-buckets.sh new file mode 100755 index 0000000..f1a8cb7 --- /dev/null +++ b/storage/caios-to-caios/create-buckets.sh @@ -0,0 +1,109 @@ +#!/bin/bash +# +# Create the source and destination CAIOS buckets, each pinned to its own +# Availability Zone via LocationConstraint. Idempotent: a bucket that already +# exists and is owned by you is left untouched. +# +# Prerequisites: +# - AWS CLI installed. +# - CAIOS access key available, either exported in the environment +# (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) or in caios-keys.env, which +# this script will source automatically. +# +# Required configuration (set via environment variables): +# SRC_BUCKET source bucket name (globally unique) +# SRC_AZ source Availability Zone (e.g. US-EAST-04A) +# DST_BUCKET destination bucket name (globally unique) +# DST_AZ dest Availability Zone (a different region's AZ) +# +# See the supported AZ list: +# https://docs.coreweave.com/products/storage/object-storage/buckets/create-bucket +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${SCRIPT_DIR}/caios-keys.env" + +# CAIOS primary endpoint (used from outside a cluster, e.g. your laptop/CI). +ENDPOINT_URL="https://cwobject.com" + +SRC_BUCKET="${SRC_BUCKET:-}" +SRC_AZ="${SRC_AZ:-}" +DST_BUCKET="${DST_BUCKET:-}" +DST_AZ="${DST_AZ:-}" + +RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m' + +if ! command -v aws &> /dev/null; then + echo -e "${RED}Error: aws CLI is not installed or not in PATH${NC}" + exit 1 +fi + +# All four parameters are required — there is no universal default. +missing="" +[ -z "$SRC_BUCKET" ] && missing="$missing SRC_BUCKET" +[ -z "$SRC_AZ" ] && missing="$missing SRC_AZ" +[ -z "$DST_BUCKET" ] && missing="$missing DST_BUCKET" +[ -z "$DST_AZ" ] && missing="$missing DST_AZ" +if [ -n "$missing" ]; then + echo -e "${RED}Error: missing required variable(s):${NC}${missing}" + echo "Set them first, e.g.:" + echo " export SRC_BUCKET=my-source-bucket SRC_AZ=US-EAST-04A" + echo " export DST_BUCKET=my-dest-bucket DST_AZ=US-WEST-01A" + exit 1 +fi + +# Load the access key from caios-keys.env if not already in the environment. +if [ -z "${AWS_ACCESS_KEY_ID:-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY:-}" ]; then + if [ -f "$ENV_FILE" ]; then + set -a; # shellcheck disable=SC1090 + source "$ENV_FILE"; set +a + fi +fi +if [ -z "${AWS_ACCESS_KEY_ID:-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY:-}" ]; then + echo -e "${RED}Error: CAIOS credentials not found.${NC}" + echo "Run ./setup-credentials.sh first, or export AWS_ACCESS_KEY_ID and" + echo "AWS_SECRET_ACCESS_KEY, or create $ENV_FILE from the template." + exit 1 +fi + +# CAIOS requires virtual-hosted-style addressing. The AWS CLI only reads this +# from a config file, so write a throwaway one scoped to this script. +AWS_CONFIG_FILE="$(mktemp)" +export AWS_CONFIG_FILE +cat > "$AWS_CONFIG_FILE" << 'EOF' +[default] +s3 = + addressing_style = virtual +EOF +trap 'rm -f "$AWS_CONFIG_FILE"' EXIT + +# create_bucket +create_bucket() { + local bucket="$1" az="$2" + echo -e "Creating ${YELLOW}${bucket}${NC} in ${YELLOW}${az}${NC}..." + + # Already exists and owned by us? head-bucket returns 0. + if aws s3api head-bucket --bucket "$bucket" --region "$az" \ + --endpoint-url "$ENDPOINT_URL" &> /dev/null; then + echo -e "${GREEN}✓ ${bucket} already exists — skipping${NC}" + return 0 + fi + + aws s3api create-bucket \ + --bucket "$bucket" \ + --region "$az" \ + --create-bucket-configuration "LocationConstraint=${az}" \ + --endpoint-url "$ENDPOINT_URL" + echo -e "${GREEN}✓ Created ${bucket}${NC}" +} + +create_bucket "$SRC_BUCKET" "$SRC_AZ" +create_bucket "$DST_BUCKET" "$DST_AZ" + +echo "" +echo -e "${GREEN}Done.${NC}" +echo " source: ${SRC_BUCKET} (${SRC_AZ})" +echo " destination: ${DST_BUCKET} (${DST_AZ})" +echo "" +echo -e "${YELLOW}Note:${NC} new buckets can take ~1 minute to become listable due to DNS caching." +echo "If a follow-up command returns 'InvalidRegion ... Region does not match', wait and retry." diff --git a/storage/caios-to-caios/job.yaml.template b/storage/caios-to-caios/job.yaml.template new file mode 100644 index 0000000..82e0af4 --- /dev/null +++ b/storage/caios-to-caios/job.yaml.template @@ -0,0 +1,69 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: caios-xregion-copy + namespace: data-migration +spec: + backoffLimit: 6 + ttlSecondsAfterFinished: 345600 # 4 days + template: + spec: + restartPolicy: OnFailure + containers: + - name: s5cmd-copy + image: alpine:3.20 + # s5cmd S3->S3 copy is server-side (CopyObject): bytes move between AZs + # on the backend, not through this pod. Resources can stay modest; the + # pod just drives concurrent copy requests. + resources: + requests: + cpu: "2" + memory: "4Gi" + limits: + memory: "8Gi" + envFrom: + - secretRef: + name: caios-credentials # AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY + env: + # Cross-region copy goes through the primary endpoint (TLS v1.3). LOTA + # (http://cwlota.com) only caches reads and is not used for the copy. + - name: ENDPOINT_URL + value: https://cwobject.com + - name: SRC_BUCKET + value: # UPDATE THIS + - name: DST_BUCKET + value: # UPDATE THIS + # Tune for your object-size mix: many small files -> raise NUMWORKERS; + # few large files -> lower NUMWORKERS and raise CONCURRENCY. + - name: NUMWORKERS + value: "256" + - name: CONCURRENCY + value: "5" + # Pinned CoreWeave s5cmd fork release (defaults to virtual-host + # addressing for cwobject.com / cwlota.com, required by AI Object Storage). + - name: S5CMD_TAG + value: v2.3.0-acb67716 + - name: S5CMD_VER + value: 2.3.0-acb67716 + command: ["/bin/sh", "-c"] + args: + - | + set -eu + apk add --no-cache ca-certificates tar >/dev/null + case "$(uname -m)" in + x86_64) ASSET="Linux-64bit" ;; + aarch64) ASSET="Linux-arm64" ;; + *) echo "unsupported arch: $(uname -m)"; exit 1 ;; + esac + URL="https://github.com/coreweave/s5cmd/releases/download/${S5CMD_TAG}/s5cmd_${S5CMD_VER}_${ASSET}.tar.gz" + echo "[setup] downloading CoreWeave s5cmd fork: ${URL}" + cd /tmp + wget -q "$URL" + tar xzf "s5cmd_${S5CMD_VER}_${ASSET}.tar.gz" s5cmd + chmod +x s5cmd + echo "[copy] ${SRC_BUCKET} -> ${DST_BUCKET} via ${ENDPOINT_URL}" + # sync (not cp) so re-runs are incremental and the Job is resumable. + exec ./s5cmd --endpoint-url "$ENDPOINT_URL" \ + --numworkers "$NUMWORKERS" \ + sync --concurrency "$CONCURRENCY" \ + "s3://${SRC_BUCKET}/*" "s3://${DST_BUCKET}/" diff --git a/storage/caios-to-caios/setup-credentials.sh b/storage/caios-to-caios/setup-credentials.sh new file mode 100755 index 0000000..9ada746 --- /dev/null +++ b/storage/caios-to-caios/setup-credentials.sh @@ -0,0 +1,78 @@ +#!/bin/bash +# +# Mint a CAIOS access key for the current organization and write it to +# caios-keys.env for use by create-buckets.sh and the rclone transfer. +# +# Prerequisites: +# - kubectl configured against a kubeconfig for the org, for a user with +# cwobject:createaccesskey permission. +# - curl and jq installed. +# +# The resulting access key is organization-scoped: it can read and write every +# CAIOS bucket in the org, in every Availability Zone. You only need one. +# +# Alternative: create the key in the Cloud Console +# (https://console.coreweave.com/object-storage/access-keys), then copy +# caios-keys.env.template to caios-keys.env and paste in the values. +set -euo pipefail + +API_ENDPOINT="https://api.coreweave.com/v1/cwobject/access-key" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${SCRIPT_DIR}/caios-keys.env" + +RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m' + +for cmd in kubectl curl jq; do + if ! command -v "$cmd" &> /dev/null; then + echo -e "${RED}Error: $cmd is not installed or not in PATH${NC}" + [ "$cmd" = "jq" ] && echo "Install jq with: brew install jq" + exit 1 + fi +done + +if [ -f "$ENV_FILE" ]; then + echo -e "${RED}Error: $ENV_FILE already exists. Will not overwrite.${NC}" + echo "Delete it first if you want to mint a new key." + exit 1 +fi + +echo "Extracting API token from the current kubeconfig context..." +API_ACCESS_TOKEN="$(kubectl config view --raw -o jsonpath='{.users[0].user.token}')" +if [ -z "$API_ACCESS_TOKEN" ]; then + echo -e "${RED}Error: Could not extract a token from kubeconfig.${NC}" + echo "Make sure kubectl is pointed at the org's kubeconfig." + exit 1 +fi + +echo "Requesting a CAIOS access key from the CoreWeave API..." +RESPONSE="$(curl -s -X POST "$API_ENDPOINT" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer ${API_ACCESS_TOKEN}" \ + -d '{"durationSeconds": 0}')" + +ACCESS_KEY="$(echo "$RESPONSE" | jq -r '.accessKeyId')" +SECRET_KEY="$(echo "$RESPONSE" | jq -r '.secretKey')" + +if [ -z "$ACCESS_KEY" ] || [ "$ACCESS_KEY" = "null" ] || \ + [ -z "$SECRET_KEY" ] || [ "$SECRET_KEY" = "null" ]; then + echo -e "${RED}Error: Failed to obtain credentials from the API.${NC}" + echo "Response: $RESPONSE" + exit 1 +fi + +echo -e "${GREEN}✓ Access key retrieved successfully${NC}" + +cat > "$ENV_FILE" << EOF +# CAIOS organization access key — minted by setup-credentials.sh +# Organization-scoped: works for every bucket in this org, in every AZ. +# Treat this file like a password. It is git-ignored. +AWS_ACCESS_KEY_ID=${ACCESS_KEY} +AWS_SECRET_ACCESS_KEY=${SECRET_KEY} +EOF +chmod 600 "$ENV_FILE" + +echo -e "${GREEN}✓ Wrote ${ENV_FILE}${NC}" +echo "" +echo "Next steps:" +echo -e " ${YELLOW}set -a; source ${ENV_FILE}; set +a${NC} # load the key into your shell" +echo -e " ${YELLOW}./create-buckets.sh${NC} # create the source + destination buckets"