Singularity pulling problems:

[mbelouso]

Hey guys, when running a faMPNN run, when it tried to pull the container I get this error:

ERROR ~ Error executing process > 'RunFAMPNN (1)'

Caused by:
  Failed to pull singularity image
    command: singularity pull  --name ghcr.io-papenfusslab-proteindj-fampnn-v2.2.img.pulling.1783734803555 oras://ghcr.io/papenfusslab/proteindj/fampnn:v2.2 > /dev/null
    status : 255
    hint   : Try and increase singularity.pullTimeout in the config (current is "20m")
    message:
      INFO:    Downloading oras image
      FATAL:   While pulling image from oci registry: error fetching image to cache: unable to Download Image: error writing layer: stream error: stream ID 1; PROTOCOL_ERROR; received from peer

 -- Check '.nextflow.log' file for details

I tried pulling manually:

singularity pull --name ghcr.io-papenfusslab-proteindj-fampnn-v2.2.img.pulling.1783734803555 oras://ghcr.io/papenfusslab/proteindj/fampnn:v2.2

And get the same error:

INFO:    Downloading oras image
4.2GiB / 4.5GiB [=====================================================================>------] 93 % 10.9 MiB/s 30s
FATAL:   While pulling image from oci registry: error fetching image to cache: unable to Download Image: error writing layer: stream error: stream ID 1; PROTOCOL_ERROR; received from peer

Weirdly it worked on my workstation..... So I just copied the image to the server... and it all worked out. Just thought you could look into this and see if it something more general, or just that my server has some major networking issues. Perhaps a script if the users want to predownload all the singularity containers (or update them etc...) might be helpful.

[JoshuaMHardy]

Hi Matt,

I am also having issues with the GitHub Container repository, and I am unable to download some of the containers on our HPC. It looks like an issue with the repository itself; maybe the download is being interrupted due to multiple requests from the same IP address.

I got the same error with the fampnn container, but adding GODEBUG=http2client=0 fixed it for me.

→ apptainer pull oras://ghcr.io/papenfusslab/proteindj/fampnn:v2.2
INFO:    Downloading oras image
176.0MiB / 4.5GiB [===>-------------------------------------------------------------------------------------------------] 4 % 9.7 MiB/s 7m3
178.7MiB / 4.5GiB [===>-------------------------------------------------------------------------------------------------] 4 % 9.8 MiB/s 7m2
180.0MiB / 4.5GiB [===>-------------------------------------------------------------------------------------------------] 4 % 9.8 MiB/s 7m2
180.0MiB / 4.5GiB [===>-------------------------------------------------------------------------------------------------] 4 % 9.7 MiB/s 7m3
1.8GiB / 4.5GiB [=======================================>-----------------------------------------------------------] 40 
1.8GiB / 4.5GiB [=======================================>-----------------------------------------------------------] 40 
1.8GiB / 4.5GiB [=======================================>-----------------------------------------------------------] 40 
1.9GiB / 4.5GiB [====================================>-------------------------------------------------] 43 % 12.5 MiB/s 3m2
1.9GiB / 4.5GiB [====================================>-------------------------------------------------] 43 % 12.5 MiB/s 3m2
1.9GiB / 4.5GiB [====================================>-------------------------------------------------] 43 % 12.5 MiB/s 3m2
4.0GiB / 4.5GiB [===========================================================================>----------] 89 % 12.3 MiB/s 41s
FATAL:   While pulling image from oci registry: error fetching image to cache: unable to Download Image: error writing layer: stream error: stream ID 1; PROTOCOL_ERROR; received from peer
[ble: exit 255][ble: elapsed 331.287s (CPU 12.2%)] apptainer pull oras://ghcr.io/papenfusslab/proteindj/fampnn:v2.2

→ GODEBUG=http2client=0 apptainer pull oras://ghcr.io/papenfusslab/proteindj/fampnn:v2.2
INFO:    Downloading oras image
4.5GiB / 4.5GiB [======================================================================================] 100 % 12.5 MiB/s 0s
[ble: elapsed 556.802s (CPU 11.6%)] GODEBUG=http2client=0 apptainer pull oras://ghcr.io/papenfusslab/proteindj/fampnn:v2.2

With the larger containers, I got an 'unexpected EOF' error, but trying the command again worked:
FATAL: While pulling image from oci registry: error fetching image to cache: unable to Download Image: error writing layer: unexpected EOF

I have made a download script as you suggested, which I'll share here in the meantime while I troubleshoot. It automatically retries the download if interrupted:

#!/bin/bash

# download_containers.sh
# Pre-downloads all ProteinDJ containers to a local Apptainer cache directory.
# This avoids HTTP/2 PROTOCOL_ERROR and unexpected-EOF failures during Nextflow runs.
#
# Usage: ./download_containers.sh [version] [cache_dir]
#
# Arguments (both optional, positional):
#   version    Container version tag (default: extracted from ../nextflow.config)
#   cache_dir  Destination directory  (default: $APPTAINER_CACHEDIR)
#
# Examples:
#   ./download_containers.sh
#   ./download_containers.sh v2.2
#   ./download_containers.sh v2.2 /path/to/cache

set -euo pipefail

MAX_RETRIES=5   # number of attempts to pull each container
RETRY_DELAY=30  # seconds between retries

# ── Paths ────────────────────────────────────────────────────────────────────
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
NEXTFLOW_CONFIG="${SCRIPT_DIR}/../nextflow.config"

# ── Registry ─────────────────────────────────────────────────────────────────
REGISTRY="ghcr.io/papenfusslab/proteindj"

CONTAINERS=(
    "af2"
    "bindcraft"
    "boltz2"
    "dl_binder_design"
    "fampnn"
    "python_tools"
    "pyrosetta_tools"
    "rfdiffusion"
)

# ── Resolve version ───────────────────────────────────────────────────────────
if [[ -n "${1:-}" ]]; then
    VERSION="$1"
else
    if [[ ! -f "$NEXTFLOW_CONFIG" ]]; then
        echo "Error: nextflow.config not found at ${NEXTFLOW_CONFIG}"
        echo "       Provide version as first argument: $0 <version>"
        exit 1
    fi
    VERSION=$(grep -E "^\s*container_version\s*=" "$NEXTFLOW_CONFIG" \
              | head -1 \
              | sed "s/.*=\s*['\"]//; s/['\"].*//")
    if [[ -z "$VERSION" ]]; then
        echo "Error: could not extract container_version from ${NEXTFLOW_CONFIG}"
        exit 1
    fi
fi

# ── Resolve cache directory ───────────────────────────────────────────────────
if [[ -n "${2:-}" ]]; then
    CACHE_DIR="$2"
elif [[ -n "${APPTAINER_CACHEDIR:-}" ]]; then
    CACHE_DIR="$APPTAINER_CACHEDIR"
else
    echo "Error: no cache directory specified and \$APPTAINER_CACHEDIR is not set."
    echo "       Usage: $0 [version] <cache_dir>"
    exit 1
fi

mkdir -p "$CACHE_DIR"

# ── Disable HTTP/2 to avoid PROTOCOL_ERROR with ORAS/OCI registries ──────────
export GODEBUG=http2client=0

# ── Summary ───────────────────────────────────────────────────────────────────
echo "========================================="
echo "Downloading ProteinDJ containers"
echo "  Version   : ${VERSION}"
echo "  Cache dir : ${CACHE_DIR}"
echo "  Registry  : oras://${REGISTRY}"
echo "========================================="
echo ""

TOTAL=${#CONTAINERS[@]}
COUNT=0
SKIPPED=0
FAILED=0

for CONTAINER in "${CONTAINERS[@]}"; do
    COUNT=$((COUNT + 1))

    # Nextflow SingularityCache naming: replace / and : with -
    IMG_NAME="${REGISTRY}/${CONTAINER}:${VERSION}"
    IMG_FILE="${CACHE_DIR}/$(echo "${IMG_NAME}" | tr '/:' '-').img"

    echo "[${COUNT}/${TOTAL}] ${CONTAINER}:${VERSION}"

    if [[ -f "$IMG_FILE" ]]; then
        echo "  ✓ Already cached: ${IMG_FILE}"
        SKIPPED=$((SKIPPED + 1))
        continue
    fi

    echo "  Pulling oras://${IMG_NAME} ..."
    PULL_TMP="${IMG_FILE}.pulling.$$"

    ATTEMPT=0
    SUCCESS=false
    while [[ $ATTEMPT -lt $MAX_RETRIES ]]; do
        ATTEMPT=$((ATTEMPT + 1))
        [[ $ATTEMPT -gt 1 ]] && echo "  Retry ${ATTEMPT}/${MAX_RETRIES} after ${RETRY_DELAY}s..." && sleep "$RETRY_DELAY"

        # Pull directly to target path; apptainer accepts a positional output path
        rm -f "$PULL_TMP"
        if apptainer pull "$PULL_TMP" "oras://${IMG_NAME}" 2>&1 | sed 's/^/    /'; then
            mv "$PULL_TMP" "$IMG_FILE"
            echo "  ✓ Saved to: ${IMG_FILE}"
            SUCCESS=true
            break
        else
            rm -f "$PULL_TMP" 2>/dev/null || true
            echo "  ✗ Attempt ${ATTEMPT} failed"
        fi
    done

    if [[ "$SUCCESS" != true ]]; then
        echo "  ✗ All ${MAX_RETRIES} attempts failed for ${CONTAINER}:${VERSION}"
        FAILED=$((FAILED + 1))
    fi

    echo ""
done

# ── Final report ──────────────────────────────────────────────────────────────
echo "========================================="
echo "Done."
echo "  Downloaded : $((COUNT - SKIPPED - FAILED))"
echo "  Skipped    : ${SKIPPED} (already cached)"
echo "  Failed     : ${FAILED}"
echo "========================================="

[[ $FAILED -eq 0 ]]