Update icl-jupyterhub image to 0.0.23 for Prefect 3.x

[aregm]

What

Bumps the JupyterHub single-user image to pbchekin/icl-jupyterhub:0.0.23 and fixes everything needed to actually build and roll it out:

• Tag bump to 0.0.23 in terraform/icl/variables.tf (jupyterhub_singleuser_default_image), the docker-icl-jupyterhub workflow default, the Triton image base (FROM), and scripts/etc/kind/images.txt (which still listed 0.0.18).
• Dockerfile build fix: official python:3.12 images no longer bundle setuptools/wheel, so the jupyter_codeserver_proxy step (setup.py bdist_wheel) failed with ModuleNotFoundError: No module named 'setuptools'. Now installed explicitly in the builder stage.
• entrypoint.sh upgrade handling: user home directories are persistent volumes, so existing users kept the old python-3.9 conda env (with a Prefect 2.13-era infractl) forever — the entrypoint only extracted the conda template on first boot. Now, if the expected python-3.12 kernel env is missing, the old prefix is backed up to ~/.conda.bak.<timestamp> (not deleted), the new template is extracted, and .profile is repointed. The kernel display-name fix is also idempotent now instead of first-boot-only.
• ICL_JUPYTERHUB_IMAGE override in kind.sh so a locally built image (e.g. loaded with kind load docker-image) can be deployed without pushing to Docker Hub.
• readme.md: dropped PREFECT_VERSION/MODIN_VERSION/RAY_VERSION build args that the Dockerfile no longer accepts.

Why

The published 0.0.22 image dates from October 2024 and predates the Prefect 3.6 migration: its kernel ships infractl with Prefect 2.13, which cannot talk to the Prefect 3.6.17 server that terraform/icl now deploys. Users following the README example hit griffe/circular-import errors from the stale client, and even redeploying with a new image would not have fixed existing sessions because of the persistent-home issue above.

Notes for reviewers

• 0.0.23 does not exist on Docker Hub yet — the image needs to be built and pushed (workflow docker-icl-jupyterhub.yaml) before/alongside merging, or deployed locally via the new ICL_JUPYTERHUB_IMAGE override.
• The fixed build step was verified in a clean python:3.12-bookworm container; the infractl wheel baked into the image was verified to require prefect>=3.6,<4.
• The GPU profile images (icl-jupyterhub-gpu:0.0.21, Intel/NVIDIA variants) are still on old tags and need the same treatment separately.

🤖 Generated with Claude Code

[aregm]

Pushed a second commit: full migration of the prefect_runtime plugin to Prefect 3.x (75b0978).

After deploying the new image, running the README example surfaced that the plugin still used the Prefect 2 deployment model (deployments.Deployment.build_from_flow, infrastructure.KubernetesJob), all removed in Prefect 3 — importing the default runtime raised PrefectImportError. The earlier partial migration had skipped the affected test modules via collect_ignore_glob to keep CI green.

Key changes:

• Deployments are created via PrefectClient.create_flow/create_deployment targeting the default-pool kubernetes work pool (matching the worker installed by terraform/modules/prefect).
• Code and runtime files are delivered with native pull steps (pull_with_block → run_shell_script → set_working_directory) instead of overriding the job command.
• Per-deployment Kubernetes manifest surgery (customizations, manifest_filter, gpus, prefect_shared_volume_mount) is not expressible in the Prefect 3 work pool model and now raises PrefectRuntimeError with guidance. Follow-up: extend the work pool base job template to restore GPU/shared-volume support and migrate the remote docker builder.
• docker/icl-prefect now builds on the Prefect 3 / Python 3.12 base (Modin/Ray optional, obsolete cryptography pin dropped).

Verified: 44 unit tests pass on Prefect 3.6.17, and both README examples (plain Python program and Prefect flow) run to completion end-to-end on a kind cluster against Prefect server 3.6.17.

Note: pbchekin/icl-prefect:3.6.17-py3.12-icl0.0.4 (the default flow-run image in defaults.py) is not on Docker Hub — the newest published tag is 2.20.16-py3.9. It needs to be built and pushed (there is no CI workflow for this image); until then, build locally and kind load it.

🤖 Generated with Claude Code