wip(vllm-tensorizer): pre-fetch FlashKDA + llm-multimodal for Kimi K3 (vllm-project/vllm#50000) - #195
wip(vllm-tensorizer): pre-fetch FlashKDA + llm-multimodal for Kimi K3 (vllm-project/vllm#50000)#195nehashah-eng wants to merge 3 commits into
Conversation
…vllm-project/vllm#50000) vLLM's still-open Kimi K3 PR adds two dependencies fetched live at build time instead of via a pinned commit like every other external source in this Dockerfile: - cmake/external_projects/flashkda.cmake FetchContent-clones github.com/vllm-project/FlashKDA from inside the vllm-builder CMake configure step. - rust/Cargo.toml adds `llm-multimodal` as a git dependency on github.com/smg-project, fetched by cargo during vllm-rust-builder's build_rust.sh. Add dedicated flashkda-downloader / llm-multimodal-downloader stages (same pattern as flashinfer-downloader et al.), and point the existing builder stages at the pre-fetched sources: FLASH_KDA_SRC_DIR (which flashkda.cmake already supports as a FetchContent override) and a cargo git-source replacement pointing at the local checkout. Untested: PR #50000 is unmerged upstream, so the pinned VLLM_COMMIT here still doesn't reference either dependency. To build against the Kimi K3 branch itself, override VLLM_COMMIT=658f2f56e557e8141f8726b4626c51496d78d914 at build time. Once the PR merges and VLLM_COMMIT is bumped past it, this wiring becomes load-bearing for the default build.
…oject/vllm#50000 head Wires FLASHKDA_COMMIT/LLM_MULTIMODAL_COMMIT through the workflow and adds a third, non-renovate-managed matrix entry pinning vllm-commit to the Kimi K3 PR's current branch head (658f2f56e557e8141f8726b4626c51496d78d914), so CI actually builds against it and exercises the flashkda-downloader/llm-multimodal-downloader wiring end to end. This is genuinely experimental: #50000 is unmerged, open, and marked needs-rebase upstream, so this SHA can go stale (or the build can just fail) at any time. Tag-suffix naturally differs from the production entries since it's derived from vllm-commit, so this pushes a distinctly-tagged image rather than touching the v0.25.1 production tags.
Align the explicit CUTLASS DSL, TVM FFI, Quack, and NIXL pins with the vLLM 0.26.0 dependency set while preserving newer compatible runtime pins.
|
@nehashah-eng Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/30302295970 |
|
@nehashah-eng Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/30302295970 |
|
@nehashah-eng Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/30302295970 |
|
@nehashah-eng Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/30299982407 |
|
@nehashah-eng Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/30299982407 |
2 participants
Why
Was going to add Kimi K3 support to
vllm-tensorizeroff vllm-project/vllm#50000. That PR's description says "this branch has some private dependencies" — turns out that's not literal (verified FlashKDA, llm-multimodal, the DSpark HF model, and thekimi-k3Docker tag are all public), but it does introduce two dependencies that are fetched live over the network from inside the actual compile stage, rather than through this repo's usual pinned-commit*-downloaderstage pattern:cmake/external_projects/flashkda.cmake(new file) doesFetchContent_Declare(flashkda GIT_REPOSITORY https://github.com/vllm-project/FlashKDA.git ...)— fired from insidevllm-builder's CMake configure step.rust/Cargo.tomladdsllm-multimodalas a git dependency ongithub.com/smg-project— fetched bycargoinsidevllm-rust-builder'sbuild_rust.sh.Every other external source in this Dockerfile (flashinfer, LMCache, InfiniStore, DeepGEMM, NIXL) is instead cloned in its own dedicated
alpine/git-based downloader stage with an explicit pinnedARG *_COMMIT, then bind-mounted into the compile stage. This PR brings FlashKDA and llm-multimodal into that same pattern.What
flashkda-downloaderstage;vllm-buildernow bind-mounts it and setsFLASH_KDA_SRC_DIR, whichflashkda.cmakealready honors as aFetchContentoverride.llm-multimodal-downloaderstage;vllm-rust-buildernow bind-mounts it and adds a cargo git-source replacement (file:///opt/llm-multimodal-src) socargo buildresolves it locally instead of hittinggithub.com/smg-project..github/configurations/vllm-tensorizer.ymlpinsvllm-commitdirectly to [New model] Kimi K3 vllm-project/vllm#50000's current branch head (658f2f56e557e8141f8726b4626c51496d78d914), with its ownflashkda-commit/llm-multimodal-commit, so CI actually exercises this end to end instead of just adding unused stages.Status: experimental, unmerged upstream
vllm-project/vllm#50000 is unmerged (open,
needs-rebase, dirty mergeable_state). The experimental matrix entry pins to its current head SHA, which can go stale (force-push/rebase) or simply fail to build at any time — that's expected for now. The two existing production entries (v0.25.1, renovate-managed) are untouched and get a distinct tag automatically (tag-suffix is derived fromvllm-commit), so this won't overwrite or affect the productionvllm-tensorizertags.Heads up: pushing this triggers real CI on shared self-hosted (
ml-amd/ml-arm) GPU runners for all three matrix entries, including this new experimental one — expect a long build, and a real chance it fails outright since #50000 is WIP/unreviewed upstream.Test plan
flashkda-downloader/llm-multimodal-downloader/vllm-rust-builder/vllm-buildervllm-commitpast it instead