Deduplicate concurrent pulls for identical model references across pods with node-level singleflight

[xujihui1985]

The current pull deduplication key is based on volumeName and mountID, not on model reference.
For static inline volumes, nodePublishVolumeStaticInlineVolume calls worker.PullModel(..., volumeName, "", reference, modelDir, ...), and Worker.pullModel uses:

inflightKey := fmt.Sprintf("pull-%s/%s", volumeName, mountID)

As a result, if two pods on the same node publish inline volumes with the same model reference but different volume names, they will not share the same in-flight pull.

That means on a single node:
two pods requesting the same model reference can still trigger duplicate pulls if they use different volume identities

there should be 2 level pull concurrency control, singleflight by model reference, semaphore for global download parallelism.

1. Per-reference deduplication with singleflight
   • If multiple requests on the same node pull the same model reference concurrently, only one real pull should run.
   • Other requests should wait on the same in-flight result.
2. Global concurrency limiting with a semaphore
   • If multiple requests pull different model references concurrently, the node should limit how many model pulls can run at once.
   • This should prevent uncontrolled fan-out of network and disk IO.

[aftersnow]

Thanks for raising this. There is already an active PR that addresses the main duplicate-pull problem: #34. It implements digest-based cross-volume pull deduplication so concurrent requests for the same model content on the same node can reuse an existing pull via hardlinks instead of downloading again.

Could you please check whether #34 covers the issue you reported here? There is also an older related issue, #24, which describes the same volumeName/mountID inflight-key limitation.

[xujihui1985]

Thanks for raising this. There is already an active PR that addresses the main duplicate-pull problem: #34. It implements digest-based cross-volume pull deduplication so concurrent requests for the same model content on the same node can reuse an existing pull via hardlinks instead of downloading again.

Could you please check whether #34 covers the issue you reported here? There is also an older related issue, #24, which describes the same volumeName/mountID inflight-key limitation.

Thanks! I have checked #34, this should work, do we need to take second level concurrent control that is a global semaphore to prevent too many model pulls into consideration?

[aftersnow]

Thanks for checking. I agree that #34 covers the duplicate-pull case for the same model content.

The global semaphore is a separate concern. pull_config.concurrency currently controls concurrency inside a single pull, not the number of concurrent model pulls on a node. I think we should track the node-level pull semaphore as a follow-up item, either in this issue or a separate PR.

@rishi-jat, since you were assigned to the older related issue #24, would you be interested in taking this follow-up work?

[rishi-jat]

Yes, I’d be interested in taking the follow-up work for node-level pull concurrency control.

From the discussion, #34 addresses deduplication for identical model content, but it does not limit the number of concurrent model pulls running on a node. A semaphore-based concurrency layer for node-level pull scheduling makes sense to avoid excessive network and disk IO under high parallel workloads.

I’ll investigate the current pull execution flow and propose an implementation approach for node-level pull concurrency limiting as a follow-up to #34.

[imeoer]

I recommend addressing this issue at the layer level rather than the image level.

We can maintain a map like layer digest -> []path to enable layer-level cache reuse:

1. Insert layer digest -> []path in AfterPullLayer hook.
2. Inspect image layers on new pulling, and try create hardlink by layer digest -> []path.
3. If hardlinking fails in step 2, fall back to modctl pull.

This map key / values need also be cleaned up during volume cleanup, when daemonsets are upgraded / restarted, this map can be rebuilt on startup.

For layer-level pull singleinflight, maybe we can keep only one backend.New instance and do singleinflight in modctl. cc @chlins

For layer-level flow control of network and disk I/O, we can implement global rate limiting using BeforePullLayer (we can get layer size in the hook func).

[rishi-jat]

@imeoer @xujihui1985 - #39 PTAL. thanks!