Skip to content

fix(dvcr): avoid software GOST TLS for DVCR import/upload#2552

Merged
universal-itengineer merged 8 commits into
mainfrom
fix/dvcr-importer-uncompressed-layer
Jun 30, 2026
Merged

fix(dvcr): avoid software GOST TLS for DVCR import/upload#2552
universal-itengineer merged 8 commits into
mainfrom
fix/dvcr-importer-uncompressed-layer

Conversation

@universal-itengineer

@universal-itengineer universal-itengineer commented Jun 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

Two changes in images/dvcr-artifact:

  1. Avoid software GOST TLS for importer/uploader → DVCR (main fix). The GOST Go toolchain advertises GOST (Kuznyechik/MGM) TLS 1.3 suites, and DVCR prefers them, so uploads ran through a pure-software cipher capped at a few MB/s. New pkg/gosttls calls tls.SetAllowedTLS13CipherSuites(...) to advertise only AES-GCM/ChaCha20, so DVCR negotiates hardware-accelerated AES. It's behind the dvcr_no_gost_tls build tag (no-op stub otherwise → standard Go / golangci-lint unaffected), enabled for both go builds in werf.inc.yaml.
  2. Uncompressed upload layer. Replaces single-threaded gzip.BestSpeed (stream.NewLayer) with an uncompressed tar layer; disk images barely compress, and consumers read it transparently.

Why do we need it, and what problem does it solve?

DVCR import was ~3–5 MB/s while the network did 73–133 MB/s. CPU profiling showed gost3412128.l at 76% flat CPU (~95% in the GOST seal path): the importer was CPU-bound on software GOST encryption of the upload, not on I/O. Dropping gzip alone didn't help — incompressible images send the same bytes through the cipher.

What is the expected result?

Import/upload runs at AES-NI speed, no longer CPU-bound on GOST. Measured on a cluster (default 750m limit): importer ~3 → ~57 MB/s peak, uploader (1 GB type: Upload) ~5 → ~20.6 MB/s; pprof shows no gost3412128.* frames.

Reproduce: create a ContainerRegistry VirtualImage/VirtualDisk from HTTP (or a type: Upload image) and watch .status.downloadSpeed and importer pod CPU.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: module
type: fix
summary: Speed up DVCR image import and upload by negotiating hardware-accelerated AES-GCM TLS instead of the software GOST cipher and by uploading uncompressed layers.
impact_level: low

@universal-itengineer
fix(dvcr): upload images as uncompressed layers to remove gzip bottle…
9466482 
…neck

The DVCR importer compressed each image into a single gzip layer via
go-containerregistry stream.NewLayer (gzip.BestSpeed, single goroutine).
gzip is CPU-bound and the provisioning pod is limited to 750m CPU, so
imports of large disk images were capped at ~4 MB/s.

Measured on cluster: the same source URL downloads at 133 MB/s while the
importer container sits pegged at exactly its 750m CPU limit, so the
network and DVCR are not the bottleneck. Disk images barely compress
(qcow2 626MB->598Mi, ISO 2.9GB->2.7Gi), so an uncompressed tar layer is
roughly the same size with near-zero CPU.

Replace stream.NewLayer with a custom single-pass streaming uncompressed
layer (application/vnd.docker.image.rootfs.diff.tar). containerd and
CDI/containers-image read uncompressed layers transparently.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer universal-itengineer force-pushed the fix/dvcr-importer-uncompressed-layer branch from 1c27ac7 to 9466482 Compare June 29, 2026 14:11
@universal-itengineer
fix(dvcr): return stream sentinels from uncompressed layer
de003d6 
remote's pusher detects a streaming layer via
errors.Is(err, stream.ErrNotComputed) (pusher.go writeLayer/manifest
paths) and only then routes the upload through its lazy
digest-after-consume path. The uncompressed layer returned its own
sentinel errors, so the pusher treated "value not computed until stream
is consumed" as fatal and the import failed after retries.

Return go-containerregistry's stream.ErrNotComputed (Digest/DiffID/Size)
and stream.ErrConsumed (Compressed/Uncompressed) so the layer is handled
identically to stream.Layer. The unit test now asserts the exported
sentinels, guarding against this regression.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer
chore(dvcr): add temporary pprof server to importer for profiling
9d090ac 
TEMPORARY debug hook (to be removed before merge): start net/http/pprof
on :6060 in the dvcr-importer so a CPU profile can be captured during an
import via kubectl port-forward + go tool pprof. Used to locate the
single-threaded ~1-core bottleneck that caps import throughput at
~4.6 MB/s regardless of the provisioning pod CPU limit.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer
perf(dvcr): force AES-GCM TLS for importer->DVCR upload
6b05d33 
CPU profiling of the importer showed ~95% of CPU spent in GOST
(Kuznyechik/MGM) TLS encryption of the layer upload to the in-cluster
DVCR: gost3412128.l alone was 76% flat. The GOST cipher is pure-software
(no hardware acceleration) and caps upload throughput at a few MB/s on a
single core, which is the real DVCR import bottleneck (gzip was a red
herring: incompressible disk images send ~the same bytes through the
cipher either way).

Pin the destination TLS client to TLS 1.2 with AES-GCM cipher suites so
the AES-NI accelerated cipher is negotiated instead of GOST. TLS 1.3
ignores CipherSuites, so the version is capped for the list to apply.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer
perf(dvcr): drop GOST TLS via toolchain API instead of pinning TLS 1.2
2f1938c 
The deckhouse GOST Go toolchain unconditionally installs GOST TLS 1.3
cipher suites (crypto/tls init -> GOSTInstall), so the importer always
advertises them and the GOST-preferring DVCR selects software GOST,
which caps upload at a few MB/s on one core.

Replace the previous TLS 1.2 + AES-GCM pin with the toolchain's intended
API: SetAllowedTLS13CipherSuites(AES/ChaCha20), keeping TLS 1.3 while
removing GOST from the advertised suites. The call uses a toolchain-only
API, so it is isolated in cmd/dvcr-importer behind the dvcr_no_gost_tls
build tag (added to the importer go build in werf.inc.yaml); standard-Go
builds exclude the file. destRemoteOptions is reverted to its original
TLS config.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer
perf(dvcr): drop GOST TLS for uploader too via shared gosttls package
6c18dae 
The dvcr-uploader pushes to DVCR through the same registry.DataProcessor
as the importer, so it hits the identical software-GOST TLS bottleneck.

Move the TLS 1.3 cipher override out of cmd/dvcr-importer into a shared
pkg/gosttls package (toolchain-only SetAllowedTLS13CipherSuites behind the
dvcr_no_gost_tls build tag, with a no-op stub when the tag is absent) and
blank-import it from both the importer and uploader mains. Add the build
tag to the uploader go build in werf.inc.yaml as well.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer universal-itengineer added this to the v1.9.2 milestone Jun 30, 2026
@universal-itengineer
chore(dvcr): remove temporary pprof hook and stale gost_tls_off.go
76276aa 
Drop the temporary net/http/pprof server from the importer main (used
only to profile the GOST TLS bottleneck), and remove the obsolete
cmd/dvcr-importer/gost_tls_off.go whose logic now lives in the shared
pkg/gosttls package.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer
style(dvcr): fix gci import order in importer/uploader mains
8242d57 
The blank gosttls import must precede the other deckhouse-prefixed
imports (gci alphabetical order within the prefix group). Caught by the
containerized golangci-lint; the macOS run masked it behind the
libnbd/cgo typecheck failure.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer universal-itengineer changed the title fix(dvcr): upload images as uncompressed layers to remove gzip bottle… fix(dvcr): avoid software GOST TLS for DVCR import/upload Jun 30, 2026
@universal-itengineer universal-itengineer marked this pull request as ready for review June 30, 2026 12:16
@universal-itengineer universal-itengineer merged commit e7d7dbe into main Jun 30, 2026
65 of 68 checks passed
@universal-itengineer universal-itengineer deleted the fix/dvcr-importer-uncompressed-layer branch June 30, 2026 13:04
deckhouse-BOaTswain added a commit that referenced this pull request Jun 30, 2026
@deckhouse-BOaTswain @universal-itengineer
Backport: fix(dvcr): avoid software GOST TLS for DVCR import/upload (#…
30aadda 
…2560)

fix(dvcr): avoid software GOST TLS for DVCR import/upload (#2552)

Description
Two changes in images/dvcr-artifact:

Avoid software GOST TLS for importer/uploader → DVCR (main fix). The GOST Go toolchain advertises GOST (Kuznyechik/MGM) TLS 1.3 suites, and DVCR prefers them, so uploads ran through a pure-software cipher capped at a few MB/s. New pkg/gosttls calls tls.SetAllowedTLS13CipherSuites(...) to advertise only AES-GCM/ChaCha20, so DVCR negotiates hardware-accelerated AES. It's behind the dvcr_no_gost_tls build tag (no-op stub otherwise → standard Go / golangci-lint unaffected), enabled for both go builds in werf.inc.yaml.
Uncompressed upload layer. Replaces single-threaded gzip.BestSpeed (stream.NewLayer) with an uncompressed tar layer; disk images barely compress, and consumers read it transparently.

---------

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Co-authored-by: Nikita Korolev <141920865+universal-itengineer@users.noreply.github.com>
@deckhouse-BOaTswain

deckhouse-BOaTswain commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Cherry pick PR 2560 to the branch release-1.9 successful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@danilrwx danilrwx danilrwx approved these changes
@nevermarine nevermarine Awaiting requested review from nevermarine nevermarine is a code owner
@Isteb4k Isteb4k Awaiting requested review from Isteb4k Isteb4k is a code owner
@yaroslavborbat yaroslavborbat Awaiting requested review from yaroslavborbat yaroslavborbat is a code owner
@diafour diafour Awaiting requested review from diafour diafour is a code owner

Assignees

@universal-itengineer universal-itengineer

Labels

status/backport/success

Projects

None yet

Milestone

v1.9.2

Development

Successfully merging this pull request may close these issues.

3 participants

@universal-itengineer @deckhouse-BOaTswain @danilrwx