Skip to content

Preserve flow ordering with Linux TAP multiqueue - #2609

Open
alatteri wants to merge 1 commit into
zerotier:devfrom
instinctual:agent/linux-tap-multiqueue-flow-ordering
Open

Preserve flow ordering with Linux TAP multiqueue#2609
alatteri wants to merge 1 commit into
zerotier:devfrom
instinctual:agent/linux-tap-multiqueue-flow-ordering

Conversation

@alatteri

Copy link
Copy Markdown

Summary

Use Linux TAP multiqueue when the configured receive concurrency is greater
than one. Each TAP receive worker gets its own /dev/net/tun descriptor instead
of having every worker read the same descriptor.

This change:

  • creates the interface with IFF_MULTI_QUEUE for multi-worker operation;
  • attaches one queue descriptor per TAP receive worker;
  • keeps each descriptor nonblocking and close-on-exec;
  • shuts down the workers before closing their queue descriptors;
  • leaves single-worker operation unchanged; and
  • documents the behavior and existing configuration controls.

Motivation and observed failure mode

The current Linux multicore path starts several receive workers, but all of
them read one TAP descriptor. Packets are dequeued from one FIFO, then handled
by different threads. Adjacent packets from a single high-rate flow can
therefore complete processing in a different order.

This was reproducible with sustained, low-latency SRT/UDP video traffic. A
single TAP reader preserved ordering but could not drain the TAP ring quickly
enough and the interface's tx_dropped counter increased. Multiple readers on
the shared descriptor improved drain rate but introduced packet reordering,
which appeared to SRT as gaps and unnecessary retransmissions.

Linux multiqueue provides the intended middle ground: the kernel assigns a
flow to a queue, each queue has one reader, and unrelated flows can still be
processed in parallel. This preserves ordering within a flow without removing
multicore TAP processing.

Implementation details

When concurrency > 1, the first TUNSETIFF uses IFF_MULTI_QUEUE. Additional
descriptors are opened and attached to the same interface name with the same
flags. Each receive thread selects and reads only its assigned descriptor.

If an additional queue cannot be attached, interface creation fails with the
underlying error rather than silently returning to multiple readers on a
shared FIFO. The existing multicoreEnabled, concurrency, and
cpuPinningEnabled settings remain the only configuration; this introduces no
new setting and makes no wire-protocol, routing, encryption, or controller
change.

Validation

  • Built successfully from the current dev branch with make -j4.
  • Built and ran zerotier-selftest; all tests passed, including 10,000-packet
    UDP and 10 MB TCP transport tests.
  • Built and deployed as a Debian package based on 1.16.2 on Ubuntu 26.04.
  • Verified two TAP queue descriptors with multicoreEnabled=true and
    concurrency=2.
  • During sustained approximately 70 Mb/s SRT video traffic, the ZeroTier TAP
    tx_dropped counter remained unchanged through the observation interval.
  • Verified startup logging reports the active queue count.

Compatibility and scope

  • Linux only.
  • Single-worker behavior is unchanged.
  • Requires Linux TAP multiqueue support when concurrency is greater than one.
  • No new user-facing configuration is required.
  • No ZeroTier packet format or network interoperability changes.

@CLAassistant

CLAassistant commented Aug 14, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@alatteri
alatteri marked this pull request as ready for review August 14, 2026 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants