Add SONiC containerlab (docker-sonic-vs) support - #3680
Conversation
07be6d2 to
281a6ae
Compare
| SSH plays no part in configuration deployment. | ||
| * Most FRR daemons ship disabled in `/etc/frr/daemons` by default (to save resources); the | ||
| initial configuration enables the ones the configured modules need and restarts FRR once. | ||
| * MPLS/SR-MPLS labs need a one-time host prerequisite before `netlab up`: `sudo modprobe |
There was a problem hiding this comment.
We have a mechanism for this, see kmods:
Line 42 in 9acc322
Since Sonic runs FRR inside, it may be better to inherit the FRR settings where possible, rather than redefine them
| # Configure interfaces | ||
| # | ||
| config hostname {{ inventory_hostname.replace("_","-") }} | ||
| {% for l in netlab_interfaces|default([]) if l.type == 'lag' %} |
There was a problem hiding this comment.
There is a recently added mechanism for running module specific initialization during init, see VyOS templates for an example
| config portchannel add {{ l.ifname }} | ||
| fi | ||
| {% endfor %} | ||
| {% if vlans is defined %} |
There was a problem hiding this comment.
This belongs in the vlan module, you could use vlan/sonic.initial.j2 similar to vyos
| # | ||
| # And now let's configure the interfaces | ||
| # | ||
| cat >/tmp/netlab-initial.frr <<CONFIG |
There was a problem hiding this comment.
This should probably reuse FRR scripting
| @@ -0,0 +1,33 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
should be named sonic-clab.j2 I believe (i.e. dash)
| @@ -0,0 +1,75 @@ | |||
| #!/bin/bash | |||
| @@ -0,0 +1,91 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
If possible reuse FRR scripts
| vrf: [ bgpd ] | ||
| gateway: [ vrrpd ] | ||
| mpls: [ ldpd ] | ||
| clab: |
There was a problem hiding this comment.
add empty kmods: key to enable kernel module loading support
| circuit_type: true | ||
| unnumbered: { ipv4: true, ipv6: true, network: true } | ||
| import: [ static ] | ||
| mpls: |
There was a problem hiding this comment.
It's up to Ivan, but I imagine it might be better to break this up into smaller PRs: First add initial support, then add OSPF, BGP, etc.
| dellos10: | ||
| features.gateway.vrrp.version: True | ||
| # the meta device used by tests/check-integration-tests.sh has to accept the plugin | ||
| none: |
There was a problem hiding this comment.
This belongs in a separate PR
| interface {{ intf.ifname }} | ||
| vrrp {{ intf.gateway.vrrp.group }} shutdown | ||
| vrrp {{ intf.gateway.vrrp.group }} version {{ gateway.vrrp.version|default(3) }} | ||
| no vrrp {{ intf.gateway.vrrp.group }} shutdown |
There was a problem hiding this comment.
The user could have requested that the interface be shutdown, the logic should check that
| #!/bin/bash | ||
| # | ||
| # vrrp.version plugin — FRR vrrpd supports both VRRP v2 and v3. netlab | ||
| # renders this as a separate per-node config step; FRR requires the vrrp instance |
There was a problem hiding this comment.
afaik none of the existing devices supports changing VRRP versions mid-flight, it's a nice detail but also a bit overkill?
jbemmel
left a comment
jbemmel
left a comment
There was a problem hiding this comment.
Thanks a lot for your sizable contribution, it's appreciated!
Overall, I think our preference would be to reuse FRR logic and templates as much as possible, rather than reimplement the same thing in parallel. You already do that in some places like validation scripts, since Sonic is running standard FRR (I presume) the configuration should be fairly similar (if not identical)
I also think a 50 file PR is perhaps a bit too big of a chunk to accept in one go, but I'll let Ivan comment
281a6ae to
1d885ff
Compare
Add a sonic_clab device (parent: sonic) for the community docker-sonic-vs image running under containerlab, alongside the existing libvirt SONiC VM device. Since SONiC runs standard FRR, the device reuses FRR wherever the config is FRR: every routing module -- including bgp -- is delegated to the in-tree FRR templates via ansible_network_os: frr (no parallel routing templates), the single validation plugin re-exports the FRR checks (from netsim.validate.frr import *), and a clab.kmods declaration (mirroring frr.yml) lets netlab auto-load the mpls/vrf/vxlan kernel modules. What stays SONiC-specific is only what genuinely is not FRR: interface, VLAN and PortChannel objects are created through SONiC config_db (the config CLI), FRR daemons that ship disabled in docker-sonic-vs are enabled per configured module, configuration is deployed over the Ansible docker connection (docker exec), and a deterministic CONFIG_DB->kernel sync backs the modules the VS orchestration agents do not program. VLAN and PortChannel creation lives in the vlan/lag module-init hooks (vlan|lag/sonic_clab.initial.j2), pulled into initial config by extra_module_initial(). Module coverage: OSPF/OSPFv3, IS-IS, BGP (+ bgp.session/policy/originate/domain, ebgp.multihop), RIPv2, BFD, VRF, VLAN, LAG, VXLAN/EVPN (incl. symmetric IRB and multihoming), MPLS L3VPN, SR-MPLS, SRv6, gateway/VRRP, routing, tunnel.gre. No netlab-core changes.
Device bring-up and per-module smoke tests for the sonic_clab device, each verified live against a docker-sonic-vs container over the docker connection plugin. Covers initial, OSPF/OSPFv3, IS-IS (v4/v6), BGP and its plugins, RIPv2, BFD, VRF and route-leaking (v4/v6), VLAN, LAG, VXLAN/EVPN, symmetric IRB, EVPN multihoming, MPLS/SR/L3VPN, SRv6, GRE tunnels, and a files-plugin worked example. Several topologies carry native validate: sections.
Add the sonic_clab device to the platform tables (device list, configuration deployment, initial config, routing/dataplane/EVPN/layer-2/IPv6 module support), a caveats section covering the image/connection model, automatic kernel-module loading for MPLS/SR, and the SRv6 control-plane limit, a containerlab usage section in the SONiC lab notes, and a 26.07 release note.
1d885ff to
bf087fc
Compare
|
Thanks for the thorough review — the "reuse FRR logic and templates rather than reimplement in parallel" principle is exactly right, and I've taken it as the primary lens for this revision, not just the individually-flagged lines. Summary of what collapsed into FRR reuse vs what stays genuinely SONiC-specific, then the per-comment detail. Reused from FRR (no parallel implementation):
Genuinely SONiC-specific (kept, because it isn't FRR):
Per comment: kmods (initial.j2 modprobe / Module init hooks (LAG PortChannel loop, VLAN creation): Moved out of the monolithic initial template into the per-module init hooks Reuse FRR scripting (initial.j2 interface block) and vxlan template: Assessed against the FRR templates. The routing and EVPN control planes already reuse FRR entirely ( vrrp.version (separate PR / buggy shutdown logic / overkill): Agreed on all three — dropped entirely from this PR. Removed the template, the test, and the Rename Break into smaller PRs: Agreed it's a sizable change. I'll leave the split call to you/Ivan — happy to restructure into staged PRs (e.g. device+initial first, then per-module) if that's easier to review. |
There was a problem hiding this comment.
Pull request overview
This PR adds a new sonic_clab device to support SONiC on the containerlab provider using the community docker-sonic-vs image, with configuration/validation driven via Ansible’s docker connection (docker exec) and FRR vtysh where applicable. It also introduces a platform-specific integration smoke-test suite and updates user documentation to advertise the new platform and its module coverage/caveats.
Changes:
- Added
sonic_clabas a child device ofsonicwith containerlab-specific settings, feature flags, and Ansible group vars. - Added SONiC-containerlab-specific initial/VLAN/LAG/VXLAN templates plus deploy/readiness Ansible task lists.
- Added a comprehensive
tests/integration/platform/sonic_clab/topology set and updated docs/release/platform/caveats/lab docs accordingly.
Reviewed changes
Copilot reviewed 44 out of 44 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
netsim/devices/sonic_clab.yml |
New device definition for SONiC on containerlab (docker-sonic-vs) with features/caveats and clab settings |
netsim/ansible/templates/initial/sonic_clab.j2 |
Initial configuration logic for monolithic SONiC container + FRR daemon enablement |
netsim/ansible/templates/vlan/sonic_clab.initial.j2 |
VLAN init hook to create VLAN objects before SVI addressing |
netsim/ansible/templates/vlan/sonic_clab.j2 |
VLAN membership + deterministic CONFIG_DB→kernel sync logic |
netsim/ansible/templates/lag/sonic_clab.initial.j2 |
LAG init hook to create PortChannels before addressing |
netsim/ansible/templates/lag/sonic_clab.j2 |
LAG membership configuration + defensive kernel/teamd sync |
netsim/ansible/templates/vxlan/sonic_clab.j2 |
VXLAN/EVPN kernel dataplane setup for docker-sonic-vs |
netsim/ansible/tasks/readiness-check/sonic_clab.yml |
Readiness wait for vtysh over docker connection |
netsim/ansible/tasks/deploy-config/sonic_clab.yml |
Config deployment via bash or vtysh -f inside the container |
netsim/validate/sonic_clab.py |
Validation aliasing to reuse FRR validation for sonic_clab |
tests/integration/platform/sonic_clab/README.md |
Overview of platform smoke tests and what each topology verifies |
tests/integration/platform/sonic_clab/01-initial.yml |
Initial bring-up smoke topology |
tests/integration/platform/sonic_clab/02-ospf.yml |
OSPF smoke topology + validation |
tests/integration/platform/sonic_clab/03-bgp.yml |
eBGP smoke topology + validation |
tests/integration/platform/sonic_clab/04-isis.yml |
IS-IS smoke topology + validation |
tests/integration/platform/sonic_clab/05-vrf.yml |
VRF + OSPF per-VRF smoke topology |
tests/integration/platform/sonic_clab/06-mpls-sr-l3vpn.yml |
MPLS/SR/L3VPN integration topology and verification notes |
tests/integration/platform/sonic_clab/07-srv6.yml |
SRv6 integration topology and verification notes |
tests/integration/platform/sonic_clab/08-vlan.yml |
VLAN/SVI dataplane integration topology |
tests/integration/platform/sonic_clab/09-lag.yml |
LAG integration topology |
tests/integration/platform/sonic_clab/11-vxlan-evpn.yml |
VXLAN/EVPN L2VNI integration topology |
tests/integration/platform/sonic_clab/12-evpn-irb-l3vni.yml |
EVPN symmetric-IRB (L3VNI) integration topology |
tests/integration/platform/sonic_clab/13-vrf-leak.yml |
VRF route-leaking integration topology |
tests/integration/platform/sonic_clab/14-bgp-session.yml |
bgp.session plugin integration topology |
tests/integration/platform/sonic_clab/15-bgp-policy.yml |
bgp.policy plugin integration topology |
tests/integration/platform/sonic_clab/16-ebgp-multihop.yml |
ebgp.multihop plugin integration topology |
tests/integration/platform/sonic_clab/17-ospf-areas.yml |
ospf.areas plugin integration topology |
tests/integration/platform/sonic_clab/18-bgp-originate.yml |
BGP originate integration topology |
tests/integration/platform/sonic_clab/19-bfd.yml |
BFD under OSPF integration topology |
tests/integration/platform/sonic_clab/20-ripv2.yml |
RIPv2 integration topology |
tests/integration/platform/sonic_clab/21-evpn-multihoming.yml |
EVPN multihoming plugin integration topology (+ host workaround) |
tests/integration/platform/sonic_clab/22-ospfv3.yml |
Dual-stack OSPFv3 integration topology |
tests/integration/platform/sonic_clab/23-isis-v6-bgp-v6.yml |
IPv6 IS-IS + IPv6 BGP AF integration topology |
tests/integration/platform/sonic_clab/24-routing-v6.yml |
IPv6 routing/prefix-list integration topology |
tests/integration/platform/sonic_clab/25-vrf-v6-leak.yml |
IPv6 VRF route-leaking integration topology |
tests/integration/platform/sonic_clab/26-vrf-v6-ospfv3.yml |
IPv6 per-VRF OSPFv3 integration topology |
tests/integration/platform/sonic_clab/27-tunnel-gre.yml |
GRE tunnel plugin integration topology |
tests/integration/platform/sonic_clab/28-files-maxprefix.yml |
files plugin configlet escape-hatch topology |
tests/integration/platform/sonic_clab/29-bgp-domain.yml |
bgp.domain plugin integration topology |
tests/integration/platform/sonic_clab/h1-fixaddr/linux.j2 |
Custom config workaround for a linux host addressing edge case |
docs/platforms.md |
Platform/support table updates to include SONiC (containerlab) |
docs/labs/sonic.md |
Lab guide updates describing how to run SONiC under containerlab |
docs/caveats.md |
Added caveats section for sonic_clab (image, connection model, limitations) |
docs/release/26.07.md |
Release notes entry announcing sonic_clab support and module coverage |
Comments suppressed due to low confidence (1)
netsim/devices/sonic_clab.yml:110
- This comment contradicts the earlier
clab.kmodsnote (and the docs) that MPLS kernel modules are auto-loaded for clab labs, enabling a real MPLS label FIB; as written it suggests MPLS can’t work in containerlab.
mpls:
ldp: true # LDP control-plane; no kernel MPLS label FIB in a clab container
vpn: true # unlocks BGP VPNv4/VPNv6 L3VPN import
| | `03-bgp.yml` | bgp | eBGP session Established (frr fallback) | | ||
| | `04-isis.yml` | isis | adjacency Up (frr fallback) | | ||
| | `05-vrf.yml` | vrf, ospf | per-VRF OSPF adjacency Full (frr fallback) | | ||
| | `06-mpls-sr-l3vpn.yml` | isis, bgp, mpls, sr, vrf | LDP Operational, real kernel MPLS label FIB, VPNv4 Established, L3VPN ping 0% loss. **Needs a one-time host prereq: `sudo modprobe mpls_router mpls_iptunnel`** | |
| # mpls_router/mpls_iptunnel modules are loaded ANYWHERE on the host: | ||
| # sudo modprobe mpls_router mpls_iptunnel | ||
| # Once loaded, /proc/sys/net/mpls/* appears in every container's netns immediately | ||
| # (even already-running ones) and mpls/frr.j2's `sysctl -w net.mpls.platform_labels` | ||
| # succeeds -- so this gets a REAL kernel label FIB (`ip -M route`), not just a | ||
| # control-plane proof. This is a one-time HOST prerequisite, not something the | ||
| # topology YAML can express -- document it, don't assume it's already loaded. |
| docker-sonic-vs ships sshd + host keys but starts neither sshd nor a login user; | ||
| templates/initial/sonic_clab.j2 bootstraps both (admin / ansible_ssh_pass below) for | ||
| interactive access ('netlab connect', ad-hoc troubleshooting) -- SSH is NOT used for config | ||
| deployment, only ansible_connection: docker is. |
| # We DO bootstrap sshd + an 'admin' user below (docker-sonic-vs ships /usr/sbin/sshd and host | ||
| # keys, just never starts them, and has no login user) purely for operator SSH access parity | ||
| # with the parent 'sonic' device (interactive login, 'netlab connect', ad-hoc troubleshooting) -- | ||
| # it is not part of the config-deployment path. |
| * `docker-sonic-vs` ships `sshd` and host keys but starts neither `sshd` nor a login user; the | ||
| initial configuration bootstraps both (`admin`/`YourPaSsWoRd`, matching the `sonic` device's | ||
| own credentials) purely for interactive access (`netlab connect`, ad-hoc troubleshooting) -- | ||
| SSH plays no part in configuration deployment. |
ipspace
left a comment
ipspace
left a comment
There was a problem hiding this comment.
I didn't go into the details (yet ;). However, I think that:
- There is no need for a new device. You can define extra (or disabled) features in clab.features and extra parameters in clab.group_vars.
- It would be nice to have a way to get a recent working SoNIC VM, maybe declare
frras SoNIC parent device (so we get all FRR control-plane features), and then turn off the features that are not supported in SoNIC VM or container. - You have to use "sonic_clab" templates because that's how you defined your device. Using "sonic" device and have different clab templates ("sonic-clab") would be a better alternative (see (1)).
- The platforms are tested with the existing tests/integration. There is no need to write new platform integration tests.
| ``` | ||
|
|
||
| (labs-sonic-clab)= | ||
| ## Using the containerlab Provider (docker-sonic-vs) |
There was a problem hiding this comment.
Change to "Using the SoNIC containers"
| Apart from the libvirt Vagrant box above, SONiC can run under **containerlab** using the community | ||
| `docker-sonic-vs` image via the `sonic_clab` device (parent: `sonic`). No box build is needed -- | ||
| supply your own `docker-sonic-vs:latest` image (build it from the | ||
| [sonic-buildimage](https://github.com/sonic-net/sonic-buildimage) `docker-sonic-vs` target, or pull a |
There was a problem hiding this comment.
It might be a good idea to provide the download URL
| * The [**WireGuard tunnel** plugin](plugin-tunnel-wireguard) supports WireGuard tunnels on FRR. | ||
| * The [**bgp.session** plugin](plugin-bgp-session) and the [OSPF module](module-ospf) support graceful restart on Arista EOS, BIRD, FortiOS, and FRR | ||
| * The [**bgp.policy** plugin](plugin-bgp-policy) supports the **bgp.role** attribute on FRR and BIRD. | ||
| * SONiC can run under *containerlab* with the community `docker-sonic-vs` image via the new [`sonic_clab` device](caveats-sonic-clab), expanding SONiC coverage well beyond the libvirt device (OSPF/OSPFv3, IS-IS, BGP with the bgp.session/policy/originate/domain and ebgp.multihop plugins, RIPv2, BFD, VRF, VLAN, LAG, VXLAN/EVPN including symmetric IRB and multihoming, MPLS L3VPN, SR-MPLS, SRv6, VRRP, and GRE tunnels). |
| interchangeable: `docker-sonic-vs` is a single monolithic container (FRR's `vtysh` runs | ||
| directly in it), while the VM runs FRR inside a nested `bgp` sub-container. | ||
|
|
||
| * You supply your own `docker-sonic-vs:latest` image (build it from the |
There was a problem hiding this comment.
This is in the "labs/sonic" document. Just add a link to that document
| * You supply your own `docker-sonic-vs:latest` image (build it from the | ||
| [sonic-buildimage](https://github.com/sonic-net/sonic-buildimage) `docker-sonic-vs` target, or | ||
| pull a prebuilt one); *netlab* does not ship or distribute it. | ||
| * Configuration is deployed over Ansible's built-in `docker` connection plugin (`docker exec`), |
There was a problem hiding this comment.
Too many details (but as before, you can move them to labs/sonic.md document. "it uses docker exec not network_cli" is good enough.
| SSH plays no part in configuration deployment. | ||
| * Most FRR daemons ship disabled in `/etc/frr/daemons` by default (to save resources); the | ||
| initial configuration enables the ones the configured modules need and restarts FRR once. | ||
| * MPLS/SR-MPLS get a real kernel MPLS label FIB (`ip -M route`) with no manual setup: because |
There was a problem hiding this comment.
If you think it's needed, move it to labs/sonic.md. No need to explain what works as expected in caveats ;)
| SONiC runs FRR, the device reuses netlab's FRR kernel-module handling (the `clab.kmods` | ||
| declaration), so `netlab up` loads `mpls_router`/`mpls_iptunnel` on the host automatically when | ||
| an MPLS or SR lab is deployed, the same way the in-tree `frr` containerlab device does. | ||
| * Module coverage is broad and FRR-delegated (`ospf`, `bgp`, `isis`, `vrf`, `bfd`, `mpls`, `sr`, |
There was a problem hiding this comment.
We have the "platforms.md" document for this.
| are advertised in the IS-IS LSDB and installed as real kernel `seg6local` routes (`ip -6 route`), | ||
| but end-to-end SRv6-OAM datapath (ping to a `uN` End SID) is not resolved on `docker-sonic-vs` | ||
| -- the same open item as the FRR/IS-IS SRv6 result on other platforms. | ||
| * Genuinely unsupported on this image (not just untested): DHCP relay/client (no |
There was a problem hiding this comment.
If things don't work, turn them off in features.
ipspace
added
the
needs changes
…lab device Addresses the review of ipspace#3680: there is no need for a second device. SONiC runs FRR for its whole control plane, so the sonic device now inherits FRR (parent: frr) and turns OFF what SONiC cannot do, rather than re-enumerating the FRR feature set -- which is how the two drifted apart in the first place: the old sonic_clab device rendered routing/frr.j2 byte-identically yet declared far fewer features than frr does with that same template. * netsim/devices/sonic_clab.yml is removed; sonic.yml gains parent: frr, the clab: block, and a disable list. * container-vs-VM differences move into clab.features and clab.group_vars. * templates the container needs of its own become <module>/sonic-clab.j2, which netlab looks up before <module>/sonic.j2 (defaults/paths.yml t_files), and the deploy task becomes deploy-config/sonic-clab.yml (tasks_generic, same order). Precedent: initial/linux-clab.j2, initial/bird-clab.j2, deploy-config/linux-clab.yml. * tests/integration/platform/sonic_clab/ is removed -- the platform is tested with the existing tests/integration. Turned off for every SONiC deployment, each with a measured reason recorded in support.caveats: stp config-plane only, no stpd daemon dhcp no dhcrelay/dhcp6relay binary in the image routing.policy.match.nexthop never exercised on this platform routing.policy.set.community.extended never exercised on this platform The last two were previously *absent* from the device file. Under a parent device absence is inheritance, so they are now disabled explicitly: silence is a claim. vlan, lag, vxlan and evpn are disabled at device level and re-enabled in clab.features, because their configuration is built from config_db/redis-cli sequences that have only ever been exercised on docker-sonic-vs, never on the VM. Inheriting a parent brings its DEPLOYMENT MECHANISM, not just its features, and three FRR settings had to be overridden or the device does not configure at all: * clab.node.config_templates -- FRR bind-mounts /etc/frr/daemons and /etc/hosts; docker-sonic-vs ships its own daemons file that initial/sonic-clab.j2 edits in place, and mounting FRR's over it undoes that. * netlab_config_mode / netlab_default_shebang -- FRR selects netlab's native "sh" config mode with a '#!/usr/bin/vtysh -f' shebang, which bypasses deploy-config/sonic-clab.yml and feeds SONiC's bash 'config' script to vtysh. Measured symptom: "RuntimeError: Unable to connect to redis - Connection refused". * netlab_mgmt_vrf -- FRR sets it, SONiC did not. Verified on docker-sonic-vs after the restructure: * ospf/ospfv2/01-network SUCCESS 4/4 -- adjacencies Full, prefixes present, with OSPF rendered by the inherited routing/frr.j2 (no sonic OSPF template exists). * the MTU fix survives and still renders: "ip link set Ethernet0 mtu 1500". * feature resolution checked directly: stp/dhcp/vlan/lag/vxlan/evpn False at device level, re-enabled under clab, match.nexthop and set.community.extended False, and set.community.large True by inheritance -- the five declarations verified in the previous branch now come from the parent for free. KNOWN OPEN ITEM, not fixed here: routing/08-community-large is 9/11. BGP sessions establish and prefixes propagate, and dut.routing.cfg renders "set large-community 65000:0:101 65000:2:202" correctly, but applying it fails: vtysh -f /tmp/config.sh -> rc 13 "Failure to communicate[13] to bgpd, line: no router bgp" "% No BGP process is configured" The routing module is configured before bgp, and the cleanup line "no router bgp" that routing/frr.j2 emits is rejected because no BGP process exists yet. This is an ordering interaction with the inherited FRR routing template, not a feature declaration problem.
bgp/sonic.j2 opens with "no router bgp". Before the device was parented to FRR the container had no bgp template of its own and fell through to bgp/frr.j2; after the rename the <device>.j2 lookup selects bgp/sonic.j2, and vtysh rejects that reset on a fresh docker-sonic-vs with rc 13 -- Failure to communicate[13] to bgpd, line: no router bgp % No BGP process is configured -- which fails the entire config deploy, not just the BGP module. Observed in the first restructured suite run as UP_FAIL on ospf/ospfv2/20-default, ospf/ospfv2/10-import, ospf/ospfv2/11-import-policy, ospf/ospfv2/22-default-vrf, ospf/ospfv3/10-import and initial/06-bridge, and as the 9/11 on routing/08-community-large reported with the restructure. One cause, not two: the earlier report attributed it to module ordering against routing/frr.j2, which was wrong -- routing/frr.j2 never emits that line. bgp/sonic-clab.j2 restores the pre-restructure behaviour, scoped to the provider that needs it, and leaves the VM template untouched.
…s not work
Parenting the device to FRR made it claim gateway.protocol: anycast, which the
hand-maintained sonic_clab device never declared. Measured on docker-sonic-vs:
gateway/01-anycast 10 of 11 checks pass, then ping_dup fails -- duplicate
packets, i.e. both gateways answering, which is precisely
the condition an anycast gateway exists to avoid
lag/11-mlag-anycast fails earlier: "X1 cannot establish a LAG with both
switches in MLAG pair" -- the documented no-mclagd limit
Both tests were correctly refused before the restructure. Restricting the
protocol list to [ vrrp ] restores that refusal rather than leaving netlab to
build a topology the platform cannot serve.
FRR declares initial.roles [ host, router, bridge ]; the hand-maintained
sonic_clab device declared none of the extra roles, and neither works on
docker-sonic-vs:
initial/06-bridge does not deploy at all -- one failed task, the node is
never configured (UP_FAIL)
initial/05-host 3 of 5 checks pass, then the IPv6 ping and the
"DUT is sending IPv6 RA, H4 got a default route" check fail
Both were correctly refused before the restructure.
Inherited from FRR by the reparenting and not verified on docker-sonic-vs. bgp.policy/70-bgp-roles gets 19 of its 22 checks, and all three failures are "The prefix ... should not be in the BGP table" -- the role-based route-leak prevention that is the whole purpose of the feature is not enforced. Sessions establish and the roles are configured; the filtering they should drive does not happen. Disabled pending verification rather than claimed on the strength of the checks that do pass -- a feature that configures but does not filter is the kind of silent wrong-path failure this device set has produced before.
…elect libvirt Not a feature claim and nothing is disabled for it. The previous sonic_clab device had a clab block only; the reparented sonic device has both libvirt and clab, so a topology that does not pin a provider can select libvirt and fail with "KVM is not installed or does not include kvm-ok utility" on a container-only host. tests/integration/initial/08-ra does exactly that. This is an inherent property of the architecture, so it is documented rather than worked around.
|
Thanks — I've restructured it as you suggested, and your architecture measurably beats what I had.
Results on the full integration suite, same harness throughout:
+21 over the structure I was defending. Inheriting FRR's feature surface reached tests my hand-maintained declarations never did — OSPF is now rendered entirely by the inherited Three things worth reporting back, since they weren't obvious: 1. Inheriting a parent brings its deployment mechanism, not just its features. Three settings had to be overridden or the device wouldn't configure at all:
None of these is a feature flag, so a features-only review would not catch them. Also worth knowing: 2. Inheritance converts under-declaration into over-declaration. 19 tests began attempting things they previously refused; 14 passed, 6 did not. I have explicitly disabled four capabilities FRR declares that SONiC does not have — anycast gateway ( 3. The device now has both |
Thank you!
Perfect.
I'm usually not merging things that don't pass the integration tests. I can look at what's going on and maybe tweak a few things, but to do that, I'd appreciate having the "download from this URL" mention in the documentation. Speaking about that, there are still a few unresolved documentation comments.
Thanks a million. If I ever decide to add an "inheriting from another device" section to developer documentation, these will definitely go in there.
That's a common behavior you can experience with many devices 🤷🏻♂️ |
Per review: if things do not work, turn them off in features rather than writing them up. docker-sonic-vs has no mclagd, so an MLAG pair never forms -- lag/10-mlag fails "X1 cannot establish a LAG with both switches in MLAG pair" and then loses the datapath. The config plane renders (a static /31 peer-link, the EOS pattern) but nothing negotiates over it. Honest limitation of this change: it does not on its own make netlab refuse an MLAG topology. netsim/modules/lag.py:77 gates the peer-link machinery on "features.lag.mlag OR features.evpn.multihoming.lag", and evpn.multihoming.lag is still declared. That declaration is itself unverified -- no evpn.multihoming test runs in the integration suite on this platform -- but removing it is a separate decision that deserves its own measurement, so it is left alone here and noted for follow-up. Verified no collateral: lag/01-l3-lag, lag/02-lag-vlan-trunk, lag/03-l3-lag-passive and lag/04-lag-vlan-routed-trunk all still render.
…e container section Review blocker: the reviewer offered to debug the remaining integration failures but has no way to obtain the image. Documents both routes to it -- downloading a published build artefact via sonic.software, and building it from sonic-buildimage -- with the load/tag steps and the tag the device expects (docker-sonic-vs:latest, which is what this submission was tested with). Also retitles the section to "Using the SoNIC containers" as requested, and updates the invocation to the restructured device: the sonic_clab device no longer exists, so it is now "netlab up -d sonic -p clab".
Addresses five review comments on docs/caveats.md, all the same theme: the
section was carrying material that belongs elsewhere. 50 lines -> 21.
* image acquisition and the VM-vs-container architecture: already in
labs/sonic.md, so this now links there instead of repeating it
* the network_cli explanation: cut to "uses docker exec, not network_cli"
* the MPLS/SR kernel-module paragraph: removed -- it described something that
works as expected, which is not what caveats are for
* the module coverage list: removed -- platforms.md carries that
* everything documented as broken: replaced by one sentence saying those
capabilities are turned off in features, which they now are
That last one is the reviewer point that "if things do not work, turn them off in
features". STP, DHCP, MLAG, anycast gateways, the host/bridge roles, BGP roles
and the two unverified routing-policy operations are all feature disables now, so
netlab refuses those topologies rather than the documentation warning about them
after the fact.
Review asked to reuse FRR scripting where possible; both flagged sites now say what was found, at the line, so the next reader does not re-derive it. initial/sonic-clab.j2 -- NOT delegated, with reason: initial/frr.j2 is itself a complete bash script (shebang, set -e, bond creation, per-interface sysctls) with its vtysh configuration inline rather than in a macro, and much of that setup conflicts with the config_db path this device uses. Delegating means factoring that vtysh portion into a macro on the FRR device -- out of scope here and needing FRR re-verification. vxlan/sonic-clab.j2 -- CAN be delegated and should be. vxlan/frr.j2 exposes create_vxlan_interface(vni, br_name, vrf, mtu), which builds exactly the per-VNI bridge plus kernel vxlan netdev this template builds by hand. Not changed here only because it alters the rendered dataplane and needs the EVPN-VXLAN datapath re-verified on two VS nodes -- a measured change, not a documentation one.
Per review, this lands in 26.08 rather than 26.07. Reworded for the restructured device: it is no longer a new sonic_clab device but a clab deployment of the existing sonic device, which now inherits FRR -- which is also why the module list is the FRR control-plane set rather than an enumeration of our own.
…ferences netsim/validate/sonic_clab.py was named for a device that no longer exists, so it could never be selected: validation for the sonic device resolves through ansible_network_os (frr) to netsim/validate/frr.py, which is what the 181-test run actually used. Dead code that reads as live. Also updates comments in the container templates that still pointed at sonic_clab.yml, sonic_clab.initial.j2, initial/sonic_clab.j2 and tests/integration/platform/sonic_clab/ -- all of which moved or were removed in the restructure. The only surviving mentions are the caveats/labs anchor names and one deliberate historical note in sonic.yml explaining what changed.
The table lists one row per device name and sonic_clab no longer exists, so the separate containerlab row named a device that cannot be selected. SONiC is now a single device with two deployments -- the same shape as the Arista row covering vEOS and cEOS. Support level left at minimal, matching the device file: the container deployment is well exercised but the libvirt VM is not, and a single row cannot express that split. The caveats and labs documents carry the distinction.
The retag line tagged the image to itself, which does nothing. docker load restores whatever tag the artefact carries, so tell the reader to check it and retag only if it differs from what the device definition expects.
The previous text made sonic.software the primary source and said it indexes docker-sonic-vs.gz. It does not: it indexes SONiC installation images (sonic-vs.img), which is what the Vagrant box above is built from, not the container artefact. docker-sonic-vs.gz is published by the SONiC Azure build pipelines. Gives the pipeline URL and the navigation to the artefact, and links containerlab, which documents the same path for its sonic-vs kind.
# Conflicts: # docs/release/26.08.md
* sudo command is no longer available; modify /etc/sudoers.d only when it exists * retry the 'config hostname' command until redis is ready * always restart FRR daemon (it starts in a weird state)
|
Had to make a few changes to make the code work with SONiC build 2605 (I'd expect people to pull down recent containers):
I did not change the deployment settings yet (your call), but I'd strongly recommend we stay as far away from Ansible as feasible (plus, the error messages are way easier to decode). Running full set of integration tests, let's see how that works out ;) |
ipspace
left a comment
ipspace
left a comment
There was a problem hiding this comment.
OK, let's move this forward. I'm merging it, but there are tons of things that still need to be fixed:
- Using Linux interfaces (and ip commands) to configure dataplane features makes no sense. The only reason to have a SONiC device is to try out features that SONiC supports, and that can be used on real hardware. So, whatever is configurable with config commands is fair game, but nothing more.
- Ideally, we'd use config commands to configure FRR as well, and only enable the features that can be configured that way. Looking at the "config" command help, it looks like SONiC only supports BGP, so maybe we should limit ourselves to that (please note that I reused FRR scripts because I wanted to invest minimal time in having a working SONiC device).
- The VLAN and LAG modules failed so many tests that I disabled them. Please figure out what works; I'll help you figure out whether that can be covered with device features or quirks, and then we can turn them back on. However, using a mix of config and ip commands that the current VLAN configuration template uses is NOT ok.
Minor details:
- With the minor changes I made, we can go to the "direct script execution" deployment model and bypass Ansible.
- It looks like the interface MTU is not set correctly
- The failed OSPF test is caused by an old FRR version; I'll add a caveat.
- Once we agree on what makes sense to support, we have to update the "platforms.md" support tables, as well as the support tables in individual modules.
With all that in mind, I'm merging this PR to move things forward. Thanks again for all the work you invested into this feature!
* Device reload does not work * Configurable RA parameters do not work as they should on the old FRR version * Caveats in device definition are displayed in the "netlab show devices" command. What you had in that field belongs partly to "caveats.md", and mostly to "implementation notes".
|
One more thing: test results are @ https://tests.netlab.tools/_html/sonic |
…oy path The redis-readiness gate added in 48c59d0 removes the reason SONiC was overriding FRR's deployment mechanism, so stop overriding it: * remove the commented-out netlab_config_mode / netlab_default_shebang lines rather than leaving them commented, and rewrite the comment block that explained why we did NOT inherit FRR's mechanism -- it now explains that we do, and records the redis gate that made it possible * correct the same stale claim at the top of initial/sonic-clab.j2 * document what tasks/deploy-config/sonic-clab.yml is still for. It is NOT dead: 'netlab config' clears netlab_config_mode (cli/config.py) and reloads custom templates through the Ansible playbook, and the lookup order puts the VM's deploy-config/sonic.yml (become + 'docker exec bgp') ahead of frr.yml, so the container needs its own task. Both branches verified live. * delete the dead readiness-check/sonic_clab.yml -- the device is 'sonic', so that filename has never been matched by the readiness lookup Verified live on docker-sonic-vs (vlan/33-vlan-irb-trunk, 9/9 checks): the DUT reports "Script: initial,vlan" and the container holds /etc/config/01-initial.sh and /etc/config/02-vlan.sh.
|
Thanks for merging it, and for going hands-on with the container — the readiness gate on Your direction is clear and I agree with it: if it can't be done with config commands, it isn't really SONiC support. Taking that as the rule going forward. Two consequences I'll act on: VLAN 1 is a good test of the rule. Chasing the I'll take you up on the VLAN/LAG offer. Rather than argue for turning them back on, I'll work out what's actually achievable with config commands only — per operation (access, trunk, native, SVI/IRB) and per LAG operation — and come back with a list of what can pass under that restriction and what can't be expressed at all. I expect that means rewriting the VLAN and LAG templates rather than patching them, since the current ones mix On the MTU — that one surprises me, because interface MTU was the single biggest fix in this work (it was why every OSPF adjacency sat in On scope — you raised that I'll leave |
👍
We don't deal with "VLAN 1 is broken" scenario in the core, but it's so common that we have a "check_tagged_vlan_1" common quirk for switches. Some devices with router configuration mindset (= create subinterfaces, connect them to a bridge) also have problems; you might want to check IOSv and RouterOS7 quirks. Also: check the very last column in https://tests.netlab.tools/_html/coverage.vlan ("VLAN 1 in trunks"). Devices that fail it correctly (with "not supported") usually do it via device quirk.
Perfect.
Exactly.
That approach could result in a never-ending whack-a-mole exercise. Do whatever you can with the config commands, and if it doesn't work, we'll add a testing/documentation caveat. If they manage to fix it (like FRR team fixed the OSPFv3 prefixes on loopbacks), the testing caveat will eventually "miraculously" disappear.
Correct. Please do not use ip commands. Also, who knows how the SONiC config commands stomp on interface settings and how that might change. As for readiness gate -- once the config commands work, we should be good to use them, right?
Well, you should do a VLAN/LAG rewrite anyway ;) Using dataplane ip commands makes no sense as they probably won't do much good on hardware platforms. From the control-plane perspective, I think we should give users a fair assessment of what SONiC can do out of the box (= BGP). As it uses FRR in the background, and we can configure that, we could document how to enable additional control-plane features if the user wants them, with an added warning that those features configure FRR directly, which might defy the whole purpose of having SONiC. Alternatively, we could eventually go for a configurable control plane, like what @jbemmel did with VPP.
👍 |
Add SONiC containerlab (docker-sonic-vs) support
Summary
netlab already ships a
sonicdevice, but only as a libvirt Vagrant box (the Azure/VMSONiC image) with
minimalsupport declared — initial configuration and BGP. This PR addsSONiC support under containerlab using the community
docker-sonic-vsimage, and expands thedeclared/validated SONiC module coverage well beyond initial + BGP.
The new module coverage is attached to a child device
sonic_clab(parent:sonic) ratherthan to the existing libvirt device, because that coverage was developed and validated against
the
docker-sonic-vscontainerlab image and its access model specifically — a single monolithiccontainer with FRR
vtyshrunning directly in it, configured over Ansible'sdockerconnectionplugin. The libvirt SONiC VM has a different access model (FRR runs in a nested
bgpsub-container reached via
docker exec bgp vtysh), so the templates and config-push path hereare not the ones exercised on the VM. Scoping the new support to the child device keeps each
device advertising only what has actually been validated on its image; it is not a claim that
the VM is incapable of these features.
sonic_clabinherits interface/loopback naming and the base BGP feature set fromsonic,adds a
clab:block, and provides its owninitial/bgp/vlan/lag/vxlan/vrrp.versiontemplates where the monolithic image requires it. Every other module is delegated to the in-tree
FRR templates via the inherited
ansible_network_os: frr.No netlab-core changes.
netsim/cli/,netsim/augment/,netsim/attributes.yml,netsim/providers/, andnetsim/modules/are byte-identical todev.mypy -p netsimreportsSuccess: no issues found in 339 source files and the transformation test suite passes
284 passed, 1 skipped, 0 failed (run in a clean venv from
requirements.txt+requirements-dev.txt).Design note
Added as a child device
sonic_clab(parent:sonic), following the existingiol/ioll2,csr/cat8000v,vjunos-router/vjunos-switchprecedent — same NOS, adifferent deployment image/shape and validated feature set, expressed as a separate device so the
new coverage attaches to the containerlab image without changing the existing libvirt
sonicdevice. Happy to rename or restructure if you'd prefer a different approach (e.g. a different
child-device name, or modelling it as a provider/image variant of
sonic) — just let me know.Connection / config-deployment model
Configuration is deployed over Ansible's built-in
dockerconnection plugin (docker exec),not
network_cli— the same pattern netlab already uses for the in-treefrrcontainerlabdevice.
docker-sonic-vsships no runningsshdand no cliconf-compatible CLI (the only SONiCcliconf Ansible ships,
dellemc.enterprise_sonic, targets Dell's licensed Management-FrameworkCLI, which the community image does not contain). Routing config is rendered as FRR
vtyshscripts;
initial/vlan/laguse SONiC's nativeconfigCLI plus a deterministicCONFIG_DB→kernel sync where the VS orchestration agents don't program the datapath.
FRR daemons that ship disabled in
/etc/frr/daemonsare enabled per configured module duringinitial configuration.
Supported modules
ospf(incl. OSPFv3,ospf.areas),isis(v4/v6),bgp(+bgp.session,bgp.policy,bgp.originate,bgp.domain,ebgp.multihop),ripv2,bfd,vrf(incl. route-leaking,per-VRF OSPFv2/OSPFv3/IS-IS/BGP),
vlan(L3 switch / SVIs),lag,vxlan,evpn(VXLAN andMPLS transport, symmetric IRB / L3VNI,
evpn.multihoming),mpls(LDP + L3VPN),sr(SR-MPLSvia IS-IS),
srv6,gateway/vrrp.version,routing(static/prefix/aspath/community/policy),and
tunnel.gre.Support level: best-effort — broad module coverage, each module carrying a device
integration topology under
tests/integration/platform/sonic_clab/; the image is user-suppliedand the device is not (yet) wired into the automated per-release integration matrix.
Verification and provenance
Live-verified for this submission against a real
docker-sonic-vs:latestcontainer oncontainerlab, config + validation over
docker exec:netlab validate(native docker-exec)netlab validate(native docker-exec)netlab validate(native docker-exec)redmaster Bridge state forwarding(CONFIG_DB→kernel bridge sync confirmed)Supported, with integration topologies included but not re-run for this submission: the
remaining modules (EVPN / symmetric-IRB /
evpn.multihoming, MPLS L3VPN, SR-MPLS, SRv6, RIPv2,BFD,
tunnel.gre, the BGP plugins, and the IPv4/IPv6 dual-stack variants) each ship a topologyunder
tests/integration/platform/sonic_clab/(several with nativevalidate:sections) that wasexercised during the device's development. They are presented as supported, not as re-verified in
this PR.
Caveats / limits
Documented in
docs/caveats.md(caveats-sonic-clab) and flagged in the platform supporttables:
docker-sonic-vs:latestimage; netlab does not ship it.sudo modprobe mpls_router mpls_iptunnel(loadsnet.mplsinto every container netns → real kernel label FIB). Flaggedwith a caveat marker on the MPLS/SR-MPLS cells in the support tables.
IS-IS LSDB and installed as real kernel
seg6localroutes, but end-to-end SRv6-OAM datapath(ping to a
uNEnd SID) is not resolved — the same open item seen with FRR/IS-IS SRv6elsewhere. Marked with a caveat in the support table.
dhcrelay/dhcp6relaybinary), real STP port-blocking (nostpddaemon), and themlag.vtepactive-active EVPN datapath (nomclagd). These are shown as ✗/absent in thetables, not claimed.
Files
netsim/devices/sonic_clab.yml— device definition (parentsonic,clab:block, features).netsim/ansible/templates/{initial,bgp,vlan,lag,vxlan}/sonic_clab.j2,netsim/extra/vrrp/version/sonic_clab.j2— device-specific templates.netsim/ansible/tasks/{deploy-config,readiness-check}/sonic_clab.yml— docker-exec deploy.netsim/validate/{,ospf,bgp,isis,route}/sonic_clab.py— FRR validation re-exports.tests/integration/platform/sonic_clab/— integration topologies + README.docs/platforms.md,docs/caveats.md,docs/labs/sonic.md,docs/release/26.07.md— docs.netsim/extra/vrrp/version/defaults.yml— registers thenonemeta-device for thevrrp.versionplugin sotests/check-integration-tests.shcan transform the10-vrrp-versiontopology under
-d none, mirroring the existingevpn.multihomingplugin. This is the onlychange outside the
sonic_clabdevice + docs.