diff --git a/docs/caveats.md b/docs/caveats.md index 98085a4a18..f1359c2724 100644 --- a/docs/caveats.md +++ b/docs/caveats.md @@ -650,14 +650,23 @@ See also [](caveats-sros) caveats for further details. * _netlab_ RIPv2/RIPng template implements route redistribution, but only for static and connected prefixes * The device role on nodes with a loopback interface is automatically changed to **router** (contrary to most other network devices, OpenBSD does not allow you to reach non-connected IP addresses unless the IPv4/IPv6 forwarding is enabled). -(caveats-sonic)= -## Sonic - -* Sonic implementation was tested with Azure sonic-vs VM image (release 2023-11) with FRR running in a container. Other Sonic distributions might use different approaches that would require significant modifications to the configuration deployment process. -* BGP is the only routing protocol running on Azure Sonic. The choice is hardcoded in FRR compilation flags. -* You cannot use IBGP as there's no IGP protocol to resolve IBGP next hops, unless you believe in running IBGP over EBGP. -* The Azure Sonic VM image has to be started with a preconfigured BGP AS number (specified in **config_db.json**); otherwise, it does not start the FRR container. That BGP process is removed during the initial BGP configuration and replaced with the actual BGP AS number specified in the lab topology. -* _netlab_ configures BGP on Sonic through vtysh, not through **config_db**. +(caveats-sonic-vm)= +## SONiC Virtual Machine + +* SONiC implementation was tested with Azure `sonic-vs` VM image (release 2023-11) with FRR running in a container. Other SONiC distributions might use different approaches that would require significant modifications to the configuration deployment process. +* BGP is the only routing protocol running on Azure SONiC. The choice is hardcoded in FRR compilation flags. +* You cannot use IBGP, as there's no IGP protocol to resolve IBGP next hops, unless you believe in running IBGP over EBGP. +* The Azure SONiC VM image has to be started with a preconfigured BGP AS number (specified in **config_db.json**); otherwise, it does not start the FRR container. That BGP process is removed during the initial BGP configuration and replaced with the actual BGP AS number specified in the lab topology. +* _netlab_ configures BGP on SONiC through vtysh, not through **config_db**. + +(caveats-sonic-clab)= +## SONiC Container + +The `sonic` device also runs under *containerlab* with the community `docker-sonic-vs` image; see [](build-sonic-container) for how to obtain it and how the two deployments differ. + +* Configuration is deployed with **docker exec** commands, not over an SSH session. +* `docker-sonic-vs` ships `sshd` but does not start it. +* `srv6` is control-plane and kernel-plane only: the locator and End/End.X SIDs are advertised in the IS-IS LSDB and installed as kernel `seg6local` routes, but the end-to-end SRv6 datapath does not resolve -- the same open item as FRR/IS-IS SRv6 on other platforms. (caveats-vyos)= ## VyOS diff --git a/docs/labs/sonic.md b/docs/labs/sonic.md index d66ad4a41a..800577b033 100644 --- a/docs/labs/sonic.md +++ b/docs/labs/sonic.md @@ -1,17 +1,22 @@ (build-sonic)= -# Building a Sonic Vagrant Libvirt Box +# Preparing a SONiC Box or Container -You can use the **netlab libvirt package** command to build a Sonic Vagrant box for a Sonic virtual machine: +_netlab_ supports SONiC running in a VM or in a container. Unfortunately, there's no ready-to-use Vagrant box or Docker container that you could pull down from a public registry; you have to [build the box](build-sonic-box) or [download and install the container](build-sonic-container) manually. -* Download the **sonic-vs.img.gz** image from Azure or [sonic.software](https://sonic.software/) into an empty directory. +(build-sonic-box)= +## Building a SONiC Vagrant Box + +You can use the **netlab libvirt package** command to build a SONiC Vagrant box for a SONiC virtual machine: + +* Download the **sonic-vs.img.gz** image from Azure or [SONiC.software](https://SONiC.software/) into an empty directory. * Unzip image with **gunzip _gz-file-name_**. * Execute **netlab libvirt package sonic _img-file-name_** and follow the instructions ```{warning} -If you're using a *‌netlab* release older than 1.8.2, or if you're using a Linux distribution other than Ubuntu, please [read the box-building caveats first](libvirt-box-caveats.md). +If you're using a Linux distribution other than Ubuntu, please [read the box-building caveats first](libvirt-box-caveats.md). ``` -## Initial Device Configuration +### Initial Device Configuration During the box-building process, you might have to disable ZTP or clean up the initial configuration database. The **netlab libvirt config sonic** command displays the build recipe: @@ -19,3 +24,55 @@ During the box-building process, you might have to disable ZTP or clean up the i .. include:: sonic.txt :literal: ``` + +(build-sonic-container)= +## Downloading and Installing SONiC containers + +SONiC also runs under *containerlab* using the community `docker-sonic-vs` container. The container is published as +a build artifact of the [sonic-buildimage](https://github.com/sonic-net/sonic-buildimage) project, so you either download a build or make one. + +### Download a Published Build + +The SONiC container image is published as an artifact of the SONiC Azure build pipelines. From : + +* Scroll to the bottom of the pipeline list, where the **vs** platform is listed; +* Pick a branch (for example `202405`) and open **Build History**; +* Choose the latest build whose *Result* is successful and open **Artifacts**; +* Open the artifact, scroll to **target/docker-sonic-vs.gz**, and download it. + +*containerlab* documents the same path for its +[`sonic-vs` kind](https://containerlab.dev/manual/kinds/sonic-vs/), which uses this image. [sonic.software](https://SONiC.software/) is an unofficial index that is sometimes offered as an alternative, but it carries SONiC *installation* images (`sonic-vs.img`, used for the Vagrant box +above) rather than the container artifact. + +After downloading the container, unpack and load it: + +``` +gunzip docker-sonic-vs.gz +docker load -i docker-sonic-vs +``` + +Check the tag `docker load` restored with **docker images**; retag it to `docker-sonic-vs:latest` if necessary. + +### Build a SONiC Container + +Use this process in an empty directory to build a SONiC container from the `sonic-buildimage` repository: + +``` +git clone --recurse-submodules https://github.com/sonic-net/sonic-buildimage.git +cd sonic-buildimage +make init +make configure PLATFORM=vs +make target/docker-sonic-vs.gz +docker load -i target/docker-sonic-vs.gz +``` + +The device definition expects the image to be tagged **`docker-sonic-vs:latest`**; override +`defaults.devices.sonic.clab.image` in your topology if yours is tagged differently. + +### How SONiC Container Works + +`docker-sonic-vs` is a single monolithic container running FRR (`vtysh`) (unlike the VM, which runs FRR in a nested `bgp` container). The container does not start the SSH daemon; *netlab* pushes device configuration and runs validation with **docker exec** commands. + +The device inherits from the `frr` device and uses FRR control-plane configuration templates. + +See the [SONiC caveats](caveats-sonic-clab) for what is and is not supported. diff --git a/docs/platforms.md b/docs/platforms.md index 99aecdd5ef..53bf721a2a 100644 --- a/docs/platforms.md +++ b/docs/platforms.md @@ -47,7 +47,7 @@ | Nokia SR OS [❗](caveats-sros) | sros | best effort[^SROSBE] | | Nokia SR-SIM [❗](caveats-srsim) | srsim | full | | OpenBSD [❗](caveats-openbsd) | openbsd | best effort | -| Sonic [❗](caveats-sonic) | sonic | minimal | +| SONiC [❗](caveats-sonic-vm) | sonic | minimal | | VyOS 1.4 [❗](caveats-vyos) | vyos | full | [^SROSBE]: With the launch of the Nokia SR SIM, we stopped running integration tests for the SR-OS VM, assuming the behavior of the two products would be nearly identical. @@ -144,7 +144,7 @@ You cannot use all supported network devices with all virtualization providers. | Nokia SR OS | ❌ | ✅ | | Nokia SR-SIM | ❌ | ✅ | | OpenBSD | [✅](build-openbsd) | [✅](clab-vrnetlab) | -| Sonic | [✅](build-sonic) | ❌ | +| SONiC | [✅](build-sonic-box) | [✅](build-sonic-container) | | VyOS | ✅ | ✅[❗](caveats-vyos) | **Note:** @@ -211,6 +211,7 @@ Ansible playbooks included with **netlab** can deploy and collect device configu | Nokia SR OS[^SROS] | ✅ | ✅ | | OpenBSD | ✅ | ❌ | | Sonic | ✅ | ✅ | +| Sonic (containerlab) | ✅ | ✅ | | VyOS | ✅ | ✅ | **Note:** *netlab* can deploy daemon configurations, but cannot collect them. Use the **netlab initial -o** command to create daemon configuration files in a custom directory. @@ -236,6 +237,7 @@ _netlab_ uses Ansible playbooks and device-specific task lists to deploy device | Junos cRPD | clab | **bash** scripts[^cRBS] | | KinD | clab | **bash** scripts copied into and executed in containers | | linux | clab | host- or container-side **bash** scripts[^LBS] | +| Sonic (containerlab) | clab | **bash** or **vtysh** scripts[^FRRBV] over **docker exec** | [^FRRBV]: Configurations starting with a *shebang* are assumed to be Linux scripts; all other configurations are assumed to be **vtysh** scripts and get a `#!/usr/bin/vtysh -f` shebang prepended to them. @@ -297,6 +299,7 @@ The following system-wide features are configured on supported network operating | Nokia SR OS[^SROS] | ✅ | ✅ | ✅ | ✅ | ✅ | | OpenBSD | ✅ | ✅ | ❌ | ✅ | ✅ | | Sonic | ✅ | ✅ | ❌ | ✅ | ✅ | +| Sonic (containerlab) | ✅ | ✅ | ❌ | ✅ | ✅ | | VyOS | ✅ | ✅ | ✅ | ✅ | ✅ | [^HIF]: Some Linux-based devices can also use interface names in host names. See [/etc/hosts file on Linux](linux-hosts) for more details. @@ -328,6 +331,7 @@ The following interface parameters are configured on supported network operating | Nokia SR OS[^SROS] | ✅ | ❌ | ✅ | ✅ | | OpenBSD | ✅ | ❌ | ✅ | ❌ | | Sonic | ✅ | ✅ | ✅ | ✅ | +| Sonic (containerlab) | ✅ | ✅ | ✅ | ✅ | | VyOS | ✅ | ❌ | ✅ | ✅ | (platform-initial-addresses)= @@ -358,6 +362,7 @@ The following interface addresses are supported on various platforms; most daemo | Nokia SR OS[^SROS] | ✅ | ✅ | ✅ | ❌ | | OpenBSD | ✅ | ✅ | ❌ | ❌ | | Sonic | ✅ | ✅ | ✅ | ❌ | +| Sonic (containerlab) | ✅ | ✅ | ✅ | ❌ | | VyOS | ✅ | ✅ | ✅ | ❌ | ```{tip} @@ -404,6 +409,7 @@ Routing protocol [configuration modules](module-reference.md) are supported on t | Nokia SR OS[^SROS] | ✅ | ✅ | ❌ | ✅ | ✅ | | OpenBSD | ✅ | ❌ | ❌ | ✅ | ✅ | | Sonic | ❌ | ❌ | ❌ | ✅ | ❌ | +| Sonic (containerlab) | ✅ | ✅ | ❌ | ✅ | ✅ | | VyOS | ✅ | ✅ | ❌ | ✅ | ❌ | These devices support additional control-plane protocols or BGP address families: @@ -431,6 +437,7 @@ These devices support additional control-plane protocols or BGP address families | Mikrotik RouterOS 7 | ✅ | ❌ | ✅ | ❌ | | Nokia SR Linux | ✅ | ✅ | ✅ | ✅ | | Nokia SR OS[^SROS] | ✅ | ✅ | ✅ | ✅ | +| Sonic (containerlab) | ✅ | ✅ | ✅ | ✅ | | VyOS | ✅ | ✅ | ✅ | ❌ | **Notes:** @@ -450,6 +457,7 @@ The layer-2 control plane [configuration modules](module-reference.md) are suppo | dnsmasq | ❌ | ✅ | | FRR | ✅ | ✅ | | Linux | ❌ | ✅ | +| Sonic (containerlab) | ❌ | ✅ | (platform-dataplane-support)= The data plane [configuration modules](module-reference.md) are supported on these devices[^NSM]: @@ -482,6 +490,7 @@ The data plane [configuration modules](module-reference.md) are supported on the | Nokia SR Linux | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | | Nokia SR OS[^SROS] | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | OpenBSD | ✅ | ❌ | ✅ | ❌ | ❌ | +| Sonic (containerlab) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅[❗](caveats-sonic-clab) | | VyOS | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | (platform-services-support)= @@ -525,6 +534,7 @@ Core *netlab* functionality and all multi-protocol routing protocol configuratio | Nokia SR OS[^SROS] | ✅ | ✅ | ❌ | ✅ | ✅ | | OpenBSD | ✅ | ❌ | ❌ | ✅ | ❌ | | Sonic | ❌ | ❌ | ❌ | ✅ | ❌ | +| Sonic (containerlab) | ✅ | ✅ | ❌ | ✅ | ✅ | | VyOS | ✅ | ✅ | ❌ | ✅ | ❌ | (platform-unknown)= diff --git a/docs/release/26.08.md b/docs/release/26.08.md index 01ee313772..bce22c74e6 100644 --- a/docs/release/26.08.md +++ b/docs/release/26.08.md @@ -9,8 +9,6 @@ (release-26.08)= ## New Functionality -* Something new - **Minor improvements** * Something new @@ -21,6 +19,9 @@ Arista EOS: * Something new +SONiC: +* SONiC can run under *containerlab* with the community `docker-sonic-vs` container ([details](build-sonic-container), [caveats](caveats-sonic-clab)). + (release-26.08-device-fixes)= ## Fixes in Device Settings and Configuration Templates diff --git a/netsim/ansible/tasks/deploy-config/sonic-clab.yml b/netsim/ansible/tasks/deploy-config/sonic-clab.yml new file mode 100644 index 0000000000..c4953a34ee --- /dev/null +++ b/netsim/ansible/tasks/deploy-config/sonic-clab.yml @@ -0,0 +1,32 @@ +# SONiC (containerlab, docker-sonic-vs) config deploy. +# +# NOT used by 'netlab up'/'netlab initial' -- those deploy through netlab's native "sh" +# config mode inherited from FRR (see the clab: block in devices/sonic.yml). This task is +# the Ansible fallback that 'netlab config' still needs: cli/config.py clears +# netlab_config_mode before reloading a custom configuration template, which routes the +# deployment back through the Ansible playbook. +# +# It cannot simply fall through to deploy-config/frr.yml: the lookup order puts the VM's +# deploy-config/sonic.yml (which uses 'become' and 'docker exec bgp') ahead of it, and the +# container has neither sudo nor a separate bgp container. +# +# Reached over Ansible's 'docker' connection plugin (ansible_connection: docker in +# sonic.yml), so 'command'/'shell' here already execute *inside* the node container -- +# no docker exec wrapping needed. +# +# SONiC config is either a bash script of 'config' CLI commands (initial: starts with +# #!/bin/bash) or an FRR vtysh config (routing modules), same convention as the package +# 'frr'/'sonic' devices. +- template: + src: "{{ config_template }}" + dest: /tmp/config.sh + +- set_fact: deployed_config={{ lookup('template',config_template) }} + +- name: "run /tmp/config.sh to deploy {{ netsim_action }} config from {{ config_template }}" + command: bash /tmp/config.sh + when: not ansible_check_mode and ("#!/bin/bash" in deployed_config or "#!/bin/sh" in deployed_config) + +- name: "run vtysh -f to deploy {{ netsim_action }} config from {{ config_template }}" + command: vtysh -f /tmp/config.sh + when: not ansible_check_mode and not ("#!/bin/bash" in deployed_config or "#!/bin/sh" in deployed_config) diff --git a/netsim/ansible/templates/bgp/sonic-clab.j2 b/netsim/ansible/templates/bgp/sonic-clab.j2 new file mode 100644 index 0000000000..f3de798f47 --- /dev/null +++ b/netsim/ansible/templates/bgp/sonic-clab.j2 @@ -0,0 +1,12 @@ +{# + SONiC BGP, container deployment. + + The VM template (bgp/sonic.j2) opens with a "no router bgp" reset. docker-sonic-vs starts with no + BGP process at all, and vtysh rejects that line with rc 13 ("Failure to communicate[13] to bgpd, + line: no router bgp" / "% No BGP process is configured"), which fails the whole config deploy. + + Before the device was parented to FRR the container had no bgp template of its own and fell + through to bgp/frr.j2. This file keeps that behaviour now that bgp/sonic.j2 would otherwise be + selected by the .j2 lookup. +#} +{% include "frr.j2" +%} diff --git a/netsim/ansible/templates/initial/sonic-clab.j2 b/netsim/ansible/templates/initial/sonic-clab.j2 new file mode 100644 index 0000000000..ffcb3b1a27 --- /dev/null +++ b/netsim/ansible/templates/initial/sonic-clab.j2 @@ -0,0 +1,207 @@ +{% from '_extra_initial.j2' import extra_module_initial with context %} +#!/bin/bash +# +# SONiC (containerlab, docker-sonic-vs) initial configuration. +# +# Adapted from the package 'sonic' (libvirt) device's initial/sonic.j2, with changes forced +# by docker-sonic-vs's monolithic architecture (see sonic.yml for why): +# * vtysh runs directly in this container -- no 'docker exec bgp' indirection. +# * most FRR daemons ship disabled (=no in /etc/frr/daemons) and must be enabled here. +# * config_db VLANs (`Vlan`) and PortChannels (`PortChannel`) must be created BEFORE any +# SVI/aggregate is addressed. Those live in the vlan/lag module-init hooks +# (vlan|lag/sonic.initial.j2), pulled in below via extra_module_initial() at exactly this +# ordering point -- the same mechanism VyOS uses for its VLAN bridge setup. +# This script is deployed with netlab's native "sh" config mode (inherited from the FRR parent +# device): netlab bind-mounts it into the container as /etc/config/01-initial.sh and runs it with +# 'docker exec', so it already executes *inside* the node container. No Ansible in the deployment +# path. The 'config hostname' retry below is what makes that safe -- it blocks until config_db +# answers, instead of assuming the container has finished booting. +# +# CONNECTION MODEL: what still goes through Ansible -- configuration collection, validation, and +# the 'netlab config' custom-template reload -- stays on ansible_connection: docker, not +# network_cli. network_cli was evaluated and is not usable here: it requires a cliconf plugin, and +# the only SONiC cliconf shipped by Ansible (dellemc.enterprise_sonic) targets Dell's licensed +# "Management Framework" CLI (sonic-cli/klish) on Dell PowerSwitch hardware -- docker-sonic-vs +# has no such binary (verified: /etc/passwd has no admin user, no sonic-cli/klish anywhere in +# the image; the only shell is bash, and 'config'/'show'/'vtysh' are plain CLI utilities, not a +# paging network CLI a cliconf plugin can drive). This matches the rest of netlab: even the +# in-tree 'frr' clab device and the libvirt 'sonic' device (which HAS a full sshd) push FRR +# config as raw text over a generic command/shell task, never network_cli. +# +# We DO bootstrap sshd + an 'admin' user below (docker-sonic-vs ships /usr/sbin/sshd and host +# keys, just never starts them, and has no login user) purely for operator SSH access parity +# with the parent 'sonic' device (interactive login, 'netlab connect', ad-hoc troubleshooting) -- +# it is not part of the config-deployment path. +set -e +set -x +{% include 'linux/bash_profile.j2' +%} +{% include 'linux/hosts.j2' +%} +# +# Try to set hostname through Redis, wait for up to 30 seconds to succeed +# +timeout 30 bash -c ' + while ! config hostname {{ inventory_hostname.replace("_","-") }}; do + echo "Waiting for redis to start" + sleep 1 + done +' +# +# Enable sshd + an 'admin' user (idempotent) -- interactive access only, see note above. +# +id admin >/dev/null 2>&1 || useradd -m -s /bin/bash -G sudo admin +echo 'admin:{{ ansible_ssh_pass | default("YourPaSsWoRd") }}' | chpasswd +if which sudo; then + echo 'admin ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/admin +fi +mkdir -p /run/sshd +pgrep -x sshd >/dev/null || /usr/sbin/sshd +# +# Disable IPv6 (for IPv4-only interfaces) or SLAAC (if the device is a router) +# +{% for l in interfaces if l.type in ['lan','p2p','stub'] %} +{% if l.ipv6 is not defined and l.vlan.access_id is not defined and l.vlan.trunk_id is not defined and l.lag._parentindex is not defined %} +{# skip L2 switchports -- access, trunk, and LAG member ports: once a port is a + VLAN/PortChannel member the command errors out ("Cannot configure the IPv6 link + local mode!"), breaking re-deploys #} +config interface ipv6 disable use-link-local-only {{ l.ifname }} +{% endif %} +{% endfor %} +# Create config_db PortChannels (lag) and VLANs (vlan) before any aggregate/SVI is addressed +# below. These live in the per-module init hooks lag/vlan sonic.initial.j2 and are pulled +# in here, at the correct ordering point, by netlab's extra_module_initial() macro. +# +{{ extra_module_initial(['lag','vlan']) }} +# +# Configure interfaces +# +{% for l in netlab_interfaces %} +{% if l.type in ['loopback'] %} +if ip link|grep {{ l.ifname }}; then + echo {{ l.ifname }} already exists +else + config loopback add {{ l.ifname }} +fi +{% endif %} +{# Tunnel interfaces (tunnel.gre plugin) don't exist as a kernel netdev yet -- the plugin's + own template creates and addresses them directly, and the `config` CLI only accepts + Ethernet/PortChannel/Vlan/Loopback names anyway. A VNI-backed SVI (EVPN symmetric-IRB) is + the same story: its config_db `Vlan` is intentionally never created above, and + vlan/vxlan sonic-clab.j2 build/address the real bridge netdev later (vxlan runs after vrf, + which runs after initial) -- addressing it here would hit "Vlan does not exist". #} +{% set _vni_svi = l.type == 'svi' and l.vlan.name|default('') in vlans and vlans[l.vlan.name].vni is defined %} +{% if l.ipv4 is defined and (l.ipv4 is string or l._parent_ipv4 is defined) and l.type != 'tunnel' and not _vni_svi %} +{% set addr = l.ipv4 if l.ipv4 is string else l._parent_ipv4 %} +if show ip interface|grep {{ l.ifname }}|grep {{ addr }}; then + echo {{ addr }} already configured on {{ l.ifname }} +else + config interface ip add {{ l.ifname }} {{ addr }} +fi +{% endif %} +{% if l.ipv6 is defined %} +{% if not (l.virtual_interface|default(False)) %} +config interface ipv6 enable use-link-local-only {{ l.ifname }} +{% endif %} +{% if l.ipv6 is string and l.ipv6|ansible.utils.ipv6 %} +if show ipv6 interface|grep {{ l.ifname }}|grep {{ l.ipv6 }}; then + echo {{ l.ipv6 }} already configured on {{ l.ifname }} +else + config interface ip add {{ l.ifname }} {{ l.ipv6 }} +fi +{% endif %} +{% endif %} +{# Set the MTU on the kernel netdev, not through "config interface mtu": on the interfaces + containerlab injects for netlab links that command writes CONFIG_DB and never reaches the + netdev, exactly like the routed-port IPs below -- measured, the interface stayed at 9500 + while CONFIG_DB claimed otherwise. The value is netlab MTU with no adjustment: + docker-sonic-vs programs it literally (1548 in -> 1548 on the netdev), so the +48 this + used to add left every interface 48 bytes above its peers and stalled OSPF in ExStart. #} +{# L2 switchports (LAG members and VLAN access/trunk ports) reject a direct MTU set once + they are members ("'interface_name' is in portchannel!" / "is in vlan"); the + PortChannel/port MTU propagates, so skip them #} +{% if l.mtu is defined and l.mtu >= 1500 and not (l.virtual_interface|default(False)) and l.lag._parentindex is not defined and l.vlan.access_id is not defined and l.vlan.trunk_id is not defined %} +ip link set {{ l.ifname }} mtu {{ l.mtu }} +{% endif %} +! +{% endfor %} +# +# docker-sonic-vs's intfmgrd/orchagent do not program routed-port IPs from CONFIG_DB onto the +# kernel netdev (loopbacks work; routed ports do not), so BGP/OSPF over links can't come up +# without this. Bring the data ports admin-up and mirror the CONFIG_DB INTERFACE IPs onto the +# kernel netdev. Idempotent. +# +for p in $(redis-cli -n 4 keys "PORT|Ethernet*" 2>/dev/null | sed 's/PORT|//'); do + config interface startup "$p" >/dev/null 2>&1 || true +done +redis-cli -n 4 keys "INTERFACE|Ethernet*|*" 2>/dev/null | while IFS= read -r k; do + ifc=$(echo "$k" | cut -d'|' -f2); addr=$(echo "$k" | cut -d'|' -f3) + [ -n "$addr" ] && ip addr add "$addr" dev "$ifc" 2>/dev/null || true +done +# +# Enable the FRR daemons needed by the configured modules (docker-sonic-vs ships most of them +# disabled) and restart FRR once, only if something actually changed. +# +{% set daemons_needed = [] %} +{% for m in module|default([]) if netlab_frr_daemons[m] is defined %} +{% for frr_d in netlab_frr_daemons[m] %} +{% if daemons_needed.append(frr_d) %}{% endif %} +{% endfor %} +{% endfor %} +{% if daemons_needed %} +if grep -qE '^({{ daemons_needed|unique|join('|') }})=no' /etc/frr/daemons; then +{% for frr_d in daemons_needed|unique %} + sed -i 's/^{{ frr_d }}=no/{{ frr_d }}=yes/' /etc/frr/daemons +{% endfor %} +fi +{% endif %} +supervisorctl restart frr >/dev/null 2>&1 || service frr restart >/dev/null 2>&1 || true +sleep 2 +# +# And now let's configure the interfaces +# +{# REUSE NOTE (review: "reuse FRR scripting where possible"). The vtysh block below duplicates what + initial/frr.j2 renders for the same interfaces. It is not delegated because initial/frr.j2 is + itself a complete bash script -- shebang, set -e, bond creation, per-interface sysctls -- with + the vtysh configuration inline rather than in a macro, and much of that setup conflicts with the + config_db path this device uses. Delegating would mean factoring the vtysh portion of + initial/frr.j2 into a macro, i.e. a change to the FRR device, out of scope for this submission + and needing FRR re-verification. Worth doing separately. #} +cat >/tmp/netlab-initial.frr </dev/null | grep -q "{{ ch.ifname }}"; then + echo {{ ch.ifname }} already a member +else + # member MTU must equal the PortChannel MTU or member-add refuses + PCMTU=$(cat /sys/class/net/{{ intf.ifname }}/mtu) + config interface mtu {{ ch.ifname }} "$PCMTU" || true + config portchannel member add {{ intf.ifname }} {{ ch.ifname }} +fi +{% endfor %} +{% endfor %} +# +# Defensive sync: teammgrd programs the kernel bond + LACP reliably (unlike vlanmgrd), but +# mirror any PORTCHANNEL_MEMBER row the config CLI loop above didn't manage to enslave. +# Idempotent (already-enslaved ports fail the add silently); port must be link-down to join. +# +redis-cli --raw -n 4 keys "PORTCHANNEL_MEMBER|*" 2>/dev/null | while IFS= read -r k; do + pc=$(echo "$k" | cut -d'|' -f2); port=$(echo "$k" | cut -d'|' -f3) + [ -n "$port" ] || continue + if ! teamdctl "$pc" state 2>/dev/null | grep -q "^ $port"; then + ip link set "$port" down 2>/dev/null || true + teamdctl "$pc" port add "$port" 2>/dev/null || true + ip link set "$port" up 2>/dev/null || true + fi +done +exit 0 diff --git a/netsim/ansible/templates/lag/sonic.initial.j2 b/netsim/ansible/templates/lag/sonic.initial.j2 new file mode 100644 index 0000000000..86cbe76955 --- /dev/null +++ b/netsim/ansible/templates/lag/sonic.initial.j2 @@ -0,0 +1,12 @@ +# +# SONiC lag module init: create the config_db PortChannel(s) before the aggregate is addressed +# in the device initial template (create-before-address ordering). Pulled into initial config by +# extra_module_initial() -- the same per-module init-hook mechanism VyOS uses. +# +{% for l in netlab_interfaces|default([]) if l.type == 'lag' %} +if ip link show {{ l.ifname }} >/dev/null 2>&1; then + echo {{ l.ifname }} already exists +else + config portchannel add {{ l.ifname }} +fi +{% endfor %} diff --git a/netsim/ansible/templates/vlan/sonic-clab.j2 b/netsim/ansible/templates/vlan/sonic-clab.j2 new file mode 100644 index 0000000000..f907dcb155 --- /dev/null +++ b/netsim/ansible/templates/vlan/sonic-clab.j2 @@ -0,0 +1,75 @@ +#!/bin/bash +# +# SONiC vlan module — switchport membership via the config CLI (access and trunk). +# VLANs themselves (config_db `Vlan`) are created by initial/sonic-clab.j2 BEFORE the +# SVIs get addressed; this template adds port membership, THEN syncs it to the kernel itself +# (see the sync block below) -- docker-sonic-vs's vlanmgrd races at boot and often never +# mirrors VLAN_MEMBER rows written during the first deploy (a vlanmgrd restart also WIPES +# bridge state without replaying CONFIG_DB), so relying on it alone silently leaves the L2 +# datapath down (confirmed live: config_db had the VLAN_MEMBER row, but Ethernet0 had no +# `master Bridge` and cross-node ping failed). Idempotency for the config_db writes is checked +# against config_db (VLAN_MEMBER rows); the kernel sync below is separately idempotent. +set -e +set -x +{# access (untagged) members #} +{% set vni_ids = (vlans|default({})).values()|selectattr('vni','defined')|map(attribute='id')|list %} +{% for l in interfaces if l.vlan.access_id is defined and l.vlan.access_id not in vni_ids %} +if [ "$(redis-cli -n 4 exists "VLAN_MEMBER|Vlan{{ l.vlan.access_id }}|{{ l.ifname }}")" = "1" ]; then + echo {{ l.ifname }} already a member of Vlan{{ l.vlan.access_id }} +else + config vlan member add -u {{ l.vlan.access_id }} {{ l.ifname }} +fi +{% endfor %} +{# trunk (tagged) members — skip the native id if this port also has an access/native vlan #} +{% for l in interfaces if l.vlan.trunk_id is defined %} +{% for vid in l.vlan.trunk_id if vid != l.vlan.access_id|default(-1) %} +if [ "$(redis-cli -n 4 exists "VLAN_MEMBER|Vlan{{ vid }}|{{ l.ifname }}")" = "1" ]; then + echo {{ l.ifname }} already a tagged member of Vlan{{ vid }} +else + config vlan member add {{ vid }} {{ l.ifname }} +fi +{% endfor %} +{% endfor %} +{# EVPN symmetric-IRB pre-create step: vlan module runs BEFORE vrf + (vrf.config_after includes vlan), and vrf's generic frr.data-plane.j2 enslaves any + `interfaces` entry with `.vrf` set (including our SVI, ifname Vlan) into the vrf + device with `set -e` -- if that netdev doesn't exist yet, the enslave command fails + and aborts vrf's WHOLE deploy script (including the vrf's FRR/BGP config that + follows in the same script). vxlan module runs AFTER vrf (vxlan.config_after + includes vrf) and is what actually attaches the vxlan device + addresses this + bridge -- so pre-create it here, empty, just so it EXISTS in time. Capital "Vlan" + matches svi_interface_name so vrf's enslave-by-ifname logic finds it by name. #} +{% for vname, v in vlans.items() if v.vni is defined and v.mode|default('') == 'irb' and v.vrf is defined %} +if [ ! -e /sys/devices/virtual/net/Vlan{{ v.id }} ]; then + ip link add Vlan{{ v.id }} type bridge +fi +ip link set Vlan{{ v.id }} type bridge stp_state 0 +ip link set up dev Vlan{{ v.id }} +{% endfor %} +# +# Sync CONFIG_DB VLAN/VLAN_MEMBER/VLAN_INTERFACE onto the kernel bridge deterministically -- +# vlanmgrd quirk, see the header comment. Idempotent (all commands are safe to re-run). +# +ip link show Bridge >/dev/null 2>&1 || { ip link add Bridge type bridge; ip link set Bridge type bridge vlan_filtering 1; } +ip link set Bridge up 2>/dev/null || true +for k in $(redis-cli -n 4 keys "VLAN|Vlan*" 2>/dev/null); do + vid=${k#VLAN|Vlan} + bridge vlan add vid "$vid" dev Bridge self 2>/dev/null || true + ip link show "Vlan$vid" >/dev/null 2>&1 || ip link add link Bridge name "Vlan$vid" type vlan id "$vid" + ip link set "Vlan$vid" up 2>/dev/null || true +done +redis-cli -n 4 keys "VLAN_MEMBER|*" 2>/dev/null | while IFS= read -r k; do + v=$(echo "$k" | cut -d'|' -f2); port=$(echo "$k" | cut -d'|' -f3); vid=${v#Vlan} + mode=$(redis-cli -n 4 hget "$k" tagging_mode 2>/dev/null) + ip link set "$port" master Bridge 2>/dev/null || true + if [ "$mode" = "tagged" ]; then + bridge vlan add vid "$vid" dev "$port" 2>/dev/null || true + else + bridge vlan add vid "$vid" dev "$port" pvid untagged 2>/dev/null || true + fi +done +redis-cli -n 4 keys "VLAN_INTERFACE|Vlan*|*" 2>/dev/null | while IFS= read -r k; do + svi=$(echo "$k" | cut -d'|' -f2); addr=$(echo "$k" | cut -d'|' -f3) + [ -n "$addr" ] && ip addr add "$addr" dev "$svi" 2>/dev/null || true +done +exit 0 diff --git a/netsim/ansible/templates/vlan/sonic.initial.j2 b/netsim/ansible/templates/vlan/sonic.initial.j2 new file mode 100644 index 0000000000..8a55a4231d --- /dev/null +++ b/netsim/ansible/templates/vlan/sonic.initial.j2 @@ -0,0 +1,15 @@ +# +# SONiC vlan module init: create config_db VLANs (`Vlan`) before any SVI is addressed in the +# device initial template. VNI-backed VLANs are skipped -- the vxlan module builds those on the +# FRR/kernel bridge path, so no config_db Vlan should compete for the access ports. Pulled into +# initial config by extra_module_initial() -- the same per-module init-hook mechanism VyOS uses. +# +{% if vlans is defined %} +{% for vname, v in vlans.items() if v.vni is not defined %} +if show vlan brief 2>/dev/null | grep -qE "^\|\s+{{ v.id }} "; then + echo Vlan{{ v.id }} already exists +else + config vlan add {{ v.id }} +fi +{% endfor %} +{% endif %} diff --git a/netsim/ansible/templates/vxlan/sonic-clab.j2 b/netsim/ansible/templates/vxlan/sonic-clab.j2 new file mode 100644 index 0000000000..69738dd449 --- /dev/null +++ b/netsim/ansible/templates/vxlan/sonic-clab.j2 @@ -0,0 +1,96 @@ +#!/bin/bash +{# REUSE NOTE (review: "reuse FRR scripting where possible"). This one CAN be delegated and should + be: vxlan/frr.j2 exposes create_vxlan_interface(vni, br_name, vrf, mtu), which builds exactly the + per-VNI bridge + kernel vxlan netdev this template builds by hand. Left as-is only because + switching to the macro changes the rendered dataplane and needs the EVPN-VXLAN datapath + re-verified on two VS nodes (VNI up, remote VTEP learned, host-to-host ping across the tunnel) -- + a measured change, not a documentation one. Next step for this file. #} +# SONiC EVPN-VXLAN L2VNI — FRR/kernel data-plane. +# +# docker-sonic-vs's orchagent/vxlanmgrd do NOT program the VXLAN dataplane (same class +# of gap as routed-port IPs and VLAN members), so the config_db VXLAN_TUNNEL model gives +# no datapath on the VS. Instead we build the L2VNI exactly the way FRR/Cumulus does on +# plain Linux — a traditional per-VNI bridge holding the access port(s) + a kernel vxlan +# netdev — and let FRR zebra drive EVPN off that kernel state (advertise-all-vni). This +# is the same path proven live between two VS nodes (VNI up, remote VTEP learned, iBGP +# l2vpn evpn Established, host-to-host ping across the tunnel). +# +# config_db is intentionally bypassed for VNI-backed VLANs: initial/sonic-clab.j2 skips +# `config vlan add` for them and vlan/sonic-clab.j2 skips their member-add, so nothing competes +# for the access ports here. +# +# EVPN symmetric-IRB: an `irb` VNI-backed vlan with a `vrf` gets its L2VNI +# bridge NAMED "Vlan" (capital, matching svi_interface_name) instead of the plain +# bridge-mode "vlan" -- vlan/sonic-clab.j2 pre-creates that exact netdev (empty) BEFORE +# vrf deploys, so vrf's generic frr.data-plane.j2 (`ip link set Vlan master `, +# matched by ifname against netlab's own SVI interface record) can enslave it; this +# template (running AFTER vrf, vxlan.config_after includes vrf) then attaches the vxlan +# device and addresses the bridge itself as the IRB gateway -- same "bridge IS the SVI" +# pattern plain Linux/Cumulus symmetric-IRB uses. The L3VNI (VRF transit VNI) is a +# separate vxlan netdev enslaved DIRECTLY into the vrf device (no bridge -- pure L3 +# transit, no learning/flooding needed); zebra's advertise-all-vni auto-binds it to the +# vrf from that kernel state, same as the L2VNI auto-discovery. +set -e +set -x +{% set _mtu = mtu|default(1500) %} +{% if vxlan.vlans is defined %} +{% for vname in vxlan.vlans if vlans[vname].vni is defined %} +{% set vlan = vlans[vname] %} +{% set irb = vlan.mode|default('') == 'irb' and vlan.vrf is defined %} +{% set br = ("Vlan" if irb else "vlan") + vlan.id|string %} +# --- L2VNI {{ vlan.vni }} (VLAN {{ vname }}, bridge {{ br }}{{ ", IRB vrf " + vlan.vrf if irb else "" }}) --- +if [ ! -e /sys/devices/virtual/net/{{ br }} ]; then + ip link add {{ br }} type bridge +fi +ip link set {{ br }} type bridge stp_state 0 +ip link set up dev {{ br }} +if ! ip link show vxlan{{ vlan.vni }} >/dev/null 2>&1; then + ip link add vxlan{{ vlan.vni }} type vxlan id {{ vlan.vni }} dstport 4789 local {{ vxlan.vtep }} nolearning +fi +ip link set dev vxlan{{ vlan.vni }} master {{ br }} +bridge link set dev vxlan{{ vlan.vni }} learning off +ip link set mtu {{ vlan.mtu|default(_mtu) }} addrgenmode none dev vxlan{{ vlan.vni }} +ip link set up dev vxlan{{ vlan.vni }} +{% for l in interfaces if l.vlan.access_id|default(0) == vlan.id %} +# access port {{ l.ifname }} -> bridge {{ br }} (L2, no IP) +ip addr flush dev {{ l.ifname }} 2>/dev/null || true +ip link set dev {{ l.ifname }} master {{ br }} +ip link set up dev {{ l.ifname }} +{% endfor %} +{% if irb %} +{% for i in interfaces if i.type|default('') == 'svi' and i.vlan.name|default('') == vname %} +# IRB gateway: address the bridge itself (it IS the SVI) then enslave into vrf {{ vlan.vrf }} +if ! ip addr show {{ br }} | grep -q " {{ i.ipv4 }} "; then + ip addr add {{ i.ipv4 }} dev {{ br }} +fi +ip link set {{ br }} master {{ vlan.vrf }} +{% endfor %} +{% endif %} +{% endfor %} +{% endif %} +{% for vrf_name in vxlan.l3vnis|default([]) %} +{% set l3vni = vrfs[vrf_name].evpn.transit_vni %} +{% set l3br = "l3vni" + l3vni|string %} +# --- L3VNI {{ l3vni }} (VRF {{ vrf_name }} transit) -- vxlan10099 enslaved into a +# dedicated bridge, and the BRIDGE ITSELF (no address, no vlan sub-interface -- unlike +# the earlier broken attempt) enslaved into the vrf. Same two-level "bridge is the +# vrf-facing device" shape that already works for the L2VNI IRB SVI (see above), just +# unaddressed since an L3VNI has no gateway IP of its own -- this gives zebra a real +# bridge-level "System MAC" for Router MAC derivation instead of raw vxlan-device +# enslavement, which left the L3VNI State: Down (proven live, not assumed -- see the +# vxlan.j2 git history for the two rejected attempts before this one). +if [ ! -e /sys/devices/virtual/net/{{ l3br }} ]; then + ip link add {{ l3br }} type bridge +fi +ip link set {{ l3br }} type bridge stp_state 0 +ip link set up dev {{ l3br }} +if ! ip link show vxlan{{ l3vni }} >/dev/null 2>&1; then + ip link add vxlan{{ l3vni }} type vxlan id {{ l3vni }} dstport 4789 local {{ vxlan.vtep }} nolearning +fi +ip link set dev vxlan{{ l3vni }} master {{ l3br }} +bridge link set dev vxlan{{ l3vni }} learning off +ip link set mtu {{ _mtu }} addrgenmode none dev vxlan{{ l3vni }} +ip link set up dev vxlan{{ l3vni }} +ip link set {{ l3br }} master {{ vrf_name }} +{% endfor %} +exit 0 diff --git a/netsim/devices/sonic.yml b/netsim/devices/sonic.yml index 33e7eed5fc..76b6c85916 100644 --- a/netsim/devices/sonic.yml +++ b/netsim/devices/sonic.yml @@ -1,15 +1,80 @@ --- -description: Sonic VM +# SONiC. The device runs FRR for its whole control plane, so it inherits the FRR device +# (parent: frr) and then turns OFF what SONiC cannot do -- rather than re-enumerating the +# FRR feature set, which is how the two drifted apart in the first place. +# +# Two deployments share this device: +# * libvirt -- the SONiC VM (support level: minimal, largely untested) +# * clab -- docker-sonic-vs, a monolithic container (vtysh runs directly, no 'docker exec bgp') +# Everything specific to the container lives in the clab: block below, and the templates the +# container needs of its own are named /sonic-clab.j2 (netlab looks up +# "/-.j2" before "/.j2", see defaults/paths.yml). +parent: frr +description: SONiC support: level: minimal interface_name: Ethernet{ifindex * 4} ifindex_offset: 0 mgmt_if: eth0 loopback_interface_name: Loopback{ifindex} +lag_interface_name: "PortChannel{lag.ifindex}" +tunnel_interface_name: tun{ifindex} group_vars: - ansible_network_os: frr ansible_python_interpreter: auto_silent netlab_device_type: sonic +# +# Turned OFF for every SONiC deployment (see support.caveats for the evidence). +# +features: + stp: false # config-plane only, no stpd daemon -- no port blocking to verify + services: false # no DNS client + tunnel: false # no tunnels + sr: false # no segment routing + mpls: false # no MPLS data plane + vrf: false # should be configured with SONiC commands (if available) + ripv2: false # no RIPv2 support + srv6: false # no SRv6 data plane + + bgp: + # RFC 9234 BGP roles. Inherited from FRR and NOT verified here: bgp.policy/70-bgp-roles gets + # 19 of its 22 checks, and the three that fail are all "The prefix ... should not be in the + # BGP table" -- i.e. the role-based leak prevention that is the entire point of the feature + # is not being enforced. The sessions come up and the roles are configured; the filtering + # they are supposed to drive does not happen. Disabled pending verification rather than + # claimed on the strength of the checks that do pass. + role: false + vrf_local_as: false # Feature not supported by the old FRR release used in SONiC + password: false + timers: false + + initial: + # FRR declares roles [ host, router, bridge ]. Neither extra role works here: + # initial/06-bridge fails to deploy at all (one failed task, node never configured), and + # initial/05-host reaches 3 of 5 checks then fails the IPv6 ping and the RA default-route + # check. Only the router role is claimed. + roles: [ router ] + ra: false # Missing RA features in the old FRR release used in SONiC + reload: false + gateway: + # FRR declares anycast; SONiC does not deliver it. gateway/01-anycast reaches 10 of 11 checks + # and then fails ping_dup -- duplicate packets, i.e. both gateways answering, which is the + # failure anycast exists to avoid. lag/11-mlag-anycast fails earlier still ("X1 cannot + # establish a LAG with both switches in MLAG pair"), the documented no-mclagd limitation. + protocol: [ vrrp ] + dhcp: false # docker-sonic-vs ships no dhcrelay/dhcp6relay binary + routing: + policy: + match: + nexthop: false # never exercised on this platform -> not claimed + set: + community: + extended: false # never exercised on this platform -> not claimed + # Built with config_db/redis-cli sequences that only the container has been tested with; + # re-enabled under clab: below. + vlan: false + lag: false + vxlan: false + evpn: false libvirt: image: netlab/sonic mtu: 1500 @@ -19,27 +84,83 @@ libvirt: ansible_user: admin ansible_ssh_pass: YourPaSsWoRd netlab_show_command: [ sudo, vtysh, -c, 'show $@' ] +clab: + # The generic templates/provider/clab/clab.j2 renders every clab. below through its + # key-value passthrough loop, so this device needs no provider template of its own. + image: docker-sonic-vs:latest + mtu: 1500 + node: + kind: sonic-vs + # Do NOT inherit FRR's bind-mounted /etc/frr/daemons and /etc/hosts config templates: + # docker-sonic-vs ships its own daemons file (initial/sonic-clab.j2 enables the daemons the + # configured modules need, in place) and mounting FRR's over it would undo that. + config_templates: + # SONiC runs FRR, so reuse FRR's kernel-module handling: an 'mpls'/'sr' lab auto-loads + # mpls_router + mpls_iptunnel, 'vxlan' loads the vxlan modules, 'vrf' the vrf module. + kmods: + initial: [ 'vrf?' ] + group_vars: + ansible_connection: docker + ansible_user: root + ansible_ssh_pass: YourPaSsWoRd # bootstrapped 'admin' login, interactive use only + netlab_ready: [ ansible ] # skip the SSH readiness wait (sshd isn't up yet) + netlab_show_command: [ vtysh, -c, 'show $@' ] + # Inherit FRR's deployment mechanism -- netlab's native "sh" config mode + # (netlab_config_mode / netlab_default_shebang come from frr.yml). Each rendered snippet + # is bind-mounted into the container as /etc/config/NN-.sh and run with + # "docker exec", so the shebang picks the interpreter: the SONiC-specific templates open + # with '#!/bin/bash' and drive SONiC's 'config' CLI, and everything inherited from FRR + # gets FRR's '#!/usr/bin/vtysh -f' default and goes to vtysh. No Ansible in the + # deployment path -- the failures are also far easier to read this way. + # + # This only works because initial/sonic-clab.j2 waits for config_db: 'config hostname' is + # retried until redis answers. Without that gate the script ran before the container had + # finished booting and died with + # "RuntimeError: Unable to connect to redis - Connection refused", which is what made the + # Ansible deploy task look necessary. + netlab_mgmt_vrf: False # FRR sets it; SONiC has no management VRF + # docker-sonic-vs ships most FRR daemons disabled in /etc/frr/daemons; initial/sonic-clab.j2 + # enables the ones the configured modules need. watchfrr, zebra and staticd are always + # started by FRR itself and are deliberately absent from this map. + netlab_frr_daemons: + bgp: [ bgpd ] + ospf: [ ospfd, ospf6d ] + isis: [ isisd ] + bfd: [ bfdd ] + ripv2: [ ripd ] + vrf: [ bgpd ] + gateway: [ vrrpd ] + mpls: [ ldpd ] + # + # Verified on docker-sonic-vs and therefore re-enabled for the container only. + # + features: + vlan: False +# model: l3-switch +# svi_interface_name: "Vlan{vlan}" +# subif_name: "{ifname}.{subif_index}" +# native_routed: true + lag: False +# passive: true + # MLAG does not work: docker-sonic-vs has no mclagd, so the pair never forms + # (lag/10-mlag: "X1 cannot establish a LAG with both switches in MLAG pair", then the + # datapath is lost). Declared False rather than described in the caveats -- a feature that + # does not work belongs turned off. Note this does not on its own make netlab refuse an + # MLAG topology: netsim/modules/lag.py gates the peer-link machinery on + # "lag.mlag OR evpn.multihoming.lag", and evpn.multihoming.lag is still declared below. +# mlag: False + vxlan: false # Use SONiC commands to configure it + evpn: + transport: [ vxlan, mpls ] + irb: true + multihoming: + lag: true + initial: + collect: true # ansible_connection: docker unblocks this (no sshd needed) + # Repeated from the global features above: FRR declares clab.features.services as a dict, + # and the provider block wins over the global value, so 'services: false' has to be + # restated here to survive the merge. + services: false # no DNS client external: image: none -features: - initial: - ipv4: - unnumbered: true - ipv6: - lla: true - reload: false - collect: true - bgp: - activate_af: true - advertise: true - ipv6_lla: true - local_as: true - local_as_ibgp: true - vrf_local_as: true - community: - standard: [ standard, large ] - large: [ large ] - extended: [ extended ] - 2octet: [ standard ] - graphite.icon: router