diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/_index.md b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/_index.md new file mode 100644 index 0000000000..06ba6bc2fc --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/_index.md @@ -0,0 +1,75 @@ +--- +title: Run parallel vision inference on an Alif Ensemble E8 with Zephyr + +description: Build a power-conscious live camera demo that drives Ethos-U55 and Ethos-U85 from one Cortex-M55 MCU. + +minutes_to_complete: 90 + +who_is_this_for: This is an advanced topic for embedded ML developers who want to run two ExecuTorch models concurrently on separate Ethos-U NPUs under Zephyr. + +learning_objectives: + - Explain how one MCU can coordinate two NPUs while avoiding the power and system cost of a second MCU or application processor + - Configure an Alif Ensemble E8 DevKit for native Zephyr camera, ISP, display, and dual-NPU operation + - Build, package, and flash an ExecuTorch application that targets Ethos-U55 and Ethos-U85 + - Validate live camera capture, model results, and parallel inference timing + +prerequisites: + - Experience with C/C++, embedded systems, and Zephyr build concepts + - A development machine running macOS on Apple Silicon with Homebrew and the Xcode Command Line Tools installed + - An [Alif Ensemble E8 DevKit](https://alifsemi.com/support/kits/ensemble-e8devkit/) with an MT9M114 camera connected to J16 and an MW405 display + - Alif SEROM 1.105.65 and SERAM 1.110.0 installed on the board + - Alif SEToolkit 1.10 installed on the development machine + +author: Varun Chari + +generate_summary_faq: false +rerun_summary: false +rerun_faqs: false + +skilllevels: Advanced +subjects: ML +armips: + - Cortex-M55 + - Ethos-U55 + - Ethos-U85 +tools_software_languages: + - ExecuTorch + - Zephyr + - Python + - GCC +operatingsystems: + - macOS + - RTOS + +further_reading: + - resource: + title: Alif Ensemble E8 DevKit support page + link: https://alifsemi.com/support/kits/ensemble-e8devkit/ + type: website + - resource: + title: Alif SDK pull request 879 + link: https://github.com/alifsemi/sdk-alif/pull/879 + type: website + - resource: + title: Ethos-U core driver multi-variant merge + link: https://gitlab.arm.com/artificial-intelligence/ethos-u/ethos-u-core-driver/-/commit/b7cd193afde80afe8bbae9a26d2ca6586554f054 + type: website + - resource: + title: ExecuTorch Arm Ethos-U NPU backend tutorial + link: https://docs.pytorch.org/executorch/stable/tutorial-arm-ethos-u.html + type: documentation + - resource: + title: Run image classification on an Alif Ensemble E8 DevKit using ExecuTorch and Ethos-U85 + link: /learning-paths/embedded-and-microcontrollers/alif-image-classification/ + type: documentation + - resource: + title: Dual-NPU live vision sample source + link: https://github.com/varunchariArm/sdk-alif/tree/dual-npu-main-integration/samples/modules/executorch/dual_npu_vision + type: website + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 +layout: "learningpathall" +learning_path_main_page: "yes" +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/_next-steps.md b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/_next-steps.md new file mode 100644 index 0000000000..e20dfa6d43 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 +title: "Next Steps" +layout: "learningpathall" +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/build-application.md b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/build-application.md new file mode 100644 index 0000000000..5355610302 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/build-application.md @@ -0,0 +1,106 @@ +--- +title: Build the dual-NPU application +description: Configure the Zephyr overlays and build the firmware and model payload for the E8 high-performance core. +weight: 4 +layout: "learningpathall" +--- + +The sample includes compiled PTE models for Ethos-U55 and Ethos-U85. It also includes a startup image that verifies both inference pipelines before the application starts the live camera. + +## Define the build inputs + +From the west workspace root, activate the Python environment and define the sample paths: + +```bash +cd $HOME/alif-dual-npu +source .venv/bin/activate +APP=$PWD/sdk-alif/samples/modules/executorch/dual_npu_vision +OD=$PWD/sdk-alif/samples/modules/tflite-micro/alif_object_detection +MODULES=$(west list -f '{abspath}' | grep -v '/modules/lib/executorch$' | paste -sd';' -) +MODULES="$MODULES;$PWD/modules/lib/executorch;$PWD/modules/ethos-u-core-driver-src" +``` + +The overlay order matters. The final `isp_route.overlay` file disables the CPI memory endpoint and routes camera frames exclusively through the ISP. + +## Build the firmware + +Run the complete build command: + +```bash +west build \ + -b alif_e8_dk/ae822fa0e5597xx0/rtss_hp \ + -d build-dual-npu-vision \ + "$APP" --pristine -- \ + -DZEPHYR_MODULES="$MODULES" \ + -DPYTHON_EXECUTABLE="$PWD/.venv-executorch/bin/python" \ + -DPython3_EXECUTABLE="$PWD/.venv-executorch/bin/python" \ + -DCMSIS_NN_LOCAL_PATH="$PWD/modules/cmsis-nn-src" \ + -DDTC_OVERLAY_FILE="$APP/boards/dual_npu_e8.overlay;$OD/boards/alif_e8_dk_ae822fa0e5597xx0_rtss_hp.overlay;$OD/serial_camera.overlay;$OD/serial_camera_mt9m114.overlay;$OD/serial_camera_isp.overlay;$OD/serial_camera_mt9m114_isp.overlay;$APP/isp_route.overlay" \ + -DOVERLAY_CONFIG="$APP/vision.conf" +``` + +A successful build ends with messages showing that Zephyr linked the ELF file and generated the binary. + +Confirm that the three expected outputs exist: + +```bash +ls -lh build-dual-npu-vision/zephyr/zephyr.bin \ + build-dual-npu-vision/zephyr/zephyr.elf \ + build-dual-npu-vision/model_assets.bin +``` + +The files have these roles: + +| File | Purpose | +| --- | --- | +| `zephyr.bin` | Cortex-M55 Zephyr application | +| `zephyr.elf` | Symbols and debug information | +| `model_assets.bin` | U85 PTE, U55 PTE, startup image, and ImageNet labels | + +The checked-in PTE files let you build this Learning Path without regenerating models. The original artifacts were generated on a Linux development host, but the model inputs are public and do not depend on that host. + +Create a directory for the source weights: + +```bash +cd $HOME/alif-dual-npu +mkdir -p model-weights +``` + +Download the trained [SSD-Slim int8 model](https://github.com/emza-vs/ModelZoo/blob/59fcdb2aab865a8a8d93a9d419b3c5490a5508e4/Models/Object_detection/SSD/ssd_slim_120x160x1_v1_int8.tflite) and the official [torchvision MobileNetV2 checkpoint](https://download.pytorch.org/models/mobilenet_v2-7ebf99e0.pth). The SSD URL is pinned to the commit that added the model: + +```bash +curl -L \ + https://raw.githubusercontent.com/emza-vs/ModelZoo/59fcdb2aab865a8a8d93a9d419b3c5490a5508e4/Models/Object_detection/SSD/ssd_slim_120x160x1_v1_int8.tflite \ + -o model-weights/ssd_slim_120x160x1_v1_int8.tflite +curl -L \ + https://download.pytorch.org/models/mobilenet_v2-7ebf99e0.pth \ + -o model-weights/mobilenet_v2-7ebf99e0.pth +``` + +Verify both downloads before using them: + +```bash +echo "64fcc31aa517798d0e798551418c85bc0a5ed03a75c45c4e47fc7ee41e5ea51f model-weights/ssd_slim_120x160x1_v1_int8.tflite" | shasum -a 256 -c - +echo "7ebf99e03e254b273379b23edca7ec0da9f48273b23a332b93c1c99d49e86e8f model-weights/mobilenet_v2-7ebf99e0.pth" | shasum -a 256 -c - +``` + +The SSD source repository does not publish a PyTorch checkpoint. Install TensorFlow in the ExecuTorch Python environment, then use the sample's importer to convert the trained, quantized constants into the common PyTorch checkpoint: + +```bash +.venv-executorch/bin/python -m pip install tensorflow==2.20.0 +.venv-executorch/bin/python \ + "$APP/tools/import_ssd_slim_tflite.py" \ + --source model-weights/ssd_slim_120x160x1_v1_int8.tflite \ + --output model-weights/ssd_slim_common.pth +``` + +Generate both PTE files with the Alif Vela configuration included in the SDK: + +```bash +"$APP/tools/generate_pte_models.sh" \ + "$PWD/sdk-alif/samples/modules/executorch/ensemble_vela.ini" \ + "$PWD/model-weights/ssd_slim_common.pth" \ + "$PWD/model-weights/mobilenet_v2-7ebf99e0.pth" +``` + +The script emits `comparable_ssd_slim_u55.pte` and `mobilenet_v2_imagenet_u85.pte` in the sample's `models` directory. The export uses PT2E quantization, deterministic representative inputs, the ExecuTorch Ethos-U partitioner, and Vela. Both programs expose int8 tensors and require complete delegation, with no Cortex-M fallback operators. The SSD checkpoint is produced by the supplied offline weight-import tool. Both deployed files are native ExecuTorch PTE programs. diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/dual-npu-memory-layout.svg b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/dual-npu-memory-layout.svg new file mode 100644 index 0000000000..0c496045c7 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/dual-npu-memory-layout.svg @@ -0,0 +1,79 @@ + + Memory layout for the dual-NPU live vision application + Four columns show MRAM, high-performance Cortex-M55 DTCM, shared SRAM0 and shared SRAM1. MRAM stores the two PTE models, startup image, labels, and Zephyr firmware. DTCM stores the U55 method pool and runtime state. SRAM0 stores U55 delegate scratch. SRAM1 stores the display framebuffer, U85 working memory, and video buffers. + + + + + Dual-NPU application memory layout + + MRAM + persistent across reset + + + Model assets, 3,888,654 bytes + 0x80008000 U85 MobileNetV2 + 0x80357520 U55 SSD-Slim + 0x8039FCE0 BMP + labels + + Zephyr firmware + 0x80400000, RTSS-HP XIP + unused gaps are not to scale + + HP DTCM + 0x20000000, 1 MiB private RAM + + + U55 inference state + 640 KiB method pool + 4 KiB U55 metadata pool + 4 KiB U85 metadata pool + per-NPU fast scratch + + Zephyr and ExecuTorch + worker stacks and semaphores + two driver objects and IRQ state + U85 metadata and fast scratch + CPU-local addresses require global aliases + + Shared SRAM0 + 0x02000000-0x023FFFFF, 4 MiB + + + U55 temporary pool + 0x02000000, 3,686,400 bytes + SSD-Slim delegate scratch + + Unallocated headroom + 0x02384000-0x023FFFFF + visible to CPU and U55 + + Shared SRAM1 + 0x02400000-0x027FFFFF, 4 MiB + + + MW405 framebuffer + 0x02400000, 768,000 bytes + + U85 command/weight mirror + 0x024BB800, 557,056 bytes + + U85 temporary pool + 0x02543800, 1,509,968 bytes + + U85 method pool + 0x026B4250, 304 KiB + + video-buffer heap + 0x02700250, 560 KiB + display, U85, and camera DMA visible + + Addresses and sizes are taken from the validated build; linked DTCM symbol addresses can move when the application changes. + diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/dual-npu-system.svg b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/dual-npu-system.svg new file mode 100644 index 0000000000..d218d4ad61 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/dual-npu-system.svg @@ -0,0 +1,101 @@ + + Alif Ensemble E8 dual-NPU live vision architecture + An MT9M114 camera feeds the CSI-2 camera interface and ISP. A Cortex-M55 coordinator sends each frame to parallel Zephyr workers. One worker runs SSD-Slim on Ethos-U55 and the other runs MobileNetV2 on Ethos-U85. Results are composed and sent through CDC200 and MIPI DSI to the MW405 display. + + + + + + + + + Alif Ensemble E8 dual-NPU live vision + + + MT9M114 + camera on J16 + MIPI CSI-2, D-PHY 0 + + + CSI-2 + CPI + 1288 x 728 RAW10 + native Zephyr video drivers + + + Hardware ISP + crop and demosaic + 192 x 192 planar RGB888 + + + Video buffer pool + five queued buffers + SRAM0, cache managed + + + + + + + Zephyr on the RTSS-HP Cortex-M55 + + + Frame coordinator + capture, preprocess, barriers + timestamps start, completion and overlap + + + + U55 worker, priority 5 + ExecuTorch SSD-Slim + 120 x 160 grayscale input + + + U85 worker, priority 4 + ExecuTorch MobileNetV2 + 224 x 224 RGB input + + + + + + Multi-variant driver registry + reserve(product, 256 MACs) + + + + + Ethos-U55 + MMIO 0x400E1000 + IRQ 55, independent driver + + + Ethos-U85 + MMIO 0x49042000 + IRQ 366, independent driver + + + + + + UI compositor + preview, boxes, class + rolling U55/U85 timing + + + CDC200 + DSI + 480 x 800 RGB565 + MW405, D-PHY 1 + + + + + diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/flash-application.md b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/flash-application.md new file mode 100644 index 0000000000..dd6c0d1a7a --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/flash-application.md @@ -0,0 +1,64 @@ +--- +title: Package and flash the application +description: Use Alif SEToolkit to package the Zephyr image and model payload and write them to MRAM. +weight: 5 +layout: "learningpathall" +--- + +The Alif boot flow uses a table of contents to load and start each processor image. The sample provides a SEToolkit JSON file that assigns the application and model payload to their validated MRAM addresses. + +## Stage the images + +Set `ALIF_SE_TOOLS_DIR` to your SEToolkit 1.10 application directory. This example uses the macOS package name: + +```bash +cd $HOME/alif-dual-npu +export ALIF_SE_TOOLS_DIR=$HOME/Alif/alif_se_toolkit_110/app-release-exec-macos +APP=$PWD/sdk-alif/samples/modules/executorch/dual_npu_vision +``` + +Confirm that SEToolkit and the support objects referenced by the sample's package configuration are present: + +```bash +test -x "$ALIF_SE_TOOLS_DIR/app-gen-toc" +test -x "$ALIF_SE_TOOLS_DIR/app-write-mram" +test -f "$ALIF_SE_TOOLS_DIR/build/config/app-device-config.json" +test -f "$ALIF_SE_TOOLS_DIR/build/images/a32_stub_0.bin" +test -f "$ALIF_SE_TOOLS_DIR/build/images/m55_stub_he.bin" +``` + +Each command completes without output when the file is present. Obtain the validated SEToolkit and firmware updates through the [Alif E8 DevKit support page](https://alifsemi.com/support/kits/ensemble-e8devkit/) if any check fails. + +Copy the images and package configuration into SEToolkit: + +```bash +cp build-dual-npu-vision/zephyr/zephyr.bin \ + build-dual-npu-vision/model_assets.bin \ + "$ALIF_SE_TOOLS_DIR/build/images/" +cp "$APP/flash/dual-npu-vision.json" \ + "$ALIF_SE_TOOLS_DIR/build/config/" +``` + +## Generate and write the package + +Close serial terminals connected to the board. Confirm that the boot switch is in the SE position, then run: + +```bash +cd "$ALIF_SE_TOOLS_DIR" +./app-gen-toc -f build/config/dual-npu-vision.json +./app-write-mram -p +``` + +Wait until `app-write-mram` reports that the write completed. Do not reset or disconnect the board during this operation. + +## Boot the application + +Move the switch from SE to U4. Open the U4 serial port at 115200 baud, 8 data bits, no parity, and 1 stop bit. Reset the board. + +The secure-enclave log shows entries for `HP_APP` and `U85MOD`. The U4 log starts with the Zephyr boot banner and the application name: + +```output +*** dual ExecuTorch parallel SSD(U55) + MobileNetV2(U85) *** +``` + +The application is now ready to run the startup test and live camera pipeline. diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/prepare-workspace.md b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/prepare-workspace.md new file mode 100644 index 0000000000..0947ec4532 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/prepare-workspace.md @@ -0,0 +1,160 @@ +--- +title: Prepare the board and workspace +description: Connect the E8 hardware and create a west workspace with the validated dual-NPU dependencies. +weight: 3 +layout: "learningpathall" +--- + +## Connect the hardware + +Power off the E8 DevKit before changing camera or display connections. + +1. Connect the MT9M114 camera module to the bottom-side J16 connector. +2. Connect the MW405 display to the display connector. +3. Connect the board's USB ports for power, SE UART, and U4 UART. +4. Confirm that the board runs SEROM 1.105.65 and SERAM 1.110.0. +5. Move the boot switch to the SE position before flashing. + +{{% notice Note %}} +The supplied overlay targets the J16 selfie-camera connection. J22 uses a different I2C address and device-tree route. +{{% /notice %}} + +## Install the host tools + +Confirm that the Xcode Command Line Tools are installed: + +```bash +xcode-select -p +``` + +If the command reports that the tools are missing, install them before you continue: + +```bash +xcode-select --install +``` + +Install Git, CMake, and Python 3.12 with Homebrew. Then create the west Python environment: + +```bash +brew install git cmake python@3.12 +mkdir -p $HOME/alif-dual-npu +cd $HOME/alif-dual-npu +python3.12 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip +python -m pip install west pyelftools fdt ninja +``` + +Confirm that west and Ninja are available: + +```bash +west --version +ninja --version +``` + +Both commands print a version number. + +## Create the west workspace + +Clone the SDK fork that contains the dual-NPU sample at the validated revision, +then initialize a local west workspace from that checkout. +The fork's ``main`` branch stays synchronized with the Alif SDK ``main`` +branch. The dual-NPU application is maintained separately on the +``dual-npu-main-integration`` branch, which also includes the merged MT9M114, +ISP, and MW405 changes from pull request 879: + +```bash +cd $HOME/alif-dual-npu +source .venv/bin/activate +git clone --branch dual-npu-main-integration --single-branch \ + https://github.com/varunchariArm/sdk-alif.git sdk-alif +git -C sdk-alif checkout fb6d0e61ebcad3098dc6298bc40386cacc4ad38a +west init -l sdk-alif +west config manifest.project-filter +executorch +west update +python -m pip install -r zephyr/scripts/requirements.txt +west sdk install +``` + +The manifest project appears at `sdk-alif`, and the remaining projects appear under `modules`, `bootloader`, `tools`, and `zephyr`. + +{{% notice Note %}} +Commit ``fb6d0e61ebcad3098dc6298bc40386cacc4ad38a`` on the +``dual-npu-main-integration`` branch contains everything required for this +Learning Path: the camera and display support from the upstream Alif SDK +``main`` branch, plus the ``dual_npu_vision`` application. Use the fork and +revision shown in the command. Do not initialize from +``alifsemi/sdk-alif`` directly because the dual-NPU application has not yet +been merged there. The fork's ``main`` branch remains synchronized with the +upstream Alif SDK and does not contain the application. +{{% /notice %}} + +Initialize the ExecuTorch submodules: + +```bash +git -C modules/lib/executorch submodule update --init --recursive +``` + +## Add the multi-variant dependencies + +The demo uses the multi-variant support merged into the Ethos-U core driver +``main`` branch. This support allows one Cortex-M55 to manage the U55 and U85 +through one driver registry, avoiding the system power overhead of assigning +each NPU to a separate MCU. Clone the current ``main`` branch: + +```bash +git clone --branch main \ + https://gitlab.arm.com/artificial-intelligence/ethos-u/ethos-u-core-driver.git \ + modules/ethos-u-core-driver-src +git -C modules/ethos-u-core-driver-src merge-base --is-ancestor \ + b7cd193afde80afe8bbae9a26d2ca6586554f054 HEAD +``` + +The Alif west manifest also downloads Zephyr's `hal_ethos_u` module. That +module is a separately maintained snapshot and its manifest revision does not +yet contain the merged multi-variant implementation. The explicit clone above +therefore remains necessary. The ancestor test is a guard rather than a pin: +it permits newer `main` revisions while rejecting an old or stale checkout +that cannot run U55 and U85 through the same driver registry. + +Clone and pin CMSIS-NN: + +```bash +git clone https://github.com/ARM-software/CMSIS-NN.git \ + modules/cmsis-nn-src +git -C modules/cmsis-nn-src checkout \ + d933672e7ca97eec70ef43230baee7b20c2a28ae +``` + +Create the Python environment used by the ExecuTorch CMake integration: + +```bash +python3.12 -m venv .venv-executorch +cd modules/lib/executorch +../../../.venv-executorch/bin/python -m pip install \ + -r requirements-examples.txt +env -u DEBUG ./install_executorch.sh +cd ../../.. +``` + +Python 3.12 is used for compatibility with the pinned ExecuTorch revision. +Removing a host `DEBUG` variable prevents ExecuTorch from interpreting a +non-numeric shell value as its numeric build option. You do not need the +optional `ethos_u` Python dependency group for the firmware build. + +Apply the sample's ExecuTorch integration patch and check the dependencies: + +```bash +./sdk-alif/samples/modules/executorch/dual_npu_vision/setup_workspace.sh +``` + +You see output similar to: + +```output +Applied ExecuTorch dual-NPU patch. +Ethos-U core driver main: ... +Workspace dependencies are ready. +``` + +If you run the script again, it reports that the patch is already applied. You +now have the sources and dependencies required for the build. diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/troubleshoot.md b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/troubleshoot.md new file mode 100644 index 0000000000..552fd953ad --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/troubleshoot.md @@ -0,0 +1,66 @@ +--- +title: Troubleshoot the demo +description: Diagnose common workspace, camera, display, packaging, and inference failures. +weight: 7 +layout: "learningpathall" +--- + +## Ninja is not available + +If CMake reports that it cannot find Ninja, install it in the active virtual environment: + +```bash +source $HOME/alif-dual-npu/.venv/bin/activate +python -m pip install ninja +``` + +Run the build again with `--pristine`. + +## The Alif flash runner cannot import fdt + +Install the Python `fdt` module in the environment that runs west: + +```bash +source $HOME/alif-dual-npu/.venv/bin/activate +python -m pip install fdt +``` + +## The camera reports chip ID 0000 or I2C error -5 + +Power off the board and check the camera connection. The supplied overlay expects the MT9M114 on J16 at the selfie-camera I2C address. Reseat the flex cable and confirm that its contacts face the correct direction. + +Do not combine a J16 overlay with a camera connected to J22. J22 uses the standard camera address and route. + +## The display D-PHY does not lock + +Confirm that the board runs SERAM 1.110.0 and that SEToolkit 1.10 generated the package. Rebuild with the full overlay list from the build section. A missing board overlay can prevent the display power and clock configuration from being applied before the Zephyr display driver starts. + +## The ISP reports no empty video buffer + +Confirm that `isp_route.overlay` is the final device-tree overlay in the build command. The application circulates five buffers and must return each processed buffer to the ISP queue. Do not remove the ISP configuration fragments from `vision.conf`. + +## The live preview is grainy or monochrome + +This symptom usually means the application is displaying packed sensor data as RGB565. Confirm that the build includes both ISP overlays and that the selected video endpoint is `isp@49046000`. The native pipeline requests planar RGB888 output from the ISP before it creates the display preview and model inputs. + +## The startup test works but the live result does not change + +Look for this message: + +```output +dual-et: live dual-NPU pipeline started +``` + +Then confirm that live-frame messages and NPU interrupt counters continue to increase. If capture errors repeat, power-cycle the board, flash the package again, and recheck the camera connector and overlay order. + +## The log reports an invalid MRAM payload + +The firmware contains offsets derived from the exact PTE, image, and label sizes used by CMake. Do not combine `zephyr.bin` from one build with `model_assets.bin` from another build. + +Remove the old build output, rebuild with `--pristine`, and copy both newly generated files into SEToolkit before running `app-gen-toc` again. + +## Model preparation reports an allocation failure + +An allocation failure usually means that the PTE files do not match the memory-pool sizes in the checked-out sample. Confirm that the SDK is at the validated commit from the workspace section and run a pristine build. If you regenerated the PTE files, confirm that you used the supplied scripts, model inputs, and `ensemble_vela.ini` file. + +These checks cover the failures most likely to affect this exact hardware and software configuration. diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/understand-architecture.md b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/understand-architecture.md new file mode 100644 index 0000000000..d27a256257 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/understand-architecture.md @@ -0,0 +1,171 @@ +--- +title: Understand the dual-NPU architecture +description: Learn how the E8 hardware, Zephyr threads, camera pipeline, memory regions, and ExecuTorch models work together. +weight: 2 +layout: "learningpathall" +--- + +The Alif Ensemble E8 combines two Arm Cortex-M55 cores with two different Ethos-U NPUs. This demo runs on the high-performance Cortex-M55 core and assigns one model to each NPU: + +| Workload | NPU | Input tensor | Output | +| --- | --- | --- | --- | +| SSD-Slim face detection | Ethos-U55-256 | 1 x 1 x 120 x 160 int8 grayscale | 1,118 box regressions and face/background logits | +| MobileNetV2 image classification | Ethos-U85-256 | 1 x 3 x 224 x 224 int8 RGB | 1,000 ImageNet class scores | + +## Reduce the system power cost + +Power-constrained embedded products often need more than one machine learning +workload, but adding another MCU also adds its active and idle power. It can +also add external memory, interprocessor communication, and board-level power +domains. The E8 avoids that duplication by letting one Cortex-M55 coordinate +both on-chip NPUs. Camera capture, preprocessing, result fusion, and the user +interface stay on one MCU while each NPU runs the workload it handles best. + +This arrangement does not make two active NPUs consume less instantaneous +power than one active NPU. It reduces system-level overhead compared with a +design that needs a separate MCU or application processor for each NPU. It can +also finish concurrent workloads sooner, allowing the system to return to an +idle or lower-power state earlier. Measure energy on the final hardware to +quantify the saving for a specific duty cycle. + +The application uses the native Zephyr drivers in the Alif SDK ``main`` branch; +the MT9M114, ISP, and MW405 changes were merged through pull request 879. The +following diagram extends the original multi-NPU prototype with the live +camera, ISP, and display pipeline used by this demo. + +![System diagram showing the MT9M114 camera and ISP feeding parallel U55 and U85 ExecuTorch workers before results are composed on the MW405 display.](dual-npu-system.svg) + +## Separate the hardware responsibilities + +One Zephyr application owns two driver objects, two register windows, two interrupt handlers, and two timing records: + +| Resource | Ethos-U55 | Ethos-U85 | +| --- | ---: | ---: | +| MMIO base | `0x400E1000` | `0x49042000` | +| NVIC interrupt | 55 | 366 | +| Worker priority | 5 | 4 | +| Primary workload | SSD-Slim | MobileNetV2 | +| Method pool | 640 KiB in HP DTCM | 304 KiB in SRAM1 | +| Temporary pool | 3,686,400 bytes in SRAM0 | 1,509,968 bytes in SRAM1 | + +The two interrupt service routines call `ethosu_irq_handler()` with the matching driver. A U85 completion therefore cannot release the U55 wait object, and a U55 completion cannot change the U85 timing record. + +## Select the NPU from model metadata + +The application enables the Ethos-U core driver's multi-variant support with +`ETHOSU_MULTI_VARIANT`. This support is part of the core-driver ``main`` +branch. During initialization, `ethosu_init_ex()` registers both physical +devices with their product descriptors. + +The main motivation for this multi-variant driver support is to let one MCU +manage different Ethos-U products in the same system. Without it, software +integration tends toward separate driver instances or separate processing +domains for each NPU variant. A common registry keeps device discovery, +interrupt handling, and workload dispatch in one Zephyr application. This +supports the lower-overhead system architecture described earlier. + +Vela stores a COP1 optimizer record in each delegated program. The record identifies the target product and MAC configuration: + +| Optimizer value | Selection | +| --- | --- | +| Product 0, log2 MACs 8 | Ethos-U55 with 256 MACs | +| Product 2, log2 MACs 8 | Ethos-U85 with 256 MACs | + +The ExecuTorch backend parses this record and asks the registry for a compatible free driver. Device selection is therefore a property of the compiled PTE model, not a hard-coded assumption in the worker thread. + +## Follow one frame through the application + +The camera and display use different MIPI D-PHY instances so they can operate at the same time. The J16 MT9M114 camera uses D-PHY 0, while the MW405 display uses D-PHY 1. + +For every live frame, the application performs these steps: + +1. The MT9M114 sends 1288 x 728 RAW10 data over MIPI CSI-2. +2. The hardware ISP crops and demosaics the image into a 192 x 192 planar RGB888 buffer. +3. Zephyr maintains five video buffers so capture continues while one frame is processed. +4. The coordinator creates a 120 x 160 grayscale SSD tensor and a 224 x 224 RGB MobileNetV2 tensor. +5. The coordinator releases both persistent worker threads through separate semaphores. +6. Each worker copies its input into its prepared ExecuTorch method and submits the delegated command stream. +7. The coordinator waits for both completion semaphores, records timing, and returns the captured buffer to the ISP queue. +8. The UI compositor draws a 480 x 352 RGB565 preview, face boxes, the classification result, and rolling timing values in the 480 x 800 framebuffer. + +ExecuTorch program and method construction is serialized because that setup path contains shared runtime state. After setup, each worker retains an independent immutable `Method`, allocator set, and NPU backend. Only prepared model execution runs in parallel; camera capture, preprocessing, result fusion, and display updates remain coordinated by the Cortex-M55. + +## Coordinate the workers + +The coordinator uses three synchronization stages: + +1. Each worker signals that its `Program` and `Method` are prepared. +2. The coordinator gives both execute semaphores for the current frame. +3. Each worker signals once it has copied the input and again after inference completes. + +The early input-copy signal lets the coordinator return the camera buffer to the ISP queue without waiting for both NPUs. The later completion signal protects result processing and timing calculations. + +The application records execution start and end cycles inside each worker, close to `Method::execute()`. It derives four live metrics: + +| Metric | Calculation | +| --- | --- | +| U55 | U55 end minus U55 start | +| U85 | U85 end minus U85 start | +| Span | Latest end minus earliest start | +| Overlap | Earlier end minus later start, or zero when executions do not overlap | + +These measurements isolate delegated execution from camera capture and UART output. They do not represent complete camera-to-display latency. + +## Understand the memory layout + +The application separates persistent artifacts, CPU-private state, display and video buffers, and U85-visible working memory. + +![Memory diagram showing models and firmware in MRAM, U55 state in HP DTCM, U55 temporary storage in SRAM0, and the display, U85 working memory, and camera buffers in SRAM1.](dual-npu-memory-layout.svg) + +### Persistent MRAM payload + +SEToolKit writes `model_assets.bin` at `0x80008000` and the execute-in-place Zephyr image at `0x80400000`: + +| Address range | Size | Contents | +| --- | ---: | --- | +| `0x80008000`-`0x8035751F` | 3,470,624 bytes | Ethos-U85 MobileNetV2 PTE | +| `0x80357520`-`0x8039FCDF` | 296,896 bytes | Ethos-U55 SSD-Slim PTE | +| `0x8039FCE0`-`0x803BAD15` | 110,646 bytes | Grace Hopper startup image | +| `0x803BAD16`-`0x803BD60D` | 10,488 bytes | ImageNet class labels | +| From `0x80400000` | Build-dependent | RTSS-HP Zephyr firmware | + +The combined `model_assets.bin` payload is 3,888,654 bytes. CMake packs both PTE files, the startup image, and the labels in this order, then generates a header containing the artifact sizes. Changing a model causes CMake to reconfigure so the compiled offsets cannot silently disagree with the payload. + +### HP DTCM + +The 1 MiB HP DTCM region starts at `0x20000000`. It contains the 640 KiB U55 method pool, both 4 KiB metadata pools, the fast-scratch arrays, Zephyr worker stacks, semaphores, driver objects, and ISP library state. The larger U55 temporary arena is placed in SRAM0. The linker can move individual DTCM symbols as code changes, so the application fixes their sizes rather than their exact addresses. + +### Shared SRAM0 + +SRAM0 spans `0x02000000`-`0x023FFFFF`. The validated build dedicates most of it to the U55 delegate scratch arena: + +| Address range | Reserved size | Use | +| --- | ---: | --- | +| `0x02000000`-`0x02383FFF` | 3,686,400 bytes | U55 temporary allocator pool | + +SSD-Slim requires a larger delegated scratch plan than the earlier face detector. Keeping this arena outside HP DTCM leaves enough private memory for the method pool and Zephyr runtime. + +### Shared SRAM1 + +SRAM1 spans `0x02400000`-`0x027FFFFF` and holds the display, U85-visible working set, and camera buffers: + +| Address range | Size | Use | +| --- | ---: | --- | +| `0x02400000`-`0x024BB7FF` | 768,000 bytes | CDC200 480 x 800 RGB565 framebuffer | +| `0x024BB800`-`0x025437FF` | 557,056 bytes | U85 command and weight mirror reservation | +| `0x02543800`-`0x026B424F` | 1,509,968 bytes | U85 temporary allocator pool | +| `0x026B4250`-`0x026F024F` | 304 KiB | U85 ExecuTorch method pool | +| `0x02700250`-`0x0278C24F` | 560 KiB | Five-buffer Zephyr video heap | + +The active ISP output uses five 192 x 192 x 3-byte RGB888 buffers of 110,592 bytes each. The heap allows up to 114,688 bytes per buffer. The backend copies delegated command or weight data into the mirror reservation when its original address is not directly usable by U85. Shared SRAM also avoids consuming the limited HP DTCM with the U85 working set. + +## Maintain cache and address visibility + +The Cortex-M55 data cache is not coherent with the camera, display controller, or either NPU. The application follows two rules: + +- Clean or flush CPU-written input tensors and framebuffer pixels before a device reads them. +- Invalidate captured frames and NPU-written output tensors before the CPU reads them. + +The U55 and U85 also require system-visible addresses. The platform layer translates CPU-local addresses to their Alif global aliases and selects the correct AXI memory attributes for MRAM, HP DTCM, SRAM0, and SRAM1. Incorrect address translation or memory attributes can produce an NPU bus error even when the CPU can read the same bytes. + +The next section prepares the board and a clean west workspace for this architecture. diff --git a/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/validate-inference.md b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/validate-inference.md new file mode 100644 index 0000000000..bd273c0406 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/alif-dual-npu-vision/validate-inference.md @@ -0,0 +1,72 @@ +--- +title: Validate live parallel inference +description: Verify the startup image, live camera UI, NPU interrupts, and rolling parallel timing values. +weight: 6 +layout: "learningpathall" +--- + +The application validates the inference path before it consumes camera frames. This separates model or NPU failures from camera and ISP failures. + +## Check the startup test + +After reset, the display shows the bundled Grace Hopper image. The U55 model draws one green face box, and the U85 result identifies an ImageNet class such as `ACADEMIC GOWN`. + +The U4 log includes model preparation and isolated preflight messages similar to: + +```output +dual-et: starting parallel worker threads +dual-et: SSD persons=1 candidates=... +dual-et: frame=0 CLASS=400 ACADEMIC GOWN confidence=...% +dual-et: isolated U55 done irqs=1/0 +dual-et: isolated U85 done irqs=1/1 +dual-et: startup self-test passed; switching to camera in 5 seconds +``` + +The exact inference times and memory addresses vary between builds. + +## Check the live camera UI + +After five seconds, the application clears the test image and starts the MT9M114 stream. Confirm these results: + +- The 480 x 352 live preview appears near the center of the MW405 display. +- The preview is in color and updates when you move the camera. +- Green boxes track faces in the live frame. +- The classification label and confidence change with the scene. +- Face boxes appear when SSD-Slim detects a face. +- The lower status area shows rolling U55, U85, span, and overlap values. + +The log confirms the transition: + +```output +dual-et: live dual-NPU pipeline started +``` + +## Confirm both NPUs execute for each frame + +The coordinator wakes both Zephyr worker threads before waiting for their completion. Every tenth frame, the application prints the current and average timing values: + +```output +dual-et: PAR class=... frame=10 sample=10 U55=... U85=... span=... overlap=... us avg U55/U85/span/overlap=.../.../.../... us +``` + +Use the fields as follows: + +| Field | Meaning | +| --- | --- | +| `U55` | SSD-Slim `Method::execute()` wall time on Ethos-U55 | +| `U85` | MobileNetV2 `Method::execute()` wall time on Ethos-U85 | +| `span` | Time from the first worker starting to the last worker finishing | +| `overlap` | Time during which both model executions overlap | + +These are Cortex-M cycle-counter wall-time measurements around delegated `Method::execute()`, not NPU PMU active-cycle counts. Input preparation and result decoding are outside these intervals. The two NPU interrupt counters also increase during live operation: + +```output +dual-et: live frame=30 IRQs=.../... +``` + +Changing the scene changes the preview, tensor checksums, and detection results. If the image and results remain fixed, the application is still using the startup test input instead of live frames. + +You have now built, flashed, and validated a Zephyr application that drives +two Ethos-U NPUs with separate persistent worker threads on one Cortex-M55. +This design runs both ML workloads without adding a second MCU and its +associated system power overhead.