diff --git a/.gitignore b/.gitignore index bd58df0f..f3829b9c 100644 --- a/.gitignore +++ b/.gitignore @@ -125,3 +125,8 @@ __pycache__/ # the site by moondeck/docs/mkdocs_hooks.py (via gen_api.py). Regenerated every build. /docs/moonmodules/core/moxygen/ /docs/moonmodules/light/moxygen/ + +# Read-only hardware snapshots (flash/NVS dumps pulled off reverse-engineered boards). +# Root-anchored + dir-scoped, like /build/ above: a bare `.snapshots` would also swallow any +# nested path of that name anywhere in the tree. +/.snapshots/ diff --git a/CLAUDE.md b/CLAUDE.md index 1b29349b..f446b6a4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -79,6 +79,8 @@ The **one exception** is `esp32/main/CMakeLists.txt`: ESP-IDF builds use IDF's b **The one nuance: a *rigorous* change gets a heads-up first.** "Flash freely" is the default for ordinary iteration (a code fix, a UI tweak, a normal reflash to see it run). But when the change is *rigorous* — it could leave a board in a bad state or is disruptive to recover from: erasing/repartitioning flash, a bootloader / partition-table / sdkconfig change, changing the flash-size variant, a first-flash of an untested board, a long full-erase, or anything that risks bricking or a bootloop — **say what you're about to do and why, and wait for the go-ahead.** The test is *reversibility*: a routine reflash is a keystroke to redo, so just do it; a change that could cost real recovery effort (or hardware) is worth one sentence of confirmation first. When unsure which side a change falls on, ask. +**Invite the product owner to test — then STOP and wait.** Whenever a change produces something the product owner can *see or judge* (LEDs on a bench board, a UI screen, a rendered effect, a live device behaviour), the agent's job ends at "it's running on , here's what to look at" — **not** at the agent's own verdict. Say what changed, where to look, and what would count as good or bad, then **stop and wait for their observation before drawing conclusions, writing them into docs, or moving to the next step.** The product owner's eyes are the measurement; serial logs and API reads are supporting evidence, not a substitute. This is the *[Agent Roles](#agent-roles)* division made concrete: the product owner tests on hardware before approving, and an agent that races ahead — reaching a conclusion, updating a doc, starting the next task — has quietly taken that decision away from them and is often *wrong* (this rule exists because it happened repeatedly: conclusions written up from a bench the product owner never got to look at). The trigger is simply *"could the product owner see this?"* — if yes, hand it over and wait. Doubly so before anything irreversible-ish (a revert, a reconfigure, a reflash) that would destroy the very state they were about to look at: leave it running. + The full gate lists per lifecycle event (commit, push, PR merge, release) live in **Lifecycle Events** below. **Mandatory subtraction.** Periodically review and remove code and docs that no longer earn their place. If nothing can be removed, justify why. This applies to `docs/backlog/` and `docs/history/` too: both grow *and shrink*. A backlog item that ships is deleted (it lives in the code now); a history entry whose lesson has been absorbed — folded into a principle, a doc, or simply internalised — gets pruned. The permanent record is the git commits; these two folders are a working narrative layered on top, kept only as long as they still earn it. Don't treat either as append-only. @@ -203,6 +205,7 @@ docs/ building.md ← how to build, flash, run for every target testing.md ← test inventory and strategy performance.md ← per-module timing, memory, sizeof for each platform + MIGRATING.md ← breaking-change log (newest first): what changed + the action it costs the user backlog/ ← forward-looking: what to build next (not present-tense) README.md ← landing page: overview of every item + index (the rest of the system links here, not into items) backlog-core.md ← to-build list, core / infrastructure domain (+ UI) diff --git a/CMakeLists.txt b/CMakeLists.txt index 0bb2093d..a0c03095 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -77,14 +77,17 @@ target_include_directories(mm_platform PUBLIC src/ src/platform/desktop/) # Winsock for the desktop socket surface on Windows. target_link_libraries(mm_platform PUBLIC $<$:ws2_32>) -# Generate build_info.h from library.json (carries version, build date, board name). -add_custom_command( - OUTPUT ${CMAKE_SOURCE_DIR}/src/core/build_info.h +# Generate build_info.h from library.json + git (carries version, build id, build date, board name). +# ALWAYS out-of-date on purpose — see the same rule in esp32/main/CMakeLists.txt for why: MM_BUILD_ID +# must track the git hash on every build, and pinning this to library.json's mtime is what made the +# reported build stamp go stale, which reads as a failed flash and sends debugging at the wrong binary. The generator +# rewrites the header only when its content changes, so an unchanged tree triggers no rebuild. +add_custom_target(build_info_gen ALL COMMAND ${UV_EXECUTABLE} run python ${CMAKE_SOURCE_DIR}/moondeck/build/generate_build_info.py - DEPENDS ${CMAKE_SOURCE_DIR}/library.json ${CMAKE_SOURCE_DIR}/moondeck/build/generate_build_info.py - COMMENT "Generating build_info.h" + BYPRODUCTS ${CMAKE_SOURCE_DIR}/src/core/build_info.h + COMMENT "Generating build_info.h (git build id)" + VERBATIM ) -add_custom_target(build_info_gen DEPENDS ${CMAKE_SOURCE_DIR}/src/core/build_info.h) # Embed UI files as C arrays. Pass UV_EXECUTABLE through as a single value # (no semicolons on the command line — those would either get split by `make` diff --git a/docs/MIGRATING.md b/docs/MIGRATING.md new file mode 100644 index 00000000..ca45b37f --- /dev/null +++ b/docs/MIGRATING.md @@ -0,0 +1,78 @@ +# Migrating + +The log of **breaking changes** — what changed between versions, and the action to take. + +projectMM ships **no migration code**: the persistence layer is robust by default (an absent key keeps the control's default, a stale value clamps to the new bounds, an unknown key is ignored), which absorbs almost all schema drift with zero migration-specific code. The rare change that a robust reader *cannot* absorb is **documented here instead of migrated** — see [ADR-0013](adr/0013-no-migration-code-robust-persistence-plus-documented-breaks.md) for the decision and its rationale. + +**Read this when upgrading a device that already holds persisted state.** Entries are newest first. Each says what changed and what to do; most need nothing at all, because the lost value re-populates on next use. + +**Action legend** — how much work an entry costs you: + +| Action | Meaning | +|---|---| +| *nothing* | Self-heals. The value re-populates on next use, or the default is correct. | +| *re-set a control* | One value resets to its default; set it again in the UI if you had changed it. | +| *re-add a module* | The module vanishes from the tree on boot; add it again and re-enter its controls. | +| *update a file* | An on-device file must be edited or replaced. | +| *erase flash* | A full flash erase is required (the heaviest — a full reconfigure follows). | + +--- + +## Unreleased (`next-iteration`) + +### `MoonLedDriver`: `forceRing` → `useRing`, and the ring's geometry is now settable (2026-07-17) + +The pin-expander path selector was a three-option Select (`auto` / `ring` / `wholeFrame`) named for a *diagnostic override*. The auto-router is gone — at the size the expander exists for (48 strands × 256 lights) a whole frame never fits internal DMA RAM, so "auto" had exactly one right answer while presenting itself as a choice, and its silent fallback hid which path was actually running. What remains is the honest question, as a switch: + +| Old | New | +|---|---| +| control `forceRing` (Select: `auto`/`ring`/`wholeFrame`) | `useRing` (a switch: on = ring, off = whole frame) | +| — | `ringRows` (new: lights per DMA buffer, 1..64) | +| — | `ringBufs` (new: buffers the DMA circulates, 2..32) | + +**Action: re-set `useRing` if you had `forceRing` on `wholeFrame`.** + +`forceRing` reads as absent → ignored, and `useRing` takes its default (**on**, the ring). A device that had explicitly selected whole-frame therefore comes up on the ring; flip `useRing` off to get it back. `ringRows`/`ringBufs` default to 16 and 12 — the geometry the driver effectively ran. (It shipped with a pool of 16, but 16 buffers never fit the S3's internal DMA heap, so the ring build failed its own fit check and the driver quietly fell back to whole-frame; 12 is what actually held. A config on the old defaults may therefore start *ringing* where it used to fall back.) They exist so the RAM / encode-overhead / interrupt-rate / lap-time trade-off can be swept on a live board rather than fixed at compile time. + +### LED driver + control rename — a human-readable UI (2026-07-16) + +The LED driver module types and several controls were renamed so the UI reads in plain language rather than peripheral jargon (the UI shows a control's name verbatim, so the name *is* the label). + +| Old | New | +|---|---| +| module type `I80LedDriver` | `MultiPinLedDriver` | +| module type `MoonI80LedDriver` | `MoonLedDriver` | +| control `shiftRegister` | `pinExpander` | +| control `asyncTransmit` | `doubleBuffer` | +| read-only `wireUs` | `frameTime` | +| read-only `stall` (Drivers) | `renderWait` | + +**Action: re-add the module, then re-set `pinExpander` / `doubleBuffer` if you had changed them.** + +A device whose persisted config names the old module type loads a module type that no longer exists — the unknown type is ignored, so **the driver is absent from the tree on boot**. Re-add it (`MultiPinLedDriver` or `MoonLedDriver`) and re-enter its controls. Within a re-added driver, the two renamed *settable* controls (`pinExpander`, `doubleBuffer`) read as absent → they take their defaults (`pinExpander` off, `doubleBuffer` on); set them again if your board needs otherwise. `frameTime` and `renderWait` are read-only KPIs — nothing to restore. + +`RmtLedDriver` and `ParlioLedDriver` are unchanged. The `pins` / `ledsPerPin` / `clockPin` / `latchPin` / `loopback*` controls are unchanged. + +--- + +## Earlier + +These pre-date this log and were recorded in ADR-0013's Consequences list. A device that persisted state on an older build and loads a newer one loses only the noted value, which re-populates on next use. + +### Container config filenames (`LayoutGroup.json` → `Layouts.json`) + +`.config/LayoutGroup.json` → `Layouts.json`, `.config/DriverGroup.json` → `Drivers.json` (container type rename). The stale files are ignored (unknown-type config isn't loaded); they linger harmlessly on disk until a flash erase. + +**Action: nothing.** Lost: the container's `enabled` flag (defaults back on). + +### UI last-selected module (`mm.selectedModule` → `mm_selected`) + +The browser localStorage key for the UI's last-selected module. + +**Action: nothing.** Lost: the remembered selection resets to the first module. + +### Device-list `colour` → `color` (US-spelling rename) + +The DevicesModule persisted-list key for a Hue bridge's color-capable light count (`DevicesModule::restoreList()`). A device list persisted under the old key reads the count as absent → 0. + +**Action: nothing.** The cached bridge count resets to 0 until the bridge is re-heard live and re-populates it. diff --git a/docs/adr/0013-no-migration-code-robust-persistence-plus-documented-breaks.md b/docs/adr/0013-no-migration-code-robust-persistence-plus-documented-breaks.md index 90956305..42823da5 100644 --- a/docs/adr/0013-no-migration-code-robust-persistence-plus-documented-breaks.md +++ b/docs/adr/0013-no-migration-code-robust-persistence-plus-documented-breaks.md @@ -12,7 +12,7 @@ The realization that settles it: **the persistence layer is already robust to sc ## Decision -**No migration code in the system.** Persistence stays robust-by-default (absent → default, stale → clamp, unknown → ignore), which absorbs almost all schema drift with zero migration-specific code. A breaking format change (a persisted key, config filename, wire key, or localStorage key that a *released* version wrote and a new version reads differently) is **documented, not migrated** — recorded in this ADR's Consequences list below, one line per break, so the record is present-tense prose rather than past-narrating code. +**No migration code in the system.** Persistence stays robust-by-default (absent → default, stale → clamp, unknown → ignore), which absorbs almost all schema drift with zero migration-specific code. A breaking format change (a persisted key, config filename, wire key, or localStorage key that a *released* version wrote and a new version reads differently) is **documented, not migrated** — recorded in [MIGRATING.md](../MIGRATING.md), one entry per break with the action it costs the user, so the record is present-tense prose rather than past-narrating code. A **patching framework is explicitly deferred**, not rejected forever: it becomes the right tool post-1.0 *if* breaking format changes become frequent enough that ad-hoc losses pile up (a rough bar: >5 across a few releases) *and* users hold persisted state too valuable to re-derive. At that point build the recognizable version-stamp + ordered-patch-chain pattern (SQLite `user_version`, Rails/Room migrations), not a bespoke one. Until then, the cost of not migrating is a re-populated setting, which the robust-by-default layer already makes cheap. @@ -20,8 +20,5 @@ A **patching framework is explicitly deferred**, not rejected forever: it become - The two existing migrations were removed: `FilesystemModule::migrateRenamedConfigs()` (and its call + declaration) and the `lsRead` legacy-key parameter. The proposed `colour`-fallback was not added. - **Robust-by-default is the contract**: a schema change must degrade gracefully through absent-default / clamp / ignore, never crash or wedge (the *Robust to any input* principle already guards this in tests). A change that would lose data silently is documented here instead of papered over with a fallback. -- **Known breaking changes** (a device that persisted state on an older build and loads a newer one loses only the noted value, which re-populates on next use): - - `.config/LayoutGroup.json` → `Layouts.json`, `.config/DriverGroup.json` → `Drivers.json` (container type rename). The stale files are ignored (unknown-type config isn't loaded); they linger harmlessly on disk until a flash erase. Lost: the container's `enabled` flag (defaults back on). - - localStorage `mm.selectedModule` → `mm_selected` (UI last-selected module). Lost: the remembered selection resets to the first module. - - DevicesModule persisted-list key `colour` → `color` (a Hue bridge's color-capable light count, in `DevicesModule::restoreList()`). A device list persisted under the old key reads the count as absent → 0; the cached bridge count resets to 0 until the bridge is re-heard live and re-populates it. +- **Known breaking changes are logged in [MIGRATING.md](../MIGRATING.md)** — one entry per break, newest first, each with the action it costs the user (usually none: the value re-populates on next use). That log is the home for this decision's output; it lives outside the ADR because it *grows* with every break, and an ADR is immutable (superseded, never edited). - The lesson that generalizes: a robust reader is worth more than a migration — build the loader to tolerate drift, and most migrations never need to exist. Reach for a patching framework only when real, frequent, high-value breaks prove it earns its complexity. diff --git a/docs/adr/0014-own-i80-dma-driver-below-esp-lcd.md b/docs/adr/0014-own-i80-dma-driver-below-esp-lcd.md new file mode 100644 index 00000000..7438ad51 --- /dev/null +++ b/docs/adr/0014-own-i80-dma-driver-below-esp-lcd.md @@ -0,0 +1,72 @@ +# 14. Our own i80 DMA driver, one level below esp_lcd + +Date: 2026-07-14 + +## Status + +Accepted + +## Context + +`I80LedDriver` drives parallel WS2812 output through ESP-IDF's `esp_lcd` i80 bus. It pre-encodes a whole frame and sends it as **one** `esp_lcd_panel_io_tx_color` transaction, which is gapless and correct — the peripheral streams the buffer and stops. + +That single-transaction design is also the driver's ceiling. The DMA must read the whole frame in one unbroken stream, so the frame has to be somewhere the DMA can sustain, in one contiguous allocation. The measured consequences: + +- The **74HCT595 expander** renders correctly only while its ×8 frame fits internal DMA RAM — about **96 lights per strand** on the ESP32-S3. Above that the frame lands in PSRAM and the strands garble. +- **Parlio** caps at ~**4,096 lights**: a hardware 65,535-byte single-transfer limit, and a contiguous-block limit below that. +- The **classic ESP32** caps at ~**2,048 lights**: its i80 backend is the I2S peripheral, whose DMA cannot address PSRAM at all. + +The obvious fix is to split the frame into several transactions. **We built that, and it does not work.** On a dense frame the strands flash full-brightness white; on a sparse frame the fault hides. The cause is in IDF, not in our code — every transaction starts with `lcd_start_transaction` (`esp_lcd/i80/esp_lcd_panel_io_i80.c`): + +```c +lcd_ll_reset(bus->hal.dev); // reset the LCD peripheral +lcd_ll_fifo_reset(bus->hal.dev); // flush the FIFO +gdma_start(...); +esp_rom_delay_us(4); // hard-coded busy-wait +lcd_ll_start(bus->hal.dev); +``` + +**`esp_lcd` resets the peripheral between transactions.** For an LCD panel that is harmless — a panel is addressed, not clocked continuously. For WS2812 it is fatal: the protocol is one unbroken self-clocked bit stream, and a mid-frame reset corrupts everything after it. IDF's Parlio driver resets its FIFO in the same place (`esp_driver_parlio/src/parlio_tx.c`). So **no chunking strategy inside IDF's LED-adjacent drivers can be gapless**, at any chunk size or boundary. This is why hpwit's driver — the reference implementation for this class of LED output — hand-rolls its DMA rather than using IDF's. + +There is, however, an opening. The LCD peripheral has **no data-length register**: `lcd_ll_set_phase_cycles()` sets `lcd_dout` as a *boolean enable*, and IDF's own comment reads *"Number of data phase cycles are controlled by DMA buffer length"*. The peripheral clocks out exactly what the DMA feeds it and stops when the chain ends. Therefore **one `gdma_start()` over an arbitrarily long descriptor chain, plus one `lcd_ll_start()`, is a single continuous gapless stream spanning as many buffers as we like.** `esp_lcd` discards that capability by re-arming per transaction; the hardware never required it. + +The descriptors are almost free: a 144 KB frame (16 lanes × 1,024 lights) needs 37 descriptors — 444 bytes. + +## Decision + +**Build a second i80 implementation, `MoonI80`, on IDF's HAL and GDMA link-list APIs (`lcd_ll_*`, `gdma_link_*`) — one level below `esp_lcd` — and ship it alongside the existing driver rather than replacing it.** + +Three parts to the decision: + +1. **One level below `esp_lcd`, not down to the registers.** `gdma_link_*` and the LCD HAL are the APIs IDF's own drivers are built on. We are declining `esp_lcd`'s transaction *policy*, not its abstractions. No raw register pokes. + +2. **The whole frame in one descriptor chain (phase 1).** We already pre-encode the frame, so the DMA can simply read it — no ISR refill, no ring, no real-time deadline, and therefore no underrun for WiFi to cause. This is strictly simpler than hpwit's design, which needs a CPU refill only because it transposes per-LED. + +3. **Both drivers ship.** `I80LedDriver` remains the default and the **reference implementation**: correct, memory-capped, and the thing MoonI80 is measured against. MoonI80 is the challenger. It replaces the reference only when it demonstrably beats it on the same bench. Both are registered module types, so the A/B is a swap in the UI with no reflash. + +An **internal-RAM ring with CPU refill** (hpwit's shape, and the only thing that can ever work on the classic ESP32) is deferred to a phase 2, and **gated on phase 1 measuring that the silicon — not `esp_lcd` — is the wall.** The ring is a superset of the same descriptor machinery (`GDMA_FINAL_LINK_TO_HEAD` closes the chain), so it is an extension, not a rewrite. + +## Consequences + +**What we gain in phase 1, precisely.** We own the descriptor chain and fire it with a single `gdma_start` + `lcd_ll_start`, so `esp_lcd`'s per-transaction re-arming (the peripheral reset that corrupts a WS2812 stream) is gone, and with it the whole `lli full` mount-failure class — the chain is mounted once, owner-checking off. **That is all phase 1 gains.** It does *not* yet lift a memory ceiling: it still mounts one contiguous frame buffer per transfer, exactly as `esp_lcd` did. And it does not touch Parlio, whose 65,535-byte and contiguous-block caps are its own peripheral's, not `esp_lcd`'s. + +What phase 1 *buys* is the **capability** the ceilings need: a chain we control, which phase 2 closes into a ring over small internal buffers. Only then does the frame stop needing to be one contiguous DMA-reachable block. + +**What we give up, and it is real.** This diverges from *[Industry standards, our own code](../../CLAUDE.md#principles)* — we are leaving a maintained IDF driver for code we own. The justification is that the maintained driver **cannot express the behaviour the hardware supports and WS2812 requires**, and that is demonstrated from IDF's source, not assumed. But we now carry: the GPIO/clock/bus setup `esp_lcd` was doing for us, the interrupt plumbing, and the risk of drifting against future IDF versions. Keeping `I80LedDriver` as the reference is the mitigation — if MoonI80 rots, the working path is still there and still default. + +**What phase 1 measured (2026-07-14, board B — the question this ADR was written to settle).** + +MoonI80 renders correctly on real hardware: the SE16 at 4,096 lights (direct, 16 lanes) and board B through the 74HCT595 expander, both confirmed by eye, with **zero GDMA mount failures** and a wire time matching the `esp_lcd` reference to within 0.15% (19,646 µs vs 19,674 µs — so our own peripheral configuration produces the same waveform). + +And it answered the open question, by removing the suspect rather than reasoning about it: + +| pixel clock | frame in PSRAM | result | +|---|---|---| +| **2.67 MHz** (direct) | 2,048 lights | **drives** — 7,712 µs on the wire | +| **26.67 MHz** (expander) | *any* size — 54 KB or 144 KB | **never completes** | + +Same board, same PSRAM, same descriptor chain, same driver. The only variable is the clock. **The S3's GDMA cannot sustain a PSRAM read at the expander's 10× rate** — a '595 is serial-in, so each WS2812 slot is shifted out over 8 bus words, and the bus must run ten times faster to keep the slot's duration. + +This **kills the hypothesis that motivated the build**: the `esp_lcd` path failed with thousands of `lli full` descriptor-mount errors, which pointed hard at its descriptor handling. MoonI80 removes that mechanism entirely (own chain, mounted once, owner-checking off) — the mount errors are gone, and the transfer still never completes. The `lli full` storm was a **symptom, not the cause**. Six earlier hypotheses about this bug were proposed and refuted (see [lessons.md](../history/lessons.md)); this is the first one killed by a controlled experiment with a working control condition rather than by a story that stopped fitting. + +**Consequence: phase 2 is now justified by measurement, and this driver is its foundation.** The fix is the internal-RAM ring — close the chain into a ring (`GDMA_FINAL_LINK_TO_HEAD`) over small *internal* buffers and refill them from the PSRAM frame in our own EOF callback, a bulk sequential CPU read, so the DMA never reads PSRAM at the expander's clock at all. Every piece of that (our own link list, our own EOF hook, one continuous `lcd_ll_start` that is never re-armed) exists *only* because we own the DMA; `esp_lcd` can express none of it. The ring extends this machinery rather than replacing it, exactly as the decision above anticipated. diff --git a/docs/adr/0015-library-is-a-tag-not-a-folder.md b/docs/adr/0015-library-is-a-tag-not-a-folder.md new file mode 100644 index 00000000..cfb8e3fc --- /dev/null +++ b/docs/adr/0015-library-is-a-tag-not-a-folder.md @@ -0,0 +1,35 @@ +# 15. The source tree splits by domain/type; library origin is a tag, not a folder + +Date: 2026-07-06 + +## Status + +Accepted + +## Context + +A module carries three orthogonal axes: its **domain** (`core` vs `light`), its **type** (effect / modifier / layout / driver), and its **library** (the origin it was learned from (MoonLight, WLED, MoonModules, projectMM-native). The `src/`, `docs/`, `test/`, and `assets/` trees all had to pick which axes become folders. + +Domain and type are unambiguous: every module has exactly one domain and one type. Library is not: an effect's origin is frequently *blended*, not a single fact: `DistortionWavesEffect` cites MoonLight + WLED + v1 + v2; `GameOfLifeEffect` cites MoonLight + MoonModules + v1; several modules have no clear single origin. A folder axis forces one answer to a multi-valued question, and a wrong or shifting answer costs a multi-file move (src + assets + tests + the registered doc path). Library also duplicates a dimension the `tags()` emoji already carries, and the emoji can carry *several* origins where a folder cannot. The end user does not care about a module's library except as a UI filter, which the emoji chip already provides. + +## Decision + +The tree is **` / / Module`**, flat within a type. Library does **not** become a folder level; it rides where it is free and non-duplicative: + +- **In code / assets / tests:** the `tags()` emoji (drives the UI origin-filter; may be multi-valued). Leaf files are flat within their type folder: `src/light/effects/DistortionWaves.h`, `docs/assets/light/effects/DistortionWaves.gif`, `test/unit/light/unit_DistortionWaves.cpp`. +- **In docs:** library rides in the **page** dimension, not a folder: one catalog page per type (`effects.md`) with library *sections* inside, splitting to per-library page *names* (`effects_wled.md`) only when a section outgrows its page. A doc page is forgiving about fuzzy origin: a blended-lineage effect goes on one page with its full origin in the row's tags, and mis-filing is a one-line edit, not a multi-file move. `docs` is thus the one area where `type` is expressed as part of a page name rather than a folder, because the docs compact to per-type/per-library pages, and library, the only axis with an explosion problem, rides along in that name. + +| | core/light | type | leaf | library | +|---|---|---|---|---| +| **src** | `light/` | `effects/` | `DistortionWaves.h` | tag in `tags()` | +| **assets** | `light/` | `effects/` | `DistortionWaves.gif` | — | +| **tests** | `light/` | `effects/` | `unit_DistortionWaves.cpp` | — | +| **docs** | `light/` | the page name (`effects.md`, later `effects_.md`) | (row inside) | the page split | + +## Consequences + +Every drawback of library-as-folder is dropped at once: fuzzy-origin filing, two-places-disagree, reclassification churn, sparse subfolders, and deep paths all disappear. A module's origin can be blended or can change without a file move; only the `tags()` emoji and, at most, a one-line catalog-row edit change. + +The one open growth path (non-blocking): when a library's section outgrows its catalog page, split it to a per-library page name (`effects_wled.md`, …), a lift, not a rewrite, since the flat page names and within-page sections are already in place for it. + +The live catalog pages (`docs/moonmodules/light/{effects,modifiers,layouts}.md`) and `docs/coding-standards.md` cite this decision for *why* the tree is shaped the way it is. diff --git a/docs/adr/README.md b/docs/adr/README.md index e20c6bfa..723d75e6 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -23,3 +23,5 @@ Agents do not read this directory automatically, only when a decision's rational | [0011](0011-data-exchange-pull-and-prepare-pass-not-pubsub.md) | Inter-module data/events: pull + prepare-pass, not pub/sub | Accepted | | [0012](0012-ha-discovery-wled-default-mqtt-opt-in.md) | HA discovery: WLED by default, MQTT discovery opt-in | Accepted | | [0013](0013-no-migration-code-robust-persistence-plus-documented-breaks.md) | No migration code — robust persistence + documented breaks | Accepted | +| [0014](0014-own-i80-dma-driver-below-esp-lcd.md) | Our own i80 DMA driver, one level below esp_lcd | Accepted | +| [0015](0015-library-is-a-tag-not-a-folder.md) | The source tree splits by domain/type; library origin is a tag, not a folder | Accepted | diff --git a/docs/architecture.md b/docs/architecture.md index 4a1cc445..e8159115 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -469,14 +469,14 @@ The shared output buffer is necessary when blend+map writes to arbitrary physica Each driver (a MoonModule) speaks one protocol: -- **LED drivers**: WS2812 via RMT (multi-pin), plus two parallel-output paths on the newer chips. The S3's LCD_CAM i80 bus ([LcdLedDriver](moonmodules/light/moxygen/LcdLedDriver.md)) drives exactly 8 data GPIOs — the i80 bus claims every data line of its width, so a partial set is rejected. The P4's Parlio peripheral ([ParlioLedDriver](moonmodules/light/moxygen/ParlioLedDriver.md)) drives 1–8 lanes — it takes the data GPIOs directly, so any count up to 8 is valid. Both are DMA-driven. Platform-specific; all behind the platform boundary. +- **LED drivers**: WS2812 via RMT (multi-pin), plus two parallel-output paths on the newer chips. The S3's LCD_CAM i80 bus ([MultiPinLedDriver](moonmodules/light/moxygen/MultiPinLedDriver.md)) drives exactly 8 data GPIOs — the i80 bus claims every data line of its width, so a partial set is rejected. The P4's Parlio peripheral ([ParlioLedDriver](moonmodules/light/moxygen/ParlioLedDriver.md)) drives 1–8 lanes — it takes the data GPIOs directly, so any count up to 8 is valid. Both are DMA-driven. Platform-specific; all behind the platform boundary. - **DMX / ArtNet**: sends DMX over UDP. Supports addressable LEDs and conventional DMX fixtures (pars, moving heads, dimmers). - **Preview**: streams light data to the web UI via WebSocket. - **Desktop output**: SDL2 or terminal for visual preview. Desktop also serves as a high-speed processing node, driving lights via ArtNet/DDP over the network. Each driver child reads from the Drivers container's output buffer. Everything before the Drivers container is platform-independent. -**Output correction** turns logical RGB into the physical signal: **brightness** scaling, channel **reorder** (RGB→GRB via a *light preset*), and **white** derivation for RGBW. The Drivers container owns the shared correction state (a brightness LUT + the light-preset, exposed as `brightness` / `lightPreset` controls) and hands each physical driver a `const Correction*`; the driver applies it per-light into its own buffer/packet. Preview is exempt (it shows the raw logical buffer). The brightness LUT rebuilds on the cheap `onControlChanged` tier ([§ Event triggering](#event-triggering-between-modules)), so the slider stays fluent. +**Output correction** turns logical RGB into the physical signal: **brightness** scaling, channel **reorder** (RGB→GRB via a *light preset*), and **white** derivation for RGBW. The Drivers container owns the global `brightness`; each driver picks its own light preset (its `preset` control) and applies the correction per-light into its own buffer/packet, so two strips on one device can be wired differently. Preview is exempt (it shows the raw logical buffer). The brightness LUT rebuilds on the cheap `onControlChanged` tier ([§ Event triggering](#event-triggering-between-modules)), so the slider stays fluent. Network-based drivers (ArtNet, E1.31, DDP) pace their output with a **non-blocking elapsed-time gate**, never a blocking wait (no `delay`/`vTaskDelay` — that would stall the single-threaded tick, the hot-path rule). The gate is the `lastSendTime`/`millis()` pattern: `if (now − lastSendTime < interval) return;` early-exits the tick so every other module's loop keeps running, exactly how FPS limiting works (`NetworkSendDriver`, `fps` control). **Frame-rate pacing is required** and implemented this way. **Inter-packet pacing** (spacing the universes within one frame) uses the same non-blocking gate *if* a receiver drops packets under a burst — it is not needed by default (the bench ArtNet matrix test runs clean bursting the universes), so it is added only when a target requires it, never as a busy-wait between packets. diff --git a/docs/assets/deviceModels/hpwit-shift-register-board.jpg b/docs/assets/deviceModels/hpwit-shift-register-board.jpg new file mode 100644 index 00000000..47d3260b Binary files /dev/null and b/docs/assets/deviceModels/hpwit-shift-register-board.jpg differ diff --git a/docs/backlog/README.md b/docs/backlog/README.md index 5d6d95c7..32742235 100644 --- a/docs/backlog/README.md +++ b/docs/backlog/README.md @@ -21,7 +21,7 @@ A map of everything in the three files, by theme. ### Core ([backlog-core.md](backlog-core.md)) - **Distribution** — remaining platforms (Linux, Teensy, RPi), code-signing (macOS/Windows), live RMII Ethernet reconfigure, installer UX polish, P4 DHCP-hostname recheck, S31 web-flash (waiting on esptool-js); DevicesModule interop growth (more plugins, the command half, live peer state). -- **ESP32 performance & memory** — E1.31 multicast (IGMP), WiFi ArtNet perf matrix, async ArtNet send (PSRAM-only), network round-trip drop/reorder test, slow eth bring-up, non-PSRAM memory ceiling + boot-time buffer degradation, task core-pinning; ops: static IP on STA, MoonDeck doc-asset hardening, CI SHA-pinning. +- **ESP32 performance & memory** — E1.31 multicast (IGMP), WiFi ArtNet perf matrix, async ArtNet send (PSRAM-only), network round-trip drop/reorder test, slow eth bring-up, non-PSRAM memory ceiling + boot-time buffer degradation; ops: static IP on STA, MoonDeck doc-asset hardening, CI SHA-pinning. - **Architecture** — disable-releases-resources, cross-module pin-uniqueness check, Improv-child-of-NetworkModule, `std::span` platform API, Improv-as-REST follow-ups, **live scripting** (on-device authored effects/layouts/modifiers/drivers/sensor logic — design phase, see the bottom-up survey); composition/config: runtime board presets, per-layout coordinate offset. - **HTTP & OTA** — HTTP file serving off the render tick; generic control + state topics over MQTT (the automation escape hatch beside the semantic HomeKit surface). - **Testing** — additional coverage (UI load time, teardown memory, JS harness), live full-suite state leak. @@ -30,15 +30,15 @@ A map of everything in the three files, by theme. ### Light ([backlog-light.md](backlog-light.md)) -- **Drivers** — extract shared lane-driver scaffolding (on the 3rd backend), 1..8-pin LCD output, classic ESP32 I2S 16-lane driver. -- **LED drivers — deferred** — sigrok flicker cross-check, core-1 driver task, fuller RMT error handling, per-driver buffer window, 16-bit/dither, moving-head preview interpreter. +- **Drivers** — MoonI80 ring open instruments (multi-strand loopback, teardown leak/fragmentation, white-flash soak, shift-mode host coverage), classic-ESP32 shift ring on raw I2S (WANTED), P4 Parlio streaming ring (WANTED), shared lane-driver scaffolding (on the 3rd backend), ArtPoll discovery, RS-485/DMX wired output. +- **LED drivers — deferred** — sigrok flicker cross-check, chunked/staged transfer (the 16K lever), fuller RMT error handling, per-driver buffer window, 16-bit/dither, moving-head preview interpreter. - **LCD / DMA driver work** — drop the i80 WR/DC sacrificial pins, LCD/Parlio DMA buffer → PSRAM. - **Effects & preview** — real z-axis in 2D effects, full-density interpolated preview, self-describing frame header, RGBW preview, fixture model (moving heads/beams), extract the resumable transport. - **Sensors & audio-reactive input** — audio follow-ups (per-band noise floor, adaptive gate), GyroDriver → core Peripheral move, Raspberry Pi 5 sensor input (mic/IMU/line-in). ### Mixed ([backlog-mixed.md](backlog-mixed.md)) -- MultiplyModifier mapping-LUT memory at large grids; intermittent ~0.5 s RMT LED pauses; NoiseEffect simplex cost on ESP32. +- MultiplyModifier mapping-LUT memory at large grids; NoiseEffect simplex cost on ESP32. ## In-flight draft specs @@ -48,12 +48,8 @@ A spec for a not-yet-built module can live here as a plain draft `.md` (alongsid One-off research documents that informed a future direction, kept for the reasoning rather than as living specs. -- [leddriver-analysis-top-down.md](leddriver-analysis-top-down.md) — reasons from the end goal (driving WS2812-class LEDs from a GPIO pin) toward a generic driver architecture, per-platform implementation, and a testing strategy. -- [leddriver-analysis-bottom-up.md](leddriver-analysis-bottom-up.md) — the companion landscape survey: catalogues the existing LED-driver libraries across ESP32, Teensy, Raspberry Pi, and desktop, and recommends a path. - [livescripts-analysis-bottom-up.md](livescripts-analysis-bottom-up.md) — live scripting (run user-authored effects/layouts/modifiers/drivers/sensor logic on-device without a reflash), Stage-1 survey. Deep-reads the ESPLiveScript fork (hpwit's native-Xtensa JIT), surveys the field (ARTI-FX interpreter by ewowi, embedded VMs, WASM/WAMR), and records the product-owner direction. - [livescripts-analysis-top-down.md](livescripts-analysis-top-down.md) — the Stage-2 redesign: a native-codegen engine, Xtensa-first behind an IR seam (WASM/WAMR the per-target fallback), a C-subset language that ports an effect near-verbatim, the MoonModule binding, and a staged spike plan along the MoonLight effects-tutorial ladder. -- [multicore-analysis-bottom-up.md](multicore-analysis-bottom-up.md) — multicore & driver scaling, Stage-1 survey. Combines MoonLight's documented two-task architecture with **hardware measurement on the P4 at 16384 lights**: the surprise is that the driver cost is the CPU **WS2812 encode (~24 ms, 85%)**, not the DMA wait (~0). Gives the per-driver max-lights-per-pin (RMT/LCD have no hard cap — fps/memory only; Parlio hits a 65535-byte/lane hardware single-transfer ceiling (897 RGB lights)), documents two driver bugs found + fixed (Parlio over-limit silent-fail, bus-not-rebuilt-on-shrink), and re-aims multicore at parallelising the encode. Consolidates the scattered multicore notes; records frame-pacing-decided-against. -- [multicore-analysis-top-down.md](multicore-analysis-top-down.md) — the Stage-2 build plan on those findings: the ordered steps — (1) encode optimisation (the SWAR transpose — **shipped**), (1.5) the per-driver DMA double-buffer hiding the wire (`asyncTransmit` — **shipped**), (2a) the render↔encode core split, the whole output stage on core 1 (`multicore` — **shipped**, +44 % fps, 85 % of the output stage off the render core), (2b) the ping-pong second output buffer — **deferred, and the `stall` KPI now scopes it**: it buys nothing under a heavy effect (~1–15 µs stall) and only pays in the light-effect/many-lights corner (6–11 ms) — and (3) the ceiling raises, where the fps wall (~1024 LEDs/lane at 30 µs/light) makes **lanes, not per-lane depth**, the real lever. Carries the dependency graph and the settled non-steps (frame pacing, split-encode, the DMA-wait). ## Project transition diff --git a/docs/backlog/backlog-core.md b/docs/backlog/backlog-core.md index 972f064b..1cfb3028 100644 --- a/docs/backlog/backlog-core.md +++ b/docs/backlog/backlog-core.md @@ -87,7 +87,7 @@ The real fix is a **dedicated send task**: `loop()` snapshots the corrected fram So the PSRAM gate isn't conservative; it's a hard requirement. PSRAM boards (S3/S2, Olimex-with-PSRAM variants) have megabytes for the handoff buffer via `heap_caps_malloc(..., MALLOC_CAP_SPIRAM)`; non-PSRAM boards keep the synchronous send and the documented "use Ethernet / smaller grid for high FPS at large grids" guidance ([NetworkSendDriver.md](../moonmodules/light/moxygen/NetworkSendDriver.md)). -When implemented: `if constexpr (platform::hasPsram)` (or a runtime `hasPsram()` check) selects the async path; the buffer lives in PSRAM; the send task pins to the core opposite the render task (see [Task core-pinning](#task-core-pinning-backlog)). Non-PSRAM keeps `loop()`'s inline send unchanged. One handoff buffer + a binary semaphore/notification is the minimal shape — don't build a ring of frames until a second consumer needs it. +Acceptance criteria: `if constexpr (platform::hasPsram)` (or a runtime `hasPsram()` check) selects the async path; the buffer lives in PSRAM; the send task pins to the core opposite the render task (the same pattern as the shipped render↔encode split's worker). Non-PSRAM keeps `loop()`'s inline send unchanged. The single handoff buffer needs an explicit ownership contract, exactly like the shipped render↔encode split's `outputBuffer_`: the render path may write frame N+1 only after the send task signals it has finished reading frame N (a binary semaphore/notification is the handoff), so the two never touch the buffer at once. When the sender still owns the buffer at the next render tick, the render path drops (or coalesces onto) that frame rather than overwriting a live read — a dropped frame is acceptable, a torn one is not. That single-buffer contract is the minimal shape; a second buffer (double-buffering) or a ring of frames is only warranted if measurement shows the drop rate hurts, or a second consumer appears. ### `esp32-eth` slow Ethernet bring-up vs `esp32-eth-wifi` (investigation) @@ -166,10 +166,6 @@ The annoyance is purely that the device boots degraded and needs a poke to recov Related: this is the render/output-buffer face of the same non-PSRAM fragmentation cliff the paged `MappingLUT` already addressed for the *LUT*. The buffers themselves still allocate as single contiguous blocks. -### Task core-pinning (backlog) - -No FreeRTOS tasks are pinned today. At 16K LEDs the render task takes ~52 ms/tick; if OTA download or Improv scan causes tick-variance spikes, pin render → core 1, OTA/Improv → core 0 (where WiFi already lives via `CONFIG_ESP_WIFI_TASK_PINNED_TO_CORE_0=y`). Defer until contention is observed — neither OTA nor Improv runs during normal operation. - ## Architecture ### WiFi runtime disable — open design question (undesigned) @@ -197,7 +193,7 @@ Today the eth-only build profile compiles WiFi out (`MM_NO_WIFI`). Turning WiFi **Hardware-limit tail (not covered by the pin check).** Pin-uniqueness rejects the common case but not the controller-count limit: the S3 has **2 I2S controllers** regardless of pins, so a 3rd mic on distinct pins passes the pin check yet fails `i2s_new_channel` at runtime. That tail is already handled — the platform I2S init returns false on failure (no panic, module stays `inited_=false`); verified live (4 pinned AudioModules → error spam, no crash). So scope = pin-uniqueness check + the existing graceful-degrade; don't try to make the pin check also model controller counts. -**Related:** [§ Disabling a module should release its resources](#disabling-a-module-should-release-its-resources-not-just-stop-its-loop-backlog) — a disabled module freeing its pins is what lets the same GPIO be reassigned live without a conflict-reject. +**Related:** the shipped "disabling releases resources" work (see docs/history/plans/) — a disabled module freeing its pins is what lets the same GPIO be reassigned live without a conflict-reject. ### PinsModule — strict reject-on-add mode (the one remaining increment) @@ -226,15 +222,15 @@ Board preset catalog + upload (later, when the runtime config has real consumers **Prior art — MoonLight's per-board pin database** ([ModuleIO.h](https://github.com/ewowi/MoonLight/blob/main/src/MoonBase/Modules/ModuleIO.h)). MoonLight (our own project) already models exactly this for ~25 boards across ESP32-D0 / S3 / P4: a `pins[]` array of `{GPIO, usage, index}` plus board-level `maxPower`, `ethernetType`, `ethPhyAddr`, `ethClkMode`. Don't copy the file or paste its tables here — read it when building the catalog and write our own. Its `usage` enum enumerates the hardware functionalities a projectMM board preset *could* drive once the device-side consumers exist (each needs its own module/control before the corresponding `deviceModels.json` / catalog field earns its keep — none exist today beyond `System.deviceModel` + `Network.txPowerSetting`): - **LED output pins** — per-strip data GPIOs (1–16 outputs/board); the first real consumer (a Driver pin control) unblocks multi-output boards (QuinLED Dig-Quad/Octa, SE16, LightCrafter). **This consumer now exists** (the `pins` control on every LED driver; e.g. QuinLED Dig-Quad ships `"pins": "16,3,1,4"` in `deviceModels.json`), so the field earns its keep — but only up to **8 lanes** today (`kMaxLanes = 8` / `kMaxPins = 8`). The parallel drivers are moving to **16 lanes (choose 1..16)**; when they do, this becomes a real gap with two halves: - - **Per-model usable-GPIO map (the data).** Identify **which up to 16 GPIOs each device model actually exposes for LED output** — not the chip's full pin count, but the pins broken out to a usable header/connector AND safe to drive (exclude strapping, flash/PSRAM, input-only, and pins already owned by eth-RMII / I²C / the onboard LED). The codebase knows the chip *ceiling* (`MM_MAX_GPIO` from `CONFIG_SOC_GPIO_PIN_COUNT`, [Control.h:13](../../src/core/Control.h)) and the live *ownership/reserved* grading ([PinsModule](../moonmodules/core/system.md)), but NOT the per-board *exposed-and-safe* set — that is board knowledge (schematic/pinout per model). The authoritative source is the **annotated pinout image per model in [`docs/assets/deviceModels/`](../assets/deviceModels/)** (and MoonLight's `ModuleIO.h` `pins[]` as prior art — read it, write our own against `deviceModels.json`). **Worked example — ESP32-S3-N16R8 dev board** (from `esp32-s3-n16r8-dev.png`): the 16 safe LED-output GPIOs are **`4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,21`**, derived by excluding — octal flash+PSRAM (the R8/N16 part reserves `26–37`, the `SPIIO4-7/SPIDQS/SPICLK` pins), USB (`19,20`), UART0 console (`43,44`), the onboard RGB LED (`38`), and the strapping pins (`0,3,45,46`). That leaves exactly 16 clean I/O — enough for the full 16-lane target. Do the same read per model. - - **Encode as per-model defaults + coverage (the catalog).** The 16 usable lane GPIOs become the model's default `pins` string, so a fresh flash of e.g. "Serg UniShield V5" comes up with the *right* lane pins pre-filled, not blank/generic. Coverage today is uneven: of 25 models, **7 carry no LED-pin config** (Olimex Gateway, LOLIN D32, Generic ESP32 Dev, ESP32-S3 N16R8 Dev, LightCrafter 16, SE 16 V1, and any bare dev board) — those default to nothing and force the user to guess. Fill every model's LED-capable pin default, and **document the per-model map** (the annotated-pin images the § below already reserves are the natural home). **The per-board usable set is wider than the exposed header:** a pin the board commits to a peripheral it *doesn't mount* is fair game (the Olimex Gateway leaves 6 clean LED pins only if you count the **unmounted micro-SD** pins 4/13/14 — see [gpio-usage § Usable LED-output GPIOs](../reference/gpio-usage.md) and the bench note in memory), so the per-model map must record *which non-exposed/repurposable pins are safe on this board*, not just the header breakout. Scope: **16 pins max** — do not over-generalize past the peripheral lane ceilings ([multicore analysis](../backlog/multicore-analysis-bottom-up.md): Parlio 65535 bytes/lane single-shot (897 RGB lights at 8 lanes, ~448 at 16), LCD **8 or 16 lanes** on S3 (the widening shipped), RMT up to 8 TX channels). **Note the i80 gotcha:** an `LcdLedDriver` model should pick `clockPin`/`dcPin` clear of its data lanes — an overlap is a *warning*, not a blocker (the driver still runs; that one lane just carries the clock/DC waveform, which is fine for an unused parked lane but garbles an active strand), so a catalog default that overlaps an *active* lane would ship a subtly-broken board. + - **Per-model usable-GPIO map (the data).** Identify **which up to 16 GPIOs each device model actually exposes for LED output** — not the chip's full pin count, but the pins broken out to a usable header/connector AND safe to drive (exclude strapping, flash/PSRAM, input-only, and pins already owned by eth-RMII / I²C / the onboard LED). The codebase knows the chip *ceiling* (`MM_MAX_GPIO` from `CONFIG_SOC_GPIO_PIN_COUNT`, [Control.h:13](../../src/core/Control.h)) and the live *ownership/reserved* grading ([PinsModule](../moonmodules/core/system.md)), but NOT the per-board *exposed-and-safe* set — that is board knowledge (schematic/pinout per model). The authoritative source is the **annotated pinout image per model under `docs/assets/deviceModels/`** (e.g. [`esp32-s3-n16r8-dev.png`](../assets/deviceModels/esp32-s3-n16r8-dev.png)) (and MoonLight's `ModuleIO.h` `pins[]` as prior art — read it, write our own against `deviceModels.json`). **Worked example — ESP32-S3-N16R8 dev board** (from `esp32-s3-n16r8-dev.png`): the 16 safe LED-output GPIOs are **`4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,21`**, derived by excluding — octal flash+PSRAM (the R8/N16 part reserves `26–37`, the `SPIIO4-7/SPIDQS/SPICLK` pins), USB (`19,20`), UART0 console (`43,44`), the onboard RGB LED (`38`), and the strapping pins (`0,3,45,46`). That leaves exactly 16 clean I/O — enough for the full 16-lane target. Do the same read per model. + - **Encode as per-model defaults + coverage (the catalog).** The 16 usable lane GPIOs become the model's default `pins` string, so a fresh flash of e.g. "Serg UniShield V5" comes up with the *right* lane pins pre-filled, not blank/generic. Coverage today is uneven: of 25 models, **7 carry no LED-pin config** (Olimex Gateway, LOLIN D32, Generic ESP32 Dev, ESP32-S3 N16R8 Dev, LightCrafter 16, SE 16 V1, and any bare dev board) — those default to nothing and force the user to guess. Fill every model's LED-capable pin default, and **document the per-model map** (the annotated-pin images the § below already reserves are the natural home). **The per-board usable set is wider than the exposed header:** a pin the board commits to a peripheral it *doesn't mount* is fair game (the Olimex Gateway leaves 6 clean LED pins only if you count the **unmounted micro-SD** pins 4/13/14 — see [gpio-usage § Usable LED-output GPIOs](../reference/gpio-usage.md) and the bench note in memory), so the per-model map must record *which non-exposed/repurposable pins are safe on this board*, not just the header breakout. Scope: **16 pins max** — do not over-generalize past the peripheral lane ceilings ([measured lane ceilings](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid): Parlio 65535 bytes/lane single-shot (897 RGB lights at 8 lanes, ~448 at 16), LCD **8 or 16 lanes** on S3 (the widening shipped), RMT up to 8 TX channels). **Note the i80 gotcha:** an `LcdLedDriver` model should pick `clockPin`/`dcPin` clear of its data lanes — an overlap is a *warning*, not a blocker (the driver still runs; that one lane just carries the clock/DC waveform, which is fine for an unused parked lane but garbles an active strand), so a catalog default that overlaps an *active* lane would ship a subtly-broken board. - **Ethernet PHY config** — LAN8720/RMII (MDC/MDIO/CLK/power-pin/PHY-addr/clock-mode) vs W5500/SPI (MISO/MOSI/SCK/CS/IRQ); the consumer is the runtime `Network.eth_*` controls listed above, replacing the hardcoded Olimex pins. - **Power budget** — `maxPower` (Watts) per board, for a future current-limit / brightness-cap control. - **Audio / I2S** — SD/WS/SCK/MCLK pins, the input side of audio-reactive effects (Pi-5 sensor note is the desktop counterpart). - **Buttons & inputs** — push/toggle/lights-on, PIR, digital-input; needs an input-event concept the firmware doesn't have yet. - **Relays & power control** — relay / lights-on / high-low pins. - **Infrared** — IR receive pin (remote control). -- **RS485 / DMX** — TX/RX/DE pins (wired DMX-512 output beyond the current Art-Net path). The driver that consumes this is the [RS-485 / DMX-512 wired output item](backlog-light.md#rs-485--dmx-512-wired-output-future--the-physical-dmx-driver) in the light backlog. +- **RS485 / DMX** — TX/RX/DE pins (wired DMX-512 output beyond the current Art-Net path). The driver that consumes this is the [RS-485 / DMX-512 wired output item](backlog-light.md#rs-485-dmx-512-wired-output-future-the-physical-dmx-driver) in the light backlog. - **Sensing** — voltage / current / battery / temperature ADC pins. - **Onboard LED / key, exposed / reserved pins** — board-housekeeping and conflict-avoidance metadata. @@ -481,3 +477,19 @@ Legend: **Adopt-1.0** (small, high value) · **Defer-1.x** (needs engine work or They are **not** CI failures (CI is Debug) and each one inspected so far is a false positive of the same shape: a bounds check exists, but GCC cannot prove the degenerate case (e.g. `HttpServerModule.cpp:2400` warns "writing 1 byte into a region of size 0" on a line *guarded* by `if (wsLen + headerLen > sizeof(hdr)) return false;`). Still worth clearing: each is either a genuinely unprovable bound (fix the code) or a bound the code knows but does not state (make it explicit). Doing it in one pass beats letting them mask a real one later. Not done with the multi-destination/tab-UI merge because 17 warnings across four core files is its own change, not a tail on someone else's. + +## Preview stream: tail byte-phase desync under backpressure + +**Symptom (board B, 2026-07-14):** on a large grid the preview's bottom region (the frame tail) shows per-pixel noise; with a SOLID effect it flickers pure R / pure G / pure B at max brightness. Pure primaries from solid content = the frame bytes read at an offset that is not a multiple of 3 — tearing alone cannot discolor identical frames, so the resumable sender's offset accounting slips. Survives a browser refresh; occurs "occasionally" (per-frame), worst when the device is network-starved. + +**Suspect:** `HttpServerModule::sendBufferedFrame` (the zero-copy resumable send whose body is the live producer buffer, draining across transport ticks) — the resume-after-partial-socket-write path. Verify by logging the resume offset vs bytes actually written on EWOULDBLOCK; the fix is offset accounting, plus a resync rule (a client that missed bytes gets a fresh frame header, not a phase-shifted tail — the *robust to any input* bar). + +**Not** the shift-register transport bug and **not** buffer corruption: the composite buffer is proven correct (Solid writes every light; the preview alone garbles). + +## MoonLive core/platform layering + JIT sdkconfig scoping (CodeRabbit #29, 4 findings) + +Four 🟠 Major boundary findings from the PR #29 review are real but each is its own scoped change, not a tail on the ring branch. The Critical sibling (a `cpl<3` overflow guard in the MoonLive effect's `tick`) landed with the branch it was found on; these four are backlogged: + +- **Core includes platform, compiled core in `mm_core`.** `src/core/moonlive/MoonLive.cpp` `#include`s `platform/platform.h` and calls the exec-memory API directly, and the root `CMakeLists.txt` compiles `MoonLive.cpp`/`MoonLiveCompiler.cpp` into `mm_core` and links `mm_core → mm_platform` — violating the header-only-core / no-platform-includes contract both files declare. The runtime exec-memory placement layer wants a core-neutral injected interface (or to move out of `src/core`), so the compiled/platform-dependent surface sits behind `mm_platform` and `mm_core` stays INTERFACE-only. These two are one change (same boundary). +- **W^X disabled in the board default.** `esp32/sdkconfig.defaults.esp32s3-n16r8` turns off `CONFIG_ESP_SYSTEM_MEMPROT_FEATURE` and enables `CONFIG_HEAP_HAS_EXEC_HEAP` for *every* build on that board, even with no MoonLive effect installed. The JIT genuinely needs a writable-then-executable heap, but that belongs in a dedicated MoonLive/JIT opt-in overlay or an explicit build profile, not the board default — so a stock build keeps memory protection on. +- **A scenario rides timing + network.** `test/scenarios/light/scenario_modifier_chain.json` carries `tick_us` baselines (host-performance dependent) and routes a modifier-chain-composition test through `NetworkSendDriver` (pulls network-path behavior into a test that is not about the network). It wants an in-process sink and structural assertions so it stays hermetic, per the `test/**` "no timing or network dependence" rule. diff --git a/docs/backlog/backlog-light.md b/docs/backlog/backlog-light.md index 9f9b8b19..dde0ded1 100644 --- a/docs/backlog/backlog-light.md +++ b/docs/backlog/backlog-light.md @@ -4,29 +4,73 @@ Forward-looking to-build items for the **light domain** (`src/light/`: drivers, ## Drivers -### Extract shared lane-driver scaffolding when the 3rd parallel backend lands (deferred) +### MoonI80 streaming ring — 48×256 shipped; open instruments and cleanups -The `I80LedDriver` (classic-ESP32 I2S + S3/P4 LCD_CAM, both via the i80 bus) and `ParlioLedDriver` (P4 Parlio) share ~245 of 362 lines, and their platform-side loopback capture+verify is ~100 lines byte-for-byte identical (`platform_esp32_parlio.cpp` even notes "The RX capture half is byte-for-byte identical" to the i80 one). The status-string lifecycle (`failBuf_` / `configErr_` / `clearFailBuf` / `clearConfigErr`) is triplicated across all three LED drivers (RMT/i80/Parlio), ~60 lines. The branch deliberately extracted the *encoders* (`ParallelSlots.h` shared by i80+Parlio, `RmtSymbol.h`, `PinList.h`) on the "extract when the second user lands" rule, but stopped at the lifecycle/loopback scaffolding. **Accepted for this merge** (the reviewer agreed driver-level extraction can wait): the duplication is in mechanical lifecycle/test scaffolding, not domain logic, and a DriverBase-level refactor touching three drivers is riskier than the duplication it removes. **Do it when the third parallel backend arrives** (16-lane widening, or Teensy FlexIO), at which point the pattern is proven three ways: (a) a `detail::` platform helper for capture+verify (the only per-peripheral difference is the transmit call, pass a callback, beside the already-shared `loopbackJumperOk`), and (b) a small owned-status helper or DriverBase members for the fail/config strings. Until then the cost is line count, not correctness. +The ring's two regimes ship and are wall-verified through 48 strands × 256 (12,288 lights): prime-only when the frame fits the pool, the clock-oracle lapping ring above it (the near-prime pool — the ISR encodes only `nSlices − ringBufs` slices per frame), with `ringAuto` deriving the geometry per config and `shiftOverclock` trading the fps ceiling against '595 shift margin. The mechanism lives in the code + the technical page; the design arc in `docs/history/plans/` (the MoonI80 plans, all marked). Open items: + +- **Last-8-panels white flash — the "44-46 flash" (OPEN, cause not yet found; 6 theories ruled out).** On the 48×256 wall, brief WHITE/bright flashes (not wrong colors) over an otherwise-correct image, confined to strands 40-47 (the last 8 panels = the last physical data pin, GPIO 17 = bus bit 5 = the 6th and final 74HC595 in the daisy chain), mostly panels 44-46, wandering within the last 8. **Brightness-gated: clean below 5, flashes at ≥5** — since brightness scales pixel values through a LUT, below 5 the frame collapses to near-all-zero bits, so the gate really means *the flash needs SET (one) data bits on that pin*. **The same physical wall runs clean on hpwit's driver, so it is OUR driver, not the hardware.** Ruled out by live hardware tests (do NOT re-chase): (1) the encode source (PSRAM vs internal — the ISR-source staging fix did nothing, reverted); (2) the ISR/lapping regime (the flash is on prime-encoded rows too); (3) it being a tail-*rows* phenomenon (it is per-strand-group); (4) shift-clock margin via `shiftOverclock` (already at the 20 MHz OFF setting, still flashes); (5) the encoder itself (`ParallelSlots.h` proven byte-for-byte correct for this geometry by a host compile, incl. a pin-5 walking-one test — no all-HIGH "white" value ever appears in a pin-5 data word); (6) bus-bit latch adjacency (a `latchBitHigh` diagnostic moved the latch off bit-5's neighbour to bit 7 — still flashes). **Reopened clue:** the shift-register analysis doc records hpwit running the '595 SRCLK at **19.2 MHz**, *slower* than our `shiftOverclock`-OFF **20 MHz** — we treated 20 MHz as "the slow floor" but it is above his proven-good rate; the ~4% could be the deepest chip's margin. Untested angles to try next: driving the '595 clock below 20 MHz (needs a new divider — 16 MHz is all-white, so the window is narrow); GPIO-17 drive-strength / edge-rate specifically; whether it follows bit-5-*position* or GPIO-17 (swap which strands ride bit 5 via the pin order); the `ringPad` inter-slice settle window; and a direct A/B of our per-slice frame timing vs hpwit's for the last chip. All diagnostics from this investigation (the `latchBitHigh` toggle, the ISR-staging code) are reverted — the tree is clean. +- **Ring bus init hard-fails instead of stepping down when `ringBufs` is raised past what RAM allocates** (wall went dark until the control was lowered again; the pool alloc steps down but a later allocation, likely the descriptor link list, does not). A control change must degrade, never dark the output. +- **~1-frame white/colored flash every ~5 s** seen at some configs — plausibly fixed by the frame-close latch word (a strand whose last data bit ended HIGH missed its reset that frame); soak-observe on the wall before closing. +- **Prime barrier fps cost.** The ring's prime holds a busy-wait until the previous frame's deterministic wire end (`waitWireDrained` in `primeRingRange` — the barrier that keeps the next prime off buffers the DMA is still draining; the frame's last slices lap into the FIRST buffers, so the prime hits them first and no counter sees the repaint). The wait is ~0 when the snapshot + render gap already span the wire, but on fast frames it serializes wire → prime and caps fps at 1/(wire + snapshot + prime). If the fps work wants that overlap back, the barrier can go finer-grained (per-buffer: buffer b is safe once the drain passes slice `b + nSlices − ringBufs`) — measure first. Do NOT drain-gate `done` in the ISR instead (deadlocks; wall-measured as flicker-then-"no LED output"). +- **Multi-strand loopback**: the instrument drives one `loopbackStrand`, so it is structurally blind to a multi-strand fault (it passed while the wall was visibly corrupt). +- **Shift-mode loopback host coverage**: nothing in the suite drives shift-mode loopback end-to-end through the mock bus — the detection gap that let two loopback bugs live unnoticed. (The stall bug itself is fixed: capture-first alloc + pool step-down; verified on two boards.) +- **Loopback teardown leak + heap fragmentation**: cycling the private-bus loopback drops ~80 KB internal and the heap stays fragmented (measured maxBlock 13–44 KB with 240–310 KB free). Capture-first + step-down made runs reliable despite it, but the leak itself stands — chase when the loopback is next touched. +- **Ride-the-live-ring loopback — parked**: built (`loopbackIntrusive` + the snapshot pattern hold) but an RMT-RX cannot capture on a GPIO the LCD_CAM is actively driving (reads 0 sym). Revival path: briefly output-detach just the RX pin for the capture window. +- **Diagnostic surface** (`ringDbg` incl. tw/ts/tp + sg/se, the dbg statics, the `kCyPerUs = 240` hardcode): kept deliberately through the tuning era; gate/remove when the ring fps work closes. +- **fps header reports the module TICK rate**, not the frame rate (`frameTime` is the real one); any fps claim predating 2026-07-17 reads with that in mind. +- **`ringAuto` "just works" — verify the geometry pick on smaller configs.** `ringAuto` (default on) derives `ringRows`/`ringBufs`/`ringPadUs`, and those manual knobs are already dev-only (`setAdvanced`, expert mode). The pick is wall-verified at 48×256; the smaller common configs (16×256 and below) still need a bench pass. If any has a strictly better geometry `ringAuto` misses, teach it that ONE *measured* rule (no per-effect/per-density heuristics — the flicker was never geometry). `ringAuto` itself stays visible as the recourse until this proves it always picks right, then it can go expert-only too. (The fork-join flicker that once blocked this shipped fixed — snapshot fork removed.) + +### PSRAM-at-shift-clock: verified NOT a viable lever for more lights/strand (research, 2026-07-16) + +A recurring idea is to "borrow from direct mode": direct mode streams a huge frame straight from PSRAM (2048 lights, clean; SE16 drives 8192 lights direct/whole-frame at ~19.5 ms), so could shift mode run a lower pclk and stream its whole frame from PSRAM too, trading fps for unlimited length? **Verified answer: no.** The shift pclk is bounded by the WS2812 waveform, not by the '595 and not by divider elegance: slot = 8 bus words / pclk, and *lowering* the clock lengthens T0H toward the max-white washout. The practical floor is the `shiftOverclock`-OFF rate, 20 MHz (T0H 400 ns — wall-verified; 16 MHz is already all-white), which barely dents the PSRAM demand. There is no shift pclk that is both slow enough to stream from contended PSRAM and fast enough to keep T0H under the 0-vs-1 threshold. + +The bandwidth arithmetic (datasheet-derived): DMA demand = bus-bytes × pclk. Direct 8/16-bit = 2.67/5.33 MB/s; shift 8/16-bit = **26.7 / 53.3 MB/s**. S3 OPI PSRAM (octal, 80 MHz DDR) is 160 MB/s *theoretical* but only **~40–84 MB/s sustained/contended** in practice (Espressif's external-RAM guide: DMA-to-PSRAM bandwidth "is very limited, especially when the core is trying to access external RAM at the same time"; PSRAM shares the flash cache region). So direct demand sits far under the floor (streams fine — proven), while shift 16-bit demand *exceeds* the ~40 MB/s contended floor and shift 8-bit sits inside the underrun zone once WiFi/HTTP/CPU cache traffic competes. Because WS2812 is one unbroken self-clocked stream, one FIFO underrun garbles the rest of the frame. This is **datasheet-consistent with**, and MEASURED to match, ADR-0014's controlled A/B (board B, same PSRAM/chain, only the clock varied: 2.67 MHz PSRAM drives, 26.67 MHz PSRAM never completes at any size) and the 2026-07-16 `forceRing` re-confirmation (whole-frame at 2880 stalls). **Proven:** the effect (PSRAM stalls at the shift clock, drives at the direct clock). **Not instrumented (needs a bench measurement if ever doubted):** the exact mechanism — contended-sustained-rate FIFO underrun vs PSRAM read latency vs cache/MMU contention — was inferred from the clock being the sole variable, never isolated with underrun/bandwidth counters. + +**Conclusion — this does not open a new path; the proper ring fix already is the path.** The internal-RAM footprint of the ring is NOT set by light count: the ring transposes from a PSRAM-resident source into a small fixed internal buffer pool, so PSRAM is never on the DMA's read path at all. The 240-light wall is the `kRingBufs=16` no-reuse stopgap (the wrap read-while-write race), NOT the ring's design — and "more buffers" is a confirmed dead end. The shipped ring (above) holds internal RAM constant at arbitrary light count, which is exactly the "unlimited lights/strand" the PSRAM-hybrid idea was reaching for — obtained the correct way, at the mandatory shift clock, without PSRAM on the read path. **Action: none — the ring shipped; the "lower shift pclk + PSRAM whole-frame" hybrid is closed as physically blocked and should not be re-attempted.** (If the mechanism is ever contested, the one bench measurement worth doing is registering GDMA underrun/FIFO-empty counters at 26.67 MHz whole-frame-PSRAM to distinguish underrun from latency — but it would not change the conclusion.) + +### MoonI80 ring — boot / first-frame-after-rebuild trips a transient give-up status (2026-07-16) + +A cosmetic residual left after the rebuild-wedge fix (below): on boot, and for a beat after any shift-ring rebuild, the driver shows **"output stalled"** even though `wireUs` reports live completions — the *first* frame after a fresh ring build occasionally misses its completion window and trips the dead-frame give-up before the ring settles, so the stale error latches until the next interaction clears it. It is NOT the old wedge (that stayed dead until reboot; this self-clears on any control edit and the ring is genuinely driving underneath). Two clean fixes to weigh: (a) don't count the very first frame after a rebuild toward `deadFrames_` (give the fresh ring one grace frame), or (b) have the give-up retry re-derive status the moment `wireUs` shows a real completion. Low priority — the LEDs drive correctly; only the status string is briefly wrong. + +### Graceful blank-on-stall — a stalled bus should DARKEN the strip, not leave it lit with garbage (2026-07-15) + +When the bus stalls mid-frame the WS2812 strip is left holding **random / max-brightness lights** (often all-white — the all-ones failure pattern) that only a **power cycle** clears. That is a robustness gap: WS2812s latch their last received color and hold it until re-clocked or power-cycled, so a frame that dies mid-stream leaves every light past the failure point stuck bright. The give-up guard today stops *spending the render thread* on a dead bus (correct) but does nothing about the *strip's* state, so the user sees a wall of garbage LEDs and reaches for the plug. + +**The fix: on give-up (and on a rebuild that SHRINKS the reachable range), clock out ONE clean all-black frame** — every lane LOW → every light receives 0,0,0 → the strip goes dark. This turns "stall = a wall of random bright LEDs until power-cycle" into "stall = strip cleanly dark," which is the honest *degraded, not crashed* state the *[Robustness](../architecture.md#robustness)* rule asks for. It also covers the PO's specific case (drop `ledsPerPin` 256→128 and the abandoned 128–256 range stays lit): a full-length black frame on the shrinking rebuild blacks the whole physical strip once, no boundary to compute. + +**The load-bearing caveat:** if the bus is wedged *because it cannot complete a transfer*, a black frame may not clock out either — so this is a best-effort **attempt**, not a guarantee: try the black frame on give-up; if it clocks, the strip darkens; if the DMA is truly dead, we are no worse off than today (and the rebuild-wedge fix above is the real cure for *that* class). Note it must be a genuine transmitted frame (all lanes driven LOW through the normal encode+transmit), not merely zeroing the DMA buffer — the strip only changes on a clocked frame. Pin it with a test: after `kDeadFramesBeforeGiveUp` dead frames, the driver emits one all-zero frame through the transmit seam (the mock asserts a zero frame was handed to the bus), and a subsequent recovery resumes normal content. + + +### Classic-ESP32 shift-register ring on raw I2S — the high-light-count classic driver (WANTED) + +**The PO wants shift-register output on classic ESP32 boards at the MAXIMUM light count**, which the current whole-frame `I80LedDriver` (esp_lcd i80 → I2S on classic) cannot deliver: its frame buffer must be **internal RAM** (the classic I2S DMA cannot read PSRAM — IDF rejects it outright), so it scales against a ~76 KB wall and caps at **~2048 lights**. Above that, classic needs the **hpwit refill-ring model**: a handful of tiny internal DMA buffers (~1.2 KB total, light-count-independent) refilled by the I2S EOF ISR, which transposes the next pixel row out of a **PSRAM** framebuffer as it goes. The DMA never touches the source array — only the CPU/ISR does — so the source lives in PSRAM and internal RAM stays constant. This is how hpwit (and WLED-MM/MoonLight on his driver) reach **48×256 ≈ 12K lights at ~100 fps on classic silicon with WiFi up**, with the same '595 expander. + +**Why it must be a SEPARATE driver, not a flag on `I80LedDriver`:** `esp_lcd` owns the classic I2S DMA and only does whole-frame, so the ring cannot be bolted onto the esp_lcd path — it needs a second classic driver written on the **raw I2S registers** (`i2s_ll` / the LCD-mode register file directly, below esp_lcd, the way the S3/P4 MoonI80 backend sits below esp_lcd on LCD_CAM). Its ISR is small enough to fit the classic's ~70 KB IRAM (hpwit's does), and — the key move — it registers the interrupt **without** `ESP_INTR_FLAG_IRAM` (his source comment: removed "to avoid Cache Disabled but Cached Memory Region Accessed") so the refill ISR is legally permitted to read the PSRAM framebuffer. The trade it accepts vs. our whole-frame path: it **gives up the whole-frame path's WiFi-underrun immunity** (a WiFi burst that starves the refill ISR can glitch a frame), which the ring mitigates with a tunable buffer-count cushion (`nbDmaBuffer`, hpwit's default 6). For a ≤2K-light WiFi-busy install the whole-frame i80 is still the better choice; the ring is for the >2K-light case classic cannot otherwise reach. -### 16-lane parallel output — SHIPPED (2026-07-12) +**Reference (study, don't copy — write fresh against our architecture):** the line-by-line source read is in [led-driver-psram-ring-analysis.md](led-driver-psram-ring-analysis.md); the ADR framing is [ADR-0014](../adr/0014-own-i80-dma-driver-below-esp-lcd.md) (which calls the internal-RAM-ring-with-CPU-refill "the only thing that can ever work on the classic ESP32," deferred to a phase 2). The S3/P4 MoonI80 ring is the closest in-tree prior art for the ring mechanics (linear self-terminating chain, per-drain refill, drain-count termination) — but its refill is a task and its buffers are internal-only *because the LCD_CAM GDMA can't sustain a PSRAM read at the shift clock*; the classic I2S ring is the inverse (PSRAM framebuffer legal, ISR refill mandatory), so it borrows the *shape* but not the constraints. Do the S3/P4 **ISR-refill + `MM_HOT`** work first (it proves the ISR-refill pattern in-tree on the friendlier unified-DIRAM chips); the classic raw-I2S ring is the next tier up, reusing that pattern where IRAM is genuinely tight. -The LCD_CAM (S3/P4) and Parlio (P4) drivers drive **up to 16 lanes**, with the bus width DERIVED from the pin count: ≤8 pins → an 8-bit bus (uint8 slots, the original path), 9..16 pins → a 16-bit bus (uint16 slots). LCD_CAM accepts exactly 8 or 16 real pins (IDF caps `bus_width` to those, and rejects an NC data line — a sub-16 board parks unused lanes on the WR "ghost pin"); Parlio accepts any 1..16 (unused lanes idle NC). The 16-lane transpose is the two-pass `transposeLanes16x8` (two 8×8 SWAR passes → a uint16 plane, low byte = lanes 0..7, high = 8..15) in `ParallelSlots.h`; `frameBytesFor` and the loopback thread a `slotBytes` factor. The DMA buffer is PSRAM-first (the 16-bit frame doubles the footprint). **Consumers:** SE 16 V1 + LightCrafter 16 (both S3/LCD_CAM, wired to `I80LedDriver` in the catalog). **Verified live on the LightCrafter 16:** the 16-bit i80 bus inits, the PSRAM DMA buffer allocates, and the 16×8 transpose + 16-bit encode runs (`LcdLed:19348 µs`). Design record: `docs/history/plans/Plan-20260712 - 16-lane parallel LED output (shipped).md`. +### P4 Parlio streaming ring — lift the P4 Parlio ceiling past ~21K to light-count-independent (WANTED) -**Alternatives considered and NOT chosen** (kept here as the design-intent record): -- **A direct-register LCD_CAM driver** (bypassing esp_lcd to drop the sacrificial WR/DC pins and allow odd lane counts) — rejected. Research showed the established S3 parallel drivers build on the same esp_lcd i80 component we do (the "I2SClockless" name is classic-ESP32 lineage; the S3 path is LCD_CAM). A register-level version is the recognized way to reclaim the WR/DC pins, but it's tightly coupled to a given IDF's register layout and has to be re-proven across IDF major bumps; on IDF v6 (which we already fight for platform drift) it would be the most bump-sensitive file in the tree for the gain of *one* pin. The **ghost-pin trick** (WR+DC+unused data lanes on one GPIO) captures the pin benefit inside the stable esp_lcd API with zero register code. -- **>16 lanes** — physically impossible on S3 (LCD_CAM hard-capped at 16 data lines); only reachable on P4 via the *RGB-panel* peripheral (24 lanes, `esp_lcd_new_rgb_panel`, still esp_lcd, still a real PCLK pin). Its own future item, not this widening. +**Port the ring concept to the P4 Parlio path**, to drive far more than its current whole-frame ceiling. troyhacks' MoonLight Parlio driver reaches **~21K LEDs RGB (~16K RGBW)** at 16 lanes — but NOT by materialising the whole encoded frame: he stages into a **fixed ~512 KB PSRAM buffer** and DMAs it out in **64 KB chunks** (`max_transfer_size = 65535`), so the DMA never needs the whole frame contiguous. That is the SAME idea as our MoonI80 ring, applied to Parlio on the P4 (where — unlike the S3 shift clock — the DMA *can* sustain PSRAM reads at the WS2812 rate). His ceiling is a *chosen buffer size*, not a hardware wall, so it caps at ~21K. -**Remaining follow-ups** (separate items, not this increment): the Parlio chunked-transfer (to drive past the 65535-byte/lane single-shot ceiling — halved to ~448 RGB lights/lane at 16-bit); and the transpose SIMD/fused-128-bit refinement *if* profiling ever shows the two-pass combine is the ceiling (drop-in behind the same signature + test). +**Two tiers, in order:** +1. **Match him (moderate):** chunk our Parlio DMA the way he does (bounded PSRAM staging + 64 KB bursts) instead of one contiguous encoded frame — this alone lifts our P4 Parlio ceiling to his ~16–21K. Our Parlio path already uses PSRAM; the change is the chunked-DMA transfer shape, not a new architecture. +2. **Beat him (the ring):** run our MoonI80 **streaming ring** on P4 Parlio — small internal buffers, refilled per drain from a PSRAM source, holding internal RAM *constant* regardless of light count (the hpwit model our ring already implements). troyhacks' 512 KB buffer is still *bounded*; the ring is not, so this goes past ~21K to light-count-independent. It reuses the ring mechanics we already have (linear self-terminating chain, per-drain refill, drain-count termination); the P4 Parlio backend gets a ring variant beside its whole-frame path, the way MoonI80 has both. -### Classic ESP32 parallel LED driver (I2S i80) — SHIPPED (2026-07-13) +**Prerequisite / sequencing:** tier 2 wants the **ISR-refill + `MM_HOT`** work done first (same as the classic ring above), since a high-rate Parlio refill benefits from the ISR-grade determinism. Confirm the CURRENT P4 Parlio tested ceiling first (re-measure on the P4 bench) before claiming a head-to-head — the ~21K figure is troyhacks' *buffer-fit* limit, not a verified-on-wire number, and his code steps the clock down at 256/512 LEDs per lane (signal integrity at long strands), which our own measurement must account for. Reference: the parlio source is `src/MoonLight/Nodes/Drivers/parlio.cpp` in MoonLight; our Parlio backend is `platform_esp32_parlio.cpp` + `ParlioLedDriver.h`. -The **classic ESP32 has 8 RMT TX channels**, so `RmtLedDriver` covers ≤8 parallel outputs (verified 8×256 = 2048 lights on the LOLIN D32). For **>8 lanes on classic ESP32**, `I80LedDriver` now drives the **I2S peripheral in i80 mode** — the *same* driver + platform seam (`platform_esp32_i80.cpp`) the S3/P4 use for LCD_CAM, because IDF's `esp_lcd` i80 API is one public interface routed by IDF's own CMake to I2S on the classic chip and LCD_CAM on the S3/P4 (selected by `CONFIG_SOC_LCDCAM_I80_LCD_SUPPORTED`). The classic gate is `i2sLanes` in `platform_config.h` (beside `rmtTxChannels`/`lcdLanes`/`parlioLanes`), and `main.cpp` registers `I80LedDriver` under `#if defined(CONFIG_SOC_LCD_I80_SUPPORTED)` (true on classic + S3 + P4). **Verified live on the ESP32-WROVER:** 8 lanes incl. pin 2 drive a real strip, scaling ~7.6 µs/light. **Measured ceiling: 2048 lights at 8 lanes** (8×256 drives — 2048 lights / 8192 RGBW channels; 8×384 already fails). The I2S DMA can't reach PSRAM, so the frame buffer must fit internal RAM (~76 KB largest block); an over-ceiling config degrades with `i80 bus init failed`, never crashes. Neither a smaller grid (that buffer is PSRAM) nor `asyncTransmit` off (one DMA buffer instead of two) lifts it — 2048 is the honest classic maximum, exactly the parallel-I2S acceptance floor. 16 lanes work electrically (the I2S peripheral does the 16-bit i80 bus), but the WROVER has too few non-strap pins to wire them safely. Full sweep + quirks: [performance.md § Multi-pin LED driving](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid). Design record: `docs/history/plans/Plan-20260713 - Classic-ESP32 I2S i80 LED driver (shipped).md`. -**The chunk-streaming-ring design was superseded — and the 2026-07 research confirms that was right, with one caveat.** The original plan (a bespoke ISR-refilled ring of small DMA buffers with a tunable `nbDmaBuffer` flicker cushion, the hpwit I2SClockless model) assumed the classic I2S peripheral has no whole-frame transfer. It does via `esp_lcd`: the i80 backend does **whole-frame chained DMA** (`esp_dma_calculate_node_count` over `LCD_DMA_DESCRIPTOR_BUFFER_MAX_SIZE` chunks), so it is **WiFi-underrun-immune by construction** — no ring, no cushion, no flicker problem, exactly like the LCD_CAM/Parlio path. So the driver is a thin reuse of the shared `ParallelLedDriver` base, not a new ISR driver. Three classic-only integration quirks the seam handles (each `#if`-gated to the I2S backend): I2S DMA can't read PSRAM (internal-only draw buffer), the I2S tx has an unconditional command phase whose busy-wait hangs to a watchdog reset unless given a real 8-bit command (`lcd_cmd_bits=8` / `kI80Cmd=0`), and the done-ISR is `IRAM_ATTR`. +### Extract shared lane-driver scaffolding when the 3rd parallel backend lands (deferred) + +The `I80LedDriver` (classic-ESP32 I2S + S3/P4 LCD_CAM, both via the i80 bus) and `ParlioLedDriver` (P4 Parlio) share ~245 of 362 lines, and their platform-side loopback capture+verify is ~100 lines byte-for-byte identical (`platform_esp32_parlio.cpp` even notes "The RX capture half is byte-for-byte identical" to the i80 one). The status-string lifecycle (`failBuf_` / `configErr_` / `clearFailBuf` / `clearConfigErr`) is triplicated across all three LED drivers (RMT/i80/Parlio), ~60 lines. The branch deliberately extracted the *encoders* (`ParallelSlots.h` shared by i80+Parlio, `RmtSymbol.h`, `PinList.h`) on the "extract when the second user lands" rule, but stopped at the lifecycle/loopback scaffolding. **Accepted for this merge** (the reviewer agreed driver-level extraction can wait): the duplication is in mechanical lifecycle/test scaffolding, not domain logic, and a DriverBase-level refactor touching three drivers is riskier than the duplication it removes. **Do it when the third parallel backend arrives** (16-lane widening, or Teensy FlexIO), at which point the pattern is proven three ways: (a) a `detail::` platform helper for capture+verify (the only per-peripheral difference is the transmit call, pass a callback, beside the already-shared `loopbackJumperOk`), and (b) a small owned-status helper or DriverBase members for the fail/config strings. Until then the cost is line count, not correctness. + +### Multicore Step 2b — ping-pong second output buffer (deferred, workload-gated) + +The shipped render↔encode split (Step 2a, `multicore` control) uses one `Drivers::outputBuffer_`, so the composite is serialized at the boundary. A **second** output buffer would overlap even that (core 0 composites into B while core 1 encodes A), at the cost of one full frame buffer (~48 KB at 16K lights). The read-only `stall` KPI was built as the trigger metric, and it splits cleanly: a **heavy effect** (render-bound) shows `stall` ~1 µs (S3) / ~15 µs (classic) — 2b recovers *nothing*, core 0 already fills the encode window; a **light effect on many lights** (output-bound — a solid colour or slow gradient across 16K lights) shows `stall` 6–11 ms — 2b recovers that whole wait. So it's a real but narrow win: **gated on the product owner naming that light-effect/many-lights workload as worth the 48 KB**, not on a generic fps gain. (Design context in the shipped Plan-20260713 - Multicore Step 2.) -**The caveat: whole-frame bought that immunity at the price of the 2048 ceiling** — because the frame buffer must be internal RAM, it scales with the light count against a ~76 KB wall. A refill ring keeps the framebuffer in **PSRAM** and holds internal RAM ~constant (~1.6 KB), which is how hpwit — and WLED-MM/MoonLight on his driver — reach far higher counts on the same silicon. **That remains a real, unclaimed lift, and it is PARKED, not rejected**: it would be a *second* classic driver on the raw I2S registers (`esp_lcd` owns the DMA and only does whole-frame, so it cannot be a flag on `I80LedDriver`), it trades away the underrun-immunity we just gained, and **nothing planned needs it**. Notably **FastLED does not beat us here either** — its rewritten classic-ESP32 I2S engine keeps hpwit's register-level lineage but *dropped the refill ring*, so it is whole-frame with no PSRAM framebuffer path and shares our ceiling; the party that beats us is hpwit (and WLED-MM/MoonLight riding his driver). **The trigger that would un-park it: classic-ESP32 above ~2048 lights becoming a shipping requirement.** Until then, note the virtual driver below does NOT need it (it targets S3/P4 first, where the DMA already reaches PSRAM), so the ring is not on anyone's critical path. +### Frame pacing — decided against (record) -Still future: the **virtual (shift-register) driver** — fan one i80 lane out to 8+ physical strands via 74HC595-class latches (hpwit's I2SClocklessVirtual lineage, ~120 outputs), **acceptance floor 48×256 = 12288 lights**. Triggered when a ≥17-output board arrives. **Build it on the shipped i80/Parlio base, S3/P4-first:** there the DMA buffer comes from PSRAM (the SE16 already drives 16,384 lights), so the 12288 floor is reachable on the *existing* whole-frame model with no new memory model. On the **classic** chip that floor is out of reach (internal-RAM-bound at 2048) — a documented limit of that chip, and the only thing that would lift it is the parked ring above. So the virtual driver is **not** blocked on the ring; do not sequence them that way. +MoonLight targets a fixed 60 fps; projectMM deliberately does not (settled with the PO 2026-07-12). The architecture is *render-uncapped + time-aware effects* (`beatsin8`/`millis()`-driven, a CLAUDE.md hard rule), so a whole-engine fps cap is redundant with that rule and would only *reduce* quality below the hardware ceiling; the LED wire rate already paces render physically (30 µs/light), and UI/WiFi responsiveness comes from the per-tick `vTaskDelay(1)` yield, not frame-rate control. Parked as a ~15-line opt-in (`targetFps=0` = unlimited default) *only if* a genuinely CPU-starved device ever appears. ### ArtPoll discovery — know which tubes are alive (next increment on NetworkSendDriver) @@ -44,7 +88,7 @@ Do it as its own increment. The multi-destination unicast it builds on has shipp ### RS-485 / DMX-512 wired output (future) — the physical-DMX driver -projectMM already speaks DMX **over the network** (Art-Net / sACN via `NetworkReceiveEffect`). The missing half is **wired DMX-512 out**: driving DMX fixtures (moving heads, par cans, wired pixel controllers) directly over an RS-485 differential pair, which is what the RS-485 hardware on carrier boards like the [MHC-WLED ESP32-P4 shield](../../reference/mhc-wled-esp32-p4-shield.md) is *for*. DMX-512 is a 250 kbps async serial frame (a break + mark-after-break + 513 bytes: start code + 512 channels) shipped over RS-485 — the textbook fixture-control transport. A DMX driver would map the light buffer (or a fixture/attribute model — see the [Fixture model — moving heads, beams](#fixture-model--moving-heads-beams-long-term) item below) to DMX channels and clock the frame out a UART in RS-485 mode. +projectMM already speaks DMX **over the network** (Art-Net / sACN via `NetworkReceiveEffect`). The missing half is **wired DMX-512 out**: driving DMX fixtures (moving heads, par cans, wired pixel controllers) directly over an RS-485 differential pair, which is what the RS-485 hardware on carrier boards like the [MHC-WLED ESP32-P4 shield](../reference/mhc-wled-esp32-p4-shield.md) is *for*. DMX-512 is a 250 kbps async serial frame (a break + mark-after-break + 513 bytes: start code + 512 channels) shipped over RS-485 — the textbook fixture-control transport. A DMX driver would map the light buffer (or a fixture/attribute model — see the [Fixture model — moving heads, beams](#fixture-model-moving-heads-beams-long-term) item below) to DMX channels and clock the frame out a UART in RS-485 mode. **What it needs that we don't have yet:** - **A `platform::` UART-RS485 seam.** The ESP32 UART has a hardware RS-485 half-duplex mode (`uart_set_mode(UART_MODE_RS485_HALF_DUPLEX)`) that auto-drives the transceiver's **DE/RE** (driver-enable / receiver-enable) line — the thing our current pin handling has no concept of (we drive pins as plain GPIO). A DMX driver is where DE/RE control first earns its place. The bench insight that surfaced this: the P4-shield loopback couldn't work partly *because* we have no DE/RE handling to flip its RS-485 transceivers Tx↔Rx. @@ -53,9 +97,9 @@ projectMM already speaks DMX **over the network** (Art-Net / sACN via `NetworkRe **The channel-mapping half is now unblocked.** The per-light encode path handles an arbitrary channel count as of 2026-07-13 (the WS2812 drivers' per-light scratch is heap-sized to `outChannels`, no fixed cap — the fix from the multi-channel-preset bootloop, see [lessons.md](../history/lessons.md)). A DMX universe is exactly that model: a light with `channelsPerLight = ` (16-ch moving head, 7-ch par, …), and the buffer's bytes ARE the DMX channel values. So a DMX driver's "map the buffer to channels" step is now the trivial part — it ships the light buffer's bytes straight into the 512-channel frame. What remains genuinely new is the **transport** (the RS-485 UART seam + break/MAB timing) and the **typed-fixture model** (naming which channel is Pan vs Dimmer — the moving-head fixture item), not the encode. -**Can a board drive XLR fixtures directly? Yes, with an RS-485 transceiver — that's the one required part.** DMX-512 is RS-485: a *differential* pair (D+/D−, ±2–6 V), not the 3.3 V single-ended UART the MCU emits, so an MCU TX pin can NOT wire straight to XLR. A transceiver chip (MAX485 / SN75176 / THVD-class, ~$0.50) sits between the UART and the connector and drives the differential pair; the DE/RE line (the UART-RS485 seam above) flips it Tx↔Rx. **3-pin XLR** carries it: pin 1 = ground, pin 2 = D−, pin 3 = D+. With a transceiver present, daisy-chaining ~10 moving heads (10 × 16 ch = 160, inside one 512-channel universe) over standard DMX in→out is well within the RS-485 limits (32 unit loads / 1200 m); the last fixture wants a 120 Ω terminator (a fixture/cable concern, not the MCU). So whether a catalog board can drive XLR *directly* hinges on one schematic question: does it carry an RS-485 transceiver + XLR/terminal (then yes, direct), or only the WS2812 level-shifted outputs (then a ~$0.50 breakout is needed). Confirm against the [MHC-WLED ESP32-P4 shield](../../reference/mhc-wled-esp32-p4-shield.md) schematic before treating direct-XLR as a shipping capability. +**Can a board drive XLR fixtures directly? Yes, with an RS-485 transceiver — that's the one required part.** DMX-512 is RS-485: a *differential* pair (D+/D−, ±2–6 V), not the 3.3 V single-ended UART the MCU emits, so an MCU TX pin can NOT wire straight to XLR. A transceiver chip (MAX485 / SN75176 / THVD-class, ~$0.50) sits between the UART and the connector and drives the differential pair; the DE/RE line (the UART-RS485 seam above) flips it Tx↔Rx. **3-pin XLR** carries it: pin 1 = ground, pin 2 = D−, pin 3 = D+. With a transceiver present, daisy-chaining ~10 moving heads (10 × 16 ch = 160, inside one 512-channel universe) over standard DMX in→out is well within the RS-485 limits (32 unit loads / 1200 m); the last fixture wants a 120 Ω terminator (a fixture/cable concern, not the MCU). So whether a catalog board can drive XLR *directly* hinges on one schematic question: does it carry an RS-485 transceiver + XLR/terminal (then yes, direct), or only the WS2812 level-shifted outputs (then a ~$0.50 breakout is needed). Confirm against the [MHC-WLED ESP32-P4 shield](../reference/mhc-wled-esp32-p4-shield.md) schematic before treating direct-XLR as a shipping capability. -Sequencing: it's a **driver** (`src/light/drivers/`) + a platform UART-RS485 seam + a fixture model shared with the Art-Net path — the buffer→channel encode is already done. Plan when a DMX fixture is actually on the bench and a catalog board's `supported`/`planned` list points at wired DMX. The [pin-functionality catalog](backlog-core.md#pin-functionality-catalog) already reserves the RS485/DMX TX/RX/DE slot; this is the driver that consumes it. +Sequencing: it's a **driver** (`src/light/drivers/`) + a platform UART-RS485 seam + a fixture model shared with the Art-Net path — the buffer→channel encode is already done. Plan when a DMX fixture is actually on the bench and a catalog board's `supported`/`planned` list points at wired DMX. The [PinsModule pin-assignment work](backlog-core.md#pinsmodule-strict-reject-on-add-mode-the-one-remaining-increment) covers the RS485/DMX TX/RX/DE slot; this is the driver that consumes it. ## Sensors and audio-reactive input @@ -153,6 +197,12 @@ Today a "light" is a point at a static coordinate with a colour. A **moving head **Sub-item — per-emitter targets, and fixing the Yellow/UV RGB-synthesis placeholder.** `Correction::apply()` today synthesizes every non-RGB emitter (White, WarmWhite, Yellow, UV) from RGB via the one `whiteMode` gate. That conflates two different physics: **White/WarmWhite are broadband illumination** with a real achromatic basis (`min(R,G,B)` is a sound approximation — warm vs cold differ only in phosphor CCT a byte value can't express), but **Yellow/Amber (~590 nm) and UV (~400 nm) are saturated/out-of-gamut emitters with no honest RGB pre-image.** Yellow's `min(R,G)` stand-in reads greener than a true amber die AND fires on far too much (any red+green content — yellows, whites, skin tones — muddies the fixture); UV's blue-excess is an eyeball hack. Amber is a *real, common, targetable* emitter (RGBA / RGBAW PARs ship a dedicated ~590 nm die), so the right model is an **effect that targets amber DIRECTLY** (a typed attribute), not RGB-and-synthesize. The fixture model is where that lands. **Decision to make when built:** whether Yellow/UV should default to **off** (0, like `whiteMode::None`) until an effect drives them — more honest than always firing a wrong approximation — vs. keeping the "light it up to eyeball the wiring" placeholder. **Practical test to run then:** on a real RGBA/RGBAW+UV PAR, compare the synthesized amber against a directly-driven amber die (does `min(R,G)` look acceptably yellow, or muddy?), and confirm the White subtraction under `Accurate` reads correct with all emitters present. Rationale + the current formulas live in `Correction.h`'s emitter-field comment. +### Spacer / gapped-strand layout — dark gaps between physical LED segments (user request) + +**User request (via PO):** driving a slat wall on the P4 where each data pin's physical strand has BLACK GAPS between addressed segments — e.g. pin 1's strand is 550 LEDs physically, but column 1 is LEDs 0–250, column 2 is LEDs 300–550, and 251–299 are unaddressed spacer LEDs that must stay dark. Today's layouts map a contiguous run of pixels per strand; there is no way to express "skip N physical LEDs here, then resume." + +The need is a layout (or layout modifier) that inserts **spacer regions** into a strand's pixel-to-physical mapping: the effect/grid still sees a contiguous logical space, but the driver's per-strand output leaves the gap LEDs at zero (dark). Design questions to settle before building: is this a new **layout type** (a "SlatWall"/"SpacedStrip" that takes segment lengths + gap lengths), a **modifier** on an existing layout (insert gaps into the coordinate stream), or a **driver-level** per-strand offset map? The driver already has a window (`start`/`count`) per strand; a gap is essentially multiple windows per strand. Physics constraint whatever the design: a serial WS2812 strand cannot skip positions (LED n+1's data passes through LED n), so every spacer LED must receive explicitly clocked ZERO bytes each frame — "unmapped" coordinates still occupy wire slots, and the design must guarantee those slots carry zeros, not stale buffer content. Spec it (a `docs/moonmodules/light/layouts` entry) before code; the panels/grid layout code is the reference for how coordinates map to the buffer. + ### Mixing light types in one Layouts — open design question (undesigned) Today each layout child describes one light type (all LED strips, or all par lights), and the current model is one Layouts container per light type. Whether a single Layouts should hold mixed types (LED strips + par lights together), and how the per-channel layout would reconcile across them, isn't designed. Deferred until a concrete need forces it; it's adjacent to the fixture model above (a real fixture/attribute model may reframe how mixed types are expressed). (Moved from architecture.md § What we leave undesigned; a deferred design decision, not a settled 🚧 one.) @@ -163,29 +213,51 @@ The preview's transport — resumable cross-tick send from a stable buffer + new ## LCD / DMA driver work -### Drop the i80 WR/DC sacrificial pins (S3 I80LedDriver) via direct LCD_CAM +### Drop the i80 WR/DC sacrificial pins — done for MoonI80, open for I80 + +**Shipped for `MoonI80LedDriver` (2026-07-14).** Owning the GPIO matrix is what bought it: the matrix is a routing fabric, so a peripheral signal that is never connected to a pad simply stays inside the peripheral. `dcPin` is **gone entirely** (DC separates command from data bytes for an LCD panel; WS2812 has no such concept, and the peripheral holds it at a constant level), and WR is routed **only** when a '595 needs it as SRCLK — in direct mode WS2812 is self-clocked, nothing reads WR, and the pin stays free for a strand. Same trick frees the *spare data lanes*: they are simply not routed, rather than parked on a "ghost" GPIO. So a direct-mode MoonI80 board spends its GPIOs on strands alone. -The S3 i80 LED path costs **two GPIOs the LEDs never use**: the IDF `esp_lcd` i80 bus hard-requires a WR (pixel clock) and a DC pin on real GPIOs (`esp_lcd_panel_io_i80.c`: `wr_gpio_num >= 0 && dc_gpio_num >= 0`), even though WS2812 strands ignore both. Today `I80LedDriver` keeps overridable defaults (clockPin=10, dcPin=11) — peripheral-required, not user-strand wiring, so a default cannot do harm. **Two ways to reclaim the pins, neither trivial:** -- **Cannot reuse a data pin for WR/DC.** A GPIO carries exactly one peripheral signal (`esp_rom_gpio_connect_out_signal` binds data_sig[i] / wr_sig / dc_sig each to its own pin); routing WR onto a data lane would clock the *clock* waveform onto that strand instead of its colour bytes. WR/DC must be distinct *physical* pins from the 8 data pins. (You CAN already point them at any otherwise-free or unstrapped GPIO via the controls — that's the "reuse a pin you're not using" answer; it's the *spare* pin you avoid, not a data pin.) -- **Zero WR/DC pins needs bypassing esp_lcd** and driving the LCD_CAM peripheral's registers directly (hpwit's I2SClockless approach — legacy parallel mode has no DC concept and emits WR without a dedicated config pin). That's the only path to 8-pins-total on the S3. Cost: leaving the recognisable IDF `esp_lcd` API for register-banging (a *Common patterns first* hit), re-proving the driver bit-perfect on hardware (the loopback self-test is the proof). Benefit: 2 GPIOs back on a tight S3 board. Its own increment, not a pin-default tweak. Parlio (P4) already needs no extra pins (`clk_out_gpio_num = GPIO_NUM_NC`), so this is S3-i80-only. +**Still open for `I80LedDriver`, and it cannot be fixed there.** IDF's `esp_lcd` hard-requires both (`esp_lcd_panel_io_i80.c`: `wr_gpio_num >= 0 && dc_gpio_num >= 0`) and rejects an NC data pin, which is *why* it must park spare lanes on a real GPIO. Reclaiming the two pins on that driver means leaving `esp_lcd` — which is precisely what MoonI80 already is. So this is not a change to make to `I80LedDriver`; it is a reason to prefer MoonI80 once it is proven. Parlio (P4) never needed the pins (`clk_out_gpio_num = GPIO_NUM_NC`). ### Parlio DMA frame buffer → PSRAM (free internal SRAM for big frames) For driving **lots of LEDs**, internal SRAM is the scarce resource and the parallel-driver DMA frame buffer is the biggest consumer (8 lanes × lights × outCh × 24 slot-bytes + latch pad). The **i80 driver already allocates PSRAM-first on the LCD_CAM chips** (S3/P4) — `platform_esp32_i80.cpp` tries `MALLOC_CAP_DMA | MALLOC_CAP_SPIRAM` under `#if SOC_LCDCAM_I80_LCD_SUPPORTED`, falling back to internal — which is why the SE16 reaches the full 16384-light frame (see [performance.md § Multi-pin](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid)). The classic-ESP32 I2S i80 backend stays internal-only (its DMA can't reach PSRAM — a hardware limit, not a TODO). **Parlio still allocates internal-only** (`platform_esp32_parlio.cpp`), so a large Parlio frame can exhaust DRAM while PSRAM sits unused; the IDF confirms Parlio's GDMA can burst from PSRAM (`esp_driver_parlio/src/parlio_tx.c` sets `access_ext_mem = true // support transmit PSRAM buffer`). (RMT already does the right thing — its symbol buffer goes through `platform::alloc`, PSRAM-first with an internal fallback.) -**The change (Parlio only):** allocate the Parlio buffer `MALLOC_CAP_DMA | MALLOC_CAP_SPIRAM` first, falling back to internal when PSRAM is absent/full, using the **external-memory alignment** the IDF requires (`gdma_get_alignment_constraints` → `ext_mem_align`, typically the cache line) and keeping the buffer cache-aligned + its size a multiple of that alignment. **Why its own increment:** it changes the proven hot DMA path, PSRAM DMA has real caveats (cache-line alignment, write-back/coherence on the encode→DMA handoff, and lower PSRAM bandwidth that the IDF guards with a CPU-MAX DFS lock during transmit), and it **must be re-proven on P4 hardware** (the loopback self-test bit-verifies it, then a real strip). It also raises the Parlio ceiling toward the [chunked-transfer](#led-drivers--deferred) goal. Measure the bandwidth headroom too: a very wide, long frame at speed may want internal SRAM regardless. +**The change (Parlio only):** allocate the Parlio buffer `MALLOC_CAP_DMA | MALLOC_CAP_SPIRAM` first, falling back to internal when PSRAM is absent/full, using the **external-memory alignment** the IDF requires (`gdma_get_alignment_constraints` → `ext_mem_align`, typically the cache line) and keeping the buffer cache-aligned + its size a multiple of that alignment. **Why its own increment:** it changes the proven hot DMA path, PSRAM DMA has real caveats (cache-line alignment, write-back/coherence on the encode→DMA handoff, and lower PSRAM bandwidth that the IDF guards with a CPU-MAX DFS lock during transmit), and it **must be re-proven on P4 hardware** (the loopback self-test bit-verifies it, then a real strip). It also raises the Parlio ceiling toward the [chunked-transfer](#led-drivers-deferred) goal. Measure the bandwidth headroom too: a very wide, long frame at speed may want internal SRAM regardless. ## LED drivers — deferred -The LED-driver increments **shipped**: increment 1 (RMT/WS2812B single-strand on classic ESP32 — [`RmtLedDriver.h`](../../src/light/drivers/RmtLedDriver.h), `RmtSymbol.h`, `platform_esp32_rmt.cpp`) and increment 2 (2a multi-pin RMT, 2b parallel LCD_CAM on the S3 — [`LcdLedDriver.h`](../../src/light/drivers/LcdLedDriver.h) via [`ParallelLedDriver.h`](../../src/light/drivers/ParallelLedDriver.h), `platform_esp32_lcd.cpp`), all with host + on-board-loopback tests, hardware-proven. The locked decisions, file-by-file phases, the WiFi-flicker test-rig analysis, and the bench deviations (8-GPIO i80 bus, 2.67 MHz slot clock, SOC-macro gate, real-frame loopback) are in [lessons.md](../history/lessons.md), the [driver docs](../moonmodules/light/moxygen/RmtLedDriver.md), and the [analysis docs](leddriver-analysis-top-down.md). What remains here is only the work that has **not** shipped and is tracked nowhere else. +The LED-driver increments **shipped**: increment 1 (RMT/WS2812B single-strand on classic ESP32 — [`RmtLedDriver.h`](../../src/light/drivers/RmtLedDriver.h), `RmtSymbol.h`, `platform_esp32_rmt.cpp`) and increment 2 (2a multi-pin RMT, 2b parallel LCD_CAM on the S3 — [`LcdLedDriver.h`](../../src/light/drivers/LcdLedDriver.h) via [`ParallelLedDriver.h`](../../src/light/drivers/ParallelLedDriver.h), `platform_esp32_lcd.cpp`), all with host + on-board-loopback tests, hardware-proven. The locked decisions, file-by-file phases, the WiFi-flicker test-rig analysis, and the bench deviations (8-GPIO i80 bus, 2.67 MHz slot clock, SOC-macro gate, real-frame loopback) are in [lessons.md](../history/lessons.md), the [driver docs](../moonmodules/light/moxygen/RmtLedDriver.md), and the [analysis docs](../history/leddriver-analysis-top-down.md). What remains here is only the work that has **not** shipped and is tracked nowhere else. + +- **sigrok/fx2lafw cross-check + MoonDeck "LED driver test" Python script** — the independent-clock proof and the run-from-MoonDeck flow ([analysis §5.3](../history/leddriver-analysis-top-down.md)). The on-board RMT-RX loopback (shipped) is the cheap CI correctness gate but a *compromised witness* for WiFi-induced flicker — the RX capture runs on the same ESP32 whose WiFi causes the glitch. The real flicker test is a **sustained capture (seconds) with WiFi associated + a packet flood**, decoding every frame for a byte-slip or reset-gap deviation; it validates the SHIPPED render↔encode split's WiFi isolation (drivers tick on core 1; WiFi lives on core 0). A DSLogic Plus (100 MS/s) upgrade is reactive — only if a flicker reproduces that 24 MS/s can't resolve. +- **Chunked transfer (Step 4) — the 16K lever, and now the ONE mechanism behind three separate ceilings.** Split a frame into transactions the DMA can actually swallow, feeding them back-to-back. It was scoped as a Parlio fix; it is really a **core-path** fix, and the shift-register expander is only its third beneficiary. + + **The three ceilings it lifts, all unshifted-first:** + 1. **P4 Parlio: ~4,096 lights.** The 2026-07-12 16-lane sweep found the single-DMA ceiling at 256/lane × 16, reproduced within 0.3% on a second P4 — and the cause is **not** the 65,535-byte cap (256/lane is far under the 897/lane limit). The P4 has 33 MB free heap but the largest *contiguous internal block* is ~368 KB, and a single-shot 16-bit DMA buffer needs one contiguous block: at 512/lane init fails outright. So chunking is **the only path to the 16×1024 = 16,384 lights the 16-lane widening promised.** This is the headline win and has nothing to do with shift registers. + 2. **Classic ESP32: 2,048 lights.** Its I2S DMA cannot reach PSRAM at all, so the frame must fit internal RAM. *(This entry previously said chunking "would not lift that" — that assumed chunking a PSRAM frame the DMA still had to read. It does not hold for the **staged** form below: if the DMA only ever reads small INTERNAL chunks that the CPU fills from a PSRAM frame, the classic chip is lifted too.)* + 3. **The 74HCT595 expander.** Currently capped at ~96 lights/strand because its ×8 frame only renders correctly from internal RAM ([§ 7.5](../history/shift-register-driver-analysis.md)). An add-on, and explicitly **not** the reason to build this. + + **Two distinct limits, one idea — keep them straight.** For **Parlio** the constraint is *transaction size* (contiguous block + 65,535 bytes), so chunking means smaller transactions. For **i80** there is no single-shot cap at all (it chains DMA descriptors) — there the constraint is *where the DMA reads from*, so the win comes from **staging**: keep the frame in PSRAM, but have the CPU copy it a chunk at a time into small internal-RAM buffers that the DMA reads. Same mechanism, different reason, and conflating the two is what muddled the shift-register investigation. + + **Prior art, and why staging is the shape.** hpwit's drivers never DMA from PSRAM: the pixel data may live there, but an EOF ISR transposes it into small **internal bounce buffers** that the DMA clocks out (`owner_check = false`, a fixed ring, built once). Espressif do the same thing under a different name — the RGB-LCD driver's **bounce buffers**. The one place hpwit *does* DMA straight from PSRAM is the **P4 Parlio** path, and he chunks it (≤64 KB via IDF's parlio driver) — which is consistent with our own finding that the P4 handles a PSRAM whole-frame shift transfer fine while the S3 does not. The staged form gets hpwit's underrun immunity **without leaving `esp_lcd`** (still IDF-maintained, still one code path across classic/S3/P4, no raw-register driver) and **without putting the CPU back on a per-LED ISR deadline** — a bulk sequential PSRAM→internal copy per chunk is what PSRAM is good at, unlike a real-time streaming read. + + **The two mechanisms do DIFFERENT jobs, and the 100 fps target needs both.** The WS2812 wire time is a physical constant — 30 µs per light, serial down each strand — so the *only* lever on frame rate is **lights per strand**, never the chip: + + | strands × lights | total | wire time | fps ceiling | + |---|---|---|---| + | 16 × 1024 (direct, 16 GPIOs) | 16,384 | 30.7 ms | **33 fps** | + | **48 × 256 (6 pins × '595)** | **12,288** | **7.7 ms** | **130 fps** ✅ *(hpwit + PO proved this in practice — StarLight)* | + | **64 × 256 (8 pins × '595)** | **16,384** | **7.7 ms** | **130 fps** ✅ | + + So: **chunking/staging buys LIGHTS (memory); the expander buys FRAMES PER SECOND (short strands).** Neither substitutes for the other, and 12–16K at 100+ fps needs both — the expander's own ~145 KB frame is exactly the one that does not work from PSRAM today, so it *depends* on the staging fix. (A third piece is needed too: at 130 fps the CPU encode must finish in <7.7 ms, and it measures ~24 ms at 16K — that is what the multicore render↔encode split exists for.) + + **The classic ESP32 is a target, not a write-off.** It is routinely dismissed for work like this, and the dismissal is wrong: hpwit and the PO have *run* 48 strands × 256 at ~100 fps on classic silicon (StarLight), with the same '595 expander, while WiFi was up. 240 MHz, two cores, and a 30 µs/light budget is a lot of headroom. The thing that would stop us is **our own encode cost**, which is software we control, not a property of the chip — and the ~24 ms/16K figure quoted in the multicore analysis is (a) measured on a **P4**, not a classic, and (b) **pre-dates the SWAR transpose** that shipped since. Do not carry that number into a classic-ESP32 feasibility argument; measure the real one on the real chip. What the classic genuinely needs is the **staged** form of chunking (its I2S DMA cannot reach PSRAM at all), which this item provides. -- **sigrok/fx2lafw cross-check + MoonDeck "LED driver test" Python script** — the independent-clock proof and the run-from-MoonDeck flow ([analysis §5.3](leddriver-analysis-top-down.md)). The on-board RMT-RX loopback (shipped) is the cheap CI correctness gate but a *compromised witness* for WiFi-induced flicker — the RX capture runs on the same ESP32 whose WiFi causes the glitch. The real flicker test is a **sustained capture (seconds) with WiFi associated + a packet flood**, decoding every frame for a byte-slip or reset-gap deviation; it belongs with the core-1 driver-task work below, since that task pinning is the *fix* it validates. A DSLogic Plus (100 MS/s) upgrade is reactive — only if a flicker reproduces that 24 MS/s can't resolve. -- **Dedicated core-1 driver task + per-module core-affinity control** ([analysis §7.2](leddriver-analysis-top-down.md)) — the WiFi-glitch mitigation, shared across all the LED drivers. (See also [backlog-core § Task core-pinning](backlog-core.md#task-core-pinning-backlog) for the general task-pinning question.) The [multicore analysis](multicore-analysis-top-down.md#step-2--multicore-effects-on-core-0-encodetransmit-on-core-1-the-pipeline--shape-decided) re-aims this: the driver's cost is the CPU **WS2812 encode (~24 ms at 16384 lights, 85%)**, so the second core's real win is **overlapping render↔encode** (render frame N+1 on core 0 while core 1 encodes+transmits N — the pipeline). Note this is a *different* overlap than Step 1.5's encode↔transmit double-buffer: the earlier "DMA wait ~0" reading was wrong (the transmit currently blocks — see Step 1.5), so recovering the wire-ceiling fps is Step 1.5's one-core job, and *this* core-1 task is about hiding effect-render cost + WiFi-timing isolation for the transmit. -- **Parlio chunked transfer (Step 4) — the 16K lever, contiguous-block-bound.** The 2026-07-12 16-lane sweep found the P4 Parlio single-DMA ceiling is **~4096 total lights (256/lane × 16)**, reproduced within 0.3% on a second P4 — and the cause is **not** the byte cap (256/lane is far under the 897/lane `PARLIO_LL_TX_MAX_BITS_PER_FRAME` = 65535-byte limit). The P4 has 33 MB free heap but the largest *contiguous internal block* is ~368 KB, and the single-shot 16-bit DMA buffer needs one contiguous block: at 512/lane (8192) init fails (`Parlio init failed — check pins / memory`). So the real limit is contiguous memory, hit at ~4096 lights. This makes chunking the **only** path to the 16×1024 = 16384 animated total the 16-lane widening promised: **split each lane's frame into transactions that each fit an allocatable contiguous block** (and ≤65535 bytes), with correct WS2812 inter-chunk timing (idle-LOW < 300 µs so the strand doesn't latch mid-frame). The driver already rejects an over-limit frame with a loud status (fixed). RMT (streams via ping-pong) and i80 (chains DMA descriptors) have **no** single-shot cap, so this is Parlio-specific. The classic-ESP32 I2S i80 backend (shipped) has no single-shot cap either — it chains descriptors like the LCD_CAM one — but it is capped for a *different* reason: its DMA can't reach PSRAM, so the frame buffer must fit internal RAM (2048 lights at 8 lanes, measured). Chunking would not lift that; only a chip whose DMA reaches PSRAM does. Pairs with the 16-lane extension; measured detail in [performance.md § Multi-pin](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid). + **Build and prove chunking on the unshifted path first** (Parlio 4,096 → 16,384 is the measurable win, on proven code), then let shift mode inherit it — that is a sequencing rule about *where to de-risk the mechanism*, **not** a claim that the expander is optional. It is not: it is the only route to 100 fps at this scale without spending 48+ GPIOs. Correct WS2812 inter-chunk timing is the one hard constraint: the lines must idle LOW for < 300 µs between chunks or the strand latches mid-frame. The driver already rejects an over-limit frame with a loud status. Measured detail: [performance.md § Multi-pin](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid). - **`rmtWs2812Show` fuller error handling** (deferred from PR #17 / 🐇 CodeRabbit). The shipped path has a finite `rmt_tx_wait_all_done` timeout (1 s) so a wedged DMA can't hang the render tick forever, and a dropped frame self-heals (the driver re-encodes the whole frame next tick). The fuller version — `rmt_transmit` return check, `rmt_tx_stop` to cancel an in-flight transfer on timeout, `show()` returning failure so `loop()` won't reuse `symbols_` mid-transmit — belongs with the **core-1 driver-task** work, since that task owns the buffer lifetime and in-flight state the cancel logic needs. -- **Surface RMT symbol-buffer alloc failure as a status** (bench-found 2026-07-12, [multicore analysis](multicore-analysis-bottom-up.md#multi-pin-driving-results-across-all-three-peripherals-128128-grid-2026-07-12)). `resizeSymbols()` sizes for the driver's `count` window, so on a classic ESP32 (~90 KB heap) a whole-grid window (`count=0` on a 128×128 grid ≈ 1.5 MB) fails to allocate: `symbols_` stays null, `tick()` bails at its `!symbols_` guard, and the strip goes **dark with no status** — the user sees nothing lit and no error. The fix mirrors the Parlio over-limit guard (already loud): when the symbol alloc returns null, set a clear "not enough memory — reduce lights or use start/count" status instead of silently idling. Small, robustness-principle work; pairs with the fuller RMT error handling above. -- **Auto-derived DMA buffer count** (7 / 30 / 75 per [analysis §7.4](leddriver-analysis-top-down.md)), **16-bit pipeline + dither** ([§7.3](leddriver-analysis-top-down.md)), **shift-register expander stubs** ([§7.5](leddriver-analysis-top-down.md)). +- **Surface RMT symbol-buffer alloc failure as a status** (bench-found 2026-07-12, [multi-pin driving results](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid)). `resizeSymbols()` sizes for the driver's `count` window, so on a classic ESP32 (~90 KB heap) a whole-grid window (`count=0` on a 128×128 grid ≈ 1.5 MB) fails to allocate: `symbols_` stays null, `tick()` bails at its `!symbols_` guard, and the strip goes **dark with no status** — the user sees nothing lit and no error. The fix mirrors the Parlio over-limit guard (already loud): when the symbol alloc returns null, set a clear "not enough memory — reduce lights or use start/count" status instead of silently idling. Small, robustness-principle work; pairs with the fuller RMT error handling above. +- **Auto-derived DMA buffer count** (7 / 30 / 75 per [analysis §7.4](../history/leddriver-analysis-top-down.md)), **16-bit pipeline + dither** ([§7.3](../history/leddriver-analysis-top-down.md)), **shift-register expander stubs** ([§7.5](../history/leddriver-analysis-top-down.md)). - **IR RX live-reconfigure recovery — unconfirmed, park until it recurs** (bench 2026-07-13, SE16). IR reception on the SE16 (`IrService` pin 5) went dead mid-session and only a **hard reset** brought it back; a warm/API path did not. **Ruled out:** not hardware (hard reset fixed it, receiver+switch+wiring fine), not LED-count (IR survives the full 16384-light / 8 fps load — a received code still toggled a control at max load), not a regression from the i80 commit (`platform_esp32_ir.cpp` untouched, the 1250 ns glitch-filter fix intact). **Prime suspect (unproven):** the session's live pin churn — including transiently setting the i80 `clockPin` to **5, which IS the IR pin** — left GPIO 5 routed to the wrong peripheral, and the RMT-RX channel (a pin-keyed static behind `platform::irStop`/`ensureChannel`) didn't re-acquire cleanly on the next `irRead`; only a full GPIO re-init (hard reset) cleared it. This may be pure test artifact (nothing in a *normal* user flow points two live modules at GPIO 5). **To conclude:** from a fresh hard reset (IR working), in isolation set i80 `clockPin=5` then restore `clockPin=8` and check whether IR dies and whether it self-recovers *without* a hard reset — self-recovers → no bug (test artifact); stays dead → a real live-reconfigure gap in the IR channel re-acquire worth fixing (per *No reboot to apply a configuration change*). Small robustness/repro work; do it only if IR breaks again in real use. - **Moving-head preview = peer interpreter.** When moving heads land, the previewer must interpret channel semantics (pan/tilt/RGBW-at-arbitrary-indices) to render a moving fixture — the same light-preset model physical drivers use, interpreted to screen. This is *why* the increments named the abstraction "interpret the preset" rather than "apply correction / opt out": so Preview becomes a full peer here without a rename. Its own design plan when moving-head support starts. - **Sparse light-preset editor.** A LightPresets row currently shows one role Select per channel across the whole `channels` width — including the unmapped `—` gaps a wide moving head has between its functions. For a fixture you usually only care about the few channels you drive (rgb, pan, tilt). The refinement: show only the *mapped* channels + an "add channel" affordance (pick a role → fills the first gap or grows the fixture), over the unchanged dense `roles[]` storage. A first attempt shipped and was reverted for edit bugs; redo it cleanly (the dense editor is the reliable interim). Prior art: GDTF / QLC+ fixture profiles (a fixture is a sparse `{channel → function}` map, not a dense per-channel array). diff --git a/docs/backlog/backlog-mixed.md b/docs/backlog/backlog-mixed.md index 5569c19c..58668e6c 100644 --- a/docs/backlog/backlog-mixed.md +++ b/docs/backlog/backlog-mixed.md @@ -22,10 +22,6 @@ Not a dependency of palettes — a separate feature, its natural home the Lights **This is NOT a no-PSRAM blocker** — 16K Noise + Multiply has run on a classic ESP32 (no PSRAM, 320KB internal) before at **10–20 FPS** (WiFi vs Ethernet), sending frames out over **ArtNet to a display, not physical LED drivers**. It works there because classic's `nrOfLightsType` is `uint16_t` (half the LUT size) and the modifier shrinks the logical render grid. So the action is **re-verify the working classic setup when a classic board is connected** (find the config — grid, mirror, ArtNet target — that reproduces the historical 10–20 FPS), not "fix an impossibility." Worth investigating only if that re-verification shows the LUT memory has regressed since: the destinations array is the obvious lever (it stores a `nrOfLightsType` per physical destination; a 2× kaleidoscope is 1:1 in *count* so the LUT need not store fan-out > the physical count — confirm it isn't over-allocating to `maxMultiplier()` when the effective fan-out is 1). Capture the classic numbers into performance.md's multi-board table first. -### Intermittent ~0.5 s LED pauses with the RMT driver (RESOLVED — suspect #1 fixed) - -Observed on the bench (2026-06): LED output on the RMT driver occasionally froze for ~half a second. **Resolved by the WiFi-power-save fix in commit `7d0cfaf1` ("Disable WiFi power-save")** — the strongest of the three original suspects. The IDF default `WIFI_PS_MIN_MODEM` puts the radio into DTIM sleep, causing exactly this class of intermittent multi-hundred-ms stall; `platform_esp32.cpp` now calls `esp_wifi_set_ps(WIFI_PS_NONE)` after association. **Bench re-verification (2026-07-10):** watched the per-tick KPI log on all four boards — S3 (WiFi), MM-Classic olimex (WiFi, single-core Xtensa, the most pause-prone), S31 and P4 (Ethernet) — for cumulative minutes; **zero pauses**, tick steady within a few-µs jitter and FPS flat (classic held 216–229 FPS over 90 s, worst tick 4.6 ms). The two lesser suspects (NetworkSend re-ARPing a dead ArtNet destination; the `rmt_tx_wait_all_done` 1 s timeout) never manifested and are moot with sleep disabled. See [lessons.md](../history/lessons.md) for the diagnostic method. Item kept only as a breadcrumb; delete on the next subtraction pass. - ### NoiseEffect simplex cost on ESP32 (investigation) With mirror XY at 128×128, NoiseEffect renders the 64×64 logical quadrant in **~11 ms/tick** on the Olimex (measured) — the simplex math dominates, since the Xtensa LX6 has no FPU and float math is software-emulated. (RainbowEffect on the same pipeline is much cheaper.) This is correct, non-degraded behaviour; it's only worth revisiting if a deployment needs Noise faster than ~11 ms at this grid. diff --git a/docs/backlog/folder-structure-proposal.md b/docs/backlog/folder-structure-proposal.md deleted file mode 100644 index 2655b7c8..00000000 --- a/docs/backlog/folder-structure-proposal.md +++ /dev/null @@ -1,58 +0,0 @@ -# Folder-structure decision — library is a tag, not a folder - -A *Refactor for simplicity* decision (per CLAUDE.md), recorded here because the live catalog pages -(`effects.md` / `modifiers.md` / `layouts.md`) cite it for *why* the tree is shaped the way it is. The -execution has shipped (assets + tests type-split; docs flattened to `domain/type` with catalog + -generated `moxygen/` pages); this is the surviving **rationale**, not a to-build list. - -## The three axes - -1. **Domain** — `core` vs `light`. Structured in `src/`, `docs/`, `test/`, `assets/`. -2. **Module type** — effects / modifiers / layouts / drivers. Structured in each of those areas. -3. **Library** — a module's *origin* (MoonLight, WLED, MoonModules, projectMM-native). **A tag and a - doc split — NOT a folder axis.** - -## Decision: `domain / type` folders; library is a tag (+ a doc split) - -The structure is **` / / Module`**, flat within type. Library does **not** become a -folder level. - -**Why library is not a folder** (the deciding analysis, still true): an effect's origin is frequently -*blended*, not a single fact — `DistortionWavesEffect` cites MoonLight + WLED + v1 + v2; -`GameOfLifeEffect` cites MoonLight + MoonModules + v1; several have no clear origin. A folder forces -one answer to a multi-valued question, and a *wrong* or *shifting* answer means a multi-file move -(src + assets + tests + the registered doc path). It also duplicates the dimension the `tags()` emoji -already carries (and the emoji can carry *several* origins; a folder can't). **The end user does not -care about a module's library** — they filter by the emoji chip in the UI if they want origin at all. -So library stays where it's free and non-duplicative: - -- **In code / assets / tests:** the `tags()` emoji (drives the UI origin-filter; can be multi-valued). - The leaf files are flat within their type folder (`src/light/effects/DistortionWaves.h`, - `docs/assets/light/effects/DistortionWaves.gif`, `test/unit/light/unit_DistortionWaves.cpp`). -- **In docs:** library rides in the **page** dimension, not a folder — a catalog page per type - (`effects.md`) with library *sections* inside today, splitting to per-library page *names* - (`effects_wled.md`) as a section outgrows its page. This is the one area with a doc-explosion - problem, and a doc page is *forgiving* about fuzzy origin (a blended-lineage effect goes on one page - with its full origin in the row's tags; mis-filing is a one-line edit, not a multi-file move). So - the drawbacks that make library-as-*folder* bad are soft for library-as-*doc-page*. - -This drops every drawback of library-as-folder (fuzzy-origin filing, two-places-disagree, -reclassification churn, sparse subfolders, deep paths) at once. - -## The one rule, across all four areas - -| | core/light | type | leaf | library | -|---|---|---|---|---| -| **src** | `light/` | `effects/` | `DistortionWaves.h` | tag in `tags()` | -| **assets** | `light/` | `effects/` | `DistortionWaves.gif` | — | -| **tests** | `light/` | `effects/` | `unit_DistortionWaves.cpp` | — | -| **docs** | `light/` | the page name (`effects.md`, later `effects_.md`) | (row inside) | the page split | - -`docs` is the one area where `type` is expressed as part of a **page name** rather than a folder, -because the docs compact to per-type/per-library pages — and library, the only axis with an explosion -problem, rides along in that name. Everywhere else: plain `domain/type` folders, library as a tag. - -## Still open (future growth, not blocking) - -- **Per-library page split** (`effects_wled.md`, …) — a lift-not-rewrite when a library's section - outgrows its page; the flat page names + within-page sections are already in place for it. diff --git a/docs/backlog/led-driver-psram-ring-analysis.md b/docs/backlog/led-driver-psram-ring-analysis.md index 6328524f..794c7f0b 100644 --- a/docs/backlog/led-driver-psram-ring-analysis.md +++ b/docs/backlog/led-driver-psram-ring-analysis.md @@ -2,13 +2,13 @@ > **Forward-looking research document — exception to the CLAUDE.md present-tense rule.** One question: **can PSRAM lift the classic-ESP32 LED ceiling projectMM measured at 2048 lights?** Researched 2026-07 against primary sources — hpwit's driver source read line-by-line, plus Espressif's own docs. Every load-bearing claim is cited in § 4. > -> **Status: PARKED FINDING — not a queued task.** The answer is **yes, via a refill ring** — but that is a **second classic-ESP32 driver** (it cannot be a flag on the shipped i80 driver), and **nothing currently planned needs it**: the virtual driver builds on the existing i80/Parlio base, S3/P4-first, where the DMA already reaches PSRAM. **§ 3 holds the verdict and the trigger that would un-park it.** Read § 3 before treating anything here as a plan. +> **Status: PARKED FINDING — not a queued task.** The answer is **yes, via a refill ring** — but that is a **second classic-ESP32 driver** (it cannot be a flag on the shipped i80 driver), and **nothing currently planned needs it**: the shift-register driver builds on the existing i80/Parlio base, S3/P4-first, where the DMA already reaches PSRAM. **§ 3 holds the verdict and the trigger that would un-park it.** Read § 3 before treating anything here as a plan. ## 1. Verdict -**Yes — PSRAM can lift the classic-ESP32 ceiling, and the mechanism is a memory-model change, not a virtual-driver feature.** Our 2048-light wall is a direct consequence of the *whole-frame* DMA model: `esp_lcd`'s i80 backend materialises every light's WS2812 waveform in one contiguous DMA-capable block before the transfer starts, and on the classic ESP32 that block **cannot be PSRAM** — the IDF source rejects it outright (`esp_lcd_panel_io_i2s.c`: `ESP_RETURN_ON_FALSE((caps & MALLOC_CAP_SPIRAM) == 0, NULL, TAG, "external memory is not supported")`), because the classic chip's I2S DMA physically cannot address the external-RAM window. So the DMA buffer scales linearly with the light count against a ~76 KB internal wall. +**Yes — PSRAM can lift the classic-ESP32 ceiling, and the mechanism is a memory-model change, not a shift-register feature.** Our 2048-light wall is a direct consequence of the *whole-frame* DMA model: `esp_lcd`'s i80 backend materialises every light's WS2812 waveform in one contiguous DMA-capable block before the transfer starts, and on the classic ESP32 that block **cannot be PSRAM** — the IDF source rejects it outright (`esp_lcd_panel_io_i2s.c`: `ESP_RETURN_ON_FALSE((caps & MALLOC_CAP_SPIRAM) == 0, NULL, TAG, "external memory is not supported")`), because the classic chip's I2S DMA physically cannot address the external-RAM window. So the DMA buffer scales linearly with the light count against a ~76 KB internal wall. -hpwit's driver uses a **streaming refill ring** instead: a handful of tiny DMA buffers (144 bytes each for RGB — *one LED slot across all 16 lanes*), refilled on the fly by the I2S completion ISR, which transposes the next pixel row out of the source color array as it goes. **The DMA never touches the source array — the CPU does, inside the ISR** — so the source array is under no DMA constraint and **can live in PSRAM**. Internal RAM becomes **constant (~1.2 KB at the default `nbDmaBuffer = 6`), independent of the light count**. That is the whole ballgame: the ring **decouples LED count from internal RAM**, for **any** driver built on it, virtual or not. +hpwit's driver uses a **streaming refill ring** instead: a handful of tiny DMA buffers (144 bytes each for RGB — *one LED slot across all 16 lanes*), refilled on the fly by the I2S completion ISR, which transposes the next pixel row out of the source color array as it goes. **The DMA never touches the source array — the CPU does, inside the ISR** — so the source array is under no DMA constraint and **can live in PSRAM**. Internal RAM becomes **constant (~1.2 KB at the default `nbDmaBuffer = 6`), independent of the light count**. That is the whole ballgame: the ring **decouples LED count from internal RAM**, for **any** driver built on it, expander or not. **The ring must be a second classic driver, not a flag on the existing one.** Our i80 driver goes through IDF's `esp_lcd`, which *owns* the DMA and only does whole-frame. **No IDF API can feed PSRAM into classic-ESP32 I2S DMA** — Espressif's own workaround *is* the ring ("add a buffer in internal memory that is DMA'able, and use memcpy to move data from/to that buffer"). So a ring driver means going under `esp_lcd` to the raw I2S registers, exactly as hpwit does, and it lands **alongside** the i80 driver as a user choice: **i80 = simple, WiFi-underrun-immune, ~2 K lights; ring = PSRAM-fed, many-K lights, needs the `nbDmaBuffer` flicker cushion.** @@ -95,24 +95,24 @@ With the internal-RAM wall gone, the binding constraints become, in order: The ring's weakness is the exact mirror of whole-frame's strength. Every buffer completion needs the ISR to refill *before* the DMA consumes the next one. If a higher-priority ISR (WiFi), or a flash-cache-off window, or a PSRAM stall delays it, the DMA runs off the end of valid data and **the strip glitches** — WS2812 latches on a gap of only ~150 µs. -`nbDmaBuffer` **is** the cushion: it buys `nbDmaBuffer × (one pixel's wire time)` of slack. hpwit says so directly in the virtual driver's README — *"Sometimes interrupts can disturb the pixel buffer calculations hence making some artifacts. A solution … is to calculate several buffers in advance. By default we have 2 dma buffers. this can be increased"* — and notes the ceiling case: pushing it to `NUM_LEDS_PER_STRIP` is the full frame, "but it requires too much memory" (i.e. it degenerates back into our model). Mitigations visible in his source: a deeper ring, `_DMA_EXTENSTION` idle padding, **core pinning** (`enableShowPixelsOnCore()`), and **Level-3 interrupt priority** so the refill preempts ordinary WiFi ISRs. +`nbDmaBuffer` **is** the cushion: it buys `nbDmaBuffer × (one pixel's wire time)` of slack. hpwit says so directly in the shift-register driver's README — *"Sometimes interrupts can disturb the pixel buffer calculations hence making some artifacts. A solution … is to calculate several buffers in advance. By default we have 2 dma buffers. this can be increased"* — and notes the ceiling case: pushing it to `NUM_LEDS_PER_STRIP` is the full frame, "but it requires too much memory" (i.e. it degenerates back into our model). Mitigations visible in his source: a deeper ring, `_DMA_EXTENSTION` idle padding, **core pinning** (`enableShowPixelsOnCore()`), and **Level-3 interrupt priority** so the refill preempts ordinary WiFi ISRs. This is not a niche worry — every ESP32 LED library that streams rather than whole-frames fights it, and they all expose the same remedy: a **tunable count of DMA buffers**, raised when interrupt load (typically WiFi) causes flicker. So the cushion depth is a **recognised, standard tuning parameter, not a bespoke hack** — which matters under [*Common patterns first*](../../CLAUDE.md#principles): if we build the ring, `nbDmaBuffer` is the knob users already expect. **The trade, plainly:** our i80 driver's underrun-immunity is a real, defensible property we would be giving up on the ring path — which is exactly why the ring should be an *additional* driver, not a replacement. A 2 K-light install on a WiFi-busy board is *better served* by i80. A 16 K-light install cannot use it at all. -### 2.6 The virtual (74HC595) driver — and why the ring is NOT its prerequisite +### 2.6 The shift-register (74HC595) driver — and why the ring is NOT its prerequisite -**In hpwit's lineage the virtual driver does inherit the ring's PSRAM property.** It multiplies lanes by shifting each ESP32 pin's byte out through a 74HC595 (one per virtual pin, a 74HC245 level-shifting LATCH/CLOCK): **"8 strips out of one single pin … for 15 esp32 pins which gives you 8x15=120 strips."** That multiplication happens *downstream on the wire*; upstream it is the **same I2S ring, the same ISR-side transpose, the same caller-supplied framebuffer** (`__NB_DMA_BUFFER`, default 2, buffers of `(NUM_VIRT_PINS+1) * nb_components * 8 * 3 * 2` ≈ 1152 B). +**In hpwit's lineage the shift-register driver does inherit the ring's PSRAM property.** It multiplies lanes by shifting each ESP32 pin's byte out through a 74HC595 (one per data pin, a 74HC245 level-shifting LATCH/CLOCK): **"8 strips out of one single pin … for 15 esp32 pins which gives you 8x15=120 strips."** That multiplication happens *downstream on the wire*; upstream it is the **same I2S ring, the same ISR-side transpose, the same caller-supplied framebuffer** (`__NB_DMA_BUFFER`, default 2, buffers of `(NUM_VIRT_PINS+1) * nb_components * 8 * 3 * 2` ≈ 1152 B). -**But that does not make the ring a prerequisite for *our* virtual driver — the chip does.** Our backlog sets the virtual driver's acceptance floor at **48 × 256 = 12,288 lights**. Under the whole-frame model that floor is: +**But that does not make the ring a prerequisite for *our* shift-register driver — the chip does.** Our backlog sets the shift-register driver's acceptance floor at **48 × 256 = 12,288 lights**. Under the whole-frame model that floor is: -| Chip | Whole-frame DMA buffer for 48 virtual lanes | Reachable today? | +| Chip | Whole-frame DMA buffer for 48 strands | Reachable today? | |---|---|---| | **Classic ESP32** | must be **internal** RAM (I2S DMA cannot address PSRAM) → past the ~76 KB wall | ❌ **needs the ring** | | **ESP32-S3 / P4** | `esp_lcd` allocates it **from PSRAM** (EDMA reaches external RAM in hardware) | ✅ **no ring needed** | -So the honest conclusion — and the correction to an earlier draft of this document, which asserted a blanket prerequisite — is: **the whole-frame model cannot meet the virtual driver's floor on the classic chip, and comfortably meets it on the S3 and P4.** The S3 i80 driver *already* drives 16,384 lights on a PSRAM DMA buffer (measured, SE16). A virtual driver built on the **existing i80/Parlio base**, targeting **S3/P4 first**, is therefore **not memory-bound** and needs no new memory model. The classic chip's ~2,048-light whole-frame ceiling becomes a **documented limit of that chip**, not a blocker for the feature. +So the honest conclusion — and the correction to an earlier draft of this document, which asserted a blanket prerequisite — is: **the whole-frame model cannot meet the shift-register driver's floor on the classic chip, and comfortably meets it on the S3 and P4.** The S3 i80 driver *already* drives 16,384 lights on a PSRAM DMA buffer (measured, SE16). A shift-register driver built on the **existing i80/Parlio base**, targeting **S3/P4 first**, is therefore **not memory-bound** and needs no new memory model. The classic chip's ~2,048-light whole-frame ceiling becomes a **documented limit of that chip**, not a blocker for the feature. This also resolves — rather than reopens — the loop in our own backlog. `backlog-light.md` records that the chunk-streaming-ring design was **superseded** by the `esp_lcd` whole-frame path, on the reasoning that whole-frame is underrun-immune and reuses the shared `ParallelLedDriver` base. That decision **stands**: it bought immunity at the price of a *classic-chip* ceiling, and the chips that matter for a 12 K-light virtual install already reach PSRAM. The ring is what would lift the **classic** chip specifically — see § 5 for the trigger. @@ -122,11 +122,11 @@ This also resolves — rather than reopens — the loop in our own backlog. `bac **1. It is a whole second classic driver, and it cuts against what just shipped.** `esp_lcd` **owns the DMA** and only does whole-frame, so the ring **cannot be a flag** on `I80LedDriver` (§ 2.1). It means going under `esp_lcd` to the raw classic I2S registers — a second, register-level, ISR-driven, underrun-*prone* classic path standing next to the i80 driver we just hardened and measured across three chips. Forking the classic path on the strength of an analysis, right after proving the simpler path works, is the wrong trade of risk for capability. -**2. It is NOT a prerequisite for the virtual driver — that was this document's own error.** An earlier draft of this section claimed the virtual driver's 12,288-light floor forces the ring first. That is true **only on the classic ESP32**. On the **S3 and P4** the `esp_lcd`/LCD_CAM backend already draws its DMA buffer **from PSRAM** (hardware EDMA — § 3, and the measured 16,384 lights on the SE16), so a virtual driver built on the existing i80/Parlio base is **not memory-bound there at all**. The correct scoping is therefore: build the virtual driver **on the drivers we have, S3/P4-first**, and treat the classic chip's ~2,048-light whole-frame ceiling as **a documented limit of that chip**, not a defect blocking the feature. Step 3's virtual-driver bullet already says "write our own against the shared slots encoder" — that instinct was right; this document was wrong to contradict it. +**2. It is NOT a prerequisite for the shift-register driver — that was this document's own error.** An earlier draft of this section claimed the shift-register driver's 12,288-light floor forces the ring first. That is true **only on the classic ESP32**. On the **S3 and P4** the `esp_lcd`/LCD_CAM backend already draws its DMA buffer **from PSRAM** (hardware EDMA — § 3, and the measured 16,384 lights on the SE16), so a shift-register driver built on the existing i80/Parlio base is **not memory-bound there at all**. The correct scoping is therefore: build the shift-register driver **on the drivers we have, S3/P4-first**, and treat the classic chip's ~2,048-light whole-frame ceiling as **a documented limit of that chip**, not a defect blocking the feature. Step 3's shift-register-driver bullet already says "write our own against the shared slots encoder" — that instinct was right; this document was wrong to contradict it. -**3. Nothing currently planned requires it.** With the virtual driver targeting S3/P4, no roadmap item is blocked by the classic memory model. The ring buys the classic chip a 4–8× light-count lift and a competitive win — genuinely valuable, but *optional*, and optional work does not preempt work that is on the critical path. +**3. Nothing currently planned requires it.** With the shift-register driver targeting S3/P4, no roadmap item is blocked by the classic memory model. The ring buys the classic chip a 4–8× light-count lift and a competitive win — genuinely valuable, but *optional*, and optional work does not preempt work that is on the critical path. -**The trigger that would un-park it (name it, so this isn't an indefinite maybe):** **classic-ESP32 above ~2,048 lights becomes a shipping requirement** — a board, a user, or a virtual-driver install that must run on classic silicon at high light counts. Until then this document is the finished homework, not a queued task: when the trigger fires, § 2's memory math, § 3's competitive position, and § 5's bench spike are ready to go without re-deriving anything. +**The trigger that would un-park it (name it, so this isn't an indefinite maybe):** **classic-ESP32 above ~2,048 lights becomes a shipping requirement** — a board, a user, or a shift-register install that must run on classic silicon at high light counts. Until then this document is the finished homework, not a queued task: when the trigger fires, § 2's memory math, § 3's competitive position, and § 5's bench spike are ready to go without re-deriving anything. **Related but distinct — don't conflate.** Two other levers raise the classic chip's *lane* count without touching the memory model: Step 3's **parallel-I2S driver** (hpwit `I2SClocklessLedDriver` lineage — the classic I2S peripheral can clock well past our current 8 lanes) and the **virtual/shift-register fan-out**. Those are lane-count work on the existing whole-frame model. The ring is *memory-model* work. They are independent; either can ship without the other. @@ -172,4 +172,4 @@ Keeping i80 is **not** legacy baggage: for a ≤2 K install on a WiFi-busy board **projectMM's own measurements:** - [performance.md § Multi-pin LED driving](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid) — classic i80 2048-light ceiling + `esp_lcd_i80_alloc_draw_buffer` rejecting `MALLOC_CAP_SPIRAM`; S3 16,384 @ ~34 fps; P4 Parlio 4096, 139 fps @ 1024; the `multicore` +44 % table. -- [backlog-light.md](backlog-light.md) — the superseded chunk-streaming-ring decision; the virtual driver's **48 × 256 = 12,288** acceptance floor. +- [backlog-light.md](backlog-light.md) — the superseded chunk-streaming-ring decision; the shift-register driver's **48 × 256 = 12,288** acceptance floor. diff --git a/docs/backlog/livescripts-analysis-bottom-up.md b/docs/backlog/livescripts-analysis-bottom-up.md index 41ebe90b..cf1acd4e 100644 --- a/docs/backlog/livescripts-analysis-bottom-up.md +++ b/docs/backlog/livescripts-analysis-bottom-up.md @@ -1,6 +1,6 @@ # MoonLive — live-script engine landscape analysis -> **Forward-looking research document — exception to CLAUDE.md present-tense rule.** This is a Stage-1 bottom-up survey of *live scripting* for projectMM: running user-authored scripts (LED effects, layouts, modifiers, drivers, sensor logic) on a running device without a recompile-and-flash cycle. It deep-reads one reference implementation — the [ewowi/ESPLiveScript `fix-warnings` fork](https://github.com/ewowi/ESPLiveScript/tree/fix-warnings) of [hpwit/ESPLiveScript](https://github.com/hpwit/ESPLiveScript) — at HEAD on **2026-06-25**, surveys the comparable field (WLED ARTI-FX, embedded VMs, WASM), and extracts the architectural primitives a clean projectMM redesign must decide. Companion to the monthly digest [history/hpwit-ESPLiveScript.md](../history/hpwit-ESPLiveScript.md) (credits + activity log). The **top-down** redesign document ([livescripts-analysis-top-down.md](livescripts-analysis-top-down.md)) expands the decisions recorded here into the build spec. Source citations use `file:line` against the cloned fork; inferred claims are marked *(inferred)*. Modelled on [leddriver-analysis-bottom-up.md](leddriver-analysis-bottom-up.md). +> **Forward-looking research document — exception to CLAUDE.md present-tense rule.** This is a Stage-1 bottom-up survey of *live scripting* for projectMM: running user-authored scripts (LED effects, layouts, modifiers, drivers, sensor logic) on a running device without a recompile-and-flash cycle. It deep-reads one reference implementation — the [ewowi/ESPLiveScript `fix-warnings` fork](https://github.com/ewowi/ESPLiveScript/tree/fix-warnings) of [hpwit/ESPLiveScript](https://github.com/hpwit/ESPLiveScript) — at HEAD on **2026-06-25**, surveys the comparable field (WLED ARTI-FX, embedded VMs, WASM), and extracts the architectural primitives a clean projectMM redesign must decide. Companion to the monthly digest [history/hpwit-ESPLiveScript.md](../history/hpwit-ESPLiveScript.md) (credits + activity log). The **top-down** redesign document ([livescripts-analysis-top-down.md](livescripts-analysis-top-down.md)) expands the decisions recorded here into the build spec. Source citations use `file:line` against the cloned fork; inferred claims are marked *(inferred)*. Modelled on [leddriver-analysis-bottom-up.md](../history/leddriver-analysis-bottom-up.md). ## TL;DR diff --git a/docs/backlog/livescripts-analysis-top-down.md b/docs/backlog/livescripts-analysis-top-down.md index 30bf0c40..a633af3f 100644 --- a/docs/backlog/livescripts-analysis-top-down.md +++ b/docs/backlog/livescripts-analysis-top-down.md @@ -1,6 +1,6 @@ # MoonLive — live-script engine, top-down redesign -> **Forward-looking research document — exception to CLAUDE.md present-tense rule.** **MoonLive** is projectMM's live-script engine (the Moon family: MoonLight, MoonDeck, MoonLive — author an effect as text, see it live). Stage-2 companion to [livescripts-analysis-bottom-up.md](livescripts-analysis-bottom-up.md) (read first: it deep-reads the ESPLiveScript fork, surveys WLED ARTI-FX, the embedded-VM field, and a portable WASM fallback, and ends with the product-owner-direction decisions this document expands). It reasons from projectMM's end goal — *author a script as text, run it on a running device on the next tick* — down to a reference architecture, a concrete API, a performance budget, and a staged spike plan. Modelled on [leddriver-analysis-top-down.md](leddriver-analysis-top-down.md). This expands the eight decisions already made; it does not re-open them. All design is written fresh against projectMM's architecture — prior art (ESPLiveScript, ARTI-FX, MoonLight) is credited, not traced. +> **Forward-looking research document — exception to CLAUDE.md present-tense rule.** **MoonLive** is projectMM's live-script engine (the Moon family: MoonLight, MoonDeck, MoonLive — author an effect as text, see it live). Stage-2 companion to [livescripts-analysis-bottom-up.md](livescripts-analysis-bottom-up.md) (read first: it deep-reads the ESPLiveScript fork, surveys WLED ARTI-FX, the embedded-VM field, and a portable WASM fallback, and ends with the product-owner-direction decisions this document expands). It reasons from projectMM's end goal — *author a script as text, run it on a running device on the next tick* — down to a reference architecture, a concrete API, a performance budget, and a staged spike plan. Modelled on [leddriver-analysis-top-down.md](../history/leddriver-analysis-top-down.md). This expands the eight decisions already made; it does not re-open them. All design is written fresh against projectMM's architecture — prior art (ESPLiveScript, ARTI-FX, MoonLight) is credited, not traced. ## TL;DR @@ -427,7 +427,7 @@ Credits also live in the bottom-up's *Prior art & credits* and the digest [histo ### Public credit — to lift into `docs/moonmodules/core/MoonLive.md` when the module spec is written -The credits above are the analysis's internal record. The block below is the **user-facing** version for the eventual `MoonLive.md` "Prior art" section. Drop it in when MoonLive ships; matches the house style of the other modules' Prior-art sections (e.g. AudioService.md, [LcdLedDriver.md](../moonmodules/light/moxygen/LcdLedDriver.md)). +The credits above are the analysis's internal record. The block below is the **user-facing** version for the eventual `MoonLive.md` "Prior art" section. Drop it in when MoonLive ships; matches the house style of the other modules' Prior-art sections (e.g. AudioService.md, [MoonLedDriver.md](../moonmodules/light/moxygen/MoonLedDriver.md)). > MoonLive's native-codegen approach — compile a small C-like language straight to machine code and call it as a function, so a live-authored effect runs at near hand-written speed — was pioneered by **Yves Bazin (hpwit)** in **[ESPLiveScript](https://github.com/hpwit/ESPLiveScript)**: a from-scratch tokenizer, parser, and Xtensa code generator that drives a 12,288-LED panel at ~85 fps where interpreted languages (Lua, Gravity) managed 3–10. That result is what makes "go native, not interpreted" the right call, and ESPLiveScript is the reference MoonLive is built against — studied closely, credited, and written fresh against projectMM's architecture, never copied, per [*Industry standards, our own code*](../../CLAUDE.md#principles). MoonLive carries the idea forward where ESPLiveScript stops: a multi-ISA backend behind an IR seam (Xtensa, then RISC-V / ARM / desktop) and a binding that makes a script a first-class MoonModule. > diff --git a/docs/backlog/moonlight-effect-inventory.md b/docs/backlog/moonlight-effect-inventory.md index 8b4ed1bc..7a34c197 100644 --- a/docs/backlog/moonlight-effect-inventory.md +++ b/docs/backlog/moonlight-effect-inventory.md @@ -1,6 +1,6 @@ # MoonLight effect inventory (migration reference) -The full set of MoonLight effects to migrate, grouped by **origin library** (a *section* within the shipped `effects.md` catalog page; a per-library page `effects_.md` only when a section outgrows it — see the [folder-structure decision](folder-structure-proposal.md)), with audio/3D markers. Source: [MoonLight effects.md](https://github.com/MoonModules/MoonLight/blob/main/docs/moonlight/effects.md) + the `E_*.h` source files — studied for *behaviour*, reimplemented fresh per the migration plan's *Industry standards, our own code* rule. This reference feeds the [migration plan's](../history/plans/Plan-20260630%20-%20MoonLight%20migration%20(multi-stage).md) Stage-3 batches; it is *what to build*, not a copy of how. +The full set of MoonLight effects to migrate, grouped by **origin library** (a *section* within the shipped `effects.md` catalog page; a per-library page `effects_.md` only when a section outgrows it — see the [folder-structure decision](../adr/0015-library-is-a-tag-not-a-folder.md)), with audio/3D markers. Source: [MoonLight effects.md](https://github.com/MoonModules/MoonLight/blob/main/docs/moonlight/effects.md) + the `E_*.h` source files — studied for *behaviour*, reimplemented fresh per the migration plan's *Industry standards, our own code* rule. This reference feeds the [migration plan's](../history/plans/Plan-20260630%20-%20MoonLight%20migration%20(multi-stage).md) Stage-3 batches; it is *what to build*, not a copy of how. **Markers:** ♫ / ♪ audio-reactive · 🧊 native 3D. **Status:** ✅ already in projectMM · ⬜ to migrate. @@ -62,7 +62,7 @@ The full set of MoonLight effects to migrate, grouped by **origin library** (a * Already in projectMM, our own (not from a MoonLight library — kept here so the inventory is complete): AudioSpectrumEffect ♫, AudioVolumeEffect ♫, FireEffect, GlowParticlesEffect, LavaLampEffect, MetaballsEffect, NetworkReceiveEffect, PlasmaEffect, PlasmaPaletteEffect, RingsEffect, SpiralEffect, CheckerboardEffect. -*(Several have a MoonLight/WLED lineage in their prior-art notes; "origin" here is the page they'll file under — settle per-effect at migration time, per the [folder-structure decision](folder-structure-proposal.md): the page is the primary-steward bucket, the `tags()` emoji carries full lineage.)* +*(Several have a MoonLight/WLED lineage in their prior-art notes; "origin" here is the page they'll file under — settle per-effect at migration time, per the [folder-structure decision](../adr/0015-library-is-a-tag-not-a-folder.md): the page is the primary-steward bucket, the `tags()` emoji carries full lineage.)* ## Tally diff --git a/docs/backlog/multicore-analysis-bottom-up.md b/docs/backlog/multicore-analysis-bottom-up.md deleted file mode 100644 index 21c06521..00000000 --- a/docs/backlog/multicore-analysis-bottom-up.md +++ /dev/null @@ -1,155 +0,0 @@ -# Multicore & driver scaling — landscape analysis - -> **⚠️ DATED SURVEY (2026-07-12) — several conclusions have since been SUPERSEDED by shipped work. Read this header before trusting any number below.** -> -> What still holds: the Parlio 65535-byte/lane single-transfer ceiling and its bytes-per-light math; the two driver bugs found + fixed; MoonLight's model as read; the frame-pacing decision (Appendix A). What has been overturned: -> -> | This doc says | Reality (shipped since) | -> |---|---| -> | "the DMA transmit-wait is ~0 µs — the driver does **not** block on the transmit" | **False.** The transmit *did* block. Step 1.5 (async double-buffer, `asyncTransmit`) shipped to overlap it, worth P4 48 → 76 fps. The measurement below missed it because the instrumented build sampled the wait in the wrong place. | -> | 8 lanes; `LcdLedDriver` is S3-only | **16 lanes** shipped for both parallel drivers; the driver is `I80LedDriver` and runs on **classic ESP32 (I2S) + S3/P4 (LCD_CAM)**. | -> | a parallel-I2S driver is future work | **Shipped** (classic ESP32, I2S i80). Its ceiling is internal-RAM, not a transfer cap: 2048 lights at 8 lanes. | -> | "multicore last, gated on need" | **Step 2a shipped** — the whole output stage runs on core 1 (`multicore` on Drivers): +44 % fps, 85 % of the output off the render core. | -> | Parlio must use a refill ring | It uses the **same whole-frame double-buffer** as i80; the ring is unneeded and deferred indefinitely. | -> -> Current, present-tense numbers live in [performance.md § Multi-pin LED driving](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid) and [§ Multicore](../performance.md#multicore-the-whole-output-stage-on-core-1-multicore-step-2). The build plan and its outcome are in the [top-down](multicore-analysis-top-down.md). This document is kept as the **design-intent record of how we got there** — including the wrong turn, which is the point. - -> **Forward-looking research document — exception to CLAUDE.md present-tense rule.** A Stage-1 bottom-up survey of *scaling the render pipeline* for projectMM: how large a display each driver can drive, where the time actually goes at scale, and whether a second core (à la MoonLight) earns its place. It combines a read of **MoonLight's documented dual-core architecture** ([moonmodules.org/MoonLight/develop/architecture](https://moonmodules.org/MoonLight/develop/architecture/), read **2026-07-12**) with **hardware measurement on the ESP32-P4 at 128×128 = 16384 lights** the same day, and the [frame-pacing decision](#appendix-a--frame-pacing-decided-against) settled with the product owner. Per *[Industry standards, our own code](../../CLAUDE.md#principles)*: study the reference, measure our own, write our own recommendation. Consolidates the multicore threads scattered across the backlog (see [§ Existing backlog](#existing-backlog-this-consolidates)). Citations use `file:line` against projectMM `HEAD`; the ESP-IDF ceilings cite the IDF HAL; MoonLight claims cite the architecture page. - -## TL;DR - -- **The headline capability question — "how many LEDs can each driver drive?" — has a measured answer, and it surprised us.** The bound is *not* DMA bandwidth or lane count; it is (1) a per-peripheral single-transfer ceiling and (2) the CPU **encode** cost. Measured on the P4: rendering 16384 lights takes ~2.5 ms and the driver's per-frame **WS2812 transpose encode takes ~24 ms** (85% of it). **The "and the DMA wait is ~0" reading below was WRONG** — the transmit really did block; Step 1.5's async double-buffer later recovered it (P4 48 → 76 fps). The durable half of the finding stands: the encode is CPU-bound and dominates. -- **Per-driver maximum lights per pin (the number the product owner asked to pin down):** - - | Driver | Chips | Lanes/pins | **Hard per-pin ceiling** | Practical per-pin | Bound by | - |---|---|---|---|---|---| - | **RmtLedDriver** | classic ESP32, S3, P4 | up to 8 (RMT TX channels) | **none** — streams via ping-pong; only `nrOfLightsType` (65535 on classic uint16, ~4 B on PSRAM uint32) + memory | ~1024 (fps) | fps / memory | - | **LcdLedDriver** (i80) | S3 | exactly 8 | **none** — chains DMA descriptors; type + memory only | ~1024 (fps) | fps / memory | - | **ParlioLedDriver** | P4 | 1..8 | **65535 BYTES/lane HARD** — one-shot transfer capped at `PARLIO_LL_TX_MAX_BITS_PER_FRAME` = 524287 bits = 65535 bytes. In *lights* this depends on channels/light (see below): **897 RGB**, ~673 RGBW, ~538 RGBCCT | 897 RGB (hardware) | Parlio single-transfer register | - - **The Parlio ceiling is a BYTE limit, not a light limit** — because the DMA buffer holds the WS2812 *waveform*, not colour bytes: one light = channels × 8 bits × 3 slots = **24 bytes/channel** (RGB = 72 B/light, RGBW = 96, RGBCCT = 120, a 24× expansion). Plus a ~864-byte latch pad. So max lights/lane ≈ (65535 − 864) / (channels × 24): **897 RGB, 673 RGBW, 538 RGBCCT**. State it as bytes-per-lane, not a fixed light count, since wider fixtures fit fewer. **RMT and LCD have no hard per-pin maximum** — the `kMaxWs2812LedsPerPin = 2048` in the code is a *clamp-and-warn* (a chosen "still animates ≥16 fps" floor, [PinList.h:23](../../src/light/drivers/PinList.h)), NOT a hardware wall: exceed it and the driver clamps + warns, it does not reject. The only true ceilings there are `nrOfLightsType` width and memory. **Parlio is the exception**: its 65535-byte/lane single-shot limit is a genuine hardware register cap (proven below). -- **Max total lights, one driver, one shot, today (8 lanes):** RMT / LCD reach **8 × 2048 = 16384** RGB at the soft-clamp (and could go higher per pin at lower fps); Parlio reaches **8 × 897 ≈ 7176** RGB (its hardware wall; fewer for wider fixtures). **The product owner's "16K over 8 lanes" claim is real for RMT/LCD; for Parlio it needs the chunked-transfer enhancement (below).** -- **fps, not lanes, is the real ceiling above ~1000 lights/lane.** WS2812 clocks 30 µs/light (24 bits × 1.25 µs). All lanes clock in parallel, so 8×2048 = 16384 transmits in the time of one 2048 lane = 2048 × 30 µs ≈ **61 ms ≈ 16 fps** — the product owner's own `800 kHz / 2048 / 24 = 16.27 fps` is exactly this. Add the ~24 ms encode and 16K is a ~10–16 fps display: usable for slow/ambient, a slideshow for fast animation. **Driving *more* lights is a lane-count + memory question; driving them *fast* is an encode + wire-rate question** — and the encode is the CPU-bound part a second core can attack. -- **Where multicore actually helps — the encode, not the wait (corrected).** MoonLight's model (Effect on core 0, Driver on core 1, double-buffered handoff) is sound, but the *reason* it helps is not "hide the transmit wait" (our measurement shows the wait is ~0). It is that the **~24 ms WS2812 transpose is CPU-bound and embarrassingly parallel** (per-row, per-lane). A second core can (a) encode+transmit frame N while core 0 renders effect frame N+1, or (b) split the encode across both cores (each does half the lanes/rows → ~halve the 24 ms). *That* is the win. Plus the classic WiFi-timing-isolation benefit for the transmit. -- **Half the multicore isolation is already ours.** projectMM already pins WiFi to core 0 (`CONFIG_ESP_WIFI_TASK_PINNED_TO_CORE_0=y`), and the render loop already yields cooperatively every tick (`vTaskDelay(pdMS_TO_TICKS(1))`, [platform_esp32.cpp:169](../../src/platform/esp32/platform_esp32.cpp)) — the same conclusion MoonLight reaches ("`taskYIELD()` is not good enough… we need `vTaskDelay(1)`"). So the gap to MoonLight is narrow: spawn a core-1 task for the timing-critical **encode+transmit** and hand it frames. -- **The producer/consumer seam already exists.** Effects produce into a `Buffer`; `Drivers` composites into `outputBuffer_` **only when needed** (≥2 enabled layers OR a LUT-mapped single layer, [Drivers.h:217](../../src/light/drivers/Drivers.h)); a lone no-LUT layer is handed its buffer zero-copy. MoonLight's core-0-produces / core-1-consumes is that same seam across a thread boundary. The one genuinely new piece is the **cross-core handoff** (a double-buffer + notification) and the **structural-change lock** it forces (a mutation while the driver task reads the buffer — the single-task loop never had to guard this; MoonLight uses `layerMutex`, we should consider a frame-boundary pointer-swap first). -- **Memory model: available-memory, not `hasPsram`-branch.** projectMM allocates PSRAM-first-else-DRAM (`platform::alloc` tries `MALLOC_CAP_SPIRAM`, falls back, [platform_esp32.cpp:98](../../src/platform/esp32/platform_esp32.cpp)) and *degrades* on failure (`Buffer::allocate` returns null, `tick()` checks). So the multicore double-buffer should be **allocate-and-degrade** ("try the second buffer; if it fails, run the inline single-task path"), matching the existing pattern — **not** an `if constexpr (hasPsram)` gate. (`hasPsram` is used only to widen `nrOfLightsType`, [light_types.h:54](../../src/light/light_types.h).) -- **Recommendation: fix the driver correctness first (done), optimise the encode next, multicore last, all gated on need.** (1) The two driver bugs this investigation found are **fixed** (below). (2) The biggest single lever is the **~24 ms encode** — before a second core, ask whether the WS2812 transpose can be cheaper (a bit→slot lookup table, or skip-when-frame-unchanged). (3) Multicore (encode+transmit on core 1) is the parallelism play once single-core encode optimisation is exhausted — and only when a measured flicker/throughput ceiling justifies the second task + handoff machinery, per the [Task core-pinning note](backlog-core.md)'s "defer until contention is observed." -- **Future work that reshapes the ceilings** (product-owner-flagged, 2026-07-12): the axis that matters is **lanes, not per-lane depth** — 30 µs/light caps a lane at ~1024 LEDs for animation (~33 fps; 2048/lane is 16 fps, a slideshow), so *animated* total scales with lane count. **16 lanes (choose 1..16)** for all parallel drivers is the valuable raise (LCD/RMT → 16×1024 = 16384 animated; Parlio → 8×897 RGB one-shot per its byte-limit); a **virtual (shift-register) driver** (hpwit `I2SClocklessVirtualLedDriver` style, ~120 outputs via 74HC595-class fan-out) multiplies lanes past the pin count — the right lever for still-more animated LEDs. A **Parlio chunked-transfer** enhancement only closes the ~897→1024 band (marginal, since >1024/lane is a slideshow anyway); a **parallel-I2S driver** (hpwit `I2SClocklessLedDriver` style) adds another `ParallelLedDriver` peripheral. All inherit the shared-base fixes below. The build ordering is in the [top-down § Step 3](multicore-analysis-top-down.md#step-3--lanebyte-ceiling-raises-independent-track). -- **Out of scope for this survey (→ [top-down](multicore-analysis-top-down.md)).** The build spec — the encode-optimisation schemes, the core-1 task-creation + handoff primitive + buffer-swap, the structural-change race resolution, and how the ceiling-raises fold into the driver backlog — all live in the top-down. This document is the findings + the ceilings + the ordered *what*; the top-down is the *how*. - -## The hardware measurement (P4, 16384 lights, 2026-07-12) - -Grid set to 128×128 on the P4 (`MM-P4`, `esp32p4-eth`), ParlioLed driver, instrumented build. Serial per-second timing, stable across samples: - -| Stage | Time | Notes | -|---|---|---| -| Effect render (`Layer`, BouncingBalls) | ~2.5 ms | scales with grid, cheap | -| Driver `correction.apply` (brightness LUT + reorder + white) | **~4.4 ms** | 15% of the encode | -| Driver `encodeWs2812LcdSlots` (RGB byte → 3-slot bus bytes × lanes) | **~24 ms** | **85% of the encode — the bottleneck** | -| `busWait` (DMA clock-out) | ~0 µs *(MIS-MEASURED — see the header)* | The instrumented build sampled the wait where it could not land. The transmit **did** block (~7.5 ms/frame at 16×256), which Step 1.5's async double-buffer later hid. | - -So the driver's per-frame cost is dominated by the **CPU encode** (the WS2812 transpose, 5.4:1 over `correction.apply`) — that part held up and drove the SWAR work. **The "DMA wait contributes nothing" half did not**: the transmit blocked, and hiding it behind a double-buffer (Step 1.5) was worth 48 → 76 fps on the P4. Both stages were real; this survey only saw one of them. - -The measured multi-pin driving results across all three peripherals (Parlio/LCD/RMT at 128×128, with the bench pins used) are recorded permanently in [performance.md § Multi-pin LED driving](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid) — present-tense, so they outlive this forward-looking analysis. The SWAR win headline: the Parlio `Drivers` tick dropped 35961 µs → ~30100 µs (−16%) at 16384 lights. - -## The two driver bugs this investigation found — FIXED - -Driving the P4 at 8 lanes surfaced two real defects (both fixed in this change; both HW-verified with LEDs burning at 8×896 = 7168 lights): - -1. **Parlio single-transfer hardware ceiling was hit silently.** 8 lanes × 2048 lights = a 148 352-byte frame = 1 186 816 bits, over the P4 Parlio `PARLIO_LL_TX_MAX_BITS_PER_FRAME` = 0x7FFFF = **524 287-bit** single-transfer limit. `parlio_tx_unit_transmit` returned `ESP_ERR_INVALID_ARG` and **nothing lit, with no error surfaced**. This ceiling is **Parlio-specific**: LCD i80 *chains* DMA descriptors (no frame cap), RMT *streams* through a ping-pong buffer (no frame cap) — verified against the IDF HAL. **Fix:** Parlio's `busInit` now rejects a frame over `kParlioMaxTransferBytes` (65535) up front and reports the init failure as a driver status, instead of creating a unit that fails every transmit ([platform_esp32_parlio.cpp](../../src/platform/esp32/platform_esp32_parlio.cpp)). Per the product owner: **document the 65535-byte/lane (897 RGB) ceiling, don't guard the UI input** — the driver surfaces a clear status; the user's remedy is fewer lights/lane, the start/count window, or (future) chunked transfer. -2. **The parallel-driver bus was reused when the frame SHRANK, keeping an invalid unit.** `reinit()` reused the bus whenever `busCapacity() >= frameBytes_` (grow-only). A shrink (2048 → 896 lights) kept the *oversized* unit whose configured `max_transfer_size` still exceeded the hardware limit — so every transmit kept failing silently even at the smaller, valid size. **Fix:** the shared `ParallelLedDriver::reinit()` now reuses the bus only on an **exact** size match (`==`, not `>=`); any grow or shrink rebuilds, so the bus is always valid-or-rebuilt ([ParallelLedDriver.h](../../src/light/drivers/ParallelLedDriver.h)). This is a **shared-base fix** — every `ParallelLedDriver` inherits it: the i80 driver (LCD_CAM on S3/P4, I2S on the classic ESP32) and Parlio. - -## MoonLight's model, as read (2026-07-12) - -Quoted where specific: - -- **Core assignment.** Core 0: WiFi/BT (prio 23), lwIP (prio 18), **Effect Task (prio 3)**. Core 1: **Driver Task (prio 3)**, ESP32SvelteKit UI (prio 2). "If Driver Task were on Core 0, WiFi would constantly preempt it… DMA/I2S/LCD/PARLIO require uninterrupted timing." -- **Effect vs driver.** Effect = "pure computation… tolerant to preemption," ~60 fps. Driver = "timing-critical… requires uninterrupted execution." -- **Handoff (double buffer).** Effect writes per-layer `virtualChannels` → mutex → `compositeLayers()` → `channelsD` → `newFrameReady=true` → release (~10 µs). Driver captures `channelsD`, DMA-sends, gives `channelsDFreeSemaphore`. "Double buffering overhead is negligible (<1%)." -- **Structural lock.** `layerMutex` guards `mapLayout()`, `Node::onSizeChanged/loop`, `NodeManager::onUpdate` against the two render tasks. -- **Watchdog.** At 16K+ LEDs: `esp_task_wdt_reset()` + `vTaskDelay(1)`; "taskYIELD() is not good enough… only yields to tasks of equal or higher priority." - -## Mapping onto projectMM - -| MoonLight | projectMM today | Verdict | -|---|---|---| -| WiFi pinned to core 0 | already pinned | ✅ same | -| `vTaskDelay(1)` yield | already `vTaskDelay(1)` | ✅ convergent | -| Effect (c0) + Driver (c1), two tasks | one task (`mm_main` loop) | ❌ the core of the work | -| `channelsD` + semaphore handoff | one `outputBuffer_`, no cross-core handoff | ❌ need double-buffer + notification | -| `layerMutex` for structural changes | structural changes inline in one loop | ⚠️ becomes necessary going multi-task — consider a frame-boundary pointer-swap over a hot-path mutex | -| Producer/consumer across cores | producer/consumer within one task | ✅ same model, not yet across a thread | -| Fixed 60 fps | uncapped + time-aware effects | 🔀 deliberate divergence — Appendix A | -| Blocks on transmit | **does NOT — encode dominates, wait ~0** | 🔬 our measurement; multicore targets the *encode* | - -## Buffering models compared (2026-07-12) - -Before adding buffers (Step 1.5 double-buffer, Step 2 cross-core, Step 4 chunked), we surveyed the buffering techniques the field uses, so our choice is *Industry standards, our own code* and not a guess. **There are THREE distinct buffering models, and which one a peripheral needs is forced by its DMA hardware — not a free choice.** Studied under [*Industry standards, our own code*](../../CLAUDE.md#principles): the models below are the recognized techniques; the source links are where to study each; we carry the technique and write our own code against our architecture, not trace any implementation's structure. - -| Model | Where the technique appears | How it works | WiFi-flicker risk | Buffers | -|---|---|---|---|---| -| **Whole-frame → one burst** | **projectMM (LCD + Parlio today)** | encode the WHOLE frame into a DMA buffer, then fire ONE autonomous transfer | **ZERO mid-frame risk** — the buffer is complete before the DMA starts; nothing races it | 1 big | -| **Whole-frame ping-pong** | the standard double-buffer / deferred-wait pipeline (e.g. FastLED's LCD_CAM path) | two whole-frame buffers; encode N+1 into the back while DMA drains N from the front; wait-at-*start*-of-next-`show()` | zero mid-frame; adds **1 frame latency** | 2 big | -| **Chunk-streaming ring** | the standard DMA-ring / transpose-on-the-fly technique (e.g. FastLED's Parlio path, the I2S-clockless family with a tunable `nbDmaBuffer`) | the frame does NOT fit one DMA transfer, so it's split; a small ring of chunk buffers is **refilled on-the-fly by an ISR** as each drains (transpose-on-the-fly) | **HIGH** — a late refill ISR underruns mid-frame → a glitch pulse latches the strand early → **flicker** | 3–**75** small | - -**The key finding — why whole-frame buys flicker-immunity for free.** The chunk-streaming model has a conveyor-belt property: the DMA reads while an ISR refills behind it, so **if WiFi preempts the refill ISR for tens of µs, the DMA drains an unfilled chunk → underrun → flicker.** The standard mitigation is a *timing cushion* — a deep DMA ring (a large `nbDmaBuffer`); we ran one at **75** on the S3 in the StarLight era (75 LED-rows of pre-filled runway so a WiFi interrupt burst can't catch the DMA up to an empty slot). **projectMM's whole-frame-then-burst model doesn't hit this at all**: the DMA reads a *finished, self-contained* buffer, so no ISR races it and WiFi cannot underrun a frame mid-transfer. That is why we never needed — and never discuss — a large `nbDmaBuffer`: our peripheral choice (LCD_CAM chains DMA descriptors, Parlio does one autonomous transfer) sidesteps the case a deep ring exists to cover. - -**Two DISTINCT WiFi-vs-LED failure modes — don't conflate them:** -- **DMA underrun** (the deep-ring case): WiFi preempts the *refill ISR* mid-frame → glitch. **Only affects chunk-streaming.** Whole-frame is immune. -- **Core-0 starvation** (the LC16 finding, [this doc's measurement + top-down § Step 2](multicore-analysis-top-down.md)): a heavy *encode* hogs core 0 so the *network stack* (also core 0) starves → HTTP dies, and on a device whose LED timing shares core 0, the render can hitch. **Affects whole-frame too** — it's the encode, not the DMA. The fix is the multicore pipeline (encode on core 1), a *different* fix than more buffers. - -**Buffer count is not pipeline depth.** A driver can hold several buffers yet still serialize if it waits on the previous transfer at the *top* of each frame (`wait_all_done(portMAX_DELAY)` before encoding) — the buffers exist but encode↔transmit never overlap (frame latency = encode + transmit, not `max`). The overlap comes from wait-*placement*, not buffer count: our Step 1.5 waits at the *start* of the next `show()` and encodes into the back buffer, which is what produces the overlap. (Reference to study for the ring machinery: [troyhacks/MoonLight parlio.cpp](https://github.com/MoonModules/MoonLight/blob/main/src/MoonLight/Nodes/Drivers/parlio.cpp).) - -**Latency vs throughput — the sound-reactive tradeoff (measured).** Buffering trades latency for throughput. One frame is imperceptible for animation (so it's rarely called out), but projectMM's **sound-reactive** priority makes it worth stating: - -| Pipeline | Added latency | Frame @125 fps | Sound-reactive impact | -|---|---|---|---| -| **current (single-buffer, synchronous)** | **0 frames** — sample→photons in the same tick | — | ✅ best possible | -| **+ Step 1.5 (double-buffer DMA)** | +1 frame (DMA of N finishes during tick N+1) | ~8 ms | negligible (beat ≈ 500 ms; A/V sync tolerance ≈ 50–80 ms) | -| **+ Step 2 (render↔encode across cores)** | +1 more frame | ~8–22 ms | small | - -So 1–2 frames (~16–30 ms) is well under human A/V-sync tolerance — but it is a real cost. **Consequence for the design: keep the double-buffer OPT-OUT** (a driver/global flag), so a latency-critical sound-reactive setup can keep the 0-latency synchronous path. This opt-out is *our* choice, driven by the sound-reactive requirement. - -**What this means per step (folds into [top-down](multicore-analysis-top-down.md)):** -- **LCD (S3): whole-frame ping-pong** (Step 1.5) — the standard double-buffer pipeline; validated as correct. -- **Parlio (P4): a chunk-streaming ring, NOT a full-frame ping-pong** — two full 16-lane frames don't even fit the 65535-byte cap, so the ring is *forced*, and it merges Step 1.5 + Step 4 into one structure (a small ring + underrun counter, refilled by a worker ISR). -- **Classic-I2S (backlog, below): inherits the underrun/flicker problem** — it's chunk-streaming by nature (no LCD_CAM/Parlio on classic), so it needs the ring **and** the `nbDmaBuffer` flicker-cushion tuning the I2S-clockless family carries. Written into that backlog item so we don't rediscover it. -- **Encode buffer stays SINGLE** in every model — only the DMA target is doubled/ringed. Our plan already does this. - -Sources (technique study, not code to trace): [FastLED LCD_CAM engine](https://github.com/FastLED/FastLED/tree/master/src/platforms/esp/32/drivers/lcd_cam), [FastLED Parlio engine](https://github.com/FastLED/FastLED/tree/master/src/platforms/esp/32/drivers/parlio), [hpwit I2SClocklessLedDriver](https://github.com/hpwit/I2SClocklessLedDriver/blob/main/src/I2SClocklessLedDriver.h), [troyhacks/MoonLight parlio.cpp](https://github.com/MoonModules/MoonLight/blob/main/src/MoonLight/Nodes/Drivers/parlio.cpp). - -## Existing backlog this consolidates - -| Note | Where | Now | -|---|---|---| -| Core-1 driver task + per-module core-affinity | [backlog-light.md](backlog-light.md) | This doc's multicore recommendation, re-aimed at the **encode** | -| Task core-pinning ("defer until contention observed") | [backlog-core.md](backlog-core.md) | Carried forward — gate on measured need | -| Async ArtNet send (PSRAM handoff) | [backlog-core.md](backlog-core.md) | Shares the same handoff primitive — build once | -| sigrok flicker cross-check | [backlog-light.md](backlog-light.md) | The measurement that opens the multicore gate | -| `rmtWs2812Show` fuller error handling | [backlog-light.md](backlog-light.md) | Dependent of the driver-task work | - -**New backlog item this creates: Parlio chunked transfer** — split a lane's frame into ≤65535-byte transactions with correct WS2812 inter-chunk timing (no false latch), lifting Parlio's 65535-byte/lane (897 RGB) ceiling to the soft cap so 8×2048 (and later 16×2048) works. Prerequisite for the product owner's full 16K-over-Parlio claim. - -## Recommendation → the build plan - -The ordered implementation plan lives in the companion **[multicore-analysis-top-down.md](multicore-analysis-top-down.md)** (same bottom-up/top-down split as the LED-driver and live-script analyses). In brief, in dependency order: - -1. **Driver correctness — DONE** (the two bugs above). -2. **Encode optimisation (one core)** — the ~24 ms transpose is the bottleneck; skip-when-unchanged → bit→slot lookup → SIMD. The prerequisite that unblocks or obviates the rest. -3. **Multicore (encode+transmit on core 1)** — only if a *measured* flicker/throughput ceiling justifies it after Step 2. -4. **Lane/byte ceiling raises** (16-lane, Parlio chunking, parallel-I2S) — independent axis, raises maxima not fps. - -The top-down expands each: the encode schemes, the task/handoff primitive, the buffer-swap + structural-change race, and how Steps 3-4 fold into the existing [backlog-light](backlog-light.md) driver items. - -## Appendix A — frame pacing, decided against - -MoonLight targets a fixed 60 fps; projectMM deliberately does not (settled with the product owner 2026-07-12 while assessing PR #45): - -- **Architecture is "render uncapped + effects are time-aware"** (`beatsin8`/`millis()`-driven), a CLAUDE.md hard rule. A whole-engine fps cap is redundant with that rule and papers over any effect that breaks it. -- **Higher fps is smoother, so a cap *reduces* quality** below the hardware ceiling. -- **The LED transmit already paces the render physically** (30 µs/light; the wire rate is the natural limiter, confirmed by the 16.27-fps-at-2048/lane math). -- **UI/WiFi responsiveness is already guaranteed by the per-tick `vTaskDelay(1)` yield** ([main.cpp:583](../../src/main.cpp)), not by frame-rate control. Skipping whole frames adds no yield points; a single long-blocking tick is the only real starvation risk, and pacing doesn't fix that either (a rendered frame blocks its full duration regardless). -- **Parked as a ~15-line opt-in** (`targetFps=0` default = unlimited, an elapsed-time gate) *only if* a genuinely CPU-starved device appears. - -Multicore addresses the real ceiling (the CPU encode + WiFi-timing isolation); frame pacing addressed a problem the yield already solves. Unrelated; only the first is wanted. diff --git a/docs/backlog/multicore-analysis-top-down.md b/docs/backlog/multicore-analysis-top-down.md deleted file mode 100644 index 356ee887..00000000 --- a/docs/backlog/multicore-analysis-top-down.md +++ /dev/null @@ -1,154 +0,0 @@ -# Multicore & driver scaling — top-down build plan - -> **Forward-looking build spec — exception to CLAUDE.md present-tense rule.** The Stage-2 implementation plan for scaling the render pipeline, built on the measured findings in the companion [multicore-analysis-bottom-up.md](multicore-analysis-bottom-up.md) (read that first: it establishes *what the bottleneck is* — the ~24 ms CPU WS2812 encode, not the DMA wait — and *why*; this document is *what to build, in what order*). Same bottom-up/top-down split as the [LED-driver](leddriver-analysis-top-down.md) and [live-script](livescripts-analysis-top-down.md) analyses. Each step below is its own increment: spec → `/plan` → implement → hardware re-verify. Citations use `file:line` against projectMM `HEAD`. - -## The one-line thesis - -The measured bottleneck is the **CPU encode** (the WS2812 bit→slot transpose, ~24 ms of a ~28 ms driver frame at 16384 lights). So the ordered levers are: **make the encode cheaper (one core) → overlap the encode with the transmit (one core, double-buffer) → then parallelise the encode/render (two cores, only if measured need) → then raise the ceilings (independent)**. Fix-the-encode is the prerequisite that unblocks or obviates everything after it. And the ceiling that matters is **lanes, not per-lane depth**: ~1024 LEDs/lane is the practical animated max (30 µs/light → ~33 fps), so more *animated* total comes from more lanes (16-lane, then shift-register fan-out), not from packing a lane deeper (2048/lane = 16 fps, a slideshow). - -**Correction (measured 2026-07-12, Parlio 16-lane sweep, [performance.md § Multi-pin](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid)):** an earlier draft of this thesis claimed "DMA transmit ~0 µs — the encode of frame N+1 already overlaps the clock-out of frame N for free." **That overlap does not exist in the current driver.** The 16-lane sweep decomposes the tick cleanly as `encode (CPU) + transmit (wire), run serially`: at 256 LEDs/lane the tick is 10787 µs = 2807 µs encode + 7980 µs WS2812 wire, giving 93 fps against a 125 fps wire ceiling. The driver **blocks** on the DMA transmit rather than encoding the next frame during it. Recovering that 93→125 is a distinct one-core lever (Step 1.5 below), *separate* from the render/encode multicore pipeline (Step 2) — the two overlap different pairs. - -## Dependency graph - -```text -Step 0 Driver correctness ─── DONE (bottom-up § bugs) - │ -Step 1 Encode optimization (1 core) ── SWAR transpose DONE ──► MEASURE ──► if still short: Step 2 (multicore) - │ │ - ├── Step 1.5 Async transmit + double-buffer (1 core) ─► SHIPPED: P4 48→76 fps (asyncTransmit) - │ │ - └── the fast transpose feeds ────────────────────────┴──► Step 3 (16-lane widening) - │ - Step 4 Chunked transfer (independent) ─► past 4096 lights → 16K -``` - -- **Step 1 unblocks Step 3**: 16 lanes doubles the transpose cost, so it *needs* the fast transpose first — now shipped (the SWAR transpose), so Step 3's 16-lane widening is unblocked. -- **Step 1.5 SHIPPED (2026-07-13): the double-buffer overlaps the blocking wire wait, one core.** Confirmed on both peripherals (P4/Parlio 48→76 fps, S3/LCD driver −32%); the `asyncTransmit` control (default ON) is the knob, the sync path is a provable non-regression, and a new `wireUs` KPI measures the wire floor explicitly. It changed what Step 2 measures: with the wire hidden, the P4 tick is now **effect ~7.3 ms + driver ~3.8 ms serial → 76 fps**, so the *effect render* is the bottleneck Step 2 attacks. See § Step 1.5 for the full result + the `tickTimeUs`-is-not-frame-time trap. -- **Step 1 did NOT obviate Step 2, and Step 1.5 confirms it's the real lever**: with Step 1.5 shipped, the driver is only ~3.8 ms of the P4's ~13 ms tick — the rest is effect render (~7.3 ms) + services. Step 2 puts the driver on core 1 so it overlaps the effect on core 0: system tick → `max(effect 7.3 ms, driver 3.8 ms + wire) ≈ 7.3 ms ≈ the wireUs 133 fps ceiling`. Step 2 overlaps *render↔encode/transmit* (a different pair than Step 1.5's *encode↔transmit*); its shape is the pipeline (render on c0, driver on c1). See Step 2. -- **Step 4 (chunked transfer) is independent**: it raises the *per-frame light ceiling* (Parlio caps a single DMA transfer at ~4096 lights on the P4 — a contiguous-block limit — so 16K needs the frame split across bursts). It's about *how many lights*, not fps; orthogonal to Steps 1.5/2. -- So **Step 1's core (the transpose) is done and re-measured**; Step 1.5 and Step 4 are unblocked one-core levers; Step 2a is **SHIPPED** (the pipeline: the whole output stage on core 1); Step 3 is independently unblocked. - ---- - -## Step 0 — Driver correctness (DONE) - -The two bugs the P4 investigation surfaced are fixed + HW-verified (details in the bottom-up § "two driver bugs found — FIXED"): the Parlio over-limit **fail-loud guard** and the shared **bus-rebuild-on-shrink** fix. The shared-base reinit fix benefits every `ParallelLedDriver` — the i80 driver (LCD_CAM on S3/P4, I2S on the classic ESP32; both shipped) and Parlio. Nothing further here — listed so the sequence is complete. - -## Step 1 — Encode optimisation (the biggest single lever, one core) — SWAR transpose DONE - -**Target:** the ~24 ms `encodeWs2812LcdSlots` transpose (85% of driver time; the `correction.apply` LUT is the other ~4.4 ms and already cheap). One-core, helps **every driver on every target**, no cross-core machinery. - -**What shipped: the zero-memory SWAR transpose (`transposeLanes8x8` in `ParallelSlots.h`).** The data slot is an 8×8 bit-matrix transpose (8 lane bytes → 8 bit-plane bytes); the hot loop now does it with the branch-free 3-delta-swap SWAR trick (Warren, *Hacker's Delight* §7-3; the same shape FastLED's `transpose8x1` uses) instead of a per-bit-per-lane gather. **~an order fewer ops on the 85% slice, zero table, less code.** Pinned bit-perfect by an exhaustive `unit_ParallelSlots.cpp` case (SWAR == naive gather over all lane patterns × masks) plus the on-device loopback self-test. - -**Measured (P4 .133, 128×128 = 16384 lights, 8 Parlio lanes, 2026-07-12):** the `Drivers` container tick dropped **35961 µs → ~30100 µs (−16%, −5.9 ms)**, whole-device fps **25 → 30 (+20%)** — the only change was the transpose, so the whole delta is the SWAR win. Below the theoretical op-count ceiling because `correction.apply` (~4.4 ms) and DMA setup are untouched and the RISC-V compiler doesn't vectorise the 64-bit swaps as hard as hand-SIMD would; a free 20% fps on every frame (animated or static) from a zero-memory, less-code change validates the minimalism call — skip-when-unchanged would have been 0% on this moving content. - -**LCD_CAM path verified (S3 n16r8 .159, 8-lane i80, 2026-07-12):** the *same* `encodeWs2812LcdSlots` runs on the S3's i80 peripheral driving a real 8×8 panel (pin 18) — healthy (`status: None`), encode scales linearly ~6 µs/light (8×64 = 512 → 3.8 ms; 8×512 = 4096 → 23 ms; 8×1024 = 8192 → 50 ms). **Side finding — the S3 LCD single-DMA-buffer ceiling is between 8192 and 12288 lights:** 8×1024 inits, 8×1536 fails with "LCD init failed — check pins / memory" (the ~1 MB single DMA allocation the i80 bus needs won't fit). The i80 bus fixes its transfer size at creation, so this is a hard per-config init limit, not a soft clamp — a candidate for the [LCD/Parlio DMA buffer → PSRAM](backlog-light.md) item. - -**Two candidates were considered and rejected — the minimalism record:** - -- **Skip-when-unchanged (rejected).** A per-driver frame-signature gate that skips the re-encode when the source bytes match last tick. It optimises the *rare* case (static/paused content) while *taxing the common* case (an animated effect changes every frame, so the hash never matches and the gate is pure per-tick overhead), and leaves the actual bottleneck — the animated-frame transpose — untouched. Net-negative against the hot-path rules (unconditional per-tick work + a fragile "every cold-path rebuild must invalidate" contract) for a narrow static-only win. Not built. -- **Bit→slot lookup table (rejected in favour of SWAR).** A 256-entry table trading ~2 KB for the per-bit math still needs the per-lane OR loop, so it's a *modest* win at a memory cost. The branch-free SWAR transpose beats it on every axis — memory (0 vs 2 KB), code size, speed, *and* recognizability (it's the textbook 8×8 transpose) — so the doc's old "lookup table → then SIMD" two-step collapses into the one SWAR step above. - -**Still open (only if measurement shows the transpose is still the ceiling):** a wider SWAR path for the 16-lane case ("N strips × 8 bits" → "8 bit-planes × N-lane words") folds into [backlog-light § 16-lane](backlog-light.md) — study **troyhacks' claimed-faster Parlio transpose** and FastLED's `parallel_transpose.h` there, write our own, credit by name. The single-8-lane transpose is done; 16-lane widens the same construct. - -**Verification:** re-run the 128×128 P4 measurement (the bottom-up's instrumented method); the transpose ms is the KPI. Loopback self-test stays bit-perfect after every change. - -## Step 1.5 — Async transmit + double-buffer (one core) — SHIPPED (2026-07-13) - -**Shipped** as the `asyncTransmit` control on both parallel peripherals, default ON. Measured, same board + config, only the toggle flipped: - -| board / peripheral | driver tick OFF → ON | system fps OFF → ON | -|---|---|---| -| P4 / Parlio (16×256) | 10,820 → 3,790 µs | **48 → 76 fps** | -| S3 / LCD_CAM (16×144) | 17,200 → 11,700 µs | ~15 → ~16 (masked) | - -**What was confirmed, and the thesis correction that mattered.** The double-buffer *does* work — it overlaps the WS2812 wire wait, dropping the driver's CPU tick on both peripherals (P4 −65%, S3 −32%), each matching the measured wire time. The whole-board win is clean on the P4 (48→76 fps, +58%); on the SE16 the driver gain is real but *masked* by ~50 ms of other per-tick render overhead. **The trap this step nearly died on:** `tickTimeUs` measures CPU-in-the-call (a *blocking* wait is inside it, a *deferred* wait is not), so async-ON's ~3,790 µs driver tick reads as "264 fps" — physically impossible past the wire ceiling. The real rate is the *system* tick; the driver number is CPU headroom. Full write-up + the git-worktree baseline that killed the false-regression scare: [lessons.md](../history/lessons.md). - -**The reframed bottleneck (this is what Step 2 now attacks).** With the wire wait overlapped, the P4 tick decomposes as **effect render ~7.3 ms + driver ~3.8 ms**, serial on one core → ~76 fps. The effect (a DistortionWaves at 128×128) is now the dominant cost, not the driver. The measured wire floor is **`wireUs` = ~7,474 µs (133 fps)** on the P4 — a new read-only KPI (the DMA transfer duration, start→done) that makes the ceiling explicit and tracks an overclocked slot rate directly. - -**The design as shipped** (differs from the plan below in three ways): (1) `tick()` is two explicit branches — `tickSync` (the literal original encode→transmit→wait, provably no regression) and `tickAsync` (deferred-wait) — so OFF is byte-for-byte the old timing; (2) allocation follows the flag — OFF allocates ONE DMA buffer, ON requests the second and degrades to sync if it won't fit — so OFF costs nothing; (3) Parlio uses the **same whole-frame double-buffer as LCD**, not a ring: the KPI 4096-light frame fits one transfer, so the ring (Step 4) is unneeded for the fps goal and, per the "beyond ~65K lights is a network-distribution problem, not a single-chip one" call, is deferred indefinitely. **RMT stayed deferred** (its shared per-pin symbol buffer isn't a small double-buffer delta). Opt-out kept: OFF is the synchronous path (ON adds ~1 frame). ON is simply the better configuration and the switch exists to A/B it — the bottom-up analysis measured that one frame as *imperceptible* (well under A/V-sync tolerance), so it is not sound-reactive guidance to turn it off. Full record: [Plan-20260712 - Step 1.5 async transmit double-buffer (shipped)](../history/plans/Plan-20260712%20-%20Step%201.5%20async%20transmit%20double-buffer%20(shipped).md). - -## Step 2 — Multicore: effects on core 0, encode+transmit on core 1 (the pipeline) — Step 2a SHIPPED (2026-07-13) - -**Step 2a shipped: the whole output stage on core 1, one switch, one output buffer.** A core-1 FreeRTOS task (`platform::spawnPinnedTask` + a direct-to-task notification — the new domain-neutral worker seam in `platform_esp32_worker.cpp` / `platform_desktop.cpp`, which the async-ArtNet item wants too) runs **every** driver's `tick()`, while core 0 renders the next frame and services HTTP/WiFi/WS. **One rule, no per-driver predicate:** when the `multicore` control on Drivers is on, the LED encode, the ArtNet packet build and the preview frame build all move — the container owns the mechanism (the handoff buffer, the task, the frame boundary), so there is one split, not one per driver. A driver that writes a socket still hands its bytes to lwIP on core 0 (that is where the stack is pinned): the CPU half offloads, the send does not move, and that is the intent rather than a leak. - -The handoff reuses the single `Drivers::outputBuffer_` (no second buffer): core 0 waits `encodeDone_` before overwriting it, so the cheap composite is the only serialization and the two heavy stages overlap. Allocate-and-degrade: the split engages only when a driver exists AND the buffer allocates — in the identity single-layer case (otherwise zero-copy) it claims one, and if that fails it **does not engage at all**, so every driver ticks inline on core 0 exactly as before and the driver reads the layer buffer directly. There is deliberately no half-split state in which the two cores could wait on each other; the memory-tight board keeps `asyncTransmit`'s DMA overlap (which needs no handoff buffer) and simply runs single-core. The switch stays on and the split re-engages by itself when the memory is there. Live-reconfigure engages/disengages with no reboot (a one-frame quiesce guards the realloc). The read-only `stall` KPI reports the worst core-0 wait at the boundary in the last second — the Step 2b trigger. Design record: `docs/history/plans/Plan-20260713 - Multicore Step 2 render-encode pipeline.md`. - -**Measured by flipping the switch live (SE16, 64² grid, 16×256 = 4096 lights; reproduced ON→OFF→ON):** whole-board fps **32 → 46 (+44 %)**, system tick 30,857 → 21,316 µs, and **`Drivers` on core 0: 15,404 → 2,261 µs — 85 % of the output stage left the render core.** At 16,384 lights the effect is starker: the encode alone is ~49,916 µs, and on core 1 it is fully hidden behind the render (fps ~14 → 37). **Calling lwIP from core 1 costs ~100 µs/frame** (Preview 49 → 91 µs, HttpServer 348 → 409 µs — the TCPIP core-lock plus cache bouncing) against ~13,000 µs of output work removed: a 130:1 trade, which is why no driver is special-cased. The contention that gated this work is gone — an HTTP hammer during a heavy 8192-light encode holds (77 requests, 163 ms median, 1 timeout), where a ~19 ms inline encode used to drop the LC16's Ethernet link. Structural churn (grid resize, ledsPerPin, driver enable/disable = live engage/disengage) across the S3, classic and P4 left every board stable — uptime climbed, no reset. - -**Step 2b (deferred — conditional on a named workload, not on a general fps win): the ping-pong second output buffer.** 2a's one-buffer boundary serializes the composite; a second `outputBuffer_` would overlap even that (core 0 composites into B while core 1 encodes A), costing one full frame buffer (~48 KB at 16K). The `stall` KPI was built as the trigger metric for exactly this decision, and it splits the answer cleanly in two: - -| Workload | measured `stall` | What 2b would recover | -|---|---|---| -| **Heavy effect** (Noise @ 128² — the render-bound case) | **~1 µs** (S3), **~7 µs** (P4), **~15 µs** (classic) | **nothing** — core 0's render already fills the encode window, so it never idles at the boundary | -| **Light effect, many lights** (the output-bound case) | **6–11 ms** (7,628 µs seen at 8192 lights) | **that whole stall** — core 0 finishes rendering early and sits waiting for core 1 to release the buffer | - -So 2b is **not dead, but it is not a general win either**: it pays *only* in the light-effect/many-lights corner. That is a real scenario (a solid colour or a slow gradient across 16 K lights), so the item stays open — but it is gated on the product owner naming that workload as one worth 48 KB, not on a generic "more fps." - -**The memory shape works in its favour.** The extra buffer is CPU-read (not DMA), so it is `platform::alloc` → **PSRAM-first**: on the S3/P4 it is nearly free, and on the memory-tight classic board it simply **fails to allocate and degrades to today's single-buffer path** — the same allocate-and-degrade already used for `outputBuffer_` and the second DMA buffer. Combined with 2a's path being a strict subset (front == back), 2b is a **small additive change, not a rewrite**, and it cannot hurt the board least able to afford it. - ---- - -**The shape (settled 2026-07-12): the pipeline — render effect frame N+1 on core 0 while core 1 encodes+transmits frame N.** Not the split-encode alternative (both cores transposing one frame); the reasoning, from the post-SWAR re-measure + the effect-weight range, is below. "90% of the benefit for 10% of the cost." - -**Why the pipeline, not split-encode — the re-measure conclusion.** The frame cost is `render + encode` on one core; two cores can make it `max(render, encode)` (pipeline) or, in the best case, `~max(render, encode/2)` (a fork-join hybrid). The numbers that decide it (P4, 16384 lights, post-SWAR: encode ≈ 22 ms, DMA-wait ≈ 0): - -*(Model rebuilt on the post-Step-1.5 numbers: the wire is already hidden by the async double-buffer, so the pair the pipeline overlaps is render ↔ encode. P4 at 16K: encode ≈ 22 ms of CPU.)* - -| Effect weight (P4 render, 16K) | 1 core `render+encode` | Pipeline `max(render,encode)` | Hybrid `~max(render,encode/2)` | -|---|---|---|---| -| **Light** (Checkerboard, ~2 ms) | ~24 ms → 41 fps | ~22 ms → **45 fps** | ~11 ms → 89 fps | -| **Heavy** (Noise, ~17 ms) | ~40 ms → 25 fps | ~22 ms → **45 fps** | ~17 ms → 57 fps | - -**How it actually came out (SE16, measured 2026-07-13):** the pipeline shipped and behaves as this model predicts — `Drivers` on core 0 fell 15,404 → 2,261 µs and whole-board fps rose 32 → 46 (+44 %). The predicted "encode-bound floor" is real but *chip-dependent*: on the S3 at 16K the encode (~50 ms) is the floor; on a classic ESP32 at 16K the **render** (~511 ms) dwarfs it, so no overlap helps there — when one stage is 20× the other, the pipeline can only hide the smaller one. - -Render is **not** a rounding error — a heavy effect (Noise) is ~17 ms at 16K on the P4 ([performance.md § Effect compute](../performance.md)), comparable to the encode. Three things fall out: (1) the pipeline gives a **~45 fps floor across the whole effect-weight range** (its `max(render,encode)` is encode-bound at ~22 ms whether the effect is light or heavy); (2) the hybrid's big win is **light-effects-only** (89 vs 45) — for a heavy effect core 0 is busy rendering the whole frame, so there's no spare time to "join" the encode and the hybrid collapses toward the pipeline (57 vs 45); (3) the hybrid costs far more (a fork-join mid-encode, a render→encode phase handoff) **and** carries the WiFi-asymmetry risk on classic/S3 (core 0 rejoining the encode gets WiFi-jittered; the halves finish unevenly and the join stalls on the slower core). The pipeline puts the *whole* encode on core 1, away from WiFi, so it's asymmetry-free on every platform. So: pipeline now; the hybrid is a possible *later* light-effect optimisation, gated on a real need. - -**What the pipeline overlaps (corrected).** The original draft justified the pipeline with a "DMA-wait ≈ 0, so there is nothing to overlap on the transmit" reading — **that measurement was wrong** (the transmit *did* block; see the bottom-up header). The correct account is that there are TWO overlaps and they are handled by two different mechanisms, which **stack**: Step 1.5's per-driver DMA double-buffer (`asyncTransmit`) hides the **wire** behind the encode on one core; Step 2's pipeline (`multicore`) hides the **encode** behind the render on the other core. Turning `render + encode + wire` into `max(render, encode, wire)`. The pipeline's value never depended on the wait being zero — it was always about the encode. - -**Audio / sensors don't need a core.** Audio FFT is ~13–22 µs on a non-completing tick and ~3 ms on the ~1-in-N tick that finishes a 512-sample block ([performance.md](../performance.md); a 22 kHz block spans ~23 ms, longer than a tick, so it completes roughly once per frame); the gyro/IMU is a 50 Hz I²C poll, not compute. Both fit in slack on either core — the encode (~22 ms every frame) dwarfs them. So the core split is just render↔encode; keep audio on the same (WiFi-free-*enough*) core as render so a completing-block tick doesn't land on the timing-critical encode core. - -**Platform notes (the WiFi-asymmetry map):** on the **P4** WiFi runs on a *separate on-board ESP32-C6* co-processor (esp-hosted; the `esp32p4-eth` variant has no WiFi at all), so **both P4 cores are jitter-free** — the pipeline is clean and even the hybrid would be asymmetry-free here. On **classic ESP32 / S3** WiFi is on-die pinned to core 0, so the encode MUST live on core 1 (which the pipeline does anyway) — this is *why* the pipeline is the portable choice. - -**The gate has now been hit (2026-07-12, LightCrafter 16).** Driving 16 lanes at a ~128×128-scale grid, the ~19 ms LcdLed encode (both encode+transmit run inline on core 0) **starved the network stack** — the W5500 SPI-Ethernet servicing (also core 0) got no CPU during the encode, the link dropped, and HTTP timed out, even though the render loop itself kept ticking at 40 fps. This is the measured contention the gate waited for, and it's NOT WiFi-specific: *any* network interface sharing core 0 with a heavy encode starves (the S3 has no separate radio co-processor like the P4's C6). So the pipeline is now justified for the classic/S3 targets — move the encode+transmit to core 1 so the network stack on core 0 keeps breathing. (The P4 is unaffected: WiFi is on the C6, Ethernet is DMA-RMII, and both P4 cores are free — but the pipeline still buys it the fps floor.) - -**The gate is OPEN and Step 2a has SHIPPED (2026-07-13).** One acceptance criterion decided it: the **measured network starvation above**. The originally-mooted sigrok LED-glitch cross-check was *not* required in the end — Step 1.5's whole-frame DMA already makes the output underrun-immune by construction, so an LED glitch was never the thing the pipeline had to fix; the fps and the network responsiveness were, and both are measured. (That skipped criterion is recorded in the plan, per the auditable-gate rule.) What was already ours (so the gap was narrow): WiFi pinned to core 0 on classic/S3; the loop's `vTaskDelay(1)`; the producer/consumer seam. What was new: the thread boundary + the handoff — both now shipped. See § Step 2a above for what landed and the measured outcome. - -**Design (as built):** - -- **Split.** A **core-1 task owns encode+transmit** (the CPU-bound, timing-critical half — WiFi is on core 0, so the transmit is never preempted). Core-0's main loop renders the next effect frame + services HTTP/WiFi/WS. This is MoonLight's Effect-c0/Driver-c1 shape (bottom-up § MoonLight), re-aimed: *both* the encode and the transmit move to c1 (our encode is the cost, not just the transmit). -- **Handoff — one double-buffer + a task notification.** Core-0 composites frame N+1 into buffer B while the core-1 task encodes+transmits frame N from buffer A; swap the pointers at the frame boundary and `xTaskNotify` the core-1 task. A notification, **not** a full mutex+semaphore pair (lighter; the async-ArtNet-send item needs the *same* primitive — **build it once**, shared). -- **Memory — allocate-and-degrade, no `hasPsram` gate.** The second buffer is `platform::alloc` (PSRAM-first-else-DRAM); if it won't fit, **fall back to the inline single-task path** (the existing `Buffer::allocate`-returns-null → `tick()` checks pattern). No `if constexpr (hasPsram)` branch — the bottom-up establishes we reason from available memory, not a PSRAM flag. -- **Structural-change race — frame-boundary pointer-swap before a mutex.** Going multi-task, a structural change (add/delete a module, resize a grid, `prepareTree`) can land while the core-1 task reads the buffer — the single-task loop never had to guard this. MoonLight uses a `layerMutex`; **prefer a frame-boundary pointer-swap first** (the core-1 task reads the frame it was handed until handed a new one; a structural change lands on the *next* composite, never mid-read). Our `applyState`/`prepareTree` run at well-defined points, so verify a pointer-swap is sufficient before reaching for a hot-path mutex. -- **Watchdog.** At 16K+ lights a long encode can trip the task WDT — keep the `vTaskDelay(1)`/`esp_task_wdt_reset()` discipline MoonLight documents (we already yield `vTaskDelay(1)`). - -**Split-encode — considered and set aside (see the re-measure table above).** Splitting one frame's transpose across both cores (~halve the encode → up to 89 fps for a *light* effect) was the tempting alternative, but it competes with rendering N+1 for the same two cores, collapses toward the pipeline for heavy effects, adds fork-join complexity, and inherits the classic/S3 WiFi asymmetry. Kept as a *possible later* light-effect-only optimisation on the P4 (where both cores are jitter-free), not the Step 2 shape. - -**Out of scope for Step 2:** per-module core-affinity controls (a later refinement, only if a specific module needs pinning); desktop/Teensy equivalents (desktop is OS-threaded, Teensy single-core). - -## Step 3 — Lane/byte ceiling raises (independent track) - -These raise the *maxima* (more lights), not the *fps* — a separate axis from Steps 1-2, done when a board or need demands. All inherit the shared `ParallelLedDriver` base + Step 1's fast transpose. - -**The fps wall sets the priority — lanes are the lever, per-lane is not.** WS2812 clocks 30 µs/light and all lanes clock in parallel, so the *per-lane* count alone sets the frame rate (bottom-up § TL;DR): - -| LEDs/lane | fps (transmit-bound) | Verdict | -|---|---|---| -| 512 | 65 | smooth | -| **~1024** | **~33** (≈23 with encode) | **the practical animated ceiling** | -| 2048 | 16 | ambient/slideshow only | -| 4096 | 8 | slideshow | - -So more LEDs *per lane* past ~1024 buys no usable animation — it's slower, not bigger-at-speed. The total that scales **at a usable frame rate** comes from **more lanes** (`~1024/lane × lane-count`): 16 lanes × 1024 = **16384 animated**. That reorders the items below by real value: **16-lane widening is the valuable one** (it's the only lever that raises the *total* without dropping fps); Parlio chunking is **marginal** (it only matters to reach the ~897→1024 band a single Parlio shot can't); per-lane beyond ~1024 is a static-install concern, not an animation one. - -- **16-lane widening — SHIPPED (2026-07-12).** The LCD_CAM + Parlio drivers now drive up to 16 lanes (bus width derived from the pin count; 16-bit slots + the two-pass `transposeLanes16x8` + PSRAM DMA buffer above 8 lanes), verified live on the LightCrafter 16. This is the lever that raised the *animated* total (16×1024 = 16384 at ~33 fps) — it adds parallelism, not per-lane depth. Full record in [backlog-light § 16-lane parallel output](backlog-light.md) (incl. the direct-driver + >16-lane alternatives not chosen). -- **Virtual (shift-register) driver — the multiplier beyond 16 lanes.** Where 16 native lanes aren't enough, a virtual driver clocks external shift registers (74HC595-class) so **one GPIO fans out to 8+ physical strands** — hpwit's virtual-driver work claims up to ~120 outputs. This is the *right* answer to "more total animated LEDs" past the pin count: it multiplies lanes (throughput-preserving parallelism) rather than deepening a lane (fps-killing). In the pipeline as its own `ParallelLedDriver` peripheral, downstream of the 16-lane base. Study hpwit's `I2SClocklessVirtualLedDriver` prior art hard, write our own against `ParallelSlots.h`, credit by name. - - **Build it on the drivers we have — S3/P4 first.** The acceptance floor is **48 × 256 = 12,288 lights**, and which chip it runs on decides whether that floor is reachable on the *existing* whole-frame model. On the **S3/P4** it is: `esp_lcd`/LCD_CAM allocates the DMA buffer **from PSRAM** (the SE16 already drives 16,384 lights that way), so the virtual driver is **not memory-bound there** and needs no new memory model — it is fan-out work on top of the shipped i80/Parlio base. On the **classic ESP32** it is not: I2S DMA cannot address PSRAM, so the whole-frame buffer is internal-only and walls at ~2,048 lights. **That is a documented limit of the classic chip, not a blocker for the feature** — the classic-chip lift would need the PSRAM refill ring, which is a *second classic driver* and is **parked** (see [backlog-light § classic ESP32 I2S](backlog-light.md); the trigger to un-park it is classic-ESP32 above ~2048 lights becoming a shipping requirement). Do not sequence the ring ahead of this item. -- **Step 4 — Parlio chunked transfer — the 16K lever (measured, NOT marginal).** An earlier draft called this marginal, bounding it at the 65535-byte/lane byte cap (897 lights/lane). **The 2026-07-12 16-lane sweep found a lower, harder ceiling: ~4096 total lights (256/lane × 16).** At 512/lane (8192) Parlio init fails — and the cause is *not* the byte cap (256/lane is far under 897): the P4 has 33 MB free heap but the largest *contiguous internal block* is ~368 KB, and the single-shot 16-bit DMA buffer needs one contiguous block. So the real limit is **contiguous memory, hit at ~4096 lights**, well before the byte cap. This makes chunking the path for **Parlio** to reach the 16×1024 = 16384 total the 16-lane widening promised: split each lane's frame into transactions that each fit an allocatable contiguous block (and ≤65535 bytes), with correct WS2812 inter-chunk timing (idle-LOW < 300 µs so the strand doesn't latch mid-frame). **LCD_CAM already reaches 16384 without chunking** — the 2026-07-12 SE16 sweep drove the full 16K (16×1024), because `esp_lcd_i80_alloc_draw_buffer` allocates the DMA buffer *from PSRAM* and so isn't bound by the ~368 KB contiguous-internal-block limit that caps Parlio. So this Step is **Parlio-specific parity work**, not a universal 16K blocker (RMT streams, LCD is PSRAM-backed — neither needs it). Reproduced within 0.3% on a second P4, so the Parlio cap is the peripheral, not a board. Note the *fps* caveat is shared regardless of peripheral: 1024 LEDs/lane is ~13 fps (LCD, measured) — reaching 16K is a *capacity* win, not an *animation* one (see Step 3's per-lane wall + Steps 1.5/2 for the fps levers). Specced in [backlog-light § Parlio chunked transfer](backlog-light.md); measured detail in [performance.md § Multi-pin](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid). -- **Parallel-I2S driver** (classic ESP32 >8 lanes, hpwit `I2SClocklessLedDriver` lineage) — a new `ParallelLedDriver` peripheral; [backlog-light § classic ESP32 I2S](backlog-light.md). The shift-register virtual driver above is the I2S lineage's fan-out extension. **This is the classic chip's *lane-count* lever, and it is independent of the memory model** — the shipped i80 driver caps at **8 lanes** (the `esp_lcd` i80 bus takes exactly 8 or 16, and a WROVER hasn't the free pins for 16), where FastLED's classic I2S engine reaches **24**. That gap is about lanes, not RAM: raising it does **not** need the parked PSRAM ring, and the ring does **not** raise it. Don't conflate the two. -- **Per-model deviceModels pin defaults** — the catalog data that lets a fresh flash pre-fill the right lane GPIOs; the per-MCU usable-pin reference is in [gpio-usage.md § Usable LED-output GPIOs](../reference/gpio-usage.md), the per-device mapping is tracked in [backlog-core § LED output pins](backlog-core.md). - -## What stays out (settled decisions, not steps) - -- **Frame pacing** — decided against (bottom-up § Appendix A): redundant with time-aware effects, the wire rate already paces, the yield already guarantees responsiveness. Parked as a ~15-line opt-in only if a CPU-starved device appears. -- **The DMA-done-wait — REOPENED as Step 1.5 (was wrongly "settled ~0").** The original investigation concluded the transmit wait was ~0 because "the encode of frame N+1 overlaps the clock-out of frame N for free." The 2026-07-12 16-lane sweep **disproved that for the current driver**: it *blocks* on the transmit (tick = encode + wire, serial), so the wait is the full ~8 ms wire time, not 0 — 26% of the frame at 256/lane. The overlap the original claim assumed has to be *built* (a double-buffer + async transmit), which is exactly Step 1.5. So this is no longer "out"; it's the cheapest fps lever. The encode is still *a* lever (Steps 1/2), but it is not the *only* one. diff --git a/docs/backlog/nrf-zephyr-target-analysis.md b/docs/backlog/nrf-zephyr-target-analysis.md index 3c0164fa..8108bebd 100644 --- a/docs/backlog/nrf-zephyr-target-analysis.md +++ b/docs/backlog/nrf-zephyr-target-analysis.md @@ -39,7 +39,7 @@ Zephyr fundamentals check out: **C++20 is a supported standard** (`CONFIG_STD_CP | Time (`millis`/`micros`), yield/delay, `reboot` | `k_uptime_get`, `k_cycle_get_32`, `k_yield`, `k_msleep`, `k_busy_wait`, `sys_reboot` | **Easy** — direct 1:1 | | Heap (`alloc`/`free`, introspection) | `k_malloc` / `sys_heap` + runtime stats | **Easy**; `hasPsram = false`, one heap — the `maxAllocBlock`-style diagnostics need `sys_heap` stats plumbing | | `allocExec`/`writeExec` (MoonLive JIT) | Cortex-M executes from RAM; no W^X obstacle | **Easy mechanically, but pointless today**: MoonLive has Xtensa/RISC-V/arm64 backends, no ARM Thumb-2 emitter — scripted modules fail cleanly (run dark) until a Thumb-2 backend exists, exactly the per-ISA cost the MoonLive IR-seam design anticipated | -| RTOS introspection (`taskSnapshot`), `spawnPinnedTask`/`notifyTask` | `k_thread_foreach`, `k_thread_create` + `k_event`/`k_sem` | **Easy** — but every viable nRF is **single application core** (the 5340/54H20 second core belongs to the radio stack), so the multicore Step 2 render/encode split has no core to pin to. The seam already handles this: `spawnPinnedTask` returns false → the caller runs inline, the documented allocate-and-degrade fallback. Same stance as Teensy ("desktop is OS-threaded, Teensy single-core" — multicore-analysis) | +| RTOS introspection (`taskSnapshot`), `spawnPinnedTask`/`notifyTask` | `k_thread_foreach`, `k_thread_create` + `k_event`/`k_sem` | **Easy** — but every viable nRF is **single application core** (the 5340/54H20 second core belongs to the radio stack), so the multicore Step 2 render/encode split has no core to pin to. The seam already handles this: `spawnPinnedTask` returns false → the caller runs inline, the documented allocate-and-degrade fallback. Same stance as Teensy ("desktop is OS-threaded, Teensy single-core" — measured on the P4, see performance.md) | | GPIO capability/live state | Zephyr `gpio` API + hand-written per-pin caps table | **With effort** — same per-chip table work every target needs | | **LED DMA seams** (`rmtWs2812*`, `i80Ws2812*`, `parlioWs2812*`) | **None of these peripherals exist** | **New seam required**: an `nrfPwm`/`i2s` WS2812 function pair in the same init/buffer/transmit/wait shape. The capability-flag pattern absorbs this cleanly (`rmtTxChannels = 0`, `lcdLanes = 0`, `parlioLanes = 0`, a new lane constant) — the existing drivers go inert exactly as they do on desktop, and a new driver guards on the new flag. Loopback self-test possible via GPIOTE+timer capture, but bespoke | | `UdpSocket`/`TcpConnection`/`TcpServer` | BSD sockets subset, fd-based | **Easy port — IF an IP link exists at all** (the crux, next section) | diff --git a/docs/backlog/pins-analysis-bottom-up.md b/docs/backlog/pins-analysis-bottom-up.md index 6bd5852f..c18afd2e 100644 --- a/docs/backlog/pins-analysis-bottom-up.md +++ b/docs/backlog/pins-analysis-bottom-up.md @@ -78,7 +78,7 @@ ESPHome ([pin schema](https://esphome.io/guides/configuration-types/), [pin-reus **Live pin state is a TESTING tool, not just a UI toy — this is the projectMM-specific reframe.** projectMM already leans hard on hardware self-verification, and live pin state feeds directly into two existing mechanisms: -- **HAL / loopback driver tests.** The LED drivers already do an on-board RMT-RX loopback to prove the output byte-stream (see [backlog-light § LED drivers](backlog-light.md)); a live pin-state view is the *human-facing* counterpart — watch a GPIO actually toggle while a driver runs, confirm the lane is wired where the config says, catch a dead/mis-wired lane a green unit test can't. The [leddriver top-down](leddriver-analysis-top-down.md) argues the real flicker/correctness proof is watching the *actual pin* under load; live pin state is that, surfaced. +- **HAL / loopback driver tests.** The LED drivers already do an on-board RMT-RX loopback to prove the output byte-stream (see [backlog-light § LED drivers](backlog-light.md)); a live pin-state view is the *human-facing* counterpart — watch a GPIO actually toggle while a driver runs, confirm the lane is wired where the config says, catch a dead/mis-wired lane a green unit test can't. The [leddriver top-down](../history/leddriver-analysis-top-down.md) argues the real flicker/correctness proof is watching the *actual pin* under load; live pin state is that, surfaced. - **The mic-health diagnostic** (shipped today: "no samples" = clocks dead / "data line silent" = SD dead). That diagnosis is *inferred* from the sample stream. Live pin state on the mic's SCK/WS/SD would let a user *see* which line is toggling — the exact "which wire is at fault" answer, made direct rather than inferred. The mic debug that cost an afternoon (a strap-pin misread) would have been a glance: SD not toggling → wrong pin. So live pin state is **shared infrastructure for the test framework**, not a cosmetic layer. It's still a *distinct axis* from the ownership map (different question, different data source), and still deferred to its own effort — but it earns its place as a testing/bring-up tool, which raises its priority above "optional polish." The top-down should treat it as a first-class (if separately-built) sibling of the ownership map, wired into the HAL-test story. diff --git a/docs/backlog/pins-analysis-top-down.md b/docs/backlog/pins-analysis-top-down.md index 65ef78e8..61e9e37c 100644 --- a/docs/backlog/pins-analysis-top-down.md +++ b/docs/backlog/pins-analysis-top-down.md @@ -74,7 +74,7 @@ Unclaimed GPIOs need not be listed (or a compact "free: …" summary), the way T ### What phase 1 is NOT -No arbitration, no reassignment, no writing. It *shows* the picture the pin-uniqueness check computes; it does not own or enforce it. That authority is phase 2. Keeping phase 1 read-only is what makes it a small, safe first increment (the same staging the [PinsModule backlog entry](backlog-core.md#pinsmodule--strict-reject-on-add-mode-the-one-remaining-increment) already draws). +No arbitration, no reassignment, no writing. It *shows* the picture the pin-uniqueness check computes; it does not own or enforce it. That authority is phase 2. Keeping phase 1 read-only is what makes it a small, safe first increment (the same staging the [PinsModule backlog entry](backlog-core.md#pinsmodule-strict-reject-on-add-mode-the-one-remaining-increment) already draws). ## 4. The conflict authority (phase 2) @@ -87,7 +87,7 @@ Today two modules can claim the same GPIO and nothing stops it (`RmtLedDriver.pi **Recommendation: soft-flag as the default** (robustness), with the map making the conflict loud, plus an ESPHome-style **explicit shared-pin opt-out** (`allow_other_uses`) for the rare legitimate case (two consumers reading one input). Reject-on-add stays available for the installer/catalog path where a clean tree is wanted. -**Later still**: live pin *reassignment* — the "swap two drivers' pins" case the uniqueness item flags as needing a free intermediate; a coordinator can broker the swap. And it pairs with [disabling-releases-resources](backlog-core.md#disabling-a-module-should-release-its-resources-not-just-stop-its-loop-backlog) so a disabled module's pins show as freed. +**Later still**: live pin *reassignment* — the "swap two drivers' pins" case the uniqueness item flags as needing a free intermediate; a coordinator can broker the swap. And it pairs with the shipped "disabling releases resources" work so a disabled module's pins show as freed. ## 5. Validity: wiring `gpio-usage.md` to a check diff --git a/docs/backlog/system-modules.md b/docs/backlog/system-modules.md index 9bc42150..18becd49 100644 --- a/docs/backlog/system-modules.md +++ b/docs/backlog/system-modules.md @@ -12,7 +12,7 @@ They are projectMM's **Task Manager / Activity Monitor / Device Manager** — th |---|---|---| | Processes / Details (per-process CPU, memory) | **Tasks** — RTOS tasks, modules nested, per-module cost | shipped | | Performance → Memory (`free`, `vmstat`: used/free by type, largest block) | **Memory** — internal vs PSRAM, used/free/largest, per-module `dynamicBytes` | proposed | -| Device Manager (what hardware is present + how it's wired) | **Pins** — the GPIO map, who owns each pin | [backlogged](backlog-core.md#pinsmodule-one-place-that-coordinates-gpio-assignment-backlog) | +| Device Manager (what hardware is present + how it's wired) | **Pins** — the GPIO map, who owns each pin | [backlogged](backlog-core.md#pinsmodule-strict-reject-on-add-mode-the-one-remaining-increment) | The load-bearing lesson from those tools: **they sample existing OS accounting cheaply and always-on; the heavy per-event tracking (UMDH, `malloc_history`, Valgrind, ESP-IDF `heap_trace`) is a separate opt-in profiler.** projectMM already learned this on TasksModule (CPU% is ~5% tick → build-flag opt-in; the task list is a cheap sample → always-on). The same tier applies to Memory (below). diff --git a/docs/coding-standards.md b/docs/coding-standards.md index da093f2b..c98489db 100644 --- a/docs/coding-standards.md +++ b/docs/coding-standards.md @@ -14,6 +14,7 @@ Decided once; not re-derived per file. - **Semantic variable names.** Name variables for what they represent, not just their type. `availableHeap` not `available`, `internalHeap` not `internal`, `lutBytes` not `bytes`. A reader should understand the variable without looking at its assignment. - **No hard line wraps in markdown.** Let the editor soft-wrap. Hard wraps make diffs noisier than they need to be. - **No em-dashes (` — `) in prose.** Use a comma, semicolon, colon, parentheses, or a full stop instead, whichever the clause actually calls for. Applies to docs, comments, and commit messages. (Code is exempt: a literal `—` in a UI string or test fixture stays.) Existing em-dashes get replaced as files are touched, not in a single sweep. +- **A conditional control is registered BELOW the control it depends on.** When a control's visibility (or range, or presence) is gated on another control's value, `addX()` it *after* that control in `defineDriverControls()`/`defineControls()`, so registration order matches the dependency: the driving control first, the dependent one under it. This makes the UI read top-down as cause then effect (toggle `pinExpander` on, the `latchPin` that depends on it appears just below), and it keeps the source order self-documenting: a reader sees the gate, then the thing it gates. Controls that are mutually exclusive (never visible together) still follow the rule by their shared gate, grouped after it. - **American English spelling, everywhere.** Code identifiers, JSON/wire keys, comments, docs, and UI strings all use US spelling: `color` (not `colour`), `serialize`, `optimize`, `initialize`, `normalize`, `behavior`, `center`, `gray`, `canceled`, `analyze`. Two reasons: (1) it's the dominant technical-project convention (LLVM, Chromium, the Linux kernel are American throughout), and (2) the graphics/LED ecosystem we interop with is uniformly American (`CRGB`, `color`, `colorFromPalette`, CSS `color`), so a British identifier fights the whole toolchain. The real hazard mixing creates: a grep for `color` silently misses `colour`, and a wire key that drifts between dialects (`"colour"` vs `"color"`) breaks a cross-device contract without a compile error. So one dialect, chosen to match the ecosystem. Watch-outs: `analysis` is already US (keep it; only `analyse`→`analyze`); a proper noun keeps its own spelling (`Travelrouter`, a product name). Existing British spellings in older prose get converted **opportunistically when a file is touched**, not in one big sweep (same as the em-dash rule above); a code identifier or wire key in the wrong dialect is the higher-priority fix, since it's a correctness hazard, not just style. ## Prefer integers, store values in their native shape @@ -130,7 +131,7 @@ Where each kind of fact lives. The guiding rule: **document a thing once, in the ### The two surfaces -1. **Summary page (hand-written, end-user).** One `.md` per module *group*, a 4-column table — **name + description · gif/image · controls · links**. One row per module, authored as `###` prose blocks that a build-time hook ([`moondeck/docs/mkdocs_hooks.py`](../moondeck/docs/mkdocs_hooks.py)) renders as the table. Catalog controls live here because a catalog module's user surface is its runtime `controls_.add(...)` calls, which no static tool sees. Each group is one flat page under its domain (a group's `type` rides in the *page name*, not a subfolder — the [folder-structure decision](backlog/folder-structure-proposal.md)): +1. **Summary page (hand-written, end-user).** One `.md` per module *group*, a 4-column table — **name + description · gif/image · controls · links**. One row per module, authored as `###` prose blocks that a build-time hook ([`moondeck/docs/mkdocs_hooks.py`](../moondeck/docs/mkdocs_hooks.py)) renders as the table. Catalog controls live here because a catalog module's user surface is its runtime `controls_.add(...)` calls, which no static tool sees. Each group is one flat page under its domain (a group's `type` rides in the *page name*, not a subfolder — the [folder-structure decision](adr/0015-library-is-a-tag-not-a-folder.md)): - `light/{effects,modifiers,layouts,drivers,supporting}.md` — the light-catalog + light-supporting pages (a type may later split by library into `effects_wled.md` / `effects_moonmodules.md`, still flat). - `core/{services,supporting,ui}.md` — the core-services (user-facing modules), core-supporting, and web-UI summary pages. @@ -140,7 +141,7 @@ Where each kind of fact lives. The guiding rule: **document a thing once, in the 2. **Technical page (generated).** `docs/moonmodules/{core,light}/moxygen/.md`, produced from the `.h` by [`moondeck/docs/gen_api.py`](../moondeck/docs/gen_api.py): **Doxygen** (the de-facto-standard C++ parser) emits XML, **moxygen** renders Markdown through a custom Handlebars template ([`moondeck/docs/moxygen-templates/`](../moondeck/docs/moxygen-templates/)). Each page carries the module's **description, variables, and members** from their `///` comments and links to its `.h`; the summary page's per-module `Detail: [technical]` link points here. Two shaping levers beyond raw moxygen, both to keep the page lean: - **Template** (`class.md`): the base class is a one-line `> **Inherits:** [Base]` link — the full inherited-member list is *not* re-dumped on every subclass (it lives once on the base's own page; re-listing `MoonModule`/`EffectBase`'s large surface everywhere is bloat — *No duplication*). - - **Post-process** in `gen_api.py`: a `@card ` directive in a class `///` comment renders to an `` of the module's UI-card screenshot. Doxygen with `GENERATE_HTML=NO` drops `\image`/`@htmlonly`/raw `` from the XML but preserves plain text, so `@card` survives and is rendered here (a missing asset drops the directive — no broken link). A second post-process wraps each member signature's declared **name** in `…name…` so the theme ([`extra.css`](assets/extra.css)) can highlight the name (accent, bold) while the type and arguments stay muted — moxygen emits a flat `` string with no internal markup, so 'colour only the name' can't be done in CSS alone. Site-wide, `extra.css` also colours `h1`/`h2`/`h3` with the theme's primary/accent (the slate theme otherwise renders every heading the same near-white, flattening the hierarchy). + - **Post-process** in `gen_api.py`: three plain-text directives survive Doxygen's `GENERATE_HTML=NO` XML (which drops `\image`/`@htmlonly`/raw `` and relative links but keeps plain text) and are rendered here — **`@card `** → an `` of the module's UI-card screenshot (a missing asset drops the directive, no broken link); **`@moreinfo`** → splits the class description, relocating everything after it to a `## More info` section below the member lists (the deep-dive-at-the-bottom shape); **`@xref{anchor|label}`** → a page-local `[label](#anchor)` link (deliberately not `@ref`, which is a real Doxygen command). A further post-process wraps each member signature's declared **name** in `…name…` so the theme ([`extra.css`](assets/extra.css)) can highlight the name (accent, bold) while the type and arguments stay muted — moxygen emits a flat `` string with no internal markup, so 'colour only the name' can't be done in CSS alone. Site-wide, `extra.css` also colours `h1`/`h2`/`h3` with the theme's primary/accent (the slate theme otherwise renders every heading the same near-white, flattening the hierarchy). The pages are **gitignored, regenerated on build** (flipping to committed-and-drift-gated is a one-line `.gitignore` change plus a gate like `check_firmwares.py`, if PR-review of the generated output ever earns it). Doxygen (a brew/apt binary) and moxygen (via npx) are the one justified non-uv dependency, like ESP-IDF's Python (see CLAUDE.md); absent locally the pages skip and the rest of the site builds, present in CI they render. @@ -150,8 +151,9 @@ A module's per-entity detail — the module description, each variable, each mem - **`///`, not `//`.** Only `///` (Doxygen) comments generate; a plain `//` comment is invisible on the technical page (it's an implementation note for the source reader only). Class-level rationale you want documented goes in `///`. - **Distribute detail to the member it describes; keep the class comment compact.** The class `///` states what the module *is* and its one defining contract — not a wall of per-control / per-method prose. Every public attribute (the config controls) and every public method gets its OWN `///`, leading with a tight one-sentence brief (the generated page shows that first sentence as the member's summary, so the reader scans a list of named declarations each with its purpose). Detail that belongs to one control (its range, what changing it does) lives on that control; detail that belongs to one method (its lifecycle role, its guarantees) lives on that method. This spreads the same information across the entities it documents instead of piling it into the class blurb, so nothing is repeated and each member is self-describing on hover and on the page. Don't restate what a superclass already documents (a `DriverBase` hook's contract lives on `DriverBase`, not re-explained on every driver). Compact, dense prose — relocate detail, don't delete it. +- **Deep dives go under `@moreinfo`, referenced from the brief — never bloat the lead.** When a class or a member genuinely needs *more* than its compact description (a mechanism explainer, a wiring diagram, a legend table for a diagnostic control), it does NOT swell the lead comment. Put an `@moreinfo` line in the **class** `///`; everything after it is deep-dive reference. Doxygen renders the class description *before* the attribute/method lists (a fixed order with no trailing slot), so a post-process (`gen_api.py`) relocates the `@moreinfo` tail to a **`## More info`** section at the bottom of the generated page, below the members — the reader gets a lean lead, the depth is one scroll away. The lead (and any member brief) then **refers** to it with an in-page link: `@xref{|