mpbuild integration

[andrewleech]

Note: this is stacked on top of #36 so includes that commit too.

Add --build flag for local MicroPython firmware building

Overview

This PR adds a --build flag to mpflash that integrates with mpbuild to build MicroPython firmware locally. This solves a critical compatibility issue where pyOCD SWD/JTAG programming requires .hex/.bin/.elf files, but only .dfu files are available from micropython.org downloads.

Problem Statement

When using --method pyocd for SWD/JTAG programming, users encounter errors like:

❌ Flash programming failed: unknown file format 'dfu'

This happens because:

• pyOCD needs .hex, .bin, or .elf firmware files
• micropython.org only provides .dfu files for most STM32 boards
• No local build option existed in mpflash

Solution

The --build flag enables local firmware building that generates all formats needed by different flash methods:

mpflash flash --build --method pyocd --board NUCLEO_H563ZI

Key Features

🏗️ Complete Build Integration

• Seamless mpbuild integration with Docker containerized builds
• Automatic MicroPython repository detection and management
• Generates all firmware formats: .dfu, .hex, .bin, .elf

⚡ Smart Caching System

• Intelligent build caching (5-30 minute builds cached indefinitely)
• Cache invalidation based on board and version
• Avoids rebuilding identical firmware

🔄 Database Integration

• Built firmware automatically imported to mpflash database
• Works seamlessly with existing firmware selection logic
• Method-aware format selection (pyOCD prefers .hex, DFU prefers .dfu)

🛡️ Robust Error Handling

• Python 3.10+ requirement detection with clear error messages
• Docker availability validation
• Graceful degradation on unsupported systems

🎯 Backward Compatibility

• Zero breaking changes to existing functionality
• Works with all existing flash methods
• Optional feature - existing workflows unchanged

Technical Implementation

Core Architecture

class BuildManager:
    """Manages MicroPython firmware builds with caching."""

    def get_or_build(self, board: str, version: str) -> List[Path]:
        # Check cache first
        if cached := self._find_cached(board, version):
            return cached
        # Build using mpbuild
        return self._build_firmware(board, version)

CLI Integration

The --build flag is added to the flash command:

@click.option(
    "--build",
    default=False,
    is_flag=True,
    help="Build MicroPython firmware locally using mpbuild before flashing. Generates all formats (.dfu, .hex, .bin, .elf). Requires Docker.",
)

Dependencies

• mpbuild: Added as optional dependency (uv sync --extra build)
• Python 3.10+: Required due to mpbuild's union type syntax
• Docker: Required for containerized MicroPython builds

Usage Examples

Basic Local Build

# Build and flash latest firmware
mpflash flash --build --board NUCLEO_H563ZI --method pyocd

# Build specific version
mpflash flash --build --board RPI_PICO --version v1.26.0

pyOCD SWD/JTAG Programming

# Now works! Generates .hex files for pyOCD
mpflash flash --build --method pyocd --board STM32F4DISC

Force Rebuild

# Bypass cache and rebuild
mpflash flash --build --force --board NUCLEO_H563ZI

Error Handling

Python Version Check

❌ Build functionality not available: mpbuild requires Python 3.10 or newer (current: Python 3.9).
The --build flag is not available on this Python version.

Docker Requirement

❌ Build functionality not available: Docker is not installed or not running.
Building requires Docker to be available.

Missing mpbuild

❌ Build functionality not available: mpbuild is not installed. Install with: uv sync --extra build
Note: mpbuild requires Docker to build MicroPython firmware.

Files Changed

• mpflash/build.py (NEW, 410 lines): Complete build management system
• mpflash/cli_flash.py (+40 lines): CLI integration and build workflow
• pyproject.toml (+3 lines): Optional mpbuild dependency
• uv.lock (+66 lines): Dependency lockfile updates

Testing

The implementation includes comprehensive error handling and has been tested with:

• ✅ Python 3.9 (clear error message about version requirement)
• ✅ Python 3.12 + Docker (successful build integration)
• ✅ Missing dependencies (helpful installation guidance)
• ✅ Method-aware firmware selection (pyOCD gets .hex, DFU gets .dfu)

Benefits

1. Solves pyOCD Compatibility: No more "unknown file format 'dfu'" errors
2. Local Development: Build custom firmware with local changes
3. All Flash Methods: Works with pyOCD, DFU, UF2, esptool
4. Performance: Smart caching avoids redundant builds
5. User Experience: Single command from problem to solution

Migration Path

This is a purely additive feature:

• Existing users: No changes required, everything works as before
• New capability: Add --build flag when needed
• Optional dependency: Only install mpbuild if you need local builds

Future Enhancements

• Build configuration options (debug/release, specific features)
• Build artifact management and cleanup
• Integration with Git worktrees and custom MicroPython forks
• Parallel builds for multiple boards

---

Before this PR: pyOCD fails with DFU files ❌
After this PR: --build generates all formats ✅

This addresses a fundamental compatibility gap and provides a foundation for advanced local development workflows.

[andrewleech]

Again, I haven't code reviewed this at all yet, so feel free to not look at it at all either until I do so and un-draft the PR!

[Josverl]

Thanks for making a start on this, I'll be patient 😁

[codecov]

Codecov Report

❌ Patch coverage is 17.57402% with 863 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.34%. Comparing base (d0dc6de) to head (dbc8523).

Files with missing lines	Patch %	Lines
mpflash/flash/pyocd_core.py	11.26%	252 Missing ⚠️
mpflash/build.py	0.00%	190 Missing ⚠️
mpflash/flash/pyocd_flash.py	18.27%	152 Missing ⚠️
mpflash/cli_pyocd.py	0.00%	146 Missing ⚠️
mpflash/flash/__init__.py	34.48%	32 Missing and 6 partials ⚠️
mpflash/flash/debug_probe.py	41.93%	35 Missing and 1 partial ⚠️
mpflash/cli_flash.py	61.53%	19 Missing and 1 partial ⚠️
mpflash/flash/worklist.py	55.55%	12 Missing and 4 partials ⚠️
mpflash/flash/pyocd_probe.py	45.83%	13 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #37       +/-   ##
===========================================
- Coverage   75.51%   60.34%   -15.17%     
===========================================
  Files          54       60        +6     
  Lines        3026     4035     +1009     
  Branches      488      655      +167     
===========================================
+ Hits         2285     2435      +150     
- Misses        621     1471      +850     
- Partials      120      129        +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Josverl]

In process of rebasing this on the changes I have been doing - may take a few iterations