Proposal: Migration of colcon-core to PEP 517/518/621/660 Standards

[KmoM88]

Hi everyone, after some investigation/experimentation I have compiled a detailed, phased proposal to bring native PEP 517/518/621/660 support to colcon. As platforms move away from direct setup.py invocations, modernizing python builds is essential. This proposal aims to bridge the standards gap while fully preserving backward compatibility, bootstrapping safety, and the rapid iteration loops expected. I would love to get your thoughts and feedback on the proposed stages and data-linking designs below! I don't claim to be the foremost expert on colcon and this proposal was done with gemini as a help to understand some internal logic, so please feel free to call out any errors or oversights in my analysis, your feedback is incredibly welcome!

1. Context & Legacy Workflow Analysis

1.1 setup.cfg Package Identification and Discovery

colcon-core natively handles Python package discovery and metadata extraction through static parsing of configuration files. The core modules colcon_core.package_identification.python and colcon_core.package_augmentation.python expect the presence of a standard setup.cfg file along with a companion setup.py shim.

• Discovery: The discovery step imports read_configuration from setuptools.config.setupcfg (with fallbacks to setuptools.config depending on the installed version of setuptools) to parse the INI-style settings in setup.cfg directly on the main thread, resolving the package type as python and extracting its name.
• Dependency Resolution: Augmentation maps configuration keys in setup.cfg to unified colcon dependencies:
  • setup_requires is mapped to build dependencies.
  • install_requires is mapped to run dependencies.
  • tests_require / extras_require (specifically test-related keys) are mapped to test dependencies.
• The Static vs. Dynamic Dichotomy: This core design enforces a strict separation between static configuration (fast, safe, and easily parsed on the main thread without side effects) and dynamic script execution (which can contain arbitrary Python code). Consequently, colcon-core natively ignores packages that rely on custom dynamic code inside setup.py files. All dynamic configurations are offloaded to the external colcon-python-setup-py extension, which executes the script in an isolated subprocess running a dry-run invocation of distutils.core.run_setup to prevent execution of installation routines.

1.2 Legacy Build Execution

During the execution phase, PythonBuildTask in colcon_core.task.python.build spawns direct command-line invocations of the setup.py script:

• Static Install: Spawns python setup.py build followed by python setup.py install --single-version-externally-managed.
• Symlink (Editable) Install: Mirrors the module directories from the source tree into the build base using physical OS symlinks, and then runs python setup.py develop --editable pointing to the build base directory. This is augmented by symlink_data (a custom distutils command class) to symlink non-Python data files directly into the target prefix (install/).

During a legacy editable installation (--symlink-install), the build task coordinates three distinct categories of filesystem link operations:

1. temp_symlinks (Build-Time Source Mirroring): Temporary filesystem links created in the build base directory (e.g. /build/my_package/my_module $\rightarrow$ /src/my_package/my_module). These mirror the source package modules so setuptools can execute the develop command inside the isolated build space, keeping the raw source repository clean of .egg-info directories and caches. These symlinks are unlinked immediately after the subprocess execution completes.
2. Editable Install Link (Python .pth Link): A persistent link registered in the target python environment (e.g. /install/my_package/lib/python3.10/site-packages/easy-install.pth) containing the path to the build directory (e.g. /build/my_package), directing Python's sys.path resolver to load modules dynamically.
3. symlink_data (Target Data-File Symlinking): Persistent links created inside the target installation share directory pointing back to assets in the source tree (e.g. /install/my_package/share/my_package/launch/my_launch.py $\rightarrow$ /src/my_package/launch/my_launch.py). Injected via a custom distutils command class, it bypasses standard static data copying so launch configurations, models, and markers remain dynamically editable.

Symlink Category	Scope / Created In	Target / Points To	Lifetime	Operational Purpose
temp_symlinks	Temporary Build Base (build/)	Source Tree (src/)	Temporary (Build-time only)	Isolates setuptools execution to keep source repositories clean of build artifacts.
Editable Install Link	Target site-packages	Temporary Build Base (build/)	Persistent	Registers the package path inside the Python interpreter's sys.path.
symlink_data	Install prefix (install/share/)	Source Tree (src/)	Persistent	Resolves non-Python assets (launch files, URDFs, configurations) dynamically.

1.3 The Bootstrapping Mechanism

To allow developing colcon using its own source repositories without depending on pre-installed instances, colcon employs a bootstrapping script at src/colcon-core/bin/colcon.

• Because colcon is not yet installed in the development virtual environment, standard importlib.metadata entry point hooks are unavailable.
• To bypass this, bin/colcon manipulates sys.path and overrides the module-level function load_extension_points in the colcon_core.extension_point module. It directly registers the local checkouts of verbs, discovery extensions, and task classes.
• During this bootstrapping phase, colcon-core relies entirely on its native static setup.cfg package discovery logic to identify, sort, and build its own checkouts from source.

1.4 Why Upgrade to standard PEP 517/518/621/660 Environments?

Maintaining this legacy model poses immediate operational risks to the colcon and ROS developer communities:

1. Upstream Deprecations: Mainstream Python packaging tools are deprecating direct invocations of setup.py install and setup.py develop. Modern Linux distributions are removing legacy macros, rendering support for legacy setup.py installations in their package managers incompatible with newer Python interpreters.
2. Standards Adoption: Modern Python developers are adopting PEP 517/518 packaging setups using standard backends like Hatchling or Flit. Currently, colcon cannot compile these packages natively.
3. Modern Metadata (PEP 621): Standards specify that package metadata and dependencies should be statically declared in pyproject.toml under the [project] and [build-system] tables. colcon must be updated to parse this modern, standardized layout.

---

2. Current Status of the Upstream Migration

2.1 Upstream PR #732: Core Contributions

The colcon-core PR #732 introduces the foundational building blocks inside colcon_core/python_project/ to support modern PEP 517 build backend hooks:

• AsyncHookCaller: Implements the subprocess execution transport needed to interface with standard build backends (e.g. setuptools.build_meta, hatchling.build, flit_core.buildapi) defined in pyproject.toml.
• Subprocess IPC Channels: Uses unidirectional OS pipes and standard pickle serialization to transmit kwargs and return structures (like requirement arrays) between the parent colcon thread and the isolated child script _call_hook.py. This ensures high reliability on both Linux and Windows (using msvcrt handles).
• TOML Parsing Gating (PR Add function for reading and caching PEP 518 spec #668 ): Introduces a unified TOML parser loader (toml_loads in colcon_core/python_project/spec.py). It dynamically queries the python environment, using the standard library's tomllib in Python 3.11+ while safely falling back to third-party packages tomli or toml in older runtime environments.
• HookCallerDecoratorExtensionPoint: Registers a new extension point allowing downstream plugins to intercept and wrap hook invocations, enabling backend-specific configurations.

2.2 What comes after PR #732?

While PR #732 establishes the underlying hook caller execution and decorator registries, it does not integrate them into the colcon runtime workflow.

As it stands, PR #732 is purely foundational and does not modify:

1. Package Identification: Core discovery still ignores packages that lack a setup.cfg/setup.py pair.
2. Package Augmentation: Core still parses dependencies exclusively from setup.cfg fields.
3. Build Task Execution: PythonBuildTask still executes legacy setup.py command arrays, causing builds of pure PEP 517 packages to fail.

---

3. Key Migration Pitfall: Editable Installs and the Data File Gap

The most significant risk during this migration revolves around the --symlink-install (editable) workflow, which is heavily relied upon by the ROS developer community.

                              Editable Package Installation
                                             │
                      ┌──────────────────────┴──────────────────────┐
                      ▼                                             ▼
            [Python Source Code]                          [Non-Python Data Files]
      (e.g., my_node/subscriber.py)                   (e.g., launch.py, package.xml)
                      │                                             │
          Dynamic Linking Supported!                     Static Copies Packaged!
        Resolved by PEP 660 (.pth)                   ambiguous in PEP 660 standard.
                      │                                             │
                      ▼                                             ▼
          Developer edits take effect                     Developer edits are lost
             without rebuilding.                         until next manual build.

3.1 Python Code Editability under PEP 660

PEP 660 establishes standardized hooks (build_editable) for generating editable wheels. Backends successfully resolve Python code editability by inserting redirection files (such as .pth files pointing to the source directory) or dynamic path hooks into sys.path. Changes to Python source modules take effect instantly without rebuilding.

3.2 The Non-Python Data File Specification Gap & Metadata Loss

The PEP 660 standard is designed with Python source executables in mind. It does not specify any standard mechanism to dynamically link or live-update non-Python data files (such as XML/YAML launch files, URDF robot description models, shared parameters, or custom package index markers).

• The Backend Fallback: Standard modern backends (like Setuptools, Hatch, or Flit) treat these data files as static resources. When compiling an editable wheel, they package these shared resources directly inside the wheel's ZIP archive.
• The PEP 427 ZIP Packaging Barrier: According to the standard wheel packaging format (PEP 427), the packaging backend translates files declared in data_files and places them under a dedicated <package>-<version>.data/data/ or share/ directory structure inside the ZIP archive. During this compilation, all context regarding the original source repository paths is completely lost. The ZIP metadata only preserves the target installation path.
• The Static Installation: When colcon installs the wheel, the installer tool unzips the archive and copies the data files statically from .data/data/ into the target directory (install/share/...). Since the installation engine receives the wheel as a standalone ZIP archive, it is structurally unable to trace these files back to the developer's original source repository.
• The Broken Workflow: Any edits made to a launch file or configuration script inside the source directory will not take effect inside the active workspace environment. The workspace continues to execute the stale static copy inside the install space, forcing developers to run colcon build after every single non-Python file modification. This introduces substantial friction and destroys the rapid iteration loops expected by developers.

---

4. Proposed Phases of Development & Adoption

To execute this migration without breaking the active ROS 2 and wider developer communities, development and adoption must be structured in sequential phases that guarantee backward compatibility and bootstrapping safety at every single step.

┌─────────────────────────────────────────────────────────────────────────────┐
│ Phase 1: Dual Identification, Augmentation, and Gating                      │
├─────────────────────────────────────────────────────────────────────────────┤
│ Phase 2: Standard PEP 517 Build Task Adoption                               │
├─────────────────────────────────────────────────────────────────────────────┤
│ Phase 3: PEP 660 Editable Symlink Restorations                              │
├─────────────────────────────────────────────────────────────────────────────┤
│ Phase 4: Strict Core Separation & Legacy Extension Extraction (Target State)│
└─────────────────────────────────────────────────────────────────────────────┘

Phase 1: Dual Identification, Augmentation, and Gating

• Objective: Update colcon-core to discover and augment modern PEP 517/518 packages, while maintaining absolute backward compatibility with legacy setup.cfg packages.
• Implementation Steps:
  1. Update PythonPackageIdentification to inspect the directory for pyproject.toml. If found, parse it using standard TOML loaders. If the [build-system] or [project] tables are defined, identify it as a python package.
  2. Update PythonPackageAugmentation to resolve modern PEP 518/621 dependency groups (mapping build-system.requires to build dependencies, project.dependencies to run dependencies, and project.optional-dependencies.test to test dependencies).
  3. The Legacy Feature Gate: Introduce an environment variable (COLCON_ENABLE_LEGACY_SETUP_CFG, defaulting to true). If set to false, the legacy setup.cfg/setup.py static identification fallback is bypassed. This provides developers a clean path to opt-out and test modern-only environments.
  4. Bootstrapping Safety: Ensure that the developer bootstrap script (bin/colcon) is updated to register the new PEP 517 identification and augmentation classes alongside the legacy classes. This guarantees that colcon-core can safely discover, sort, and build its own source tree during development bootstrapping.

Phase 2: Standard PEP 517 Build Task Adoption

• Objective: Implement standard, backend-agnostic wheel compilation and target installation inside the core build task, bypassing legacy commands.
• Implementation Steps:
  1. Refactor PythonBuildTask to branch into a PEP 517 execution path (_build_pep517) when pyproject.toml is present and setup.py is absent.
  2. Initialize the build backend hook caller. Invoke the build_wheel hook of the declared backend to compile the package into a standard wheel (.whl) within the temporary <build_base>/wheel workspace directory.
  3. Predictable Target Installations: Execute the active virtual environment's interpreter inside a subprocess to target-install the wheel:

     python -m pip install --no-index --no-deps --target <install_base>/<python_lib> <wheel_path>

     • Why --target is highly preferred over --prefix: Under standard --prefix installations, installers are subject to operating system and packaging scheme overrides (such as Ubuntu/Debian overriding python install layouts to place files in dist-packages instead of site-packages). By redirecting standard installations with the --target flag, colcon completely bypasses these platform-specific layout overrides, ensuring that files are unpacked into a predictable, platform-agnostic directory structure which can be cleanly sourced by environmental setup scripts.
  4. Entry Point Script Wrapping: Parse the unpacked package's .dist-info/entry_points.txt metadata and dynamically generate standard executable python scripts inside the target bin/ directory with 0o755 permissions, ensuring console script executables work natively.

Phase 3: PEP 660 Editable Symlink Restorations

• Objective: Integrate build_editable editable installation hooks and restore the dynamic --symlink-install experience for non-Python data files.
• Implementation Steps:
  1. Update the build task to check if the backend implements the build_editable hook. If not, log a warning and fall back to a standard wheel build.
  2. If editable installations are requested and supported, invoke build_editable to get an editable wheel, and install it.
  3. Deploy a symlinking strategy (such as backend-specific decorators or frontend out-of-band linking) to replace the statically copied files inside install/share/ with dynamic symlinks pointing straight back to the original files in the source tree.

Phase 4: Strict Core Separation & Legacy Extension Extraction (Target State)

• Objective: Purge all historical configuration code from colcon-core, rendering the core package 100% standards-compliant and highly maintainable.
• Implementation Steps:
  1. Create a new, retroactive standalone extension package: colcon-python-setup-cfg.
  2. Extract the static setup.cfg package identification, augmentation, and dependency parsing logic out of colcon-core and place it inside colcon-python-setup-cfg.
  3. Remove the COLCON_ENABLE_LEGACY_SETUP_CFG feature gate and all associated legacy static code from colcon-core.
  4. At this point, colcon-core becomes purely standards-compliant (PEP 517/518/621/660). Users requiring legacy configurations can seamlessly maintain compatibility by installing the standalone colcon-python-setup-py and the new colcon-python-setup-cfg extensions.

---

5. Proposed Build Pipeline Modifications

5.1 Package Identification & Augmentation

The package identification step must inspect packages in priority order:

1. Check for Modern Specification: If pyproject.toml is present, parse the file. Verify that a [build-system] table is present. Extract the package name from the [project] static metadata table, and mark the descriptor type as python. Caches the parsed TOML structure inside desc.metadata['python_project_spec'] to prevent redundant disk I/O in the downstream augmentation step.
2. Legacy Fallback: If pyproject.toml is absent, and the environment gate COLCON_ENABLE_LEGACY_SETUP_CFG is active, check for the presence of the setup.cfg/setup.py pair and fall back to legacy static parsing.

The package augmentation step must extract dependencies in a unified format:

• Static modern resolution: If the metadata was pre-cached from pyproject.toml, read and resolve standard dependency arrays under the [project] table directly on the parent thread.
• Dynamic hook fallback: If dependencies are declared as dynamic (e.g. dynamic = ["dependencies"] under the [project] table), use the AsyncHookCaller subprocess to invoke get_requires_for_build_wheel to retrieve requirements dynamically.

5.2 Build Task (PythonBuildTask)

The build task execution block will be updated to execute the following logic:

graph TD
    Start[Start Build Task] --> Check{Is pyproject.toml present and setup.py absent?}
    Check --> |Yes| PEP517[PEP 517 Build Pipeline]
    Check --> |No| Legacy[Legacy setup.cfg Build Pipeline]
    
    PEP517 --> Hook[Invoke backend build_wheel hook via AsyncHookCaller]
    Hook --> Install[Target-install wheel into install_base via pip subprocess]
    Install --> Scripts[Parse .dist-info/entry_points.txt and generate bin/ console wrappers]
    Scripts --> Environment[Create colcon environment hooks & setup scripts]
    
    Legacy --> RunLegacy[Spawn setup.py build, install, or develop]
    RunLegacy --> Environment
    
    Environment --> End[End Build Task]

Loading

---

6. Options for Implementing Editable Data-File Linking under PEP 660

To resolve the data file specification gap during Phase 3, two distinct architectural options can be considered.

Option A: Hook Caller Decorator Interception

This option utilizes the HookCallerDecoratorExtensionPoint plugin registry introduced in PR #732 to intercept the backend.

• Implementation: A decorator tailored specifically for the target backend (e.g. setuptools.build_meta) intercepts the build_editable execution. It inspects the package's configuration files to parse source-to-destination mappings for data_files. Once the wheel is generated and installed, the decorator intercepts the install directory, deletes the static data files, and replaces them with direct OS symlinks pointing back to the source tree.
• Pros: Highly modular. Avoids polluting colcon-core with backend-specific code. Standard Python installers remain unaffected.
• Cons: Backend-specific. A decorator must be written and maintained for every PEP 517 backend (Setuptools, Hatch, Flit) used in the workspace. Introspecting non-setuptools configs to resolve original source paths might be complex.

Option B: Frontend-Driven Out-of-Band Symlinking

This option resolves symlinking entirely within colcon's core build task, completely independent of the backend hook execution.

• Implementation: During metadata augmentation, colcon parses the package's metadata and caches a backend-agnostic, unified mapping of all declared data files in PackageDescriptor.metadata. During build execution, the task runs standard PEP 660 editable installations and target-installs the wheel. In the post-install phase, the task reads the cached mapping, locates the target files in the install space, deletes the static copies, and replaces them with OS symlinks pointing back to the recorded source files.
• Pros: Unified, single-point implementation inside colcon-core. The backend remains a clean "black box" during hook execution.
• Cons: Shifting the responsibility of parsing proprietary backend TOML configurations (Hatchling, Flit, etc.) to the colcon core discovery system increases code complexity.

Some planning and milestones surely still need to be discussed, but this summary should bring everyone up to speed. I welcome any feedback, corrections, thoughts, or suggestions on the proposed next phases.