From 59983c178dc33a94e360f16805325837b8a996fe Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Mon, 17 Nov 2025 15:09:57 +0100 Subject: [PATCH 01/75] Add Nextflow Development Constitution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce comprehensive development constitution documenting core principles and practices for Nextflow development including modular architecture, test-driven quality assurance, dataflow programming model, licensing compliance, DCO requirements, semantic versioning, and Groovy code standards. The constitution codifies existing best practices from CLAUDE.md and CONTRIBUTING.md to provide clear governance and quality standards for the project. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Signed-off-by: Paolo Di Tommaso --- .specify/memory/constitution.md | 216 ++++++++++++++++++++++++++------ 1 file changed, 181 insertions(+), 35 deletions(-) diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index a4670ff469..5ecfd189e9 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,50 +1,196 @@ -# [PROJECT_NAME] Constitution - +# Nextflow Development Constitution + + ## Core Principles -### [PRINCIPLE_1_NAME] - -[PRINCIPLE_1_DESCRIPTION] - +### I. Modular Architecture + +Nextflow MUST maintain a clear separation between core functionality and extensions through its modular architecture: + +- **Core modules** (`modules/`) contain essential functionality: workflow engine (nextflow), shared utilities (nf-commons), language parsing (nf-lang), HTTP filesystem support (nf-httpfs), and lineage tracking (nf-lineage) +- **Plugin system** (`plugins/`) provides cloud provider integrations (AWS, Azure, GCP), execution platforms (Kubernetes), and specialized services (Seqera Platform, Wave container management) +- New features MUST be evaluated for placement: core features belong in `modules/`, specialized/cloud-specific features belong in `plugins/` +- Each module and plugin MUST be independently buildable and testable +- Plugin dependencies MUST be explicitly declared in `build.gradle` with semantic versioning + +**Rationale**: This architecture enables independent development of cloud provider features without core engine changes, supports third-party plugin development, and maintains a clean separation of concerns across a large multi-module codebase. + +### II. Test-Driven Quality Assurance (NON-NEGOTIABLE) + +Testing MUST be comprehensive and multi-layered before any code is merged: + +- **Unit tests** MUST use Spock Framework for all Groovy code, be independently executable, and achieve meaningful coverage (measured via JaCoCo) +- **Integration tests** (`tests/` directory) MUST validate end-to-end workflows using actual `.nf` scripts with expected outputs +- **Smoke tests** (`make smoke` or `NXF_SMOKE=1`) MUST be available to skip long-running and cloud-dependent tests during rapid development +- **Cloud validation tests** (`validation/` directory) MUST verify cloud provider integrations end-to-end before release +- **Documentation tests** (`docs/snippets/`) MUST ensure all documentation examples remain functional +- All tests MUST pass before commits, and `make test` MUST be run locally before pushing + +**Rationale**: Scientific workflows demand reliability and reproducibility. Multi-layered testing catches issues at appropriate levels: unit tests for logic, integration tests for workflow correctness, and validation tests for cloud provider compatibility. + +### III. Dataflow Programming Model + +Nextflow's core abstraction MUST adhere to the dataflow programming model: + +- Workflows are defined as dataflow graphs where data flows between processes +- Processes MUST be stateless, side-effect-free transformations that communicate via channels +- The DSL MUST prioritize expressiveness for concurrent and parallel pipeline definition +- Changes to the language parser (ANTLR grammars in `nf-lang`) MUST preserve backward compatibility with existing pipelines unless explicitly versioned (DSL1 vs DSL2) +- Concurrency primitives (GPars actors/dataflow) MUST be used correctly to maintain the dataflow semantics + +**Rationale**: The dataflow model is Nextflow's fundamental value proposition, enabling automatic parallelization and distribution. Preserving this model ensures existing scientific pipelines continue to work and users can reason about workflow behavior. + +### IV. Apache 2.0 License Compliance + +All source code MUST include Apache 2.0 license headers: + +- Every source file MUST begin with the Apache 2.0 license header +- All contributions MUST comply with Apache 2.0 terms +- Third-party dependencies MUST use compatible licenses +- License compliance MUST be verified during code review + +**Rationale**: Legal clarity protects both contributors and users. Consistent licensing enables academic and commercial use, which is critical for scientific software adoption. + +### V. Developer Certificate of Origin (DCO) Sign-off + +All commits MUST be signed with DCO certification: + +- Contributors MUST certify they have the right to submit the code by using `git commit -s` or `git commit --signoff` +- Every commit message MUST include a `Signed-off-by` line +- The DCO bot MUST verify sign-off before any PR can be merged +- Contributors MUST NOT bypass the DCO requirement -### [PRINCIPLE_2_NAME] - -[PRINCIPLE_2_DESCRIPTION] - +**Rationale**: DCO provides legal protection and clear chain of custody for contributions, which is essential for open-source projects with diverse contributors. -### [PRINCIPLE_3_NAME] - -[PRINCIPLE_3_DESCRIPTION] - +### VI. Semantic Versioning and Release Discipline -### [PRINCIPLE_4_NAME] - -[PRINCIPLE_4_DESCRIPTION] - +Version management MUST follow strict semantic versioning with calendar-based releases: -### [PRINCIPLE_5_NAME] - -[PRINCIPLE_5_DESCRIPTION] - +- **Project versions** use calendar-based scheme: `YY.MM.PATCH` where April (`.04.`) and October (`.10.`) are stable releases, all other months use `-edge` suffix (e.g., `25.09.0-edge`) +- **Plugin versions** MUST use semantic versioning (`MAJOR.MINOR.PATCH`) +- Version changes MUST be documented in `changelog.txt` files (both project root and per-plugin) +- Breaking changes MUST increment MAJOR version for plugins and be clearly documented +- Release process MUST follow the documented procedure in `CLAUDE.md` including: updating changelogs, version files, running `make releaseInfo`, using `[release]` tag in commit message -## [SECTION_2_NAME] - +**Rationale**: Predictable versioning enables users to understand compatibility and stability expectations. Calendar-based versioning for the main project makes release timing transparent, while semantic versioning for plugins enables clear communication of breaking changes. -[SECTION_2_CONTENT] - +### VII. Groovy Idioms and Code Standards -## [SECTION_3_NAME] - +Code MUST follow Groovy best practices and Nextflow conventions: -[SECTION_3_CONTENT] - +- Use Groovy idioms (closures, operator overloading, DSL builders) appropriately +- Follow existing code patterns and conventions from similar modules +- Leverage Groovy's dynamic capabilities judiciously without sacrificing type safety where beneficial +- Use Groovy's `@CompileStatic` where performance is critical or type safety is desired +- AST transformations (in `modules/nextflow`) MUST be well-documented due to their compile-time magic +- Code MUST be formatted consistently (consider CodeNarc configuration in `gradle/codenarc.groovy`) + +**Rationale**: Groovy enables powerful DSL capabilities that make Nextflow's language expressive, but requires discipline to maintain readability and debuggability. Consistency across the large codebase improves maintainability. + +## Development Workflow + +### Build and Development Process + +- **Build tool**: Gradle with wrapper (`./gradlew`) is the authoritative build system +- **Quick commands**: Makefile provides convenience targets (`make compile`, `make test`, `make assemble`, `make check`, `make clean`) +- **Development testing**: Use `./launch.sh run script.nf` for testing changes against real workflows without full installation +- **Local installation**: `make install` publishes to Maven local for integration testing +- **Dependency management**: All dependencies MUST be declared in `build.gradle` with explicit versions; use `make deps` to analyze dependency trees + +### Git Workflow + +- **Branch management**: Work on feature branches, never commit directly to `master` +- **Commit sign-off**: Always use `git commit -s` to add DCO sign-off +- **CI control tags**: Use special commit message tags to control CI behavior: + - `[ci skip]` - Skip CI tests entirely + - `[ci fast]` - Run only unit tests, skip integration tests + - `[e2e stage]` - Run end-to-end tests against Seqera platform staging environment + - `[e2e prod]` - Run end-to-end tests against production platform + - `[release]` - Trigger release automation +- **Pull requests**: Must pass all CI checks, require code review, and have DCO verification + +### Architecture Decision Records (ADRs) + +- Significant structural and technical decisions MUST be documented as ADRs in the `adr/` directory +- ADRs MUST follow the template format: date prefix + descriptive name (e.g., `20251114-module-system.md`) +- ADRs provide historical context for why architectural decisions were made +- When changing fundamental architecture, review existing ADRs and create new ones documenting the rationale + +## Quality Standards + +### Code Review Requirements + +- All changes MUST go through pull request review +- Reviewers MUST verify: + - Tests are included and passing + - Code follows Groovy idioms and project conventions + - License headers are present + - DCO sign-off is present + - Changes align with modular architecture principles + - Breaking changes are appropriately versioned and documented + +### Testing Gates + +- `make test` MUST pass before committing locally +- All CI tests MUST pass before merging +- Integration tests MUST be run for changes affecting workflow execution +- Cloud validation tests MUST be run before releases touching cloud provider plugins +- Smoke tests enable rapid iteration but MUST NOT replace full test execution + +### Performance and Compatibility + +- Target platform: Java 17 runtime compatibility (development uses Java 21 toolchain) +- Performance-critical paths SHOULD be profiled and optimized +- Memory usage SHOULD be monitored for large-scale workflows +- Backward compatibility MUST be maintained for existing DSL features unless a new DSL version is introduced ## Governance - -[GOVERNANCE_RULES] - +### Amendment Process + +This constitution supersedes all other development practices. Amendments require: + +1. **Proposal**: Submit amendment proposal via GitHub issue or pull request +2. **Discussion**: Community discussion period (minimum 1 week for major changes) +3. **Approval**: Approval from core maintainers +4. **Documentation**: Update this constitution with version bump following semantic versioning: + - **MAJOR**: Backward incompatible governance changes, principle removal/redefinition + - **MINOR**: New principle added or materially expanded guidance + - **PATCH**: Clarifications, wording improvements, typo fixes + +### Compliance and Review + +- All pull requests and code reviews MUST verify compliance with these principles +- Deviations from principles MUST be explicitly justified in PR description +- Complexity additions MUST be justified against the "simplicity first" principle +- Constitution compliance is enforced through code review and CI automation where possible + +### Ratification and Version History + +**Version**: 1.0.0 | **Ratified**: 2025-11-17 | **Last Amended**: 2025-11-17 -**Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE] - +This constitution was derived from the Nextflow project's documented practices in `CLAUDE.md`, `CONTRIBUTING.md`, and the project's existing architectural patterns. It codifies the development principles that have made Nextflow a successful scientific workflow management system. From fccfc7444cba0b8fc674b626c0be2d1ed502a4d3 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Mon, 24 Nov 2025 12:55:45 +0100 Subject: [PATCH 02/75] Module adr v1 Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 254 ++++++++++++++++++++++++++++++++++ 1 file changed, 254 insertions(+) create mode 100644 adr/20251114-module-system.md diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md new file mode 100644 index 0000000000..2b69b3e803 --- /dev/null +++ b/adr/20251114-module-system.md @@ -0,0 +1,254 @@ +# Module System for Nextflow + +- Authors: Paolo Di Tommaso +- Status: draft +- Date: 2025-11-14 +- Tags: modules, dsl, registry, versioning, architecture + +## Context and Problem Statement + +Nextflow supports local script inclusion via `include` directive but lacks standardized mechanisms for package management, versioning, and distribution of reusable process definitions. This limits code reuse and reproducibility across the ecosystem. + +## Decision + +Implement a module system with four core capabilities: + +1. **Remote module inclusion** via registry +2. **Semantic versioning** with dependency resolution +3. **Unified Nextflow Registry** (rebrand existing plugin registry) +4. **First-class CLI support** (pull, push, search, run) + +## Core Capabilities + +### 1. Remote Module Inclusion + +**DSL Syntax**: +```groovy +// Import from registry (version resolved from meta.yaml) +include { BWA_ALIGN } from module 'bwa/align' + +// Existing file-based includes remain supported +include { MY_PROCESS } from './modules/my-process.nf' +``` + +**Version Resolution**: Module versions and dependencies declared in the **module's own meta.yaml**, not in include statements or separate lock files. + +**Resolution**: Modules resolved at workflow parse time (after plugin resolution at startup). + +**Caching**: Downloaded modules cached in `$NXF_HOME/modules/cache/` organized by name/version. + +### 2. Semantic Versioning + +**Version Format**: MAJOR.MINOR.PATCH +- **MAJOR**: Breaking changes to process signatures, inputs, or outputs +- **MINOR**: New processes, backward-compatible enhancements +- **PATCH**: Bug fixes, documentation updates + +**Version Declaration**: Module versions and dependency constraints declared in **meta.yaml**: +```yaml +name: bwa/align +version: 1.2.4 # This module's version + +dependencies: + "samtools/view": "^1.0.0" # Semantic version constraint + "samtools/sort": "~2.1.0" # Tilde constraint + "picard/mark": "3.0.1" # Exact version +``` + +**Version Constraints**: +- `1.2.3`: Exact version +- `^1.2.3`: Compatible with >=1.2.3 <2.0.0 (caret - allow minor/patch) +- `~1.2.3`: Compatible with >=1.2.3 <1.3.0 (tilde - allow patch only) + +**Dependency Resolution**: When a module is imported, resolve its dependencies recursively using constraints from each module's meta.yaml. Use minimal version selection algorithm (choose minimum version satisfying all constraints). + +### 3. Unified Nextflow Registry + +**Architecture Decision**: Extend existing plugin registry at `registry.nextflow.io` to host both plugins and modules. + +**Current Plugin API** (reference: https://registry.nextflow.io/openapi/): +``` +GET /api/v1/plugins # List/search plugins +GET /api/v1/plugins/{pluginId} # Get plugin + all releases +GET /api/v1/plugins/{pluginId}/{version} # Get specific release +GET /api/v1/plugins/{pluginId}/{version}/download/{fileName} # Download artifact +POST /api/v1/plugins/release # Create draft release +POST /api/v1/plugins/release/{releaseId}/upload # Upload artifact +``` + +**Proposed Module API Extension** (same pattern): +``` +GET /api/v1/modules # List/search modules +GET /api/v1/modules/{moduleId} # Get module + all releases +GET /api/v1/modules/{moduleId}/{version} # Get specific release +GET /api/v1/modules/{moduleId}/{version}/download/{fileName} # Download source archive +POST /api/v1/modules/release # Create draft release +POST /api/v1/modules/release/{releaseId}/upload # Upload module archive +``` + +**Internal API** (already exists for modules - currently semantic search): +``` +GET /internal/modules # Natural language search +GET /internal/modules/{name} # Retrieve module metadata +``` + +**Registry URL**: `registry.nextflow.io` + +**Artifact Types**: +- **Plugins**: JAR files with JSON metadata, resolved at startup +- **Modules**: Source archives (.nf + meta.yaml), resolved at parse time + +**Benefits**: +- Reuses existing infrastructure (HTTP service, S3 storage, authentication) +- Consistent API patterns for both artifact types +- Operational simplicity (one service vs. two) +- Internal module API already partially implemented + +### 4. First-Class CLI Support + +**Commands**: +```bash +nextflow module search # Search registry +nextflow module pull # Download to cache +nextflow module push # Publish to registry (requires auth) +nextflow module run [args] # Execute module directly +``` + +**Key Features**: +- `search`: Find modules by name, description, tags +- `pull`: Download module and transitive dependencies +- `push`: Validate meta.yaml, authenticate, upload to registry +- `run`: Execute module with auto-generated CLI flags from meta.yaml + +## Module Structure + +**Directory Layout**: +``` +my-module/ +├── meta.yaml # Module manifest (metadata, dependencies, I/O specs) +├── main.nf # Process definitions +├── tests/ # Optional test workflows +└── README.md # Optional documentation +``` + +**Module Manifest** (`meta.yaml`): +```yaml +name: bwa/align +version: 1.2.4 # This module's version +description: Align reads using BWA-MEM +author: nf-core community +license: MIT + +requires: + nextflow: ">=24.04.0" + plugins: + - nf-amazon@2.0.0 + +processes: + - name: BWA_ALIGN + inputs: + - name: reads + type: path + format: fastq + ontology: edam:format_1930 + outputs: + - name: bam + type: path + format: bam + ontology: edam:format_2572 + +dependencies: # Module dependencies with version constraints + "samtools/view": "^1.0.0" + "samtools/sort": "~2.1.0" +``` + +## Implementation Strategy + +**Phase 1**: Module manifest schema, local module loading, validation tools + +**Phase 2**: Extend plugin registry for modules, implement caching, add `pull` and `search` commands + +**Phase 3**: Extend DSL parser for `from module` syntax, implement dependency resolution from meta.yaml + +**Phase 4**: Implement `push` command with authentication and `run` command + +**Phase 5**: Advanced features (search UI, language server integration, ontology validation) + +## Technical Details + +**Dependency Resolution Flow**: +1. Parse `include` statements → extract module names (no version in include) +2. Check local cache for latest or fetch module meta.yaml from registry +3. Read module's meta.yaml → get version and dependencies +4. Recursively resolve dependencies using version constraints in each meta.yaml +5. Download missing modules (minimal version selection) +6. Parse module scripts → make processes available + +**Security**: +- SHA-256 checksum verification for all downloads +- Authentication required for publishing +- Support for private registries + +**Integration with Plugin System**: +- Modules can declare plugin dependencies in meta.yaml +- Both plugins and modules query same registry +- Single authentication system +- Separate cache locations: `$NXF_HOME/plugins/` vs `$NXF_HOME/modules/` + +## Comparison: Plugins vs. Modules + +| Aspect | Plugins | Modules | +|--------|---------|---------| +| Purpose | Extend runtime | Reusable processes | +| Format | JAR files | Source code (.nf) | +| Resolution | Startup | Parse time | +| Metadata | JSON spec | YAML manifest | +| Registry Path | `/v1/artifacts/plugins/` | `/v1/artifacts/modules/` | + +## Rationale + +**Why unified registry?** +- Reuses battle-tested infrastructure (HTTP API, S3, auth) +- Single discovery experience for ecosystem +- Lower operational overhead +- Type-specific handling maintains separation of concerns + +**Why versions in meta.yaml instead of lock files?** +- Each module is self-contained with its own dependencies +- Simpler model: no separate lock file to manage +- Module author controls exact versions of dependencies +- Reproducibility guaranteed by module version itself + +**Why parse-time resolution?** +- Modules are source code, not compiled artifacts +- Allows inspection/modification for reproducibility +- Enables dependency analysis before execution + +**Why semantic versioning?** +- Clear compatibility guarantees +- Enables automated dependency resolution +- Industry standard (npm, cargo, Go modules) + +## Consequences + +**Positive**: +- Enables ecosystem-wide code reuse +- Reproducible workflows (versions pinned in module meta.yaml) +- Centralized discovery and distribution +- Minimal operational overhead (single registry) +- No separate lock file needed (versions self-contained in modules) + +**Negative**: +- Registry becomes critical infrastructure (requires HA setup) +- Type-specific handling adds registry complexity +- Parse-time resolution adds latency to workflow startup + +**Neutral**: +- Modules and plugins conceptually distinct but share infrastructure +- Different resolution timing supported by same API + +## Links + +- Related: [Plugin Spec ADR](20250922-plugin-spec.md) +- Inspired by: [Go Modules](https://go.dev/ref/mod), [npm](https://docs.npmjs.com), [Cargo](https://doc.rust-lang.org/cargo/) +- Related: [nf-core modules](https://nf-co.re/modules) From 20020a73ea843ecc5e1ddd93437a615353f17d94 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Tue, 25 Nov 2025 11:05:58 +0100 Subject: [PATCH 03/75] Module adr v2 Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 189 ++++++++++++++++++++++++---------- 1 file changed, 133 insertions(+), 56 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 2b69b3e803..ef16942012 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -24,43 +24,81 @@ Implement a module system with four core capabilities: **DSL Syntax**: ```groovy -// Import from registry (version resolved from meta.yaml) -include { BWA_ALIGN } from module 'bwa/align' +// Import from registry (scoped module name, detected by @scope prefix) +include { BWA_ALIGN } from '@nf-core/bwa-align' // Existing file-based includes remain supported include { MY_PROCESS } from './modules/my-process.nf' ``` -**Version Resolution**: Module versions and dependencies declared in the **module's own meta.yaml**, not in include statements or separate lock files. +**Module Naming**: NPM-style scoped packages `@scope/name` (e.g., `@nf-core/salmon`, `@myorg/custom`). Unscoped names supported for legacy compatibility. No nested paths allowed - each module must have a `main.nf` as the entry point. -**Resolution**: Modules resolved at workflow parse time (after plugin resolution at startup). +**Version Resolution**: Module versions pinned in `nextflow.config`. If not specified, uses latest available locally in `modules/` directory, or falls back to latest from registry. -**Caching**: Downloaded modules cached in `$NXF_HOME/modules/cache/` organized by name/version. +**Resolution Order**: +1. Check `nextflow.config` for pinned version +2. Check local `modules/@scope/name/` for any cached version +3. Query registry for latest version if not found +4. Warn if transitive dependencies are not pinned -### 2. Semantic Versioning +**Resolution Timing**: Modules resolved at workflow parse time (after plugin resolution at startup). + +**Local Storage**: Downloaded modules stored in `modules/@scope/name@version/` directory in project root (not global cache). Each module must contain a `main.nf` file as the required entry point. + +### 2. Semantic Versioning and Configuration **Version Format**: MAJOR.MINOR.PATCH - **MAJOR**: Breaking changes to process signatures, inputs, or outputs - **MINOR**: New processes, backward-compatible enhancements - **PATCH**: Bug fixes, documentation updates -**Version Declaration**: Module versions and dependency constraints declared in **meta.yaml**: +**Workflow Configuration** (`nextflow.config`): +```groovy +// Module versions (exact versions only, no ranges) +modules { + '@nf-core/salmon' = '1.1.0' // Simple syntax + '@nf-core/bwa-align' = [ // Extended syntax (with checksum) + version: '1.2.0', + checksum: 'sha256-abc123...' + ] +} + +// Registry configuration (separate block) +registry { + url = 'https://registry.nextflow.io' // Default registry + + scopes { + '@myorg' = 'https://npm.myorg.com' // Custom registry for @myorg scope + '@private' = 'https://private.com' + } + + auth { + 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' + 'npm.myorg.com' = '${MYORG_TOKEN}' + } +} +``` + +**Module Manifest** (`meta.yaml`): ```yaml -name: bwa/align +name: "@nf-core/bwa-align" version: 1.2.4 # This module's version -dependencies: - "samtools/view": "^1.0.0" # Semantic version constraint - "samtools/sort": "~2.1.0" # Tilde constraint - "picard/mark": "3.0.1" # Exact version +dependencies: # Transitive dependencies (version constraints) + "@nf-core/samtools-view": "^1.0.0" + "@nf-core/samtools-sort": "~2.1.0" ``` -**Version Constraints**: +**Version Constraints** (in module's meta.yaml only): - `1.2.3`: Exact version - `^1.2.3`: Compatible with >=1.2.3 <2.0.0 (caret - allow minor/patch) - `~1.2.3`: Compatible with >=1.2.3 <1.3.0 (tilde - allow patch only) -**Dependency Resolution**: When a module is imported, resolve its dependencies recursively using constraints from each module's meta.yaml. Use minimal version selection algorithm (choose minimum version satisfying all constraints). +**Dependency Resolution**: +- Workflow's `nextflow.config` specifies exact versions for direct dependencies +- Transitive dependencies resolved from each module's `meta.yaml` using version constraints +- Warn if transitive dependencies are not pinned in workflow config +- Use `nextflow module freeze` to pin all transitive dependencies with checksums ### 3. Unified Nextflow Registry @@ -78,18 +116,12 @@ POST /api/v1/plugins/release/{releaseId}/upload # Upload artifact **Proposed Module API Extension** (same pattern): ``` -GET /api/v1/modules # List/search modules -GET /api/v1/modules/{moduleId} # Get module + all releases -GET /api/v1/modules/{moduleId}/{version} # Get specific release -GET /api/v1/modules/{moduleId}/{version}/download/{fileName} # Download source archive -POST /api/v1/modules/release # Create draft release -POST /api/v1/modules/release/{releaseId}/upload # Upload module archive -``` - -**Internal API** (already exists for modules - currently semantic search): -``` -GET /internal/modules # Natural language search -GET /internal/modules/{name} # Retrieve module metadata +GET /api/v1/modules # List/search modules +GET /api/v1/modules/{scope}/{name} # Get module + all releases +GET /api/v1/modules/{scope}/{name}/{version} # Get specific release +GET /api/v1/modules/{scope}/{name}/{version}/download # Download source archive +POST /api/v1/modules/release # Create draft release +POST /api/v1/modules/release/{releaseId}/upload # Upload module archive ``` **Registry URL**: `registry.nextflow.io` @@ -108,17 +140,23 @@ GET /internal/modules/{name} # Retrieve module metadata **Commands**: ```bash -nextflow module search # Search registry -nextflow module pull # Download to cache -nextflow module push # Publish to registry (requires auth) -nextflow module run [args] # Execute module directly +nextflow module search # Search registry +nextflow module install [@scope/name] # Install all from config, or specific module +nextflow module update [@scope/name] # Update specific module or all +nextflow module freeze # Pin all versions + checksums to config +nextflow module list # Show installed vs configured +nextflow module remove @scope/name # Remove from config + local cache +nextflow module publish @scope/name # Publish to registry (requires api key) ``` **Key Features**: - `search`: Find modules by name, description, tags -- `pull`: Download module and transitive dependencies -- `push`: Validate meta.yaml, authenticate, upload to registry -- `run`: Execute module with auto-generated CLI flags from meta.yaml +- `install`: Download modules to local `modules/` directory, auto-download on `nextflow run` if missing +- `update`: Re-query registry for latest version, update config +- `freeze`: Scan `modules/` directory, write all versions + checksums to config (extended syntax) +- `list`: Compare config versions vs installed versions +- `publish`: Validate meta.yaml, authenticate, upload to registry +- **Warnings**: Unpinned transitive dependencies generate warnings during resolution ## Module Structure @@ -126,14 +164,14 @@ nextflow module run [args] # Execute module directly ``` my-module/ ├── meta.yaml # Module manifest (metadata, dependencies, I/O specs) -├── main.nf # Process definitions +├── main.nf # Required: entry point for module ├── tests/ # Optional test workflows └── README.md # Optional documentation ``` **Module Manifest** (`meta.yaml`): ```yaml -name: bwa/align +name: "@nf-core/bwa-align" version: 1.2.4 # This module's version description: Align reads using BWA-MEM author: nf-core community @@ -157,9 +195,28 @@ processes: format: bam ontology: edam:format_2572 -dependencies: # Module dependencies with version constraints - "samtools/view": "^1.0.0" - "samtools/sort": "~2.1.0" +dependencies: # Transitive dependencies with version constraints + "@nf-core/samtools-view": "^1.0.0" + "@nf-core/samtools-sort": "~2.1.0" +``` + +**Local Storage Structure**: +``` +project-root/ +├── nextflow.config +├── main.nf +└── modules/ # Local module cache + ├── @nf-core/ + │ ├── bwa-align@1.2.4/ + │ │ ├── meta.yaml + │ │ └── main.nf # Required entry point + │ └── samtools-view@1.0.5/ + │ ├── meta.yaml + │ └── main.nf # Required entry point + └── @myorg/ + └── custom-process@2.0.0/ + ├── meta.yaml + └── main.nf # Required entry point ``` ## Implementation Strategy @@ -177,12 +234,16 @@ dependencies: # Module dependencies with version constraints ## Technical Details **Dependency Resolution Flow**: -1. Parse `include` statements → extract module names (no version in include) -2. Check local cache for latest or fetch module meta.yaml from registry -3. Read module's meta.yaml → get version and dependencies -4. Recursively resolve dependencies using version constraints in each meta.yaml -5. Download missing modules (minimal version selection) -6. Parse module scripts → make processes available +1. Parse `include` statements → extract module names (e.g., `@nf-core/bwa-align`) +2. For each module: + a. Check `nextflow.config` modules section for pinned version + b. If not pinned: check local `modules/@scope/name/` for any cached version (use latest local) + c. If not found locally: query registry for latest version + d. Warn if module not pinned in config (especially transitive dependencies) +3. Download missing modules to `modules/@scope/name@version/` +4. Read module's `meta.yaml` → resolve transitive dependencies recursively +5. Verify checksums (if present in config) +6. Parse module's `main.nf` file → make processes available **Security**: - SHA-256 checksum verification for all downloads @@ -193,7 +254,7 @@ dependencies: # Module dependencies with version constraints - Modules can declare plugin dependencies in meta.yaml - Both plugins and modules query same registry - Single authentication system -- Separate cache locations: `$NXF_HOME/plugins/` vs `$NXF_HOME/modules/` +- Separate cache locations: `$NXF_HOME/plugins/` (global) vs `modules/` (per-project) ## Comparison: Plugins vs. Modules @@ -203,7 +264,10 @@ dependencies: # Module dependencies with version constraints | Format | JAR files | Source code (.nf) | | Resolution | Startup | Parse time | | Metadata | JSON spec | YAML manifest | -| Registry Path | `/v1/artifacts/plugins/` | `/v1/artifacts/modules/` | +| Naming | `nf-amazon` | `@nf-core/salmon` | +| Cache Location | `$NXF_HOME/plugins/` | `modules/@scope/name@version/` | +| Version Config | `plugins {}` in config | `modules {}` in config | +| Registry Path | `/api/v1/plugins/` | `/api/v1/modules/@scope/name` | ## Rationale @@ -213,35 +277,48 @@ dependencies: # Module dependencies with version constraints - Lower operational overhead - Type-specific handling maintains separation of concerns -**Why versions in meta.yaml instead of lock files?** -- Each module is self-contained with its own dependencies -- Simpler model: no separate lock file to manage -- Module author controls exact versions of dependencies -- Reproducibility guaranteed by module version itself +**Why versions in nextflow.config instead of separate lock file?** +- Single source of truth for workflow dependencies +- Simple: exact versions in config, no separate lock file to manage +- Transitive dependencies resolved from module's meta.yaml with version constraints +- Use `nextflow module freeze` to pin all versions + checksums when needed +- Reproducibility via explicit version pinning in config **Why parse-time resolution?** - Modules are source code, not compiled artifacts - Allows inspection/modification for reproducibility - Enables dependency analysis before execution +**Why NPM-style scoped packages?** +- Organization namespacing prevents name collisions (`@nf-core/salmon` vs `@myorg/salmon`) +- Clear ownership and provenance of modules +- Supports private registries per scope +- Industry-standard pattern (NPM, Terraform, others) +- Enables ecosystem organization by maintainer/organization + **Why semantic versioning?** - Clear compatibility guarantees -- Enables automated dependency resolution +- Enables automated dependency resolution for transitive dependencies - Industry standard (npm, cargo, Go modules) ## Consequences **Positive**: - Enables ecosystem-wide code reuse -- Reproducible workflows (versions pinned in module meta.yaml) -- Centralized discovery and distribution -- Minimal operational overhead (single registry) -- No separate lock file needed (versions self-contained in modules) +- Reproducible workflows (exact versions pinned in nextflow.config) +- Centralized discovery and distribution via unified registry +- Minimal operational overhead (single registry for both plugins and modules) +- NPM-style scoping enables organization namespaces and private registries +- Local `modules/` directory provides project isolation +- Simple config model: no separate lock file unless using `freeze` +- Simple module structure: each module has single `main.nf` entry point **Negative**: - Registry becomes critical infrastructure (requires HA setup) - Type-specific handling adds registry complexity - Parse-time resolution adds latency to workflow startup +- Local `modules/` directory duplicates storage across projects (unlike global cache) +- Warnings for unpinned transitive dependencies may be noisy initially **Neutral**: - Modules and plugins conceptually distinct but share infrastructure From 226d4eec51778d374e6e2058cd9880ffe0e74106 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 11 Dec 2025 16:15:45 +0100 Subject: [PATCH 04/75] ADR: Nextflow remote modules system v2.1 Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 68 ++++++++++++++++++++--------------- 1 file changed, 40 insertions(+), 28 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index ef16942012..e85802cf33 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -2,13 +2,16 @@ - Authors: Paolo Di Tommaso - Status: draft -- Date: 2025-11-14 +- Date: 2025-12-11 - Tags: modules, dsl, registry, versioning, architecture +- Version: 2.1 ## Context and Problem Statement Nextflow supports local script inclusion via `include` directive but lacks standardized mechanisms for package management, versioning, and distribution of reusable process definitions. This limits code reuse and reproducibility across the ecosystem. +Discussion/request goes back to at least 2019, see GitHub issues [#1376](https://github.com/nextflow-io/nextflow/issues/1376), [#1463](https://github.com/nextflow-io/nextflow/issues/1463) and [#4122](https://github.com/nextflow-io/nextflow/issues/4112). + ## Decision Implement a module system with four core capabilities: @@ -31,9 +34,9 @@ include { BWA_ALIGN } from '@nf-core/bwa-align' include { MY_PROCESS } from './modules/my-process.nf' ``` -**Module Naming**: NPM-style scoped packages `@scope/name` (e.g., `@nf-core/salmon`, `@myorg/custom`). Unscoped names supported for legacy compatibility. No nested paths allowed - each module must have a `main.nf` as the entry point. +**Module Naming**: NPM-style scoped packages `@scope/name` (e.g., `@nf-core/salmon`, `@myorg/custom`). Unscoped names (eg. local paths) supported for legacy compatibility. No nested paths with the module are allowed - each module must have a `main.nf` as the entry point. -**Version Resolution**: Module versions pinned in `nextflow.config`. If not specified, uses latest available locally in `modules/` directory, or falls back to latest from registry. +**Version Resolution**: Module versions pinned in `nextflow.config`. If not specified, use the latest available locally in `modules/` directory, or downloaded and cached in the `modules/` directory. **Resolution Order**: 1. Check `nextflow.config` for pinned version @@ -43,7 +46,7 @@ include { MY_PROCESS } from './modules/my-process.nf' **Resolution Timing**: Modules resolved at workflow parse time (after plugin resolution at startup). -**Local Storage**: Downloaded modules stored in `modules/@scope/name@version/` directory in project root (not global cache). Each module must contain a `main.nf` file as the required entry point. +**Local Storage**: Downloaded modules stored in `modules/@scope/name@version/` directory in project root (not global cache). Each module must contain a `main.nf` file as the required entry point. It is intended that module source code will be committed to the pipeline git repository. ### 2. Semantic Versioning and Configuration @@ -67,10 +70,9 @@ modules { registry { url = 'https://registry.nextflow.io' // Default registry - scopes { - '@myorg' = 'https://npm.myorg.com' // Custom registry for @myorg scope - '@private' = 'https://private.com' - } + // allow the use of multiple registry url for resolving module + // across custom registries, e.g. + // url = [ 'https://custom.registry.com', 'https://registry.nextflow.io' ] auth { 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' @@ -91,8 +93,30 @@ dependencies: # Transitive dependencies (version constraints **Version Constraints** (in module's meta.yaml only): - `1.2.3`: Exact version -- `^1.2.3`: Compatible with >=1.2.3 <2.0.0 (caret - allow minor/patch) -- `~1.2.3`: Compatible with >=1.2.3 <1.3.0 (tilde - allow patch only) +- `>=1.2.3`: Greater or equal +- `>=1.2.3, <2.0.0`: Range (comma-separated) - equivalent to npm's `^1.2.3` +- `>=1.2.3, <1.3.0`: Range (comma-separated) - equivalent to npm's `~1.2.3` + +**Version Notation Consistency**: + +Modules use the same version constraint syntax already supported by both `nextflowVersion` and plugins: + +| Notation | Meaning | nextflowVersion | Plugins | Modules | +| :---- | :---- | :---- | :---- | :---- | +| 1.2.3 | Exact version | ✓ | ✓ | ✓ | +| >=1.2.3 | Greater or equal | ✓ | ✓ | ✓ | +| <=1.2.3 | Less or equal | ✓ | ✓ | ✓ | +| >1.2.3 | Greater than | ✓ | ✓ | ✓ | +| <1.2.3 | Less than | ✓ | ✓ | ✓ | +| >=1.2, <2.0 | Range (comma) | ✓ | ✓ | ✓ | +| !=1.2.3 | Not equal | ✓ | - | - | +| 1.2+ | >=1.2.x <2.0 | ✓ | - | - | +| 1.2.+ | >=1.2.0 <1.3.0 | ✓ | - | - | +| ~1.2.3 | >=1.2.3 <1.3.0 | - | ✓ | - | + +Using comparison operators (`>=`, `<`) with comma-separated ranges provides the same expressive power as +npm-style `^` and `~` notation while maintaining consistency with existing Nextflow version constraint syntax. +This avoids introducing new notation that would require additional parser support. **Dependency Resolution**: - Workflow's `nextflow.config` specifies exact versions for direct dependencies @@ -161,15 +185,16 @@ nextflow module publish @scope/name # Publish to registry (requires api ## Module Structure **Directory Layout**: +Everything within the module directory should be uploaded. Module bundle should not exceed 1MB (uncompressed). Typically this is expected to look something like this: ``` my-module/ -├── meta.yaml # Module manifest (metadata, dependencies, I/O specs) ├── main.nf # Required: entry point for module -├── tests/ # Optional test workflows -└── README.md # Optional documentation +├── meta.yaml # Optional: Module spec (metadata, dependencies, I/O specs) +├── README.md # Required: Module description +└── tests/ # Optional test workflows ``` -**Module Manifest** (`meta.yaml`): +**Module Spec extension** (`meta.yaml`): ```yaml name: "@nf-core/bwa-align" version: 1.2.4 # This module's version @@ -182,20 +207,7 @@ requires: plugins: - nf-amazon@2.0.0 -processes: - - name: BWA_ALIGN - inputs: - - name: reads - type: path - format: fastq - ontology: edam:format_1930 - outputs: - - name: bam - type: path - format: bam - ontology: edam:format_2572 - -dependencies: # Transitive dependencies with version constraints +dependencies: "@nf-core/samtools-view": "^1.0.0" "@nf-core/samtools-sort": "~2.1.0" ``` From 8e681e965cb7d844f8711d2635592aaae9f0d46f Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Mon, 15 Dec 2025 12:31:50 +0100 Subject: [PATCH 05/75] Add detailed CLI command descriptions to module system ADR [ci skip] - Add comprehensive documentation for all module CLI commands - Add `nextflow module run` command for standalone module execution - Remove `module update` command to simplify the design - Use single-dash prefix for Nextflow options, double-dash for module inputs - Remove @ prefix from scope in CLI commands (keep only in DSL syntax) Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 226 ++++++++++++++++++++++++++++++++-- 1 file changed, 214 insertions(+), 12 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index e85802cf33..34fce4af1d 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -164,23 +164,225 @@ POST /api/v1/modules/release/{releaseId}/upload # Upload module arc **Commands**: ```bash +nextflow module run scope/name # Run a module directly without a wrapper script nextflow module search # Search registry -nextflow module install [@scope/name] # Install all from config, or specific module -nextflow module update [@scope/name] # Update specific module or all +nextflow module install [scope/name] # Install all from config, or specific module nextflow module freeze # Pin all versions + checksums to config nextflow module list # Show installed vs configured -nextflow module remove @scope/name # Remove from config + local cache -nextflow module publish @scope/name # Publish to registry (requires api key) +nextflow module remove scope/name # Remove from config + local cache +nextflow module publish scope/name # Publish to registry (requires api key) ``` -**Key Features**: -- `search`: Find modules by name, description, tags -- `install`: Download modules to local `modules/` directory, auto-download on `nextflow run` if missing -- `update`: Re-query registry for latest version, update config -- `freeze`: Scan `modules/` directory, write all versions + checksums to config (extended syntax) -- `list`: Compare config versions vs installed versions -- `publish`: Validate meta.yaml, authenticate, upload to registry -- **Warnings**: Unpinned transitive dependencies generate warnings during resolution +#### `nextflow module run scope/name` + +Run a module directly without requiring a wrapper workflow script. This command enables standalone execution of any module by automatically mapping command-line arguments to the module's process inputs. If the module is not available locally, it is automatically installed before execution. + +**Arguments**: +- `scope/name`: Module identifier to run (required) + +**Options**: +- `-version `: Run a specific version (default: latest or configured version) +- `-- `: Map value to the corresponding module process input channel +- All standard `nextflow run` options (e.g., `-profile`, `-work-dir`, `-resume`, etc.) + +**Behavior**: +1. Checks if module is installed locally; if not, downloads from registry +2. Parses the module's `main.nf` to identify the main process and its input declarations +3. Validates command-line arguments against the process input schema +4. Generates an implicit workflow that wires CLI arguments to process inputs +5. Executes the workflow using standard Nextflow runtime + +**Input Mapping**: +- Named arguments (`--reads`, `--reference`) are mapped to corresponding process inputs +- File paths are automatically converted to file channels +- Multiple values can be provided for inputs expecting collections +- Required inputs without defaults must be provided; optional inputs use declared defaults + +**Example**: +```bash +# Run BWA alignment module with input files +nextflow module run nf-core/bwa-align \ + --reads 'samples/*_{1,2}.fastq.gz' \ + --reference genome.fa + +# Run a specific version with Nextflow options +nextflow module run nf-core/fastqc -version 1.0.0 \ + --input 'data/*.fastq.gz' \ + -profile docker \ + -resume + +# Run with work directory and output specification +nextflow module run nf-core/salmon \ + --reads reads.fq \ + --index salmon_index \ + -work-dir /tmp/work \ + --outdir results/ +``` + +--- + +#### `nextflow module search ` + +Search the Nextflow registry for available modules matching the specified query. The search operates against module names, descriptions, tags, and author information. Results are displayed with module name, latest version, description, and download statistics. + +**Arguments**: +- ``: Search term (required) - matches against module metadata + +**Options**: +- `-limit `: Maximum number of results to return (default: 10) +- `-json`: Output results in JSON format for programmatic use + +**Example**: +```bash +nextflow module search bwa +nextflow module search "alignment" -limit 50 +``` + +--- + +#### `nextflow module install [scope/name]` + +Download and install modules to the local `modules/` directory. When called without arguments, installs all modules declared in `nextflow.config`. When a specific module is provided, installs that module and adds it to the configuration. + +**Arguments**: +- `[scope/name]`: Optional module identifier. If omitted, installs all modules from config + +**Options**: +- `-version `: Install a specific version (default: latest) +- `-force`: Re-download even if already installed locally + +**Behavior**: +1. Resolves the module version from `nextflow.config` or queries registry for latest +2. Downloads the module archive from the registry +3. Extracts to `modules/@scope/name@version/` directory +4. Recursively installs transitive dependencies declared in `meta.yaml` +5. Updates `nextflow.config` if installing a new module not already configured + +**Example**: +```bash +nextflow module install # Install all from config +nextflow module install nf-core/bwa-align # Install specific module (latest) +nextflow module install nf-core/salmon -version 1.2.0 +``` + +--- + +#### `nextflow module freeze` + +Lock all module versions by writing exact versions and SHA-256 checksums to `nextflow.config`. This ensures fully reproducible builds by capturing the precise state of all dependencies. + +**Options**: +- `-verify`: Verify existing checksums without updating + +**Behavior**: +1. Scans the `modules/` directory for all installed modules +2. Computes SHA-256 checksums for each module archive +3. Converts simple version syntax to extended syntax with checksums in `nextflow.config` +4. Includes transitive dependencies not explicitly declared + +**Output** (in `nextflow.config`): +```groovy +modules { + '@nf-core/bwa-align' = [ + version: '1.2.4', + checksum: 'sha256-a1b2c3d4e5f6...' + ] +} +``` + +**Example**: +```bash +nextflow module freeze # Pin all versions + checksums +nextflow module freeze -verify # Verify checksums match +``` + +--- + +#### `nextflow module list` + +Display the status of all modules, comparing what is configured in `nextflow.config` against what is actually installed in the `modules/` directory. + +**Options**: +- `-json`: Output in JSON format +- `-outdated`: Only show modules with available updates + +**Output columns**: +- Module name (`@scope/name`) +- Configured version (from `nextflow.config`) +- Installed version (from `modules/` directory) +- Latest available version (from registry) +- Status indicator (up-to-date, outdated, missing, not configured) + +**Example**: +```bash +nextflow module list +nextflow module list -outdated +``` + +--- + +#### `nextflow module remove scope/name` + +Remove a module from both the local `modules/` directory and the `nextflow.config` configuration. Also removes orphaned transitive dependencies that are no longer required by other modules. + +**Arguments**: +- `scope/name`: Module identifier to remove (required) + +**Options**: +- `-keep-config`: Remove local files but keep the entry in `nextflow.config` +- `-keep-files`: Remove from config but keep local files + +**Behavior**: +1. Removes the module directory from `modules/@scope/name@version/` +2. Removes the module entry from the `modules {}` block in `nextflow.config` +3. Identifies and optionally removes orphaned transitive dependencies +4. Warns if the module is still referenced in workflow files + +**Example**: +```bash +nextflow module remove nf-core/bwa-align +nextflow module remove myorg/custom -keep-files +``` + +--- + +#### `nextflow module publish scope/name` + +Publish a module to the Nextflow registry, making it available for others to install. Requires authentication via API key and appropriate permissions for the target scope. + +**Arguments**: +- `scope/name`: Module identifier to publish (required) + +**Options**: +- `-registry `: Target registry URL (default: `registry.nextflow.io`) +- `-tag `: Additional tags for discoverability +- `-dry-run`: Validate without publishing + +**Behavior**: +1. Validates `meta.yaml` schema and required fields (name, version, description) +2. Verifies `main.nf` exists and is valid Nextflow syntax +3. Checks that `README.md` documentation is present +4. Authenticates with registry using configured credentials +5. Creates a release draft and uploads the module archive +6. Publishes the release, making it available for installation + +**Requirements**: +- Valid `meta.yaml` with name, version, and description +- `main.nf` entry point file +- `README.md` documentation +- Authentication token configured in `registry.auth` or `NXF_REGISTRY_TOKEN` +- Write permission for the target scope + +**Example**: +```bash +nextflow module publish myorg/my-process +nextflow module publish myorg/my-process -dry-run +``` + +**General Notes**: +- All commands respect the `registry.url` configuration for custom registries +- Unpinned transitive dependencies generate warnings during resolution +- Modules are automatically downloaded on `nextflow run` if missing but configured ## Module Structure From 5df12cd4a64e79612e98b92fac1aa2caf827874e Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Tue, 6 Jan 2026 18:33:05 +0700 Subject: [PATCH 06/75] Simplify module spec: unify dependencies under requires MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove separate `dependencies` and `components` fields - Expand `requires` to include: - `nextflow`: version constraint (unchanged) - `plugins`: array with name@constraint syntax - `modules`: array of module dependencies - `workflows`: array of workflow dependencies - Unified version constraint syntax: `[scope/]name[@constraint]` - Mark `components` as deprecated (use requires.modules) - Update all examples in ADR and schema Signed-off-by: Paolo Di Tommaso 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 508 ++++++++++++++++++++++++++++++++- adr/module-spec-schema.json | 521 ++++++++++++++++++++++++++++++++++ 2 files changed, 1015 insertions(+), 14 deletions(-) create mode 100644 adr/module-spec-schema.json diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 34fce4af1d..31d1a59d77 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -83,19 +83,21 @@ registry { **Module Manifest** (`meta.yaml`): ```yaml -name: "@nf-core/bwa-align" +name: nf-core/bwa-align version: 1.2.4 # This module's version -dependencies: # Transitive dependencies (version constraints) - "@nf-core/samtools-view": "^1.0.0" - "@nf-core/samtools-sort": "~2.1.0" +requires: + nextflow: ">=24.04.0" + modules: # Required modules (version constraints) + - nf-core/samtools/view@>=1.0.0,<2.0.0 + - nf-core/samtools/sort@>=2.1.0,<2.2.0 ``` -**Version Constraints** (in module's meta.yaml only): -- `1.2.3`: Exact version -- `>=1.2.3`: Greater or equal -- `>=1.2.3, <2.0.0`: Range (comma-separated) - equivalent to npm's `^1.2.3` -- `>=1.2.3, <1.3.0`: Range (comma-separated) - equivalent to npm's `~1.2.3` +**Version Constraints** (unified `name@constraint` syntax): +- `name`: Any version (latest) +- `name@1.2.3`: Exact version +- `name@>=1.2.3`: Greater or equal +- `name@>=1.2.3,<2.0.0`: Range (comma-separated) **Version Notation Consistency**: @@ -398,7 +400,7 @@ my-module/ **Module Spec extension** (`meta.yaml`): ```yaml -name: "@nf-core/bwa-align" +name: nf-core/bwa-align version: 1.2.4 # This module's version description: Align reads using BWA-MEM author: nf-core community @@ -408,10 +410,9 @@ requires: nextflow: ">=24.04.0" plugins: - nf-amazon@2.0.0 - -dependencies: - "@nf-core/samtools-view": "^1.0.0" - "@nf-core/samtools-sort": "~2.1.0" + modules: + - nf-core/samtools/view@>=1.0.0,<2.0.0 + - nf-core/samtools/sort@>=2.1.0,<2.2.0 ``` **Local Storage Structure**: @@ -543,3 +544,482 @@ project-root/ - Related: [Plugin Spec ADR](20250922-plugin-spec.md) - Inspired by: [Go Modules](https://go.dev/ref/mod), [npm](https://docs.npmjs.com), [Cargo](https://doc.rust-lang.org/cargo/) - Related: [nf-core modules](https://nf-co.re/modules) + +--- + +## Appendix A: Module Metadata Schema Specification + +This appendix defines the JSON schema for module `meta.yaml` files. The schema maintains backward compatibility with existing nf-core module metadata patterns while supporting the new Nextflow module system features. + +**Schema File:** [module-spec-schema.json](module-spec-schema.json) +**Published URL:** `https://registry.nextflow.io/schemas/module-spec/v1.0.0` + +### Field Reference + +#### Core Fields (Existing nf-core Pattern) + +These fields are already widely adopted in the nf-core community and remain fully supported: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | Module identifier | +| `description` | string | Yes | Brief description of module functionality | +| `keywords` | array[string] | Recommended | Discovery and categorization keywords | +| `authors` | array[string] | Recommended | Original authors (GitHub handles) | +| `maintainers` | array[string] | Recommended | Current maintainers | +| `tools` | array[object] | Conditional | Software tools wrapped by the module | +| `input` | array/object | Recommended | Input channel specifications | +| `output` | object/array | Recommended | Output channel specifications | + +#### Extension Fields (Nextflow Module System) + +These fields extend the schema to support the new Nextflow module system: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `version` | string | Registry | Semantic version (MAJOR.MINOR.PATCH) | +| `license` | string | Registry | SPDX license identifier for module code | +| `requires` | object | Optional | All requirements: runtime, plugins, and dependencies | +| `requires.nextflow` | string | Optional | Nextflow version constraint | +| `requires.plugins` | array[string] | Optional | Required Nextflow plugins | +| `requires.modules` | array[string] | Optional | Required modules (processes) | +| `requires.workflows` | array[string] | Optional | Required workflows/subworkflows | + +### Detailed Field Specifications + +#### `name` + +The module name must be a fully qualified scoped identifier in `scope/name` format: + +```yaml +name: nf-core/fastqc +name: nf-core/bwa-mem +name: myorg/custom-aligner +``` + +**Naming Rules:** +- Format: `scope/name` (e.g., `nf-core/salmon`, `myorg/custom`) +- Scope: lowercase alphanumeric with hyphens (organization/owner identifier) +- Name: lowercase alphanumeric with underscores/hyphens (module identifier) +- Pattern: `^[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$` + +**Note:** The `@` prefix is used only in Nextflow DSL `include` statements (e.g., `include { FASTQC } from '@nf-core/fastqc'`) to distinguish registry modules from local file paths. The meta.yaml `name` field should not include the `@` prefix. + +#### `version` + +Semantic version following [SemVer 2.0.0](https://semver.org/): + +```yaml +version: "1.0.0" +version: "2.3.1" +version: "1.0.0-beta.1" +``` + +**Version Semantics:** +- **MAJOR:** Breaking changes to process signatures, inputs, or outputs +- **MINOR:** New processes, backward-compatible enhancements +- **PATCH:** Bug fixes, documentation updates + +**Requirement:** Mandatory for registry-published modules (scoped names in `scope/name` format). + +#### `requires` + +Specifies all requirements for the module: runtime environment, plugins, and dependencies. + +```yaml +requires: + nextflow: ">=24.04.0" + plugins: + - nf-amazon@2.0.0 + - nf-wave@>=1.5.0 + modules: + - nf-core/fastqc@>=1.0.0 + - nf-core/samtools/sort@>=2.1.0,<3.0.0 + - bwa/mem + workflows: + - nf-core/fastq-align-bwa@1.0.0 +``` + +**Unified Version Constraint Syntax:** + +All requirements (except `nextflow`) use a unified `name@constraint` format: + +| Format | Meaning | Example | +|--------|---------|---------| +| `name` | Any version (latest) | `bwa/mem` | +| `name@1.2.3` | Exact version | `nf-core/fastqc@1.0.0` | +| `name@>=1.2.3` | Greater or equal | `nf-core/fastqc@>=1.0.0` | +| `name@>=1.2.3,<2.0.0` | Range constraint | `nf-core/samtools/sort@>=2.1.0,<3.0.0` | + +**`requires.nextflow`** - Nextflow version constraint: +```yaml +requires: + nextflow: ">=24.04.0" # minimum version + nextflow: ">=24.04.0,<25.0.0" # version range +``` + +**`requires.plugins`** - Required Nextflow plugins: +```yaml +requires: + plugins: + - nf-amazon@2.0.0 # exact version + - nf-wave@>=1.5.0 # minimum version + - nf-azure # any version +``` + +**`requires.modules`** - Required modules (processes): +```yaml +requires: + modules: + - nf-core/fastqc@>=1.0.0 # registry module with constraint + - nf-core/samtools/sort@>=2.1.0 # nested module path + - bwa/mem # local or registry (no constraint) +``` + +**`requires.workflows`** - Required workflows/subworkflows: +```yaml +requires: + workflows: + - nf-core/fastq-align-bwa@1.0.0 # registry workflow + - my-local-workflow # local workflow +``` + +**Resolution:** +1. The resolver looks up dependencies locally first, then in configured registries +2. Version constraints are resolved transitively +3. Pinned versions are recorded in `nextflow.config` for reproducibility + +#### `tools` + +Documents the software tools wrapped by the module: + +```yaml +tools: + - bwa: + description: | + BWA is a software package for mapping DNA sequences + against a large reference genome. + homepage: http://bio-bwa.sourceforge.net/ + documentation: https://bio-bwa.sourceforge.net/bwa.shtml + doi: 10.1093/bioinformatics/btp324 + arxiv: arXiv:1303.3997 + licence: ["GPL-3.0-or-later"] + identifier: biotools:bwa +``` + +**Tool Properties:** + +| Property | Required | Description | +|----------|----------|-------------| +| `description` | Yes | Tool description | +| `homepage` | One of these | Tool homepage URL | +| `documentation` | One of these | Documentation URL | +| `tool_dev_url` | One of these | Development/source URL | +| `doi` | One of these | Publication DOI | +| `arxiv` | No | arXiv identifier | +| `licence` | Recommended | SPDX license(s) | +| `identifier` | Recommended | bio.tools identifier | +| `manual` | No | User manual URL | + +#### `input` and `output` + +The schema supports both nf-core patterns to ensure backward compatibility: + +**Module Pattern (Tuple-based):** +```yaml +input: + - - meta: + type: map + description: Sample metadata + - reads: + type: file + description: Input FastQ files + ontologies: + - edam: "http://edamontology.org/format_1930" + - - index: + type: directory + description: Reference index + +output: + bam: + - - meta: + type: map + description: Sample metadata + - "*.bam": + type: file + description: Aligned BAM file + pattern: "*.bam" + versions: + - versions.yml: + type: file + description: Software versions +``` + +**Subworkflow Pattern (Simplified):** +```yaml +input: + - ch_reads: + description: | + Input FastQ files + Structure: [ val(meta), [ path(reads) ] ] + - ch_index: + description: BWA index files + type: file + +output: + - bam: + description: Aligned BAM files + - versions: + description: Software versions +``` + +**Channel Element Properties:** + +| Property | Type | Description | +|----------|------|-------------| +| `type` | string | Data type: `map`, `file`, `directory`, `string`, `integer`, `float`, `boolean`, `list`, `val` | +| `description` | string | Human-readable description | +| `pattern` | string | File glob pattern or value pattern | +| `optional` | boolean | Whether input is optional (default: false) | +| `default` | any | Default value if not provided | +| `enum` | array | List of allowed values | +| `ontologies` | array | EDAM or other ontology annotations | + +### Migration Guide + +#### From nf-core Module to Registry Module + +**Before (nf-core local):** +```yaml +name: bwa_mem +description: Align reads using BWA-MEM +keywords: + - alignment + - bwa +tools: + - bwa: + description: BWA software + homepage: http://bio-bwa.sourceforge.net/ + licence: ["GPL-3.0-or-later"] + identifier: biotools:bwa +authors: + - "@drpatelh" +maintainers: + - "@drpatelh" +input: + # ... existing input spec +output: + # ... existing output spec +``` + +**After (Registry-ready):** +```yaml +name: nf-core/bwa-mem # Added scope prefix +version: "1.0.0" # Added version +description: Align reads using BWA-MEM +keywords: + - alignment + - bwa +license: MIT # Added module license +requires: # Added requirements + nextflow: ">=24.04.0" + modules: # Added module dependencies (if any) + - nf-core/samtools/sort@>=1.0.0 +tools: + - bwa: + description: BWA software + homepage: http://bio-bwa.sourceforge.net/ + licence: ["GPL-3.0-or-later"] + identifier: biotools:bwa +authors: + - "@drpatelh" +maintainers: + - "@drpatelh" +input: + # ... unchanged +output: + # ... unchanged +``` + +#### Schema Validation + +Use the schema reference in your `meta.yaml`: + +```yaml +# yaml-language-server: $schema=https://registry.nextflow.io/schemas/module-spec/v1.0.0 + +name: nf-core/my-module +version: "1.0.0" +# ... +``` + +### Compatibility Matrix + +| Feature | nf-core Current | Nextflow Module System | +|---------|-----------------|------------------------| +| Simple names | Yes | Yes (local only) | +| Scoped names | No | Yes (registry) | +| Version field | No | Yes (required for registry) | +| `tools` section | Yes | Yes | +| `components` | Yes (subworkflows) | Deprecated → use `requires.modules` | +| `requires` | No | Yes (unified requirements field) | +| I/O specifications | Yes | Yes | +| Ontologies | Yes | Yes | + +### Unsupported nf-core Attributes + +The following attributes from the nf-core meta schema are **not supported** in the Nextflow module system: + +| Attribute | Reason | Future | +|-----------|--------|--------| +| `extra_args` | Not adopted in practice by nf-core modules | Will be redesigned as part of the `tools` schema attribute to document tool-specific arguments and configuration options | +| `components` | Replaced by unified `requires.modules` | Use `requires.modules` for all module dependencies (local and registry) | + +### Complete Examples + +#### Minimal nf-core Module + +```yaml +name: fastqc +description: Run FastQC on sequenced reads +keywords: + - quality control + - qc + - fastq +tools: + - fastqc: + description: FastQC quality metrics + homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/ + licence: ["GPL-2.0-only"] + identifier: biotools:fastqc +authors: + - "@drpatelh" +maintainers: + - "@drpatelh" +output: + html: + - "*.html": + type: file + description: FastQC HTML report + versions: + - versions.yml: + type: file + description: Software versions +``` + +#### Full Registry Module + +```yaml +name: nf-core/bwa-align +version: "1.2.4" +description: Align reads to reference genome using BWA-MEM algorithm +keywords: + - alignment + - mapping + - bwa + - bam + - fastq +license: MIT + +requires: + nextflow: ">=24.04.0" + plugins: + - nf-wave@1.5.0 + modules: + - nf-core/samtools/view@>=1.0.0,<2.0.0 + - nf-core/samtools/sort@>=2.1.0,<2.2.0 + +tools: + - bwa: + description: | + BWA is a software package for mapping DNA sequences + against a large reference genome. + homepage: http://bio-bwa.sourceforge.net/ + documentation: https://bio-bwa.sourceforge.net/bwa.shtml + doi: 10.1093/bioinformatics/btp324 + licence: ["GPL-3.0-or-later"] + identifier: biotools:bwa + +authors: + - "@nf-core" +maintainers: + - "@drpatelh" + - "@maxulysse" + +input: + - - meta: + type: map + description: Sample metadata map (e.g., [ id:'sample1', single_end:false ]) + - reads: + type: file + description: Input FastQ files + ontologies: + - edam: "http://edamontology.org/format_1930" + - - meta2: + type: map + description: Reference metadata + - index: + type: directory + description: BWA index directory + ontologies: + - edam: "http://edamontology.org/data_3210" + +output: + bam: + - - meta: + type: map + description: Sample metadata + - "*.bam": + type: file + description: Aligned BAM file + pattern: "*.bam" + ontologies: + - edam: "http://edamontology.org/format_2572" + versions: + - versions.yml: + type: file + description: Software versions + pattern: "versions.yml" +``` + +#### Subworkflow with Module Dependencies + +```yaml +name: fastq_align_bwa +description: Align reads with BWA and generate statistics +keywords: + - alignment + - bwa + - samtools + - statistics + +requires: + modules: + - bwa/mem + - samtools/sort + - samtools/index + - samtools/stats + +authors: + - "@JoseEspinosa" +maintainers: + - "@JoseEspinosa" + +input: + - ch_reads: + description: | + Input FastQ files + Structure: [ val(meta), [ path(reads) ] ] + - ch_index: + description: BWA index files + +output: + - bam: + description: Sorted BAM files + - bai: + description: BAM index files + - stats: + description: Alignment statistics + - versions: + description: Software versions +``` diff --git a/adr/module-spec-schema.json b/adr/module-spec-schema.json new file mode 100644 index 0000000000..6e54cf6bba --- /dev/null +++ b/adr/module-spec-schema.json @@ -0,0 +1,521 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://registry.nextflow.io/schemas/module-spec/v1.0.0", + "title": "Nextflow Module Metadata Schema", + "description": "Schema for Nextflow module meta.yaml files, supporting both nf-core community patterns and the Nextflow module system", + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "Module name. Can be a simple identifier (e.g., 'fastqc', 'bwa_mem') for local/nf-core modules, or a fully qualified scoped name (e.g., 'nf-core/fastqc', 'myorg/custom') for registry modules. Note: The '@' prefix is only used in DSL include statements, not in meta.yaml", + "examples": ["fastqc", "bwa_mem", "nf-core/fastqc", "myorg/salmon-quant"], + "pattern": "^([a-z0-9][a-z0-9-]*/)?[a-z][a-z0-9_-]*$" + }, + "version": { + "type": "string", + "description": "Semantic version of the module (MAJOR.MINOR.PATCH). Required for registry publication", + "pattern": "^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)(-[0-9A-Za-z-]+(\\.[0-9A-Za-z-]+)*)?(\\+[0-9A-Za-z-]+(\\.[0-9A-Za-z-]+)*)?$", + "examples": ["1.0.0", "2.1.3", "1.0.0-beta.1"] + }, + "description": { + "type": "string", + "description": "Brief description of what the module does", + "minLength": 10, + "maxLength": 500 + }, + "keywords": { + "type": "array", + "description": "Keywords for discovery and categorization", + "items": { + "type": "string", + "minLength": 2 + }, + "minItems": 1, + "uniqueItems": true + }, + "license": { + "type": "string", + "description": "SPDX license identifier for the module code itself", + "examples": ["MIT", "Apache-2.0", "GPL-3.0-or-later"] + }, + "authors": { + "type": "array", + "description": "Original authors of the module (GitHub handles preferred)", + "items": { + "type": "string", + "pattern": "^@?[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?$" + }, + "minItems": 1 + }, + "maintainers": { + "type": "array", + "description": "Current maintainers of the module (GitHub handles preferred)", + "items": { + "type": "string", + "pattern": "^@?[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?$" + } + }, + "requires": { + "type": "object", + "description": "All requirements for the module: runtime environment, plugins, and dependencies", + "properties": { + "nextflow": { + "type": "string", + "description": "Nextflow version constraint using comparison operators", + "examples": [">=24.04.0", ">=24.04.0,<25.0.0"], + "pattern": "^[<>=!]+[0-9]+\\.[0-9]+\\.[0-9]+(-[a-zA-Z0-9]+)?(,\\s*[<>=!]+[0-9]+\\.[0-9]+\\.[0-9]+(-[a-zA-Z0-9]+)?)*$" + }, + "plugins": { + "type": "array", + "description": "Required Nextflow plugins with optional version constraints", + "items": { + "type": "string", + "description": "Plugin reference in format 'plugin-name' or 'plugin-name@constraint'", + "pattern": "^[a-z][a-z0-9-]*(@[<>=,0-9.]+)?$", + "examples": ["nf-amazon@2.0.0", "nf-wave@>=1.5.0", "nf-azure"] + } + }, + "modules": { + "type": "array", + "description": "Required modules (processes) with optional version constraints", + "items": { + "type": "string", + "description": "Module reference in format '[scope/]name' or '[scope/]name@constraint'", + "pattern": "^([a-z0-9][a-z0-9-]*/)?[a-z][a-z0-9_/-]*(@[<>=,0-9.]+)?$", + "examples": ["nf-core/fastqc@>=1.0.0", "nf-core/samtools/sort@>=2.1.0,<3.0.0", "bwa/mem"] + } + }, + "workflows": { + "type": "array", + "description": "Required workflows/subworkflows with optional version constraints", + "items": { + "type": "string", + "description": "Workflow reference in format '[scope/]name' or '[scope/]name@constraint'", + "pattern": "^([a-z0-9][a-z0-9-]*/)?[a-z][a-z0-9_/-]*(@[<>=,0-9.]+)?$", + "examples": ["nf-core/fastq-align-bwa@1.0.0", "my-subworkflow"] + } + } + }, + "additionalProperties": false + }, + "tools": { + "type": "array", + "description": "Software tools wrapped by this module with their metadata", + "items": { + "type": "object", + "minProperties": 1, + "maxProperties": 1, + "patternProperties": { + "^[a-zA-Z][a-zA-Z0-9_-]*$": { + "$ref": "#/$defs/toolSpec" + } + } + } + }, + "input": { + "description": "Input channel specifications for the module's process(es)", + "oneOf": [ + { + "type": "array", + "description": "Array-based input specification (nf-core modules pattern)", + "items": { + "$ref": "#/$defs/inputChannelItem" + } + }, + { + "type": "object", + "description": "Object-based input specification (simplified pattern)", + "patternProperties": { + "^[a-zA-Z_][a-zA-Z0-9_]*$": { + "$ref": "#/$defs/channelElementSpec" + } + } + } + ] + }, + "output": { + "description": "Output channel specifications for the module's process(es)", + "oneOf": [ + { + "type": "object", + "description": "Object-based output specification (nf-core modules pattern)", + "patternProperties": { + "^[a-zA-Z_][a-zA-Z0-9_]*$": { + "$ref": "#/$defs/outputChannelDef" + } + } + }, + { + "type": "array", + "description": "Array-based output specification (nf-core subworkflows pattern)", + "items": { + "type": "object", + "patternProperties": { + "^[a-zA-Z_][a-zA-Z0-9_]*$": { + "$ref": "#/$defs/channelElementSpec" + } + } + } + } + ] + } + }, + "required": ["name", "description"], + "$defs": { + "toolSpec": { + "type": "object", + "description": "Specification for a software tool used by the module", + "properties": { + "description": { + "type": "string", + "description": "Description of the tool and its purpose" + }, + "homepage": { + "type": "string", + "format": "uri", + "description": "Tool's homepage URL", + "pattern": "^https?://.*$" + }, + "documentation": { + "type": "string", + "format": "uri", + "description": "Documentation URL", + "pattern": "^(https?|ftp)://.*$" + }, + "tool_dev_url": { + "type": "string", + "format": "uri", + "description": "Development/source code URL", + "pattern": "^https?://.*$" + }, + "doi": { + "description": "Digital Object Identifier for the tool's publication", + "oneOf": [ + { + "type": "string", + "pattern": "^10\\.\\d{4,9}/[^,]+$" + }, + { + "type": "string", + "const": "no DOI available" + } + ] + }, + "arxiv": { + "type": "string", + "description": "arXiv identifier", + "pattern": "^arXiv:\\d{4}\\.\\d{4,5}(v\\d+)?$" + }, + "licence": { + "type": "array", + "description": "SPDX license identifier(s) for the tool", + "items": { + "type": "string" + }, + "minItems": 1, + "uniqueItems": true + }, + "identifier": { + "description": "bio.tools identifier or empty string", + "oneOf": [ + { + "type": "string", + "pattern": "^biotools:[a-zA-Z0-9_-]+$" + }, + { + "type": "string", + "maxLength": 0 + } + ] + }, + "manual": { + "type": "string", + "format": "uri", + "description": "Manual/user guide URL" + } + }, + "required": ["description"], + "anyOf": [ + { "required": ["homepage"] }, + { "required": ["documentation"] }, + { "required": ["tool_dev_url"] }, + { "required": ["doi"] } + ] + }, + "channelElementSpec": { + "type": "object", + "description": "Specification for a channel element (input or output)", + "properties": { + "type": { + "type": "string", + "description": "Data type of the channel element", + "enum": ["map", "file", "directory", "string", "integer", "float", "boolean", "list", "val"] + }, + "description": { + "type": "string", + "description": "Human-readable description of the channel element" + }, + "pattern": { + "type": "string", + "description": "File pattern in glob syntax or allowed values pattern" + }, + "optional": { + "type": "boolean", + "description": "Whether this input is optional", + "default": false + }, + "default": { + "description": "Default value if not provided" + }, + "enum": { + "type": "array", + "description": "List of allowed values", + "uniqueItems": true + }, + "ontologies": { + "type": "array", + "description": "Ontology annotations (e.g., EDAM)", + "items": { + "type": "object", + "patternProperties": { + "^[a-zA-Z]+$": { + "type": "string", + "format": "uri", + "description": "Ontology URI" + } + } + }, + "uniqueItems": true + } + }, + "required": ["description"] + }, + "inputChannelItem": { + "description": "Input channel item - can be a tuple (array) or single element (object)", + "oneOf": [ + { + "type": "array", + "description": "Tuple-style input channel (multiple elements per emission)", + "items": { + "type": "object", + "patternProperties": { + "^[a-zA-Z_][a-zA-Z0-9_]*$|^\\*\\..*$": { + "$ref": "#/$defs/channelElementSpec" + } + } + } + }, + { + "type": "object", + "description": "Single-element input channel", + "patternProperties": { + "^[a-zA-Z_][a-zA-Z0-9_]*$": { + "$ref": "#/$defs/channelElementSpec" + } + } + } + ] + }, + "outputChannelDef": { + "type": "array", + "description": "Output channel definition - array of emission patterns", + "items": { + "oneOf": [ + { + "type": "object", + "description": "Single output element", + "patternProperties": { + "^[a-zA-Z_$][a-zA-Z0-9_{}.$*\"']*$": { + "$ref": "#/$defs/channelElementSpec" + } + } + }, + { + "type": "array", + "description": "Tuple output (multiple elements per emission)", + "items": { + "type": "object", + "patternProperties": { + "^[a-zA-Z_$][a-zA-Z0-9_{}.$*\"']*$|^\\*\\..*$": { + "$ref": "#/$defs/channelElementSpec" + } + } + } + } + ] + } + } + }, + "allOf": [ + { + "if": { + "properties": { + "name": { + "pattern": "^[a-z0-9][a-z0-9-]*/" + } + }, + "required": ["name"] + }, + "then": { + "required": ["name", "description", "version"], + "properties": { + "version": { + "description": "Version is required for scoped/registry modules (scope/name format)" + } + } + } + } + ], + "examples": [ + { + "name": "fastqc", + "description": "Run FastQC on sequenced reads", + "keywords": ["quality control", "qc", "adapters", "fastq"], + "tools": [ + { + "fastqc": { + "description": "FastQC gives general quality metrics about your reads.", + "homepage": "https://www.bioinformatics.babraham.ac.uk/projects/fastqc/", + "documentation": "https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/", + "licence": ["GPL-2.0-only"], + "identifier": "biotools:fastqc" + } + } + ], + "input": [ + [ + { + "meta": { + "type": "map", + "description": "Groovy Map containing sample information" + } + }, + { + "reads": { + "type": "file", + "description": "Input FastQ files", + "ontologies": [] + } + } + ] + ], + "output": { + "html": [ + [ + { + "meta": { + "type": "map", + "description": "Sample information" + } + }, + { + "*.html": { + "type": "file", + "description": "FastQC report", + "pattern": "*_{fastqc.html}", + "ontologies": [] + } + } + ] + ], + "versions": [ + { + "versions.yml": { + "type": "file", + "description": "File containing software versions", + "pattern": "versions.yml" + } + } + ] + }, + "authors": ["@drpatelh", "@ewels"], + "maintainers": ["@drpatelh", "@ewels"] + }, + { + "name": "nf-core/bwa-align", + "version": "1.2.4", + "description": "Align reads using BWA-MEM algorithm", + "keywords": ["alignment", "bwa", "mapping", "fastq", "bam"], + "license": "MIT", + "authors": ["@nf-core"], + "maintainers": ["@nf-core"], + "requires": { + "nextflow": ">=24.04.0", + "plugins": [ + "nf-amazon@2.0.0" + ], + "modules": [ + "nf-core/samtools/view@>=1.0.0,<2.0.0", + "nf-core/samtools/sort@>=2.1.0,<2.2.0" + ] + }, + "tools": [ + { + "bwa": { + "description": "BWA is a software package for mapping DNA sequences against a large reference genome.", + "homepage": "http://bio-bwa.sourceforge.net/", + "documentation": "https://bio-bwa.sourceforge.net/bwa.shtml", + "doi": "10.1093/bioinformatics/btp324", + "licence": ["GPL-3.0-or-later"], + "identifier": "biotools:bwa" + } + } + ], + "input": [ + [ + { + "meta": { + "type": "map", + "description": "Sample metadata map" + } + }, + { + "reads": { + "type": "file", + "description": "Input FastQ files", + "ontologies": [ + { "edam": "http://edamontology.org/format_1930" } + ] + } + } + ], + { + "index": { + "type": "directory", + "description": "BWA index directory" + } + } + ], + "output": { + "bam": [ + [ + { + "meta": { + "type": "map", + "description": "Sample metadata" + } + }, + { + "*.bam": { + "type": "file", + "description": "Aligned BAM file", + "pattern": "*.bam", + "ontologies": [ + { "edam": "http://edamontology.org/format_2572" } + ] + } + } + ] + ], + "versions": [ + { + "versions.yml": { + "type": "file", + "description": "Software versions" + } + } + ] + } + } + ] +} \ No newline at end of file From 01c777aaf92a52ad9aaf2d465ff4cbe1ea5a17ec Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Tue, 6 Jan 2026 19:03:07 +0700 Subject: [PATCH 07/75] Add structured tool arguments to module spec schema - Add `args` property to tools section for type-safe argument configuration - Define toolArgSpec with flag, type, description, default, enum, required - Support implicit variable `tools..args.` returning formatted flag+value (e.g., "-K 100000000") - Support `tools..args` to return all args concatenated - Document deprecation of ext.args/ext.args2/ext.args3 pattern - Update ADR with Tool Arguments Configuration section and appendix [ci skip] Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 152 +++++++++++++++++++++++++++++++--- adr/module-spec-schema.json | 76 ++++++++++++++++- 2 files changed, 214 insertions(+), 14 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 31d1a59d77..cfc5f1effc 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -2,9 +2,22 @@ - Authors: Paolo Di Tommaso - Status: draft -- Date: 2025-12-11 +- Date: 2025-01-06 - Tags: modules, dsl, registry, versioning, architecture -- Version: 2.1 +- Version: 2.2 + +## Updates + +### Version 2.2 (2025-01-06) +- **Structured tool arguments**: Added `args` property to `tools` section for type-safe argument configuration +- **New implicit variables**: `tools..args.` returns formatted flag+value; `tools..args` returns all args concatenated +- **Deprecation**: `ext.args`, `ext.args2`, `ext.args3` pattern deprecated in favor of structured tool arguments + +### Version 2.1 (2025-12-11) +- **Unified dependencies**: Consolidated `components`, `dependencies`, and `requires` into single `requires` field +- **New sub-properties**: `requires.modules` and `requires.workflows` for declaring module dependencies +- **Unified version syntax**: `[scope/]name[@constraint]` format across plugins, modules, and workflows +- **Deprecation**: `components` field deprecated (use `requires.modules` instead) ## Context and Problem Statement @@ -471,6 +484,99 @@ project-root/ - Single authentication system - Separate cache locations: `$NXF_HOME/plugins/` (global) vs `modules/` (per-project) +## Tool Arguments Configuration + +The module system introduces a structured approach to tool argument configuration, replacing the legacy `ext.args` pattern with type-safe, documented argument specifications. + +### Current Pattern (Deprecated) + +The traditional nf-core pattern uses `ext.args` strings in config files: + +```groovy +// Config file +withName: 'BWA_MEM' { + ext.args = "-K 100000000 -Y -B 3 -R ${meta.read_group}" + ext.args2 = "--output-fmt cram" +} + +// Module script +def args = task.ext.args ?: '' +def args2 = task.ext.args2 ?: '' +bwa mem $args -t $task.cpus $index $reads | samtools sort $args2 -o out.bam - +``` + +**Limitations:** +- No documentation of available arguments +- No validation or type checking +- Unclear which `ext.argsN` maps to which tool +- No IDE autocompletion support + +### New Pattern: Structured Tool Arguments + +Modules declare available arguments in `meta.yaml` under each tool's `args` property: + +```yaml +tools: + - bwa: + description: BWA aligner + homepage: http://bio-bwa.sourceforge.net/ + args: + K: + flag: "-K" + type: integer + description: "Process INT input bases in each batch" + Y: + flag: "-Y" + type: boolean + description: "Use soft clipping for supplementary alignments" + + - samtools: + description: SAMtools + homepage: http://www.htslib.org/ + args: + output_fmt: + flag: "--output-fmt" + type: string + enum: ["sam", "bam", "cram"] + description: "Output format" +``` + +### Configuration Usage + +Arguments are configured using `tools..args.`: + +```groovy +withName: 'BWA_MEM' { + tools.bwa.args.K = 100000000 + tools.bwa.args.Y = true + tools.samtools.args.output_fmt = "cram" +} +``` + +### Script Usage + +In module scripts, access arguments via the `tools` implicit variable: + +```groovy +// tools.bwa.args.K → "-K 100000000" +// tools.bwa.args.Y → "-Y" +// tools.bwa.args → "-K 100000000 -Y" (all args concatenated) + +bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ + | samtools sort ${tools.samtools.args} -o ${prefix}.bam - +``` + +### Benefits + +| Aspect | `ext.args` (Legacy) | `tools.*.args` (New) | +|--------|---------------------|----------------------| +| Documentation | None | In meta.yaml | +| Type Safety | None | Validated | +| IDE Support | None | Autocompletion | +| Multi-tool | Confusing (`ext.args2`) | Clear (`tools.samtools.args`) | +| Defaults | Manual | Schema-defined | +| Enums | None | Validated | + ## Comparison: Plugins vs. Modules | Aspect | Plugins | Modules | @@ -691,20 +797,23 @@ requires: #### `tools` -Documents the software tools wrapped by the module: +Documents the software tools wrapped by the module, including their command-line arguments: ```yaml tools: - bwa: - description: | - BWA is a software package for mapping DNA sequences - against a large reference genome. + description: BWA aligner homepage: http://bio-bwa.sourceforge.net/ - documentation: https://bio-bwa.sourceforge.net/bwa.shtml - doi: 10.1093/bioinformatics/btp324 - arxiv: arXiv:1303.3997 licence: ["GPL-3.0-or-later"] - identifier: biotools:bwa + args: + K: + flag: "-K" + type: integer + description: "Process INT input bases in each batch" + Y: + flag: "-Y" + type: boolean + description: "Use soft clipping for supplementary alignments" ``` **Tool Properties:** @@ -720,6 +829,29 @@ tools: | `licence` | Recommended | SPDX license(s) | | `identifier` | Recommended | bio.tools identifier | | `manual` | No | User manual URL | +| `args` | No | Command-line argument specifications | + +**Argument Properties (`args.`):** + +The `args` object maps argument names to their specifications. Argument names become accessible in scripts via `tools..args.`. + +| Property | Required | Description | +|----------|----------|-------------| +| `flag` | Yes | CLI flag (e.g., `-K`, `--output-fmt`) | +| `type` | Yes | Data type: `boolean`, `integer`, `float`, `string`, `file`, `path` | +| `description` | Yes | Human-readable description | +| `default` | No | Default value | +| `enum` | No | List of allowed values | +| `required` | No | Whether the argument is mandatory (default: false) | + +**Argument Type Behavior:** + +| Type | Config Example | Output | +|------|----------------|--------| +| `boolean` | `tools.bwa.args.Y = true` | `-Y` | +| `integer` | `tools.bwa.args.K = 100000` | `-K 100000` | +| `string` | `tools.bwa.args.R = "@RG\tID:s1"` | `-R @RG\tID:s1` | +| `string` + `enum` | `tools.samtools.args.output_fmt = "cram"` | `--output-fmt cram` | #### `input` and `output` diff --git a/adr/module-spec-schema.json b/adr/module-spec-schema.json index 6e54cf6bba..6f54ed6fed 100644 --- a/adr/module-spec-schema.json +++ b/adr/module-spec-schema.json @@ -232,6 +232,16 @@ "type": "string", "format": "uri", "description": "Manual/user guide URL" + }, + "args": { + "type": "object", + "description": "Command-line arguments supported by the tool. Keys are argument names accessible via tool..args.", + "patternProperties": { + "^[a-zA-Z_][a-zA-Z0-9_]*$": { + "$ref": "#/$defs/toolArgSpec" + } + }, + "additionalProperties": false } }, "required": ["description"], @@ -242,6 +252,40 @@ { "required": ["doi"] } ] }, + "toolArgSpec": { + "type": "object", + "description": "Specification for a tool command-line argument", + "properties": { + "flag": { + "type": "string", + "description": "The CLI flag (e.g., '-n', '--output-fmt')", + "pattern": "^--?[a-zA-Z][a-zA-Z0-9_-]*$" + }, + "type": { + "type": "string", + "description": "Data type of the argument value", + "enum": ["boolean", "integer", "float", "string", "file", "path"] + }, + "description": { + "type": "string", + "description": "Human-readable description of the argument" + }, + "default": { + "description": "Default value for the argument" + }, + "enum": { + "type": "array", + "description": "List of allowed values", + "uniqueItems": true + }, + "required": { + "type": "boolean", + "description": "Whether this argument is required", + "default": false + } + }, + "required": ["flag", "type", "description"] + }, "channelElementSpec": { "type": "object", "description": "Specification for a channel element (input or output)", @@ -452,12 +496,36 @@ "tools": [ { "bwa": { - "description": "BWA is a software package for mapping DNA sequences against a large reference genome.", + "description": "BWA aligner", "homepage": "http://bio-bwa.sourceforge.net/", - "documentation": "https://bio-bwa.sourceforge.net/bwa.shtml", - "doi": "10.1093/bioinformatics/btp324", "licence": ["GPL-3.0-or-later"], - "identifier": "biotools:bwa" + "args": { + "K": { + "flag": "-K", + "type": "integer", + "description": "Process INT input bases in each batch" + }, + "Y": { + "flag": "-Y", + "type": "boolean", + "description": "Use soft clipping for supplementary alignments" + } + } + } + }, + { + "samtools": { + "description": "SAMtools", + "homepage": "http://www.htslib.org/", + "licence": ["MIT"], + "args": { + "output_fmt": { + "flag": "--output-fmt", + "type": "string", + "description": "Output format", + "enum": ["sam", "bam", "cram"] + } + } } } ], From 7e12f3a6fa40b1a6c3b84ec40e15e6783a937585 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 7 Jan 2026 22:59:50 +0700 Subject: [PATCH 08/75] Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index cfc5f1effc..094fd2aa7b 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -32,7 +32,7 @@ Implement a module system with four core capabilities: 1. **Remote module inclusion** via registry 2. **Semantic versioning** with dependency resolution 3. **Unified Nextflow Registry** (rebrand existing plugin registry) -4. **First-class CLI support** (pull, push, search, run) +4. **First-class CLI support** (install, publish, search, list, remove, freeze, run) ## Core Capabilities From 5a01d00374dac09c9df2bd0aa9f08582730ad924 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 7 Jan 2026 23:00:18 +0700 Subject: [PATCH 09/75] Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 094fd2aa7b..d9b2a3f763 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -455,7 +455,7 @@ project-root/ **Phase 3**: Extend DSL parser for `from module` syntax, implement dependency resolution from meta.yaml -**Phase 4**: Implement `push` command with authentication and `run` command +**Phase 4**: Implement `publish` command with authentication and `run` command **Phase 5**: Advanced features (search UI, language server integration, ontology validation) From 4779d5091e0bbe3ab223d833edc5cbe41871f80c Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 7 Jan 2026 23:00:35 +0700 Subject: [PATCH 10/75] Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index d9b2a3f763..c110adf7c7 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -451,7 +451,7 @@ project-root/ **Phase 1**: Module manifest schema, local module loading, validation tools -**Phase 2**: Extend plugin registry for modules, implement caching, add `pull` and `search` commands +**Phase 2**: Extend plugin registry for modules, implement caching, add `install` and `search` commands **Phase 3**: Extend DSL parser for `from module` syntax, implement dependency resolution from meta.yaml From 5b1d4e5030dae783928ac7d62b0fe227b6fb959b Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 14 Jan 2026 15:48:17 +0100 Subject: [PATCH 11/75] Update adr/20251114-module-system.md [ci skip] Expand deprecation notice to cover all ext.* custom directives Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index c110adf7c7..8dabbf2dea 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -11,7 +11,7 @@ ### Version 2.2 (2025-01-06) - **Structured tool arguments**: Added `args` property to `tools` section for type-safe argument configuration - **New implicit variables**: `tools..args.` returns formatted flag+value; `tools..args` returns all args concatenated -- **Deprecation**: `ext.args`, `ext.args2`, `ext.args3` pattern deprecated in favor of structured tool arguments +- **Deprecation**: All `ext.*` custom directives (e.g., `ext.args`, `ext.args2`, `ext.args3`, `ext.prefix`, `ext.suffix`) deprecated in favor of structured tool arguments ### Version 2.1 (2025-12-11) - **Unified dependencies**: Consolidated `components`, `dependencies`, and `requires` into single `requires` field From a06ab7a0f45b48b2dc51adb71fee5a61ae6102ca Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 14 Jan 2026 15:58:08 +0100 Subject: [PATCH 12/75] Fix inconsistencies in module system ADR [ci skip] - Use consistent module path format with version: modules/@scope/name@version/ - Fix directory structure example: samtools-view -> samtools/view - Standardize on 'license' spelling (American English) - Fix author -> authors (plural array format) Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 8dabbf2dea..f11e7f0e8a 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -53,7 +53,7 @@ include { MY_PROCESS } from './modules/my-process.nf' **Resolution Order**: 1. Check `nextflow.config` for pinned version -2. Check local `modules/@scope/name/` for any cached version +2. Check local `modules/@scope/name@version/` for any cached version 3. Query registry for latest version if not found 4. Warn if transitive dependencies are not pinned @@ -416,7 +416,8 @@ my-module/ name: nf-core/bwa-align version: 1.2.4 # This module's version description: Align reads using BWA-MEM -author: nf-core community +authors: + - nf-core community license: MIT requires: @@ -438,7 +439,7 @@ project-root/ │ ├── bwa-align@1.2.4/ │ │ ├── meta.yaml │ │ └── main.nf # Required entry point - │ └── samtools-view@1.0.5/ + │ └── samtools/view@1.0.5/ │ ├── meta.yaml │ └── main.nf # Required entry point └── @myorg/ @@ -465,7 +466,7 @@ project-root/ 1. Parse `include` statements → extract module names (e.g., `@nf-core/bwa-align`) 2. For each module: a. Check `nextflow.config` modules section for pinned version - b. If not pinned: check local `modules/@scope/name/` for any cached version (use latest local) + b. If not pinned: check local `modules/@scope/name@version/` for any cached version (use latest local) c. If not found locally: query registry for latest version d. Warn if module not pinned in config (especially transitive dependencies) 3. Download missing modules to `modules/@scope/name@version/` @@ -804,7 +805,7 @@ tools: - bwa: description: BWA aligner homepage: http://bio-bwa.sourceforge.net/ - licence: ["GPL-3.0-or-later"] + license: ["GPL-3.0-or-later"] args: K: flag: "-K" @@ -826,7 +827,7 @@ tools: | `tool_dev_url` | One of these | Development/source URL | | `doi` | One of these | Publication DOI | | `arxiv` | No | arXiv identifier | -| `licence` | Recommended | SPDX license(s) | +| `license` | Recommended | SPDX license(s) | | `identifier` | Recommended | bio.tools identifier | | `manual` | No | User manual URL | | `args` | No | Command-line argument specifications | @@ -932,7 +933,7 @@ tools: - bwa: description: BWA software homepage: http://bio-bwa.sourceforge.net/ - licence: ["GPL-3.0-or-later"] + license: ["GPL-3.0-or-later"] identifier: biotools:bwa authors: - "@drpatelh" @@ -961,7 +962,7 @@ tools: - bwa: description: BWA software homepage: http://bio-bwa.sourceforge.net/ - licence: ["GPL-3.0-or-later"] + license: ["GPL-3.0-or-later"] identifier: biotools:bwa authors: - "@drpatelh" @@ -1022,7 +1023,7 @@ tools: - fastqc: description: FastQC quality metrics homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/ - licence: ["GPL-2.0-only"] + license: ["GPL-2.0-only"] identifier: biotools:fastqc authors: - "@drpatelh" @@ -1069,7 +1070,7 @@ tools: homepage: http://bio-bwa.sourceforge.net/ documentation: https://bio-bwa.sourceforge.net/bwa.shtml doi: 10.1093/bioinformatics/btp324 - licence: ["GPL-3.0-or-later"] + license: ["GPL-3.0-or-later"] identifier: biotools:bwa authors: From f7dab8d58a7ccbd28c2fa8d83a4567f5663fb0fa Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 14 Jan 2026 16:00:34 +0100 Subject: [PATCH 13/75] Fix registry API path in comparison table [ci skip] Remove @ prefix from scope in API path to match API definition Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index f11e7f0e8a..ea2f6c25ef 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -589,7 +589,7 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ | Naming | `nf-amazon` | `@nf-core/salmon` | | Cache Location | `$NXF_HOME/plugins/` | `modules/@scope/name@version/` | | Version Config | `plugins {}` in config | `modules {}` in config | -| Registry Path | `/api/v1/plugins/` | `/api/v1/modules/@scope/name` | +| Registry Path | `/api/v1/plugins/` | `/api/v1/modules/{scope}/{name}` | ## Rationale From 0242a4ab78c8cf754d53ccf5e2b27429201cc5b1 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 14 Jan 2026 16:05:14 +0100 Subject: [PATCH 14/75] Fix checksum format: sha256- -> sha256: [ci skip] Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index ea2f6c25ef..d0cf0a272b 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -75,7 +75,7 @@ modules { '@nf-core/salmon' = '1.1.0' // Simple syntax '@nf-core/bwa-align' = [ // Extended syntax (with checksum) version: '1.2.0', - checksum: 'sha256-abc123...' + checksum: 'sha256:abc123...' ] } @@ -300,7 +300,7 @@ Lock all module versions by writing exact versions and SHA-256 checksums to `nex modules { '@nf-core/bwa-align' = [ version: '1.2.4', - checksum: 'sha256-a1b2c3d4e5f6...' + checksum: 'sha256:a1b2c3d4e5f6...' ] } ``` From 5461f5b56ce2f87bffd51013ad940bcf51bb192a Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 14 Jan 2026 16:15:37 +0100 Subject: [PATCH 15/75] Update module API to match registry implementation [ci skip] - Update API endpoints to match seqeralabs/plugin-registry#266 - Use /api/modules base path (no v1 prefix) - Use single {name} parameter with namespace (e.g., "nf-core/fastqc") - Add separate /releases endpoint - Simplify publish to single POST endpoint Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index d0cf0a272b..cee7acfd8c 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -31,7 +31,7 @@ Implement a module system with four core capabilities: 1. **Remote module inclusion** via registry 2. **Semantic versioning** with dependency resolution -3. **Unified Nextflow Registry** (rebrand existing plugin registry) +3. **Unified Nextflow Registry** (rebrand existing Nextflow registry) 4. **First-class CLI support** (install, publish, search, list, remove, freeze, run) ## Core Capabilities @@ -141,7 +141,7 @@ This avoids introducing new notation that would require additional parser suppor ### 3. Unified Nextflow Registry -**Architecture Decision**: Extend existing plugin registry at `registry.nextflow.io` to host both plugins and modules. +**Architecture Decision**: Extend existing Nextflow registry at `registry.nextflow.io` to host both plugins and modules. **Current Plugin API** (reference: https://registry.nextflow.io/openapi/): ``` @@ -153,16 +153,18 @@ POST /api/v1/plugins/release # Create draft release POST /api/v1/plugins/release/{releaseId}/upload # Upload artifact ``` -**Proposed Module API Extension** (same pattern): +**Module API** (reference: https://github.com/seqeralabs/plugin-registry/pull/266): ``` -GET /api/v1/modules # List/search modules -GET /api/v1/modules/{scope}/{name} # Get module + all releases -GET /api/v1/modules/{scope}/{name}/{version} # Get specific release -GET /api/v1/modules/{scope}/{name}/{version}/download # Download source archive -POST /api/v1/modules/release # Create draft release -POST /api/v1/modules/release/{releaseId}/upload # Upload module archive +GET /api/modules?query= # Search modules (semantic search) +GET /api/modules/{name} # Get module + latest release +GET /api/modules/{name}/releases # List all releases +GET /api/modules/{name}/{version} # Get specific release +GET /api/modules/{name}/{version}/download # Download module bundle +POST /api/modules/{name} # Publish module version (authenticated) ``` +Note: The `{name}` parameter includes the namespace prefix (e.g., "nf-core/fastqc"). + **Registry URL**: `registry.nextflow.io` **Artifact Types**: @@ -452,7 +454,7 @@ project-root/ **Phase 1**: Module manifest schema, local module loading, validation tools -**Phase 2**: Extend plugin registry for modules, implement caching, add `install` and `search` commands +**Phase 2**: Extend Nextflow registry for modules, implement caching, add `install` and `search` commands **Phase 3**: Extend DSL parser for `from module` syntax, implement dependency resolution from meta.yaml @@ -589,7 +591,7 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ | Naming | `nf-amazon` | `@nf-core/salmon` | | Cache Location | `$NXF_HOME/plugins/` | `modules/@scope/name@version/` | | Version Config | `plugins {}` in config | `modules {}` in config | -| Registry Path | `/api/v1/plugins/` | `/api/v1/modules/{scope}/{name}` | +| Registry Path | `/api/v1/plugins/` | `/api/modules/{name}` | ## Rationale From 926052ce8a5542f013d90ce33575aa990d21a448 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 15 Jan 2026 09:20:47 +0100 Subject: [PATCH 16/75] Update adr [ci skip] Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 36 +++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index cee7acfd8c..d3c54b6330 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -200,14 +200,16 @@ Run a module directly without requiring a wrapper workflow script. This command **Options**: - `-version `: Run a specific version (default: latest or configured version) - `-- `: Map value to the corresponding module process input channel +- `--tools:: `: Configure tool-specific arguments (validated against meta.yaml schema) - All standard `nextflow run` options (e.g., `-profile`, `-work-dir`, `-resume`, etc.) **Behavior**: 1. Checks if module is installed locally; if not, downloads from registry 2. Parses the module's `main.nf` to identify the main process and its input declarations 3. Validates command-line arguments against the process input schema -4. Generates an implicit workflow that wires CLI arguments to process inputs -5. Executes the workflow using standard Nextflow runtime +4. Validates tool arguments against the `tools.*.args` schema in `meta.yaml` +5. Generates an implicit workflow that wires CLI arguments to process inputs +6. Executes the workflow using standard Nextflow runtime **Input Mapping**: - Named arguments (`--reads`, `--reference`) are mapped to corresponding process inputs @@ -215,6 +217,13 @@ Run a module directly without requiring a wrapper workflow script. This command - Multiple values can be provided for inputs expecting collections - Required inputs without defaults must be provided; optional inputs use declared defaults +**Tool Arguments**: +- Arguments prefixed with `--tools:` configure tool-specific parameters +- Format: `--tools:: ` (e.g., `--tools:bwa:K 100000000`) +- Boolean flags can be specified without value (e.g., `--tools:bwa:Y`) +- Arguments are validated against the tool's `args` schema in `meta.yaml` +- Invalid argument names or values that fail type/enum validation produce errors + **Example**: ```bash # Run BWA alignment module with input files @@ -234,6 +243,14 @@ nextflow module run nf-core/salmon \ --index salmon_index \ -work-dir /tmp/work \ --outdir results/ + +# Run with tool-specific arguments +nextflow module run nf-core/bwa-align \ + --reads 'samples/*_{1,2}.fastq.gz' \ + --reference genome.fa \ + --tools:bwa:K 100000000 \ + --tools:bwa:Y \ + --tools:samtools:output_fmt cram ``` --- @@ -654,6 +671,21 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ - Inspired by: [Go Modules](https://go.dev/ref/mod), [npm](https://docs.npmjs.com), [Cargo](https://doc.rust-lang.org/cargo/) - Related: [nf-core modules](https://nf-co.re/modules) +## Open Questions + +1. **Local vs managed module distinction**: Should local modules use the `@` prefix in include statements, or should a dot file (e.g., `.nf-modules`) be used to distinguish local modules from managed/remote modules? + +2. **Tool arguments CLI syntax**: What is the preferred syntax for tool arguments on the command line? + - Colon-separated: `--tools:: ` + - Dot-separated: `--tools.. ` + +3. **Module version configuration**: Should pipeline module versions be specified in `nextflow.config` or in a dedicated pipeline spec file (e.g., `pipeline.yaml`)? + +4. **Local module checksum verification**: How should Nextflow verify that a locally installed module matches the registry checksum? Options include: + - Store a `.checksum` file alongside the downloaded module + - Query the registry API during `freeze -verify` to fetch the expected checksum + - Note: Storing the checksum in `meta.yaml` is not viable as it violates content-addressable storage principles (the checksum would change the content) + --- ## Appendix A: Module Metadata Schema Specification From ec187fd4bb23b5e22cd297463f6ef0ecac4b7ad9 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 15 Jan 2026 10:47:24 +0100 Subject: [PATCH 17/75] Simplify module storage and verification model [ci skip] - Remove checksums from nextflow.config (registry is source of truth) - Use single version per module locally (no version in directory path) - Add .checksum file in module directory for integrity verification - Simplify freeze command to only pin transitive dependency versions - Checksum mismatch reports warning instead of automatic re-download - Remove resolved open question about checksum verification Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 93 +++++++++++++++++------------------ 1 file changed, 46 insertions(+), 47 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index d3c54b6330..3d194221c2 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -52,14 +52,16 @@ include { MY_PROCESS } from './modules/my-process.nf' **Version Resolution**: Module versions pinned in `nextflow.config`. If not specified, use the latest available locally in `modules/` directory, or downloaded and cached in the `modules/` directory. **Resolution Order**: -1. Check `nextflow.config` for pinned version -2. Check local `modules/@scope/name@version/` for any cached version -3. Query registry for latest version if not found -4. Warn if transitive dependencies are not pinned +1. Check `nextflow.config` for declared version +2. Check local `modules/@scope/name/` exists +3. Verify integrity against `.checksum` file +4. If missing, download from registry +5. Warn if local module exist and checksum mismatch +6. Warn if transitive dependencies are not pinned **Resolution Timing**: Modules resolved at workflow parse time (after plugin resolution at startup). -**Local Storage**: Downloaded modules stored in `modules/@scope/name@version/` directory in project root (not global cache). Each module must contain a `main.nf` file as the required entry point. It is intended that module source code will be committed to the pipeline git repository. +**Local Storage**: Downloaded modules stored in `modules/@scope/name/` directory in project root (not global cache). Each module must contain a `main.nf` file as the required entry point. It is intended that module source code will be committed to the pipeline git repository. ### 2. Semantic Versioning and Configuration @@ -72,11 +74,8 @@ include { MY_PROCESS } from './modules/my-process.nf' ```groovy // Module versions (exact versions only, no ranges) modules { - '@nf-core/salmon' = '1.1.0' // Simple syntax - '@nf-core/bwa-align' = [ // Extended syntax (with checksum) - version: '1.2.0', - checksum: 'sha256:abc123...' - ] + '@nf-core/salmon' = '1.1.0' + '@nf-core/bwa-align' = '1.2.0' } // Registry configuration (separate block) @@ -137,7 +136,7 @@ This avoids introducing new notation that would require additional parser suppor - Workflow's `nextflow.config` specifies exact versions for direct dependencies - Transitive dependencies resolved from each module's `meta.yaml` using version constraints - Warn if transitive dependencies are not pinned in workflow config -- Use `nextflow module freeze` to pin all transitive dependencies with checksums +- Use `nextflow module freeze` to pin all transitive dependencies to exact versions ### 3. Unified Nextflow Registry @@ -184,7 +183,7 @@ Note: The `{name}` parameter includes the namespace prefix (e.g., "nf-core/fastq nextflow module run scope/name # Run a module directly without a wrapper script nextflow module search # Search registry nextflow module install [scope/name] # Install all from config, or specific module -nextflow module freeze # Pin all versions + checksums to config +nextflow module freeze # Pin all transitive dependencies to config nextflow module list # Show installed vs configured nextflow module remove scope/name # Remove from config + local cache nextflow module publish scope/name # Publish to registry (requires api key) @@ -288,9 +287,10 @@ Download and install modules to the local `modules/` directory. When called with **Behavior**: 1. Resolves the module version from `nextflow.config` or queries registry for latest 2. Downloads the module archive from the registry -3. Extracts to `modules/@scope/name@version/` directory -4. Recursively installs transitive dependencies declared in `meta.yaml` -5. Updates `nextflow.config` if installing a new module not already configured +3. Extracts to `modules/@scope/name/` directory (replaces existing if version differs) +4. Stores `.checksum` file from registry's X-Checksum response header +5. Recursively installs transitive dependencies declared in `meta.yaml` +6. Updates `nextflow.config` if installing a new module not already configured **Example**: ```bash @@ -303,31 +303,25 @@ nextflow module install nf-core/salmon -version 1.2.0 #### `nextflow module freeze` -Lock all module versions by writing exact versions and SHA-256 checksums to `nextflow.config`. This ensures fully reproducible builds by capturing the precise state of all dependencies. - -**Options**: -- `-verify`: Verify existing checksums without updating +Pin all transitive module dependencies to exact versions in `nextflow.config`. This ensures reproducible builds by capturing the precise versions of all dependencies. **Behavior**: 1. Scans the `modules/` directory for all installed modules -2. Computes SHA-256 checksums for each module archive -3. Converts simple version syntax to extended syntax with checksums in `nextflow.config` -4. Includes transitive dependencies not explicitly declared +2. Reads version from each module's `meta.yaml` +3. Adds transitive dependencies not explicitly declared to `nextflow.config` **Output** (in `nextflow.config`): ```groovy modules { - '@nf-core/bwa-align' = [ - version: '1.2.4', - checksum: 'sha256:a1b2c3d4e5f6...' - ] + '@nf-core/bwa-align' = '1.2.4' + '@nf-core/samtools/view' = '1.0.5' // Transitive dependency + '@nf-core/samtools/sort' = '2.1.0' // Transitive dependency } ``` **Example**: ```bash -nextflow module freeze # Pin all versions + checksums -nextflow module freeze -verify # Verify checksums match +nextflow module freeze # Pin all transitive dependencies ``` --- @@ -367,7 +361,7 @@ Remove a module from both the local `modules/` directory and the `nextflow.confi - `-keep-files`: Remove from config but keep local files **Behavior**: -1. Removes the module directory from `modules/@scope/name@version/` +1. Removes the module directory from `modules/@scope/name/` 2. Removes the module entry from the `modules {}` block in `nextflow.config` 3. Identifies and optionally removes orphaned transitive dependencies 4. Warns if the module is still referenced in workflow files @@ -455,18 +449,27 @@ project-root/ ├── main.nf └── modules/ # Local module cache ├── @nf-core/ - │ ├── bwa-align@1.2.4/ + │ ├── bwa-align/ + │ │ ├── .checksum # Cached registry checksum │ │ ├── meta.yaml │ │ └── main.nf # Required entry point - │ └── samtools/view@1.0.5/ + │ └── samtools/view/ + │ ├── .checksum │ ├── meta.yaml │ └── main.nf # Required entry point └── @myorg/ - └── custom-process@2.0.0/ + └── custom-process/ + ├── .checksum ├── meta.yaml └── main.nf # Required entry point ``` +**Module Integrity Verification**: +- On install: `.checksum` file created from registry's X-Checksum response header +- On run: Local module checksum compared against `.checksum` file +- If match: Proceed without network call +- If mismatch: Report warning (module may have been locally modified) + ## Implementation Strategy **Phase 1**: Module manifest schema, local module loading, validation tools @@ -484,17 +487,18 @@ project-root/ **Dependency Resolution Flow**: 1. Parse `include` statements → extract module names (e.g., `@nf-core/bwa-align`) 2. For each module: - a. Check `nextflow.config` modules section for pinned version - b. If not pinned: check local `modules/@scope/name@version/` for any cached version (use latest local) - c. If not found locally: query registry for latest version - d. Warn if module not pinned in config (especially transitive dependencies) -3. Download missing modules to `modules/@scope/name@version/` + a. Check `nextflow.config` modules section for declared version + b. Check local `modules/@scope/name/` exists + c. Verify local module integrity against `.checksum` file + d. If missing: download from registry; if checksum mismatch: report warning + e. Warn if module not pinned in config (especially transitive dependencies) +3. On download: store module to `modules/@scope/name/` with `.checksum` file 4. Read module's `meta.yaml` → resolve transitive dependencies recursively -5. Verify checksums (if present in config) -6. Parse module's `main.nf` file → make processes available +5. Parse module's `main.nf` file → make processes available **Security**: -- SHA-256 checksum verification for all downloads +- SHA-256 checksum verification on download (stored in `.checksum` file) +- Integrity verification on run (local checksum vs `.checksum` file) - Authentication required for publishing - Support for private registries @@ -606,7 +610,7 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ | Resolution | Startup | Parse time | | Metadata | JSON spec | YAML manifest | | Naming | `nf-amazon` | `@nf-core/salmon` | -| Cache Location | `$NXF_HOME/plugins/` | `modules/@scope/name@version/` | +| Cache Location | `$NXF_HOME/plugins/` | `modules/@scope/name/` | | Version Config | `plugins {}` in config | `modules {}` in config | | Registry Path | `/api/v1/plugins/` | `/api/modules/{name}` | @@ -622,7 +626,7 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ - Single source of truth for workflow dependencies - Simple: exact versions in config, no separate lock file to manage - Transitive dependencies resolved from module's meta.yaml with version constraints -- Use `nextflow module freeze` to pin all versions + checksums when needed +- Use `nextflow module freeze` to pin all transitive dependencies when needed - Reproducibility via explicit version pinning in config **Why parse-time resolution?** @@ -681,11 +685,6 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ 3. **Module version configuration**: Should pipeline module versions be specified in `nextflow.config` or in a dedicated pipeline spec file (e.g., `pipeline.yaml`)? -4. **Local module checksum verification**: How should Nextflow verify that a locally installed module matches the registry checksum? Options include: - - Store a `.checksum` file alongside the downloaded module - - Query the registry API during `freeze -verify` to fetch the expected checksum - - Note: Storing the checksum in `meta.yaml` is not viable as it violates content-addressable storage principles (the checksum would change the content) - --- ## Appendix A: Module Metadata Schema Specification From 46c6060932b5bffa5909e5de7de036f2c2405e7c Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 15 Jan 2026 10:58:10 +0100 Subject: [PATCH 18/75] Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 3d194221c2..82f66452ae 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -285,7 +285,7 @@ Download and install modules to the local `modules/` directory. When called with - `-force`: Re-download even if already installed locally **Behavior**: -1. Resolves the module version from `nextflow.config` or queries registry for latest +1. If `-version' not specified, resolves the module version from `nextflow.config` or queries registry for latest 2. Downloads the module archive from the registry 3. Extracts to `modules/@scope/name/` directory (replaces existing if version differs) 4. Stores `.checksum` file from registry's X-Checksum response header From 1b0ab1a248fef6e47006c64fba036f3e01acff3b Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 15 Jan 2026 11:15:26 +0100 Subject: [PATCH 19/75] Clarify module resolution rules for version changes vs local modifications [ci skip] - Added Resolution Rules table clearly specifying behavior for each scenario: - Version change (local unmodified): automatically replace with declared version - Local modification (checksum mismatch): warn and protect local changes - Use -force flag to override locally modified modules - Updated install command behavior to reflect checksum verification - Updated Technical Details with expanded resolution flow Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 41 ++++++++++++++++++++++++++--------- 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 82f66452ae..9e31ee2278 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -55,9 +55,23 @@ include { MY_PROCESS } from './modules/my-process.nf' 1. Check `nextflow.config` for declared version 2. Check local `modules/@scope/name/` exists 3. Verify integrity against `.checksum` file -4. If missing, download from registry -5. Warn if local module exist and checksum mismatch -6. Warn if transitive dependencies are not pinned +4. Apply resolution rules (see below) +5. Warn if transitive dependencies are not pinned + +**Resolution Rules**: + +| Local State | Declared Version | Action | +|-------------|------------------|--------| +| Missing | Any | Download declared version (or latest if not declared) | +| Exists, checksum valid | Same as declared | Use local module | +| Exists, checksum valid | Different from declared | **Replace** local with declared version | +| Exists, checksum mismatch | Same as declared | **Warn**: module was locally modified, do not override | +| Exists, checksum mismatch | Different from declared | **Warn**: locally modified module will be replaced; use `-force` to override | + +**Key Behaviors**: +- **Version change**: When the declared version differs from the installed version (and local is unmodified), the local module is automatically replaced with the declared version +- **Local modification**: When the local module content was manually changed (checksum mismatch with `.checksum`), Nextflow warns and does NOT override to prevent accidental loss of local changes +- **Force flag**: Use `-force` with `nextflow module install` to override locally modified modules **Resolution Timing**: Modules resolved at workflow parse time (after plugin resolution at startup). @@ -285,12 +299,15 @@ Download and install modules to the local `modules/` directory. When called with - `-force`: Re-download even if already installed locally **Behavior**: -1. If `-version' not specified, resolves the module version from `nextflow.config` or queries registry for latest -2. Downloads the module archive from the registry -3. Extracts to `modules/@scope/name/` directory (replaces existing if version differs) -4. Stores `.checksum` file from registry's X-Checksum response header -5. Recursively installs transitive dependencies declared in `meta.yaml` -6. Updates `nextflow.config` if installing a new module not already configured +1. If `-version` not specified, resolves the module version from `nextflow.config` or queries registry for latest +2. Checks if local module exists and verifies integrity against `.checksum` file +3. If local module is unmodified and version differs: replaces with requested version +4. If local module was modified (checksum mismatch): warns and aborts unless `-force` is used +5. Downloads the module archive from the registry +6. Extracts to `modules/@scope/name/` directory +7. Stores `.checksum` file from registry's X-Checksum response header +8. Recursively installs transitive dependencies declared in `meta.yaml` +9. Updates `nextflow.config` if installing a new module not already configured **Example**: ```bash @@ -490,7 +507,11 @@ project-root/ a. Check `nextflow.config` modules section for declared version b. Check local `modules/@scope/name/` exists c. Verify local module integrity against `.checksum` file - d. If missing: download from registry; if checksum mismatch: report warning + d. Apply resolution rules: + - Missing → download declared version from registry + - Exists, checksum valid, same version → use local + - Exists, checksum valid, different version → replace with declared version + - Exists, checksum mismatch → warn and do NOT override (local changes detected) e. Warn if module not pinned in config (especially transitive dependencies) 3. On download: store module to `modules/@scope/name/` with `.checksum` file 4. Read module's `meta.yaml` → resolve transitive dependencies recursively From 7a1e62e8ec7e8199289086ff85dc98655f2bab88 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 15 Jan 2026 11:24:33 +0100 Subject: [PATCH 20/75] ADR v2.3: Fix resolution rules consistency and add version entry [ci skip] - Fixed Resolution Rules table: locally modified modules will NOT be replaced unless -force is used (was incorrectly saying "will be replaced") - Added Version 2.3 changelog entry documenting: - Resolution Rules table with clear behavior matrix - Local modification protection with -force flag - Simplified storage model (single version per module) - .checksum file for fast integrity verification Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 9e31ee2278..d17ee93b01 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -4,10 +4,16 @@ - Status: draft - Date: 2025-01-06 - Tags: modules, dsl, registry, versioning, architecture -- Version: 2.2 +- Version: 2.3 ## Updates +### Version 2.3 (2026-01-15) +- **Resolution Rules table**: Added clear table specifying behavior for each combination of local state and declared version +- **Local modification protection**: Locally modified modules (checksum mismatch) are NOT overridden unless `-force` flag is used +- **Simplified storage model**: Single version per module locally (`modules/@scope/name/` without version in path) +- **`.checksum` file**: Registry checksum cached locally for fast integrity verification without network calls + ### Version 2.2 (2025-01-06) - **Structured tool arguments**: Added `args` property to `tools` section for type-safe argument configuration - **New implicit variables**: `tools..args.` returns formatted flag+value; `tools..args` returns all args concatenated @@ -66,7 +72,7 @@ include { MY_PROCESS } from './modules/my-process.nf' | Exists, checksum valid | Same as declared | Use local module | | Exists, checksum valid | Different from declared | **Replace** local with declared version | | Exists, checksum mismatch | Same as declared | **Warn**: module was locally modified, do not override | -| Exists, checksum mismatch | Different from declared | **Warn**: locally modified module will be replaced; use `-force` to override | +| Exists, checksum mismatch | Different from declared | **Warn**: locally modified, will NOT replace unless `-force` is used | **Key Behaviors**: - **Version change**: When the declared version differs from the installed version (and local is unmodified), the local module is automatically replaced with the declared version From c6eaabb03886d415cfb29ccbc05dc24a4eaabf65 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 15 Jan 2026 11:25:23 +0100 Subject: [PATCH 21/75] Fix Version 2.1 date typo: 2025 -> 2024 [ci skip] Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index d17ee93b01..f7c2969aa6 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -19,7 +19,7 @@ - **New implicit variables**: `tools..args.` returns formatted flag+value; `tools..args` returns all args concatenated - **Deprecation**: All `ext.*` custom directives (e.g., `ext.args`, `ext.args2`, `ext.args3`, `ext.prefix`, `ext.suffix`) deprecated in favor of structured tool arguments -### Version 2.1 (2025-12-11) +### Version 2.1 (2024-12-11) - **Unified dependencies**: Consolidated `components`, `dependencies`, and `requires` into single `requires` field - **New sub-properties**: `requires.modules` and `requires.workflows` for declaring module dependencies - **Unified version syntax**: `[scope/]name[@constraint]` format across plugins, modules, and workflows From c8bc50d16f9bf21a4f558c2477cfa16d9d386808 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 15 Jan 2026 12:54:31 +0100 Subject: [PATCH 22/75] Add module system client specification [ci skip] Specification for Nextflow module system client implementation based on ADR 20251114-module-system.md. Covers: P1 (Core): - Install and use registry modules via @scope/name syntax - Run modules directly from CLI without wrapper workflow - Structured tool arguments replacing ext.args pattern P2 (Important): - Module version management and freeze command - Module integrity protection with checksum validation P3 (Nice to have): - Remove module command - Search and discover modules - Publish module to registry Registry backend is out of scope (assumed implemented). Signed-off-by: Paolo Di Tommaso --- .../checklists/requirements.md | 39 +++ specs/251117-module-system/spec.md | 258 ++++++++++++++++++ 2 files changed, 297 insertions(+) create mode 100644 specs/251117-module-system/checklists/requirements.md create mode 100644 specs/251117-module-system/spec.md diff --git a/specs/251117-module-system/checklists/requirements.md b/specs/251117-module-system/checklists/requirements.md new file mode 100644 index 0000000000..e01762d921 --- /dev/null +++ b/specs/251117-module-system/checklists/requirements.md @@ -0,0 +1,39 @@ +# Specification Quality Checklist: Nextflow Module System Client + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-01-15 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Specification is complete and ready for `/speckit.plan` +- All 8 user stories have clear acceptance scenarios +- 32 functional requirements defined across 6 categories +- 8 success criteria defined with measurable outcomes +- Edge cases documented for error handling scenarios +- Registry backend is explicitly out of scope (assumed implemented) \ No newline at end of file diff --git a/specs/251117-module-system/spec.md b/specs/251117-module-system/spec.md new file mode 100644 index 0000000000..e489d0a8ef --- /dev/null +++ b/specs/251117-module-system/spec.md @@ -0,0 +1,258 @@ +# Feature Specification: Nextflow Module System Client + +**Feature Branch**: `251117-module-system` +**Created**: 2026-01-15 +**Status**: Draft +**Input**: User description: "Implement Nextflow module system client based on ADR 20251114-module-system.md. Focus on client-side implementation only - CLI commands, DSL parser extensions, dependency resolution, and local storage. Registry backend is assumed to be already implemented." + +## Overview + +This specification covers the **Nextflow client-side implementation** of the module system, enabling pipeline developers to: +- Include remote modules from the Nextflow registry using `@scope/name` syntax +- Manage module versions through `nextflow.config` +- Use CLI commands to install, search, list, remove, freeze, publish, and run modules +- Configure tool arguments through structured `meta.yaml` definitions + +**Out of Scope**: Registry backend implementation (assumed already available at `registry.nextflow.io`) + +## User Scenarios & Testing + +### User Story 1 - Install and Use Registry Module (Priority: P1) + +A pipeline developer wants to use a pre-built module from the Nextflow registry in their workflow without manually downloading or managing module files. + +**Why this priority**: This is the core value proposition - enabling code reuse from the ecosystem. Without this, the module system provides no benefit. + +**Independent Test**: Can be fully tested by running `nextflow module install nf-core/fastqc` and then executing a workflow that includes the module. Delivers immediate value by enabling module consumption. + +**Acceptance Scenarios**: + +1. **Given** a new Nextflow project with no modules installed, **When** user runs `nextflow module install nf-core/fastqc`, **Then** the module is downloaded to `modules/@nf-core/fastqc/`, a `.checksum` file is created, and `nextflow.config` is updated with the version +2. **Given** a workflow file with `include { FASTQC } from '@nf-core/fastqc'`, **When** user runs `nextflow run main.nf`, **Then** Nextflow resolves the module from local storage and executes the process +3. **Given** a module version declared in `nextflow.config`, **When** user includes the module, **Then** the declared version is used (not latest) +4. **Given** a module with transitive dependencies in `meta.yaml`, **When** user installs the module, **Then** all transitive dependencies are also installed + +--- + +### User Story 2 - Run Module Directly (Priority: P1) + +A user wants to run a module directly from the command line without writing a wrapper workflow. + +**Why this priority**: Enables immediate productivity - users can test and execute modules without boilerplate code, essential for AI agents and quick experimentation. + +**Independent Test**: Can be tested by running `nextflow module run nf-core/fastqc --input 'data/*.fq'` and verifying the process executes. + +**Acceptance Scenarios**: + +1. **Given** a module is available (locally or in registry), **When** user runs `nextflow module run nf-core/fastqc --input 'data/*.fastq'`, **Then** the module is executed with the provided inputs mapped to process parameters +2. **Given** a module with tool arguments defined in `meta.yaml`, **When** user runs `nextflow module run nf-core/bwa-align --tools:bwa:K 100000`, **Then** the tool argument is validated and passed to the process +3. **Given** a module is not installed locally, **When** user runs `nextflow module run nf-core/salmon`, **Then** the module is automatically downloaded before execution + +--- + +### User Story 3 - Structured Tool Arguments (Priority: P1) + +A module author wants to define typed, documented tool arguments that replace the legacy `ext.args` pattern. + +**Why this priority**: Critical for module usability - provides type-safe, documented arguments that enable IDE autocompletion and validation, replacing the opaque `ext.args` pattern. + +**Independent Test**: Can be tested by configuring `tools.bwa.args.K = 100000` in config and verifying the argument is applied in the script. + +**Acceptance Scenarios**: + +1. **Given** a module with `tools.*.args` defined in `meta.yaml`, **When** user configures `tools.bwa.args.K = 100000` in config, **Then** the argument is accessible in scripts as `tools.bwa.args.K` returning `-K 100000` +2. **Given** all tool arguments are configured, **When** script uses `${tools.bwa.args}`, **Then** all configured arguments are concatenated in the output +3. **Given** an argument with enum validation, **When** user provides an invalid value, **Then** a validation error is displayed + +--- + +### User Story 4 - Module Version Management (Priority: P2) + +A pipeline developer wants to pin and manage module versions to ensure reproducible workflow executions. + +**Why this priority**: Reproducibility is important for scientific workflows - version pinning ensures consistent results. + +**Independent Test**: Can be tested by modifying `nextflow.config` module versions and verifying the correct version is used on workflow run. + +**Acceptance Scenarios**: + +1. **Given** a module is installed at version 1.0.0, **When** user changes `nextflow.config` to specify version 1.1.0 and runs the workflow, **Then** version 1.1.0 is automatically downloaded and replaces the local copy +2. **Given** modules with transitive dependencies, **When** user runs `nextflow module freeze`, **Then** all transitive dependencies are pinned to exact versions in `nextflow.config` +3. **Given** modules installed locally, **When** user runs `nextflow module list`, **Then** configured version, installed version, latest available version, and status are displayed for each module + +--- + +### User Story 5 - Module Integrity Protection (Priority: P2) + +A pipeline developer who has locally modified a module (for debugging or customization) wants to be protected from accidentally losing those changes. + +**Why this priority**: Protects user work - important for developer experience but not blocking core functionality. + +**Independent Test**: Can be tested by modifying a module's `main.nf` locally, then attempting to install a different version and verifying the warning appears. + +**Acceptance Scenarios**: + +1. **Given** a locally modified module (checksum mismatch with `.checksum`), **When** user tries to install a different version, **Then** Nextflow warns about local modifications and does NOT override +2. **Given** a locally modified module, **When** user runs `nextflow module install -force`, **Then** the local module is replaced with the registry version +3. **Given** a locally modified module, **When** user runs the workflow, **Then** a warning is displayed about checksum mismatch but execution continues + +--- + +### User Story 6 - Remove Module (Priority: P3) + +A pipeline developer wants to remove a module they no longer need. + +**Why this priority**: Housekeeping feature - useful but not blocking core workflows. + +**Independent Test**: Can be tested by running `nextflow module remove nf-core/fastqc` and verifying files are deleted and config is updated. + +**Acceptance Scenarios**: + +1. **Given** a module is installed, **When** user runs `nextflow module remove nf-core/fastqc`, **Then** the module directory is deleted and the entry is removed from `nextflow.config` +2. **Given** a module is referenced in workflow files, **When** user runs `nextflow module remove`, **Then** a warning is displayed about the reference but removal proceeds +3. **Given** orphaned transitive dependencies exist, **When** user removes a module, **Then** orphaned dependencies are identified and user is prompted to remove them + +--- + +### User Story 7 - Search and Discover Modules (Priority: P3) + +A pipeline developer wants to find available modules in the registry that match their analysis needs. + +**Why this priority**: Discovery feature - useful but users can find modules through documentation or registry web UI. + +**Independent Test**: Can be tested by running `nextflow module search bwa` and verifying results are displayed with name, version, and description. + +**Acceptance Scenarios**: + +1. **Given** modules exist in the registry, **When** user runs `nextflow module search alignment`, **Then** matching modules are displayed with name, latest version, description, and download count +2. **Given** user wants JSON output for scripting, **When** user runs `nextflow module search fastqc -json`, **Then** results are returned in parseable JSON format +3. **Given** many results exist, **When** user runs `nextflow module search quality -limit 5`, **Then** only 5 results are returned + +--- + +### User Story 8 - Publish Module to Registry (Priority: P3) + +A module author wants to publish their module to the Nextflow registry for others to use. + +**Why this priority**: Ecosystem contribution feature - important for growth but users can consume modules without publishing capability. + +**Independent Test**: Can be tested by creating a valid module structure and running `nextflow module publish -dry-run` to validate. + +**Acceptance Scenarios**: + +1. **Given** a valid module with `main.nf`, `meta.yaml`, and `README.md`, **When** user runs `nextflow module publish myorg/my-module`, **Then** the module is uploaded to the registry and becomes available for installation +2. **Given** an invalid module (missing required fields), **When** user runs `nextflow module publish`, **Then** validation errors are displayed listing the missing requirements +3. **Given** no authentication configured, **When** user runs `nextflow module publish`, **Then** a clear error message indicates authentication is required + +--- + +### Edge Cases + +- What happens when the registry is unreachable during module resolution? + - Nextflow uses locally cached modules if available, otherwise fails with a clear network error +- How does the system handle circular module dependencies? + - Dependency resolver detects cycles and fails with an error listing the cycle +- What happens when two modules require incompatible versions of the same dependency? + - Version conflict is reported with the conflicting requirements +- How are modules resolved when multiple registries are configured? + - Registries are tried in order; first match wins +- What happens when `meta.yaml` is missing from a module? + - Module is treated as having no dependencies; basic functionality works +- What happens when local module directory is corrupted or incomplete? + - Checksum mismatch triggers warning; `-force` allows re-download + +## Requirements + +### Functional Requirements + +#### DSL Parser Extension + +- **FR-001**: System MUST recognize `@scope/name` syntax in `include` statements as registry module references +- **FR-002**: System MUST distinguish between local file paths (starting with `.` or `/`) and registry modules (starting with `@`) +- **FR-003**: System MUST resolve module versions from `nextflow.config` `modules {}` block before downloading +- **FR-004**: System MUST parse and validate `meta.yaml` files for module metadata and dependencies + +#### Module Resolution + +- **FR-005**: System MUST resolve modules at workflow parse time (after plugin resolution) +- **FR-006**: System MUST check local `modules/@scope/name/` directory before querying registry +- **FR-007**: System MUST verify module integrity using `.checksum` file on every run +- **FR-008**: System MUST download modules from registry when not present locally or when version differs +- **FR-009**: System MUST NOT override locally modified modules (checksum mismatch) unless `-force` is used +- **FR-010**: System MUST recursively resolve transitive dependencies from `meta.yaml` +- **FR-011**: System MUST warn when transitive dependencies are not pinned in `nextflow.config` + +#### Local Storage + +- **FR-012**: System MUST store modules in `modules/@scope/name/` directory structure (single version per module) +- **FR-013**: System MUST create `.checksum` file from registry's X-Checksum header on download +- **FR-014**: System MUST store module's `main.nf`, `meta.yaml`, and supporting files in the module directory + +#### CLI Commands + +- **FR-015**: System MUST provide `nextflow module install [scope/name]` command to download modules +- **FR-016**: System MUST provide `nextflow module search ` command to search the registry +- **FR-017**: System MUST provide `nextflow module list` command to show installed vs configured modules +- **FR-018**: System MUST provide `nextflow module remove scope/name` command to delete modules +- **FR-019**: System MUST provide `nextflow module freeze` command to pin all transitive dependencies +- **FR-020**: System MUST provide `nextflow module publish scope/name` command to upload modules to registry +- **FR-021**: System MUST provide `nextflow module run scope/name` command to execute modules directly + +#### Configuration + +- **FR-022**: System MUST read module versions from `modules {}` block in `nextflow.config` +- **FR-023**: System MUST support `registry {}` block for configuring registry URL and authentication +- **FR-024**: System MUST support `NXF_REGISTRY_TOKEN` environment variable for authentication +- **FR-025**: System MUST support multiple registry URLs with fallback ordering + +#### Tool Arguments + +- **FR-026**: System MUST provide `tools..args.` implicit variable in module scripts +- **FR-027**: System MUST validate tool arguments against `meta.yaml` schema (type, enum) +- **FR-028**: System MUST support boolean, integer, float, string, file, and path argument types +- **FR-029**: System MUST concatenate all tool arguments when `tools..args` is accessed + +#### Registry Communication + +- **FR-030**: System MUST communicate with registry via documented Module API endpoints +- **FR-031**: System MUST handle authentication using Bearer token in Authorization header +- **FR-032**: System MUST verify SHA-256 checksum on module download + +### Key Entities + +- **Module**: A reusable Nextflow process definition with `main.nf` entry point, optional `meta.yaml` manifest, and README documentation +- **Module Reference**: A scoped identifier (`@scope/name`) pointing to a registry module +- **Module Manifest (meta.yaml)**: YAML file containing module metadata, version, dependencies, tool arguments schema +- **Checksum File (.checksum)**: Local cache of registry checksum for integrity verification +- **Registry Configuration**: Settings for registry URL, authentication, and fallback ordering + +## Success Criteria + +### Measurable Outcomes + +- **SC-001**: Pipeline developers can install and use a registry module within 5 minutes of starting a new project +- **SC-002**: Module resolution adds less than 2 seconds to workflow startup time when modules are cached locally +- **SC-003**: Users can successfully search, install, and run any module from the registry without reading documentation +- **SC-004**: 100% of module version changes in `nextflow.config` result in automatic module updates without manual intervention +- **SC-005**: Users receive clear, actionable error messages for all failure scenarios (network, validation, authentication) +- **SC-006**: Module authors can publish a new module version within 3 minutes using the CLI +- **SC-007**: Locally modified modules are never accidentally overwritten during normal operations +- **SC-008**: All transitive dependencies can be pinned with a single `nextflow module freeze` command + +## Assumptions + +- Registry backend is fully implemented and available at `registry.nextflow.io` with the Module API as documented in the ADR +- Existing plugin authentication system can be reused for module registry authentication +- Module bundle size limit of 1MB (uncompressed) is enforced by the registry +- Network connectivity is available for initial module downloads; offline operation uses local cache only +- The `modules/` directory is intended to be committed to the pipeline's git repository +- Version constraints in `meta.yaml` follow the same syntax as existing Nextflow plugin version constraints +- SHA-256 is used for all checksum operations +- Tool arguments CLI syntax uses colon-separated format: `--tools::` + +## Dependencies + +- Registry backend API (Module API endpoints as specified in ADR) +- Existing Nextflow plugin system (for authentication reuse) +- Existing DSL parser infrastructure (for `include` statement extension) +- Existing config parser (for `modules {}` and `registry {}` blocks) \ No newline at end of file From 68b202f6e82bc1fb67dcf277c2c400748e11e61e Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 15 Jan 2026 16:50:35 +0100 Subject: [PATCH 23/75] Remove transitive dependency resolution from module system [ci skip] - Remove `freeze` command from CLI - Remove transitive dependency install behavior - Remove orphaned transitive dependency removal from `remove` command - Update rationale and consequences sections - Simplify dependency resolution flow - Update ADR to version 2.4 Co-Authored-By: Claude Opus 4.5 Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 64 +++++++----------------------- specs/251117-module-system/spec.md | 13 ++---- 2 files changed, 18 insertions(+), 59 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index f7c2969aa6..01b7c38bec 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -4,10 +4,15 @@ - Status: draft - Date: 2025-01-06 - Tags: modules, dsl, registry, versioning, architecture -- Version: 2.3 +- Version: 2.4 ## Updates +### Version 2.4 (2026-01-15) +- **Removed transitive dependency resolution**: Module dependencies are explicit only; no automatic transitive resolution +- **Removed `freeze` command**: No longer needed without transitive dependency management +- **Simplified model**: Each module explicitly declares its dependencies in `nextflow.config` + ### Version 2.3 (2026-01-15) - **Resolution Rules table**: Added clear table specifying behavior for each combination of local state and declared version - **Local modification protection**: Locally modified modules (checksum mismatch) are NOT overridden unless `-force` flag is used @@ -38,7 +43,7 @@ Implement a module system with four core capabilities: 1. **Remote module inclusion** via registry 2. **Semantic versioning** with dependency resolution 3. **Unified Nextflow Registry** (rebrand existing Nextflow registry) -4. **First-class CLI support** (install, publish, search, list, remove, freeze, run) +4. **First-class CLI support** (install, publish, search, list, remove, run) ## Core Capabilities @@ -62,7 +67,6 @@ include { MY_PROCESS } from './modules/my-process.nf' 2. Check local `modules/@scope/name/` exists 3. Verify integrity against `.checksum` file 4. Apply resolution rules (see below) -5. Warn if transitive dependencies are not pinned **Resolution Rules**: @@ -153,10 +157,8 @@ npm-style `^` and `~` notation while maintaining consistency with existing Nextf This avoids introducing new notation that would require additional parser support. **Dependency Resolution**: -- Workflow's `nextflow.config` specifies exact versions for direct dependencies -- Transitive dependencies resolved from each module's `meta.yaml` using version constraints -- Warn if transitive dependencies are not pinned in workflow config -- Use `nextflow module freeze` to pin all transitive dependencies to exact versions +- Workflow's `nextflow.config` specifies exact versions for dependencies +- Module dependencies declared in `meta.yaml` using version constraints ### 3. Unified Nextflow Registry @@ -203,7 +205,6 @@ Note: The `{name}` parameter includes the namespace prefix (e.g., "nf-core/fastq nextflow module run scope/name # Run a module directly without a wrapper script nextflow module search # Search registry nextflow module install [scope/name] # Install all from config, or specific module -nextflow module freeze # Pin all transitive dependencies to config nextflow module list # Show installed vs configured nextflow module remove scope/name # Remove from config + local cache nextflow module publish scope/name # Publish to registry (requires api key) @@ -312,8 +313,7 @@ Download and install modules to the local `modules/` directory. When called with 5. Downloads the module archive from the registry 6. Extracts to `modules/@scope/name/` directory 7. Stores `.checksum` file from registry's X-Checksum response header -8. Recursively installs transitive dependencies declared in `meta.yaml` -9. Updates `nextflow.config` if installing a new module not already configured +8. Updates `nextflow.config` if installing a new module not already configured **Example**: ```bash @@ -324,31 +324,6 @@ nextflow module install nf-core/salmon -version 1.2.0 --- -#### `nextflow module freeze` - -Pin all transitive module dependencies to exact versions in `nextflow.config`. This ensures reproducible builds by capturing the precise versions of all dependencies. - -**Behavior**: -1. Scans the `modules/` directory for all installed modules -2. Reads version from each module's `meta.yaml` -3. Adds transitive dependencies not explicitly declared to `nextflow.config` - -**Output** (in `nextflow.config`): -```groovy -modules { - '@nf-core/bwa-align' = '1.2.4' - '@nf-core/samtools/view' = '1.0.5' // Transitive dependency - '@nf-core/samtools/sort' = '2.1.0' // Transitive dependency -} -``` - -**Example**: -```bash -nextflow module freeze # Pin all transitive dependencies -``` - ---- - #### `nextflow module list` Display the status of all modules, comparing what is configured in `nextflow.config` against what is actually installed in the `modules/` directory. @@ -374,7 +349,7 @@ nextflow module list -outdated #### `nextflow module remove scope/name` -Remove a module from both the local `modules/` directory and the `nextflow.config` configuration. Also removes orphaned transitive dependencies that are no longer required by other modules. +Remove a module from both the local `modules/` directory and the `nextflow.config` configuration. **Arguments**: - `scope/name`: Module identifier to remove (required) @@ -386,8 +361,7 @@ Remove a module from both the local `modules/` directory and the `nextflow.confi **Behavior**: 1. Removes the module directory from `modules/@scope/name/` 2. Removes the module entry from the `modules {}` block in `nextflow.config` -3. Identifies and optionally removes orphaned transitive dependencies -4. Warns if the module is still referenced in workflow files +3. Warns if the module is still referenced in workflow files **Example**: ```bash @@ -432,7 +406,6 @@ nextflow module publish myorg/my-process -dry-run **General Notes**: - All commands respect the `registry.url` configuration for custom registries -- Unpinned transitive dependencies generate warnings during resolution - Modules are automatically downloaded on `nextflow run` if missing but configured ## Module Structure @@ -518,10 +491,8 @@ project-root/ - Exists, checksum valid, same version → use local - Exists, checksum valid, different version → replace with declared version - Exists, checksum mismatch → warn and do NOT override (local changes detected) - e. Warn if module not pinned in config (especially transitive dependencies) 3. On download: store module to `modules/@scope/name/` with `.checksum` file -4. Read module's `meta.yaml` → resolve transitive dependencies recursively -5. Parse module's `main.nf` file → make processes available +4. Parse module's `main.nf` file → make processes available **Security**: - SHA-256 checksum verification on download (stored in `.checksum` file) @@ -652,8 +623,6 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ **Why versions in nextflow.config instead of separate lock file?** - Single source of truth for workflow dependencies - Simple: exact versions in config, no separate lock file to manage -- Transitive dependencies resolved from module's meta.yaml with version constraints -- Use `nextflow module freeze` to pin all transitive dependencies when needed - Reproducibility via explicit version pinning in config **Why parse-time resolution?** @@ -670,7 +639,6 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ **Why semantic versioning?** - Clear compatibility guarantees -- Enables automated dependency resolution for transitive dependencies - Industry standard (npm, cargo, Go modules) ## Consequences @@ -682,7 +650,7 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ - Minimal operational overhead (single registry for both plugins and modules) - NPM-style scoping enables organization namespaces and private registries - Local `modules/` directory provides project isolation -- Simple config model: no separate lock file unless using `freeze` +- Simple config model: no separate lock file - Simple module structure: each module has single `main.nf` entry point **Negative**: @@ -690,7 +658,6 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ - Type-specific handling adds registry complexity - Parse-time resolution adds latency to workflow startup - Local `modules/` directory duplicates storage across projects (unlike global cache) -- Warnings for unpinned transitive dependencies may be noisy initially **Neutral**: - Modules and plugins conceptually distinct but share infrastructure @@ -853,8 +820,7 @@ requires: **Resolution:** 1. The resolver looks up dependencies locally first, then in configured registries -2. Version constraints are resolved transitively -3. Pinned versions are recorded in `nextflow.config` for reproducibility +2. Pinned versions are recorded in `nextflow.config` for reproducibility #### `tools` diff --git a/specs/251117-module-system/spec.md b/specs/251117-module-system/spec.md index e489d0a8ef..4a538a765c 100644 --- a/specs/251117-module-system/spec.md +++ b/specs/251117-module-system/spec.md @@ -30,7 +30,6 @@ A pipeline developer wants to use a pre-built module from the Nextflow registry 1. **Given** a new Nextflow project with no modules installed, **When** user runs `nextflow module install nf-core/fastqc`, **Then** the module is downloaded to `modules/@nf-core/fastqc/`, a `.checksum` file is created, and `nextflow.config` is updated with the version 2. **Given** a workflow file with `include { FASTQC } from '@nf-core/fastqc'`, **When** user runs `nextflow run main.nf`, **Then** Nextflow resolves the module from local storage and executes the process 3. **Given** a module version declared in `nextflow.config`, **When** user includes the module, **Then** the declared version is used (not latest) -4. **Given** a module with transitive dependencies in `meta.yaml`, **When** user installs the module, **Then** all transitive dependencies are also installed --- @@ -77,8 +76,7 @@ A pipeline developer wants to pin and manage module versions to ensure reproduci **Acceptance Scenarios**: 1. **Given** a module is installed at version 1.0.0, **When** user changes `nextflow.config` to specify version 1.1.0 and runs the workflow, **Then** version 1.1.0 is automatically downloaded and replaces the local copy -2. **Given** modules with transitive dependencies, **When** user runs `nextflow module freeze`, **Then** all transitive dependencies are pinned to exact versions in `nextflow.config` -3. **Given** modules installed locally, **When** user runs `nextflow module list`, **Then** configured version, installed version, latest available version, and status are displayed for each module +2. **Given** modules installed locally, **When** user runs `nextflow module list`, **Then** configured version, installed version, latest available version, and status are displayed for each module --- @@ -110,7 +108,6 @@ A pipeline developer wants to remove a module they no longer need. 1. **Given** a module is installed, **When** user runs `nextflow module remove nf-core/fastqc`, **Then** the module directory is deleted and the entry is removed from `nextflow.config` 2. **Given** a module is referenced in workflow files, **When** user runs `nextflow module remove`, **Then** a warning is displayed about the reference but removal proceeds -3. **Given** orphaned transitive dependencies exist, **When** user removes a module, **Then** orphaned dependencies are identified and user is prompted to remove them --- @@ -179,8 +176,6 @@ A module author wants to publish their module to the Nextflow registry for other - **FR-007**: System MUST verify module integrity using `.checksum` file on every run - **FR-008**: System MUST download modules from registry when not present locally or when version differs - **FR-009**: System MUST NOT override locally modified modules (checksum mismatch) unless `-force` is used -- **FR-010**: System MUST recursively resolve transitive dependencies from `meta.yaml` -- **FR-011**: System MUST warn when transitive dependencies are not pinned in `nextflow.config` #### Local Storage @@ -194,9 +189,8 @@ A module author wants to publish their module to the Nextflow registry for other - **FR-016**: System MUST provide `nextflow module search ` command to search the registry - **FR-017**: System MUST provide `nextflow module list` command to show installed vs configured modules - **FR-018**: System MUST provide `nextflow module remove scope/name` command to delete modules -- **FR-019**: System MUST provide `nextflow module freeze` command to pin all transitive dependencies -- **FR-020**: System MUST provide `nextflow module publish scope/name` command to upload modules to registry -- **FR-021**: System MUST provide `nextflow module run scope/name` command to execute modules directly +- **FR-019**: System MUST provide `nextflow module publish scope/name` command to upload modules to registry +- **FR-020**: System MUST provide `nextflow module run scope/name` command to execute modules directly #### Configuration @@ -237,7 +231,6 @@ A module author wants to publish their module to the Nextflow registry for other - **SC-005**: Users receive clear, actionable error messages for all failure scenarios (network, validation, authentication) - **SC-006**: Module authors can publish a new module version within 3 minutes using the CLI - **SC-007**: Locally modified modules are never accidentally overwritten during normal operations -- **SC-008**: All transitive dependencies can be pinned with a single `nextflow module freeze` command ## Assumptions From d118d5adfa567737cfda85de7cb63f95fa0e5b78 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 21 Jan 2026 12:23:21 +0100 Subject: [PATCH 24/75] Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 01b7c38bec..963f995a23 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -492,7 +492,10 @@ project-root/ - Exists, checksum valid, different version → replace with declared version - Exists, checksum mismatch → warn and do NOT override (local changes detected) 3. On download: store module to `modules/@scope/name/` with `.checksum` file -4. Parse module's `main.nf` file → make processes available +4. Read `meta.yaml` file: + a. Validates Nextflow requirement → Fail if not fulfilled + b. Load Pluign requirements if not exist. +5. Parse module's `main.nf` file → make processes available``` **Security**: - SHA-256 checksum verification on download (stored in `.checksum` file) From ffe11163037a1898bc6dcddfffebcba35942df70 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 21 Jan 2026 12:24:03 +0100 Subject: [PATCH 25/75] Update adr/20251114-module-system.md [ci skip] Co-authored-by: Phil Ewels Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 963f995a23..7001576715 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -538,7 +538,8 @@ bwa mem $args -t $task.cpus $index $reads | samtools sort $args2 -o out.bam - ### New Pattern: Structured Tool Arguments -Modules declare available arguments in `meta.yaml` under each tool's `args` property: +Modules declare available arguments in `meta.yaml` under each tool's `args` property. +This list does _not_ need to be exhaustive. It should include any arguments known to be used by pipelines or that could be expected to be used by users. However, arguments can still be specified in the config even if not defined in this file, so absence does not prevent use. ```yaml tools: From 806a41f3b1ff3027d80845a57cb554819a62079e Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Mon, 19 Jan 2026 16:38:45 +0100 Subject: [PATCH 26/75] Update modules plan [ci skip] Signed-off-by: Paolo Di Tommaso --- .../checklists/requirements.md | 2 +- .../contracts/registry-api.yaml | 388 ++++++++++++++++++ specs/251117-module-system/data-model.md | 314 ++++++++++++++ specs/251117-module-system/plan.md | 101 +++++ specs/251117-module-system/quickstart.md | 315 ++++++++++++++ specs/251117-module-system/research.md | 360 ++++++++++++++++ specs/251117-module-system/spec.md | 54 +-- 7 files changed, 1510 insertions(+), 24 deletions(-) create mode 100644 specs/251117-module-system/contracts/registry-api.yaml create mode 100644 specs/251117-module-system/data-model.md create mode 100644 specs/251117-module-system/plan.md create mode 100644 specs/251117-module-system/quickstart.md create mode 100644 specs/251117-module-system/research.md diff --git a/specs/251117-module-system/checklists/requirements.md b/specs/251117-module-system/checklists/requirements.md index e01762d921..fe90e53ec3 100644 --- a/specs/251117-module-system/checklists/requirements.md +++ b/specs/251117-module-system/checklists/requirements.md @@ -33,7 +33,7 @@ - Specification is complete and ready for `/speckit.plan` - All 8 user stories have clear acceptance scenarios -- 32 functional requirements defined across 6 categories +- 29 functional requirements defined across 6 categories - 8 success criteria defined with measurable outcomes - Edge cases documented for error handling scenarios - Registry backend is explicitly out of scope (assumed implemented) \ No newline at end of file diff --git a/specs/251117-module-system/contracts/registry-api.yaml b/specs/251117-module-system/contracts/registry-api.yaml new file mode 100644 index 0000000000..2541e12c02 --- /dev/null +++ b/specs/251117-module-system/contracts/registry-api.yaml @@ -0,0 +1,388 @@ +openapi: 3.0.3 +info: + title: Nextflow Module Registry API + description: | + API specification for the Nextflow Module Registry. + This documents the endpoints that the Nextflow module system client consumes. + The registry backend is assumed to be already implemented at registry.nextflow.io. + version: 1.0.0 + contact: + name: Nextflow Team + url: https://nextflow.io + +servers: + - url: https://registry.nextflow.io/api + description: Production registry + +security: + - BearerAuth: [] + +paths: + /modules: + get: + operationId: searchModules + summary: Search modules + description: Search for modules by query text (semantic search across name, description, keywords) + tags: + - Modules + parameters: + - name: query + in: query + required: true + description: Search query text + schema: + type: string + example: "alignment" + - name: limit + in: query + required: false + description: Maximum number of results + schema: + type: integer + default: 10 + maximum: 100 + responses: + '200': + description: Search results + content: + application/json: + schema: + type: object + properties: + modules: + type: array + items: + $ref: '#/components/schemas/ModuleSummary' + total: + type: integer + description: Total matching modules + '400': + description: Invalid query parameters + + /modules/{name}: + get: + operationId: getModule + summary: Get module details + description: Get module metadata including latest release information + tags: + - Modules + parameters: + - name: name + in: path + required: true + description: Module name including scope (e.g., nf-core/fastqc) + schema: + type: string + example: "nf-core/fastqc" + responses: + '200': + description: Module details + content: + application/json: + schema: + $ref: '#/components/schemas/ModuleDetails' + '404': + description: Module not found + + post: + operationId: publishModule + summary: Publish module version + description: Upload and publish a new module version (requires authentication) + tags: + - Modules + security: + - BearerAuth: [] + parameters: + - name: name + in: path + required: true + description: Module name including scope + schema: + type: string + requestBody: + required: true + content: + multipart/form-data: + schema: + type: object + required: + - bundle + properties: + bundle: + type: string + format: binary + description: Module bundle (tar.gz archive) + tags: + type: array + items: + type: string + description: Additional tags for discoverability + responses: + '201': + description: Module published successfully + content: + application/json: + schema: + $ref: '#/components/schemas/PublishResult' + '400': + description: Invalid module bundle or manifest + content: + application/json: + schema: + $ref: '#/components/schemas/ValidationError' + '401': + description: Authentication required + '403': + description: Insufficient permissions for scope + + /modules/{name}/releases: + get: + operationId: listReleases + summary: List module releases + description: Get all available versions of a module + tags: + - Modules + parameters: + - name: name + in: path + required: true + schema: + type: string + responses: + '200': + description: List of releases + content: + application/json: + schema: + type: object + properties: + releases: + type: array + items: + $ref: '#/components/schemas/ReleaseInfo' + '404': + description: Module not found + + /modules/{name}/{version}: + get: + operationId: getRelease + summary: Get specific release + description: Get metadata for a specific module version + tags: + - Modules + parameters: + - name: name + in: path + required: true + schema: + type: string + - name: version + in: path + required: true + description: Semantic version (e.g., 1.0.0) + schema: + type: string + example: "1.0.0" + responses: + '200': + description: Release details + content: + application/json: + schema: + $ref: '#/components/schemas/ReleaseInfo' + '404': + description: Module or version not found + + /modules/{name}/{version}/download: + get: + operationId: downloadModule + summary: Download module bundle + description: Download the module source archive for a specific version + tags: + - Modules + parameters: + - name: name + in: path + required: true + schema: + type: string + - name: version + in: path + required: true + schema: + type: string + responses: + '200': + description: Module bundle + headers: + X-Checksum: + description: SHA-256 checksum of the bundle + schema: + type: string + example: "sha256:abc123..." + Content-Disposition: + description: Suggested filename + schema: + type: string + example: "attachment; filename=nf-core-fastqc-1.0.0.tar.gz" + content: + application/gzip: + schema: + type: string + format: binary + '404': + description: Module or version not found + +components: + securitySchemes: + BearerAuth: + type: http + scheme: bearer + description: | + Authentication token. Can be provided via: + - NXF_REGISTRY_TOKEN environment variable + - registry.auth config block + + schemas: + ModuleSummary: + type: object + required: + - name + - latestVersion + - description + properties: + name: + type: string + description: Full module name with scope + example: "nf-core/fastqc" + latestVersion: + type: string + description: Latest available version + example: "1.2.0" + description: + type: string + description: Short module description + example: "Run FastQC on sequenced reads" + downloadCount: + type: integer + description: Total download count + example: 15420 + keywords: + type: array + items: + type: string + example: ["quality control", "fastq"] + + ModuleDetails: + type: object + required: + - name + - latestVersion + - description + properties: + name: + type: string + example: "nf-core/fastqc" + latestVersion: + type: string + example: "1.2.0" + description: + type: string + authors: + type: array + items: + type: string + example: ["@drpatelh"] + maintainers: + type: array + items: + type: string + license: + type: string + example: "MIT" + keywords: + type: array + items: + type: string + downloadCount: + type: integer + createdAt: + type: string + format: date-time + updatedAt: + type: string + format: date-time + latestRelease: + $ref: '#/components/schemas/ReleaseInfo' + + ReleaseInfo: + type: object + required: + - version + - checksum + - publishedAt + properties: + version: + type: string + description: Semantic version + example: "1.2.0" + checksum: + type: string + description: SHA-256 checksum of bundle + example: "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + publishedAt: + type: string + format: date-time + size: + type: integer + description: Bundle size in bytes + example: 45678 + requires: + type: object + properties: + nextflow: + type: string + example: ">=24.04.0" + plugins: + type: array + items: + type: string + modules: + type: array + items: + type: string + + PublishResult: + type: object + properties: + name: + type: string + version: + type: string + checksum: + type: string + publishedAt: + type: string + format: date-time + downloadUrl: + type: string + format: uri + + ValidationError: + type: object + properties: + code: + type: string + enum: + - INVALID_MANIFEST + - MISSING_MAIN_NF + - MISSING_README + - INVALID_VERSION + - BUNDLE_TOO_LARGE + - DUPLICATE_VERSION + message: + type: string + details: + type: array + items: + type: string \ No newline at end of file diff --git a/specs/251117-module-system/data-model.md b/specs/251117-module-system/data-model.md new file mode 100644 index 0000000000..d9345e33a4 --- /dev/null +++ b/specs/251117-module-system/data-model.md @@ -0,0 +1,314 @@ +# Data Model: Nextflow Module System Client + +**Date**: 2026-01-19 +**Feature**: 251117-module-system + +## Overview + +This document defines the data entities, their attributes, relationships, and state transitions for the Nextflow module system client implementation. + +--- + +## Entity Definitions + +### 1. ModuleReference + +Represents a reference to a module in DSL `include` statements. + +```groovy +@CompileStatic +class ModuleReference { + String scope // e.g., "nf-core" + String name // e.g., "fastqc" + String fullName // e.g., "@nf-core/fastqc" + + static ModuleReference parse(String source) { + // Parses "@scope/name" format + } + + boolean isRegistryModule() { + return fullName.startsWith('@') + } +} +``` + +**Validation Rules**: +- `scope`: lowercase alphanumeric with hyphens, pattern `[a-z0-9][a-z0-9-]*` +- `name`: lowercase alphanumeric with underscores/hyphens, pattern `[a-z][a-z0-9_-]*` +- `fullName`: must match `^@[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$` + +--- + +### 2. ModuleManifest + +Parsed representation of `meta.yaml` file. + +```groovy +@CompileStatic +class ModuleManifest { + String name // e.g., "nf-core/fastqc" (without @) + String version // e.g., "1.0.0" + String description // Module description + List keywords // Discovery keywords + List authors // GitHub handles + List maintainers + String license // SPDX identifier + + ModuleRequirements requires + List tools + List input + Map output +} + +@CompileStatic +class ModuleRequirements { + String nextflow // Version constraint, e.g., ">=24.04.0" + List plugins // e.g., ["nf-amazon@2.0.0"] + List modules // e.g., ["nf-core/samtools@>=1.0.0"] + List workflows // e.g., ["nf-core/fastq-align@1.0.0"] +} + +@CompileStatic +class ToolDefinition { + String name // Tool identifier + String description + String homepage + String documentation + String doi + List license + String identifier // bio.tools identifier + Map args +} + +@CompileStatic +class ArgDefinition { + String flag // CLI flag, e.g., "-K" + String type // boolean, integer, float, string, file, path + String description + Object defaultValue + List enumValues + boolean required = false +} +``` + +**Validation Rules**: +- `version`: Must be valid SemVer (MAJOR.MINOR.PATCH) +- `type` in ArgDefinition: Must be one of: boolean, integer, float, string, file, path +- `enumValues`: If present, configured value must be in this list + +--- + +### 3. ModuleInfo + +Module metadata returned from registry API. + +```groovy +@CompileStatic +class ModuleInfo { + String name // e.g., "nf-core/fastqc" + String version // Specific version + String latestVersion // Latest available + String description + String checksum // SHA-256 of bundle + long downloadCount + Instant publishedAt + List versions // All available versions +} +``` + +--- + +### 4. InstalledModule + +Represents a module in local `modules/` directory. + +```groovy +@CompileStatic +class InstalledModule { + ModuleReference reference + Path directory // e.g., /project/modules/@nf-core/fastqc + Path mainFile // e.g., /project/modules/@nf-core/fastqc/main.nf + Path manifestFile // e.g., /project/modules/@nf-core/fastqc/meta.yaml + Path checksumFile // e.g., /project/modules/@nf-core/fastqc/.checksum + String installedVersion + String expectedChecksum + ModuleManifest manifest + + ModuleIntegrity getIntegrity() { + // Compute and compare checksum + } +} + +enum ModuleIntegrity { + VALID, // Checksum matches + MODIFIED, // Checksum mismatch (local changes) + MISSING_CHECKSUM, // No .checksum file + CORRUPTED // Missing required files +} +``` + +**State Transitions**: +``` +[NOT_INSTALLED] --install--> [VALID] +[VALID] --user edits--> [MODIFIED] +[MODIFIED] --install -force--> [VALID] +[VALID] --version change in config--> [VALID] (replaced) +[MODIFIED] --version change in config--> [MODIFIED] (blocked, warn) +``` + +--- + +### 5. ModuleConfig + +Module configuration from `nextflow.config`. + +```groovy +@CompileStatic +class ModuleConfig { + Map modules = [:] // name -> version + RegistryConfig registry +} + +@CompileStatic +class RegistryConfig { + String url = 'https://registry.nextflow.io' + List urls = [] // Multiple registries + Map auth = [:] // registry -> token expression +} +``` + +**Config Syntax**: +```groovy +modules { + '@nf-core/fastqc' = '1.0.0' + '@nf-core/bwa-align' = '1.2.0' +} + +registry { + url = 'https://registry.nextflow.io' + auth { + 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' + } +} +``` + +--- + +### 6. ToolArgsContext + +Runtime context for tool arguments in process scripts. + +```groovy +@CompileStatic +class ToolArgsContext { + private Map tools = [:] + + ToolArgs getAt(String toolName) { + return tools[toolName] + } +} + +@CompileStatic +class ToolArgs { + private Map schema + private Map values + + String getAt(String argName) { + def def = schema[argName] + def value = values[argName] + if (value == null) return '' + if (def.type == 'boolean') { + return value ? def.flag : '' + } + return "${def.flag} ${value}" + } + + String toString() { + schema.keySet() + .findAll { values.containsKey(it) && values[it] != null } + .collect { this[it] } + .findAll { it } + .join(' ') + } +} +``` + +--- + +### 7. ModuleResolutionResult + +Result of module resolution process. + +```groovy +@CompileStatic +class ModuleResolutionResult { + ModuleReference reference + Path resolvedPath // Absolute path to main.nf + ResolutionAction action + String message // Warning/info message if any + ModuleManifest manifest +} + +enum ResolutionAction { + USE_LOCAL, // Used existing local module + DOWNLOADED, // Downloaded from registry + REPLACED, // Replaced local with different version + BLOCKED_MODIFIED, // Local modified, not replaced (warning issued) + FAILED // Resolution failed (error) +} +``` + +--- + +## Relationships + +``` +ModuleConfig (1) -----> (*) ModuleReference + | + v +RegistryConfig (1) -----> (*) Registry URLs + +ModuleReference (1) -----> (0..1) InstalledModule + | + v (via registry) +ModuleInfo (1) -----> (1) ModuleManifest + +InstalledModule (1) -----> (1) ModuleManifest + -----> (*) ToolDefinition + -----> (*) ArgDefinition + +ToolArgsContext (1) -----> (*) ToolArgs + -----> (*) ArgDefinition (schema) +``` + +--- + +## Storage Layout + +``` +project-root/ +├── nextflow.config # modules{}, registry{} blocks +├── main.nf # include { X } from '@scope/name' +└── modules/ + └── @scope/ + └── name/ + ├── .checksum # SHA-256 from registry + ├── main.nf # Entry point (required) + ├── meta.yaml # Manifest (optional but recommended) + ├── README.md # Documentation + └── [other files] # Supporting files +``` + +--- + +## Validation Summary + +| Entity | Field | Validation | +|--------|-------|------------| +| ModuleReference | fullName | Pattern: `^@[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$` | +| ModuleManifest | version | SemVer: `MAJOR.MINOR.PATCH` | +| ArgDefinition | type | Enum: boolean, integer, float, string, file, path | +| ArgDefinition | enumValues | If set, value must be member | +| InstalledModule | directory | Must contain main.nf | +| ModuleConfig | modules | Keys must be valid module references | +| RegistryConfig | url | Valid HTTPS URL | \ No newline at end of file diff --git a/specs/251117-module-system/plan.md b/specs/251117-module-system/plan.md new file mode 100644 index 0000000000..8fc08ac793 --- /dev/null +++ b/specs/251117-module-system/plan.md @@ -0,0 +1,101 @@ +# Implementation Plan: Nextflow Module System Client + +**Branch**: `251117-module-system` | **Date**: 2026-01-19 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `/specs/251117-module-system/spec.md` + +## Summary + +Implement client-side module system for Nextflow enabling pipeline developers to include remote modules from the Nextflow registry using `@scope/name` syntax, manage versions via `nextflow.config`, and use CLI commands (install, search, list, remove, publish, run). Implementation extends existing DSL parser, config parser, and follows plugin system patterns for registry communication and authentication. + +## Technical Context + +**Language/Version**: Groovy 4.0.29 (targeting Java 17 runtime, Java 21 toolchain for development) +**Primary Dependencies**: +- Existing Nextflow DSL parser (nf-lang module, ANTLR) +- Existing config parser (ConfigBuilder, ConfigParser) +- Existing HTTP client (HxClient from io.seqera.http) +- Existing plugin authentication infrastructure +**Storage**: Local filesystem (`modules/@scope/name/` per-project, `.checksum` files) +**Testing**: Spock Framework for unit tests, integration tests in `tests/` directory +**Target Platform**: JVM 17+ (same as Nextflow core) +**Project Type**: Multi-module Gradle project extension (core modules + CLI) +**Performance Goals**: Module resolution adds <2 seconds to workflow startup when cached locally (SC-002) +**Constraints**: +- Module bundle size limit: 1MB uncompressed (enforced by registry) +- Backward compatibility: Must not break existing `include` statements +- Offline operation: Must work with locally cached modules +**Scale/Scope**: Ecosystem-wide module distribution; typical project: 5-20 modules + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +| Principle | Status | Evidence | +|-----------|--------|----------| +| I. Modular Architecture | PASS | Module system client belongs in `modules/nextflow` (core CLI) with potential shared utilities in `nf-commons` | +| II. Test-Driven Quality | PASS | Unit tests (Spock), integration tests planned, smoke test support | +| III. Dataflow Programming Model | PASS | Modules are process definitions; include resolution at parse time preserves dataflow semantics | +| IV. Apache 2.0 License | PASS | All new code will include Apache 2.0 headers | +| V. DCO Sign-off | PASS | All commits will use `git commit -s` | +| VI. Semantic Versioning | PASS | Modules use SemVer; plugin-compatible version constraint syntax | +| VII. Groovy Idioms | PASS | Follow existing patterns from CmdPlugin, ConfigBuilder, HttpPluginRepository | + +**Gate Status**: PASS - No violations requiring justification + +## Project Structure + +### Documentation (this feature) + +```text +specs/251117-module-system/ +├── plan.md # This file +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +├── contracts/ # Phase 1 output (API contracts) +└── tasks.md # Phase 2 output (from /speckit.tasks) +``` + +### Source Code (repository root) + +```text +modules/nextflow/src/main/groovy/nextflow/ +├── cli/ +│ └── CmdModule.groovy # NEW: Module CLI command +├── config/ +│ ├── ConfigBuilder.groovy # MODIFY: Add modules/registry DSL +│ └── parser/v1/ +│ ├── ModulesDsl.groovy # NEW: modules {} block parser +│ └── RegistryDsl.groovy # NEW: registry {} block parser +└── module/ + ├── ModuleResolver.groovy # NEW: Core resolution logic + ├── ModuleStorage.groovy # NEW: Local storage management + ├── ModuleChecksum.groovy # NEW: Checksum verification + ├── ModuleManifest.groovy # NEW: meta.yaml parser + └── HttpModuleRepository.groovy # NEW: Registry HTTP client + +modules/nf-lang/src/main/java/nextflow/script/ +└── ResolveIncludeVisitor.java # MODIFY: Add @scope/name detection + +modules/nextflow/src/test/groovy/nextflow/ +├── cli/ +│ └── CmdModuleTest.groovy # NEW: CLI unit tests +├── config/ +│ └── ModulesDslTest.groovy # NEW: Config parsing tests +└── module/ + ├── ModuleResolverTest.groovy # NEW: Resolution logic tests + ├── ModuleStorageTest.groovy # NEW: Storage tests + └── ModuleChecksumTest.groovy # NEW: Checksum tests + +tests/ +└── modules/ # NEW: Integration tests + ├── install-module.nf # Test module install + include + ├── version-resolution.nf # Test version management + └── checksum-protection.nf # Test local modification protection +``` + +**Structure Decision**: Implementation extends existing Nextflow core modules following modular architecture. New code in `modules/nextflow` for CLI and core logic. DSL parser extension in `modules/nf-lang`. No new plugins required. + +## Complexity Tracking + +No constitution violations requiring justification. \ No newline at end of file diff --git a/specs/251117-module-system/quickstart.md b/specs/251117-module-system/quickstart.md new file mode 100644 index 0000000000..f9247dcb74 --- /dev/null +++ b/specs/251117-module-system/quickstart.md @@ -0,0 +1,315 @@ +# Quickstart: Nextflow Module System + +This guide covers the essential workflows for using the Nextflow module system. + +## Prerequisites + +- Nextflow 25.x or later (with module system support) +- Network connectivity for initial module downloads +- Optional: `NXF_REGISTRY_TOKEN` for publishing + +--- + +## 1. Install and Use a Module + +### Install a module + +```bash +# Install latest version +nextflow module install nf-core/fastqc + +# Install specific version +nextflow module install nf-core/fastqc -version 1.0.0 +``` + +This downloads the module to `modules/@nf-core/fastqc/` and updates `nextflow.config`. + +### Use in your workflow + +```groovy +// main.nf +include { FASTQC } from '@nf-core/fastqc' + +workflow { + reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz') + FASTQC(reads) +} +``` + +### Run your workflow + +```bash +nextflow run main.nf +``` + +--- + +## 2. Run a Module Directly + +Execute a module without writing a wrapper workflow: + +```bash +# Basic usage +nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' + +# With tool arguments +nextflow module run nf-core/bwa-align \ + --reads 'samples/*_{1,2}.fastq.gz' \ + --reference genome.fa \ + --tools:bwa:K 100000000 + +# With Nextflow options +nextflow module run nf-core/salmon \ + --reads reads.fq \ + --index salmon_index \ + -profile docker \ + -resume +``` + +--- + +## 3. Manage Module Versions + +### Configure versions in nextflow.config + +```groovy +// nextflow.config +modules { + '@nf-core/fastqc' = '1.0.0' + '@nf-core/bwa-align' = '1.2.0' + '@nf-core/samtools' = '2.1.0' +} +``` + +### Check module status + +```bash +# List all modules +nextflow module list + +# Output: +# MODULE CONFIGURED INSTALLED LATEST STATUS +# @nf-core/fastqc 1.0.0 1.0.0 1.2.0 outdated +# @nf-core/bwa-align 1.2.0 1.2.0 1.2.0 up-to-date +# @nf-core/samtools 2.1.0 - 2.1.0 missing +``` + +### Update a module + +Change the version in `nextflow.config`, then run your workflow. Nextflow automatically downloads the new version. + +```groovy +modules { + '@nf-core/fastqc' = '1.2.0' // Changed from 1.0.0 +} +``` + +--- + +## 4. Search for Modules + +```bash +# Search by keyword +nextflow module search alignment + +# Limit results +nextflow module search "quality control" -limit 5 + +# JSON output for scripting +nextflow module search bwa -json +``` + +--- + +## 5. Configure Tool Arguments + +### Define in meta.yaml (module author) + +```yaml +# modules/@nf-core/bwa-align/meta.yaml +tools: + - bwa: + description: BWA aligner + args: + K: + flag: "-K" + type: integer + description: "Process INT input bases in each batch" + Y: + flag: "-Y" + type: boolean + description: "Use soft clipping for supplementary alignments" +``` + +### Configure in nextflow.config (user) + +```groovy +// nextflow.config +process { + withName: 'BWA_ALIGN' { + tools.bwa.args.K = 100000000 + tools.bwa.args.Y = true + } +} +``` + +### Access in script (module author) + +```groovy +// main.nf +process BWA_ALIGN { + script: + """ + bwa mem ${tools.bwa.args} -t $task.cpus $index $reads + """ +} +``` + +--- + +## 6. Work with Private Registries + +### Configure authentication + +```groovy +// nextflow.config +registry { + // Multiple registries (tried in order) + url = [ + 'https://private.registry.myorg.com', + 'https://registry.nextflow.io' + ] + + auth { + 'private.registry.myorg.com' = '${MYORG_TOKEN}' + 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' + } +} +``` + +### Or use environment variable + +```bash +export NXF_REGISTRY_TOKEN=your-token-here +nextflow module install nf-core/fastqc +``` + +--- + +## 7. Publish a Module + +### Prepare your module + +``` +my-module/ +├── main.nf # Required: entry point +├── meta.yaml # Required for registry +├── README.md # Required for registry +└── tests/ # Recommended +``` + +### Validate before publishing + +```bash +nextflow module publish myorg/my-module -dry-run +``` + +### Publish to registry + +```bash +export NXF_REGISTRY_TOKEN=your-token +nextflow module publish myorg/my-module +``` + +--- + +## 8. Handle Local Modifications + +If you modify a module locally (for debugging), Nextflow protects your changes: + +```bash +# This warns and does NOT override your changes +nextflow module install nf-core/fastqc -version 1.1.0 +# Warning: Module @nf-core/fastqc has local modifications. Use -force to override. + +# Force replacement if needed +nextflow module install nf-core/fastqc -version 1.1.0 -force +``` + +--- + +## 9. Remove a Module + +```bash +# Remove module and config entry +nextflow module remove nf-core/fastqc + +# Keep config entry (just delete local files) +nextflow module remove nf-core/fastqc -keep-config + +# Keep local files (just remove from config) +nextflow module remove nf-core/fastqc -keep-files +``` + +--- + +## Common Patterns + +### Install all configured modules + +```bash +# Installs all modules listed in nextflow.config +nextflow module install +``` + +### Offline operation + +Modules are cached locally in `modules/`. Once installed, workflows run without network access. + +### Git integration + +The `modules/` directory is intended to be committed to your git repository: + +```bash +git add modules/ +git commit -m "Add module dependencies" +``` + +--- + +## Troubleshooting + +### Module not found + +```bash +# Check if module exists in registry +nextflow module search exact-module-name + +# Verify spelling and scope +# Correct: @nf-core/fastqc +# Wrong: @nfcore/fastqc, nf-core/fastqc (without @) +``` + +### Authentication errors + +```bash +# Verify token is set +echo $NXF_REGISTRY_TOKEN + +# Check registry config +grep -A5 'registry' nextflow.config +``` + +### Version conflicts + +If two modules require incompatible versions of a dependency: +- Nextflow selects the highest compatible version automatically +- If no compatible version exists, an error lists the conflicts + +### Checksum warnings + +``` +Warning: Module @nf-core/fastqc has local modifications +``` + +This means the local module content differs from the registry version. Your changes are preserved. Use `-force` only if you want to discard local changes. \ No newline at end of file diff --git a/specs/251117-module-system/research.md b/specs/251117-module-system/research.md new file mode 100644 index 0000000000..270603da82 --- /dev/null +++ b/specs/251117-module-system/research.md @@ -0,0 +1,360 @@ +# Research: Nextflow Module System Client + +**Date**: 2026-01-19 +**Feature**: 251117-module-system + +## Overview + +This document captures technical research and decisions for implementing the Nextflow module system client. All NEEDS CLARIFICATION items from Technical Context have been resolved through codebase exploration. + +--- + +## 1. CLI Command Structure + +**Research Question**: How should `nextflow module` CLI commands be implemented? + +**Decision**: Follow CmdPlugin pattern with sub-command delegation + +**Rationale**: +- CmdPlugin.groovy provides proven pattern for multi-action commands +- Uses JCommander `@Parameters` and `@Parameter` annotations +- Sub-commands (install, search, list, remove, publish, run) handled via positional args +- PluginExecAware interface allows plugin extensibility if needed later + +**Reference Implementation**: +``` +Location: modules/nextflow/src/main/groovy/nextflow/cli/CmdPlugin.groovy +Pattern: + - Extends CmdBase + - @Parameters(commandNames = 'module', commandDescription = '...') + - @Parameter(names = ['-h', '--help']) + - args list for sub-command + module name + - run() method dispatches to install(), search(), etc. +``` + +**Alternatives Considered**: +- Separate CmdModuleInstall, CmdModuleSearch classes: Rejected - too many entry points, doesn't match existing patterns +- Plugin-based CLI extension: Rejected - module system is core functionality, not optional + +--- + +## 2. DSL Parser Extension for @scope/name + +**Research Question**: How to extend `include` statement parsing for registry modules? + +**Decision**: Extend ResolveIncludeVisitor to detect `@` prefix and delegate to ModuleResolver + +**Rationale**: +- IncludeNode already captures source path as string +- Detection: `source.startsWith('@')` distinguishes registry vs local paths +- Resolution happens at parse time (after plugin resolution) per ADR +- Preserves existing local file include behavior + +**Reference Implementation**: +``` +Location: modules/nf-lang/src/main/java/nextflow/script/ResolveIncludeVisitor.java +Extension Point: visitInclude() method +Pattern: + 1. Check if source starts with '@' + 2. If yes: call ModuleResolver.resolve(source, configuredVersion) + 3. ModuleResolver returns absolute path to modules/@scope/name/main.nf + 4. Continue with standard include processing +``` + +**Key Files**: +- `IncludeNode.java` - AST representation +- `IncludeEntryNode.java` - Individual entries +- `ResolveIncludeVisitor.java` - Visitor for resolution + +**Alternatives Considered**: +- New ANTLR grammar token for `@`: Rejected - unnecessary parser complexity +- Dot file marker for local modules: Deferred to Open Questions in ADR + +--- + +## 3. Config Parsing for modules{} and registry{} Blocks + +**Research Question**: How to add new config DSL blocks? + +**Decision**: Create ModulesDsl and RegistryDsl classes following PluginsDsl pattern + +**Rationale**: +- PluginsDsl.groovy provides exact template for DSL block handling +- ConfigBuilder already supports dynamic DSL registration +- Groovy's methodMissing enables clean config syntax + +**Reference Implementation**: +``` +Location: modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy +Pattern: + @CompileStatic + class ModulesDsl { + private Map modules = [:] + + def methodMissing(String name, args) { + // modules { '@nf-core/fastqc' = '1.0.0' } + modules[name] = args[0].toString() + } + + Map getModules() { modules } + } +``` + +**RegistryDsl Pattern**: +```groovy +class RegistryDsl { + String url = 'https://registry.nextflow.io' + List urls = [] // For multiple registries + Map auth = [:] + + void url(String value) { this.url = value } + void url(List values) { this.urls = values } + void auth(Closure config) { /* parse auth block */ } +} +``` + +**Integration Point**: ConfigBuilder.build() instantiates DSL objects + +**Alternatives Considered**: +- JSON/YAML config file: Rejected - inconsistent with Nextflow config style +- Dedicated pipeline.yaml: Deferred per ADR Open Questions + +--- + +## 4. Registry HTTP Communication + +**Research Question**: How to communicate with module registry API? + +**Decision**: Create HttpModuleRepository following HttpPluginRepository pattern + +**Rationale**: +- HttpPluginRepository provides robust HTTP client with retry logic +- Uses HxClient from io.seqera.http (already a dependency) +- Handles authentication headers consistently +- Supports connection pooling and timeout configuration + +**Reference Implementation**: +``` +Location: modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy +Pattern: + class HttpModuleRepository { + private final URI url + private final HxClient httpClient + private final String authToken + + ModuleInfo getModule(String name, String version) + List search(String query, int limit) + Path download(String name, String version, Path target) + void publish(String name, Path bundle) + } +``` + +**API Endpoints** (from ADR): +``` +GET /api/modules?query= # Search +GET /api/modules/{name} # Get module + latest release +GET /api/modules/{name}/releases # List all releases +GET /api/modules/{name}/{version} # Get specific release +GET /api/modules/{name}/{version}/download # Download bundle +POST /api/modules/{name} # Publish (authenticated) +``` + +**Alternatives Considered**: +- Direct HttpClient usage: Rejected - loses retry, pooling benefits +- gRPC protocol: Rejected - registry already uses REST + +--- + +## 5. Authentication Patterns + +**Research Question**: How to handle registry authentication? + +**Decision**: Support NXF_REGISTRY_TOKEN env var + registry.auth config block + +**Rationale**: +- Environment variable provides CI/CD compatibility +- Config block allows per-registry tokens for private registries +- Follows existing plugin auth patterns +- Bearer token in Authorization header (standard HTTP auth) + +**Reference Implementation**: +``` +Location: modules/nextflow/src/main/groovy/nextflow/cli/CmdAuth.groovy +Pattern: + 1. Check NXF_REGISTRY_TOKEN environment variable + 2. Fall back to registry.auth.'registry.nextflow.io' in config + 3. Add header: Authorization: Bearer +``` + +**Config Syntax**: +```groovy +registry { + auth { + 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' + 'private.registry.com' = '${PRIVATE_TOKEN}' + } +} +``` + +**Alternatives Considered**: +- Secrets file (~/.nextflow/secrets.json): Possible future enhancement +- OAuth flow: Rejected for CLI - token-based simpler + +--- + +## 6. Checksum Verification + +**Research Question**: How to implement module integrity verification? + +**Decision**: SHA-256 checksum stored in `.checksum` file, verified on every run + +**Rationale**: +- SHA-256 is industry standard, already used for plugin verification +- `.checksum` file stores registry-provided checksum (from X-Checksum header) +- Local checksum computed on-demand and compared +- Mismatch indicates local modification (warn, don't override) + +**Implementation Pattern**: +```groovy +class ModuleChecksum { + static final String ALGORITHM = 'SHA-256' + + static String compute(Path moduleDir) { + // Hash all files in module directory + // Exclude .checksum itself + // Return hex-encoded SHA-256 + } + + static boolean verify(Path moduleDir) { + def expected = moduleDir.resolve('.checksum').text.trim() + def actual = compute(moduleDir) + return expected == actual + } + + static void save(Path moduleDir, String checksum) { + moduleDir.resolve('.checksum').text = checksum + } +} +``` + +**Checksum Scope**: Covers all files in module directory (main.nf, meta.yaml, README.md, etc.) + +**Alternatives Considered**: +- Per-file checksums: Rejected - adds complexity, single checksum sufficient +- MD5: Rejected - SHA-256 more secure + +--- + +## 7. Version Constraint Syntax + +**Research Question**: What version constraint syntax to use for module dependencies? + +**Decision**: Reuse existing Nextflow plugin version constraint syntax + +**Rationale**: +- Already implemented and tested in plugin system +- Users familiar with existing `nextflowVersion` syntax +- Supports ranges, comparisons, exact versions +- No new parser code needed + +**Supported Syntax**: +| Notation | Meaning | Example | +|----------|---------|---------| +| `1.2.3` | Exact version | `@nf-core/fastqc@1.0.0` | +| `>=1.2.3` | Greater or equal | `@nf-core/fastqc@>=1.0.0` | +| `<=1.2.3` | Less or equal | `@nf-core/fastqc@<=2.0.0` | +| `>=1.2.0,<2.0.0` | Range | `@nf-core/samtools@>=1.0.0,<2.0.0` | + +**Reference**: Version parsing code exists in plugin system; reuse VersionNumber class + +**Alternatives Considered**: +- NPM-style `^` and `~`: Rejected - inconsistent with existing Nextflow patterns +- Always latest: Rejected - breaks reproducibility + +--- + +## 8. Tool Arguments Implementation + +**Research Question**: How to implement structured tool arguments (`tools..args`)? + +**Decision**: Implement as implicit variable in process scope, validated at parse time + +**Rationale**: +- `tools` variable accessible in script block like `task`, `params` +- Validation at parse time catches errors early (per clarification) +- Schema defined in meta.yaml, parsed by ModuleManifest +- Concatenation logic handles flag formatting + +**Implementation Pattern**: +```groovy +class ToolArgs { + private Map schema // From meta.yaml + private Map values // From config + + String getAt(String argName) { + def def = schema[argName] + def value = values[argName] + if (def.type == 'boolean' && value) { + return def.flag // e.g., "-Y" + } + return "${def.flag} ${value}" // e.g., "-K 100000" + } + + String toString() { + // Concatenate all configured args + values.collect { name, value -> + this[name] + }.join(' ') + } +} +``` + +**Config Access**: +```groovy +withName: 'BWA_MEM' { + tools.bwa.args.K = 100000 + tools.bwa.args.Y = true +} +``` + +**Script Access**: +```groovy +script: +""" +bwa mem ${tools.bwa.args} -t $task.cpus $index $reads +""" +``` + +**Alternatives Considered**: +- Runtime validation only: Rejected - late errors waste compute +- String-only values: Rejected - loses type safety benefits + +--- + +## Summary of Key Decisions + +| Area | Decision | Key Reference | +|------|----------|---------------| +| CLI | CmdModule extends CmdBase | CmdPlugin.groovy | +| DSL Parser | Extend ResolveIncludeVisitor | ResolveIncludeVisitor.java | +| Config | ModulesDsl + RegistryDsl | PluginsDsl.groovy | +| Registry HTTP | HttpModuleRepository | HttpPluginRepository.groovy | +| Authentication | NXF_REGISTRY_TOKEN + config | CmdAuth.groovy | +| Checksums | SHA-256, .checksum file | Standard Java security | +| Version Syntax | Plugin-compatible constraints | VersionNumber class | +| Tool Args | Implicit variable, parse-time validation | New implementation | + +--- + +## Open Items (Deferred) + +These items are noted in the ADR as open questions and do not block implementation: + +1. **Local vs managed module distinction**: Whether local modules use `@` prefix or dot file marker +2. **Tool arguments CLI syntax**: Colon vs dot separator (`--tools:bwa:K` vs `--tools.bwa.K`) +3. **Module version location**: nextflow.config vs dedicated pipeline.yaml + +Current implementation uses: +- `@` prefix for registry modules only (local paths start with `.` or `/`) +- Colon-separated CLI syntax per ADR assumption +- Versions in nextflow.config per ADR decision \ No newline at end of file diff --git a/specs/251117-module-system/spec.md b/specs/251117-module-system/spec.md index 4a538a765c..73d5d21427 100644 --- a/specs/251117-module-system/spec.md +++ b/specs/251117-module-system/spec.md @@ -10,7 +10,7 @@ This specification covers the **Nextflow client-side implementation** of the module system, enabling pipeline developers to: - Include remote modules from the Nextflow registry using `@scope/name` syntax - Manage module versions through `nextflow.config` -- Use CLI commands to install, search, list, remove, freeze, publish, and run modules +- Use CLI commands to install, search, list, remove, publish, and run modules - Configure tool arguments through structured `meta.yaml` definitions **Out of Scope**: Registry backend implementation (assumed already available at `registry.nextflow.io`) @@ -150,7 +150,7 @@ A module author wants to publish their module to the Nextflow registry for other - How does the system handle circular module dependencies? - Dependency resolver detects cycles and fails with an error listing the cycle - What happens when two modules require incompatible versions of the same dependency? - - Version conflict is reported with the conflicting requirements + - System automatically selects the highest compatible version; if no compatible version exists, fails with error listing conflicting requirements - How are modules resolved when multiple registries are configured? - Registries are tried in order; first match wins - What happens when `meta.yaml` is missing from a module? @@ -176,41 +176,42 @@ A module author wants to publish their module to the Nextflow registry for other - **FR-007**: System MUST verify module integrity using `.checksum` file on every run - **FR-008**: System MUST download modules from registry when not present locally or when version differs - **FR-009**: System MUST NOT override locally modified modules (checksum mismatch) unless `-force` is used +- **FR-010**: System MUST resolve version conflicts by selecting the highest compatible version; if no compatible version exists, MUST fail with error listing conflicting requirements #### Local Storage -- **FR-012**: System MUST store modules in `modules/@scope/name/` directory structure (single version per module) -- **FR-013**: System MUST create `.checksum` file from registry's X-Checksum header on download -- **FR-014**: System MUST store module's `main.nf`, `meta.yaml`, and supporting files in the module directory +- **FR-011**: System MUST store modules in `modules/@scope/name/` directory structure (single version per module) +- **FR-012**: System MUST create `.checksum` file from registry's X-Checksum header on download +- **FR-013**: System MUST store module's `main.nf`, `meta.yaml`, and supporting files in the module directory #### CLI Commands -- **FR-015**: System MUST provide `nextflow module install [scope/name]` command to download modules -- **FR-016**: System MUST provide `nextflow module search ` command to search the registry -- **FR-017**: System MUST provide `nextflow module list` command to show installed vs configured modules -- **FR-018**: System MUST provide `nextflow module remove scope/name` command to delete modules -- **FR-019**: System MUST provide `nextflow module publish scope/name` command to upload modules to registry -- **FR-020**: System MUST provide `nextflow module run scope/name` command to execute modules directly +- **FR-014**: System MUST provide `nextflow module install [scope/name]` command to download modules +- **FR-015**: System MUST provide `nextflow module search ` command to search the registry +- **FR-016**: System MUST provide `nextflow module list` command to show installed vs configured modules +- **FR-017**: System MUST provide `nextflow module remove scope/name` command to delete modules +- **FR-018**: System MUST provide `nextflow module publish scope/name` command to upload modules to registry +- **FR-019**: System MUST provide `nextflow module run scope/name` command to execute modules directly #### Configuration -- **FR-022**: System MUST read module versions from `modules {}` block in `nextflow.config` -- **FR-023**: System MUST support `registry {}` block for configuring registry URL and authentication -- **FR-024**: System MUST support `NXF_REGISTRY_TOKEN` environment variable for authentication -- **FR-025**: System MUST support multiple registry URLs with fallback ordering +- **FR-020**: System MUST read module versions from `modules {}` block in `nextflow.config` +- **FR-021**: System MUST support `registry {}` block for configuring registry URL and authentication +- **FR-022**: System MUST support `NXF_REGISTRY_TOKEN` environment variable for authentication +- **FR-023**: System MUST support multiple registry URLs with fallback ordering #### Tool Arguments -- **FR-026**: System MUST provide `tools..args.` implicit variable in module scripts -- **FR-027**: System MUST validate tool arguments against `meta.yaml` schema (type, enum) -- **FR-028**: System MUST support boolean, integer, float, string, file, and path argument types -- **FR-029**: System MUST concatenate all tool arguments when `tools..args` is accessed +- **FR-024**: System MUST provide `tools..args.` implicit variable in module scripts +- **FR-025**: System MUST validate tool arguments against `meta.yaml` schema (type, enum) at workflow parse time +- **FR-026**: System MUST support boolean, integer, float, string, file, and path argument types +- **FR-027**: System MUST concatenate all tool arguments when `tools..args` is accessed #### Registry Communication -- **FR-030**: System MUST communicate with registry via documented Module API endpoints -- **FR-031**: System MUST handle authentication using Bearer token in Authorization header -- **FR-032**: System MUST verify SHA-256 checksum on module download +- **FR-028**: System MUST communicate with registry via documented Module API endpoints +- **FR-029**: System MUST handle authentication using Bearer token in Authorization header +- **FR-030**: System MUST verify SHA-256 checksum on module download ### Key Entities @@ -248,4 +249,11 @@ A module author wants to publish their module to the Nextflow registry for other - Registry backend API (Module API endpoints as specified in ADR) - Existing Nextflow plugin system (for authentication reuse) - Existing DSL parser infrastructure (for `include` statement extension) -- Existing config parser (for `modules {}` and `registry {}` blocks) \ No newline at end of file +- Existing config parser (for `modules {}` and `registry {}` blocks) + +## Clarifications + +### Session 2026-01-19 + +- Q: What should happen when incompatible dependency versions are detected? → A: Use highest compatible version automatically, warn if none exists +- Q: When should tool argument validation occur? → A: At workflow parse time (early, before any execution) \ No newline at end of file From f7e9def37284275d030ceb2ceb61334482a9f4d8 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 21 Jan 2026 12:42:21 +0100 Subject: [PATCH 27/75] Remove tools config [ci skip] Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 7001576715..09a8635bd4 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -539,7 +539,7 @@ bwa mem $args -t $task.cpus $index $reads | samtools sort $args2 -o out.bam - ### New Pattern: Structured Tool Arguments Modules declare available arguments in `meta.yaml` under each tool's `args` property. -This list does _not_ need to be exhaustive. It should include any arguments known to be used by pipelines or that could be expected to be used by users. However, arguments can still be specified in the config even if not defined in this file, so absence does not prevent use. +This list does _not_ need to be exhaustive. It should include any arguments known to be used by pipelines or that could be expected to be used by users. ```yaml tools: From 474eb758c5a7b9510d43edc87932b2dedcd846c2 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 22 Jan 2026 18:01:48 +0100 Subject: [PATCH 28/75] Simplify module system requires definition [ci skip] - Remove plugins, modules, subworkflows from requires block - Spec now focused on process modules only - Dependencies managed via nextflow.config Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 160 ++++------------------------------ 1 file changed, 16 insertions(+), 144 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 09a8635bd4..bfb1b37ffc 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -4,10 +4,14 @@ - Status: draft - Date: 2025-01-06 - Tags: modules, dsl, registry, versioning, architecture -- Version: 2.4 +- Version: 2.5 ## Updates +### Version 2.5 (2026-01-22) +- **Simplified `requires` block**: Removed `plugins`, `modules`, and `subworkflows` sub-properties; `requires` now only contains `nextflow` version constraint +- **Process modules focus**: Removed sub-workflow references; spec is now focused on process modules only + ### Version 2.4 (2026-01-15) - **Removed transitive dependency resolution**: Module dependencies are explicit only; no automatic transitive resolution - **Removed `freeze` command**: No longer needed without transitive dependency management @@ -26,9 +30,8 @@ ### Version 2.1 (2024-12-11) - **Unified dependencies**: Consolidated `components`, `dependencies`, and `requires` into single `requires` field -- **New sub-properties**: `requires.modules` and `requires.workflows` for declaring module dependencies -- **Unified version syntax**: `[scope/]name[@constraint]` format across plugins, modules, and workflows -- **Deprecation**: `components` field deprecated (use `requires.modules` instead) +- **Unified version syntax**: `[scope/]name[@constraint]` format across plugins and modules +- **Deprecation**: `components` field deprecated (use top-level `modules` instead) ## Context and Problem Statement @@ -124,9 +127,6 @@ version: 1.2.4 # This module's version requires: nextflow: ">=24.04.0" - modules: # Required modules (version constraints) - - nf-core/samtools/view@>=1.0.0,<2.0.0 - - nf-core/samtools/sort@>=2.1.0,<2.2.0 ``` **Version Constraints** (unified `name@constraint` syntax): @@ -157,8 +157,7 @@ npm-style `^` and `~` notation while maintaining consistency with existing Nextf This avoids introducing new notation that would require additional parser support. **Dependency Resolution**: -- Workflow's `nextflow.config` specifies exact versions for dependencies -- Module dependencies declared in `meta.yaml` using version constraints +- Workflow's `nextflow.config` specifies exact versions for module dependencies ### 3. Unified Nextflow Registry @@ -415,7 +414,7 @@ Everything within the module directory should be uploaded. Module bundle should ``` my-module/ ├── main.nf # Required: entry point for module -├── meta.yaml # Optional: Module spec (metadata, dependencies, I/O specs) +├── meta.yaml # Optional: Module spec (metadata, I/O specs) ├── README.md # Required: Module description └── tests/ # Optional test workflows ``` @@ -431,11 +430,6 @@ license: MIT requires: nextflow: ">=24.04.0" - plugins: - - nf-amazon@2.0.0 - modules: - - nf-core/samtools/view@>=1.0.0,<2.0.0 - - nf-core/samtools/sort@>=2.1.0,<2.2.0 ``` **Local Storage Structure**: @@ -472,7 +466,7 @@ project-root/ **Phase 2**: Extend Nextflow registry for modules, implement caching, add `install` and `search` commands -**Phase 3**: Extend DSL parser for `from module` syntax, implement dependency resolution from meta.yaml +**Phase 3**: Extend DSL parser for `from module` syntax **Phase 4**: Implement `publish` command with authentication and `run` command @@ -492,9 +486,7 @@ project-root/ - Exists, checksum valid, different version → replace with declared version - Exists, checksum mismatch → warn and do NOT override (local changes detected) 3. On download: store module to `modules/@scope/name/` with `.checksum` file -4. Read `meta.yaml` file: - a. Validates Nextflow requirement → Fail if not fulfilled - b. Load Pluign requirements if not exist. +4. Read `meta.yaml` file: Validates Nextflow requirement → Fail if not fulfilled 5. Parse module's `main.nf` file → make processes available``` **Security**: @@ -504,7 +496,6 @@ project-root/ - Support for private registries **Integration with Plugin System**: -- Modules can declare plugin dependencies in meta.yaml - Both plugins and modules query same registry - Single authentication system - Separate cache locations: `$NXF_HOME/plugins/` (global) vs `modules/` (per-project) @@ -717,11 +708,8 @@ These fields extend the schema to support the new Nextflow module system: |-------|------|----------|-------------| | `version` | string | Registry | Semantic version (MAJOR.MINOR.PATCH) | | `license` | string | Registry | SPDX license identifier for module code | -| `requires` | object | Optional | All requirements: runtime, plugins, and dependencies | +| `requires` | object | Optional | Runtime requirements | | `requires.nextflow` | string | Optional | Nextflow version constraint | -| `requires.plugins` | array[string] | Optional | Required Nextflow plugins | -| `requires.modules` | array[string] | Optional | Required modules (processes) | -| `requires.workflows` | array[string] | Optional | Required workflows/subworkflows | ### Detailed Field Specifications @@ -762,33 +750,13 @@ version: "1.0.0-beta.1" #### `requires` -Specifies all requirements for the module: runtime environment, plugins, and dependencies. +Specifies runtime requirements for the module. ```yaml requires: nextflow: ">=24.04.0" - plugins: - - nf-amazon@2.0.0 - - nf-wave@>=1.5.0 - modules: - - nf-core/fastqc@>=1.0.0 - - nf-core/samtools/sort@>=2.1.0,<3.0.0 - - bwa/mem - workflows: - - nf-core/fastq-align-bwa@1.0.0 ``` -**Unified Version Constraint Syntax:** - -All requirements (except `nextflow`) use a unified `name@constraint` format: - -| Format | Meaning | Example | -|--------|---------|---------| -| `name` | Any version (latest) | `bwa/mem` | -| `name@1.2.3` | Exact version | `nf-core/fastqc@1.0.0` | -| `name@>=1.2.3` | Greater or equal | `nf-core/fastqc@>=1.0.0` | -| `name@>=1.2.3,<2.0.0` | Range constraint | `nf-core/samtools/sort@>=2.1.0,<3.0.0` | - **`requires.nextflow`** - Nextflow version constraint: ```yaml requires: @@ -796,36 +764,6 @@ requires: nextflow: ">=24.04.0,<25.0.0" # version range ``` -**`requires.plugins`** - Required Nextflow plugins: -```yaml -requires: - plugins: - - nf-amazon@2.0.0 # exact version - - nf-wave@>=1.5.0 # minimum version - - nf-azure # any version -``` - -**`requires.modules`** - Required modules (processes): -```yaml -requires: - modules: - - nf-core/fastqc@>=1.0.0 # registry module with constraint - - nf-core/samtools/sort@>=2.1.0 # nested module path - - bwa/mem # local or registry (no constraint) -``` - -**`requires.workflows`** - Required workflows/subworkflows: -```yaml -requires: - workflows: - - nf-core/fastq-align-bwa@1.0.0 # registry workflow - - my-local-workflow # local workflow -``` - -**Resolution:** -1. The resolver looks up dependencies locally first, then in configured registries -2. Pinned versions are recorded in `nextflow.config` for reproducibility - #### `tools` Documents the software tools wrapped by the module, including their command-line arguments: @@ -918,24 +856,6 @@ output: description: Software versions ``` -**Subworkflow Pattern (Simplified):** -```yaml -input: - - ch_reads: - description: | - Input FastQ files - Structure: [ val(meta), [ path(reads) ] ] - - ch_index: - description: BWA index files - type: file - -output: - - bam: - description: Aligned BAM files - - versions: - description: Software versions -``` - **Channel Element Properties:** | Property | Type | Description | @@ -986,8 +906,6 @@ keywords: license: MIT # Added module license requires: # Added requirements nextflow: ">=24.04.0" - modules: # Added module dependencies (if any) - - nf-core/samtools/sort@>=1.0.0 tools: - bwa: description: BWA software @@ -1024,8 +942,8 @@ version: "1.0.0" | Scoped names | No | Yes (registry) | | Version field | No | Yes (required for registry) | | `tools` section | Yes | Yes | -| `components` | Yes (subworkflows) | Deprecated → use `requires.modules` | -| `requires` | No | Yes (unified requirements field) | +| `components` | Yes | Deprecated | +| `requires` | No | Yes (Nextflow version constraint) | | I/O specifications | Yes | Yes | | Ontologies | Yes | Yes | @@ -1036,7 +954,7 @@ The following attributes from the nf-core meta schema are **not supported** in t | Attribute | Reason | Future | |-----------|--------|--------| | `extra_args` | Not adopted in practice by nf-core modules | Will be redesigned as part of the `tools` schema attribute to document tool-specific arguments and configuration options | -| `components` | Replaced by unified `requires.modules` | Use `requires.modules` for all module dependencies (local and registry) | +| `components` | No longer supported | Module dependencies are managed via `nextflow.config` | ### Complete Examples @@ -1086,11 +1004,6 @@ license: MIT requires: nextflow: ">=24.04.0" - plugins: - - nf-wave@1.5.0 - modules: - - nf-core/samtools/view@>=1.0.0,<2.0.0 - - nf-core/samtools/sort@>=2.1.0,<2.2.0 tools: - bwa: @@ -1145,44 +1058,3 @@ output: pattern: "versions.yml" ``` -#### Subworkflow with Module Dependencies - -```yaml -name: fastq_align_bwa -description: Align reads with BWA and generate statistics -keywords: - - alignment - - bwa - - samtools - - statistics - -requires: - modules: - - bwa/mem - - samtools/sort - - samtools/index - - samtools/stats - -authors: - - "@JoseEspinosa" -maintainers: - - "@JoseEspinosa" - -input: - - ch_reads: - description: | - Input FastQ files - Structure: [ val(meta), [ path(reads) ] ] - - ch_index: - description: BWA index files - -output: - - bam: - description: Sorted BAM files - - bai: - description: BAM index files - - stats: - description: Alignment statistics - - versions: - description: Software versions -``` From 55812572170389554f260107d1bec1be287d977c Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 23 Jan 2026 11:33:25 +0100 Subject: [PATCH 29/75] Replace tool arguments with module parameters in ADR [ci skip] - Module parameters defined in meta.yaml params section with name, type, description, and example attributes - Removed tools args property and specification - Updated schema and examples throughout Co-Authored-By: Claude Opus 4.5 Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 214 ++++++++++++++++------------------ adr/module-spec-schema.json | 130 +++++++-------------- 2 files changed, 138 insertions(+), 206 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index bfb1b37ffc..69682454dd 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -8,7 +8,9 @@ ## Updates -### Version 2.5 (2026-01-22) +### Version 2.5 (2026-01-23) +- **Module parameters**: Replaced structured tool arguments with general module parameters defined in `meta.yaml` +- **Simplified tools section**: Removed `args` property from tools; tool arguments now configured via module parameters - **Simplified `requires` block**: Removed `plugins`, `modules`, and `subworkflows` sub-properties; `requires` now only contains `nextflow` version constraint - **Process modules focus**: Removed sub-workflow references; spec is now focused on process modules only @@ -23,10 +25,11 @@ - **Simplified storage model**: Single version per module locally (`modules/@scope/name/` without version in path) - **`.checksum` file**: Registry checksum cached locally for fast integrity verification without network calls -### Version 2.2 (2025-01-06) +### Version 2.2 (2025-01-06) — *Superseded by v2.5* - **Structured tool arguments**: Added `args` property to `tools` section for type-safe argument configuration - **New implicit variables**: `tools..args.` returns formatted flag+value; `tools..args` returns all args concatenated - **Deprecation**: All `ext.*` custom directives (e.g., `ext.args`, `ext.args2`, `ext.args3`, `ext.prefix`, `ext.suffix`) deprecated in favor of structured tool arguments +- *Note: Tool arguments replaced by module parameters in v2.5* ### Version 2.1 (2024-12-11) - **Unified dependencies**: Consolidated `components`, `dependencies`, and `requires` into single `requires` field @@ -219,14 +222,14 @@ Run a module directly without requiring a wrapper workflow script. This command **Options**: - `-version `: Run a specific version (default: latest or configured version) - `-- `: Map value to the corresponding module process input channel -- `--tools:: `: Configure tool-specific arguments (validated against meta.yaml schema) +- `-- `: Configure module parameters (validated against meta.yaml schema) - All standard `nextflow run` options (e.g., `-profile`, `-work-dir`, `-resume`, etc.) **Behavior**: 1. Checks if module is installed locally; if not, downloads from registry 2. Parses the module's `main.nf` to identify the main process and its input declarations 3. Validates command-line arguments against the process input schema -4. Validates tool arguments against the `tools.*.args` schema in `meta.yaml` +4. Validates parameters against the `params` schema in `meta.yaml` 5. Generates an implicit workflow that wires CLI arguments to process inputs 6. Executes the workflow using standard Nextflow runtime @@ -236,12 +239,11 @@ Run a module directly without requiring a wrapper workflow script. This command - Multiple values can be provided for inputs expecting collections - Required inputs without defaults must be provided; optional inputs use declared defaults -**Tool Arguments**: -- Arguments prefixed with `--tools:` configure tool-specific parameters -- Format: `--tools:: ` (e.g., `--tools:bwa:K 100000000`) -- Boolean flags can be specified without value (e.g., `--tools:bwa:Y`) -- Arguments are validated against the tool's `args` schema in `meta.yaml` -- Invalid argument names or values that fail type/enum validation produce errors +**Module Parameters**: +- Parameters defined in `meta.yaml` can be configured via CLI arguments +- Boolean flags can be specified without value (e.g., `--use_soft_clipping`) +- Arguments are validated against the `params` schema in `meta.yaml` +- Invalid parameter names or values that fail type validation produce errors **Example**: ```bash @@ -263,13 +265,13 @@ nextflow module run nf-core/salmon \ -work-dir /tmp/work \ --outdir results/ -# Run with tool-specific arguments +# Run with module parameters nextflow module run nf-core/bwa-align \ --reads 'samples/*_{1,2}.fastq.gz' \ --reference genome.fa \ - --tools:bwa:K 100000000 \ - --tools:bwa:Y \ - --tools:samtools:output_fmt cram + --batch_size 100000000 \ + --use_soft_clipping \ + --output_format cram ``` --- @@ -414,7 +416,7 @@ Everything within the module directory should be uploaded. Module bundle should ``` my-module/ ├── main.nf # Required: entry point for module -├── meta.yaml # Optional: Module spec (metadata, I/O specs) +├── meta.yaml # Required: Module spec (version, metadata, I/O specs) ├── README.md # Required: Module description └── tests/ # Optional test workflows ``` @@ -500,99 +502,72 @@ project-root/ - Single authentication system - Separate cache locations: `$NXF_HOME/plugins/` (global) vs `modules/` (per-project) -## Tool Arguments Configuration +## Module Parameters -The module system introduces a structured approach to tool argument configuration, replacing the legacy `ext.args` pattern with type-safe, documented argument specifications. +The module system introduces a structured approach to module configuration through parameters defined in `meta.yaml`. Parameters provide a documented, type-safe way to customize module behavior. -### Current Pattern (Deprecated) +### Parameter Definition -The traditional nf-core pattern uses `ext.args` strings in config files: +Modules declare available parameters in `meta.yaml` under the `params` section. Each parameter has a name and optional attributes for type and description. -```groovy -// Config file -withName: 'BWA_MEM' { - ext.args = "-K 100000000 -Y -B 3 -R ${meta.read_group}" - ext.args2 = "--output-fmt cram" -} - -// Module script -def args = task.ext.args ?: '' -def args2 = task.ext.args2 ?: '' -bwa mem $args -t $task.cpus $index $reads | samtools sort $args2 -o out.bam - +```yaml +params: + - name: batch_size + type: integer + description: "Process INT input bases in each batch" + example: 100000000 + + - name: use_soft_clipping + type: boolean + description: "Use soft clipping for supplementary alignments" + + - name: output_format + type: string + description: "Output format (sam, bam, or cram)" ``` -**Limitations:** -- No documentation of available arguments -- No validation or type checking -- Unclear which `ext.argsN` maps to which tool -- No IDE autocompletion support - -### New Pattern: Structured Tool Arguments +### Parameter Attributes -Modules declare available arguments in `meta.yaml` under each tool's `args` property. -This list does _not_ need to be exhaustive. It should include any arguments known to be used by pipelines or that could be expected to be used by users. - -```yaml -tools: - - bwa: - description: BWA aligner - homepage: http://bio-bwa.sourceforge.net/ - args: - K: - flag: "-K" - type: integer - description: "Process INT input bases in each batch" - Y: - flag: "-Y" - type: boolean - description: "Use soft clipping for supplementary alignments" - - - samtools: - description: SAMtools - homepage: http://www.htslib.org/ - args: - output_fmt: - flag: "--output-fmt" - type: string - enum: ["sam", "bam", "cram"] - description: "Output format" -``` +| Attribute | Required | Description | +|-----------|----------|-------------| +| `name` | Yes | Parameter identifier | +| `type` | No | Data type: `boolean`, `integer`, `float`, `string`, `file`, `path` | +| `description` | No | Human-readable description | +| `example` | No | Example value for the parameter | ### Configuration Usage -Arguments are configured using `tools..args.`: +Parameters are configured using standard Nextflow params syntax: ```groovy withName: 'BWA_MEM' { - tools.bwa.args.K = 100000000 - tools.bwa.args.Y = true - tools.samtools.args.output_fmt = "cram" + params.batch_size = 100000000 + params.use_soft_clipping = true + params.output_format = "cram" } ``` ### Script Usage -In module scripts, access arguments via the `tools` implicit variable: +In module scripts, access parameters via the standard `params` variable: ```groovy -// tools.bwa.args.K → "-K 100000000" -// tools.bwa.args.Y → "-Y" -// tools.bwa.args → "-K 100000000 -Y" (all args concatenated) +def batch_arg = params.batch_size ? "-K ${params.batch_size}" : '' +def soft_clip = params.use_soft_clipping ? "-Y" : '' -bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ - | samtools sort ${tools.samtools.args} -o ${prefix}.bam - +bwa mem ${batch_arg} ${soft_clip} -t $task.cpus $index $reads \ + | samtools sort --output-fmt ${params.output_format ?: 'bam'} -o ${prefix}.bam - ``` ### Benefits -| Aspect | `ext.args` (Legacy) | `tools.*.args` (New) | -|--------|---------------------|----------------------| +| Aspect | `ext.args` (Legacy) | Module Parameters (New) | +|--------|---------------------|-------------------------| | Documentation | None | In meta.yaml | | Type Safety | None | Validated | | IDE Support | None | Autocompletion | -| Multi-tool | Confusing (`ext.args2`) | Clear (`tools.samtools.args`) | +| Clarity | Opaque strings | Named parameters | | Defaults | Manual | Schema-defined | -| Enums | None | Validated | ## Comparison: Plugins vs. Modules @@ -668,11 +643,7 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads \ 1. **Local vs managed module distinction**: Should local modules use the `@` prefix in include statements, or should a dot file (e.g., `.nf-modules`) be used to distinguish local modules from managed/remote modules? -2. **Tool arguments CLI syntax**: What is the preferred syntax for tool arguments on the command line? - - Colon-separated: `--tools:: ` - - Dot-separated: `--tools.. ` - -3. **Module version configuration**: Should pipeline module versions be specified in `nextflow.config` or in a dedicated pipeline spec file (e.g., `pipeline.yaml`)? +2. **Module version configuration**: Should pipeline module versions be specified in `nextflow.config` or in a dedicated pipeline spec file (e.g., `pipeline.yaml`)? --- @@ -710,6 +681,7 @@ These fields extend the schema to support the new Nextflow module system: | `license` | string | Registry | SPDX license identifier for module code | | `requires` | object | Optional | Runtime requirements | | `requires.nextflow` | string | Optional | Nextflow version constraint | +| `params` | array[object] | Optional | Module parameter specifications | ### Detailed Field Specifications @@ -766,7 +738,7 @@ requires: #### `tools` -Documents the software tools wrapped by the module, including their command-line arguments: +Documents the software tools wrapped by the module: ```yaml tools: @@ -774,15 +746,7 @@ tools: description: BWA aligner homepage: http://bio-bwa.sourceforge.net/ license: ["GPL-3.0-or-later"] - args: - K: - flag: "-K" - type: integer - description: "Process INT input bases in each batch" - Y: - flag: "-Y" - type: boolean - description: "Use soft clipping for supplementary alignments" + identifier: biotools:bwa ``` **Tool Properties:** @@ -798,29 +762,35 @@ tools: | `license` | Recommended | SPDX license(s) | | `identifier` | Recommended | bio.tools identifier | | `manual` | No | User manual URL | -| `args` | No | Command-line argument specifications | -**Argument Properties (`args.`):** +#### `params` -The `args` object maps argument names to their specifications. Argument names become accessible in scripts via `tools..args.`. +Defines configurable parameters for the module: + +```yaml +params: + - name: batch_size + type: integer + description: "Process INT input bases in each batch" + example: 100000000 + + - name: use_soft_clipping + type: boolean + description: "Use soft clipping for supplementary alignments" + + - name: output_format + type: string + description: "Output format (sam, bam, or cram)" +``` + +**Parameter Properties:** | Property | Required | Description | |----------|----------|-------------| -| `flag` | Yes | CLI flag (e.g., `-K`, `--output-fmt`) | -| `type` | Yes | Data type: `boolean`, `integer`, `float`, `string`, `file`, `path` | -| `description` | Yes | Human-readable description | -| `default` | No | Default value | -| `enum` | No | List of allowed values | -| `required` | No | Whether the argument is mandatory (default: false) | - -**Argument Type Behavior:** - -| Type | Config Example | Output | -|------|----------------|--------| -| `boolean` | `tools.bwa.args.Y = true` | `-Y` | -| `integer` | `tools.bwa.args.K = 100000` | `-K 100000` | -| `string` | `tools.bwa.args.R = "@RG\tID:s1"` | `-R @RG\tID:s1` | -| `string` + `enum` | `tools.samtools.args.output_fmt = "cram"` | `--output-fmt cram` | +| `name` | Yes | Parameter identifier | +| `type` | No | Data type: `boolean`, `integer`, `float`, `string`, `file`, `path` | +| `description` | No | Human-readable description | +| `example` | No | Example value for the parameter | #### `input` and `output` @@ -951,9 +921,9 @@ version: "1.0.0" The following attributes from the nf-core meta schema are **not supported** in the Nextflow module system: -| Attribute | Reason | Future | -|-----------|--------|--------| -| `extra_args` | Not adopted in practice by nf-core modules | Will be redesigned as part of the `tools` schema attribute to document tool-specific arguments and configuration options | +| Attribute | Reason | Alternative | +|-----------|--------|-------------| +| `extra_args` | Not adopted in practice by nf-core modules | Use `params` section to define module parameters | | `components` | No longer supported | Module dependencies are managed via `nextflow.config` | ### Complete Examples @@ -1016,6 +986,20 @@ tools: license: ["GPL-3.0-or-later"] identifier: biotools:bwa +params: + - name: batch_size + type: integer + description: "Process INT input bases in each batch" + example: 100000000 + + - name: use_soft_clipping + type: boolean + description: "Use soft clipping for supplementary alignments" + + - name: output_format + type: string + description: "Output format (sam, bam, or cram)" + authors: - "@nf-core" maintainers: diff --git a/adr/module-spec-schema.json b/adr/module-spec-schema.json index 6f54ed6fed..c54d9683b6 100644 --- a/adr/module-spec-schema.json +++ b/adr/module-spec-schema.json @@ -57,43 +57,13 @@ }, "requires": { "type": "object", - "description": "All requirements for the module: runtime environment, plugins, and dependencies", + "description": "Runtime requirements for the module", "properties": { "nextflow": { "type": "string", "description": "Nextflow version constraint using comparison operators", "examples": [">=24.04.0", ">=24.04.0,<25.0.0"], "pattern": "^[<>=!]+[0-9]+\\.[0-9]+\\.[0-9]+(-[a-zA-Z0-9]+)?(,\\s*[<>=!]+[0-9]+\\.[0-9]+\\.[0-9]+(-[a-zA-Z0-9]+)?)*$" - }, - "plugins": { - "type": "array", - "description": "Required Nextflow plugins with optional version constraints", - "items": { - "type": "string", - "description": "Plugin reference in format 'plugin-name' or 'plugin-name@constraint'", - "pattern": "^[a-z][a-z0-9-]*(@[<>=,0-9.]+)?$", - "examples": ["nf-amazon@2.0.0", "nf-wave@>=1.5.0", "nf-azure"] - } - }, - "modules": { - "type": "array", - "description": "Required modules (processes) with optional version constraints", - "items": { - "type": "string", - "description": "Module reference in format '[scope/]name' or '[scope/]name@constraint'", - "pattern": "^([a-z0-9][a-z0-9-]*/)?[a-z][a-z0-9_/-]*(@[<>=,0-9.]+)?$", - "examples": ["nf-core/fastqc@>=1.0.0", "nf-core/samtools/sort@>=2.1.0,<3.0.0", "bwa/mem"] - } - }, - "workflows": { - "type": "array", - "description": "Required workflows/subworkflows with optional version constraints", - "items": { - "type": "string", - "description": "Workflow reference in format '[scope/]name' or '[scope/]name@constraint'", - "pattern": "^([a-z0-9][a-z0-9-]*/)?[a-z][a-z0-9_/-]*(@[<>=,0-9.]+)?$", - "examples": ["nf-core/fastq-align-bwa@1.0.0", "my-subworkflow"] - } } }, "additionalProperties": false @@ -112,6 +82,13 @@ } } }, + "params": { + "type": "array", + "description": "Module parameter specifications", + "items": { + "$ref": "#/$defs/paramSpec" + } + }, "input": { "description": "Input channel specifications for the module's process(es)", "oneOf": [ @@ -232,16 +209,6 @@ "type": "string", "format": "uri", "description": "Manual/user guide URL" - }, - "args": { - "type": "object", - "description": "Command-line arguments supported by the tool. Keys are argument names accessible via tool..args.", - "patternProperties": { - "^[a-zA-Z_][a-zA-Z0-9_]*$": { - "$ref": "#/$defs/toolArgSpec" - } - }, - "additionalProperties": false } }, "required": ["description"], @@ -252,39 +219,29 @@ { "required": ["doi"] } ] }, - "toolArgSpec": { + "paramSpec": { "type": "object", - "description": "Specification for a tool command-line argument", + "description": "Specification for a module parameter", "properties": { - "flag": { + "name": { "type": "string", - "description": "The CLI flag (e.g., '-n', '--output-fmt')", - "pattern": "^--?[a-zA-Z][a-zA-Z0-9_-]*$" + "description": "Parameter identifier", + "pattern": "^[a-zA-Z_][a-zA-Z0-9_]*$" }, "type": { "type": "string", - "description": "Data type of the argument value", + "description": "Data type of the parameter value", "enum": ["boolean", "integer", "float", "string", "file", "path"] }, "description": { "type": "string", - "description": "Human-readable description of the argument" - }, - "default": { - "description": "Default value for the argument" + "description": "Human-readable description of the parameter" }, - "enum": { - "type": "array", - "description": "List of allowed values", - "uniqueItems": true - }, - "required": { - "type": "boolean", - "description": "Whether this argument is required", - "default": false + "example": { + "description": "Example value for the parameter" } }, - "required": ["flag", "type", "description"] + "required": ["name"] }, "channelElementSpec": { "type": "object", @@ -484,51 +441,42 @@ "authors": ["@nf-core"], "maintainers": ["@nf-core"], "requires": { - "nextflow": ">=24.04.0", - "plugins": [ - "nf-amazon@2.0.0" - ], - "modules": [ - "nf-core/samtools/view@>=1.0.0,<2.0.0", - "nf-core/samtools/sort@>=2.1.0,<2.2.0" - ] + "nextflow": ">=24.04.0" }, "tools": [ { "bwa": { "description": "BWA aligner", "homepage": "http://bio-bwa.sourceforge.net/", - "licence": ["GPL-3.0-or-later"], - "args": { - "K": { - "flag": "-K", - "type": "integer", - "description": "Process INT input bases in each batch" - }, - "Y": { - "flag": "-Y", - "type": "boolean", - "description": "Use soft clipping for supplementary alignments" - } - } + "licence": ["GPL-3.0-or-later"] } }, { "samtools": { "description": "SAMtools", "homepage": "http://www.htslib.org/", - "licence": ["MIT"], - "args": { - "output_fmt": { - "flag": "--output-fmt", - "type": "string", - "description": "Output format", - "enum": ["sam", "bam", "cram"] - } - } + "licence": ["MIT"] } } ], + "params": [ + { + "name": "batch_size", + "type": "integer", + "description": "Process INT input bases in each batch", + "example": 100000000 + }, + { + "name": "use_soft_clipping", + "type": "boolean", + "description": "Use soft clipping for supplementary alignments" + }, + { + "name": "output_format", + "type": "string", + "description": "Output format (sam, bam, or cram)" + } + ], "input": [ [ { From 903c7409152b7e8cb7a35936f8613a628b36e35e Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 23 Jan 2026 11:46:16 +0100 Subject: [PATCH 30/75] Update specs + adr [ci skip] Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 56 +++++++++++++++--------------- specs/251117-module-system/plan.md | 2 +- specs/251117-module-system/spec.md | 35 ++++++++++--------- 3 files changed, 47 insertions(+), 46 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 69682454dd..8fe8965896 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -474,34 +474,6 @@ project-root/ **Phase 5**: Advanced features (search UI, language server integration, ontology validation) -## Technical Details - -**Dependency Resolution Flow**: -1. Parse `include` statements → extract module names (e.g., `@nf-core/bwa-align`) -2. For each module: - a. Check `nextflow.config` modules section for declared version - b. Check local `modules/@scope/name/` exists - c. Verify local module integrity against `.checksum` file - d. Apply resolution rules: - - Missing → download declared version from registry - - Exists, checksum valid, same version → use local - - Exists, checksum valid, different version → replace with declared version - - Exists, checksum mismatch → warn and do NOT override (local changes detected) -3. On download: store module to `modules/@scope/name/` with `.checksum` file -4. Read `meta.yaml` file: Validates Nextflow requirement → Fail if not fulfilled -5. Parse module's `main.nf` file → make processes available``` - -**Security**: -- SHA-256 checksum verification on download (stored in `.checksum` file) -- Integrity verification on run (local checksum vs `.checksum` file) -- Authentication required for publishing -- Support for private registries - -**Integration with Plugin System**: -- Both plugins and modules query same registry -- Single authentication system -- Separate cache locations: `$NXF_HOME/plugins/` (global) vs `modules/` (per-project) - ## Module Parameters The module system introduces a structured approach to module configuration through parameters defined in `meta.yaml`. Parameters provide a documented, type-safe way to customize module behavior. @@ -569,6 +541,34 @@ bwa mem ${batch_arg} ${soft_clip} -t $task.cpus $index $reads \ | Clarity | Opaque strings | Named parameters | | Defaults | Manual | Schema-defined | +## Technical Details + +**Dependency Resolution Flow**: +1. Parse `include` statements → extract module names (e.g., `@nf-core/bwa-align`) +2. For each module: + a. Check `nextflow.config` modules section for declared version + b. Check local `modules/@scope/name/` exists + c. Verify local module integrity against `.checksum` file + d. Apply resolution rules: + - Missing → download declared version from registry + - Exists, checksum valid, same version → use local + - Exists, checksum valid, different version → replace with declared version + - Exists, checksum mismatch → warn and do NOT override (local changes detected) +3. On download: store module to `modules/@scope/name/` with `.checksum` file +4. Read `meta.yaml` file: Validates Nextflow requirement → Fail if not fulfilled +5. Parse module's `main.nf` file → make processes available``` + +**Security**: +- SHA-256 checksum verification on download (stored in `.checksum` file) +- Integrity verification on run (local checksum vs `.checksum` file) +- Authentication required for publishing +- Support for private registries + +**Integration with Plugin System**: +- Both plugins and modules query same registry +- Single authentication system +- Separate cache locations: `$NXF_HOME/plugins/` (global) vs `modules/` (per-project) + ## Comparison: Plugins vs. Modules | Aspect | Plugins | Modules | diff --git a/specs/251117-module-system/plan.md b/specs/251117-module-system/plan.md index 8fc08ac793..0d12cf3f96 100644 --- a/specs/251117-module-system/plan.md +++ b/specs/251117-module-system/plan.md @@ -5,7 +5,7 @@ ## Summary -Implement client-side module system for Nextflow enabling pipeline developers to include remote modules from the Nextflow registry using `@scope/name` syntax, manage versions via `nextflow.config`, and use CLI commands (install, search, list, remove, publish, run). Implementation extends existing DSL parser, config parser, and follows plugin system patterns for registry communication and authentication. +Implement client-side module system for Nextflow enabling pipeline developers to include remote modules from the Nextflow registry using `@scope/name` syntax, manage versions via `nextflow.config`, configure module parameters via `meta.yaml`, and use CLI commands (install, search, list, remove, publish, run). Implementation extends existing DSL parser, config parser, and follows plugin system patterns for registry communication and authentication. ## Technical Context diff --git a/specs/251117-module-system/spec.md b/specs/251117-module-system/spec.md index 73d5d21427..652397bb80 100644 --- a/specs/251117-module-system/spec.md +++ b/specs/251117-module-system/spec.md @@ -11,7 +11,7 @@ This specification covers the **Nextflow client-side implementation** of the mod - Include remote modules from the Nextflow registry using `@scope/name` syntax - Manage module versions through `nextflow.config` - Use CLI commands to install, search, list, remove, publish, and run modules -- Configure tool arguments through structured `meta.yaml` definitions +- Configure module parameters through structured `meta.yaml` definitions **Out of Scope**: Registry backend implementation (assumed already available at `registry.nextflow.io`) @@ -44,24 +44,24 @@ A user wants to run a module directly from the command line without writing a wr **Acceptance Scenarios**: 1. **Given** a module is available (locally or in registry), **When** user runs `nextflow module run nf-core/fastqc --input 'data/*.fastq'`, **Then** the module is executed with the provided inputs mapped to process parameters -2. **Given** a module with tool arguments defined in `meta.yaml`, **When** user runs `nextflow module run nf-core/bwa-align --tools:bwa:K 100000`, **Then** the tool argument is validated and passed to the process +2. **Given** a module with parameters defined in `meta.yaml`, **When** user runs `nextflow module run nf-core/bwa-align --batch_size 100000`, **Then** the parameter is validated and passed to the process 3. **Given** a module is not installed locally, **When** user runs `nextflow module run nf-core/salmon`, **Then** the module is automatically downloaded before execution --- -### User Story 3 - Structured Tool Arguments (Priority: P1) +### User Story 3 - Module Parameters (Priority: P1) -A module author wants to define typed, documented tool arguments that replace the legacy `ext.args` pattern. +A module author wants to define typed, documented parameters that provide a clear interface for module customization. -**Why this priority**: Critical for module usability - provides type-safe, documented arguments that enable IDE autocompletion and validation, replacing the opaque `ext.args` pattern. +**Why this priority**: Critical for module usability - provides type-safe, documented parameters that enable IDE autocompletion and validation, replacing the opaque `ext.args` pattern. -**Independent Test**: Can be tested by configuring `tools.bwa.args.K = 100000` in config and verifying the argument is applied in the script. +**Independent Test**: Can be tested by configuring `params.batch_size = 100000` in config and verifying the parameter is applied in the script. **Acceptance Scenarios**: -1. **Given** a module with `tools.*.args` defined in `meta.yaml`, **When** user configures `tools.bwa.args.K = 100000` in config, **Then** the argument is accessible in scripts as `tools.bwa.args.K` returning `-K 100000` -2. **Given** all tool arguments are configured, **When** script uses `${tools.bwa.args}`, **Then** all configured arguments are concatenated in the output -3. **Given** an argument with enum validation, **When** user provides an invalid value, **Then** a validation error is displayed +1. **Given** a module with `params` defined in `meta.yaml`, **When** user configures `params.batch_size = 100000` in config, **Then** the parameter is accessible in scripts via `params.batch_size` +2. **Given** a parameter with type validation, **When** user provides an invalid value type, **Then** a validation error is displayed +3. **Given** a module with documented parameters, **When** user runs `nextflow module run --help`, **Then** available parameters with descriptions are listed --- @@ -200,12 +200,12 @@ A module author wants to publish their module to the Nextflow registry for other - **FR-022**: System MUST support `NXF_REGISTRY_TOKEN` environment variable for authentication - **FR-023**: System MUST support multiple registry URLs with fallback ordering -#### Tool Arguments +#### Module Parameters -- **FR-024**: System MUST provide `tools..args.` implicit variable in module scripts -- **FR-025**: System MUST validate tool arguments against `meta.yaml` schema (type, enum) at workflow parse time -- **FR-026**: System MUST support boolean, integer, float, string, file, and path argument types -- **FR-027**: System MUST concatenate all tool arguments when `tools..args` is accessed +- **FR-024**: System MUST parse module parameters from `params` section in `meta.yaml` +- **FR-025**: System MUST validate module parameters against `meta.yaml` schema (type) at workflow parse time +- **FR-026**: System MUST support boolean, integer, float, string, file, and path parameter types +- **FR-027**: System MUST make module parameters accessible via standard `params` variable in scripts #### Registry Communication @@ -217,7 +217,8 @@ A module author wants to publish their module to the Nextflow registry for other - **Module**: A reusable Nextflow process definition with `main.nf` entry point, optional `meta.yaml` manifest, and README documentation - **Module Reference**: A scoped identifier (`@scope/name`) pointing to a registry module -- **Module Manifest (meta.yaml)**: YAML file containing module metadata, version, dependencies, tool arguments schema +- **Module Manifest (meta.yaml)**: YAML file containing module metadata, version, dependencies, and parameter definitions +- **Module Parameter**: A configurable parameter defined in `meta.yaml` with name, optional type, description, and example - **Checksum File (.checksum)**: Local cache of registry checksum for integrity verification - **Registry Configuration**: Settings for registry URL, authentication, and fallback ordering @@ -242,7 +243,7 @@ A module author wants to publish their module to the Nextflow registry for other - The `modules/` directory is intended to be committed to the pipeline's git repository - Version constraints in `meta.yaml` follow the same syntax as existing Nextflow plugin version constraints - SHA-256 is used for all checksum operations -- Tool arguments CLI syntax uses colon-separated format: `--tools::` +- Module parameters use standard `--` CLI syntax ## Dependencies @@ -256,4 +257,4 @@ A module author wants to publish their module to the Nextflow registry for other ### Session 2026-01-19 - Q: What should happen when incompatible dependency versions are detected? → A: Use highest compatible version automatically, warn if none exists -- Q: When should tool argument validation occur? → A: At workflow parse time (early, before any execution) \ No newline at end of file +- Q: When should module parameter validation occur? → A: At workflow parse time (early, before any execution) \ No newline at end of file From 82d90d6cfb27cd90f858a5845340dabeb294a493 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Sun, 25 Jan 2026 11:56:05 -0600 Subject: [PATCH 31/75] Resolve some conflicts with nextflow-io/schemas#10 Signed-off-by: Ben Sherman --- adr/20251114-module-system.md | 8 +- adr/module-spec-schema.json | 210 ++++++++++++---------------------- 2 files changed, 74 insertions(+), 144 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 8fe8965896..4608b3099b 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -123,7 +123,7 @@ registry { } ``` -**Module Manifest** (`meta.yaml`): +**Module Spec** (`meta.yaml`): ```yaml name: nf-core/bwa-align version: 1.2.4 # This module's version @@ -464,7 +464,7 @@ project-root/ ## Implementation Strategy -**Phase 1**: Module manifest schema, local module loading, validation tools +**Phase 1**: Module schema, local module loading, validation tools **Phase 2**: Extend Nextflow registry for modules, implement caching, add `install` and `search` commands @@ -576,7 +576,7 @@ bwa mem ${batch_arg} ${soft_clip} -t $task.cpus $index $reads \ | Purpose | Extend runtime | Reusable processes | | Format | JAR files | Source code (.nf) | | Resolution | Startup | Parse time | -| Metadata | JSON spec | YAML manifest | +| Metadata | JSON spec | YAML spec | | Naming | `nf-amazon` | `@nf-core/salmon` | | Cache Location | `$NXF_HOME/plugins/` | `modules/@scope/name/` | | Version Config | `plugins {}` in config | `modules {}` in config | @@ -647,7 +647,7 @@ bwa mem ${batch_arg} ${soft_clip} -t $task.cpus $index $reads \ --- -## Appendix A: Module Metadata Schema Specification +## Appendix A: Module Schema Specification This appendix defines the JSON schema for module `meta.yaml` files. The schema maintains backward compatibility with existing nf-core module metadata patterns while supporting the new Nextflow module system features. diff --git a/adr/module-spec-schema.json b/adr/module-spec-schema.json index c54d9683b6..ad972df973 100644 --- a/adr/module-spec-schema.json +++ b/adr/module-spec-schema.json @@ -1,7 +1,7 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://registry.nextflow.io/schemas/module-spec/v1.0.0", - "title": "Nextflow Module Metadata Schema", + "$id": "https://raw.githubusercontent.com/nextflow-io/schemas/main/module/v1/schema.json", + "title": "Nextflow Module Schema", "description": "Schema for Nextflow module meta.yaml files, supporting both nf-core community patterns and the Nextflow module system", "type": "object", "properties": { @@ -68,18 +68,11 @@ }, "additionalProperties": false }, - "tools": { + "input": { "type": "array", - "description": "Software tools wrapped by this module with their metadata", + "description": "Inputs of the module", "items": { - "type": "object", - "minProperties": 1, - "maxProperties": 1, - "patternProperties": { - "^[a-zA-Z][a-zA-Z0-9_-]*$": { - "$ref": "#/$defs/toolSpec" - } - } + "$ref": "#/$defs/structuredParameter" } }, "params": { @@ -89,52 +82,33 @@ "$ref": "#/$defs/paramSpec" } }, - "input": { - "description": "Input channel specifications for the module's process(es)", - "oneOf": [ - { - "type": "array", - "description": "Array-based input specification (nf-core modules pattern)", - "items": { - "$ref": "#/$defs/inputChannelItem" - } - }, - { - "type": "object", - "description": "Object-based input specification (simplified pattern)", - "patternProperties": { - "^[a-zA-Z_][a-zA-Z0-9_]*$": { - "$ref": "#/$defs/channelElementSpec" - } - } - } - ] - }, "output": { - "description": "Output channel specifications for the module's process(es)", - "oneOf": [ - { - "type": "object", - "description": "Object-based output specification (nf-core modules pattern)", - "patternProperties": { - "^[a-zA-Z_][a-zA-Z0-9_]*$": { - "$ref": "#/$defs/outputChannelDef" - } - } - }, - { - "type": "array", - "description": "Array-based output specification (nf-core subworkflows pattern)", - "items": { - "type": "object", - "patternProperties": { - "^[a-zA-Z_][a-zA-Z0-9_]*$": { - "$ref": "#/$defs/channelElementSpec" - } - } + "type": "array", + "description": "Outputs of the module", + "items": { + "$ref": "#/$defs/structuredParameter" + } + }, + "topics": { + "type": "array", + "description": "Topics of the module", + "items": { + "$ref": "#/$defs/structuredParameter" + } + }, + "tools": { + "type": "array", + "description": "Software tools wrapped by this module with their metadata", + "items": { + "type": "object", + "minProperties": 1, + "maxProperties": 1, + "patternProperties": { + "^[a-zA-Z][a-zA-Z0-9_-]*$": { + "$ref": "#/$defs/toolSpec" } } - ] + } } }, "required": ["name", "description"], @@ -213,12 +187,36 @@ }, "required": ["description"], "anyOf": [ - { "required": ["homepage"] }, - { "required": ["documentation"] }, - { "required": ["tool_dev_url"] }, - { "required": ["doi"] } + { + "required": ["homepage"] + }, + { + "required": ["documentation"] + }, + { + "required": ["tool_dev_url"] + }, + { + "required": ["doi"] + } ] }, + "structuredParameter": { + "type": "array", + "items": { + "oneOf": [ + { + "$ref": "#/$defs/paramSpec" + }, + { + "type": "array", + "items": { + "$ref": "#/$defs/paramSpec" + } + } + ] + } + }, "paramSpec": { "type": "object", "description": "Specification for a module parameter", @@ -231,43 +229,30 @@ "type": { "type": "string", "description": "Data type of the parameter value", - "enum": ["boolean", "integer", "float", "string", "file", "path"] + "enum": [ + "boolean", + "float", + "integer", + "string", + "list", + "map", + "file", + "directory" + ] }, "description": { "type": "string", "description": "Human-readable description of the parameter" }, - "example": { - "description": "Example value for the parameter" - } - }, - "required": ["name"] - }, - "channelElementSpec": { - "type": "object", - "description": "Specification for a channel element (input or output)", - "properties": { - "type": { - "type": "string", - "description": "Data type of the channel element", - "enum": ["map", "file", "directory", "string", "integer", "float", "boolean", "list", "val"] - }, - "description": { - "type": "string", - "description": "Human-readable description of the channel element" - }, "pattern": { "type": "string", - "description": "File pattern in glob syntax or allowed values pattern" + "description": "Glob pattern for file/directory parameters" }, "optional": { "type": "boolean", - "description": "Whether this input is optional", + "description": "Whether this parameter is optional", "default": false }, - "default": { - "description": "Default value if not provided" - }, "enum": { "type": "array", "description": "List of allowed values", @@ -289,62 +274,7 @@ "uniqueItems": true } }, - "required": ["description"] - }, - "inputChannelItem": { - "description": "Input channel item - can be a tuple (array) or single element (object)", - "oneOf": [ - { - "type": "array", - "description": "Tuple-style input channel (multiple elements per emission)", - "items": { - "type": "object", - "patternProperties": { - "^[a-zA-Z_][a-zA-Z0-9_]*$|^\\*\\..*$": { - "$ref": "#/$defs/channelElementSpec" - } - } - } - }, - { - "type": "object", - "description": "Single-element input channel", - "patternProperties": { - "^[a-zA-Z_][a-zA-Z0-9_]*$": { - "$ref": "#/$defs/channelElementSpec" - } - } - } - ] - }, - "outputChannelDef": { - "type": "array", - "description": "Output channel definition - array of emission patterns", - "items": { - "oneOf": [ - { - "type": "object", - "description": "Single output element", - "patternProperties": { - "^[a-zA-Z_$][a-zA-Z0-9_{}.$*\"']*$": { - "$ref": "#/$defs/channelElementSpec" - } - } - }, - { - "type": "array", - "description": "Tuple output (multiple elements per emission)", - "items": { - "type": "object", - "patternProperties": { - "^[a-zA-Z_$][a-zA-Z0-9_{}.$*\"']*$|^\\*\\..*$": { - "$ref": "#/$defs/channelElementSpec" - } - } - } - } - ] - } + "required": ["name", "type", "description"] } }, "allOf": [ From d86c2112efb0f7dbddd4e865c588b645653e0080 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Mon, 26 Jan 2026 12:07:23 +0100 Subject: [PATCH 32/75] Update adr/20251114-module-system.md [ci skip] Co-authored-by: Ben Sherman Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 4608b3099b..9d95e42433 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -263,7 +263,7 @@ nextflow module run nf-core/salmon \ --reads reads.fq \ --index salmon_index \ -work-dir /tmp/work \ - --outdir results/ + -output-dir results/ # Run with module parameters nextflow module run nf-core/bwa-align \ From 8fa6478c407c8f53e1bf476c2edc0272f4067da1 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 28 Jan 2026 11:56:09 +0100 Subject: [PATCH 33/75] Extract module parameters to separate spec [ci skip] Co-Authored-By: Claude Opus 4.5 Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 138 ++-------------------------------- 1 file changed, 7 insertions(+), 131 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 9d95e42433..3468e86207 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -4,10 +4,13 @@ - Status: draft - Date: 2025-01-06 - Tags: modules, dsl, registry, versioning, architecture -- Version: 2.5 +- Version: 2.6 ## Updates +### Version 2.6 (2026-01-28) +- **Removed module parameters**: Module parameters specification moved to separate spec document. + ### Version 2.5 (2026-01-23) - **Module parameters**: Replaced structured tool arguments with general module parameters defined in `meta.yaml` - **Simplified tools section**: Removed `args` property from tools; tool arguments now configured via module parameters @@ -222,16 +225,14 @@ Run a module directly without requiring a wrapper workflow script. This command **Options**: - `-version `: Run a specific version (default: latest or configured version) - `-- `: Map value to the corresponding module process input channel -- `-- `: Configure module parameters (validated against meta.yaml schema) - All standard `nextflow run` options (e.g., `-profile`, `-work-dir`, `-resume`, etc.) **Behavior**: 1. Checks if module is installed locally; if not, downloads from registry 2. Parses the module's `main.nf` to identify the main process and its input declarations 3. Validates command-line arguments against the process input schema -4. Validates parameters against the `params` schema in `meta.yaml` -5. Generates an implicit workflow that wires CLI arguments to process inputs -6. Executes the workflow using standard Nextflow runtime +4. Generates an implicit workflow that wires CLI arguments to process inputs +5. Executes the workflow using standard Nextflow runtime **Input Mapping**: - Named arguments (`--reads`, `--reference`) are mapped to corresponding process inputs @@ -239,12 +240,6 @@ Run a module directly without requiring a wrapper workflow script. This command - Multiple values can be provided for inputs expecting collections - Required inputs without defaults must be provided; optional inputs use declared defaults -**Module Parameters**: -- Parameters defined in `meta.yaml` can be configured via CLI arguments -- Boolean flags can be specified without value (e.g., `--use_soft_clipping`) -- Arguments are validated against the `params` schema in `meta.yaml` -- Invalid parameter names or values that fail type validation produce errors - **Example**: ```bash # Run BWA alignment module with input files @@ -264,14 +259,6 @@ nextflow module run nf-core/salmon \ --index salmon_index \ -work-dir /tmp/work \ -output-dir results/ - -# Run with module parameters -nextflow module run nf-core/bwa-align \ - --reads 'samples/*_{1,2}.fastq.gz' \ - --reference genome.fa \ - --batch_size 100000000 \ - --use_soft_clipping \ - --output_format cram ``` --- @@ -474,73 +461,6 @@ project-root/ **Phase 5**: Advanced features (search UI, language server integration, ontology validation) -## Module Parameters - -The module system introduces a structured approach to module configuration through parameters defined in `meta.yaml`. Parameters provide a documented, type-safe way to customize module behavior. - -### Parameter Definition - -Modules declare available parameters in `meta.yaml` under the `params` section. Each parameter has a name and optional attributes for type and description. - -```yaml -params: - - name: batch_size - type: integer - description: "Process INT input bases in each batch" - example: 100000000 - - - name: use_soft_clipping - type: boolean - description: "Use soft clipping for supplementary alignments" - - - name: output_format - type: string - description: "Output format (sam, bam, or cram)" -``` - -### Parameter Attributes - -| Attribute | Required | Description | -|-----------|----------|-------------| -| `name` | Yes | Parameter identifier | -| `type` | No | Data type: `boolean`, `integer`, `float`, `string`, `file`, `path` | -| `description` | No | Human-readable description | -| `example` | No | Example value for the parameter | - -### Configuration Usage - -Parameters are configured using standard Nextflow params syntax: - -```groovy -withName: 'BWA_MEM' { - params.batch_size = 100000000 - params.use_soft_clipping = true - params.output_format = "cram" -} -``` - -### Script Usage - -In module scripts, access parameters via the standard `params` variable: - -```groovy -def batch_arg = params.batch_size ? "-K ${params.batch_size}" : '' -def soft_clip = params.use_soft_clipping ? "-Y" : '' - -bwa mem ${batch_arg} ${soft_clip} -t $task.cpus $index $reads \ - | samtools sort --output-fmt ${params.output_format ?: 'bam'} -o ${prefix}.bam - -``` - -### Benefits - -| Aspect | `ext.args` (Legacy) | Module Parameters (New) | -|--------|---------------------|-------------------------| -| Documentation | None | In meta.yaml | -| Type Safety | None | Validated | -| IDE Support | None | Autocompletion | -| Clarity | Opaque strings | Named parameters | -| Defaults | Manual | Schema-defined | - ## Technical Details **Dependency Resolution Flow**: @@ -681,7 +601,6 @@ These fields extend the schema to support the new Nextflow module system: | `license` | string | Registry | SPDX license identifier for module code | | `requires` | object | Optional | Runtime requirements | | `requires.nextflow` | string | Optional | Nextflow version constraint | -| `params` | array[object] | Optional | Module parameter specifications | ### Detailed Field Specifications @@ -763,35 +682,6 @@ tools: | `identifier` | Recommended | bio.tools identifier | | `manual` | No | User manual URL | -#### `params` - -Defines configurable parameters for the module: - -```yaml -params: - - name: batch_size - type: integer - description: "Process INT input bases in each batch" - example: 100000000 - - - name: use_soft_clipping - type: boolean - description: "Use soft clipping for supplementary alignments" - - - name: output_format - type: string - description: "Output format (sam, bam, or cram)" -``` - -**Parameter Properties:** - -| Property | Required | Description | -|----------|----------|-------------| -| `name` | Yes | Parameter identifier | -| `type` | No | Data type: `boolean`, `integer`, `float`, `string`, `file`, `path` | -| `description` | No | Human-readable description | -| `example` | No | Example value for the parameter | - #### `input` and `output` The schema supports both nf-core patterns to ensure backward compatibility: @@ -923,7 +813,7 @@ The following attributes from the nf-core meta schema are **not supported** in t | Attribute | Reason | Alternative | |-----------|--------|-------------| -| `extra_args` | Not adopted in practice by nf-core modules | Use `params` section to define module parameters | +| `extra_args` | Not adopted in practice by nf-core modules | To be defined | | `components` | No longer supported | Module dependencies are managed via `nextflow.config` | ### Complete Examples @@ -986,20 +876,6 @@ tools: license: ["GPL-3.0-or-later"] identifier: biotools:bwa -params: - - name: batch_size - type: integer - description: "Process INT input bases in each batch" - example: 100000000 - - - name: use_soft_clipping - type: boolean - description: "Use soft clipping for supplementary alignments" - - - name: output_format - type: string - description: "Output format (sam, bam, or cram)" - authors: - "@nf-core" maintainers: From 1d6005b35bac96315d646bac7d52acf2cff6e334 Mon Sep 17 00:00:00 2001 From: Abi Date: Thu, 26 Feb 2026 20:59:00 +0530 Subject: [PATCH 34/75] docs: clarify path name for staged task inputs (#6869) Co-authored-by: Ben Sherman --- docs/reference/stdlib-types.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/stdlib-types.md b/docs/reference/stdlib-types.md index 6c91752015..8a690f6943 100644 --- a/docs/reference/stdlib-types.md +++ b/docs/reference/stdlib-types.md @@ -521,6 +521,7 @@ The following properties are available: `name: String` : The path name, e.g. `/some/path/file.txt` -> `file.txt`. +: For files staged as a task input, the path name is the path relative to the task directory (e.g., `my-dir/file.txt`). Use `fileName.name` for task paths to get only the file name. `parent: Path` : The path parent path, e.g. `/some/path/file.txt` -> `/some/path`. From d9952fba78136e152146839579e772040112850c Mon Sep 17 00:00:00 2001 From: Peter Kneale Date: Fri, 27 Feb 2026 05:06:43 +1100 Subject: [PATCH 35/75] Add devcontainer (#6792) Co-authored-by: Phil Ewels Co-authored-by: Ben Sherman --- .devcontainer/devcontainer.json | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 .devcontainer/devcontainer.json diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json new file mode 100644 index 0000000000..402dbad785 --- /dev/null +++ b/.devcontainer/devcontainer.json @@ -0,0 +1,16 @@ +// README at: https://github.com/devcontainers/templates/tree/main/src/java +{ + "name": "Nextflow Java dev environment", + "image": "mcr.microsoft.com/devcontainers/base:bullseye", + + "features": { + "ghcr.io/devcontainers/features/java:1": { + "version": "21", + "jdkDistro": "tem", + "installGradle": "true", + "gradleVersion": "9.3.1", + "installGroovy": "true", + "groovyVersion": "4.0.30" + } + } +} From 481b90fa59f9db20b7100926cfaeb7c316f40fe7 Mon Sep 17 00:00:00 2001 From: Peter Kneale Date: Fri, 27 Feb 2026 05:23:32 +1100 Subject: [PATCH 36/75] Fix vertical alignment in timeline report (#6794) --- .../src/main/resources/nextflow/trace/assets/d3-timeline.js | 2 +- .../main/resources/nextflow/trace/assets/d3-timeline.min.js | 2 +- .../src/test/resources/nextflow/trace/timeline-expected.html | 3 +-- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/modules/nextflow/src/main/resources/nextflow/trace/assets/d3-timeline.js b/modules/nextflow/src/main/resources/nextflow/trace/assets/d3-timeline.js index 95976c5e27..8f9cf9dafa 100644 --- a/modules/nextflow/src/main/resources/nextflow/trace/assets/d3-timeline.js +++ b/modules/nextflow/src/main/resources/nextflow/trace/assets/d3-timeline.js @@ -154,7 +154,7 @@ var appendLabel = function (gParent, yAxisMapping, index, hasLabel, datum) { var fullItemHeight = itemHeight + itemMargin; - var rowsDown = fullItemHeight + fullItemHeight * (yAxisMapping[index] || 1); + var rowsDown = margin.top + (fullItemHeight/2) + fullItemHeight * (yAxisMapping[index] || 1); gParent.append("text") .attr("class", "timeline-label") diff --git a/modules/nextflow/src/main/resources/nextflow/trace/assets/d3-timeline.min.js b/modules/nextflow/src/main/resources/nextflow/trace/assets/d3-timeline.min.js index 7e394a303b..17180656ca 100644 --- a/modules/nextflow/src/main/resources/nextflow/trace/assets/d3-timeline.min.js +++ b/modules/nextflow/src/main/resources/nextflow/trace/assets/d3-timeline.min.js @@ -1 +1 @@ -!function(){var t=d3.scale.category20c().domain(d3.range(0,20)).range();var n="#9c9c9c",e="#bdbdbd";d3.timeline=function(){var r=["circle","rect"],i=function(){},a=function(){},o=function(){},c=function(){},l=function(){},u=function(t){return t},s=function(){},f=function(){},g="bottom",h=null,d=null,m=null,p=null,x={format:d3.time.format("%I %p"),tickTime:d3.time.hours,tickInterval:1,tickSize:6},v=d3.scale.category20(),w=null,k="rect",y=0,_=0,T=0,b={left:30,right:30,top:30,bottom:30},F=!1,B=!1,A=!1,E=!1,S=20,I=5,Y=60,z=!0,M=!1,C=!1,O=!1,H={stroke:"stroke-dasharray",spacing:"4 10"},L={marginTop:25,marginBottom:0,width:1,color:v},R=!1,j={marginTop:25,marginBottom:0,width:1,color:v},D=!1,N=!1,P=!1,W="white",q={},G=function(t,n,e){D&&K(t,0,0),N&&J(t);t.append("g").attr("class","axis").attr("transform","translate(0,"+e+")").call(n)},J=function(t){var n=b.left-Y,e=(h-b.left)/6,r=h-b.right-e+Y,i=t.append("g").attr("class","axis").attr("transform","translate(0, 20)");P&&function(t){var n=y.getFullYear();y.getFullYear()!=T.getFullYear()&&(n=y.getFullYear()+"-"+T.getFullYear()),t.append("text").attr("transform","translate(20, 0)").attr("x",0).attr("y",14).attr("class","calendarYear").text(n)}(i),i.append("text").attr("transform","translate("+n+", 0)").attr("x",0).attr("y",14).attr("class","chevron").text("<").on("click",function(){return s(y,q)}),i.append("text").attr("transform","translate("+r+", 0)").attr("x",0).attr("y",14).attr("class","chevron").text(">").on("click",function(){return f(T,q)})},K=function(t,n,e){t.insert("rect").attr("class","row-green-bar").attr("x",n).attr("width",h).attr("y",e).attr("height",S).attr("fill",W)},Q=function(t,n,e){t.append("g").attr("class","axis").attr("transform","translate(0,"+(b.top+(S+I)*e)+")").attr(H.stroke,H.spacing).call(n.tickFormat("").tickSize(-(b.top+(S+I)*(e-1)+3),0,0))},U=function(t,n,e,r,i){var a=(S+I)*t[n]+b.top;e.selectAll("svg").data(r).enter().insert("rect").attr("class","row-green-bar").attr("x",E?0:b.left).attr("width",E?h:h-b.right-b.left).attr("y",a).attr("height",S).attr("fill",p instanceof Function?p(i,n):p)},V=function(t,n,e,r,i){var a=S+I,o=a+a*(n[e]||1);t.append("text").attr("class","timeline-label").attr("transform","translate("+_+","+o+")").text(r?u(i.label):i.id).on("click",function(t,n){c(t,e,i)})};function X(r){var u=r.append("g"),s=r[0][0].getBoundingClientRect(),f=d3.select(r[0][0]),v={},w=1,_=0,E=0;!function(){if(h||s.width){if(!h||!s.width)try{h=f.attr("width")}catch(t){console.log(t)}}else try{if(!(h=f.attr("width")))throw"width of the timeline is not set. As of Firefox 27, timeline().with(x) needs to be explicitly set in order to render"}catch(t){console.log(t)}}(),A&&u.each(function(t,n){t.forEach(function(t,n){t.times.forEach(function(t,e){0===n&&0===e?(originTime=t.starting_time,t.starting_time=0,t.ending_time=t.ending_time-originTime):(t.starting_time=t.starting_time-originTime,t.ending_time=t.ending_time-originTime)})})}),(F||0===T||0===y)&&(u.each(function(t,n){t.forEach(function(t,n){F&&-1==Object.keys(v).indexOf(n)&&(v[n]=w,w++),t.times.forEach(function(t,n){0===y&&(t.starting_time<_||0===_&&!1===A)&&(_=t.starting_time),0===T&&t.ending_time>E&&(E=t.ending_time)})})}),0===T&&(T=E),0===y&&(y=_));var Y=1/(T-y)*(h-b.left-b.right),H=d3.time.scale().domain([y,T]).range([b.left,h-b.right]),D=d3.svg.axis().scale(H).orient(g).tickFormat(x.format).ticks(x.numTicks||x.tickTime,x.tickInterval).tickSize(x.tickSize);u.each(function(l,s){q=l,l.forEach(function(l,s){var f=l.cached,g=l.index,d=l.times,x=void 0!==l.label;if(void 0!==l.id&&console.warn("d3Timeline Warning: Ids per dataset is deprecated in favor of a 'class' key. Ids are now per data element."),p&&U(v,s,u,d,l),u.selectAll("svg").data(d).enter().append(function(t,n){return document.createElementNS(d3.ns.prefix.svg,"display"in t?t.display:k)}).attr("x",K).attr("y",y).attr("width",function(t,n){return(t.ending_time-t.starting_time)*Y}).attr("cy",function(t,n){return y(t,n)+S/2}).attr("cx",K).attr("r",S/2).attr("height",S).style("fill",function(r,i){return 0==i?e:1==i?f?n:t[g%16]:e}).on("mousemove",function(t,n){i(t,s,l)}).on("mouseover",function(t,n){a(t,n,l)}).on("mouseout",function(t,n){o(t,n,l)}).on("click",function(t,n){c(t,s,l)}).attr("class",function(t,n){return l.class?"timelineSeries_"+l.class:"timelineSeries_"+s}).attr("id",function(t,n){return l.id&&!t.id?"timelineItem_"+l.id:t.id?t.id:"timelineItem_"+s+"_"+n}),u.selectAll("svg").data(d).enter().append("text").attr("x",X).attr("y",function(t,n){if(F)return b.top+(S+I)*v[s]+.75*S;return b.top+.75*S}).text(function(t){return t.label}),m){var w=S+I/2+b.top+(S+I)*v[s];r.append("svg:line").attr("class","row-seperator").attr("x1",0+b.left).attr("x2",h-b.right).attr("y1",w).attr("y2",w).attr("stroke-width",1).attr("stroke",m)}function y(t,n){return F?b.top+(S+I)*v[s]:b.top}x&&V(r,v,s,x,l),void 0!==l.icon&&r.append("image").attr("class","timeline-label").attr("transform","translate(0,"+(b.top+(S+I)*v[s])+")").attr("xlink:href",l.icon).attr("width",b.left).attr("height",S)})});var N=b.top+(S+I)*w,P=b.top;if(z&&G(u,D,M?P:N),O&&Q(u,D,w),h>s.width){var W=d3.behavior.zoom().x(H).on("zoom",function(){var t=Math.min(0,Math.max(s.width-h,d3.event.translate[0]));W.translate([t,0]),u.attr("transform","translate("+t+",0)"),l(t*Y,H)});r.attr("class","scrollable").call(W)}B&&u.selectAll(".tick text").attr("transform",function(t){return"rotate("+B+")translate("+(this.getBBox().width/2+10)+","+this.getBBox().height/2+")"});var J=u[0][0].getBoundingClientRect();(function(){if(d||f.attr("height"))d?f.attr("height",d):d=f.attr("height");else{if(!S)throw"height of the timeline is not set";d=J.height+J.top-s.top,d3.select(r[0][0]).attr("height",d)}}(),R&&u.each(function(t,n){t.forEach(function(t){t.times.forEach(function(t){Z(H(t.starting_time),j),Z(H(t.ending_time),j)})})}),C)&&Z(H(new Date),L);function K(t,n){return b.left+(t.starting_time-y)*Y}function X(t,n){return b.left+(t.starting_time-y)*Y+5}function Z(t,n){r.append("svg:line").attr("x1",t).attr("y1",n.marginTop).attr("x2",t).attr("y2",d-n.marginBottom).style("stroke",n.color).style("stroke-width",n.width)}}return X.margin=function(t){return arguments.length?(b=t,X):b},X.orient=function(t){return arguments.length?(g=t,X):g},X.itemHeight=function(t){return arguments.length?(S=t,X):S},X.itemMargin=function(t){return arguments.length?(I=t,X):I},X.navMargin=function(t){return arguments.length?(Y=t,X):Y},X.height=function(t){return arguments.length?(d=t,X):d},X.width=function(t){return arguments.length?(h=t,X):h},X.display=function(t){return arguments.length&&-1!=r.indexOf(t)?(k=t,X):k},X.labelFormat=function(t){return arguments.length?(u=t,X):null},X.tickFormat=function(t){return arguments.length?(x=t,X):x},X.hover=function(t){return arguments.length?(i=t,X):i},X.mouseover=function(t){return arguments.length?(a=t,X):t},X.mouseout=function(t){return arguments.length?(o=t,X):t},X.click=function(t){return arguments.length?(c=t,X):c},X.scroll=function(t){return arguments.length?(l=t,X):l},X.colors=function(t){return arguments.length?(v=t,X):v},X.beginning=function(t){return arguments.length?(y=t,X):y},X.ending=function(t){return arguments.length?(T=t,X):T},X.labelMargin=function(t){return arguments.length?(_=t,X):T},X.rotateTicks=function(t){return B=t,X},X.stack=function(){return F=!F,X},X.relativeTime=function(){return A=!A,X},X.showBorderLine=function(){return R=!R,X},X.showBorderFormat=function(t){return arguments.length?(j=t,X):j},X.showToday=function(){return C=!C,X},X.showTodayFormat=function(t){return arguments.length?(L=t,X):L},X.colorProperty=function(t){return arguments.length?(w=t,X):w},X.rowSeperators=function(t){return arguments.length?(m=t,X):m},X.background=function(t){return arguments.length?(p=t,X):p},X.showTimeAxis=function(){return z=!z,X},X.showAxisTop=function(){return M=!M,X},X.showAxisCalendarYear=function(){return P=!P,X},X.showTimeAxisTick=function(){return O=!O,X},X.fullLengthBackgrounds=function(){return E=!E,X},X.showTimeAxisTickFormat=function(t){return arguments.length?(H=t,X):H},X.showAxisHeaderBackground=function(t){return D=!D,t&&(W=t),X},X.navigate=function(t,n){return s=t,f=n,N=!N,X},X}}(); +!function(){var t=d3.scale.category20c().domain(d3.range(0,20)).range();var n="#bdbdbd";d3.timeline=function(){var e=["circle","rect"],r=function(){},i=function(){},a=function(){},o=function(){},c=function(){},l=function(t){return t},u=function(){},s=function(){},f="bottom",g=null,h=null,d=null,m=null,p={format:d3.time.format("%I %p"),tickTime:d3.time.hours,tickInterval:1,tickSize:6},x=d3.scale.category20(),v=null,w="rect",k=0,y=0,_=0,T={left:30,right:30,top:30,bottom:30},b=!1,F=!1,B=!1,A=!1,E=20,S=5,I=60,Y=!0,z=!1,M=!1,C=!1,O={stroke:"stroke-dasharray",spacing:"4 10"},H={marginTop:25,marginBottom:0,width:1,color:x},L=!1,R={marginTop:25,marginBottom:0,width:1,color:x},j=!1,D=!1,N=!1,P="white",W={},q=function(t){var n=T.left-I,e=(g-T.left)/6,r=g-T.right-e+I,i=t.append("g").attr("class","axis").attr("transform","translate(0, 20)");N&&function(t){var n=k.getFullYear();k.getFullYear()!=_.getFullYear()&&(n=k.getFullYear()+"-"+_.getFullYear()),t.append("text").attr("transform","translate(20, 0)").attr("x",0).attr("y",14).attr("class","calendarYear").text(n)}(i),i.append("text").attr("transform","translate("+n+", 0)").attr("x",0).attr("y",14).attr("class","chevron").text("<").on("click",(function(){return u(k,W)})),i.append("text").attr("transform","translate("+r+", 0)").attr("x",0).attr("y",14).attr("class","chevron").text(">").on("click",(function(){return s(_,W)}))},G=function(t,n,e){t.insert("rect").attr("class","row-green-bar").attr("x",n).attr("width",g).attr("y",e).attr("height",E).attr("fill",P)};function J(e){var u=e.append("g"),s=e[0][0].getBoundingClientRect(),x=d3.select(e[0][0]),v={},I=1,N=0,P=0;!function(){if(g||s.width){if(!g||!s.width)try{g=x.attr("width")}catch(t){console.log(t)}}else try{if(!(g=x.attr("width")))throw"width of the timeline is not set. As of Firefox 27, timeline().with(x) needs to be explicitly set in order to render"}catch(t){console.log(t)}}(),B&&u.each((function(t,n){t.forEach((function(t,n){t.times.forEach((function(t,e){0===n&&0===e?(originTime=t.starting_time,t.starting_time=0,t.ending_time=t.ending_time-originTime):(t.starting_time=t.starting_time-originTime,t.ending_time=t.ending_time-originTime)}))}))})),(b||0===_||0===k)&&(u.each((function(t,n){t.forEach((function(t,n){b&&-1==Object.keys(v).indexOf(n)&&(v[n]=I,I++),t.times.forEach((function(t,n){0===k&&(t.starting_timeP&&(P=t.ending_time)}))}))})),0===_&&(_=P),0===k&&(k=N));var J=1/(_-k)*(g-T.left-T.right),K=d3.time.scale().domain([k,_]).range([T.left,g-T.right]),Q=d3.svg.axis().scale(K).orient(f).tickFormat(p.format).ticks(p.numTicks||p.tickTime,p.tickInterval).tickSize(p.tickSize);u.each((function(c,s){W=c,c.forEach((function(c,s){var f=c.cached,h=c.index,p=c.times,x=void 0!==c.label;if(void 0!==c.id&&console.warn("d3Timeline Warning: Ids per dataset is deprecated in favor of a 'class' key. Ids are now per data element."),m&&function(t,n,e,r,i){var a=(E+S)*t[n]+T.top;e.selectAll("svg").data(r).enter().insert("rect").attr("class","row-green-bar").attr("x",A?0:T.left).attr("width",A?g:g-T.right-T.left).attr("y",a).attr("height",E).attr("fill",m instanceof Function?m(i,n):m)}(v,s,u,p,c),u.selectAll("svg").data(p).enter().append((function(t,n){return document.createElementNS(d3.ns.prefix.svg,"display"in t?t.display:w)})).attr("x",$).attr("y",_).attr("width",(function(t,n){return(t.ending_time-t.starting_time)*J})).attr("cy",(function(t,n){return _(t,n)+E/2})).attr("cx",$).attr("r",E/2).attr("height",E).style("fill",(function(e,r){return 0==r?n:1==r?f?"#9c9c9c":function(n){return t[n%16]}(h):n})).on("mousemove",(function(t,n){r(t,s,c)})).on("mouseover",(function(t,n){i(t,n,c)})).on("mouseout",(function(t,n){a(t,n,c)})).on("click",(function(t,n){o(t,s,c)})).attr("class",(function(t,n){return c.class?"timelineSeries_"+c.class:"timelineSeries_"+s})).attr("id",(function(t,n){return c.id&&!t.id?"timelineItem_"+c.id:t.id?t.id:"timelineItem_"+s+"_"+n})),u.selectAll("svg").data(p).enter().append("text").attr("x",tt).attr("y",(function(t,n){if(b)return T.top+(E+S)*v[s]+.75*E;return T.top+.75*E})).text((function(t){return t.label})),d){var k=E+S/2+T.top+(E+S)*v[s];e.append("svg:line").attr("class","row-seperator").attr("x1",0+T.left).attr("x2",g-T.right).attr("y1",k).attr("y2",k).attr("stroke-width",1).attr("stroke",d)}function _(t,n){return b?T.top+(E+S)*v[s]:T.top}x&&function(t,n,e,r,i){var a=E+S,c=T.top+a/2+a*(n[e]||1);t.append("text").attr("class","timeline-label").attr("transform","translate("+y+","+c+")").text(r?l(i.label):i.id).on("click",(function(t,n){o(t,e,i)}))}(e,v,s,x,c),void 0!==c.icon&&e.append("image").attr("class","timeline-label").attr("transform","translate(0,"+(T.top+(E+S)*v[s])+")").attr("xlink:href",c.icon).attr("width",T.left).attr("height",E)}))}));var U=T.top+(E+S)*I,V=T.top;if(Y&&function(t,n,e){j&&G(t,0,0),D&&q(t),t.append("g").attr("class","axis").attr("transform","translate(0,"+e+")").call(n)}(u,Q,z?V:U),C&&function(t,n,e){t.append("g").attr("class","axis").attr("transform","translate(0,"+(T.top+(E+S)*e)+")").attr(O.stroke,O.spacing).call(n.tickFormat("").tickSize(-(T.top+(E+S)*(e-1)+3),0,0))}(u,Q,I),g>s.width){var X=d3.behavior.zoom().x(K).on("zoom",(function(){var t=Math.min(0,Math.max(s.width-g,d3.event.translate[0]));X.translate([t,0]),u.attr("transform","translate("+t+",0)"),c(t*J,K)}));e.attr("class","scrollable").call(X)}F&&u.selectAll(".tick text").attr("transform",(function(t){return"rotate("+F+")translate("+(this.getBBox().width/2+10)+","+this.getBBox().height/2+")"}));var Z=u[0][0].getBoundingClientRect();(function(){if(h||x.attr("height"))h?x.attr("height",h):h=x.attr("height");else{if(!E)throw"height of the timeline is not set";h=Z.height+Z.top-s.top,d3.select(e[0][0]).attr("height",h)}}(),L&&u.each((function(t,n){t.forEach((function(t){t.times.forEach((function(t){nt(K(t.starting_time),R),nt(K(t.ending_time),R)}))}))})),M)&&nt(K(new Date),H);function $(t,n){return T.left+(t.starting_time-k)*J}function tt(t,n){return T.left+(t.starting_time-k)*J+5}function nt(t,n){e.append("svg:line").attr("x1",t).attr("y1",n.marginTop).attr("x2",t).attr("y2",h-n.marginBottom).style("stroke",n.color).style("stroke-width",n.width)}}return J.margin=function(t){return arguments.length?(T=t,J):T},J.orient=function(t){return arguments.length?(f=t,J):f},J.itemHeight=function(t){return arguments.length?(E=t,J):E},J.itemMargin=function(t){return arguments.length?(S=t,J):S},J.navMargin=function(t){return arguments.length?(I=t,J):I},J.height=function(t){return arguments.length?(h=t,J):h},J.width=function(t){return arguments.length?(g=t,J):g},J.display=function(t){return arguments.length&&-1!=e.indexOf(t)?(w=t,J):w},J.labelFormat=function(t){return arguments.length?(l=t,J):null},J.tickFormat=function(t){return arguments.length?(p=t,J):p},J.hover=function(t){return arguments.length?(r=t,J):r},J.mouseover=function(t){return arguments.length?(i=t,J):t},J.mouseout=function(t){return arguments.length?(a=t,J):t},J.click=function(t){return arguments.length?(o=t,J):o},J.scroll=function(t){return arguments.length?(c=t,J):c},J.colors=function(t){return arguments.length?(x=t,J):x},J.beginning=function(t){return arguments.length?(k=t,J):k},J.ending=function(t){return arguments.length?(_=t,J):_},J.labelMargin=function(t){return arguments.length?(y=t,J):_},J.rotateTicks=function(t){return F=t,J},J.stack=function(){return b=!b,J},J.relativeTime=function(){return B=!B,J},J.showBorderLine=function(){return L=!L,J},J.showBorderFormat=function(t){return arguments.length?(R=t,J):R},J.showToday=function(){return M=!M,J},J.showTodayFormat=function(t){return arguments.length?(H=t,J):H},J.colorProperty=function(t){return arguments.length?(v=t,J):v},J.rowSeperators=function(t){return arguments.length?(d=t,J):d},J.background=function(t){return arguments.length?(m=t,J):m},J.showTimeAxis=function(){return Y=!Y,J},J.showAxisTop=function(){return z=!z,J},J.showAxisCalendarYear=function(){return N=!N,J},J.showTimeAxisTick=function(){return C=!C,J},J.fullLengthBackgrounds=function(){return A=!A,J},J.showTimeAxisTickFormat=function(t){return arguments.length?(O=t,J):O},J.showAxisHeaderBackground=function(t){return j=!j,t&&(P=t),J},J.navigate=function(t,n){return u=t,s=n,D=!D,J},J}}(); \ No newline at end of file diff --git a/modules/nextflow/src/test/resources/nextflow/trace/timeline-expected.html b/modules/nextflow/src/test/resources/nextflow/trace/timeline-expected.html index eae0964d79..954741faa2 100644 --- a/modules/nextflow/src/test/resources/nextflow/trace/timeline-expected.html +++ b/modules/nextflow/src/test/resources/nextflow/trace/timeline-expected.html @@ -100,8 +100,7 @@

Processes execution timeline

-!function(){var t=d3.scale.category20c().domain(d3.range(0,20)).range();var n="#9c9c9c",e="#bdbdbd";d3.timeline=function(){var r=["circle","rect"],i=function(){},a=function(){},o=function(){},c=function(){},l=function(){},u=function(t){return t},s=function(){},f=function(){},g="bottom",h=null,d=null,m=null,p=null,x={format:d3.time.format("%I %p"),tickTime:d3.time.hours,tickInterval:1,tickSize:6},v=d3.scale.category20(),w=null,k="rect",y=0,_=0,T=0,b={left:30,right:30,top:30,bottom:30},F=!1,B=!1,A=!1,E=!1,S=20,I=5,Y=60,z=!0,M=!1,C=!1,O=!1,H={stroke:"stroke-dasharray",spacing:"4 10"},L={marginTop:25,marginBottom:0,width:1,color:v},R=!1,j={marginTop:25,marginBottom:0,width:1,color:v},D=!1,N=!1,P=!1,W="white",q={},G=function(t,n,e){D&&K(t,0,0),N&&J(t);t.append("g").attr("class","axis").attr("transform","translate(0,"+e+")").call(n)},J=function(t){var n=b.left-Y,e=(h-b.left)/6,r=h-b.right-e+Y,i=t.append("g").attr("class","axis").attr("transform","translate(0, 20)");P&&function(t){var n=y.getFullYear();y.getFullYear()!=T.getFullYear()&&(n=y.getFullYear()+"-"+T.getFullYear()),t.append("text").attr("transform","translate(20, 0)").attr("x",0).attr("y",14).attr("class","calendarYear").text(n)}(i),i.append("text").attr("transform","translate("+n+", 0)").attr("x",0).attr("y",14).attr("class","chevron").text("<").on("click",function(){return s(y,q)}),i.append("text").attr("transform","translate("+r+", 0)").attr("x",0).attr("y",14).attr("class","chevron").text(">").on("click",function(){return f(T,q)})},K=function(t,n,e){t.insert("rect").attr("class","row-green-bar").attr("x",n).attr("width",h).attr("y",e).attr("height",S).attr("fill",W)},Q=function(t,n,e){t.append("g").attr("class","axis").attr("transform","translate(0,"+(b.top+(S+I)*e)+")").attr(H.stroke,H.spacing).call(n.tickFormat("").tickSize(-(b.top+(S+I)*(e-1)+3),0,0))},U=function(t,n,e,r,i){var a=(S+I)*t[n]+b.top;e.selectAll("svg").data(r).enter().insert("rect").attr("class","row-green-bar").attr("x",E?0:b.left).attr("width",E?h:h-b.right-b.left).attr("y",a).attr("height",S).attr("fill",p instanceof Function?p(i,n):p)},V=function(t,n,e,r,i){var a=S+I,o=a+a*(n[e]||1);t.append("text").attr("class","timeline-label").attr("transform","translate("+_+","+o+")").text(r?u(i.label):i.id).on("click",function(t,n){c(t,e,i)})};function X(r){var u=r.append("g"),s=r[0][0].getBoundingClientRect(),f=d3.select(r[0][0]),v={},w=1,_=0,E=0;!function(){if(h||s.width){if(!h||!s.width)try{h=f.attr("width")}catch(t){console.log(t)}}else try{if(!(h=f.attr("width")))throw"width of the timeline is not set. As of Firefox 27, timeline().with(x) needs to be explicitly set in order to render"}catch(t){console.log(t)}}(),A&&u.each(function(t,n){t.forEach(function(t,n){t.times.forEach(function(t,e){0===n&&0===e?(originTime=t.starting_time,t.starting_time=0,t.ending_time=t.ending_time-originTime):(t.starting_time=t.starting_time-originTime,t.ending_time=t.ending_time-originTime)})})}),(F||0===T||0===y)&&(u.each(function(t,n){t.forEach(function(t,n){F&&-1==Object.keys(v).indexOf(n)&&(v[n]=w,w++),t.times.forEach(function(t,n){0===y&&(t.starting_time<_||0===_&&!1===A)&&(_=t.starting_time),0===T&&t.ending_time>E&&(E=t.ending_time)})})}),0===T&&(T=E),0===y&&(y=_));var Y=1/(T-y)*(h-b.left-b.right),H=d3.time.scale().domain([y,T]).range([b.left,h-b.right]),D=d3.svg.axis().scale(H).orient(g).tickFormat(x.format).ticks(x.numTicks||x.tickTime,x.tickInterval).tickSize(x.tickSize);u.each(function(l,s){q=l,l.forEach(function(l,s){var f=l.cached,g=l.index,d=l.times,x=void 0!==l.label;if(void 0!==l.id&&console.warn("d3Timeline Warning: Ids per dataset is deprecated in favor of a 'class' key. Ids are now per data element."),p&&U(v,s,u,d,l),u.selectAll("svg").data(d).enter().append(function(t,n){return document.createElementNS(d3.ns.prefix.svg,"display"in t?t.display:k)}).attr("x",K).attr("y",y).attr("width",function(t,n){return(t.ending_time-t.starting_time)*Y}).attr("cy",function(t,n){return y(t,n)+S/2}).attr("cx",K).attr("r",S/2).attr("height",S).style("fill",function(r,i){return 0==i?e:1==i?f?n:t[g%16]:e}).on("mousemove",function(t,n){i(t,s,l)}).on("mouseover",function(t,n){a(t,n,l)}).on("mouseout",function(t,n){o(t,n,l)}).on("click",function(t,n){c(t,s,l)}).attr("class",function(t,n){return l.class?"timelineSeries_"+l.class:"timelineSeries_"+s}).attr("id",function(t,n){return l.id&&!t.id?"timelineItem_"+l.id:t.id?t.id:"timelineItem_"+s+"_"+n}),u.selectAll("svg").data(d).enter().append("text").attr("x",X).attr("y",function(t,n){if(F)return b.top+(S+I)*v[s]+.75*S;return b.top+.75*S}).text(function(t){return t.label}),m){var w=S+I/2+b.top+(S+I)*v[s];r.append("svg:line").attr("class","row-seperator").attr("x1",0+b.left).attr("x2",h-b.right).attr("y1",w).attr("y2",w).attr("stroke-width",1).attr("stroke",m)}function y(t,n){return F?b.top+(S+I)*v[s]:b.top}x&&V(r,v,s,x,l),void 0!==l.icon&&r.append("image").attr("class","timeline-label").attr("transform","translate(0,"+(b.top+(S+I)*v[s])+")").attr("xlink:href",l.icon).attr("width",b.left).attr("height",S)})});var N=b.top+(S+I)*w,P=b.top;if(z&&G(u,D,M?P:N),O&&Q(u,D,w),h>s.width){var W=d3.behavior.zoom().x(H).on("zoom",function(){var t=Math.min(0,Math.max(s.width-h,d3.event.translate[0]));W.translate([t,0]),u.attr("transform","translate("+t+",0)"),l(t*Y,H)});r.attr("class","scrollable").call(W)}B&&u.selectAll(".tick text").attr("transform",function(t){return"rotate("+B+")translate("+(this.getBBox().width/2+10)+","+this.getBBox().height/2+")"});var J=u[0][0].getBoundingClientRect();(function(){if(d||f.attr("height"))d?f.attr("height",d):d=f.attr("height");else{if(!S)throw"height of the timeline is not set";d=J.height+J.top-s.top,d3.select(r[0][0]).attr("height",d)}}(),R&&u.each(function(t,n){t.forEach(function(t){t.times.forEach(function(t){Z(H(t.starting_time),j),Z(H(t.ending_time),j)})})}),C)&&Z(H(new Date),L);function K(t,n){return b.left+(t.starting_time-y)*Y}function X(t,n){return b.left+(t.starting_time-y)*Y+5}function Z(t,n){r.append("svg:line").attr("x1",t).attr("y1",n.marginTop).attr("x2",t).attr("y2",d-n.marginBottom).style("stroke",n.color).style("stroke-width",n.width)}}return X.margin=function(t){return arguments.length?(b=t,X):b},X.orient=function(t){return arguments.length?(g=t,X):g},X.itemHeight=function(t){return arguments.length?(S=t,X):S},X.itemMargin=function(t){return arguments.length?(I=t,X):I},X.navMargin=function(t){return arguments.length?(Y=t,X):Y},X.height=function(t){return arguments.length?(d=t,X):d},X.width=function(t){return arguments.length?(h=t,X):h},X.display=function(t){return arguments.length&&-1!=r.indexOf(t)?(k=t,X):k},X.labelFormat=function(t){return arguments.length?(u=t,X):null},X.tickFormat=function(t){return arguments.length?(x=t,X):x},X.hover=function(t){return arguments.length?(i=t,X):i},X.mouseover=function(t){return arguments.length?(a=t,X):t},X.mouseout=function(t){return arguments.length?(o=t,X):t},X.click=function(t){return arguments.length?(c=t,X):c},X.scroll=function(t){return arguments.length?(l=t,X):l},X.colors=function(t){return arguments.length?(v=t,X):v},X.beginning=function(t){return arguments.length?(y=t,X):y},X.ending=function(t){return arguments.length?(T=t,X):T},X.labelMargin=function(t){return arguments.length?(_=t,X):T},X.rotateTicks=function(t){return B=t,X},X.stack=function(){return F=!F,X},X.relativeTime=function(){return A=!A,X},X.showBorderLine=function(){return R=!R,X},X.showBorderFormat=function(t){return arguments.length?(j=t,X):j},X.showToday=function(){return C=!C,X},X.showTodayFormat=function(t){return arguments.length?(L=t,X):L},X.colorProperty=function(t){return arguments.length?(w=t,X):w},X.rowSeperators=function(t){return arguments.length?(m=t,X):m},X.background=function(t){return arguments.length?(p=t,X):p},X.showTimeAxis=function(){return z=!z,X},X.showAxisTop=function(){return M=!M,X},X.showAxisCalendarYear=function(){return P=!P,X},X.showTimeAxisTick=function(){return O=!O,X},X.fullLengthBackgrounds=function(){return E=!E,X},X.showTimeAxisTickFormat=function(t){return arguments.length?(H=t,X):H},X.showAxisHeaderBackground=function(t){return D=!D,t&&(W=t),X},X.navigate=function(t,n){return s=t,f=n,N=!N,X},X}}(); - +!function(){var t=d3.scale.category20c().domain(d3.range(0,20)).range();var n="#bdbdbd";d3.timeline=function(){var e=["circle","rect"],r=function(){},i=function(){},a=function(){},o=function(){},c=function(){},l=function(t){return t},u=function(){},s=function(){},f="bottom",g=null,h=null,d=null,m=null,p={format:d3.time.format("%I %p"),tickTime:d3.time.hours,tickInterval:1,tickSize:6},x=d3.scale.category20(),v=null,w="rect",k=0,y=0,_=0,T={left:30,right:30,top:30,bottom:30},b=!1,F=!1,B=!1,A=!1,E=20,S=5,I=60,Y=!0,z=!1,M=!1,C=!1,O={stroke:"stroke-dasharray",spacing:"4 10"},H={marginTop:25,marginBottom:0,width:1,color:x},L=!1,R={marginTop:25,marginBottom:0,width:1,color:x},j=!1,D=!1,N=!1,P="white",W={},q=function(t){var n=T.left-I,e=(g-T.left)/6,r=g-T.right-e+I,i=t.append("g").attr("class","axis").attr("transform","translate(0, 20)");N&&function(t){var n=k.getFullYear();k.getFullYear()!=_.getFullYear()&&(n=k.getFullYear()+"-"+_.getFullYear()),t.append("text").attr("transform","translate(20, 0)").attr("x",0).attr("y",14).attr("class","calendarYear").text(n)}(i),i.append("text").attr("transform","translate("+n+", 0)").attr("x",0).attr("y",14).attr("class","chevron").text("<").on("click",(function(){return u(k,W)})),i.append("text").attr("transform","translate("+r+", 0)").attr("x",0).attr("y",14).attr("class","chevron").text(">").on("click",(function(){return s(_,W)}))},G=function(t,n,e){t.insert("rect").attr("class","row-green-bar").attr("x",n).attr("width",g).attr("y",e).attr("height",E).attr("fill",P)};function J(e){var u=e.append("g"),s=e[0][0].getBoundingClientRect(),x=d3.select(e[0][0]),v={},I=1,N=0,P=0;!function(){if(g||s.width){if(!g||!s.width)try{g=x.attr("width")}catch(t){console.log(t)}}else try{if(!(g=x.attr("width")))throw"width of the timeline is not set. As of Firefox 27, timeline().with(x) needs to be explicitly set in order to render"}catch(t){console.log(t)}}(),B&&u.each((function(t,n){t.forEach((function(t,n){t.times.forEach((function(t,e){0===n&&0===e?(originTime=t.starting_time,t.starting_time=0,t.ending_time=t.ending_time-originTime):(t.starting_time=t.starting_time-originTime,t.ending_time=t.ending_time-originTime)}))}))})),(b||0===_||0===k)&&(u.each((function(t,n){t.forEach((function(t,n){b&&-1==Object.keys(v).indexOf(n)&&(v[n]=I,I++),t.times.forEach((function(t,n){0===k&&(t.starting_timeP&&(P=t.ending_time)}))}))})),0===_&&(_=P),0===k&&(k=N));var J=1/(_-k)*(g-T.left-T.right),K=d3.time.scale().domain([k,_]).range([T.left,g-T.right]),Q=d3.svg.axis().scale(K).orient(f).tickFormat(p.format).ticks(p.numTicks||p.tickTime,p.tickInterval).tickSize(p.tickSize);u.each((function(c,s){W=c,c.forEach((function(c,s){var f=c.cached,h=c.index,p=c.times,x=void 0!==c.label;if(void 0!==c.id&&console.warn("d3Timeline Warning: Ids per dataset is deprecated in favor of a 'class' key. Ids are now per data element."),m&&function(t,n,e,r,i){var a=(E+S)*t[n]+T.top;e.selectAll("svg").data(r).enter().insert("rect").attr("class","row-green-bar").attr("x",A?0:T.left).attr("width",A?g:g-T.right-T.left).attr("y",a).attr("height",E).attr("fill",m instanceof Function?m(i,n):m)}(v,s,u,p,c),u.selectAll("svg").data(p).enter().append((function(t,n){return document.createElementNS(d3.ns.prefix.svg,"display"in t?t.display:w)})).attr("x",$).attr("y",_).attr("width",(function(t,n){return(t.ending_time-t.starting_time)*J})).attr("cy",(function(t,n){return _(t,n)+E/2})).attr("cx",$).attr("r",E/2).attr("height",E).style("fill",(function(e,r){return 0==r?n:1==r?f?"#9c9c9c":function(n){return t[n%16]}(h):n})).on("mousemove",(function(t,n){r(t,s,c)})).on("mouseover",(function(t,n){i(t,n,c)})).on("mouseout",(function(t,n){a(t,n,c)})).on("click",(function(t,n){o(t,s,c)})).attr("class",(function(t,n){return c.class?"timelineSeries_"+c.class:"timelineSeries_"+s})).attr("id",(function(t,n){return c.id&&!t.id?"timelineItem_"+c.id:t.id?t.id:"timelineItem_"+s+"_"+n})),u.selectAll("svg").data(p).enter().append("text").attr("x",tt).attr("y",(function(t,n){if(b)return T.top+(E+S)*v[s]+.75*E;return T.top+.75*E})).text((function(t){return t.label})),d){var k=E+S/2+T.top+(E+S)*v[s];e.append("svg:line").attr("class","row-seperator").attr("x1",0+T.left).attr("x2",g-T.right).attr("y1",k).attr("y2",k).attr("stroke-width",1).attr("stroke",d)}function _(t,n){return b?T.top+(E+S)*v[s]:T.top}x&&function(t,n,e,r,i){var a=E+S,c=T.top+a/2+a*(n[e]||1);t.append("text").attr("class","timeline-label").attr("transform","translate("+y+","+c+")").text(r?l(i.label):i.id).on("click",(function(t,n){o(t,e,i)}))}(e,v,s,x,c),void 0!==c.icon&&e.append("image").attr("class","timeline-label").attr("transform","translate(0,"+(T.top+(E+S)*v[s])+")").attr("xlink:href",c.icon).attr("width",T.left).attr("height",E)}))}));var U=T.top+(E+S)*I,V=T.top;if(Y&&function(t,n,e){j&&G(t,0,0),D&&q(t),t.append("g").attr("class","axis").attr("transform","translate(0,"+e+")").call(n)}(u,Q,z?V:U),C&&function(t,n,e){t.append("g").attr("class","axis").attr("transform","translate(0,"+(T.top+(E+S)*e)+")").attr(O.stroke,O.spacing).call(n.tickFormat("").tickSize(-(T.top+(E+S)*(e-1)+3),0,0))}(u,Q,I),g>s.width){var X=d3.behavior.zoom().x(K).on("zoom",(function(){var t=Math.min(0,Math.max(s.width-g,d3.event.translate[0]));X.translate([t,0]),u.attr("transform","translate("+t+",0)"),c(t*J,K)}));e.attr("class","scrollable").call(X)}F&&u.selectAll(".tick text").attr("transform",(function(t){return"rotate("+F+")translate("+(this.getBBox().width/2+10)+","+this.getBBox().height/2+")"}));var Z=u[0][0].getBoundingClientRect();(function(){if(h||x.attr("height"))h?x.attr("height",h):h=x.attr("height");else{if(!E)throw"height of the timeline is not set";h=Z.height+Z.top-s.top,d3.select(e[0][0]).attr("height",h)}}(),L&&u.each((function(t,n){t.forEach((function(t){t.times.forEach((function(t){nt(K(t.starting_time),R),nt(K(t.ending_time),R)}))}))})),M)&&nt(K(new Date),H);function $(t,n){return T.left+(t.starting_time-k)*J}function tt(t,n){return T.left+(t.starting_time-k)*J+5}function nt(t,n){e.append("svg:line").attr("x1",t).attr("y1",n.marginTop).attr("x2",t).attr("y2",h-n.marginBottom).style("stroke",n.color).style("stroke-width",n.width)}}return J.margin=function(t){return arguments.length?(T=t,J):T},J.orient=function(t){return arguments.length?(f=t,J):f},J.itemHeight=function(t){return arguments.length?(E=t,J):E},J.itemMargin=function(t){return arguments.length?(S=t,J):S},J.navMargin=function(t){return arguments.length?(I=t,J):I},J.height=function(t){return arguments.length?(h=t,J):h},J.width=function(t){return arguments.length?(g=t,J):g},J.display=function(t){return arguments.length&&-1!=e.indexOf(t)?(w=t,J):w},J.labelFormat=function(t){return arguments.length?(l=t,J):null},J.tickFormat=function(t){return arguments.length?(p=t,J):p},J.hover=function(t){return arguments.length?(r=t,J):r},J.mouseover=function(t){return arguments.length?(i=t,J):t},J.mouseout=function(t){return arguments.length?(a=t,J):t},J.click=function(t){return arguments.length?(o=t,J):o},J.scroll=function(t){return arguments.length?(c=t,J):c},J.colors=function(t){return arguments.length?(x=t,J):x},J.beginning=function(t){return arguments.length?(k=t,J):k},J.ending=function(t){return arguments.length?(_=t,J):_},J.labelMargin=function(t){return arguments.length?(y=t,J):_},J.rotateTicks=function(t){return F=t,J},J.stack=function(){return b=!b,J},J.relativeTime=function(){return B=!B,J},J.showBorderLine=function(){return L=!L,J},J.showBorderFormat=function(t){return arguments.length?(R=t,J):R},J.showToday=function(){return M=!M,J},J.showTodayFormat=function(t){return arguments.length?(H=t,J):H},J.colorProperty=function(t){return arguments.length?(v=t,J):v},J.rowSeperators=function(t){return arguments.length?(d=t,J):d},J.background=function(t){return arguments.length?(m=t,J):m},J.showTimeAxis=function(){return Y=!Y,J},J.showAxisTop=function(){return z=!z,J},J.showAxisCalendarYear=function(){return N=!N,J},J.showTimeAxisTick=function(){return C=!C,J},J.fullLengthBackgrounds=function(){return A=!A,J},J.showTimeAxisTickFormat=function(t){return arguments.length?(O=t,J):O},J.showAxisHeaderBackground=function(t){return j=!j,t&&(P=t),J},J.navigate=function(t,n){return u=t,s=n,D=!D,J},J}}(); From 4ad9e3405a6af667c0787446fa738d39eaae34da Mon Sep 17 00:00:00 2001 From: Jorge Ejarque Date: Thu, 26 Feb 2026 19:38:31 +0100 Subject: [PATCH 37/75] Ensure main script is first in the WorkflowRun lineage record (#6845) --- .../nf-lineage/src/main/nextflow/lineage/LinObserver.groovy | 4 ++-- .../src/main/nextflow/lineage/model/v1beta1/Workflow.groovy | 2 +- .../src/test/nextflow/lineage/LinObserverTest.groovy | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/modules/nf-lineage/src/main/nextflow/lineage/LinObserver.groovy b/modules/nf-lineage/src/main/nextflow/lineage/LinObserver.groovy index 41b0a8cb9a..655a0c8f16 100644 --- a/modules/nf-lineage/src/main/nextflow/lineage/LinObserver.groovy +++ b/modules/nf-lineage/src/main/nextflow/lineage/LinObserver.groovy @@ -133,7 +133,7 @@ class LinObserver implements TraceObserverV2 { } protected List collectScriptDataPaths(PathNormalizer normalizer) { - final allScripts = allScriptFiles() + final allScripts = allScriptFiles().sort() final result = new ArrayList(allScripts.size()+1) // the main script result.add( new DataPath( @@ -148,7 +148,7 @@ class LinObserver implements TraceObserverV2 { final dataPath = new DataPath(normalizer.normalizePath(it.normalize()), Checksum.ofNextflow(it.text)) result.add(dataPath) } - return result.sort{it.path} + return result } protected String storeWorkflowRun(PathNormalizer normalizer) { diff --git a/modules/nf-lineage/src/main/nextflow/lineage/model/v1beta1/Workflow.groovy b/modules/nf-lineage/src/main/nextflow/lineage/model/v1beta1/Workflow.groovy index b64a59a03d..fa35f433e3 100644 --- a/modules/nf-lineage/src/main/nextflow/lineage/model/v1beta1/Workflow.groovy +++ b/modules/nf-lineage/src/main/nextflow/lineage/model/v1beta1/Workflow.groovy @@ -30,7 +30,7 @@ import nextflow.lineage.serde.LinSerializable @CompileStatic class Workflow implements LinSerializable { /** - * List of script files defining a workflow + * List of script files used by a workflow, starting with the main script */ List scriptFiles /** diff --git a/modules/nf-lineage/src/test/nextflow/lineage/LinObserverTest.groovy b/modules/nf-lineage/src/test/nextflow/lineage/LinObserverTest.groovy index d69f8c22ad..44cdd4b35b 100644 --- a/modules/nf-lineage/src/test/nextflow/lineage/LinObserverTest.groovy +++ b/modules/nf-lineage/src/test/nextflow/lineage/LinObserverTest.groovy @@ -114,8 +114,8 @@ class LinObserverTest extends Specification { def store = new DefaultLinStore(); def uniqueId = UUID.randomUUID() def scriptFile = folder.resolve("main.nf") - def module1 = folder.resolve("script1.nf"); module1.text = 'hola' - def module2 = folder.resolve("script2.nf"); module2.text = 'world' + def module1 = folder.resolve("a_script1.nf"); module1.text = 'hola' + def module2 = folder.resolve("b_script2.nf"); module2.text = 'world' and: def metadata = Mock(WorkflowMetadata){ @@ -139,7 +139,7 @@ class LinObserverTest extends Specification { when: def files = observer.collectScriptDataPaths(new PathNormalizer(metadata)) then: - observer.allScriptFiles() >> [ scriptFile, module1, module2 ] + observer.allScriptFiles() >> [ module2, scriptFile, module1 ] and: files.size() == 3 and: From 4d8ef3079f228c16220cc60097e800c230d787d4 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 26 Feb 2026 23:14:43 +0100 Subject: [PATCH 38/75] Add whitespace rules to .editorconfig (#5606) --- .editorconfig | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.editorconfig b/.editorconfig index 8ed3888e2d..2436963043 100644 --- a/.editorconfig +++ b/.editorconfig @@ -3,3 +3,6 @@ charset = utf-8 indent_size = 4 indent_style = space tab_width = 4 +end_of_line = lf +insert_final_newline = true +trim_trailing_whitespace = true From 0228a4ebf922449e6a87ac5ca7ea550bbaa611af Mon Sep 17 00:00:00 2001 From: Chris Hakkaart Date: Fri, 27 Feb 2026 12:45:57 +1300 Subject: [PATCH 39/75] docs: Update your first script to use outputs (#6500) --- docs/your-first-script.md | 52 ++++++++++++++++++++++++++++----------- 1 file changed, 38 insertions(+), 14 deletions(-) diff --git a/docs/your-first-script.md b/docs/your-first-script.md index 8c86fb93e2..a259e9e98d 100644 --- a/docs/your-first-script.md +++ b/docs/your-first-script.md @@ -12,7 +12,7 @@ This guide details fundamental skills to run a basic Nextflow pipeline. It inclu You will need the following to get started: -- Nextflow: See {ref}`install-page` for installation instructions. +- Nextflow version 25.10 or later. See {ref}`install-page` for installation instructions. If you have an older version, see {ref}`updating-nextflow` to update. ## Run a pipeline @@ -25,11 +25,9 @@ params.str = "Hello world!" // split process process split { - publishDir "results/lower" - input: val x - + output: path 'chunk_*' @@ -41,7 +39,6 @@ process split { // convert_to_upper process process convert_to_upper { - publishDir "results/upper" tag "$y" input: @@ -58,9 +55,23 @@ process convert_to_upper { // Workflow block workflow { - ch_str = channel.of(params.str) // Create a channel using parameter input - ch_chunks = split(ch_str) // Split string into chunks and create a named channel - convert_to_upper(ch_chunks.flatten()) // Convert lowercase letters to uppercase letters + main: + ch_str = channel.of(params.str) // Create a channel using parameter input + ch_chunks = split(ch_str) // Split string into chunks and create a named channel + ch_upper = convert_to_upper(ch_chunks.flatten()) // Convert lowercase letters to uppercase letters + + publish: + lower = ch_chunks.flatten() + upper = ch_upper +} + +output { + lower { + path 'lower' + } + upper { + path 'upper' + } } ``` @@ -71,7 +82,12 @@ This script defines two processes: The `split` output is emitted as a single element. The `flatten` operator splits this combined element so that each file is treated as a sole element. -The outputs from both processes are published in subdirectories (`lower` and `upper`) in the `results` directory. +The workflow block is organized into two sections: + +- `main:`: defines the workflow logic and how processes are connected via channels +- `publish:`: declares which channels should be published as workflow outputs + +The `output` block (outside the workflow) defines where and how each output should be published. In this example, the outputs from both processes are published in subdirectories (`lower` and `upper`) in the default `results` out directory. To run your pipeline: @@ -87,7 +103,7 @@ To run your pipeline: You will see output similar to the following: ```console - N E X T F L O W ~ version 24.10.3 + N E X T F L O W ~ version 25.10.0 Launching `main.nf` [big_wegener] DSL2 - revision: 13a41a8946 @@ -98,6 +114,15 @@ executor > local (3) Nextflow creates a `work` directory to store files used during a pipeline run. Each execution of a process is run as a separate task. The `split` process is run as one task and the `convert_to_upper` process is run as two tasks. The hexadecimal string, for example, `82/457482`, is the beginning of a unique hash. It is a prefix used to identify the task directory where the script was executed. +When the pipeline completes, you can view the output files in the `results` directory: + +```{code-block} bash +:class: copyable +ls -R results/ +``` + +You will see the published output files organized in the `lower` and `upper` subdirectories. + :::{tip} Run your pipeline with `-ansi-log false` to see each task printed on a separate line: @@ -109,7 +134,7 @@ nextflow run main.nf -ansi-log false You will see output similar to the following: ```console -N E X T F L O W ~ version 24.10.3 +N E X T F L O W ~ version 25.10.0 Launching `main.nf` [peaceful_watson] DSL2 - revision: 13a41a8946 [43/f1f8b5] Submitted process > split (1) [a2/5aa4b1] Submitted process > convert_to_upper (chunk_ab) @@ -132,7 +157,6 @@ You can enable resumability using the `-resume` flag when running a pipeline. To ```{code-block} groovy :class: copyable process convert_to_upper { - publishDir "results/upper" tag "$y" input: @@ -159,7 +183,7 @@ You can enable resumability using the `-resume` flag when running a pipeline. To You will see output similar to the following: ```console - N E X T F L O W ~ version 24.10.3 + N E X T F L O W ~ version 25.10.0 Launching `main.nf` [furious_curie] DSL2 - revision: 5490f13c43 @@ -190,7 +214,7 @@ You can configure the `str` parameter in your pipeline. To modify your `str` par You will see output similar to the following: ```console - N E X T F L O W ~ version 24.10.3 + N E X T F L O W ~ version 25.10.0 Launching `main.nf` [distracted_kalam] DSL2 - revision: 082867d4d6 From a1a046afba8552255962b0efee49404eea044954 Mon Sep 17 00:00:00 2001 From: Rintze Zelle <78232505+rzelle-lallemand@users.noreply.github.com> Date: Thu, 26 Feb 2026 22:03:50 -0500 Subject: [PATCH 40/75] Update AWS CLI install docs to bypass conda install TOS prompt (#6685) --- docs/aws.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/aws.md b/docs/aws.md index 820faec89d..68937f39ec 100644 --- a/docs/aws.md +++ b/docs/aws.md @@ -317,7 +317,7 @@ cd $HOME sudo yum install -y bzip2 wget wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh -b -f -p $HOME/miniconda -$HOME/miniconda/bin/conda install -c conda-forge -y awscli +$HOME/miniconda/bin/conda install -c conda-forge --override-channels -y awscli rm Miniconda3-latest-Linux-x86_64.sh ``` From 3830af3b2a5f59e9ef9a43427300f38c0017b95f Mon Sep 17 00:00:00 2001 From: "Colin J. Brislawn" Date: Thu, 26 Feb 2026 19:29:57 -0800 Subject: [PATCH 41/75] Clarify onError and onComplete handler descriptions (#6709) [ci fast] --- docs/notifications.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/notifications.md b/docs/notifications.md index 9828420cf2..44015bb540 100644 --- a/docs/notifications.md +++ b/docs/notifications.md @@ -52,7 +52,7 @@ workflow.onError { ``` :::{note} -Both the `onError` and `onComplete` handlers are invoked when an error condition is encountered. The first is called as soon as the error is raised, while the second is called just before the pipeline execution is about to terminate. When using the `finish` {ref}`process-error-strategy`, there may be a significant gap between the two, depending on the time required to complete any pending job. +Both the `onError` and `onComplete` handlers are invoked when an error condition is encountered. The `onError` handler is called as soon as the error is raised, while `onComplete` is called just before the pipeline execution is about to terminate. When using the `finish` {ref}`process-error-strategy`, there may be a significant gap between the two, depending on the time required to complete any pending job. ::: :::{versionadded} 25.10.0 From 28458b50f0178f99b9e51a274965e70d527d6b1f Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 27 Feb 2026 08:25:24 -0600 Subject: [PATCH 42/75] Use explicit memory units for LSF executor (#5217) Signed-off-by: Ben Sherman --- .../nextflow/executor/LsfExecutor.groovy | 23 +--- .../nextflow/executor/LsfExecutorTest.groovy | 111 +++++------------- .../test/resources/nextflow/executor/lsf.conf | 2 - 3 files changed, 32 insertions(+), 104 deletions(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/executor/LsfExecutor.groovy b/modules/nextflow/src/main/groovy/nextflow/executor/LsfExecutor.groovy index d7e705a5e5..eead0de136 100644 --- a/modules/nextflow/src/main/groovy/nextflow/executor/LsfExecutor.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/executor/LsfExecutor.groovy @@ -48,18 +48,8 @@ class LsfExecutor extends AbstractGridExecutor implements TaskArrayExecutor { private boolean perTaskReserve - /* - * If LSF_UNIT_FOR_LIMITS is not defined in lsf.conf, then the default setting is in KB, and for RUSAGE it is MB - * see https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.3/lsf_config_ref/lsf.conf.lsf_unit_for_limits.5.html - */ - private String memUnit = 'KB' - - private String usageUnit = 'MB' - protected boolean getPerJobMemLimit() { perJobMemLimit } protected boolean getPerTaskReserve() { perTaskReserve } - protected String getMemUnit() { memUnit } - protected String getUsageUnit() { usageUnit } /** * Gets the directives to submit the specified task to the cluster for execution @@ -96,13 +86,13 @@ class LsfExecutor extends AbstractGridExecutor implements TaskArrayExecutor { def mem1 = ( task.config.getCpus() > 1 && !perJobMemLimit ) ? mem.div(task.config.getCpus() as int) : mem def mem2 = ( task.config.getCpus() > 1 && perTaskReserve ) ? mem.div(task.config.getCpus() as int) : mem - result << '-M' << String.valueOf(mem1.toUnit(memUnit)) - result << '-R' << "select[mem>=${mem.toUnit(memUnit)}] rusage[mem=${mem2.toUnit(usageUnit)}]".toString() + result << '-M' << "${mem1.toMega()}MB".toString() + result << '-R' << "select[mem>=${mem.toMega()}MB] rusage[mem=${mem2.toMega()}MB]".toString() } def disk = task.config.getDisk() if( disk ) { - result << '-R' << "select[tmp>=${disk.toUnit(memUnit)}] rusage[tmp=${disk.toUnit(usageUnit)}]".toString() + result << '-R' << "select[tmp>=${disk.toMega()}MB] rusage[tmp=${disk.toMega()}MB]".toString() } // -- the job name @@ -310,13 +300,6 @@ class LsfExecutor extends AbstractGridExecutor implements TaskArrayExecutor { super.register() final conf = parseLsfConfig() - // lsf mem unit - // https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.3/lsf_config_ref/lsf.conf.lsf_unit_for_limits.5.html - if( conf.get('LSF_UNIT_FOR_LIMITS') ) { - memUnit = usageUnit = conf.get('LSF_UNIT_FOR_LIMITS') - log.debug "[LSF] Detected lsf.conf LSF_UNIT_FOR_LIMITS=$memUnit" - } - // per job mem limit // https://www.ibm.com/support/knowledgecenter/SSETD4_9.1.3/lsf_config_ref/lsf.conf.lsb_job_memlimit.5.dita if( conf.get('LSB_JOB_MEMLIMIT') ) { diff --git a/modules/nextflow/src/test/groovy/nextflow/executor/LsfExecutorTest.groovy b/modules/nextflow/src/test/groovy/nextflow/executor/LsfExecutorTest.groovy index 7ad66f287b..fc966d438f 100644 --- a/modules/nextflow/src/test/groovy/nextflow/executor/LsfExecutorTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/executor/LsfExecutorTest.groovy @@ -69,31 +69,8 @@ class LsfExecutorTest extends Specification { _ * task.config >> new TaskConfig(memory: '10MB') then: result == ['-o', '/work/dir/.command.log', - '-M', '10240', - '-R', 'select[mem>=10240] rusage[mem=10]', - '-J', 'foo'] - } - - def testMemDirectiveMemUnit2() { - given: - def WORK_DIR = Paths.get('/work/dir') - def executor = createExecutor() - executor.getSession() >> Mock(Session) - and: - def task = Mock(TaskRun) - task.workDir >> WORK_DIR - - when: - executor.@memUnit = 'GB' - executor.@usageUnit = 'GB' - def result = executor.getDirectives(task, []) - then: - 1 * executor.getJobNameFor(task) >> 'foo' - _ * task.config >> new TaskConfig(memory: '100GB') - then: - result == ['-o', '/work/dir/.command.log', - '-M', '100', - '-R', 'select[mem>=100] rusage[mem=100]', + '-M', '10MB', + '-R', 'select[mem>=10MB] rusage[mem=10MB]', '-J', 'foo'] } @@ -107,7 +84,6 @@ class LsfExecutorTest extends Specification { task.workDir >> WORK_DIR when: - executor.@usageUnit = 'KB' executor.@perJobMemLimit = true def result = executor.getDirectives(task, []) then: @@ -117,8 +93,8 @@ class LsfExecutorTest extends Specification { result == ['-o', '/work/dir/.command.log', '-n', '2', '-R', 'span[hosts=1]', - '-M', '10240', - '-R', 'select[mem>=10240] rusage[mem=10240]', + '-M', '10MB', + '-R', 'select[mem>=10MB] rusage[mem=10MB]', '-J', 'foo'] } @@ -134,7 +110,6 @@ class LsfExecutorTest extends Specification { when: executor.@perJobMemLimit = true executor.@perTaskReserve = true - executor.@usageUnit = 'KB' def result = executor.getDirectives(task, []) then: 1 * executor.getJobNameFor(task) >> 'foo' @@ -143,8 +118,8 @@ class LsfExecutorTest extends Specification { result == ['-o', '/work/dir/.command.log', '-n', '2', '-R', 'span[hosts=1]', - '-M', '10240', - '-R', 'select[mem>=10240] rusage[mem=5120]', + '-M', '10MB', + '-R', 'select[mem>=10MB] rusage[mem=5MB]', '-J', 'foo'] } @@ -153,8 +128,6 @@ class LsfExecutorTest extends Specification { setup: def executor = createExecutor() - executor.@memUnit = 'MB' - executor.@usageUnit = 'MB' executor.session = new Session() def proc = Mock(TaskProcessor) @@ -179,8 +152,8 @@ class LsfExecutorTest extends Specification { #BSUB -n 2 #BSUB -R "span[hosts=1]" #BSUB -W 01:30 - #BSUB -M 4096 - #BSUB -R "select[mem>=8192] rusage[mem=8192]" + #BSUB -M 4096MB + #BSUB -R "select[mem>=8192MB] rusage[mem=8192MB]" #BSUB -J nf-mapping_hola #BSUB -x 1 #BSUB -R "span[ptile=2]" @@ -201,8 +174,8 @@ class LsfExecutorTest extends Specification { #BSUB -n 2 #BSUB -R "span[hosts=1]" #BSUB -W 01:30 - #BSUB -M 4096 - #BSUB -R "select[mem>=8192] rusage[mem=8192]" + #BSUB -M 4096MB + #BSUB -R "select[mem>=8192MB] rusage[mem=8192MB]" #BSUB -J nf-mapping_hola #BSUB -x 1 #BSUB -R "span[ptile=2]" @@ -232,8 +205,8 @@ class LsfExecutorTest extends Specification { #BSUB -o /scratch/.command.log #BSUB -q alpha #BSUB -W 00:01 - #BSUB -M 10 - #BSUB -R "select[mem>=10] rusage[mem=10]" + #BSUB -M 10MB + #BSUB -R "select[mem>=10MB] rusage[mem=10MB]" #BSUB -J nf-mapping_hola ''' .stripIndent().leftTrim() @@ -249,8 +222,8 @@ class LsfExecutorTest extends Specification { #BSUB -o /scratch/.command.log #BSUB -q gamma #BSUB -W 04:00 - #BSUB -M 200 - #BSUB -R "select[mem>=200] rusage[mem=200]" + #BSUB -M 200MB + #BSUB -R "select[mem>=200MB] rusage[mem=200MB]" #BSUB -J nf-mapping_hola ''' .stripIndent().leftTrim() @@ -266,8 +239,8 @@ class LsfExecutorTest extends Specification { #BSUB -q gamma #BSUB -n 4 #BSUB -R "span[hosts=1]" - #BSUB -M 512 - #BSUB -R "select[mem>=2048] rusage[mem=2048]" + #BSUB -M 512MB + #BSUB -R "select[mem>=2048MB] rusage[mem=2048MB]" #BSUB -J nf-mapping_hola ''' .stripIndent().leftTrim() @@ -285,8 +258,8 @@ class LsfExecutorTest extends Specification { #BSUB -n 4 #BSUB -R "span[hosts=1]" #BSUB -W 24:00 - #BSUB -M 512 - #BSUB -R "select[mem>=2048] rusage[mem=2048]" + #BSUB -M 512MB + #BSUB -R "select[mem>=2048MB] rusage[mem=2048MB]" #BSUB -J nf-mapping_hola ''' .stripIndent().leftTrim() @@ -304,8 +277,8 @@ class LsfExecutorTest extends Specification { #BSUB -n 8 #BSUB -R "span[hosts=1]" #BSUB -W 48:00 - #BSUB -M 256 - #BSUB -R "select[mem>=2048] rusage[mem=2048]" + #BSUB -M 256MB + #BSUB -R "select[mem>=2048MB] rusage[mem=2048MB]" #BSUB -J nf-mapping_hola ''' .stripIndent().leftTrim() @@ -320,8 +293,8 @@ class LsfExecutorTest extends Specification { #BSUB -o /scratch/.command.log #BSUB -q delta #BSUB -W 60:05 - #BSUB -M 2048 - #BSUB -R "select[mem>=2048] rusage[mem=2048]" + #BSUB -M 2048MB + #BSUB -R "select[mem>=2048MB] rusage[mem=2048MB]" #BSUB -J nf-mapping_hola ''' .stripIndent().leftTrim() @@ -348,7 +321,6 @@ class LsfExecutorTest extends Specification { def WORKDIR = Paths.get('/my/work') def executor = createExecutor() executor.getSession() >> Mock(Session) - executor.@memUnit = 'MB' and: def task = Mock(TaskRun) @@ -359,7 +331,7 @@ class LsfExecutorTest extends Specification { task.config >> config task.name >> 'foo' and: - result.join(' ') == "-o $WORKDIR/.command.log -R select[tmp>=10240] rusage[tmp=10240] -J nf-foo" + result.join(' ') == "-o $WORKDIR/.command.log -R select[tmp>=10240MB] rusage[tmp=10240MB] -J nf-foo" } def testPerJobMemLimit() { @@ -385,7 +357,6 @@ class LsfExecutorTest extends Specification { // LSF executor def executor = createExecutor() executor.session = new Session() - executor.@memUnit = 'MB' then: executor.getHeaders(task) == ''' @@ -393,8 +364,8 @@ class LsfExecutorTest extends Specification { #BSUB -q bsc_ls #BSUB -n 4 #BSUB -R "span[hosts=1]" - #BSUB -M 2048 - #BSUB -R "select[mem>=8192] rusage[mem=8192]" + #BSUB -M 2048MB + #BSUB -R "select[mem>=8192MB] rusage[mem=8192MB]" #BSUB -J nf-mapping_hola ''' .stripIndent().leftTrim() @@ -424,7 +395,6 @@ class LsfExecutorTest extends Specification { // LSF executor def config = new ExecutorConfig(perJobMemLimit: true) def executor = createExecutor(config) - executor.@memUnit = 'MB' executor.register() then: @@ -433,8 +403,8 @@ class LsfExecutorTest extends Specification { #BSUB -q bsc_ls #BSUB -n 4 #BSUB -R "span[hosts=1]" - #BSUB -M 8192 - #BSUB -R "select[mem>=8192] rusage[mem=8192]" + #BSUB -M 8192MB + #BSUB -R "select[mem>=8192MB] rusage[mem=8192MB]" #BSUB -J nf-mapping_hola ''' .stripIndent().leftTrim() @@ -446,8 +416,6 @@ class LsfExecutorTest extends Specification { // LSF executor def executor = createExecutor() executor.session = new Session() - executor.@memUnit = 'MB' - executor.@usageUnit = 'MB' and: // mock process def proc = Mock(TaskProcessor) @@ -477,8 +445,8 @@ class LsfExecutorTest extends Specification { #BSUB -n 2 #BSUB -R "span[hosts=1]" #BSUB -W 01:30 - #BSUB -M 4096 - #BSUB -R "select[mem>=8192] rusage[mem=8192]" + #BSUB -M 4096MB + #BSUB -R "select[mem>=8192MB] rusage[mem=8192MB]" #BSUB -J nf-mapping_hola #BSUB -x 1 #BSUB -R "span[ptile=2]" @@ -608,26 +576,6 @@ class LsfExecutorTest extends Specification { executor.getSubmitCommandLine(Mock(TaskRun), Mock(Path)) == ['bsub'] } - def 'should apply lsf mem unit' () { - given: - def executor = createExecutor() - executor.session = Mock(Session) - - when: - executor.register() - then: - 1 * executor.parseLsfConfig() >> [:] - executor.memUnit == 'KB' - executor.usageUnit == 'MB' - - when: - executor.register() - then: - 1 * executor.parseLsfConfig() >> ['LSF_UNIT_FOR_LIMITS': 'GB'] - executor.memUnit == 'GB' - executor.usageUnit == 'GB' - } - def 'should apply per task reserve' () { given: @@ -738,7 +686,6 @@ class LsfExecutorTest extends Specification { config.LSF_LOGDIR == '/common/foo/bar/log' config.LSF_LOG_MASK=='LOG_WARNING' config.LSF_LIM_PORT == '7869' - config.LSF_UNIT_FOR_LIMITS == 'GB' config.LSF_STRIP_DOMAIN == '.cbio.private:.cbio.delta.org:.delta.org' config.LSF_MASTER_LIST == "omega-sched01 omega-sched02" config.LSF_API_CONNTIMEOUT == '10' diff --git a/modules/nextflow/src/test/resources/nextflow/executor/lsf.conf b/modules/nextflow/src/test/resources/nextflow/executor/lsf.conf index 03df9a093b..04a3050742 100644 --- a/modules/nextflow/src/test/resources/nextflow/executor/lsf.conf +++ b/modules/nextflow/src/test/resources/nextflow/executor/lsf.conf @@ -56,8 +56,6 @@ LSB_MOD_ALL_JOBS=N # Reduce pim update frequency LSF_PIM_SLEEPTIME_UPDATE=Y LSF_PIM_LINUX_ENHANCE=Y -LSF_UNIT_FOR_LIMITS=GB -#LSF_UNIT_FOR_LIMITS=MB # Do not lock lim when running exclusive jobs LSB_DISABLE_LIMLOCK_EXCL=Y # Display the execution host in the output of the command bsub -K From 911d39931b7fd4b1eedb07a63ba2814209706242 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 27 Feb 2026 08:49:10 -0600 Subject: [PATCH 43/75] Treat LSF job status UNKWN as HOLD (#5756) --- .../src/main/groovy/nextflow/executor/LsfExecutor.groovy | 2 +- .../src/test/groovy/nextflow/executor/LsfExecutorTest.groovy | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/executor/LsfExecutor.groovy b/modules/nextflow/src/main/groovy/nextflow/executor/LsfExecutor.groovy index eead0de136..ddf5ea72b5 100644 --- a/modules/nextflow/src/main/groovy/nextflow/executor/LsfExecutor.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/executor/LsfExecutor.groovy @@ -202,7 +202,7 @@ class LsfExecutor extends AbstractGridExecutor implements TaskArrayExecutor { 'SSUSP': QueueStatus.HOLD, 'DONE': QueueStatus.DONE, 'EXIT': QueueStatus.ERROR, - 'UNKWN': QueueStatus.ERROR, + 'UNKWN': QueueStatus.HOLD, 'ZOMBI': QueueStatus.ERROR, ] diff --git a/modules/nextflow/src/test/groovy/nextflow/executor/LsfExecutorTest.groovy b/modules/nextflow/src/test/groovy/nextflow/executor/LsfExecutorTest.groovy index fc966d438f..312d4f570a 100644 --- a/modules/nextflow/src/test/groovy/nextflow/executor/LsfExecutorTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/executor/LsfExecutorTest.groovy @@ -507,7 +507,7 @@ class LsfExecutorTest extends Specification { result['5085604'] == AbstractGridExecutor.QueueStatus.PENDING result['5085611'] == AbstractGridExecutor.QueueStatus.HOLD result['5085107'] == AbstractGridExecutor.QueueStatus.ERROR - result['5085607'] == AbstractGridExecutor.QueueStatus.ERROR + result['5085607'] == AbstractGridExecutor.QueueStatus.HOLD result['5085608'] == AbstractGridExecutor.QueueStatus.ERROR result['5085609'] == AbstractGridExecutor.QueueStatus.RUNNING result['5085702'] == AbstractGridExecutor.QueueStatus.RUNNING From d17e6c10b3fb2c566d6116e81b7bede373428d64 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 27 Feb 2026 10:32:18 -0600 Subject: [PATCH 44/75] Allow boolean params to implicitly default to false (#6764) --- docs/workflow.md | 4 ++-- .../src/main/groovy/nextflow/script/ParamsDsl.groovy | 2 ++ .../test/groovy/nextflow/script/ParamsDslTest.groovy | 8 ++++---- tests/checks/params-dsl.nf/.checks | 11 +++++++++++ tests/params-dsl.config | 1 + tests/params-dsl.nf | 8 ++++++-- 6 files changed, 26 insertions(+), 8 deletions(-) diff --git a/docs/workflow.md b/docs/workflow.md index b1e901c377..2629f2251a 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -49,7 +49,7 @@ params { input: Path // Whether to save intermediate files. - save_intermeds: Boolean = false + save_intermeds: Boolean } ``` @@ -69,7 +69,7 @@ As a best practice, parameters should only be referenced in the entry workflow o The default value can be overridden by the command line, params file, or config file. Parameters from multiple sources are resolved in the order described in {ref}`cli-params`. Parameters specified on the command line are converted to the appropriate type based on the corresponding type annotation. -A parameter that doesn't specify a default value is a *required* parameter. If a required parameter is not given a value at runtime, the run will fail. +A parameter that doesn't specify a default value is a *required* parameter. If a required parameter is not given a value at runtime, the run will fail. Boolean parameters that don't specify a default value default to `false`. :::{versionadded} 26.04.0 ::: diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ParamsDsl.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ParamsDsl.groovy index a4e28d48ab..2126909b52 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ParamsDsl.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ParamsDsl.groovy @@ -46,6 +46,8 @@ class ParamsDsl { private Map declarations = [:] void declare(String name, Class type, boolean optional, Object defaultValue = null) { + if( defaultValue == null && type == Boolean ) + defaultValue = false declarations[name] = new Param(name, type, optional, defaultValue) } diff --git a/modules/nextflow/src/test/groovy/nextflow/script/ParamsDslTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/ParamsDslTest.groovy index 67a4245aa6..d566ce86bd 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/ParamsDslTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/ParamsDslTest.groovy @@ -26,7 +26,7 @@ class ParamsDslTest extends Specification { def dsl = new ParamsDsl() dsl.declare('input', Path, false) dsl.declare('chunk_size', Integer, false, 1) - dsl.declare('save_intermeds', Boolean, false, false) + dsl.declare('save_intermeds', Boolean, false) dsl.apply(session) then: session.binding.getParams() == [input: FileHelper.asPath('./data'), chunk_size: 3, save_intermeds: false, outdir: 'results'] @@ -55,7 +55,7 @@ class ParamsDslTest extends Specification { when: def dsl = new ParamsDsl() dsl.declare('input', Path, false) - dsl.declare('save_intermeds', Boolean, false, false) + dsl.declare('save_intermeds', Boolean, false) dsl.apply(session) then: def e = thrown(ScriptRuntimeException) @@ -72,7 +72,7 @@ class ParamsDslTest extends Specification { when: def dsl = new ParamsDsl() dsl.declare('input', Path, false) - dsl.declare('save_intermeds', Boolean, false, false) + dsl.declare('save_intermeds', Boolean, false) dsl.apply(session) then: def e = thrown(ScriptRuntimeException) @@ -89,7 +89,7 @@ class ParamsDslTest extends Specification { when: def dsl = new ParamsDsl() dsl.declare('input', Path, false) - dsl.declare('save_intermeds', Boolean, false, false) + dsl.declare('save_intermeds', Boolean, false) dsl.apply(session) then: def e = thrown(ScriptRuntimeException) diff --git a/tests/checks/params-dsl.nf/.checks b/tests/checks/params-dsl.nf/.checks index 76af690189..1b4b2ae6cb 100644 --- a/tests/checks/params-dsl.nf/.checks +++ b/tests/checks/params-dsl.nf/.checks @@ -5,6 +5,7 @@ $NXF_RUN --input ./data > stdout < stdout grep -F 'params.input = [./data]' < stdout grep -F 'params.save_intermeds = false' +< stdout grep -F 'params.method = auto' echo echo "Test missing required param" @@ -17,6 +18,15 @@ set -e < stdout grep -F 'Parameter `input` is required' +echo +echo "Test overwrite script param from command line" +echo +$NXF_RUN -c ../../params-dsl.config --input 'alpha,beta' --save_intermeds --method special > stdout + +< stdout grep -F 'params.input = [alpha, beta]' +< stdout grep -F 'params.save_intermeds = true' +< stdout grep -F 'params.method = special' + echo echo "Test overwrite script param from config profile" echo @@ -24,6 +34,7 @@ $NXF_RUN -c ../../params-dsl.config -profile test > stdout < stdout grep -F 'params.input = [alpha, beta, delta]' < stdout grep -F 'params.save_intermeds = true' +< stdout grep -F 'params.method = special' echo echo "Test invalid param" diff --git a/tests/params-dsl.config b/tests/params-dsl.config index 18ba6c6c01..301b337018 100644 --- a/tests/params-dsl.config +++ b/tests/params-dsl.config @@ -5,5 +5,6 @@ profiles { test { params.input = 'alpha,beta,delta' params.save_intermeds = true + params.method = 'special' } } diff --git a/tests/params-dsl.nf b/tests/params-dsl.nf index f14088ebdb..51b9d59921 100644 --- a/tests/params-dsl.nf +++ b/tests/params-dsl.nf @@ -16,15 +16,19 @@ */ params { - // List of IDs. + // Comma-separated list of IDs. input: String // Whether to save intermediate outputs. - save_intermeds: Boolean = false + save_intermeds: Boolean + + // Method to use for analyzing samples. + method: String = 'auto' } workflow { main: println "params.input = ${params.input.tokenize(',')}" println "params.save_intermeds = ${params.save_intermeds}" + println "params.method = ${params.method}" } From d0a9fbd037236fe383847ac1a94c7e75bdfb074f Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 27 Feb 2026 18:52:18 +0100 Subject: [PATCH 45/75] Sched core implementation alpha1 (#6242) Signed-off-by: Paolo Di Tommaso Co-authored-by: Claude Opus 4.5 Co-authored-by: Lorenzo Fontana --- VERSION | 2 +- docs/executor.md | 75 ++ docs/reference/config.md | 76 ++ .../nextflow/executor/res/DiskResource.groovy | 30 +- .../nextflow/script/PlatformMetadata.groovy | 51 ++ .../nextflow/script/WorkflowMetadata.groovy | 17 +- .../groovy/nextflow/trace/TraceRecord.groovy | 9 + .../main/resources/META-INF/plugins-info.txt | 1 + .../script/PlatformMetadataTest.groovy | 74 ++ .../main/nextflow/plugin/PluginsFacade.groovy | 3 + .../nextflow/plugin/PluginsFacadeTest.groovy | 8 + packing.gradle | 4 +- plugins/nf-seqera/README.md | 64 ++ plugins/nf-seqera/VERSION | 1 + plugins/nf-seqera/build.gradle | 54 ++ .../main/io/seqera/config/ExecutorOpts.groovy | 172 ++++ .../config/MachineRequirementOpts.groovy | 171 ++++ .../main/io/seqera/config/RetryOpts.groovy | 93 +++ .../main/io/seqera/config/SeqeraConfig.groovy | 54 ++ .../seqera/executor/InputFilesProfiler.groovy | 144 ++++ .../src/main/io/seqera/executor/Labels.groovy | 110 +++ .../executor/SeqeraBatchSubmitter.groovy | 318 +++++++ .../io/seqera/executor/SeqeraExecutor.groovy | 201 +++++ .../seqera/executor/SeqeraTaskHandler.groovy | 383 +++++++++ .../main/io/seqera/plugin/SeqeraPlugin.groovy | 35 + .../io/seqera/util/SchemaMapperUtil.groovy | 253 ++++++ .../io/seqera/config/ExecutorOptsTest.groovy | 244 ++++++ .../config/MachineRequirementOptsTest.groovy | 67 ++ .../io/seqera/config/RetryOptsTest.groovy | 59 ++ .../io/seqera/config/SeqeraConfigTest.groovy | 96 +++ .../executor/InputFilesProfilerTest.groovy | 202 +++++ .../test/io/seqera/executor/LabelsTest.groovy | 201 +++++ .../executor/SeqeraBatchSubmitterTest.groovy | 593 +++++++++++++ .../seqera/executor/SeqeraExecutorTest.groovy | 129 +++ .../executor/SeqeraTaskHandlerTest.groovy | 779 ++++++++++++++++++ .../test/io/seqera/util/MapperUtilTest.groovy | 550 +++++++++++++ plugins/nf-tower/VERSION | 2 +- .../io/seqera/tower/plugin/TowerClient.groovy | 8 +- .../tower/plugin/TowerClientTest.groovy | 32 +- settings.gradle | 3 + 40 files changed, 5359 insertions(+), 9 deletions(-) create mode 100644 modules/nextflow/src/main/groovy/nextflow/script/PlatformMetadata.groovy create mode 100644 modules/nextflow/src/test/groovy/nextflow/script/PlatformMetadataTest.groovy create mode 100644 plugins/nf-seqera/README.md create mode 100644 plugins/nf-seqera/VERSION create mode 100644 plugins/nf-seqera/build.gradle create mode 100644 plugins/nf-seqera/src/main/io/seqera/config/ExecutorOpts.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/config/RetryOpts.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/config/SeqeraConfig.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/executor/InputFilesProfiler.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/executor/Labels.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/executor/SeqeraBatchSubmitter.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/executor/SeqeraExecutor.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/executor/SeqeraTaskHandler.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/plugin/SeqeraPlugin.groovy create mode 100644 plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/config/ExecutorOptsTest.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/config/MachineRequirementOptsTest.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/config/RetryOptsTest.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/config/SeqeraConfigTest.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/executor/InputFilesProfilerTest.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/executor/LabelsTest.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/executor/SeqeraBatchSubmitterTest.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/executor/SeqeraExecutorTest.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/executor/SeqeraTaskHandlerTest.groovy create mode 100644 plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy diff --git a/VERSION b/VERSION index 5f38a2f391..729e722712 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -26.01.1-edge +26.02.0-edge diff --git a/docs/executor.md b/docs/executor.md index c861301ec0..b2e90f9b72 100644 --- a/docs/executor.md +++ b/docs/executor.md @@ -414,6 +414,81 @@ Resource requests and other job characteristics can be controlled via the follow - {ref}`process-queue` - {ref}`process-time` +(seqera-executor)= + +## Seqera + +:::{versionadded} 26.04.0 +::: + +:::{warning} +*Preview feature: may change in a future release.* +::: + +The `seqera` executor allows you to run your pipeline using the [Seqera](https://seqera.io) cloud infrastructure. It enables the seamless execution of Nextflow pipelines by offloading process executions to the Seqera scheduler service. + +The pipeline processes must specify the Docker image to use by defining the `container` directive, either in the pipeline script or the `nextflow.config` file. Additionally, an S3 bucket must be used as the pipeline work directory. + +To enable this executor, set `process.executor = 'seqera'` in the `nextflow.config` file. + +Resource requests and other job characteristics can be controlled via the following process directives: + +- {ref}`process-arch` +- {ref}`process-container` +- {ref}`process-containerOptions` +- {ref}`process-cpus` +- {ref}`process-disk` +- {ref}`process-memory` +- {ref}`process-time` + +### Disk support + +When the {ref}`process-disk` directive is specified, the Seqera executor provisions storage for the task container. There are two disk allocation strategies: + +- **task** (default): A dedicated EBS volume is created for each task at launch time. This provides isolated, high-performance storage with configurable volume type, IOPS, throughput, and encryption. + +- **node**: Uses the instance storage attached at the cluster level. This is shared across tasks running on the same node and does not support EBS-specific options. + +#### Task allocation (EBS volumes) + +By default, a gp3 volume with 325 MiB/s throughput is used (Fusion recommended settings). You can customize the EBS volume configuration: + +```groovy +seqera { + executor { + machineRequirement { + diskAllocation = 'task' // Per-task EBS volume (default) + diskType = 'ebs/io1' // Use provisioned IOPS SSD + diskIops = 10000 // Required for io1/io2 + diskThroughputMiBps = 500 // Throughput for gp3 volumes + diskEncrypted = true // Enable KMS encryption + } + } +} +``` + +Supported volume types: `ebs/gp3` (default), `ebs/gp2`, `ebs/io1`, `ebs/io2`, `ebs/st1`, `ebs/sc1`. + +#### Node allocation (instance storage) + +To use instance storage instead of per-task EBS volumes: + +```groovy +seqera { + executor { + machineRequirement { + diskAllocation = 'node' // Use instance storage + } + } +} +``` + +:::{note} +When using `node` allocation, the EBS-specific options (`diskType`, `diskIops`, `diskThroughputMiBps`, `diskEncrypted`) are not applicable and will cause an error if specified. +::: + +See the {ref}`seqera scope ` for the available configuration options. + (slurm-executor)= ## SLURM diff --git a/docs/reference/config.md b/docs/reference/config.md index a776dbf95f..9d110c13cc 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -1409,6 +1409,82 @@ The following settings are available: `sarus.tty` : Allocates a pseudo-tty (default: `false`). +(config-seqera)= + +## `seqera` + +:::{versionadded} 26.04.0 +::: + +:::{warning} +*Preview feature: may change in a future release.* +::: + +The `seqera` scope allows you to configure the interactions with Seqera services. + +### `executor` + +The `seqera.executor` scope configures the Seqera scheduler service for the {ref}`seqera-executor`. + +The following settings are available: + +`seqera.executor.endpoint` +: The Seqera scheduler service endpoint URL (required). + +`seqera.executor.region` +: The AWS region for task execution (default: `'eu-central-1'`). + +`seqera.executor.autoLabels` +: When `true`, automatically adds workflow metadata labels to the session with the `nextflow.io/` prefix (default: `false`). The following labels are added: `projectName`, `userName`, `runName`, `sessionId`, `resume`, `revision`, `commitId`, `repository`, `manifestName`, `runtimeVersion`. A `seqera.io/runId` label is also added, computed as a SipHash of the session ID and run name. + +`seqera.executor.labels` +: Custom labels to apply to AWS resources for cost tracking and resource organization. Labels are propagated to ECS tasks, capacity providers, and EC2 instances. When used together with `autoLabels`, user-defined labels take precedence over auto-generated labels. + +`seqera.executor.machineRequirement.arch` +: The CPU architecture for task execution, e.g. `'x86_64'` or `'arm64'`. + +`seqera.executor.machineRequirement.provisioning` +: The instance provisioning mode. Can be `'spot'`, `'ondemand'`, or `'spotFirst'`. + +`seqera.executor.machineRequirement.maxSpotAttempts` +: The maximum number of spot retry attempts before falling back to on-demand. Only used when `provisioning` is `'spot'` or `'spotFirst'`. + +`seqera.executor.machineRequirement.machineFamilies` +: List of acceptable EC2 instance families, e.g. `['m5', 'c5', 'r5']`. + +`seqera.executor.machineRequirement.diskAllocation` +: The disk allocation strategy. Can be `'task'` (default) for per-task EBS volumes, or `'node'` for per-node instance storage. When using `'node'` allocation, EBS-specific options (`diskType`, `diskIops`, `diskThroughputMiBps`, `diskEncrypted`) are not applicable. + +`seqera.executor.machineRequirement.diskType` +: The EBS volume type for task scratch disk. Supported types: `'ebs/gp3'` (default), `'ebs/gp2'`, `'ebs/io1'`, `'ebs/io2'`, `'ebs/st1'`, `'ebs/sc1'`. Only applicable when `diskAllocation` is `'task'`. + +`seqera.executor.machineRequirement.diskThroughputMiBps` +: The throughput in MiB/s for gp3 volumes (125-1000). Default: `325` (Fusion recommended). Only applicable when `diskAllocation` is `'task'`. + +`seqera.executor.machineRequirement.diskIops` +: The IOPS for io1/io2/gp3 volumes. Required for io1/io2 volume types. Only applicable when `diskAllocation` is `'task'`. + +`seqera.executor.machineRequirement.diskEncrypted` +: Enable KMS encryption for the EBS volume (default: `false`). Only applicable when `diskAllocation` is `'task'`. + +`seqera.executor.taskEnvironment` +: Custom environment variables to apply to all tasks submitted by the Seqera executor. These are merged with the Fusion environment variables, with Fusion variables taking precedence. For example: `taskEnvironment = [MY_VAR: 'value']`. + +`seqera.executor.retryPolicy.delay` +: The initial delay when a failing HTTP request is retried (default: `'450ms'`). + +`seqera.executor.retryPolicy.maxDelay` +: The maximum delay when a failing HTTP request is retried (default: `'90s'`). + +`seqera.executor.retryPolicy.maxAttempts` +: The maximum number of retry attempts (default: `10`). + +`seqera.executor.retryPolicy.jitter` +: The jitter factor for randomizing retry delays (default: `0.25`). + +`seqera.executor.retryPolicy.multiplier` +: The multiplier for exponential backoff (default: `2.0`). + (config-shifter)= ## `shifter` diff --git a/modules/nextflow/src/main/groovy/nextflow/executor/res/DiskResource.groovy b/modules/nextflow/src/main/groovy/nextflow/executor/res/DiskResource.groovy index 25939c7e90..7364bb0ff0 100644 --- a/modules/nextflow/src/main/groovy/nextflow/executor/res/DiskResource.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/executor/res/DiskResource.groovy @@ -22,9 +22,10 @@ import groovy.transform.ToString import nextflow.util.MemoryUnit /** - * Models disk resource request - * + * Models disk resource request with support for cloud-specific options. + * * @author Ben Sherman + * @author Paolo Di Tommaso */ @ToString(includeNames = true, includePackage = false) @CompileStatic @@ -33,6 +34,11 @@ class DiskResource { final MemoryUnit request final String type + final Integer iops + final Integer throughput + final Boolean encrypted + final String filesystem + final String mountPath DiskResource( value ) { this(request: value) @@ -43,10 +49,28 @@ class DiskResource { if( opts.type ) this.type = opts.type as String + if( opts.iops ) + this.iops = opts.iops as Integer + if( opts.throughput ) + this.throughput = opts.throughput as Integer + if( opts.encrypted != null ) + this.encrypted = opts.encrypted as Boolean + if( opts.filesystem ) + this.filesystem = opts.filesystem as String + if( opts.mountPath ) + this.mountPath = opts.mountPath as String } DiskResource withRequest(MemoryUnit value) { - return new DiskResource(request: value, type: this.type) + return new DiskResource( + request: value, + type: this.type, + iops: this.iops, + throughput: this.throughput, + encrypted: this.encrypted, + filesystem: this.filesystem, + mountPath: this.mountPath + ) } private static MemoryUnit toMemoryUnit( value ) { diff --git a/modules/nextflow/src/main/groovy/nextflow/script/PlatformMetadata.groovy b/modules/nextflow/src/main/groovy/nextflow/script/PlatformMetadata.groovy new file mode 100644 index 0000000000..1130d03d36 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/script/PlatformMetadata.groovy @@ -0,0 +1,51 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package nextflow.script + +import groovy.transform.CompileStatic +import groovy.transform.EqualsAndHashCode +import groovy.transform.ToString + +/** + * Models Seqera Platform metadata for Nextflow execution + * + * @author Paolo Di Tommaso + */ +@CompileStatic +@ToString(includeNames = true, includePackage = false) +@EqualsAndHashCode +class PlatformMetadata { + + /** + * Volatile because it is written by TowerClient.onFlowCreate on the main thread + * and read by SeqeraExecutor.createRun on the executor thread. + */ + volatile String workflowId + + /** + * The Platform watch URL for the current workflow execution. + * Set by TowerClient.onFlowBegin and read by SeqeraExecutor.createRun. + */ + volatile String workflowUrl + + PlatformMetadata() {} + + PlatformMetadata(String workflowId) { + this.workflowId = workflowId + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/script/WorkflowMetadata.groovy b/modules/nextflow/src/main/groovy/nextflow/script/WorkflowMetadata.groovy index 394d39dbf9..70c56146f9 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/WorkflowMetadata.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/WorkflowMetadata.groovy @@ -34,7 +34,6 @@ import nextflow.exception.WorkflowScriptErrorException import nextflow.trace.WorkflowStats import nextflow.util.Duration import nextflow.util.TestOnly -import org.codehaus.groovy.runtime.InvokerHelper /** * Models workflow metadata properties and notification handler * @@ -218,6 +217,12 @@ class WorkflowMetadata { */ FusionMetadata fusion + /** + * Metadata specific to Seqera Platform, including: + *
  • workflowId: the Platform-assigned workflow identifier + */ + PlatformMetadata platform + /** * The list of files that concurred to create the config object */ @@ -497,4 +502,14 @@ class WorkflowMetadata { session.statsObserver.getStats() } + PlatformMetadata getPlatform() { + if( platform!=null ) + return platform + synchronized (this) { + if( platform!=null ) + return platform + platform = new PlatformMetadata() + } + return platform + } } diff --git a/modules/nextflow/src/main/groovy/nextflow/trace/TraceRecord.groovy b/modules/nextflow/src/main/groovy/nextflow/trace/TraceRecord.groovy index f35d54c599..3fc2ba1323 100644 --- a/modules/nextflow/src/main/groovy/nextflow/trace/TraceRecord.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/trace/TraceRecord.groovy @@ -124,6 +124,7 @@ class TraceRecord implements Serializable { transient private CloudMachineInfo machineInfo transient private ContainerMeta containerMeta transient private Integer numSpotInterruptions + transient private String logStreamId /** * Convert the given value to a string @@ -622,6 +623,14 @@ class TraceRecord implements Serializable { this.numSpotInterruptions = numSpotInterruptions } + String getLogStreamId() { + return logStreamId + } + + void setLogStreamId(String logStreamId) { + this.logStreamId = logStreamId + } + ContainerMeta getContainerMeta() { return containerMeta } diff --git a/modules/nextflow/src/main/resources/META-INF/plugins-info.txt b/modules/nextflow/src/main/resources/META-INF/plugins-info.txt index e08664196a..1f26a3c37c 100644 --- a/modules/nextflow/src/main/resources/META-INF/plugins-info.txt +++ b/modules/nextflow/src/main/resources/META-INF/plugins-info.txt @@ -5,5 +5,6 @@ nf-codecommit@0.5.0 nf-console@1.3.0 nf-google@1.26.0 nf-k8s@1.5.0 +nf-seqera@0.6.0 nf-tower@1.20.0 nf-wave@1.18.0 \ No newline at end of file diff --git a/modules/nextflow/src/test/groovy/nextflow/script/PlatformMetadataTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/PlatformMetadataTest.groovy new file mode 100644 index 0000000000..3fac5cbd10 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/script/PlatformMetadataTest.groovy @@ -0,0 +1,74 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package nextflow.script + +import spock.lang.Specification + +/** + * Tests for {@link PlatformMetadata} + * + * @author Paolo Di Tommaso + */ +class PlatformMetadataTest extends Specification { + + def 'should create with default constructor'() { + when: + def meta = new PlatformMetadata() + + then: + meta.workflowId == null + } + + def 'should create with workflowId'() { + when: + def meta = new PlatformMetadata('abc123') + + then: + meta.workflowId == 'abc123' + } + + def 'should allow setting workflowId after construction'() { + given: + def meta = new PlatformMetadata() + + when: + meta.workflowId = 'xyz789' + + then: + meta.workflowId == 'xyz789' + } + + def 'should create with null workflowUrl by default'() { + when: + def meta = new PlatformMetadata() + + then: + meta.workflowUrl == null + } + + def 'should allow setting workflowUrl after construction'() { + given: + def meta = new PlatformMetadata() + + when: + meta.workflowUrl = 'https://cloud.seqera.io/watch/abc123' + + then: + meta.workflowUrl == 'https://cloud.seqera.io/watch/abc123' + } +} diff --git a/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy b/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy index d5ff5c4ea9..1969f9371d 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy @@ -456,6 +456,9 @@ class PluginsFacade implements PluginStateListener { if( (Bolts.navigate(config,'wave.enabled') || Bolts.navigate(config,'fusion.enabled')) && !specs.find {it.id == 'nf-wave' } ) { specs << defaultPlugins.getPlugin('nf-wave') } + if( Bolts.navigate(config,'process.executor')=='seqera') { + specs << defaultPlugins.getPlugin('nf-seqera') + } // add cloudcache plugin when cloudcache is enabled in the config if( Bolts.navigate(config, 'cloudcache.enabled')==true ) { diff --git a/modules/nf-commons/src/test/nextflow/plugin/PluginsFacadeTest.groovy b/modules/nf-commons/src/test/nextflow/plugin/PluginsFacadeTest.groovy index 014f13e21b..3fc0e5b47f 100644 --- a/modules/nf-commons/src/test/nextflow/plugin/PluginsFacadeTest.groovy +++ b/modules/nf-commons/src/test/nextflow/plugin/PluginsFacadeTest.groovy @@ -90,6 +90,7 @@ class PluginsFacadeTest extends Specification { 'nf-amazon': new PluginRef('nf-amazon', '0.1.0'), 'nf-cloudcache': new PluginRef('nf-cloudcache', '0.1.0'), 'nf-google': new PluginRef('nf-google', '0.1.0'), + 'nf-seqera': new PluginRef('nf-seqera', '0.1.0'), 'nf-tower': new PluginRef('nf-tower', '0.1.0'), 'nf-wave': new PluginRef('nf-wave', '0.1.0') ]) @@ -174,6 +175,13 @@ class PluginsFacadeTest extends Specification { then: result == [ new PluginRef('nf-cloudcache', '0.1.0') ] + // seqera executor requires nf-seqera plugin + when: + handler = new PluginsFacade(defaultPlugins: defaults, env: [:]) + result = handler.pluginsRequirement([process:[executor:'seqera']]) + then: + result == [ new PluginRef('nf-seqera', '0.1.0') ] + } def 'should return default plugins given config' () { diff --git a/packing.gradle b/packing.gradle index dca30c9bfe..25a5a7b2e8 100644 --- a/packing.gradle +++ b/packing.gradle @@ -8,13 +8,14 @@ configurations { legacy.extendsFrom defaultCfg tower.extendsFrom defaultCfg wave.extendsFrom defaultCfg + seqera.extendsFrom defaultCfg } dependencies { api project(':nextflow') // include Ivy at runtime in order to have Grape @Grab work correctly defaultCfg "org.apache.ivy:ivy:2.5.2" - // default cfg = runtime + httpfs + lineage + amazon + tower client + wave client + // default cfg = runtime + httpfs + lineage defaultCfg project(':nf-httpfs') defaultCfg project(':nf-lineage') console project(':plugins:nf-console') @@ -23,6 +24,7 @@ dependencies { azure project(':plugins:nf-azure') tower project(':plugins:nf-tower') wave project(':plugins:nf-wave') + seqera project(':plugins:nf-seqera') } diff --git a/plugins/nf-seqera/README.md b/plugins/nf-seqera/README.md new file mode 100644 index 0000000000..f8f30ec1c8 --- /dev/null +++ b/plugins/nf-seqera/README.md @@ -0,0 +1,64 @@ +# Seqera Executor plugin for Nextflow + +The Seqera Executor plugin provides integration with Seqera Cloud for executing Nextflow tasks using Seqera's managed compute infrastructure. + +## Get Started + +To use this plugin, add it to your `nextflow.config`: + +```groovy +plugins { + id 'nf-seqera' +} +``` + +Configure the Seqera executor: + +```groovy +process.executor = 'seqera' + +seqera { + endpoint = '' + accessToken = '' +} +``` + +Alternatively, set the access token via environment variable: + +```bash +export SEQERA_ACCESS_TOKEN='' +``` + +## Examples + +### Basic Configuration + +```groovy +plugins { + id 'nf-seqera' +} + +process.executor = 'seqera' + +seqera { + endpoint = 'https://api.cloud.seqera.io' + accessToken = System.getenv('SEQERA_ACCESS_TOKEN') +} +``` + +### Custom Endpoint + +```groovy +seqera { + endpoint = 'https://seqera.mycompany.com/api' + accessToken = System.getenv('SEQERA_ACCESS_TOKEN') +} +``` + +## Resources + +- [Seqera Platform Documentation](https://docs.seqera.io/) + +## License + +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) \ No newline at end of file diff --git a/plugins/nf-seqera/VERSION b/plugins/nf-seqera/VERSION new file mode 100644 index 0000000000..ac454c6a1f --- /dev/null +++ b/plugins/nf-seqera/VERSION @@ -0,0 +1 @@ +0.12.0 diff --git a/plugins/nf-seqera/build.gradle b/plugins/nf-seqera/build.gradle new file mode 100644 index 0000000000..724cd6ff31 --- /dev/null +++ b/plugins/nf-seqera/build.gradle @@ -0,0 +1,54 @@ +/* + * Copyright (c) 2019, Seqera Labs. + * + * This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + * + * This Source Code Form is "Incompatible With Secondary Licenses", as + * defined by the Mozilla Public License, v. 2.0. + */ + +plugins { + id 'io.nextflow.nextflow-plugin' version "${nextflowPluginVersion}" + id 'java-test-fixtures' +} + +nextflowPlugin { + nextflowVersion = '26.02.0-edge' + + provider = "${nextflowPluginProvider}" + description = 'Integrates with Seqera Platform for comprehensive workflow monitoring, resource tracking, and cache management capabilities' + className = 'io.seqera.plugin.SeqeraPlugin' + useDefaultDependencies = false + generateSpec = false + extensionPoints = [ + 'io.seqera.executor.SeqeraExecutor', + 'io.seqera.config.SeqeraConfig' + ] +} + +sourceSets { + main.java.srcDirs = [] + main.groovy.srcDirs = ['src/main'] + main.resources.srcDirs = ['src/resources'] + test.groovy.srcDirs = ['src/test'] + test.java.srcDirs = [] + test.resources.srcDirs = [] +} + +configurations { + // see https://docs.gradle.org/4.1/userguide/dependency_management.html#sub:exclude_transitive_dependencies + runtimeClasspath.exclude group: 'org.slf4j', module: 'slf4j-api' +} + +dependencies { + compileOnly project(':nextflow') + compileOnly 'org.slf4j:slf4j-api:2.0.17' + compileOnly 'org.pf4j:pf4j:3.12.0' + api 'io.seqera:sched-client:0.35.0-SNAPSHOT' + + testImplementation(testFixtures(project(":nextflow"))) + testImplementation "org.apache.groovy:groovy:4.0.30" + testImplementation "org.apache.groovy:groovy-nio:4.0.30" +} diff --git a/plugins/nf-seqera/src/main/io/seqera/config/ExecutorOpts.groovy b/plugins/nf-seqera/src/main/io/seqera/config/ExecutorOpts.groovy new file mode 100644 index 0000000000..217277ab56 --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/config/ExecutorOpts.groovy @@ -0,0 +1,172 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.config + +import groovy.transform.CompileStatic +import nextflow.config.spec.ConfigOption +import nextflow.config.spec.ConfigScope +import nextflow.script.dsl.Description +import nextflow.util.Duration + +/** + * Configuration for the Seqera executor. + * + * @author Paolo Di Tommaso + */ +@Description(""" + The `seqera.executor` scope provides configuration for the Seqera compute executor. +""") +@CompileStatic +class ExecutorOpts implements ConfigScope { + + final RetryOpts retryPolicy + + @ConfigOption + @Description(""" + The Seqera scheduler service endpoint URL. + """) + final String endpoint + + @ConfigOption + @Description(""" + The AWS region for task execution (default: `eu-central-1`). + """) + final String region + + @ConfigOption + @Description(""" + The EC2 key pair name for SSH access to instances. + """) + final String keyPairName + + @ConfigOption + @Description(""" + The interval for batching task submissions (default: `1 sec`). + """) + final Duration batchFlushInterval + + @Description(""" + Machine/infrastructure requirements for session tasks. + """) + final MachineRequirementOpts machineRequirement + + @ConfigOption + @Description(""" + Custom labels to apply to AWS resources for cost tracking and resource organization. + Labels are propagated to ECS tasks, capacity providers, and EC2 instances. + """) + final Map labels + + @ConfigOption + @Description(""" + When `true`, automatically adds workflow metadata labels (e.g. project name, + run name, session ID) with the `nextflow.io/` prefix to the session (default: `false`). + """) + final boolean autoLabels + + @ConfigOption + @Description(""" + The resource prediction model to use for estimating task resource requirements + based on historical execution metrics. Supported values: `qr/v1` (quantile regression). + When not set, no resource estimation is applied. + """) + final String predictionModel + + @ConfigOption + @Description(""" + Custom environment variables to apply to all tasks submitted by the Seqera executor. + These are merged with the Fusion environment variables, with Fusion variables taking precedence. + """) + final Map taskEnvironment + + /* required by config scope -- do not remove */ + + ExecutorOpts() {} + + ExecutorOpts(Map opts) { + this.retryPolicy = new RetryOpts(opts.retryPolicy as Map ?: Map.of()) + this.endpoint = opts.endpoint as String + if (!endpoint) + throw new IllegalArgumentException("Missing Seqera endpoint - make sure to specify 'seqera.executor.endpoint' settings") + + this.region = opts.region as String ?: "eu-central-1" + this.keyPairName = opts.keyPairName as String + this.batchFlushInterval = opts.batchFlushInterval + ? Duration.of(opts.batchFlushInterval as String) + : Duration.of('1 sec') + // machine requirement settings + this.machineRequirement = new MachineRequirementOpts(opts.machineRequirement as Map ?: Map.of()) + // labels for cost tracking + this.labels = opts.labels as Map + this.autoLabels = opts.autoLabels as boolean ?: false + // prediction model + this.predictionModel = parsePredictionModel(opts.predictionModel as String) + // custom task environment variables + this.taskEnvironment = opts.taskEnvironment as Map + } + + private static final Set VALID_PREDICTION_MODELS = Set.of('qr/v1') + + private static String parsePredictionModel(String value) { + if( !value ) + return null + if( !VALID_PREDICTION_MODELS.contains(value) ) + throw new IllegalArgumentException("Invalid prediction model '${value}'. Supported values: ${VALID_PREDICTION_MODELS.join(', ')}") + return value + } + + RetryOpts retryOpts() { + this.retryPolicy + } + + String getEndpoint() { + return endpoint + } + + String getRegion() { + return region + } + + String getKeyPairName() { + return keyPairName + } + + Duration getBatchFlushInterval() { + return batchFlushInterval + } + + MachineRequirementOpts getMachineRequirement() { + return machineRequirement + } + + Map getLabels() { + return labels + } + + boolean getAutoLabels() { + return autoLabels + } + + String getPredictionModel() { + return predictionModel + } + + Map getTaskEnvironment() { + return taskEnvironment + } +} diff --git a/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy b/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy new file mode 100644 index 0000000000..6e5d40681e --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/config/MachineRequirementOpts.groovy @@ -0,0 +1,171 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.config + +import groovy.transform.CompileStatic +import nextflow.config.spec.ConfigOption +import nextflow.config.spec.ConfigScope +import nextflow.script.dsl.Description +import nextflow.util.MemoryUnit + +/** + * Machine/infrastructure requirements configuration options. + * + * @author Paolo Di Tommaso + */ +@CompileStatic +class MachineRequirementOpts implements ConfigScope { + + @ConfigOption + @Description(""" + The CPU architecture for task execution (e.g., `x86_64`, `arm64`). + """) + final String arch + + @ConfigOption + @Description(""" + The instance provisioning mode: `spot`, `ondemand`, or `spotFirst`. + """) + final String provisioning + + @ConfigOption + @Description(""" + Maximum number of spot retry attempts before falling back to on-demand. + Only used when provisioning is `spot` or `spotFirst`. + """) + final Integer maxSpotAttempts + + @ConfigOption + @Description(""" + List of acceptable machine type patterns. Supports exact types (e.g., `t3.small`), + family prefixes (e.g., `m5` matches all m5 sizes), and glob wildcards (e.g., `t*.small`). + """) + final List machineTypes + + @ConfigOption + @Description(""" + The EBS volume type for task scratch disk (e.g., `ebs/gp3`, `ebs/io1`). + Default: `ebs/gp3`. + """) + final String diskType + + @ConfigOption + @Description(""" + The throughput in MiB/s for gp3 volumes (125-1000). + Default: 325 (Fusion recommended). + """) + final Integer diskThroughputMiBps + + @ConfigOption + @Description(""" + The IOPS for io1/io2/gp3 volumes. Required for io1/io2. + """) + final Integer diskIops + + @ConfigOption + @Description(""" + Enable KMS encryption for the EBS volume. + Default: false. + """) + final Boolean diskEncrypted + + @ConfigOption + @Description(""" + The disk allocation strategy: `task` or `node`. + - `task`: Per-task EBS volume created at task launch (default) + - `node`: Per-node instance storage attached at cluster level + """) + final String diskAllocation + + @ConfigOption + @Description(""" + The disk size for session-level storage (e.g., `100.GB`). + """) + final MemoryUnit diskSize + + @ConfigOption + @Description(""" + The ECS capacity provider mode: `managed` (default) or `asg`. + - `managed`: ECS Managed Instances + - `asg`: Auto Scaling Group-backed capacity provider + """) + final String capacityMode + + /* required by config scope -- do not remove */ + MachineRequirementOpts() {} + + MachineRequirementOpts(Map opts) { + this.arch = opts.arch as String + this.provisioning = opts.provisioning as String + this.maxSpotAttempts = opts.maxSpotAttempts as Integer + this.machineTypes = (opts.machineTypes ?: opts.machineFamilies) as List + this.diskType = opts.diskType as String + this.diskThroughputMiBps = opts.diskThroughputMiBps as Integer + this.diskIops = opts.diskIops as Integer + this.diskEncrypted = opts.diskEncrypted as Boolean + this.diskAllocation = opts.diskAllocation as String + this.diskSize = opts.diskSize instanceof MemoryUnit + ? opts.diskSize as MemoryUnit + : (opts.diskSize ? MemoryUnit.of(opts.diskSize as String) : null) + this.capacityMode = opts.capacityMode as String + } + + String getArch() { + return arch + } + + String getProvisioning() { + return provisioning + } + + Integer getMaxSpotAttempts() { + return maxSpotAttempts + } + + List getMachineTypes() { + return machineTypes + } + + String getDiskType() { + return diskType + } + + Integer getDiskThroughputMiBps() { + return diskThroughputMiBps + } + + Integer getDiskIops() { + return diskIops + } + + Boolean getDiskEncrypted() { + return diskEncrypted + } + + String getDiskAllocation() { + return diskAllocation + } + + MemoryUnit getDiskSize() { + return diskSize + } + + String getCapacityMode() { + return capacityMode + } +} \ No newline at end of file diff --git a/plugins/nf-seqera/src/main/io/seqera/config/RetryOpts.groovy b/plugins/nf-seqera/src/main/io/seqera/config/RetryOpts.groovy new file mode 100644 index 0000000000..4d5107944e --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/config/RetryOpts.groovy @@ -0,0 +1,93 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.config + +import groovy.transform.CompileStatic +import groovy.transform.ToString +import io.seqera.util.retry.Retryable +import nextflow.config.spec.ConfigOption +import nextflow.config.spec.ConfigScope +import nextflow.script.dsl.Description +import nextflow.util.Duration + +/** + * Model retry options for Seqera scheduler HTTP requests. + * Implements {@link Retryable.Config} for integration with lib-retry. + */ +@ToString(includeNames = true, includePackage = false) +@CompileStatic +class RetryOpts implements ConfigScope, Retryable.Config { + + @ConfigOption + @Description(""" + The initial delay when a failing HTTP request is retried (default: `450ms`). + """) + Duration delay = Duration.of('450ms') + + @ConfigOption + @Description(""" + The max delay when a failing HTTP request is retried (default: `90s`). + """) + Duration maxDelay = Duration.of('90s') + + @ConfigOption + @Description(""" + The maximum number of retry attempts (default: `10`). + """) + int maxAttempts = 10 + + @ConfigOption + @Description(""" + The jitter factor for randomizing retry delays (default: `0.25`). + """) + double jitter = 0.25 + + @ConfigOption + @Description(""" + The multiplier for exponential backoff (default: `2.0`). + """) + double multiplier = 2.0d + + RetryOpts() { + this(Collections.emptyMap()) + } + + RetryOpts(Map config) { + if( config.delay ) + delay = config.delay as Duration + if( config.maxDelay ) + maxDelay = config.maxDelay as Duration + if( config.maxAttempts ) + maxAttempts = config.maxAttempts as int + if( config.jitter ) + jitter = config.jitter as double + if( config.multiplier ) + multiplier = config.multiplier as double + } + + // Methods required by Retryable.Config interface + @Override + java.time.Duration getDelayAsDuration() { + return java.time.Duration.ofMillis(delay.toMillis()) + } + + @Override + java.time.Duration getMaxDelayAsDuration() { + return java.time.Duration.ofMillis(maxDelay.toMillis()) + } +} diff --git a/plugins/nf-seqera/src/main/io/seqera/config/SeqeraConfig.groovy b/plugins/nf-seqera/src/main/io/seqera/config/SeqeraConfig.groovy new file mode 100644 index 0000000000..faaa36380c --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/config/SeqeraConfig.groovy @@ -0,0 +1,54 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.config + +import groovy.transform.CompileStatic +import nextflow.config.spec.ConfigScope +import nextflow.config.spec.ScopeName +import nextflow.script.dsl.Description + +/** + * Top-level configuration scope for Seqera settings. + * + * @author Paolo Di Tommaso + */ +@ScopeName("seqera") +@Description(""" + The `seqera` scope provides configuration for Seqera services. +""") +@CompileStatic +class SeqeraConfig implements ConfigScope { + + @Description(""" + Configuration for the Seqera compute executor. + """) + final ExecutorOpts executor + + /* required by config scope -- do not remove */ + SeqeraConfig() {} + + SeqeraConfig(Map opts) { + this.executor = opts.executor + ? new ExecutorOpts(opts.executor as Map) + : null + } + + ExecutorOpts getExecutor() { + return executor + } +} diff --git a/plugins/nf-seqera/src/main/io/seqera/executor/InputFilesProfiler.groovy b/plugins/nf-seqera/src/main/io/seqera/executor/InputFilesProfiler.groovy new file mode 100644 index 0000000000..92a5376a75 --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/executor/InputFilesProfiler.groovy @@ -0,0 +1,144 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import java.nio.file.FileVisitResult +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.SimpleFileVisitor +import java.nio.file.attribute.BasicFileAttributes + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.sched.api.schema.v1a1.InputFilesMetrics +import nextflow.file.FileHolder +import nextflow.processor.TaskRun + +/** + * Computes input files metrics for a task. + * Follows symlinks and recursively computes directory sizes. + * + * @author Paolo Di Tommaso + */ +@Slf4j +@CompileStatic +class InputFilesProfiler { + + /** + * Compute input files metrics for a task. + * + * @param task The task to compute metrics for + * @return InputFilesMetrics or null if no input files + */ + static InputFilesMetrics compute(TaskRun task) { + final files = task?.inputFiles + if( !files || files.isEmpty() ) + return null + + return compute0(files) + } + + /** + * Compute input files metrics from a list of file holders. + * + * @param files List of FileHolder objects + * @return InputFilesMetrics or null if list is empty + */ + static InputFilesMetrics compute(List files) { + if( !files || files.isEmpty() ) + return null + + return compute0(files) + } + + private static InputFilesMetrics compute0(List files) { + int totalCount = 0 + long totalBytes = 0 + long maxFileBytes = Long.MIN_VALUE + long minFileBytes = Long.MAX_VALUE + + for( FileHolder fh : files ) { + final long[] result = getFileStats(fh.storePath) + final count = (int) result[0] + final size = result[1] + totalCount += count + totalBytes += size + if( size > maxFileBytes ) + maxFileBytes = size + if( size < minFileBytes ) + minFileBytes = size + } + + return new InputFilesMetrics() + .count(totalCount) + .totalBytes(totalBytes) + .maxFileBytes(maxFileBytes) + .minFileBytes(minFileBytes) + } + + /** + * Get file stats for a path: file count and total size. + * For regular files, count is 1. For directories, count is the number of files within. + * + * @param path The path to measure + * @return A two-element array: [fileCount, totalSize] + */ + private static long[] getFileStats(Path path) { + if( path == null ) + return new long[]{0, 0} + + try { + if( Files.isDirectory(path) ) { + return computeDirStats(path) + } + // Files.size() follows symlinks by default + return new long[]{1, Files.size(path)} + } + catch( Exception e ) { + log.warn "Unable to determine size for input file: ${path} - ${e.message}" + return new long[]{1, 0} + } + } + + /** + * Recursively compute file count and total size of a directory. + * + * @param dir The directory path + * @return A two-element array: [fileCount, totalSize] + */ + private static long[] computeDirStats(Path dir) { + final long[] result = [0L, 0L] // [count, size] + + Files.walkFileTree(dir, new SimpleFileVisitor() { + @Override + FileVisitResult visitFile(Path file, BasicFileAttributes attrs) { + result[0]++ + result[1] += attrs.size() + return FileVisitResult.CONTINUE + } + + @Override + FileVisitResult visitFileFailed(Path file, IOException exc) { + log.warn "Unable to access file during size computation: ${file} - ${exc.message}" + return FileVisitResult.CONTINUE + } + }) + + return result + } +} diff --git a/plugins/nf-seqera/src/main/io/seqera/executor/Labels.groovy b/plugins/nf-seqera/src/main/io/seqera/executor/Labels.groovy new file mode 100644 index 0000000000..e3634c8aa3 --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/executor/Labels.groovy @@ -0,0 +1,110 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import com.google.common.hash.Hashing + +import groovy.transform.CompileStatic +import nextflow.NextflowMeta +import nextflow.script.WorkflowMetadata + +/** + * Helper class to manage run labels. + * + * Builds the labels map from workflow metadata ({@code nextflow.io/*}), + * scheduler metadata ({@code seqera:sched:*}), and user-configured labels. + * + * @author Paolo Di Tommaso + */ +@CompileStatic +class Labels { + + private final Map entries = new LinkedHashMap<>(20) + + /** + * Add {@code nextflow.io/*} labels from workflow metadata + */ + Labels withWorkflowMetadata(WorkflowMetadata workflow) { + if( workflow.projectName ) + entries.put('nextflow.io/projectName', workflow.projectName) + if( workflow.userName ) + entries.put('nextflow.io/userName', workflow.userName) + if( workflow.runName ) + entries.put('nextflow.io/runName', workflow.runName) + if( workflow.sessionId ) + entries.put('nextflow.io/sessionId', workflow.sessionId.toString()) + entries.put('nextflow.io/resume', String.valueOf(workflow.resume)) + if( workflow.revision ) + entries.put('nextflow.io/revision', workflow.revision) + if( workflow.commitId ) + entries.put('nextflow.io/commitId', workflow.commitId) + if( workflow.repository ) + entries.put('nextflow.io/repository', workflow.repository) + if( workflow.manifest?.name ) + entries.put('nextflow.io/manifestName', workflow.manifest.name) + if( NextflowMeta.instance.version ) + entries.put('nextflow.io/runtimeVersion', NextflowMeta.instance.version.toString()) + if( workflow.platform?.workflowId ) + entries.put('seqera.io/platform/workflowId', workflow.platform.workflowId) + return this + } + + /** + * Add {@code seqera:sched:*} scheduler labels + */ + Labels withSchedRunId(String runId) { + if( runId ) + entries.put('seqera:sched:runId', runId) + return this + } + + Labels withSchedClusterId(String clusterId) { + if( clusterId ) + entries.put('seqera:sched:clusterId', clusterId) + return this + } + + /** + * Add user-configured labels. These take precedence over implicit labels. + */ + Labels withUserLabels(Map labels) { + if( labels ) + entries.putAll(labels) + return this + } + + /** + * @return all labels as an unmodifiable map + */ + Map getEntries() { + return Collections.unmodifiableMap(entries) + } + + /** + * Compute a run identifier as SipHash of sessionId + runName + */ + protected static String runId(String sessionId, String runName) { + return Hashing + .sipHash24() + .newHasher() + .putUnencodedChars(sessionId) + .putUnencodedChars(runName) + .hash() + .toString() + } +} diff --git a/plugins/nf-seqera/src/main/io/seqera/executor/SeqeraBatchSubmitter.groovy b/plugins/nf-seqera/src/main/io/seqera/executor/SeqeraBatchSubmitter.groovy new file mode 100644 index 0000000000..f58550b35f --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/executor/SeqeraBatchSubmitter.groovy @@ -0,0 +1,318 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import java.util.concurrent.CompletableFuture +import java.util.concurrent.ExecutorService +import java.util.concurrent.LinkedBlockingQueue +import java.util.concurrent.TimeUnit +import java.util.concurrent.TimeoutException + +import groovy.transform.CompileStatic +import groovy.transform.TupleConstructor +import groovy.util.logging.Slf4j +import io.seqera.sched.api.schema.v1a1.InputFilesMetrics +import io.seqera.sched.api.schema.v1a1.Task +import io.seqera.sched.client.SchedClient +import nextflow.SysEnv +import nextflow.util.Duration +import nextflow.util.ThreadPoolBuilder +import nextflow.util.Threads + +/** + * Batches task submissions to the Seqera scheduler API. + * + * @author Lorenzo Fontana + */ +@Slf4j +@CompileStatic +class SeqeraBatchSubmitter { + + /** Maximum tasks per API call */ + static final int TASKS_PER_REQUEST = SysEnv.getInteger('NXF_SEQERA_TASK_PER_REQUEST', 100) + + /** Default flush interval */ + static final Duration REQUEST_INTERVAL = SysEnv.get('NXF_SEQERA_REQUEST_INTERVAL', '1 sec') as Duration + + /** Keep-alive interval - send empty submission to maintain run */ + static final Duration KEEP_ALIVE_INTERVAL = SysEnv.get('NXF_SEQERA_KEEP_ALIVE_INTERVAL', '60 sec') as Duration + + /** Timeout for waiting on metrics computation */ + static final Duration METRICS_TIMEOUT = SysEnv.get('NXF_SEQERA_METRICS_TIMEOUT', '30 sec') as Duration + + /** + * Holds a task handler, its prepared Task object, and async metrics computation + */ + @TupleConstructor + static class PendingTask { + SeqeraTaskHandler handler + Task task + CompletableFuture metricsFuture + } + + private final SchedClient client + private final String runId + private final Duration requestInterval + private final Duration keepAliveInterval + private final Closure onError + private final LinkedBlockingQueue pendingQueue = new LinkedBlockingQueue<>() + private Thread sender + private volatile boolean completed = false + + /** Executor pool for async input file metrics computation */ + private final ExecutorService metricsExecutor + + SeqeraBatchSubmitter(SchedClient client, String runId) { + this(client, runId, REQUEST_INTERVAL, KEEP_ALIVE_INTERVAL) + } + + SeqeraBatchSubmitter(SchedClient client, String runId, Duration requestInterval) { + this(client, runId, requestInterval, KEEP_ALIVE_INTERVAL) + } + + SeqeraBatchSubmitter(SchedClient client, String runId, Duration requestInterval, Duration keepAliveInterval, Closure onError=null) { + this.client = client + this.runId = runId + this.requestInterval = requestInterval + this.keepAliveInterval = keepAliveInterval + this.onError = onError + // Create a thread pool for metrics computation + this.metricsExecutor = new ThreadPoolBuilder() + .withName('seqera-metrics') + .withMinSize(0) + .withMaxSize(10) + .withKeepAliveTime(60_000L) + .withAllowCoreThreadTimeout(true) + .build() + } + + /** + * Start the sender thread that processes the batch queue + */ + void start() { + log.debug "[SEQERA] Starting batch submitter - interval=${requestInterval}" + this.sender = Threads.start('Seqera-batch-submitter', this.&sendTasks0) + } + + /** + * Enqueue a task for batch submission. + * Starts async computation of input files metrics immediately. + */ + void submit(SeqeraTaskHandler handler, Task task) { + if (completed) { + throw new IllegalStateException("Batch submitter has been shutdown") + } + + // Start async metrics computation + final taskRun = handler.task + final metricsFuture = CompletableFuture.supplyAsync( + ()-> InputFilesProfiler.compute(taskRun), + metricsExecutor + ) + + pendingQueue.add(new PendingTask(handler, task, metricsFuture)) + } + + /** + * Signal completion and wait for sender thread to finish + */ + void shutdown() { + log.debug "[SEQERA] Shutting down batch submitter" + completed = true + if (sender) { + sender.join() + } + // Shutdown metrics executor + metricsExecutor.shutdown() + try { + if (!metricsExecutor.awaitTermination(30, TimeUnit.SECONDS)) { + metricsExecutor.shutdownNow() + } + } + catch (InterruptedException e) { + metricsExecutor.shutdownNow() + Thread.currentThread().interrupt() + } + log.debug "[SEQERA] Batch submitter shutdown complete" + } + + /** + * Sender thread loop + */ + protected void sendTasks0(dummy) { + final List batch = new ArrayList<>(TASKS_PER_REQUEST) + long previous = System.currentTimeMillis() + final long period = requestInterval.millis + final long delay = period / 10 as long + + try { + while (!completed || !pendingQueue.isEmpty()) { + // Poll with timeout + final PendingTask pending = pendingQueue.poll(delay, TimeUnit.MILLISECONDS) + if (pending) { + // Start the batch timer when first task arrives + if (batch.isEmpty()) { + previous = System.currentTimeMillis() + } + batch.add(pending) + } + + // Check if we should flush + final now = System.currentTimeMillis() + final delta = now - previous + + if (!batch.isEmpty()) { + // Flush if: time elapsed OR batch full OR shutting down + if (delta > period || batch.size() >= TASKS_PER_REQUEST || completed) { + flushBatch(batch) + previous = System.currentTimeMillis() + batch.clear() + } + } + else if (delta > keepAliveInterval.millis) { + // Keep-alive: send empty submission to maintain run + try { + log.debug "[SEQERA] Sending keep-alive for run ${runId}" + client.createTasks(runId, Collections.emptyList()) + } + catch (Exception e) { + log.warn "[SEQERA] Keep-alive failed: ${e.message}" + // Don't crash the thread for keep-alive failures + } + // Always update timestamp to avoid rapid retry on failure + previous = System.currentTimeMillis() + } + } + + // Final flush of any remaining tasks + if (!batch.isEmpty()) { + flushBatch(batch) + } + } + catch (Throwable e) { + log.error "[SEQERA] Fatal error in batch submitter thread", e + // Convert Throwable to Exception for handler API + final Exception exception = e instanceof Exception ? (Exception) e : new RuntimeException(e) + // Fail any tasks in the current batch + for (PendingTask pending : batch) { + try { + pending.handler.onBatchSubmitFailure(exception) + } + catch (Exception ex) { + log.warn "[SEQERA] Error failing batch task", ex + } + } + // Drain and fail any remaining pending tasks + drainAndFailPendingTasks(exception) + // Invoke error callback to abort run + if (onError) { + try { + onError.call(e) + } + catch (Throwable t) { + log.warn "[SEQERA] Error in failure callback", t + } + } + } + } + + /** + * Drain the pending queue and fail all tasks with the given error + */ + private void drainAndFailPendingTasks(Exception cause) { + PendingTask pending + while ((pending = pendingQueue.poll()) != null) { + try { + pending.handler.onBatchSubmitFailure(cause) + } + catch (Exception e) { + log.warn "[SEQERA] Error failing pending task", e + } + } + } + + /** + * Submit a batch of tasks to the scheduler API + */ + protected void flushBatch(List batch) { + log.debug "[SEQERA] Submitting batch of ${batch.size()} tasks" + + try { + // Resolve async metrics for all tasks in batch + resolveMetrics(batch) + + // Extract Task objects for API call + final List tasks = batch.collect { it.task } + + // Submit batch to API + final response = client.createTasks(runId, tasks) + final List taskIds = response.getTaskIds() + + // Validate response + if (taskIds.size() != batch.size()) { + throw new IllegalStateException("Seqera Scheduler API returned ${taskIds.size()} task IDs but submitted ${batch.size()} tasks") + } + + // Map task IDs back to handlers + for (int i = 0; i < batch.size(); i++) { + final handler = batch[i].handler + final taskId = taskIds[i] + handler.setBatchTaskId(taskId) + } + + log.debug "[SEQERA] Batch submission complete: ${taskIds.size()} tasks submitted" + + } catch (Exception e) { + log.error "[SEQERA] Batch submission failed for ${batch.size()} tasks", e + + // Propagate failure to all handlers in this batch + for (PendingTask pending : batch) { + try { + pending.handler.onBatchSubmitFailure(e) + } catch (Exception ex) { + log.warn "[SEQERA] Error handling batch failure for task", ex + } + } + } + } + + /** + * Wait for and attach metrics to all tasks in the batch. + * Uses timeout to avoid blocking indefinitely on slow computations. + */ + private void resolveMetrics(List batch) { + final timeout = METRICS_TIMEOUT.millis + + for (PendingTask pending : batch) { + try { + final metrics = pending.metricsFuture.get(timeout, TimeUnit.MILLISECONDS) + if (metrics) { + pending.task.inputFiles(metrics) + log.debug "[SEQERA] Task `${pending.handler.task.name}` input files metrics: ${metrics}" + } + } + catch (TimeoutException e) { + log.warn "[SEQERA] Timeout computing input files metrics for task: ${pending.handler.task.name}" + pending.metricsFuture.cancel(true) + } + catch (Exception e) { + log.warn "[SEQERA] Failed to compute input files metrics for task: ${pending.handler.task.name} - ${e.message}" + } + } + } +} diff --git a/plugins/nf-seqera/src/main/io/seqera/executor/SeqeraExecutor.groovy b/plugins/nf-seqera/src/main/io/seqera/executor/SeqeraExecutor.groovy new file mode 100644 index 0000000000..1d41cf585e --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/executor/SeqeraExecutor.groovy @@ -0,0 +1,201 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.config.SeqeraConfig +import io.seqera.config.ExecutorOpts +import io.seqera.util.SchemaMapperUtil +import io.seqera.sched.api.schema.v1a1.CreateRunRequest +import io.seqera.sched.api.schema.v1a1.PredictionModel +import io.seqera.sched.client.SchedClient +import io.seqera.sched.api.schema.v1a1.TerminateRunRequest +import io.seqera.sched.client.SchedClientConfig +import nextflow.exception.AbortOperationException +import nextflow.executor.Executor +import nextflow.fusion.FusionHelper +import nextflow.platform.PlatformHelper +import nextflow.processor.TaskHandler +import nextflow.processor.TaskMonitor +import nextflow.processor.TaskPollingMonitor +import nextflow.processor.TaskRun +import nextflow.SysEnv +import nextflow.util.Duration +import nextflow.util.ServiceName +import org.pf4j.ExtensionPoint + +/** + * Nextflow executor that delegates task execution to the Seqera scheduler API. + * + *

    This executor creates a run on the Seqera scheduler, submits tasks in batches + * via {@link SeqeraBatchSubmitter}, and monitors their lifecycle through the scheduler API. + * It requires Fusion file system to be enabled and all processes to specify a container image. + * + * @author Paolo Di Tommaso + */ +@Slf4j +@ServiceName(SEQERA) +@CompileStatic +class SeqeraExecutor extends Executor implements ExtensionPoint { + + public static final String SEQERA = 'seqera' + + private ExecutorOpts seqeraConfig + + private SchedClient client + + private volatile String runId + + private SeqeraBatchSubmitter batchSubmitter + + @Override + protected void register() { + createClient() + } + + @Override + void shutdown() { + // Flush any pending batch jobs before terminating run + session.error + batchSubmitter?.shutdown() + terminateRun() + } + + protected void createClient() { + final seqera = new SeqeraConfig(session.config.seqera as Map ?: Collections.emptyMap()) + this.seqeraConfig = seqera.executor + if (!seqeraConfig) + throw new IllegalArgumentException("Missing Seqera executor configuration - make sure to specify 'seqera.executor' settings") + // Get access token and refresh token from tower config (shares authentication with Platform) + def towerConfig = session.config.tower as Map ?: Collections.emptyMap() + def accessToken = PlatformHelper.getAccessToken(towerConfig, SysEnv.get()) + def refreshToken = PlatformHelper.getRefreshToken(towerConfig, SysEnv.get()) + def platformUrl = PlatformHelper.getEndpoint(towerConfig, SysEnv.get()) + def clientConfig = SchedClientConfig.builder() + .endpoint(seqeraConfig.endpoint) + .platformUrl(platformUrl) + .accessToken(accessToken) + .refreshToken(refreshToken) + .retryConfig(seqeraConfig.retryOpts()) + .build() + this.client = new SchedClient(clientConfig) + } + + protected void createRun() { + final towerConfig = session.config.tower as Map ?: Collections.emptyMap() + final workflowId = session.workflowMetadata?.platform?.workflowId + final workflowUrl = session.workflowMetadata?.platform?.workflowUrl + final labels = new Labels() + if( seqeraConfig.autoLabels ) + labels.withWorkflowMetadata(session.workflowMetadata) + labels.withUserLabels(seqeraConfig.labels) + final predictionModel = seqeraConfig.predictionModel ? PredictionModel.fromValue(seqeraConfig.predictionModel) : null + final request = new CreateRunRequest() + .region(seqeraConfig.region) + .name(session.runName) + .machineRequirement(SchemaMapperUtil.toMachineRequirement(seqeraConfig.machineRequirement)) + .labels(labels.entries) + .workspaceId(PlatformHelper.getWorkspaceId(towerConfig, SysEnv.get()) as Long) + .workflowId(workflowId) + .workflowUrl(workflowUrl) + .predictionModel(predictionModel) + log.debug "[SEQERA] Creating run: ${request}" + final response = client.createRun(request) + this.runId = response.getRunId() + log.debug "[SEQERA] Run created id: ${runId}; workflowId: '${workflowId}'; workflowUrl: '${workflowUrl}'" + // Initialize and start batch submitter with error callback to abort on fatal errors + this.batchSubmitter = new SeqeraBatchSubmitter( + client, + runId, + seqeraConfig.batchFlushInterval, + SeqeraBatchSubmitter.KEEP_ALIVE_INTERVAL, + { Throwable t -> session.abort(t) } + ) + this.batchSubmitter.start() + } + + protected void terminateRun() { + if (!runId) { + return + } + final stopReason = truncate(session.fault?.report, 10_000) + log.debug "[SEQERA] Terminating run: ${runId}; stopReason: ${stopReason}" + client.terminateRun(runId, new TerminateRunRequest().stopReason(stopReason)) + log.debug "[SEQERA] Run terminated" + } + + @Override + protected TaskMonitor createTaskMonitor() { + TaskPollingMonitor.create(session, config, name, 1000, Duration.of('10 sec')) + } + + @Override + TaskHandler createTaskHandler(TaskRun task) { + return new SeqeraTaskHandler(task, this) + } + + /** + * @return {@code true} whenever the containerization is managed by the executor itself + */ + boolean isContainerNative() { + return true + } + + @Override + boolean isFusionEnabled() { + final enabled = FusionHelper.isFusionEnabled(session) + if (!enabled) + throw new AbortOperationException("Seqera executor requires the use of Fusion file system") + return true + } + + /** + * Lazily creates the run on first access, ensuring workflowId and labels + * are available (they are set by TowerClient.onFlowCreate before tasks are submitted). + */ + void ensureRunCreated() { + if (runId) return + synchronized (this) { + if (runId) return + createRun() + } + } + + SchedClient getClient() { + return client + } + + String getRunId() { + return runId + } + + SeqeraBatchSubmitter getBatchSubmitter() { + return batchSubmitter + } + + ExecutorOpts getSeqeraConfig() { + return seqeraConfig + } + + protected static String truncate(String value, int maxLen) { + if (!value || value.length() <= maxLen) + return value + return value.take(maxLen) + '\n.. [TRUNCATED]' + } +} diff --git a/plugins/nf-seqera/src/main/io/seqera/executor/SeqeraTaskHandler.groovy b/plugins/nf-seqera/src/main/io/seqera/executor/SeqeraTaskHandler.groovy new file mode 100644 index 0000000000..722c3e5a8e --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/executor/SeqeraTaskHandler.groovy @@ -0,0 +1,383 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import java.nio.file.Path + +import groovy.transform.CompileStatic +import groovy.transform.PackageScope +import groovy.util.logging.Slf4j +import io.seqera.sched.api.schema.v1a1.AcceleratorType +import io.seqera.sched.api.schema.v1a1.GetTaskLogsResponse +import io.seqera.sched.api.schema.v1a1.NextflowTask +import io.seqera.sched.api.schema.v1a1.ResourceLimit +import io.seqera.sched.api.schema.v1a1.ResourceRequirement +import io.seqera.sched.api.schema.v1a1.Task +import io.seqera.sched.api.schema.v1a1.TaskState as SchedTaskState +import io.seqera.sched.api.schema.v1a1.TaskStatus as SchedTaskStatus +import io.seqera.sched.client.SchedClient +import io.seqera.util.SchemaMapperUtil +import nextflow.cloud.types.CloudMachineInfo +import nextflow.exception.ProcessException +import nextflow.exception.ProcessUnrecoverableException +import nextflow.util.Duration +import nextflow.util.MemoryUnit +import nextflow.fusion.FusionAwareTask +import nextflow.processor.TaskHandler +import nextflow.processor.TaskRun +import nextflow.processor.TaskStatus +import nextflow.trace.TraceRecord +/** + * Task handler for the Seqera scheduler executor. + * + *

    Manages the lifecycle of a single task submitted to the Seqera scheduler, + * including submission via batch submitter, status polling, completion handling, + * and trace record enrichment with machine info and spot interruption metadata. + * + * @author Paolo Di Tommaso + */ +@Slf4j +@CompileStatic +class SeqeraTaskHandler extends TaskHandler implements FusionAwareTask { + + private SchedClient client + + private SeqeraExecutor executor + + private Path exitFile + + private Path outputFile + + private Path errorFile + + private volatile String taskId + + /** + * Cached task state from last describeTask call, used for trace record metadata + */ + private volatile SchedTaskState cachedTaskState + + /** + * Cached machine info extracted from task attempts + */ + private volatile CloudMachineInfo machineInfo + + SeqeraTaskHandler(TaskRun task, SeqeraExecutor executor) { + super(task) + this.client = executor.getClient() + this.executor = executor + // those files are access via NF runtime, keep based on CloudStoragePath + this.outputFile = task.workDir.resolve(TaskRun.CMD_OUTFILE) + this.errorFile = task.workDir.resolve(TaskRun.CMD_ERRFILE) + this.exitFile = task.workDir.resolve(TaskRun.CMD_EXIT) + } + + @Override + void prepareLauncher() { + assert fusionEnabled() + final launcher = fusionLauncher() + launcher.build() + } + + @Override + void submit() { + executor.ensureRunCreated() + int cpuShares = (task.config.getCpus() ?: 1) * 1024 + int memoryMiB = task.config.getMemory() ? (int) (task.config.getMemory().toBytes() / (1024 * 1024)) : 1024 + final resourceReq = new ResourceRequirement() + .cpuShares(cpuShares) + .memoryMiB(memoryMiB) + // add accelerator settings if defined + final accelerator = task.config.getAccelerator() + if( accelerator ) { + // number of accelerators requested, fallback to limit if request is not specified + resourceReq.acceleratorCount(accelerator.request ?: accelerator.limit) + // accelerator type is GPU by default (most common in scientific computing) + resourceReq.acceleratorType(AcceleratorType.GPU) + // specific accelerator model name e.g. "nvidia-tesla-v100", "nvidia-a10g" + if( accelerator.type ) + resourceReq.acceleratorName(accelerator.type) + } + // build machine requirement merging config settings with task arch, disk, and snapshot settings + final machineReq = SchemaMapperUtil.toMachineRequirement( + executor.getSeqeraConfig().machineRequirement, + task.getContainerPlatform(), + task.config.getDisk(), + fusionConfig().snapshotsEnabled() + ) + // build resource limit from process resourceLimits directive (upper bound for OOM retry scaling) + final resourceLim = toResourceLimit() + // validate container - Seqera executor requires all processes to specify a container image + final container = task.getContainer() + if( !container ) + throw new ProcessUnrecoverableException("Process `${task.lazyName()}` failed because the container image was not specified -- the Seqera executor requires all processes define a container image") + // build the scheduler task with all required attributes + final schedTask = new Task() + .name(task.lazyName()) // process name for identification + .image(container) // container image to run + .command(fusionSubmitCli()) // fusion-based command launcher + .environment(getTaskEnvironment()) // fusion + user-configured environment variables + .resourceRequirement(resourceReq) // cpu, memory, accelerators + .resourceLimit(resourceLim) // resource upper bounds for OOM retry + .machineRequirement(machineReq) // machine type and disk requirements + .nextflow(new NextflowTask() + .taskId(task.id?.intValue()) + .hash(task.hash?.toString()) + .workDir(task.getWorkDirStr())) + log.debug "[SEQERA] Enqueueing task for batch submission: ${schedTask}" + // Enqueue for batch submission - status will be set by setBatchTaskId callback + executor.getBatchSubmitter().submit(this, schedTask) + } + + /** + * Build the task environment by merging user-configured environment variables + * with Fusion environment variables. Fusion variables take precedence. + */ + protected Map getTaskEnvironment() { + final configEnv = executor.getSeqeraConfig()?.taskEnvironment + final fusionEnv = fusionLauncher().fusionEnv() + if( !configEnv ) + return fusionEnv + final result = new LinkedHashMap(configEnv) + result.putAll(fusionEnv) + return result + } + + /** + * Called by batch submitter after successful batch submission + */ + void setBatchTaskId(String taskId) { + this.taskId = taskId + this.status = TaskStatus.SUBMITTED + log.debug "[SEQERA] Process `${task.lazyName()}` submitted > taskId=$taskId; work-dir=${task.getWorkDirStr()}" + } + + /** + * Called by batch submitter when batch submission fails + */ + void onBatchSubmitFailure(Exception cause) { + log.debug "[SEQERA] Batch submission failed for task ${task.lazyName()}: ${cause.message}" + task.error = cause + this.status = TaskStatus.COMPLETED + } + + /** + * Build a {@link ResourceLimit} from the process {@code resourceLimits} directive. + * Returns {@code null} if no resource limits are defined. + */ + protected ResourceLimit toResourceLimit() { + final memoryLimit = task.config.getResourceLimit('memory') as MemoryUnit + final cpusLimit = task.config.getResourceLimit('cpus') as Integer + if( !memoryLimit && !cpusLimit ) + return null + final result = new ResourceLimit() + if( memoryLimit ) + result.memoryMiB((int)(memoryLimit.toBytes() / (1024 * 1024))) + if( cpusLimit ) + result.cpuShares(cpusLimit * 1024) + return result + } + + protected SchedTaskStatus schedTaskStatus() { + cachedTaskState = client.describeTask(taskId).getTaskState() + return cachedTaskState.getStatus() + } + + @Override + boolean checkIfRunning() { + if (isSubmitted()) { + final schedStatus = schedTaskStatus() + log.debug "[SEQERA] checkIfRunning taskId=${taskId}; status=${schedStatus}" + if (isRunningOrTerminated(schedStatus)) { + status = TaskStatus.RUNNING + return true + } + } + return false + } + + @Override + boolean checkIfCompleted() { + // Handle batch submission failure - task error was set but never reached RUNNING state + if (task.error && isCompleted()) { + return true + } + if (!isRunning()) + return false + final schedStatus = schedTaskStatus() + log.debug "[SEQERA] checkIfCompleted status=${schedStatus}" + if (isTerminated(schedStatus)) { + log.debug "[SEQERA] Process `${task.lazyName()}` - terminated taskId=$taskId; status=$schedStatus" + // finalize the task + task.exitStatus = readExitFile() + if (isFailed(schedStatus)) { + // When no exit code available, get the error message from task state + if (task.exitStatus == Integer.MAX_VALUE) { + final errorMessage = cachedTaskState?.getErrorMessage() ?: "Task failed for unknown reason" + task.error = new ProcessException(errorMessage) + } + final logs = getTaskLogs(taskId) + task.stdout = logs?.stdout ?: outputFile + task.stderr = logs?.stderr ?: errorFile + } else { + task.stdout = outputFile + task.stderr = errorFile + } + status = TaskStatus.COMPLETED + return true + } + + return false + } + + protected boolean isRunningOrTerminated(SchedTaskStatus status) { + return status == SchedTaskStatus.RUNNING || isTerminated(status) + } + + protected boolean isTerminated(SchedTaskStatus status) { + return status in [SchedTaskStatus.SUCCEEDED, SchedTaskStatus.FAILED, SchedTaskStatus.CANCELLED] + } + + protected boolean isFailed(SchedTaskStatus status) { + return status == SchedTaskStatus.FAILED + } + + protected GetTaskLogsResponse getTaskLogs(String taskId) { + return client.getTaskLogs(taskId) + } + + @Override + protected void killTask() { + if( !taskId ) { + log.trace "[SEQERA] Skip cancel - taskId not yet assigned" + return + } + log.debug "[SEQERA] Cancel taskId=${taskId}" + try { + client.cancelTask(taskId) + } + catch (Throwable t) { + log.warn "[SEQERA] Failed to cancel task ${taskId}", t + } + } + + @PackageScope + Integer readExitFile() { + try { + final result = exitFile.text as Integer + log.trace "[SEQERA] Read exit file for taskId $taskId; exit=${result}" + return result + } + catch (Exception e) { + log.debug "[SEQERA] Cannot read exit status for task: `${task.lazyName()}` - ${e.message}" + // return MAX_VALUE to signal it was unable to retrieve the exit code + return Integer.MAX_VALUE + } + } + + /** + * Get machine info for the task execution from the last task attempt. + * The machine info is cached after first retrieval. + * + * @return CloudMachineInfo containing instance type, zone, and price model, or null if not available + */ + protected CloudMachineInfo getMachineInfo() { + if (machineInfo) + return machineInfo + if (!cachedTaskState) + return null + + try { + final attempts = cachedTaskState.getAttempts() + if (!attempts || attempts.isEmpty()) + return null + + final lastAttempt = attempts.get(attempts.size() - 1) + final lastInfo = lastAttempt.getMachineInfo() + if (!lastInfo) + return null + + // Convert Sched API MachineInfo to Nextflow CloudMachineInfo + machineInfo = new CloudMachineInfo( + type: lastInfo.getType(), + zone: lastInfo.getZone(), + priceModel: SchemaMapperUtil.toPriceModel(lastInfo.getPriceModel()) + ) + log.trace "[SEQERA] taskId=$taskId => machineInfo=$machineInfo" + return machineInfo + } + catch (Exception e) { + log.debug "[SEQERA] Unable to get machine info for taskId=$taskId - ${e.message}" + return null + } + } + + /** + * Get the number of spot interruptions for this task. + * This is calculated server-side from task attempts with spot-related stop reasons. + * + * @return the count of spot interruptions, or null if not completed or not available + */ + protected Integer getNumSpotInterruptions() { + if (!taskId || !isCompleted()) + return null + if (!cachedTaskState) + return null + return cachedTaskState.getNumSpotInterruptions() + } + + /** + * Get the log stream identifier for this task. + * + * @return the log stream ID, or null if not available + */ + protected String getLogStreamId() { + return cachedTaskState?.getLogStreamId() + } + + /** + * Get the native backend ID for this task (ECS task ARN or Docker container ID). + * + * @return the native ID from the last task attempt, or null if not available + */ + protected String getNativeId() { + return cachedTaskState?.getId() + } + + protected Long getGrantedTime() { + final time = cachedTaskState?.getResourceRequirement()?.getTime() + return time != null ? Duration.of(time).toMillis() : task.config.getTime()?.toMillis() + } + + /** + * Get the trace record for this task, including machine info and spot interruptions metadata. + * + * @return the trace record with additional metadata fields + */ + @Override + TraceRecord getTraceRecord() { + final result = super.getTraceRecord() + result.put('native_id', getNativeId()) + result.machineInfo = getMachineInfo() + result.numSpotInterruptions = getNumSpotInterruptions() + result.logStreamId = getLogStreamId() + // Override executor name to include cloud backend for cost tracking + result.executorName = "${SeqeraExecutor.SEQERA}/aws" + return result + } +} diff --git a/plugins/nf-seqera/src/main/io/seqera/plugin/SeqeraPlugin.groovy b/plugins/nf-seqera/src/main/io/seqera/plugin/SeqeraPlugin.groovy new file mode 100644 index 0000000000..335bcc1f4a --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/plugin/SeqeraPlugin.groovy @@ -0,0 +1,35 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.plugin + +import groovy.transform.CompileStatic +import nextflow.plugin.BasePlugin +import org.pf4j.PluginWrapper + +/** + * Seqera plugin entry point + * + * @author Paolo Di Tommaso + */ +@CompileStatic +class SeqeraPlugin extends BasePlugin { + + SeqeraPlugin(PluginWrapper wrapper) { + super(wrapper) + } +} diff --git a/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy b/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy new file mode 100644 index 0000000000..a42d325fc0 --- /dev/null +++ b/plugins/nf-seqera/src/main/io/seqera/util/SchemaMapperUtil.groovy @@ -0,0 +1,253 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.util + +import groovy.transform.CompileStatic +import io.seqera.config.MachineRequirementOpts +import io.seqera.sched.api.schema.v1a1.DiskAllocation +import io.seqera.sched.api.schema.v1a1.DiskRequirement +import io.seqera.sched.api.schema.v1a1.EcsCapacityMode +import io.seqera.sched.api.schema.v1a1.MachineRequirement +import io.seqera.sched.api.schema.v1a1.PriceModel as SchedPriceModel +import io.seqera.sched.api.schema.v1a1.ProvisioningModel +import nextflow.cloud.types.PriceModel +import nextflow.fusion.FusionConfig +import nextflow.util.MemoryUnit + +/** + * Utility class to map Nextflow config objects to Sched API schema objects. + * + * @author Paolo Di Tommaso + */ +@CompileStatic +class SchemaMapperUtil { + + /** Default EBS volume type - gp3 provides good balance of price and performance */ + static final String DEFAULT_DISK_TYPE = 'ebs/gp3' + + /** Default throughput in MiB/s - Fusion recommended setting for optimal I/O */ + static final int DEFAULT_DISK_THROUGHPUT_MIBPS = 325 + + /** Supported EBS volume types */ + static final Set SUPPORTED_DISK_TYPES = Set.of( + 'ebs/gp3', // General purpose SSD (default) + 'ebs/gp2', // General purpose SSD (legacy) + 'ebs/io1', // Provisioned IOPS SSD + 'ebs/io2', // Provisioned IOPS SSD (higher durability) + 'ebs/st1', // Throughput optimized HDD + 'ebs/sc1' // Cold HDD + ) + + /** + * Maps MachineRequirementOpts to MachineRequirement API object. + * + * @param opts the config options (can be null) + * @return the MachineRequirement API object, or null if opts is null or has no settings + */ + static MachineRequirement toMachineRequirement(MachineRequirementOpts opts) { + if (!opts) + return null + final diskReq = toDiskRequirement(opts.diskSize, opts) + final capacityMode = toEcsCapacityMode(opts.capacityMode) + if (!opts.arch && !opts.provisioning && !opts.maxSpotAttempts && !opts.machineTypes && !diskReq && !capacityMode) + return null + new MachineRequirement() + .arch(opts.arch) + .provisioning(toProvisioningModel(opts.provisioning)) + .maxSpotAttempts(opts.maxSpotAttempts) + .machineTypes(opts.machineTypes) + .disk(diskReq) + .capacityMode(capacityMode) + } + + /** + * Maps MachineRequirementOpts to MachineRequirement API object, merging with task arch. + * Task arch overrides config arch if specified. + * + * @param opts the config options (can be null) + * @param taskArch the task container platform/arch (can be null) + * @return the MachineRequirement API object, or null if no settings + */ + static MachineRequirement toMachineRequirement(MachineRequirementOpts opts, String taskArch) { + return toMachineRequirement(opts, taskArch, null, false) + } + + /** + * Maps MachineRequirementOpts to MachineRequirement API object, merging with task arch, disk, and snapshots. + * Task arch overrides config arch if specified. + * + * @param opts the config options (can be null) + * @param taskArch the task container platform/arch (can be null) + * @param diskSize the disk size from task config (can be null) + * @param snapshotEnabled whether Fusion snapshots are enabled + * @return the MachineRequirement API object, or null if no settings + */ + static MachineRequirement toMachineRequirement(MachineRequirementOpts opts, String taskArch, MemoryUnit diskSize, boolean snapshotEnabled) { + final arch = taskArch ?: opts?.arch + final provisioning = opts?.provisioning + final maxSpotAttempts = opts?.maxSpotAttempts + ?: (snapshotEnabled ? FusionConfig.DEFAULT_SNAPSHOT_MAX_SPOT_ATTEMPTS : null) + final machineTypes = opts?.machineTypes + // task disk overrides config disk + final effectiveDiskSize = diskSize ?: opts?.diskSize + final diskReq = toDiskRequirement(effectiveDiskSize, opts) + final capacityMode = toEcsCapacityMode(opts?.capacityMode) + // return null if no settings + if (!arch && !provisioning && !maxSpotAttempts && !machineTypes && !diskReq && !snapshotEnabled && !capacityMode) + return null + new MachineRequirement() + .arch(arch) + .provisioning(toProvisioningModel(provisioning)) + .maxSpotAttempts(maxSpotAttempts) + .machineTypes(machineTypes) + .disk(diskReq) + .snapshotEnabled(snapshotEnabled ? Boolean.TRUE : null) + .capacityMode(capacityMode) + } + + /** + * Maps a disk size to DiskRequirement API object. + * Uses config options if provided, otherwise defaults to Fusion recommended settings: + * EBS gp3 volume with 325 MiB/s throughput. + * + * For 'node' allocation (default), only sizeGiB and mountPath are applicable. + * For 'task' allocation, all EBS options can be specified. + * + * @param diskSize the disk size (can be null) + * @param opts the machine requirement options with disk settings (can be null) + * @return the DiskRequirement API object, or null if diskSize is null or zero + */ + static DiskRequirement toDiskRequirement(MemoryUnit diskSize, MachineRequirementOpts opts=null) { + if (!diskSize || diskSize.toGiga() <= 0) + return null + + final allocation = toDiskAllocation(opts?.diskAllocation) ?: DiskAllocation.NODE + + // For 'node' allocation (default), only size and mountPath are valid + if (allocation == DiskAllocation.NODE) { + validateNodeAllocationOpts(opts) + final DiskRequirement req = new DiskRequirement() + req.sizeGiB(diskSize.toGiga() as Integer) + req.allocation(allocation) + return req + } + + // For 'task' allocation, apply EBS-specific options + final type = opts?.diskType ?: DEFAULT_DISK_TYPE + // Validate disk type is supported + if (!SUPPORTED_DISK_TYPES.contains(type)) { + throw new IllegalArgumentException("Invalid disk type: ${type}. Supported types: ${SUPPORTED_DISK_TYPES.join(', ')}") + } + final throughput = opts?.diskThroughputMiBps ?: DEFAULT_DISK_THROUGHPUT_MIBPS + final iops = opts?.diskIops + final encrypted = opts?.diskEncrypted ?: false + + final DiskRequirement req = new DiskRequirement() + req.sizeGiB(diskSize.toGiga() as Integer) + req.volumeType(type) + req.encrypted(encrypted) + req.allocation(allocation) + // Only set throughput for gp3 volumes + if (type == DEFAULT_DISK_TYPE) { + req.throughputMiBps(throughput) + } + // Set IOPS if provided + if (iops) { + req.iops(iops) + } + return req + } + + /** + * Validates that no EBS-specific options are set when using 'node' allocation. + * Node allocation uses instance storage, not EBS volumes. + * + * @param opts the machine requirement options + * @throws IllegalArgumentException if EBS-specific options are set with node allocation + */ + private static void validateNodeAllocationOpts(MachineRequirementOpts opts) { + if (!opts) + return + final List invalidOpts = [] + if (opts.diskType) + invalidOpts.add('diskType') + if (opts.diskThroughputMiBps) + invalidOpts.add('diskThroughputMiBps') + if (opts.diskIops) + invalidOpts.add('diskIops') + if (opts.diskEncrypted) + invalidOpts.add('diskEncrypted') + + if (invalidOpts) { + throw new IllegalArgumentException( + "The following options are not valid with 'node' disk allocation: ${invalidOpts.join(', ')}. " + + "Node allocation uses instance storage; only disk size is applicable." + ) + } + } + + /** + * Maps a disk allocation string to DiskAllocation enum. + * + * @param value the disk allocation string (task, node) + * @return the DiskAllocation enum value, or null if value is null + */ + static DiskAllocation toDiskAllocation(String value) { + value ? DiskAllocation.fromValue(value) : null + } + + /** + * Maps a capacity mode string to EcsCapacityMode enum. + * + * @param value the capacity mode string (managed, asg) + * @return the EcsCapacityMode enum value, or null if value is null + */ + static EcsCapacityMode toEcsCapacityMode(String value) { + value ? EcsCapacityMode.fromValue(value) : null + } + + /** + * Maps a provisioning string to ProvisioningModel enum. + * + * @param value the provisioning string (spot, ondemand, spotFirst) + * @return the ProvisioningModel enum value, or null if value is null + */ + static ProvisioningModel toProvisioningModel(String value) { + value ? ProvisioningModel.fromValue(value) : null + } + + /** + * Maps Sched API PriceModel to Nextflow PriceModel. + * + * @param schedPriceModel the Sched API price model + * @return the Nextflow PriceModel, or null if input is null or unknown + */ + static PriceModel toPriceModel(SchedPriceModel schedPriceModel) { + if (schedPriceModel == null) + return null + switch (schedPriceModel) { + case SchedPriceModel.SPOT: + return PriceModel.spot + case SchedPriceModel.STANDARD: + return PriceModel.standard + default: + return null + } + } + +} diff --git a/plugins/nf-seqera/src/test/io/seqera/config/ExecutorOptsTest.groovy b/plugins/nf-seqera/src/test/io/seqera/config/ExecutorOptsTest.groovy new file mode 100644 index 0000000000..1770afadc0 --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/config/ExecutorOptsTest.groovy @@ -0,0 +1,244 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.config + +import nextflow.util.Duration +import spock.lang.Specification + +/** + * Unit tests for SeqeraExecutorConfig + * + * @author Paolo Di Tommaso + */ +class ExecutorOptsTest extends Specification { + + def 'should throw error when endpoint is missing' () { + when: + new ExecutorOpts([:]) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains('Missing Seqera endpoint') + } + + def 'should create config with minimal settings' () { + when: + def config = new ExecutorOpts([endpoint: 'https://sched.example.com']) + + then: + config.endpoint == 'https://sched.example.com' + config.region == 'eu-central-1' // default + config.keyPairName == null + config.batchFlushInterval == Duration.of('1 sec') + config.machineRequirement != null + config.machineRequirement.arch == null + config.machineRequirement.provisioning == null + !config.autoLabels + } + + def 'should create config with custom region' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + region: 'us-west-2' + ]) + + then: + config.endpoint == 'https://sched.example.com' + config.region == 'us-west-2' + } + + def 'should create config with custom batch flush interval' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + batchFlushInterval: '5 sec' + ]) + + then: + config.batchFlushInterval == Duration.of('5 sec') + } + + def 'should create config with machine requirement settings' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + machineRequirement: [ + arch: 'arm64', + provisioning: 'spotFirst', + maxSpotAttempts: 3, + machineTypes: ['m6g', 'c6g'] + ] + ]) + + then: + config.machineRequirement != null + config.machineRequirement.arch == 'arm64' + config.machineRequirement.provisioning == 'spotFirst' + config.machineRequirement.maxSpotAttempts == 3 + config.machineRequirement.machineTypes == ['m6g', 'c6g'] + } + + def 'should create config with retry policy' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + retryPolicy: [maxAttempts: 5, delay: '2s'] + ]) + + then: + config.retryOpts().maxAttempts == 5 + config.retryOpts().delay == Duration.of('2s') + } + + def 'should create config with all settings' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + region: 'eu-west-1', + keyPairName: 'my-key', + batchFlushInterval: '2 sec', + machineRequirement: [ + arch: 'x86_64', + provisioning: 'spot' + ] + ]) + + then: + config.endpoint == 'https://sched.example.com' + config.region == 'eu-west-1' + config.keyPairName == 'my-key' + config.batchFlushInterval == Duration.of('2 sec') + config.machineRequirement.arch == 'x86_64' + config.machineRequirement.provisioning == 'spot' + } + + def 'should create config with labels' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + labels: [ + project: 'genomics', + team: 'research', + costCenter: 'CC-1234' + ] + ]) + + then: + config.labels == [project: 'genomics', team: 'research', costCenter: 'CC-1234'] + } + + def 'should handle null labels' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com' + ]) + + then: + config.labels == null + } + + def 'should handle empty labels' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + labels: [:] + ]) + + then: + config.labels == [:] + } + + def 'should enable auto labels' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + autoLabels: true + ]) + + then: + config.autoLabels + } + + def 'should create config with prediction model' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + predictionModel: 'qr/v1' + ]) + + then: + config.predictionModel == 'qr/v1' + } + + def 'should default prediction model to null' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com' + ]) + + then: + config.predictionModel == null + } + + def 'should create config with taskEnvironment' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + taskEnvironment: [FOO: 'bar', BAZ: 'qux'] + ]) + + then: + config.taskEnvironment == [FOO: 'bar', BAZ: 'qux'] + } + + def 'should handle null taskEnvironment' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com' + ]) + + then: + config.taskEnvironment == null + } + + def 'should handle empty taskEnvironment' () { + when: + def config = new ExecutorOpts([ + endpoint: 'https://sched.example.com', + taskEnvironment: [:] + ]) + + then: + config.taskEnvironment == [:] + } + + def 'should reject invalid prediction model' () { + when: + new ExecutorOpts([ + endpoint: 'https://sched.example.com', + predictionModel: 'invalid' + ]) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains("Invalid prediction model 'invalid'") + e.message.contains('qr/v1') + } + +} diff --git a/plugins/nf-seqera/src/test/io/seqera/config/MachineRequirementOptsTest.groovy b/plugins/nf-seqera/src/test/io/seqera/config/MachineRequirementOptsTest.groovy new file mode 100644 index 0000000000..61a7a02d65 --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/config/MachineRequirementOptsTest.groovy @@ -0,0 +1,67 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.config + +import spock.lang.Specification + +/** + * Unit tests for MachineRequirementOpts + * + * @author Paolo Di Tommaso + */ +class MachineRequirementOptsTest extends Specification { + + def 'should create with empty config' () { + when: + def opts = new MachineRequirementOpts([:]) + + then: + opts.arch == null + opts.provisioning == null + opts.maxSpotAttempts == null + opts.machineTypes == null + } + + def 'should create with all settings' () { + when: + def opts = new MachineRequirementOpts([ + arch: 'arm64', + provisioning: 'spotFirst', + maxSpotAttempts: 3, + machineTypes: ['m5', 'c5', 'r5'] + ]) + + then: + opts.arch == 'arm64' + opts.provisioning == 'spotFirst' + opts.maxSpotAttempts == 3 + opts.machineTypes == ['m5', 'c5', 'r5'] + } + + def 'should create with partial settings' () { + when: + def opts = new MachineRequirementOpts([arch: 'x86_64', provisioning: 'spot']) + + then: + opts.arch == 'x86_64' + opts.provisioning == 'spot' + opts.maxSpotAttempts == null + opts.machineTypes == null + } + +} \ No newline at end of file diff --git a/plugins/nf-seqera/src/test/io/seqera/config/RetryOptsTest.groovy b/plugins/nf-seqera/src/test/io/seqera/config/RetryOptsTest.groovy new file mode 100644 index 0000000000..b6b7b1e2b0 --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/config/RetryOptsTest.groovy @@ -0,0 +1,59 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.config + +import nextflow.util.Duration +import spock.lang.Specification + +/** + * + * @author Paolo Di Tommaso + */ +class RetryOptsTest extends Specification { + + def 'should create retry config' () { + + expect: + new RetryOpts().delay == Duration.of('450ms') + new RetryOpts().maxDelay == Duration.of('90s') + new RetryOpts().maxAttempts == 10 + new RetryOpts().jitter == 0.25d + new RetryOpts().multiplier == 2.0d + + and: + new RetryOpts([maxAttempts: 20]).maxAttempts == 20 + new RetryOpts([delay: '1s']).delay == Duration.of('1s') + new RetryOpts([maxDelay: '1m']).maxDelay == Duration.of('1m') + new RetryOpts([jitter: '0.5']).jitter == 0.5d + new RetryOpts([multiplier: '3.0']).multiplier == 3.0d + + } + + def 'should implement Retryable.Config interface' () { + when: + def opts = new RetryOpts([delay: '1s', maxDelay: '2m', maxAttempts: 5, jitter: '0.3', multiplier: '1.5']) + + then: + opts.getDelayAsDuration() == java.time.Duration.ofSeconds(1) + opts.getMaxDelayAsDuration() == java.time.Duration.ofMinutes(2) + opts.getMaxAttempts() == 5 + opts.getJitter() == 0.3d + opts.getMultiplier() == 1.5d + } + +} diff --git a/plugins/nf-seqera/src/test/io/seqera/config/SeqeraConfigTest.groovy b/plugins/nf-seqera/src/test/io/seqera/config/SeqeraConfigTest.groovy new file mode 100644 index 0000000000..fe4fe5cac3 --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/config/SeqeraConfigTest.groovy @@ -0,0 +1,96 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.config + +import nextflow.util.Duration +import spock.lang.Specification + +/** + * Unit tests for SeqeraConfig + * + * @author Paolo Di Tommaso + */ +class SeqeraConfigTest extends Specification { + + def 'should create config with no executor' () { + when: + def config = new SeqeraConfig([:]) + + then: + config.executor == null + } + + def 'should create config with executor settings' () { + when: + def config = new SeqeraConfig([ + executor: [ + endpoint: 'https://sched.example.com', + region: 'us-west-2' + ] + ]) + + then: + config.executor != null + config.executor.endpoint == 'https://sched.example.com' + config.executor.region == 'us-west-2' + } + + def 'should create config with full executor settings' () { + when: + def config = new SeqeraConfig([ + executor: [ + endpoint: 'https://sched.example.com', + region: 'eu-west-1', + keyPairName: 'my-key', + batchFlushInterval: '2 sec', + machineRequirement: [ + arch: 'arm64', + provisioning: 'spot' + ], + labels: [ + project: 'genomics', + team: 'research' + ] + ] + ]) + + then: + config.executor != null + config.executor.endpoint == 'https://sched.example.com' + config.executor.region == 'eu-west-1' + config.executor.keyPairName == 'my-key' + config.executor.batchFlushInterval == Duration.of('2 sec') + config.executor.machineRequirement.arch == 'arm64' + config.executor.machineRequirement.provisioning == 'spot' + config.executor.labels == [project: 'genomics', team: 'research'] + } + + def 'should throw error when executor endpoint is missing' () { + when: + new SeqeraConfig([ + executor: [ + region: 'us-west-2' + ] + ]) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains('Missing Seqera endpoint') + } + +} diff --git a/plugins/nf-seqera/src/test/io/seqera/executor/InputFilesProfilerTest.groovy b/plugins/nf-seqera/src/test/io/seqera/executor/InputFilesProfilerTest.groovy new file mode 100644 index 0000000000..30d4f268e3 --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/executor/InputFilesProfilerTest.groovy @@ -0,0 +1,202 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import java.nio.file.Files +import java.nio.file.Path + +import nextflow.file.FileHolder +import nextflow.processor.TaskRun +import spock.lang.Specification +import spock.lang.TempDir + +/** + * Tests for InputFilesProfiler + * + * @author Paolo Di Tommaso + */ +class InputFilesProfilerTest extends Specification { + + @TempDir + Path tempDir + + def 'should return null for null task'() { + expect: + InputFilesProfiler.compute((TaskRun) null) == null + } + + def 'should return null for task with no input files'() { + given: + def task = Mock(TaskRun) { + getInputFiles() >> [] + } + + expect: + InputFilesProfiler.compute(task) == null + } + + def 'should return null for empty file list'() { + expect: + InputFilesProfiler.compute([]) == null + } + + def 'should return null for null file list'() { + expect: + InputFilesProfiler.compute((List) null) == null + } + + def 'should compute metrics for single small file'() { + given: + def file = tempDir.resolve('small.txt') + Files.write(file, 'hello'.bytes) + def files = [new FileHolder(file)] + + when: + def metrics = InputFilesProfiler.compute(files) + + then: + metrics.count == 1 + metrics.totalBytes == 5 + metrics.maxFileBytes == 5 + metrics.minFileBytes == 5 + } + + def 'should compute metrics for multiple files'() { + given: + def smallFile = tempDir.resolve('small.txt') + Files.write(smallFile, new byte[500]) + + def mediumFile = tempDir.resolve('medium.dat') + Files.write(mediumFile, new byte[2000]) + + def largeFile = tempDir.resolve('large.dat') + Files.write(largeFile, new byte[50000]) + + def files = [ + new FileHolder(smallFile), + new FileHolder(mediumFile), + new FileHolder(largeFile) + ] + + when: + def metrics = InputFilesProfiler.compute(files) + + then: + metrics.count == 3 + metrics.totalBytes == 500 + 2000 + 50000 + metrics.maxFileBytes == 50000 + metrics.minFileBytes == 500 + } + + def 'should count files in directory recursively'() { + given: + def dir = tempDir.resolve('mydir') + Files.createDirectory(dir) + Files.write(dir.resolve('file1.txt'), new byte[100]) + Files.write(dir.resolve('file2.txt'), new byte[200]) + + def subDir = dir.resolve('subdir') + Files.createDirectory(subDir) + Files.write(subDir.resolve('file3.txt'), new byte[300]) + + def files = [new FileHolder(dir)] + + when: + def metrics = InputFilesProfiler.compute(files) + + then: + metrics.count == 3 // 3 actual files inside the directory + metrics.totalBytes == 600 // 100 + 200 + 300 + metrics.maxFileBytes == 600 + metrics.minFileBytes == 600 + } + + def 'should count files and directory contents together'() { + given: + def file1 = tempDir.resolve('input.fq') + Files.write(file1, new byte[5000]) + + def dir = tempDir.resolve('index') + Files.createDirectory(dir) + Files.write(dir.resolve('a.bin'), new byte[100]) + Files.write(dir.resolve('b.bin'), new byte[200]) + + def files = [new FileHolder(file1), new FileHolder(dir)] + + when: + def metrics = InputFilesProfiler.compute(files) + + then: + metrics.count == 3 // 1 regular file + 2 files in directory + metrics.totalBytes == 5300 + metrics.maxFileBytes == 5000 + metrics.minFileBytes == 300 + } + + def 'should follow symlinks'() { + given: + def realFile = tempDir.resolve('real.txt') + Files.write(realFile, new byte[1000]) + + def symlink = tempDir.resolve('link.txt') + Files.createSymbolicLink(symlink, realFile) + + def files = [new FileHolder(symlink)] + + when: + def metrics = InputFilesProfiler.compute(files) + + then: + metrics.count == 1 + metrics.totalBytes == 1000 + metrics.maxFileBytes == 1000 + metrics.minFileBytes == 1000 + } + + def 'should handle non-existent file gracefully'() { + given: + def missingFile = tempDir.resolve('does-not-exist.txt') + def files = [new FileHolder(missingFile)] + + when: + def metrics = InputFilesProfiler.compute(files) + + then: + metrics.count == 1 + metrics.totalBytes == 0 + } + + def 'should compute from TaskRun'() { + given: + def file = tempDir.resolve('task-input.txt') + Files.write(file, new byte[2048]) + + def task = Mock(TaskRun) { + getInputFiles() >> [new FileHolder(file)] + } + + when: + def metrics = InputFilesProfiler.compute(task) + + then: + metrics.count == 1 + metrics.totalBytes == 2048 + metrics.maxFileBytes == 2048 + metrics.minFileBytes == 2048 + } +} diff --git a/plugins/nf-seqera/src/test/io/seqera/executor/LabelsTest.groovy b/plugins/nf-seqera/src/test/io/seqera/executor/LabelsTest.groovy new file mode 100644 index 0000000000..3437afa95c --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/executor/LabelsTest.groovy @@ -0,0 +1,201 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import nextflow.NextflowMeta +import nextflow.config.Manifest +import nextflow.script.PlatformMetadata +import nextflow.script.WorkflowMetadata +import spock.lang.Specification + +/** + * Tests for Labels helper + * + * @author Paolo Di Tommaso + */ +class LabelsTest extends Specification { + + def 'should create labels with all workflow metadata'() { + given: + def sessionId = UUID.randomUUID() + def workflow = Mock(WorkflowMetadata) { + getProjectName() >> 'nf-core/rnaseq' + getUserName() >> 'pditommaso' + getRunName() >> 'crazy_darwin' + getSessionId() >> sessionId + isResume() >> true + getRevision() >> '3.12.0' + getCommitId() >> 'abc1234' + getRepository() >> 'https://github.com/nf-core/rnaseq' + getManifest() >> new Manifest([name: 'nf-core/rnaseq']) + } + + when: + def labels = new Labels() + .withWorkflowMetadata(workflow) + + then: + labels.entries['nextflow.io/projectName'] == 'nf-core/rnaseq' + labels.entries['nextflow.io/userName'] == 'pditommaso' + labels.entries['nextflow.io/runName'] == 'crazy_darwin' + labels.entries['nextflow.io/sessionId'] == sessionId.toString() + labels.entries['nextflow.io/resume'] == 'true' + labels.entries['nextflow.io/revision'] == '3.12.0' + labels.entries['nextflow.io/commitId'] == 'abc1234' + labels.entries['nextflow.io/repository'] == 'https://github.com/nf-core/rnaseq' + labels.entries['nextflow.io/manifestName'] == 'nf-core/rnaseq' + labels.entries['nextflow.io/runtimeVersion'] == NextflowMeta.instance.version.toString() + } + + def 'should compute stable runId from sessionId and runName'() { + given: + def sid = 'e2315a82-49b0-4langc3-a58a-0d7d52f7e3a1' + def runName = 'crazy_darwin' + + expect: + Labels.runId(sid, runName) == Labels.runId(sid, runName) + Labels.runId(sid, runName) != Labels.runId(sid, 'other_name') + Labels.runId(sid, runName) != Labels.runId(UUID.randomUUID().toString(), runName) + } + + def 'should omit null workflow metadata from labels'() { + given: + def workflow = Mock(WorkflowMetadata) { + getProjectName() >> 'hello' + getUserName() >> 'user1' + getRunName() >> 'happy_turing' + getSessionId() >> UUID.randomUUID() + isResume() >> false + getRevision() >> null + getCommitId() >> null + getRepository() >> null + getManifest() >> new Manifest([:]) + } + + when: + def labels = new Labels() + .withWorkflowMetadata(workflow) + + then: + labels.entries.containsKey('nextflow.io/projectName') + labels.entries.containsKey('nextflow.io/userName') + labels.entries.containsKey('nextflow.io/runName') + labels.entries.containsKey('nextflow.io/sessionId') + labels.entries['nextflow.io/resume'] == 'false' + !labels.entries.containsKey('nextflow.io/revision') + !labels.entries.containsKey('nextflow.io/commitId') + !labels.entries.containsKey('nextflow.io/repository') + !labels.entries.containsKey('nextflow.io/manifestName') + } + + def 'should add scheduler labels'() { + when: + def labels = new Labels() + .withSchedRunId('run-123') + .withSchedClusterId('cluster-456') + + then: + labels.entries['seqera:sched:runId'] == 'run-123' + labels.entries['seqera:sched:clusterId'] == 'cluster-456' + } + + def 'should skip null scheduler labels'() { + when: + def labels = new Labels() + .withSchedRunId(null) + .withSchedClusterId(null) + + then: + !labels.entries.containsKey('seqera:sched:runId') + !labels.entries.containsKey('seqera:sched:clusterId') + } + + def 'should allow user labels to override implicit labels'() { + given: + def workflow = Mock(WorkflowMetadata) { + getProjectName() >> 'hello' + getUserName() >> 'user1' + getRunName() >> 'happy_turing' + getSessionId() >> UUID.randomUUID() + isResume() >> false + getManifest() >> new Manifest([:]) + } + + when: + def labels = new Labels() + .withWorkflowMetadata(workflow) + .withUserLabels([ + 'nextflow.io/runName': 'custom_name', + 'team': 'research' + ]) + + then: + labels.entries['nextflow.io/runName'] == 'custom_name' + labels.entries['team'] == 'research' + labels.entries['nextflow.io/projectName'] == 'hello' + } + + def 'should include platform workflowId when available'() { + given: + def workflow = Mock(WorkflowMetadata) { + getProjectName() >> 'hello' + getUserName() >> 'user1' + getRunName() >> 'happy_turing' + getSessionId() >> UUID.randomUUID() + isResume() >> false + getManifest() >> new Manifest([:]) + getPlatform() >> new PlatformMetadata('wf-abc123') + } + + when: + def labels = new Labels() + .withWorkflowMetadata(workflow) + + then: + labels.entries['seqera.io/platform/workflowId'] == 'wf-abc123' + } + + def 'should omit platform workflowId when not set'() { + given: + def workflow = Mock(WorkflowMetadata) { + getProjectName() >> 'hello' + getUserName() >> 'user1' + getRunName() >> 'happy_turing' + getSessionId() >> UUID.randomUUID() + isResume() >> false + getManifest() >> new Manifest([:]) + getPlatform() >> new PlatformMetadata() + } + + when: + def labels = new Labels() + .withWorkflowMetadata(workflow) + + then: + !labels.entries.containsKey('seqera.io/platform/workflowId') + } + + def 'should handle null user labels'() { + when: + def labels = new Labels() + .withUserLabels(null) + + then: + labels.entries.isEmpty() + } +} diff --git a/plugins/nf-seqera/src/test/io/seqera/executor/SeqeraBatchSubmitterTest.groovy b/plugins/nf-seqera/src/test/io/seqera/executor/SeqeraBatchSubmitterTest.groovy new file mode 100644 index 0000000000..80dc6a37d9 --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/executor/SeqeraBatchSubmitterTest.groovy @@ -0,0 +1,593 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import java.nio.file.Files +import java.nio.file.Path +import java.util.concurrent.atomic.AtomicInteger + +import io.seqera.sched.api.schema.v1a1.CreateTasksResponse +import io.seqera.sched.api.schema.v1a1.Task +import io.seqera.sched.client.SchedClient +import nextflow.file.FileHolder +import nextflow.processor.TaskRun +import nextflow.util.Duration +import spock.lang.Specification +import spock.lang.TempDir +import spock.lang.Timeout + +/** + * Tests for SeqeraBatchSubmitter + * + * @author Lorenzo Fontana + */ +@Timeout(30) +class SeqeraBatchSubmitterTest extends Specification { + + static final String TEST_RUN = 'run-test:local' + + def 'should batch multiple tasks submitted within the interval'() { + given: + def taskIds = ['task-1', 'task-2', 'task-3'] + def capturedTasks = [] + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + capturedTasks.addAll(tasks) + Stub(CreateTasksResponse) { + getTaskIds() >> taskIds.take(tasks.size()) + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('500ms')) + def handlers = (1..3).collect { createMockHandler() } + def tasks = (1..3).collect { new Task().image("image-$it") } + + when: 'start submitter and enqueue tasks quickly' + submitter.start() + handlers.eachWithIndex { handler, i -> + submitter.submit(handler, tasks[i]) + } + // Wait for the batch interval to elapse and tasks to be submitted + sleep(800) + submitter.shutdown() + + then: 'all tasks should be submitted in a single batch' + capturedTasks.size() == 3 + capturedTasks*.image == ['image-1', 'image-2', 'image-3'] + + and: 'task IDs should be assigned to handlers' + handlers.each { handler -> + 1 * handler.setBatchTaskId(_) + } + } + + def 'should submit tasks in separate batches when interval elapses between them'() { + given: + def batchCount = new AtomicInteger() + def batches = [].asSynchronized() // thread-safe list to capture batches + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + def batchNum = batchCount.incrementAndGet() + def tasksInBatch = tasks.collect { it.image } + batches << [batch: batchNum, tasks: tasksInBatch] + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${it.image}" } + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('200ms')) + + when: 'enqueue first batch, wait for flush, then enqueue second batch' + submitter.start() + // First batch - tasks s1 and s2 + submitter.submit(createMockHandler(), new Task().image('s1')) + submitter.submit(createMockHandler(), new Task().image('s2')) + // Wait for first batch to flush (interval + buffer) + sleep(400) + // Second batch - task s3 + submitter.submit(createMockHandler(), new Task().image('s3')) + // Wait for second batch to flush + sleep(400) + submitter.shutdown() + + then: 'should have two separate batches with correct tasks' + batches.size() == 2 + and: 'first batch contains s1 and s2' + batches[0].batch == 1 + batches[0].tasks == ['s1', 's2'] + and: 'second batch contains only s3' + batches[1].batch == 2 + batches[1].tasks == ['s3'] + } + + def 'should flush batch immediately when reaching max size'() { + given: + def batchSizes = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + batchSizes << tasks.size() + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('10s')) // Long interval + + when: 'enqueue exactly TASKS_PER_REQUEST tasks' + submitter.start() + (1..SeqeraBatchSubmitter.TASKS_PER_REQUEST).each { + submitter.submit(createMockHandler(), new Task().image("img$it")) + } + // Give a small delay for the batch to be processed + sleep(200) + submitter.shutdown() + + then: 'should flush immediately due to batch size limit' + batchSizes.size() >= 1 + batchSizes[0] == SeqeraBatchSubmitter.TASKS_PER_REQUEST + } + + def 'should split into multiple batches when exceeding max size'() { + given: + def totalTasks = SeqeraBatchSubmitter.TASKS_PER_REQUEST + 20 // 120 tasks + def batchSizes = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + batchSizes << tasks.size() + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('10s')) // Long interval + + when: 'enqueue more than TASKS_PER_REQUEST tasks' + submitter.start() + (1..totalTasks).each { + submitter.submit(createMockHandler(), new Task().image("img$it")) + } + // Wait for batches to be processed + sleep(500) + submitter.shutdown() + + then: 'should create two batches: 100 + 20' + batchSizes.size() == 2 + batchSizes[0] == SeqeraBatchSubmitter.TASKS_PER_REQUEST // 100 + batchSizes[1] == 20 + } + + def 'should flush remaining tasks on shutdown'() { + given: + def capturedTasks = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + capturedTasks.addAll(tasks) + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('10s')) // Long interval + + when: 'enqueue tasks and immediately shutdown' + submitter.start() + submitter.submit(createMockHandler(), new Task().image('img1')) + submitter.submit(createMockHandler(), new Task().image('img2')) + // Shutdown without waiting for interval + submitter.shutdown() + + then: 'tasks should still be submitted' + capturedTasks.size() == 2 + } + + def 'should throw exception when enqueueing after shutdown'() { + given: + def client = Stub(SchedClient) + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('1s')) + + when: + submitter.start() + submitter.shutdown() + submitter.submit(createMockHandler(), new Task().image('img1')) + + then: + thrown(IllegalStateException) + } + + def 'should propagate failure to handlers on API error'() { + given: + def apiError = new RuntimeException('API error') + def client = Stub(SchedClient) { + createTasks(_, _) >> { throw apiError } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('100ms')) + def handler1 = Mock(SeqeraTaskHandler) + def handler2 = Mock(SeqeraTaskHandler) + + when: + submitter.start() + submitter.submit(handler1, new Task().image('img1')) + submitter.submit(handler2, new Task().image('img2')) + sleep(300) + submitter.shutdown() + + then: 'all handlers should receive the failure' + 1 * handler1.onBatchSubmitFailure(apiError) + 1 * handler2.onBatchSubmitFailure(apiError) + } + + def 'should propagate failure to handlers when API returns wrong number of task IDs'() { + given: + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + Stub(CreateTasksResponse) { + // Return fewer IDs than tasks submitted + getTaskIds() >> ['task-1'] + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('100ms')) + def handler1 = Mock(SeqeraTaskHandler) + def handler2 = Mock(SeqeraTaskHandler) + def handler3 = Mock(SeqeraTaskHandler) + + when: + submitter.start() + submitter.submit(handler1, new Task().image('img1')) + submitter.submit(handler2, new Task().image('img2')) + submitter.submit(handler3, new Task().image('img3')) + sleep(300) + submitter.shutdown() + + then: 'handlers should receive failure due to mismatched IDs' + 1 * handler1.onBatchSubmitFailure({ it instanceof IllegalStateException }) + 1 * handler2.onBatchSubmitFailure({ it instanceof IllegalStateException }) + 1 * handler3.onBatchSubmitFailure({ it instanceof IllegalStateException }) + } + + def 'should start batch timer only when first task arrives not when thread starts'() { + given: + def submitTimes = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + submitTimes << System.currentTimeMillis() + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('300ms')) + + when: 'start submitter, wait longer than interval, then enqueue tasks' + submitter.start() + // Wait longer than the batch interval before enqueueing + sleep(500) + def enqueueTime = System.currentTimeMillis() + submitter.submit(createMockHandler(), new Task().image('img1')) + submitter.submit(createMockHandler(), new Task().image('img2')) + // Wait for batch to flush + sleep(500) + submitter.shutdown() + + then: 'batch should be submitted ~300ms after enqueue, not immediately' + submitTimes.size() == 1 + // The submit should happen ~300ms after enqueue + def timeSinceEnqueue = submitTimes[0] - enqueueTime + timeSinceEnqueue >= 250 // Allow some tolerance + timeSinceEnqueue < 600 // But not too long + } + + def 'should use default interval when not specified'() { + given: + def client = Stub(SchedClient) + + when: + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN) + + then: + submitter.requestInterval == SeqeraBatchSubmitter.REQUEST_INTERVAL + submitter.keepAliveInterval == SeqeraBatchSubmitter.KEEP_ALIVE_INTERVAL + } + + def 'should send keep-alive when no tasks received within keep-alive interval'() { + given: + def submissions = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + submissions << [runId: runId, taskCount: tasks.size()] + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + // Short keep-alive interval for testing + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('10s'), Duration.of('200ms')) + + when: 'start submitter and wait for keep-alive interval without enqueueing tasks' + submitter.start() + // Wait for keep-alive to trigger (interval + buffer) + sleep(400) + submitter.shutdown() + + then: 'should have sent at least one keep-alive (empty submission)' + submissions.size() >= 1 + submissions.any { it.taskCount == 0 } + } + + def 'should not send keep-alive when tasks are being submitted regularly'() { + given: + def submissions = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + submissions << [runId: runId, taskCount: tasks.size()] + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + // Short intervals for testing + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('100ms'), Duration.of('300ms')) + + when: 'start submitter and continuously enqueue tasks' + submitter.start() + // Enqueue tasks at intervals shorter than keep-alive + 5.times { i -> + submitter.submit(createMockHandler(), new Task().image("img$i")) + sleep(80) + } + sleep(200) // Wait for final batch to flush + submitter.shutdown() + + then: 'all submissions should have tasks, no keep-alive (empty) submissions' + submissions.size() >= 1 + submissions.every { it.taskCount > 0 } + } + + def 'should invoke error callback on fatal error in thread'() { + given: + // Use an Error to simulate a truly fatal condition that escapes flushBatch + def fatalError = new OutOfMemoryError('Fatal thread error') + def errorReceived = null + def onError = { Throwable t -> errorReceived = t } + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + // Throw a fatal Error that will escape flushBatch + throw fatalError + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('100ms'), Duration.of('10s'), onError) + def handler = Mock(SeqeraTaskHandler) + + when: + submitter.start() + submitter.submit(handler, new Task().image('img1')) + sleep(300) + submitter.shutdown() + + then: 'handler should receive failure (wrapped in RuntimeException since it was an Error)' + 1 * handler.onBatchSubmitFailure({ it instanceof RuntimeException && it.cause == fatalError }) + and: 'error callback should be invoked with original error' + errorReceived == fatalError + } + + def 'should drain and fail pending tasks on fatal error'() { + given: + def fatalError = new RuntimeException('Fatal thread error') + def failedHandlers = [].asSynchronized() + def onError = { Throwable t -> /* no-op */ } + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + // Always throw to simulate fatal error + throw fatalError + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('100ms'), Duration.of('10s'), onError) + + // Create handlers that track when they receive failure notification + def handlers = (1..3).collect { + def h = Mock(SeqeraTaskHandler) { + onBatchSubmitFailure(_) >> { args -> failedHandlers << it } + } + h + } + + when: + submitter.start() + handlers.each { h -> + submitter.submit(h, new Task().image('img')) + } + sleep(300) + submitter.shutdown() + + then: 'all handlers should receive failure notification' + handlers.each { h -> + 1 * h.onBatchSubmitFailure(fatalError) + } + } + + def 'should continue after keep-alive failure'() { + given: + def keepAliveFailCount = new AtomicInteger() + def taskSubmissions = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + if (tasks.isEmpty()) { + // Keep-alive call - fail it + keepAliveFailCount.incrementAndGet() + throw new RuntimeException('Keep-alive failed') + } + // Normal task submission + taskSubmissions << tasks.size() + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + // Short keep-alive interval to trigger failures quickly + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('500ms'), Duration.of('100ms')) + + when: 'start submitter, let keep-alive fail, then submit tasks' + submitter.start() + // Wait for a few keep-alive failures + sleep(350) + // Now submit a task + submitter.submit(createMockHandler(), new Task().image('img1')) + // Wait for task to be submitted + sleep(700) + submitter.shutdown() + + then: 'keep-alive should have failed but thread should continue' + keepAliveFailCount.get() >= 1 + and: 'task submission should still succeed' + taskSubmissions.size() >= 1 + } + + def 'should work without error callback'() { + given: + def taskSubmissions = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + taskSubmissions << tasks.size() + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + // Constructor without error callback (null) + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('100ms'), Duration.of('10s'), null) + + when: + submitter.start() + submitter.submit(createMockHandler(), new Task().image('img1')) + sleep(300) + submitter.shutdown() + + then: 'should work normally' + taskSubmissions.size() >= 1 + } + + // -- input files metrics tests -- + + @TempDir + Path tempDir + + def 'should attach input files metrics to task before submission'() { + given: + def file1 = tempDir.resolve('a.txt') + Files.write(file1, new byte[500]) + def file2 = tempDir.resolve('b.txt') + Files.write(file2, new byte[3000]) + def capturedTasks = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + capturedTasks.addAll(tasks) + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('100ms')) + def taskRun = Mock(TaskRun) { + getInputFiles() >> [new FileHolder(file1), new FileHolder(file2)] + } + def handler = Mock(SeqeraTaskHandler) { + getTask() >> taskRun + } + + when: + submitter.start() + submitter.submit(handler, new Task().image('img1')) + sleep(300) + submitter.shutdown() + + then: 'metrics should be set on the submitted task' + capturedTasks.size() == 1 + def metrics = capturedTasks[0].getInputFiles() + metrics != null + metrics.count == 2 + metrics.totalBytes == 3500 + metrics.maxFileBytes == 3000 + metrics.minFileBytes == 500 + } + + def 'should submit task without metrics when no input files'() { + given: + def capturedTasks = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + capturedTasks.addAll(tasks) + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('100ms')) + def taskRun = Mock(TaskRun) { + getInputFiles() >> [] + } + def handler = Mock(SeqeraTaskHandler) { + getTask() >> taskRun + } + + when: + submitter.start() + submitter.submit(handler, new Task().image('img1')) + sleep(300) + submitter.shutdown() + + then: 'task should be submitted without metrics' + capturedTasks.size() == 1 + capturedTasks[0].getInputFiles() == null + } + + def 'should submit task even when metrics computation fails'() { + given: + def capturedTasks = [].asSynchronized() + def client = Stub(SchedClient) { + createTasks(_, _) >> { String runId, List tasks -> + capturedTasks.addAll(tasks) + Stub(CreateTasksResponse) { + getTaskIds() >> tasks.collect { "task-${System.nanoTime()}" } + } + } + } + def submitter = new SeqeraBatchSubmitter(client, TEST_RUN, Duration.of('100ms')) + def taskRun = Mock(TaskRun) { + getInputFiles() >> { throw new RuntimeException('Cannot access input files') } + } + def handler = Mock(SeqeraTaskHandler) { + getTask() >> taskRun + } + + when: + submitter.start() + submitter.submit(handler, new Task().image('img1')) + sleep(300) + submitter.shutdown() + + then: 'task should still be submitted despite metrics failure' + capturedTasks.size() == 1 + } + + /** + * Creates a mock handler that can track setBatchTaskId and onBatchSubmitFailure calls + */ + private SeqeraTaskHandler createMockHandler() { + Mock(SeqeraTaskHandler) + } +} diff --git a/plugins/nf-seqera/src/test/io/seqera/executor/SeqeraExecutorTest.groovy b/plugins/nf-seqera/src/test/io/seqera/executor/SeqeraExecutorTest.groovy new file mode 100644 index 0000000000..a220caec8d --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/executor/SeqeraExecutorTest.groovy @@ -0,0 +1,129 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import io.seqera.config.SeqeraConfig +import io.seqera.sched.client.SchedClientConfig +import nextflow.SysEnv +import nextflow.platform.PlatformHelper +import spock.lang.Specification + +/** + * Tests for SeqeraExecutor client configuration + * + * @author Paolo Di Tommaso + */ +class SeqeraExecutorTest extends Specification { + + def cleanup() { + SysEnv.pop() + } + + def 'should create client config with config settings'() { + given: + SysEnv.push([:]) + + when: + def config = buildClientConfig( + [endpoint: 'https://sched.example.com', region: 'us-west-2'], + [endpoint: 'https://api.platform.example.com', accessToken: 'config-access-token', refreshToken: 'config-refresh-token'] + ) + + then: + config.endpoint == 'https://sched.example.com' + config.platformUrl == 'https://api.platform.example.com' + config.accessToken == 'config-access-token' + config.refreshToken == 'config-refresh-token' + } + + def 'should create client config with env variable settings'() { + given: + SysEnv.push([ + TOWER_API_ENDPOINT: 'https://api.env.example.com', + TOWER_ACCESS_TOKEN: 'env-access-token', + TOWER_REFRESH_TOKEN: 'env-refresh-token' + ]) + + when: + def config = buildClientConfig( + [endpoint: 'https://sched.example.com', region: 'us-west-2'], + [:] + ) + + then: + config.endpoint == 'https://sched.example.com' + config.platformUrl == 'https://api.env.example.com' + config.accessToken == 'env-access-token' + config.refreshToken == 'env-refresh-token' + } + + def 'should use default platform url when not configured'() { + given: + SysEnv.push([:]) + + when: + def config = buildClientConfig( + [endpoint: 'https://sched.example.com', region: 'us-west-2'], + [accessToken: 'my-token'] + ) + + then: + config.endpoint == 'https://sched.example.com' + config.platformUrl == 'https://api.cloud.seqera.io' + config.accessToken == 'my-token' + config.refreshToken == null + } + + def 'should prefer config over env variables'() { + given: + SysEnv.push([ + TOWER_API_ENDPOINT: 'https://api.env.example.com', + TOWER_ACCESS_TOKEN: 'env-access-token', + TOWER_REFRESH_TOKEN: 'env-refresh-token' + ]) + + when: + def config = buildClientConfig( + [endpoint: 'https://sched.example.com', region: 'us-west-2'], + [endpoint: 'https://api.config.example.com', accessToken: 'config-access-token', refreshToken: 'config-refresh-token'] + ) + + then: + config.endpoint == 'https://sched.example.com' + config.platformUrl == 'https://api.config.example.com' + config.accessToken == 'config-access-token' + config.refreshToken == 'config-refresh-token' + } + + /** + * Builds a SchedClientConfig using the same logic as {@link SeqeraExecutor#createClient()} + */ + private SchedClientConfig buildClientConfig(Map executorOpts, Map towerConfig) { + def seqeraConfig = new SeqeraConfig([executor: executorOpts]).executor + def accessToken = PlatformHelper.getAccessToken(towerConfig, SysEnv.get()) + def refreshToken = PlatformHelper.getRefreshToken(towerConfig, SysEnv.get()) + def platformUrl = PlatformHelper.getEndpoint(towerConfig, SysEnv.get()) + return SchedClientConfig.builder() + .endpoint(seqeraConfig.endpoint) + .platformUrl(platformUrl) + .accessToken(accessToken) + .refreshToken(refreshToken) + .retryConfig(seqeraConfig.retryOpts()) + .build() + } +} diff --git a/plugins/nf-seqera/src/test/io/seqera/executor/SeqeraTaskHandlerTest.groovy b/plugins/nf-seqera/src/test/io/seqera/executor/SeqeraTaskHandlerTest.groovy new file mode 100644 index 0000000000..16decc00a1 --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/executor/SeqeraTaskHandlerTest.groovy @@ -0,0 +1,779 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.executor + +import com.google.common.hash.HashCode +import io.seqera.config.ExecutorOpts +import io.seqera.sched.api.schema.v1a1.DescribeTaskResponse +import io.seqera.sched.api.schema.v1a1.GetTaskLogsResponse +import io.seqera.sched.api.schema.v1a1.MachineInfo +import io.seqera.sched.api.schema.v1a1.NextflowTask +import io.seqera.sched.api.schema.v1a1.PriceModel as SchedPriceModel +import io.seqera.sched.api.schema.v1a1.ResourceLimit +import io.seqera.sched.api.schema.v1a1.ResourceRequirement +import io.seqera.sched.api.schema.v1a1.Task +import io.seqera.sched.api.schema.v1a1.TaskAttempt +import io.seqera.sched.api.schema.v1a1.TaskState as SchedTaskState +import io.seqera.sched.api.schema.v1a1.TaskStatus as SchedTaskStatus +import io.seqera.sched.client.SchedClient +import nextflow.cloud.types.CloudMachineInfo +import nextflow.cloud.types.PriceModel +import nextflow.util.Duration +import nextflow.util.MemoryUnit +import nextflow.exception.ProcessException +import nextflow.processor.TaskConfig +import nextflow.processor.TaskId +import nextflow.processor.TaskProcessor +import nextflow.processor.TaskRun +import nextflow.processor.TaskStatus +import spock.lang.Specification + +import java.nio.file.Paths + +/** + * Tests for SeqeraTaskHandler metadata fetching functionality + * + * @author Paolo Di Tommaso + */ +class SeqeraTaskHandlerTest extends Specification { + + def 'should return null for getMachineInfo when cachedTaskState is null'() { + given: + def handler = createHandler() + + expect: + handler.getMachineInfo() == null + } + + def 'should return null for getMachineInfo when attempts list is empty'() { + given: + def handler = createHandler() + handler.cachedTaskState = new SchedTaskState().attempts([]) + + expect: + handler.getMachineInfo() == null + } + + def 'should return null for getMachineInfo when last attempt has no machine info'() { + given: + def handler = createHandler() + def attempt = new TaskAttempt() + .index(1) + .nativeId('arn:aws:ecs:us-east-1:123:task/abc') + .status(SchedTaskStatus.SUCCEEDED) + handler.cachedTaskState = new SchedTaskState().attempts([attempt]) + + expect: + handler.getMachineInfo() == null + } + + def 'should extract machine info from last task attempt'() { + given: + def handler = createHandler() + def machineInfo = new MachineInfo() + .type('m5.large') + .zone('us-east-1a') + .priceModel(SchedPriceModel.SPOT) + def attempt = new TaskAttempt() + .index(1) + .nativeId('arn:aws:ecs:us-east-1:123:task/abc') + .status(SchedTaskStatus.SUCCEEDED) + .machineInfo(machineInfo) + handler.cachedTaskState = new SchedTaskState().attempts([attempt]) + + when: + def result = handler.getMachineInfo() + + then: + result != null + result.type == 'm5.large' + result.zone == 'us-east-1a' + result.priceModel == PriceModel.spot + } + + def 'should cache machine info after first retrieval'() { + given: + def handler = createHandler() + def machineInfo = new MachineInfo() + .type('c5.xlarge') + .zone('eu-west-1b') + .priceModel(SchedPriceModel.STANDARD) + def attempt = new TaskAttempt() + .index(1) + .nativeId('arn:aws:ecs:eu-west-1:123:task/xyz') + .status(SchedTaskStatus.SUCCEEDED) + .machineInfo(machineInfo) + handler.cachedTaskState = new SchedTaskState().attempts([attempt]) + + when: + def first = handler.getMachineInfo() + // Clear the cached task state + handler.cachedTaskState = null + def second = handler.getMachineInfo() + + then: + first.is(second) + } + + def 'should use last attempt when multiple attempts exist'() { + given: + def handler = createHandler() + def info1 = new MachineInfo().type('m5.large').zone('us-east-1a').priceModel(SchedPriceModel.SPOT) + def info2 = new MachineInfo().type('c5.xlarge').zone('us-east-1b').priceModel(SchedPriceModel.STANDARD) + def attempt1 = new TaskAttempt().index(1).nativeId('task-1').status(SchedTaskStatus.FAILED).machineInfo(info1) + def attempt2 = new TaskAttempt().index(2).nativeId('task-2').status(SchedTaskStatus.SUCCEEDED).machineInfo(info2) + handler.cachedTaskState = new SchedTaskState().attempts([attempt1, attempt2]) + + when: + def result = handler.getMachineInfo() + + then: + result.type == 'c5.xlarge' + result.zone == 'us-east-1b' + result.priceModel == PriceModel.standard + } + + def 'should return null for getNumSpotInterruptions when task not completed'() { + given: + def handler = createHandler() + handler.taskId = 'task-123' + handler.status = TaskStatus.RUNNING + handler.cachedTaskState = new SchedTaskState().numSpotInterruptions(2) + + expect: + handler.getNumSpotInterruptions() == null + } + + def 'should return null for getNumSpotInterruptions when cachedTaskState is null'() { + given: + def handler = createHandler() + handler.taskId = 'task-123' + handler.status = TaskStatus.COMPLETED + + expect: + handler.getNumSpotInterruptions() == null + } + + def 'should return spot interruptions count from task state'() { + given: + def handler = createHandler() + handler.taskId = 'task-123' + handler.status = TaskStatus.COMPLETED + handler.cachedTaskState = new SchedTaskState().numSpotInterruptions(3) + + expect: + handler.getNumSpotInterruptions() == 3 + } + + def 'should return zero spot interruptions when none occurred'() { + given: + def handler = createHandler() + handler.taskId = 'task-123' + handler.status = TaskStatus.COMPLETED + handler.cachedTaskState = new SchedTaskState().numSpotInterruptions(0) + + expect: + handler.getNumSpotInterruptions() == 0 + } + + def 'should return null for getLogStreamId when cachedTaskState is null'() { + given: + def handler = createHandler() + + expect: + handler.getLogStreamId() == null + } + + def 'should return log stream id from cached task state'() { + given: + def handler = createHandler() + handler.cachedTaskState = new SchedTaskState().logStreamId('log-stream-abc123') + + expect: + handler.getLogStreamId() == 'log-stream-abc123' + } + + def 'should return null for getNativeId when cachedTaskState is null'() { + given: + def handler = createHandler() + + expect: + handler.getNativeId() == null + } + + def 'should return task id from cached task state'() { + given: + def handler = createHandler() + handler.cachedTaskState = new SchedTaskState().id('tsk-abc123') + + expect: + handler.getNativeId() == 'tsk-abc123' + } + + def 'should populate trace record with metadata'() { + given: + def handler = createHandlerForTraceTest() + handler.taskId = 'task-123' + handler.status = TaskStatus.COMPLETED + def machineInfo = new MachineInfo() + .type('m5.large') + .zone('us-east-1a') + .priceModel(SchedPriceModel.SPOT) + def attempt = new TaskAttempt() + .index(1) + .nativeId('arn:aws:ecs:us-east-1:123:task/abc') + .status(SchedTaskStatus.SUCCEEDED) + .machineInfo(machineInfo) + handler.cachedTaskState = new SchedTaskState() + .id('tsk-xyz789') + .attempts([attempt]) + .numSpotInterruptions(2) + .logStreamId('log-stream-xyz') + + when: + def trace = handler.getTraceRecord() + + then: + trace.get('native_id') == 'tsk-xyz789' + trace.getMachineInfo().type == 'm5.large' + trace.getNumSpotInterruptions() == 2 + trace.getLogStreamId() == 'log-stream-xyz' + trace.getExecutorName() == 'seqera/aws' + } + + def 'should detect completion when batch submission fails'() { + given: + def error = new RuntimeException('Batch submission failed') + def handler = createHandlerWithError(error) + // Simulate batch submission failure - task has error and status is COMPLETED but never went through RUNNING + handler.status = TaskStatus.COMPLETED + + expect: + // checkIfCompleted should return true to allow proper error propagation + handler.checkIfCompleted() == true + } + + def 'should not detect completion for normal pending task'() { + given: + def handler = createHandler() + // Normal task that has been submitted but not yet running + handler.status = TaskStatus.SUBMITTED + + expect: + // checkIfCompleted should return false - task is still pending + handler.checkIfCompleted() == false + } + + def 'should set task error with error message when failed with no exit code'() { + given: + Throwable capturedError = null + Integer capturedExitStatus = null + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getWorkDirStr() >> '/work/ab/cd1234' + getConfig() >> Mock(TaskConfig) + lazyName() >> 'test_task' + setExitStatus(_) >> { args -> capturedExitStatus = args[0] } + getExitStatus() >> { capturedExitStatus } + setError(_) >> { args -> capturedError = args[0] } + setStdout(_) >> {} + setStderr(_) >> {} + } + def taskState = new SchedTaskState() + .status(SchedTaskStatus.FAILED) + .errorMessage('Container terminated with OOMKilled') + def describeResponse = new DescribeTaskResponse().taskState(taskState) + def client = Mock(SchedClient) { + describeTask(_) >> describeResponse + getTaskLogs(_) >> null + } + def executor = Mock(SeqeraExecutor) { + getClient() >> client + } + def handler = Spy(new SeqeraTaskHandler(taskRun, executor)) { + readExitFile() >> Integer.MAX_VALUE + } + handler.setBatchTaskId('task-123') + handler.status = TaskStatus.RUNNING + + when: + def completed = handler.checkIfCompleted() + + then: + completed + capturedExitStatus == Integer.MAX_VALUE + capturedError instanceof ProcessException + capturedError.message == 'Container terminated with OOMKilled' + } + + def 'should set task error with fallback message when failed with no exit code and no error message'() { + given: + Throwable capturedError = null + Integer capturedExitStatus = null + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getWorkDirStr() >> '/work/ab/cd1234' + getConfig() >> Mock(TaskConfig) + lazyName() >> 'test_task' + setExitStatus(_) >> { args -> capturedExitStatus = args[0] } + getExitStatus() >> { capturedExitStatus } + setError(_) >> { args -> capturedError = args[0] } + setStdout(_) >> {} + setStderr(_) >> {} + } + def taskState = new SchedTaskState() + .status(SchedTaskStatus.FAILED) + def describeResponse = new DescribeTaskResponse().taskState(taskState) + def client = Mock(SchedClient) { + describeTask(_) >> describeResponse + getTaskLogs(_) >> null + } + def executor = Mock(SeqeraExecutor) { + getClient() >> client + } + def handler = Spy(new SeqeraTaskHandler(taskRun, executor)) { + readExitFile() >> Integer.MAX_VALUE + } + handler.setBatchTaskId('task-456') + handler.status = TaskStatus.RUNNING + + when: + def completed = handler.checkIfCompleted() + + then: + completed + capturedExitStatus == Integer.MAX_VALUE + capturedError instanceof ProcessException + capturedError.message == 'Task failed for unknown reason' + } + + def 'should not set task error when failed with valid exit code'() { + given: + Throwable capturedError = null + Integer capturedExitStatus = null + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getWorkDirStr() >> '/work/ab/cd1234' + getConfig() >> Mock(TaskConfig) + lazyName() >> 'test_task' + setExitStatus(_) >> { args -> capturedExitStatus = args[0] } + getExitStatus() >> { capturedExitStatus } + setError(_) >> { args -> capturedError = args[0] } + setStdout(_) >> {} + setStderr(_) >> {} + } + def taskState = new SchedTaskState() + .status(SchedTaskStatus.FAILED) + .errorMessage('Some error') + def describeResponse = new DescribeTaskResponse().taskState(taskState) + def client = Mock(SchedClient) { + describeTask(_) >> describeResponse + getTaskLogs(_) >> null + } + def executor = Mock(SeqeraExecutor) { + getClient() >> client + } + def handler = Spy(new SeqeraTaskHandler(taskRun, executor)) { + readExitFile() >> 1 + } + handler.setBatchTaskId('task-789') + handler.status = TaskStatus.RUNNING + + when: + def completed = handler.checkIfCompleted() + + then: + completed + capturedExitStatus == 1 + capturedError == null + } + + def 'should set index and hash on submitted task'() { + given: + Task capturedTask = null + def batchSubmitter = Mock(SeqeraBatchSubmitter) { + submit(_, _) >> { handler, task -> capturedTask = task } + } + def seqeraConfig = Mock(ExecutorOpts) { + getMachineRequirement() >> null + } + def executor = Mock(SeqeraExecutor) { + getClient() >> Mock(SchedClient) + getBatchSubmitter() >> batchSubmitter + getSeqeraConfig() >> seqeraConfig + } + def taskConfig = Mock(TaskConfig) { + getCpus() >> 1 + getMemory() >> null + getAccelerator() >> null + getDisk() >> null + } + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getWorkDirStr() >> '/work/ab/cd1234' + getConfig() >> taskConfig + getContainer() >> 'ubuntu:latest' + getContainerPlatform() >> null + lazyName() >> 'test_task' + getId() >> TaskId.of(42) + getHash() >> HashCode.fromString('abcd1234') + } + def handler = Spy(new SeqeraTaskHandler(taskRun, executor)) { + fusionEnabled() >> true + fusionSubmitCli() >> ['bash', '-c', 'echo hello'] + fusionLauncher() >> Mock(nextflow.fusion.FusionScriptLauncher) { + fusionEnv() >> [:] + } + } + + when: + handler.submit() + + then: + 1 * executor.ensureRunCreated() + capturedTask != null + capturedTask.getNextflow() != null + capturedTask.getNextflow().getTaskId() == 42 + capturedTask.getNextflow().getHash() == 'abcd1234' + capturedTask.getNextflow().getWorkDir() == '/work/ab/cd1234' + } + + def 'should handle null task id and hash'() { + given: + Task capturedTask = null + def batchSubmitter = Mock(SeqeraBatchSubmitter) { + submit(_, _) >> { handler, task -> capturedTask = task } + } + def seqeraConfig = Mock(ExecutorOpts) { + getMachineRequirement() >> null + } + def executor = Mock(SeqeraExecutor) { + getClient() >> Mock(SchedClient) + getBatchSubmitter() >> batchSubmitter + getSeqeraConfig() >> seqeraConfig + } + def taskConfig = Mock(TaskConfig) { + getCpus() >> 1 + getMemory() >> null + getAccelerator() >> null + getDisk() >> null + } + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getWorkDirStr() >> '/work/ab/cd1234' + getConfig() >> taskConfig + getContainer() >> 'ubuntu:latest' + getContainerPlatform() >> null + lazyName() >> 'test_task' + getId() >> null + getHash() >> null + } + def handler = Spy(new SeqeraTaskHandler(taskRun, executor)) { + fusionEnabled() >> true + fusionSubmitCli() >> ['bash', '-c', 'echo hello'] + fusionLauncher() >> Mock(nextflow.fusion.FusionScriptLauncher) { + fusionEnv() >> [:] + } + } + + when: + handler.submit() + + then: + capturedTask != null + capturedTask.getNextflow() != null + capturedTask.getNextflow().getTaskId() == null + capturedTask.getNextflow().getHash() == null + capturedTask.getNextflow().getWorkDir() == '/work/ab/cd1234' + } + + def 'should return only fusion env when no config environment'() { + given: + def seqeraConfig = Mock(ExecutorOpts) { + getTaskEnvironment() >> null + } + def executor = Mock(SeqeraExecutor) { + getClient() >> Mock(SchedClient) + getSeqeraConfig() >> seqeraConfig + } + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getConfig() >> Mock(TaskConfig) + } + def handler = Spy(new SeqeraTaskHandler(taskRun, executor)) { + fusionLauncher() >> Mock(nextflow.fusion.FusionScriptLauncher) { + fusionEnv() >> [FUSION_KEY: 'fusion_val'] + } + } + + when: + def result = handler.getTaskEnvironment() + + then: + result == [FUSION_KEY: 'fusion_val'] + } + + def 'should merge config environment with fusion env'() { + given: + def seqeraConfig = Mock(ExecutorOpts) { + getTaskEnvironment() >> [MY_VAR: 'my_val', OTHER: 'other_val'] + } + def executor = Mock(SeqeraExecutor) { + getClient() >> Mock(SchedClient) + getSeqeraConfig() >> seqeraConfig + } + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getConfig() >> Mock(TaskConfig) + } + def handler = Spy(new SeqeraTaskHandler(taskRun, executor)) { + fusionLauncher() >> Mock(nextflow.fusion.FusionScriptLauncher) { + fusionEnv() >> [FUSION_KEY: 'fusion_val'] + } + } + + when: + def result = handler.getTaskEnvironment() + + then: + result == [MY_VAR: 'my_val', OTHER: 'other_val', FUSION_KEY: 'fusion_val'] + } + + def 'should give fusion env precedence over config environment'() { + given: + def seqeraConfig = Mock(ExecutorOpts) { + getTaskEnvironment() >> [SHARED_KEY: 'config_val', MY_VAR: 'my_val'] + } + def executor = Mock(SeqeraExecutor) { + getClient() >> Mock(SchedClient) + getSeqeraConfig() >> seqeraConfig + } + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getConfig() >> Mock(TaskConfig) + } + def handler = Spy(new SeqeraTaskHandler(taskRun, executor)) { + fusionLauncher() >> Mock(nextflow.fusion.FusionScriptLauncher) { + fusionEnv() >> [SHARED_KEY: 'fusion_val'] + } + } + + when: + def result = handler.getTaskEnvironment() + + then: + result == [MY_VAR: 'my_val', SHARED_KEY: 'fusion_val'] + } + + def 'should return granted time from resource requirement'() { + given: + def handler = createHandler() + handler.cachedTaskState = new SchedTaskState() + .resourceRequirement(new ResourceRequirement().time('2h')) + + expect: + handler.getGrantedTime() == Duration.of('2h').toMillis() + } + + def 'should fallback to config time when cachedTaskState is null'() { + given: + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getConfig() >> Mock(TaskConfig) { getTime() >> Duration.of('6h') } + } + def executor = Mock(SeqeraExecutor) { getClient() >> Mock(SchedClient) } + def handler = new SeqeraTaskHandler(taskRun, executor) + + expect: + handler.getGrantedTime() == Duration.of('6h').toMillis() + } + + def 'should build resource limit from task config'() { + given: + def taskConfig = Mock(TaskConfig) { + getResourceLimit('memory') >> MemoryUnit.of('2 GB') + getResourceLimit('cpus') >> 4 + } + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getConfig() >> taskConfig + } + def executor = Mock(SeqeraExecutor) { getClient() >> Mock(SchedClient) } + def handler = new SeqeraTaskHandler(taskRun, executor) + + when: + def result = handler.toResourceLimit() + + then: + result != null + result.memoryMiB == 2048 + result.cpuShares == 4096 + } + + def 'should build resource limit with only memory'() { + given: + def taskConfig = Mock(TaskConfig) { + getResourceLimit('memory') >> MemoryUnit.of('800 MB') + getResourceLimit('cpus') >> null + } + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getConfig() >> taskConfig + } + def executor = Mock(SeqeraExecutor) { getClient() >> Mock(SchedClient) } + def handler = new SeqeraTaskHandler(taskRun, executor) + + when: + def result = handler.toResourceLimit() + + then: + result != null + result.memoryMiB == 800 + result.cpuShares == null + } + + def 'should return null resource limit when no limits defined'() { + given: + def taskConfig = Mock(TaskConfig) { + getResourceLimit('memory') >> null + getResourceLimit('cpus') >> null + } + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getConfig() >> taskConfig + } + def executor = Mock(SeqeraExecutor) { getClient() >> Mock(SchedClient) } + def handler = new SeqeraTaskHandler(taskRun, executor) + + when: + def result = handler.toResourceLimit() + + then: + result == null + } + + def 'should include resource limit in submitted task'() { + given: + Task capturedTask = null + def batchSubmitter = Mock(SeqeraBatchSubmitter) { + submit(_, _) >> { handler, task -> capturedTask = task } + } + def seqeraConfig = Mock(ExecutorOpts) { + getMachineRequirement() >> null + } + def executor = Mock(SeqeraExecutor) { + getClient() >> Mock(SchedClient) + getBatchSubmitter() >> batchSubmitter + getSeqeraConfig() >> seqeraConfig + } + def taskConfig = Mock(TaskConfig) { + getCpus() >> 1 + getMemory() >> null + getAccelerator() >> null + getDisk() >> null + getResourceLimit('memory') >> MemoryUnit.of('2 GB') + getResourceLimit('cpus') >> null + } + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getWorkDirStr() >> '/work/ab/cd1234' + getConfig() >> taskConfig + getContainer() >> 'ubuntu:latest' + getContainerPlatform() >> null + lazyName() >> 'test_task' + getId() >> TaskId.of(1) + getHash() >> HashCode.fromString('abcd1234') + } + def handler = Spy(new SeqeraTaskHandler(taskRun, executor)) { + fusionEnabled() >> true + fusionSubmitCli() >> ['bash', '-c', 'echo hello'] + fusionLauncher() >> Mock(nextflow.fusion.FusionScriptLauncher) { + fusionEnv() >> [:] + } + } + + when: + handler.submit() + + then: + capturedTask != null + capturedTask.getResourceLimit() != null + capturedTask.getResourceLimit().memoryMiB == 2048 + capturedTask.getResourceLimit().cpuShares == null + } + + /** + * Creates a test handler with minimal mocked dependencies + */ + private SeqeraTaskHandler createHandler() { + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getConfig() >> Mock(TaskConfig) + } + def executor = Mock(SeqeraExecutor) { + getClient() >> Mock(SchedClient) + } + return new SeqeraTaskHandler(taskRun, executor) + } + + /** + * Creates a test handler with an error set on the task + */ + private SeqeraTaskHandler createHandlerWithError(Throwable error) { + def taskRun = Mock(TaskRun) { + getWorkDir() >> Paths.get('/work/ab/cd1234') + getConfig() >> Mock(TaskConfig) + getError() >> error + } + def executor = Mock(SeqeraExecutor) { + getClient() >> Mock(SchedClient) + } + return new SeqeraTaskHandler(taskRun, executor) + } + + /** + * Creates a test handler with mocks sufficient for getTraceRecord() + */ + private SeqeraTaskHandler createHandlerForTraceTest() { + def executor = Mock(SeqeraExecutor) { + getClient() >> Mock(SchedClient) + getName() >> 'seqera' + } + def processor = Mock(TaskProcessor) { + getName() >> 'test_process' + getExecutor() >> executor + } + def taskConfig = new TaskConfig(attempt: 1, cpus: 1) + def taskRun = Mock(TaskRun) { + getId() >> TaskId.of(1) + getHashLog() >> 'ab/cd1234' + getName() >> 'test_task' + getExitStatus() >> 0 + getProcessor() >> processor + getConfig() >> taskConfig + getContainer() >> 'ubuntu:latest' + getTraceScript() >> 'echo hello' + getScratch() >> null + getWorkDirStr() >> '/work/ab/cd1234' + getWorkDir() >> Paths.get('/work/ab/cd1234') + getEnvironmentStr() >> '' + containerMeta() >> null + } + return new SeqeraTaskHandler(taskRun, executor) + } +} diff --git a/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy b/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy new file mode 100644 index 0000000000..a34f70e804 --- /dev/null +++ b/plugins/nf-seqera/src/test/io/seqera/util/MapperUtilTest.groovy @@ -0,0 +1,550 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.util + +import io.seqera.config.MachineRequirementOpts +import io.seqera.sched.api.schema.v1a1.DiskAllocation +import io.seqera.sched.api.schema.v1a1.EcsCapacityMode +import io.seqera.sched.api.schema.v1a1.PriceModel as SchedPriceModel +import io.seqera.sched.api.schema.v1a1.ProvisioningModel +import nextflow.cloud.types.PriceModel +import nextflow.fusion.FusionConfig +import nextflow.util.MemoryUnit +import spock.lang.Specification + +/** + * Unit tests for SchemaMapper + * + * @author Paolo Di Tommaso + */ +class MapperUtilTest extends Specification { + + def 'should return null for null opts' () { + expect: + SchemaMapperUtil.toMachineRequirement(null) == null + } + + def 'should return null for empty opts' () { + expect: + SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([:])) == null + } + + def 'should map arch only' () { + when: + def result = SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([arch: 'arm64'])) + + then: + result.arch == 'arm64' + result.provisioning == null + result.maxSpotAttempts == null + result.machineTypes == null + } + + def 'should map all fields' () { + when: + def result = SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([ + arch: 'x86_64', + provisioning: 'spotFirst', + maxSpotAttempts: 3, + machineTypes: ['m5', 'c5'] + ])) + + then: + result.arch == 'x86_64' + result.provisioning == ProvisioningModel.SPOT_FIRST + result.maxSpotAttempts == 3 + result.machineTypes == ['m5', 'c5'] + } + + def 'should map provisioning model' () { + expect: + SchemaMapperUtil.toProvisioningModel(null) == null + SchemaMapperUtil.toProvisioningModel('spot') == ProvisioningModel.SPOT + SchemaMapperUtil.toProvisioningModel('ondemand') == ProvisioningModel.ONDEMAND + SchemaMapperUtil.toProvisioningModel('spotFirst') == ProvisioningModel.SPOT_FIRST + } + + def 'should map price model' () { + expect: + SchemaMapperUtil.toPriceModel(null) == null + SchemaMapperUtil.toPriceModel(SchedPriceModel.SPOT) == PriceModel.spot + SchemaMapperUtil.toPriceModel(SchedPriceModel.STANDARD) == PriceModel.standard + } + + // tests for toMachineRequirement with task arch + + def 'should return null when both opts and taskArch are null' () { + expect: + SchemaMapperUtil.toMachineRequirement(null, null) == null + } + + def 'should use taskArch when opts is null' () { + when: + def result = SchemaMapperUtil.toMachineRequirement(null, 'arm64') + + then: + result.arch == 'arm64' + result.provisioning == null + } + + def 'should use taskArch over config arch' () { + when: + def result = SchemaMapperUtil.toMachineRequirement( + new MachineRequirementOpts([arch: 'x86_64', provisioning: 'spot']), + 'arm64' + ) + + then: + result.arch == 'arm64' + result.provisioning == ProvisioningModel.SPOT + } + + def 'should use config arch when taskArch is null' () { + when: + def result = SchemaMapperUtil.toMachineRequirement( + new MachineRequirementOpts([arch: 'x86_64', provisioning: 'spot']), + null + ) + + then: + result.arch == 'x86_64' + result.provisioning == ProvisioningModel.SPOT + } + + def 'should merge config settings with taskArch' () { + when: + def result = SchemaMapperUtil.toMachineRequirement( + new MachineRequirementOpts([ + provisioning: 'spotFirst', + maxSpotAttempts: 3, + machineTypes: ['m5', 'c5'] + ]), + 'arm64' + ) + + then: + result.arch == 'arm64' + result.provisioning == ProvisioningModel.SPOT_FIRST + result.maxSpotAttempts == 3 + result.machineTypes == ['m5', 'c5'] + } + + // tests for disk requirement mapping + + def 'should return null disk requirement for null disk size' () { + expect: + SchemaMapperUtil.toDiskRequirement(null) == null + } + + def 'should return null disk requirement for zero disk size' () { + expect: + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of(0)) == null + } + + def 'should map disk size to disk requirement with defaults' () { + when: + def result = SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB')) + + then: 'disk size is set' + result.sizeGiB == 100 + and: 'allocation defaults to node' + result.allocation == DiskAllocation.NODE + and: 'node allocation does not set EBS options' + result.volumeType == null + result.throughputMiBps == null + result.encrypted == null + result.iops == null + } + + def 'should map disk size in different units' () { + expect: + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('1 TB')).sizeGiB == 1024 + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('50 GB')).sizeGiB == 50 + and: 'defaults to node allocation' + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('1 TB')).allocation == DiskAllocation.NODE + } + + def 'should include disk in machine requirement' () { + when: + def result = SchemaMapperUtil.toMachineRequirement( + new MachineRequirementOpts([arch: 'x86_64']), + null, + MemoryUnit.of('200 GB'), + false + ) + + then: + result.arch == 'x86_64' + result.disk != null + result.disk.sizeGiB == 200 + } + + def 'should return machine requirement with only disk' () { + when: + def result = SchemaMapperUtil.toMachineRequirement(null, null, MemoryUnit.of('100 GB'), false) + + then: + result != null + result.arch == null + result.disk != null + result.disk.sizeGiB == 100 + } + + def 'should return null when no arch, no opts, and no disk' () { + expect: + SchemaMapperUtil.toMachineRequirement(null, null, null, false) == null + } + + // tests for custom disk configuration options + + def 'should throw exception for invalid disk type' () { + given: + def opts = new MachineRequirementOpts([diskAllocation: 'task', diskType: 'local/nvme']) + + when: + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains("Invalid disk type: local/nvme") + e.message.contains("Supported types:") + } + + def 'should use custom disk type from config' () { + given: + def opts = new MachineRequirementOpts([diskAllocation: 'task', diskType: 'ebs/io1', diskIops: 10000]) + + when: + def result = SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + result.sizeGiB == 100 + result.allocation == DiskAllocation.TASK + result.volumeType == 'ebs/io1' + result.iops == 10000 + result.throughputMiBps == null // throughput only for gp3 + } + + def 'should use custom throughput from config' () { + given: + def opts = new MachineRequirementOpts([diskAllocation: 'task', diskThroughputMiBps: 500]) + + when: + def result = SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + result.allocation == DiskAllocation.TASK + result.volumeType == SchemaMapperUtil.DEFAULT_DISK_TYPE + result.throughputMiBps == 500 + } + + def 'should use encryption from config' () { + given: + def opts = new MachineRequirementOpts([diskAllocation: 'task', diskEncrypted: true]) + + when: + def result = SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + result.allocation == DiskAllocation.TASK + result.encrypted == true + } + + def 'should use all disk options from config' () { + given: + def opts = new MachineRequirementOpts([ + diskAllocation: 'task', + diskType: 'ebs/gp3', + diskThroughputMiBps: 600, + diskIops: 8000, + diskEncrypted: true + ]) + + when: + def result = SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('200 GB'), opts) + + then: + result.sizeGiB == 200 + result.allocation == DiskAllocation.TASK + result.volumeType == 'ebs/gp3' + result.throughputMiBps == 600 + result.iops == 8000 + result.encrypted == true + } + + def 'should pass disk options through machine requirement' () { + given: + def opts = new MachineRequirementOpts([ + arch: 'arm64', + diskAllocation: 'task', + diskType: 'ebs/io2', + diskIops: 15000, + diskEncrypted: true + ]) + + when: + def result = SchemaMapperUtil.toMachineRequirement(opts, null, MemoryUnit.of('500 GB'), false) + + then: + result.arch == 'arm64' + result.disk.sizeGiB == 500 + result.disk.allocation == DiskAllocation.TASK + result.disk.volumeType == 'ebs/io2' + result.disk.iops == 15000 + result.disk.encrypted == true + result.disk.throughputMiBps == null // io2 doesn't use throughput + } + + // tests for disk allocation mapping + + def 'should map disk allocation' () { + expect: + SchemaMapperUtil.toDiskAllocation(null) == null + SchemaMapperUtil.toDiskAllocation('task') == DiskAllocation.TASK + SchemaMapperUtil.toDiskAllocation('node') == DiskAllocation.NODE + } + + def 'should throw exception for invalid disk allocation' () { + when: + SchemaMapperUtil.toDiskAllocation('invalid') + + then: + thrown(IllegalArgumentException) + } + + def 'should use task disk allocation from config' () { + given: + def opts = new MachineRequirementOpts([diskAllocation: 'task']) + + when: + def result = SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + result.allocation == DiskAllocation.TASK + } + + def 'should use node disk allocation from config' () { + given: + def opts = new MachineRequirementOpts([diskAllocation: 'node']) + + when: + def result = SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + result.allocation == DiskAllocation.NODE + result.sizeGiB == 100 + result.volumeType == null // node allocation doesn't set EBS options + result.throughputMiBps == null + result.iops == null + result.encrypted == null + } + + def 'should default to node disk allocation when not specified' () { + when: + def result = SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB')) + + then: + result.allocation == DiskAllocation.NODE + } + + def 'should include disk allocation in machine requirement' () { + given: + def opts = new MachineRequirementOpts([ + arch: 'x86_64', + diskAllocation: 'node' + ]) + + when: + def result = SchemaMapperUtil.toMachineRequirement(opts, null, MemoryUnit.of('200 GB'), false) + + then: + result.arch == 'x86_64' + result.disk.sizeGiB == 200 + result.disk.allocation == DiskAllocation.NODE + } + + // tests for node allocation validation + + def 'should throw error when diskType is set with node allocation' () { + given: + def opts = new MachineRequirementOpts([ + diskAllocation: 'node', + diskType: 'ebs/gp3' + ]) + + when: + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains('diskType') + e.message.contains('node') + } + + def 'should throw error when diskIops is set with node allocation' () { + given: + def opts = new MachineRequirementOpts([ + diskAllocation: 'node', + diskIops: 10000 + ]) + + when: + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains('diskIops') + } + + def 'should throw error when diskThroughputMiBps is set with node allocation' () { + given: + def opts = new MachineRequirementOpts([ + diskAllocation: 'node', + diskThroughputMiBps: 500 + ]) + + when: + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains('diskThroughputMiBps') + } + + def 'should throw error when diskEncrypted is set with node allocation' () { + given: + def opts = new MachineRequirementOpts([ + diskAllocation: 'node', + diskEncrypted: true + ]) + + when: + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains('diskEncrypted') + } + + def 'should report all invalid options with node allocation' () { + given: + def opts = new MachineRequirementOpts([ + diskAllocation: 'node', + diskType: 'ebs/io1', + diskIops: 10000, + diskEncrypted: true + ]) + + when: + SchemaMapperUtil.toDiskRequirement(MemoryUnit.of('100 GB'), opts) + + then: + def e = thrown(IllegalArgumentException) + e.message.contains('diskType') + e.message.contains('diskIops') + e.message.contains('diskEncrypted') + } + + // tests for snapshot maxSpotAttempts defaulting + + def 'should return machine requirement with only snapshot enabled' () { + when: + def result = SchemaMapperUtil.toMachineRequirement(null, null, null, true) + + then: + result != null + result.snapshotEnabled == true + result.maxSpotAttempts == FusionConfig.DEFAULT_SNAPSHOT_MAX_SPOT_ATTEMPTS + } + + def 'should use explicit maxSpotAttempts when snapshot enabled' () { + when: + def result = SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([maxSpotAttempts: 2]), null, null, true) + + then: + result.snapshotEnabled == true + result.maxSpotAttempts == 2 + } + + def 'should not default maxSpotAttempts when snapshot disabled' () { + when: + def result = SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([arch: 'x86_64']), null, null, false) + + then: + result.snapshotEnabled == null + result.maxSpotAttempts == null + } + + // tests for capacity mode mapping + + def 'should map capacity mode' () { + expect: + SchemaMapperUtil.toEcsCapacityMode(null) == null + SchemaMapperUtil.toEcsCapacityMode('managed') == EcsCapacityMode.MANAGED + SchemaMapperUtil.toEcsCapacityMode('asg') == EcsCapacityMode.ASG + } + + def 'should throw exception for invalid capacity mode' () { + when: + SchemaMapperUtil.toEcsCapacityMode('invalid') + + then: + thrown(IllegalArgumentException) + } + + def 'should include capacity mode in machine requirement' () { + when: + def result = SchemaMapperUtil.toMachineRequirement(new MachineRequirementOpts([capacityMode: 'asg'])) + + then: + result != null + result.capacityMode == EcsCapacityMode.ASG + } + + def 'should include capacity mode in machine requirement with task arch' () { + when: + def result = SchemaMapperUtil.toMachineRequirement( + new MachineRequirementOpts([capacityMode: 'managed', arch: 'arm64']), + null, + null, + false + ) + + then: + result.arch == 'arm64' + result.capacityMode == EcsCapacityMode.MANAGED + } + + def 'should combine snapshot with other machine requirement settings' () { + when: + def result = SchemaMapperUtil.toMachineRequirement( + new MachineRequirementOpts([arch: 'arm64', provisioning: 'spot']), + null, + MemoryUnit.of('100 GB'), + true + ) + + then: + result.arch == 'arm64' + result.provisioning == ProvisioningModel.SPOT + result.disk.sizeGiB == 100 + result.snapshotEnabled == true + result.maxSpotAttempts == FusionConfig.DEFAULT_SNAPSHOT_MAX_SPOT_ATTEMPTS + } + +} diff --git a/plugins/nf-tower/VERSION b/plugins/nf-tower/VERSION index 3989355915..3500250a4b 100644 --- a/plugins/nf-tower/VERSION +++ b/plugins/nf-tower/VERSION @@ -1 +1 @@ -1.20.0 +1.21.0 diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy index 42393edafd..35caad2ee7 100644 --- a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy @@ -292,6 +292,11 @@ class TowerClient implements TraceObserverV2 { this.workflowId = ret.workflowId if( !workflowId ) throw new AbortOperationException("Invalid Seqera Platform API response - Missing workflow Id") + log.debug "Platform workflow id: $workflowId; workflow url: ${ret.watchUrl}" + session.workflowMetadata.platform.workflowId = workflowId + // note: `watchUrl` in the create response requires Platform 26.01 or later + this.watchUrl = ret.watchUrl as String + session.workflowMetadata.platform.workflowUrl = watchUrl if( ret.message ) log.warn(ret.message.toString()) @@ -382,7 +387,8 @@ class TowerClient implements TraceObserverV2 { } final payload = parseTowerResponse(resp) - this.watchUrl = payload.watchUrl + this.watchUrl ?= payload.watchUrl + session.workflowMetadata.platform.workflowUrl ?= watchUrl this.sender = Threads.start('Tower-thread', this.&sendTasks0) final msg = "Monitor the execution with Seqera Platform using this URL: ${watchUrl}" log.info(LoggerHelper.STICKY, msg) diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerClientTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerClientTest.groovy index 18558595da..93e68ec87f 100644 --- a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerClientTest.groovy +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerClientTest.groovy @@ -30,6 +30,7 @@ import nextflow.cloud.types.PriceModel import nextflow.container.DockerConfig import nextflow.container.resolver.ContainerMeta import nextflow.exception.AbortOperationException +import nextflow.script.PlatformMetadata import nextflow.script.ScriptBinding import nextflow.script.WorkflowMetadata import nextflow.trace.TraceRecord @@ -378,9 +379,11 @@ class TowerClientTest extends Specification { def 'should post create request' () { given: def uuid = UUID.randomUUID() + def platform = new PlatformMetadata() def meta = Mock(WorkflowMetadata) { getProjectName() >> 'the-project-name' getRepository() >> 'git://repo.com/foo' + getPlatform() >> platform } def session = Mock(Session) { getUniqueId() >> uuid @@ -396,14 +399,41 @@ class TowerClientTest extends Specification { then: 1 * client.getAccessToken() >> 'secret' 1 * client.makeCreateReq(session) >> [runName: 'foo'] - 1 * client.sendHttpMessage('https://api.cloud.seqera.io/trace/create', [runName: 'foo'], 'POST') >> new TowerClient.Response(200, '{"workflowId":"xyz123"}') + 1 * client.sendHttpMessage('https://api.cloud.seqera.io/trace/create', [runName: 'foo'], 'POST') >> new TowerClient.Response(200, '{"workflowId":"xyz123","watchUrl":"https://cloud.seqera.io/watch/xyz123"}') and: client.runName == 'foo_bar' client.runId == uuid.toString() and: client.workflowId == 'xyz123' + client.@watchUrl == 'https://cloud.seqera.io/watch/xyz123' !client.towerLaunch + and: + platform.workflowId == 'xyz123' + platform.workflowUrl == 'https://cloud.seqera.io/watch/xyz123' + + } + def 'should set workflowUrl on platform metadata during onFlowBegin' () { + given: + def platform = new PlatformMetadata() + def meta = Mock(WorkflowMetadata) { + getPlatform() >> platform + } + def session = Mock(Session) { + getWorkflowMetadata() >> meta + } + def config = new TowerConfig([:], [:]) + def client = Spy(new TowerClient(session, config)) + client.@workflowId = 'abc123' + + when: + client.onFlowBegin() + then: + 1 * client.makeBeginReq(session) >> [foo: 'bar'] + 1 * client.sendHttpMessage(_, [foo: 'bar'], 'PUT') >> new TowerClient.Response(200, '{"watchUrl":"https://cloud.seqera.io/watch/abc123"}') + and: + client.@watchUrl == 'https://cloud.seqera.io/watch/abc123' + platform.workflowUrl == 'https://cloud.seqera.io/watch/abc123' } def 'should get trace endpoint' () { diff --git a/settings.gradle b/settings.gradle index d8eb7aea65..1b01869d38 100644 --- a/settings.gradle +++ b/settings.gradle @@ -47,3 +47,6 @@ include 'plugins:nf-codecommit' include 'plugins:nf-wave' include 'plugins:nf-cloudcache' include 'plugins:nf-k8s' +include 'plugins:nf-seqera' + +//includeBuild '../sched' From 153d9371e39c5355ed33b425e88a96020f8105a0 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 27 Feb 2026 23:09:15 +0100 Subject: [PATCH 46/75] Revert version 26.01.1-edge Signed-off-by: Paolo Di Tommaso --- VERSION | 2 +- plugins/nf-seqera/build.gradle | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/VERSION b/VERSION index 729e722712..5f38a2f391 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -26.02.0-edge +26.01.1-edge diff --git a/plugins/nf-seqera/build.gradle b/plugins/nf-seqera/build.gradle index 724cd6ff31..21c99982f7 100644 --- a/plugins/nf-seqera/build.gradle +++ b/plugins/nf-seqera/build.gradle @@ -3,7 +3,7 @@ * * This Source Code Form is subject to the terms of the Mozilla Public * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. + * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * This Source Code Form is "Incompatible With Secondary Licenses", as * defined by the Mozilla Public License, v. 2.0. @@ -15,7 +15,7 @@ plugins { } nextflowPlugin { - nextflowVersion = '26.02.0-edge' + nextflowVersion = '26.01.1-edge' provider = "${nextflowPluginProvider}" description = 'Integrates with Seqera Platform for comprehensive workflow monitoring, resource tracking, and cache management capabilities' From 5c670904d583d429fa647c009d2ea84ace4ddd32 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 27 Feb 2026 23:22:20 +0100 Subject: [PATCH 47/75] Bump copyright 2026 Signed-off-by: Paolo Di Tommaso --- build.gradle | 14 +- compile.sh | 16 + config/codenarc/codenarc.groovy | 590 +++++++++--------- config/codenarc/codenarc.xml | 4 +- console.sh | 18 +- docker/entry.sh | 21 +- gradle/codenarc.groovy | 34 + launch.sh | 21 +- modules/nextflow/build.gradle | 16 + .../src/main/groovy/nextflow/Channel.groovy | 2 +- .../src/main/groovy/nextflow/NF.groovy | 4 +- .../src/main/groovy/nextflow/Nextflow.groovy | 4 +- .../main/groovy/nextflow/NextflowMeta.groovy | 18 +- .../src/main/groovy/nextflow/Session.groovy | 2 +- .../groovy/nextflow/ast/ASTHelpers.groovy | 2 +- .../groovy/nextflow/ast/NextflowDSL.groovy | 2 +- .../nextflow/ast/NextflowDSLImpl.groovy | 6 +- .../groovy/nextflow/ast/NextflowXform.groovy | 2 +- .../main/groovy/nextflow/ast/OpXform.groovy | 2 +- .../groovy/nextflow/ast/TaskCmdXform.groovy | 2 +- .../nextflow/ast/TaskTemplateVarsXform.groovy | 2 +- .../nextflow/ast/TaskTemplateVisitor.groovy | 2 +- .../nextflow/ast/VariableVisitor.groovy | 2 +- .../main/groovy/nextflow/cache/CacheDB.groovy | 3 +- .../groovy/nextflow/cache/CacheFactory.groovy | 5 +- .../groovy/nextflow/cache/CacheStore.groovy | 3 +- .../nextflow/cache/DefaultCacheFactory.groovy | 3 +- .../nextflow/cache/DefaultCacheStore.groovy | 3 +- .../main/groovy/nextflow/cli/CacheBase.groovy | 2 +- .../groovy/nextflow/cli/CliOptions.groovy | 2 +- .../main/groovy/nextflow/cli/CmdAuth.groovy | 2 +- .../main/groovy/nextflow/cli/CmdBase.groovy | 2 +- .../main/groovy/nextflow/cli/CmdClean.groovy | 4 +- .../main/groovy/nextflow/cli/CmdClone.groovy | 2 +- .../main/groovy/nextflow/cli/CmdConfig.groovy | 2 +- .../groovy/nextflow/cli/CmdConsole.groovy | 2 +- .../main/groovy/nextflow/cli/CmdDrop.groovy | 2 +- .../src/main/groovy/nextflow/cli/CmdFs.groovy | 4 +- .../main/groovy/nextflow/cli/CmdHelp.groovy | 2 +- .../main/groovy/nextflow/cli/CmdHelper.groovy | 2 +- .../main/groovy/nextflow/cli/CmdInfo.groovy | 2 +- .../groovy/nextflow/cli/CmdInspect.groovy | 5 +- .../groovy/nextflow/cli/CmdKubeRun.groovy | 4 +- .../main/groovy/nextflow/cli/CmdLaunch.groovy | 2 +- .../groovy/nextflow/cli/CmdLineage.groovy | 3 +- .../main/groovy/nextflow/cli/CmdLint.groovy | 2 +- .../main/groovy/nextflow/cli/CmdList.groovy | 2 +- .../main/groovy/nextflow/cli/CmdLog.groovy | 2 +- .../main/groovy/nextflow/cli/CmdNode.groovy | 2 +- .../main/groovy/nextflow/cli/CmdPlugin.groovy | 15 +- .../main/groovy/nextflow/cli/CmdPull.groovy | 2 +- .../main/groovy/nextflow/cli/CmdRun.groovy | 10 +- .../main/groovy/nextflow/cli/CmdSecret.groovy | 3 +- .../groovy/nextflow/cli/CmdSelfUpdate.groovy | 2 +- .../main/groovy/nextflow/cli/CmdView.groovy | 2 +- .../groovy/nextflow/cli/HubOptions.groovy | 2 +- .../main/groovy/nextflow/cli/Launcher.groovy | 2 +- .../nextflow/cli/PluginAbstractExec.groovy | 3 +- .../nextflow/cli/PluginExecAware.groovy | 3 +- .../groovy/nextflow/cli/UsageAware.groovy | 2 +- .../CloudSpotTerminationException.groovy | 2 +- .../cloud/CloudTransferOptions.groovy | 16 + .../nextflow/cloud/types/CloudInstance.groovy | 2 +- .../cloud/types/CloudInstanceStatus.groovy | 2 +- .../cloud/types/CloudInstanceType.groovy | 2 +- .../cloud/types/CloudMachineInfo.groovy | 2 +- .../cloud/types/CloudSpotPrice.groovy | 2 +- .../nextflow/cloud/types/PriceModel.groovy | 2 +- .../groovy/nextflow/conda/CondaCache.groovy | 10 +- .../groovy/nextflow/conda/CondaConfig.groovy | 4 +- .../nextflow/config/CascadingConfig.groovy | 2 +- .../nextflow/config/ConfigBuilder.groovy | 6 +- .../config/ConfigClosurePlaceholder.groovy | 2 +- .../groovy/nextflow/config/ConfigField.groovy | 2 +- .../groovy/nextflow/config/ConfigMap.groovy | 2 +- .../nextflow/config/ConfigParser.groovy | 2 +- .../config/ConfigParserFactory.groovy | 3 +- .../nextflow/config/ConfigValidator.groovy | 2 +- .../groovy/nextflow/config/Manifest.groovy | 2 +- .../nextflow/config/StripSecretsXform.groovy | 5 +- .../nextflow/config/WorkflowConfig.groovy | 4 +- .../config/parser/v1/ConfigBase.groovy | 2 +- .../config/parser/v1/ConfigParserV1.groovy | 18 +- .../config/parser/v1/ConfigTransform.groovy | 2 +- .../parser/v1/ConfigTransformImpl.groovy | 2 +- .../config/parser/v1/PluginsDsl.groovy | 2 +- .../parser/v2/ClosureToStringVisitor.java | 2 +- .../config/parser/v2/ConfigCompiler.java | 2 +- .../config/parser/v2/ConfigDsl.groovy | 2 +- .../config/parser/v2/ConfigParserV2.groovy | 2 +- .../config/spec/MarkdownRenderer.groovy | 2 +- .../container/ApptainerBuilder.groovy | 3 +- .../nextflow/container/ApptainerCache.groovy | 3 +- .../nextflow/container/ApptainerConfig.groovy | 2 +- .../container/CharliecloudBuilder.groovy | 12 +- .../container/CharliecloudCache.groovy | 18 +- .../container/CharliecloudConfig.groovy | 2 +- .../container/ContainerBuilder.groovy | 2 +- .../nextflow/container/ContainerConfig.groovy | 2 +- .../container/ContainerHandler.groovy | 4 +- .../nextflow/container/ContainerHelper.groovy | 2 +- .../container/ContainerNameValidator.groovy | 4 +- .../nextflow/container/DockerBuilder.groovy | 2 +- .../nextflow/container/DockerConfig.groovy | 2 +- .../nextflow/container/PodmanBuilder.groovy | 4 +- .../nextflow/container/PodmanConfig.groovy | 2 +- .../nextflow/container/SarusBuilder.groovy | 2 +- .../nextflow/container/SarusConfig.groovy | 2 +- .../nextflow/container/ShifterBuilder.groovy | 2 +- .../nextflow/container/ShifterConfig.groovy | 2 +- .../container/SingularityBuilder.groovy | 2 +- .../container/SingularityCache.groovy | 2 +- .../container/SingularityConfig.groovy | 2 +- .../inspect/ContainerInspectMode.groovy | 3 +- .../inspect/ContainersInspector.groovy | 3 +- .../container/resolver/ContainerInfo.groovy | 3 +- .../container/resolver/ContainerMeta.groovy | 3 +- .../resolver/ContainerResolver.groovy | 3 +- .../resolver/ContainerResolverProvider.groovy | 3 +- .../resolver/DefaultContainerResolver.groovy | 3 +- .../nextflow/daemon/DaemonLauncher.groovy | 2 +- .../src/main/groovy/nextflow/dag/DAG.groovy | 2 +- .../groovy/nextflow/dag/DagRenderer.groovy | 2 +- .../groovy/nextflow/dag/DotRenderer.groovy | 2 +- .../groovy/nextflow/dag/GexfRenderer.groovy | 6 +- .../nextflow/dag/GraphVizRenderer.groovy | 2 +- .../nextflow/dag/MermaidHtmlRenderer.groovy | 2 +- .../nextflow/dag/MermaidRenderer.groovy | 4 +- .../dag/MultipleInputChannelException.groovy | 2 +- .../dag/MultipleOutputChannelException.groovy | 2 +- .../groovy/nextflow/dag/NodeMarker.groovy | 2 +- .../nextflow/datasource/SraExplorer.groovy | 4 +- .../nextflow/datasource/SraRetryConfig.groovy | 3 +- .../exception/AbortRunException.groovy | 2 +- .../exception/AbortSignalException.groovy | 2 +- .../AmbiguousPipelineNameException.groovy | 4 +- .../exception/ConfigParseException.groovy | 2 +- .../DuplicateChannelNameException.groovy | 4 +- .../DuplicateModuleFunctionException.groovy | 5 +- .../DuplicateModuleIncludeException.groovy | 16 + .../DuplicateProcessInvocation.groovy | 4 +- .../exception/FailedGuardException.groovy | 2 +- .../HttpResponseLengthExceedException.groovy | 2 +- .../exception/IllegalArityException.groovy | 3 +- .../exception/IllegalConfigException.groovy | 2 +- .../IllegalDirectiveException.groovy | 2 +- .../exception/IllegalFileException.groovy | 2 +- .../IllegalInvocationException.groovy | 16 + .../exception/IllegalModulePath.groovy | 4 +- .../exception/K8sOutOfCpuException.groovy | 16 + .../exception/K8sOutOfMemoryException.groovy | 16 + .../MissingCredentialsException.groovy | 16 + .../exception/MissingFileException.groovy | 2 +- .../exception/MissingLibraryException.groovy | 2 +- .../MissingModuleComponentException.groovy | 18 +- .../exception/MissingProcessException.groovy | 18 +- .../exception/MissingValueException.groovy | 2 +- .../exception/NodeTerminationException.groovy | 3 +- .../exception/PlainExceptionMessage.groovy | 5 +- .../exception/ProcessEvalException.groovy | 5 +- .../exception/ProcessException.groovy | 2 +- .../exception/ProcessFailedException.groovy | 2 +- .../ProcessNonZeroExitStatusException.groovy | 3 +- .../ProcessRetryableException.groovy | 16 + .../exception/ProcessStageException.groovy | 4 +- .../exception/ProcessSubmitException.groovy | 2 +- .../ProcessSubmitTimeoutException.groovy | 2 +- .../exception/ProcessTemplateException.groovy | 2 +- .../ProcessUnrecoverableException.groovy | 2 +- .../RateLimitExceededException.groovy | 3 +- .../exception/ReportWarningException.groovy | 3 +- .../ScriptCompilationException.groovy | 4 +- .../exception/ScriptRuntimeException.groovy | 2 +- .../exception/ShowOnlyExceptionMessage.groovy | 2 +- .../StopSplitIterationException.groovy | 2 +- .../exception/UnexpectedException.groovy | 16 + .../WorkflowScriptErrorException.groovy | 5 +- .../executor/AbstractGridExecutor.groovy | 2 +- .../nextflow/executor/BashFunLib.groovy | 9 +- .../executor/BashTemplateEngine.groovy | 2 +- .../executor/BashWrapperBuilder.groovy | 8 +- .../nextflow/executor/BatchCleanup.groovy | 2 +- .../nextflow/executor/BridgeExecutor.groovy | 27 +- .../executor/CachedTaskHandler.groovy | 2 +- .../nextflow/executor/CondorExecutor.groovy | 2 +- .../nextflow/executor/CrgExecutor.groovy | 2 +- .../groovy/nextflow/executor/Executor.groovy | 2 +- .../nextflow/executor/ExecutorConfig.groovy | 4 +- .../nextflow/executor/ExecutorFactory.groovy | 4 +- .../executor/ExecutorRetryConfig.groovy | 2 +- .../nextflow/executor/FluxExecutor.groovy | 2 +- .../nextflow/executor/GridTaskHandler.groovy | 2 +- .../executor/HyperQueueExecutor.groovy | 3 +- .../nextflow/executor/LsfExecutor.groovy | 4 +- .../nextflow/executor/MoabExecutor.groovy | 2 +- .../nextflow/executor/NopeExecutor.groovy | 2 +- .../nextflow/executor/NqsiiExecutor.groovy | 4 +- .../nextflow/executor/OarExecutor.groovy | 6 +- .../nextflow/executor/PbsExecutor.groovy | 2 +- .../nextflow/executor/PbsProExecutor.groovy | 12 +- .../executor/ScriptFileCopyStrategy.groovy | 2 +- .../nextflow/executor/SgeExecutor.groovy | 2 +- .../executor/SimpleFileCopyStrategy.groovy | 8 +- .../nextflow/executor/SlurmExecutor.groovy | 2 +- .../executor/StoredTaskHandler.groovy | 18 +- .../executor/SupportedScriptTypes.groovy | 2 +- .../executor/TaskArrayExecutor.groovy | 2 +- .../nextflow/executor/TcsExecutor.groovy | 16 + .../executor/local/LocalExecutor.groovy | 3 +- .../executor/local/LocalTaskHandler.groovy | 5 +- .../executor/local/NativeTaskHandler.groovy | 3 +- .../executor/res/AcceleratorResource.groovy | 4 +- .../nextflow/executor/res/DiskResource.groovy | 2 +- .../groovy/nextflow/extension/BranchOp.groovy | 2 +- .../groovy/nextflow/extension/BufferOp.groovy | 4 +- .../main/groovy/nextflow/extension/CH.groovy | 18 +- .../extension/CaptureProperties.groovy | 2 +- .../nextflow/extension/ChannelEx.groovy | 2 +- .../nextflow/extension/CollectFileOp.groovy | 2 +- .../nextflow/extension/CollectOp.groovy | 2 +- .../nextflow/extension/CombineOp.groovy | 2 +- .../groovy/nextflow/extension/ConcatOp.groovy | 2 +- .../groovy/nextflow/extension/CrossOp.groovy | 2 +- .../nextflow/extension/DataflowHelper.groovy | 2 +- .../extension/DefaultMergeClosure.groovy | 2 +- .../nextflow/extension/DumpHelper.groovy | 3 +- .../groovy/nextflow/extension/DumpOp.groovy | 2 +- .../groovy/nextflow/extension/GroupKey.groovy | 2 +- .../nextflow/extension/GroupTupleOp.groovy | 2 +- .../groovy/nextflow/extension/IntoOp.groovy | 2 +- .../groovy/nextflow/extension/JoinOp.groovy | 4 +- .../groovy/nextflow/extension/KeyPair.groovy | 2 +- .../nextflow/extension/LinExtension.groovy | 2 +- .../groovy/nextflow/extension/MapOp.groovy | 2 +- .../groovy/nextflow/extension/MergeOp.groovy | 3 +- .../groovy/nextflow/extension/MixOp.groovy | 3 +- .../nextflow/extension/MultiMapOp.groovy | 2 +- .../groovy/nextflow/extension/OpCall.groovy | 18 +- .../nextflow/extension/OperatorImpl.groovy | 4 +- .../groovy/nextflow/extension/PhaseOp.groovy | 2 +- .../nextflow/extension/PublishOp.groovy | 2 +- .../nextflow/extension/RandomSampleOp.groovy | 2 +- .../groovy/nextflow/extension/SplitOp.groovy | 2 +- .../extension/SplitterMergeClosure.groovy | 2 +- .../groovy/nextflow/extension/TakeOp.groovy | 3 +- .../groovy/nextflow/extension/TapOp.groovy | 2 +- .../groovy/nextflow/extension/ToListOp.groovy | 2 +- .../nextflow/extension/TransposeOp.groovy | 2 +- .../nextflow/extension/UntilManyOp.groovy | 3 +- .../groovy/nextflow/extension/UntilOp.groovy | 5 +- .../groovy/nextflow/file/DirListener.groovy | 3 +- .../groovy/nextflow/file/DirWatcher.groovy | 2 +- .../groovy/nextflow/file/DirWatcherV2.groovy | 3 +- .../groovy/nextflow/file/FileCollector.groovy | 2 +- .../groovy/nextflow/file/FilePorter.groovy | 2 +- .../nextflow/file/LogicalDataPath.groovy | 3 +- .../groovy/nextflow/file/PathVisitor.groovy | 2 +- .../nextflow/file/SequentialFileStore.groovy | 2 +- .../nextflow/file/SimpleFileCollector.groovy | 2 +- .../groovy/nextflow/file/SlurperEx.groovy | 2 +- .../nextflow/file/SortFileCollector.groovy | 2 +- .../nextflow/fusion/FusionAwareTask.groovy | 3 +- .../nextflow/fusion/FusionConfig.groovy | 3 +- .../groovy/nextflow/fusion/FusionEnv.groovy | 3 +- .../nextflow/fusion/FusionEnvProvider.groovy | 3 +- .../nextflow/fusion/FusionHelper.groovy | 3 +- .../fusion/FusionScriptLauncher.groovy | 5 +- .../groovy/nextflow/fusion/FusionToken.groovy | 3 +- .../nextflow/fusion/FusionTokenDefault.groovy | 3 +- .../groovy/nextflow/mail/Attachment.groovy | 2 +- .../nextflow/mail/BaseMailProvider.groovy | 3 +- .../nextflow/mail/JavaMailProvider.groovy | 3 +- .../src/main/groovy/nextflow/mail/Mail.groovy | 2 +- .../groovy/nextflow/mail/MailConfig.groovy | 4 +- .../groovy/nextflow/mail/MailProvider.groovy | 3 +- .../main/groovy/nextflow/mail/Mailer.groovy | 2 +- .../groovy/nextflow/mail/Notification.groovy | 2 +- .../nextflow/mail/SendMailProvider.groovy | 3 +- .../nextflow/mail/SimpleMailProvider.groovy | 3 +- .../nextflow/platform/PlatformHelper.groovy | 16 + .../extension/ChannelFactoryInstance.groovy | 3 +- .../nextflow/plugin/extension/Factory.groovy | 3 +- .../nextflow/plugin/extension/Function.groovy | 3 +- .../nextflow/plugin/extension/Operator.groovy | 3 +- .../extension/PluginExtensionMethod.groovy | 3 +- .../extension/PluginExtensionPoint.groovy | 3 +- .../extension/PluginExtensionProvider.groovy | 5 +- .../nextflow/plugin/spec/ConfigSpec.groovy | 2 +- .../nextflow/plugin/spec/FunctionSpec.groovy | 2 +- .../nextflow/plugin/spec/PluginSpec.groovy | 2 +- .../plugin/spec/PluginSpecWriter.groovy | 2 +- .../nextflow/processor/Architecture.groovy | 12 +- .../nextflow/processor/BatchContext.groovy | 2 +- .../nextflow/processor/BatchHandler.groovy | 2 +- .../nextflow/processor/ErrorStrategy.groovy | 2 +- .../nextflow/processor/ForwardClosure.groovy | 2 +- .../processor/InvokeTaskAdapter.groovy | 2 +- .../processor/LocalPollingMonitor.groovy | 2 +- .../processor/ParallelPollingMonitor.groovy | 6 +- .../nextflow/processor/PublishDir.groovy | 4 +- .../groovy/nextflow/processor/StateObj.groovy | 2 +- .../processor/TaskArrayCollector.groovy | 4 +- .../nextflow/processor/TaskArrayRun.groovy | 2 +- .../groovy/nextflow/processor/TaskBean.groovy | 2 +- .../nextflow/processor/TaskConfig.groovy | 4 +- .../nextflow/processor/TaskContext.groovy | 2 +- .../nextflow/processor/TaskEntry.groovy | 2 +- .../processor/TaskEnvCollector.groovy | 2 +- .../processor/TaskErrorFormatter.groovy | 2 +- .../nextflow/processor/TaskFault.groovy | 2 +- .../processor/TaskFileCollector.groovy | 2 +- .../nextflow/processor/TaskHandler.groovy | 6 +- .../nextflow/processor/TaskHasher.groovy | 2 +- .../groovy/nextflow/processor/TaskId.groovy | 2 +- .../processor/TaskInputResolver.groovy | 6 +- .../nextflow/processor/TaskMonitor.groovy | 2 +- .../processor/TaskOutputResolver.groovy | 2 +- .../groovy/nextflow/processor/TaskPath.groovy | 2 +- .../processor/TaskPollingMonitor.groovy | 12 +- .../nextflow/processor/TaskProcessor.groovy | 6 +- .../groovy/nextflow/processor/TaskRun.groovy | 4 +- .../nextflow/processor/TaskStartParams.groovy | 3 +- .../nextflow/processor/TaskStatus.groovy | 2 +- .../processor/TaskTemplateEngine.groovy | 20 +- .../tip/DefaultTaskTipProvider.groovy | 3 +- .../processor/tip/TaskTipProvider.groovy | 3 +- .../scm/AbstractRepositoryStrategy.groovy | 2 +- .../groovy/nextflow/scm/AssetManager.groovy | 2 +- .../scm/AzureRepositoryProvider.groovy | 38 +- .../scm/BitbucketRepositoryProvider.groovy | 24 +- .../BitbucketServerRepositoryProvider.groovy | 4 +- .../nextflow/scm/GitReferenceHelper.groovy | 2 +- .../main/groovy/nextflow/scm/GitUrl.groovy | 2 +- .../scm/GiteaRepositoryProvider.groovy | 38 +- .../scm/GithubRepositoryProvider.groovy | 42 +- .../scm/GitlabRepositoryProvider.groovy | 26 +- .../scm/LegacyRepositoryStrategy.groovy | 2 +- .../scm/LocalRepositoryProvider.groovy | 8 +- .../MultiRevisionRepositoryStrategy.groovy | 2 +- .../groovy/nextflow/scm/ProviderConfig.groovy | 2 +- .../groovy/nextflow/scm/ProviderPath.groovy | 2 +- .../nextflow/scm/RepositoryFactory.groovy | 5 +- .../nextflow/scm/RepositoryProvider.groovy | 18 +- .../nextflow/scm/RepositoryStrategy.groovy | 2 +- .../groovy/nextflow/script/BaseScript.groovy | 2 +- .../nextflow/script/BaseScriptConsts.groovy | 3 +- .../groovy/nextflow/script/BindableDef.groovy | 2 +- .../groovy/nextflow/script/BodyDef.groovy | 2 +- .../nextflow/script/ChainableDef.groovy | 4 +- .../groovy/nextflow/script/ChannelOut.groovy | 2 +- .../nextflow/script/ComponentDef.groovy | 2 +- .../nextflow/script/CompositeDef.groovy | 2 +- .../nextflow/script/ExecutionContext.groovy | 2 +- .../nextflow/script/ExecutionStack.groovy | 2 +- .../groovy/nextflow/script/FunctionDef.groovy | 2 +- .../nextflow/script/FusionMetadata.groovy | 5 +- .../groovy/nextflow/script/IncludeDef.groovy | 2 +- .../groovy/nextflow/script/IterableDef.groovy | 7 +- .../groovy/nextflow/script/OutputDef.groovy | 2 +- .../groovy/nextflow/script/OutputDsl.groovy | 2 +- .../groovy/nextflow/script/ParamsDsl.groovy | 2 +- .../nextflow/script/PlatformMetadata.groovy | 3 +- .../nextflow/script/ProcessConfig.groovy | 2 +- .../nextflow/script/ProcessConfigV1.groovy | 2 +- .../nextflow/script/ProcessConfigV2.groovy | 2 +- .../groovy/nextflow/script/ProcessDef.groovy | 4 +- .../script/ProcessEntryHandler.groovy | 4 +- .../nextflow/script/ProcessFactory.groovy | 2 +- .../nextflow/script/ScriptBinding.groovy | 4 +- .../groovy/nextflow/script/ScriptFile.groovy | 4 +- .../nextflow/script/ScriptLoader.groovy | 2 +- .../script/ScriptLoaderFactory.groovy | 3 +- .../groovy/nextflow/script/ScriptMeta.groovy | 8 +- .../nextflow/script/ScriptRunner.groovy | 2 +- .../nextflow/script/ScriptTokens.groovy | 2 +- .../groovy/nextflow/script/ScriptType.groovy | 2 +- .../groovy/nextflow/script/TaskClosure.java | 2 +- .../nextflow/script/WaveMetadata.groovy | 5 +- .../nextflow/script/WorkflowBinding.groovy | 4 +- .../groovy/nextflow/script/WorkflowDef.groovy | 2 +- .../nextflow/script/WorkflowMetadata.groovy | 2 +- .../nextflow/script/WorkflowNotifier.groovy | 2 +- .../script/bundle/ResourcesBundle.groovy | 5 +- .../nextflow/script/dsl/ProcessBuilder.groovy | 2 +- .../script/dsl/ProcessConfigBuilder.groovy | 2 +- .../nextflow/script/dsl/ProcessDslV1.groovy | 2 +- .../nextflow/script/dsl/ProcessDslV2.groovy | 2 +- .../nextflow/script/params/ArityParam.groovy | 2 +- .../nextflow/script/params/BaseInParam.groovy | 2 +- .../script/params/BaseOutParam.groovy | 2 +- .../nextflow/script/params/BaseParam.groovy | 2 +- .../script/params/CmdEvalParam.groovy | 3 +- .../script/params/DefaultInParam.groovy | 2 +- .../script/params/DefaultOutParam.groovy | 2 +- .../nextflow/script/params/EachInParam.groovy | 4 +- .../nextflow/script/params/EnvInParam.groovy | 2 +- .../nextflow/script/params/EnvOutParam.groovy | 2 +- .../nextflow/script/params/FileInParam.groovy | 2 +- .../script/params/FileOutParam.groovy | 2 +- .../nextflow/script/params/InParam.groovy | 2 +- .../nextflow/script/params/InputsList.groovy | 2 +- .../script/params/MissingParam.groovy | 2 +- .../script/params/OptionalParam.groovy | 2 +- .../nextflow/script/params/OutParam.groovy | 2 +- .../nextflow/script/params/OutputsList.groovy | 2 +- .../script/params/PathQualifier.groovy | 4 +- .../nextflow/script/params/StdInParam.groovy | 2 +- .../nextflow/script/params/StdOutParam.groovy | 2 +- .../script/params/TupleInParam.groovy | 4 +- .../script/params/TupleOutParam.groovy | 2 +- .../script/params/ValueInParam.groovy | 2 +- .../script/params/ValueOutParam.groovy | 2 +- .../script/params/v2/ProcessFileInput.groovy | 2 +- .../script/params/v2/ProcessFileOutput.groovy | 2 +- .../script/params/v2/ProcessInput.groovy | 2 +- .../script/params/v2/ProcessInputsDef.groovy | 4 +- .../script/params/v2/ProcessOutput.groovy | 4 +- .../script/params/v2/ProcessOutputsDef.groovy | 2 +- .../script/params/v2/ProcessTopic.groovy | 4 +- .../script/params/v2/ProcessTupleInput.groovy | 2 +- .../script/parser/v1/ScriptLoaderV1.groovy | 2 +- .../script/parser/v2/ErrorListener.groovy | 2 +- .../script/parser/v2/ScriptCompiler.java | 4 +- .../script/parser/v2/ScriptLoaderV2.groovy | 2 +- .../secret/EmptySecretProvider.groovy | 27 +- .../secret/LocalSecretsProvider.groovy | 5 +- .../secret/MissingSecretException.groovy | 13 +- .../nextflow/secret/NullProvider.groovy | 3 +- .../main/groovy/nextflow/secret/Secret.groovy | 3 +- .../groovy/nextflow/secret/SecretImpl.groovy | 3 +- .../nextflow/secret/SecretsHelper.groovy | 3 +- .../nextflow/secret/SecretsLoader.groovy | 3 +- .../nextflow/secret/SecretsProvider.groovy | 3 +- .../main/groovy/nextflow/sort/BigSort.java | 2 +- .../groovy/nextflow/sort/LevelDbSort.java | 2 +- .../groovy/nextflow/spack/SpackCache.groovy | 2 +- .../groovy/nextflow/spack/SpackConfig.groovy | 4 +- .../splitter/AbstractBinarySplitter.groovy | 2 +- .../nextflow/splitter/AbstractSplitter.groovy | 2 +- .../splitter/AbstractTextSplitter.groovy | 2 +- .../nextflow/splitter/BytesSplitter.groovy | 2 +- .../splitter/CacheableCollector.groovy | 2 +- .../splitter/CharSequenceCollector.groovy | 2 +- .../splitter/CollectorStrategy.groovy | 2 +- .../nextflow/splitter/CsvSplitter.groovy | 2 +- .../nextflow/splitter/EntryCounter.groovy | 2 +- .../nextflow/splitter/FastaSplitter.groovy | 2 +- .../nextflow/splitter/FastqSplitter.groovy | 2 +- .../nextflow/splitter/HeaderCollector.groovy | 2 +- .../nextflow/splitter/JsonSplitter.groovy | 2 +- .../splitter/ObjectListCollector.groovy | 2 +- .../nextflow/splitter/SplitterEx.groovy | 2 +- .../nextflow/splitter/SplitterFactory.groovy | 2 +- .../nextflow/splitter/SplitterStrategy.groovy | 2 +- .../nextflow/splitter/StringSplitter.groovy | 2 +- .../splitter/TextFileCollector.groovy | 2 +- .../nextflow/splitter/TextSplitter.groovy | 4 +- .../nextflow/trace/AnsiLogObserver.groovy | 2 +- .../trace/DefaultObserverFactory.groovy | 4 +- .../nextflow/trace/GraphObserver.groovy | 2 +- .../nextflow/trace/ProgressRecord.groovy | 2 +- .../nextflow/trace/ProgressState.groovy | 2 +- .../nextflow/trace/ReportObserver.groovy | 2 +- .../nextflow/trace/ReportSummary.groovy | 4 +- .../nextflow/trace/ResourcesAggregator.groovy | 16 + .../nextflow/trace/TimelineObserver.groovy | 2 +- .../nextflow/trace/TraceFileObserver.groovy | 2 +- .../groovy/nextflow/trace/TraceHelper.groovy | 3 +- .../nextflow/trace/TraceObserver.groovy | 2 +- .../trace/TraceObserverFactory.groovy | 16 + .../trace/TraceObserverFactoryV2.groovy | 2 +- .../nextflow/trace/TraceObserverV2.groovy | 2 +- .../groovy/nextflow/trace/TraceRecord.groovy | 2 +- .../nextflow/trace/WorkflowStats.groovy | 2 +- .../trace/WorkflowStatsObserver.groovy | 2 +- .../nextflow/trace/config/DagConfig.groovy | 2 +- .../nextflow/trace/config/ReportConfig.groovy | 2 +- .../trace/config/TimelineConfig.groovy | 2 +- .../nextflow/trace/config/TraceConfig.groovy | 2 +- .../trace/event/FilePublishEvent.groovy | 2 +- .../nextflow/trace/event/TaskEvent.groovy | 2 +- .../trace/event/WorkflowOutputEvent.groovy | 2 +- .../main/groovy/nextflow/util/ArrayBag.groovy | 2 +- .../main/groovy/nextflow/util/Barrier.groovy | 2 +- .../nextflow/util/BlankSeparatedList.groovy | 2 +- .../nextflow/util/ClientProxyThrottler.groovy | 4 +- .../groovy/nextflow/util/ClusterConfig.groovy | 2 +- .../groovy/nextflow/util/ColorUtil.groovy | 2 +- .../groovy/nextflow/util/ConfigHelper.groovy | 2 +- .../groovy/nextflow/util/CsvWriter.groovy | 2 +- .../nextflow/util/CustomPoolFactory.groovy | 2 +- .../nextflow/util/CustomThreadFactory.groovy | 2 +- .../nextflow/util/CustomThreadPool.java | 2 +- .../nextflow/util/HardBlockingQueue.groovy | 2 +- .../groovy/nextflow/util/HexIdentity.groovy | 2 +- .../groovy/nextflow/util/HistoryFile.groovy | 2 +- .../groovy/nextflow/util/LangHelpers.java | 6 +- .../groovy/nextflow/util/LockManager.groovy | 2 +- .../groovy/nextflow/util/LoggerHelper.groovy | 2 +- .../util/MustacheTemplateEngine.groovy | 2 +- .../groovy/nextflow/util/NameGenerator.groovy | 12 +- .../nextflow/util/PathEscapeAware.groovy | 4 +- .../nextflow/util/PathNormalizer.groovy | 3 +- .../groovy/nextflow/util/PathSplitter.groovy | 5 +- .../main/groovy/nextflow/util/PathTrie.groovy | 4 +- .../groovy/nextflow/util/ProxyConfig.groovy | 3 +- .../groovy/nextflow/util/RemoteSession.groovy | 4 +- .../groovy/nextflow/util/SecretHelper.groovy | 26 +- .../nextflow/util/SerializationHelper.groovy | 2 +- .../groovy/nextflow/util/ServiceName.groovy | 2 +- .../groovy/nextflow/util/SimpleAgent.groovy | 2 +- .../nextflow/util/SimpleHttpClient.groovy | 3 +- .../groovy/nextflow/util/SpinnerUtil.groovy | 2 +- .../groovy/nextflow/util/SpuriousDeps.groovy | 2 +- .../nextflow/util/ThreadPoolBuilder.groovy | 3 +- .../nextflow/util/ThreadPoolHelper.groovy | 3 +- .../nextflow/util/ThreadPoolManager.groovy | 5 +- .../nextflow/util/ThrottlingExecutor.groovy | 8 +- .../src/main/groovy/nextflow/util/Trie.groovy | 2 +- .../groovy/nextflow/util/TupleHelper.groovy | 2 +- .../nextflow/util/VirtualThreadPool.groovy | 5 +- .../java/com/esotericsoftware/minlog/Log.java | 15 +- .../UnmodifiableCollectionsSerializer.java | 7 +- .../runtime/metaclass/ChannelFactory.java | 16 + .../CustomMetaClassCreationHandle.java | 2 +- .../runtime/metaclass/ExtensionProvider.java | 4 +- .../NextflowDelegatingMetaClass.java | 2 +- .../metaclass/NumberDelegatingMetaClass.java | 2 +- ...rg.codehaus.groovy.runtime.ExtensionModule | 2 +- .../nextflow/dag/mermaid.dag.template.html | 2 +- .../resources/nextflow/mail/notification.html | 32 +- .../nextflow/trace/ReportTemplate.html | 2 +- .../nextflow/trace/TimelineTemplate.html | 2 +- .../src/test/groovy/FunctionalTests.groovy | 2 +- .../NextflowDelegatingMetaClassTest.groovy | 2 +- .../NumberDelegatingMetaClassTest.groovy | 6 +- .../src/test/groovy/misc/ScriptEnv.groovy | 2 +- .../groovy/misc/ScriptNoInputNoOutput.groovy | 2 +- .../groovy/misc/ScriptProcessorTest.groovy | 2 +- .../test/groovy/nextflow/ChannelTest.groovy | 8 +- .../groovy/nextflow/NextflowMetaTest.groovy | 16 + .../test/groovy/nextflow/NextflowTest.groovy | 4 +- .../test/groovy/nextflow/SessionTest.groovy | 2 +- .../nextflow/ast/NextflowDSLImplTest.groovy | 42 +- .../nextflow/ast/NextflowXformImplTest.groovy | 110 ++-- .../groovy/nextflow/ast/OpXformTest.groovy | 20 +- .../ast/TaskCmdXformVisitorTest.groovy | 4 +- .../groovy/nextflow/cache/CacheDBTest.groovy | 3 +- .../cache/DefaultCacheStoreTest.groovy | 5 +- .../groovy/nextflow/cli/CmdAuthTest.groovy | 4 +- .../groovy/nextflow/cli/CmdCleanTest.groovy | 2 +- .../groovy/nextflow/cli/CmdCloneTest.groovy | 2 +- .../groovy/nextflow/cli/CmdConfigTest.groovy | 2 +- .../groovy/nextflow/cli/CmdHelperTest.groovy | 2 +- .../groovy/nextflow/cli/CmdInfoTest.groovy | 4 +- .../groovy/nextflow/cli/CmdInspectTest.groovy | 3 +- .../groovy/nextflow/cli/CmdLineageTest.groovy | 3 +- .../groovy/nextflow/cli/CmdLogTest.groovy | 4 +- .../nextflow/cli/CmdPluginCreateTest.groovy | 2 +- .../groovy/nextflow/cli/CmdPullTest.groovy | 4 +- .../groovy/nextflow/cli/CmdRunTest.groovy | 10 +- .../groovy/nextflow/cli/CmdSecretTest.groovy | 2 +- .../groovy/nextflow/cli/LauncherTest.groovy | 8 +- .../nextflow/conda/CondaCacheTest.groovy | 2 +- .../nextflow/conda/CondaConfigTest.groovy | 3 +- .../config/CascadingConfigTest.groovy | 2 +- .../nextflow/config/ConfigBuilderTest.groovy | 100 +-- .../nextflow/config/ConfigMapTest.groovy | 13 +- .../config/ConfigValidatorTest.groovy | 2 +- .../nextflow/config/ManifestTest.groovy | 20 +- .../parser/v1/ConfigParserV1Test.groovy | 2 +- .../parser/v2/ConfigParserV2Test.groovy | 2 +- .../container/ApptainerBuilderTest.groovy | 2 +- .../container/ApptainerCacheTest.groovy | 2 +- .../container/ApptainerConfigTest.groovy | 16 + .../container/CharliecloudBuilderTest.groovy | 4 +- .../container/CharliecloudCacheTest.groovy | 8 +- .../container/ContainerBuilderTest.groovy | 4 +- .../container/ContainerConfigTest.groovy | 2 +- .../container/ContainerHandlerTest.groovy | 6 +- .../ContainerNameValidatorTest.groovy | 2 +- .../container/DockerBuilderTest.groovy | 2 +- .../container/PodmanBuilderTest.groovy | 2 +- .../container/SarusBuilderTest.groovy | 2 +- .../container/ShifterBuilderTest.groovy | 2 +- .../container/SingularityBuilderTest.groovy | 2 +- .../container/SingularityCacheTest.groovy | 2 +- .../container/SingularityConfigTest.groovy | 16 + .../inspect/ContainerInspectModeTest.groovy | 3 +- .../inspect/ContainersInspectorTest.groovy | 3 +- .../resolver/ContainerInfoTest.groovy | 3 +- .../DefaultContainerResolverTest.groovy | 3 +- .../test/groovy/nextflow/dag/DAGTest.groovy | 2 +- .../nextflow/dag/DotRendererTest.groovy | 2 +- .../nextflow/dag/GexfRendererTest.groovy | 2 +- .../nextflow/dag/MermaidRendererTest.groovy | 2 +- .../MultipleInputChannelExceptionTest.groovy | 2 +- .../MultipleOutputChannelExceptionTest.groovy | 6 +- .../datasource/SraExplorerTest.groovy | 2 +- .../datasource/SraRetryConfigTest.groovy | 3 +- .../executor/AbstractGridExecutorTest.groovy | 6 +- .../nextflow/executor/BashFunLibTest.groovy | 20 +- .../executor/BashTemplateEngineTest.groovy | 2 +- .../executor/BashWrapperBuilderTest.groovy | 14 +- .../nextflow/executor/BatchCleanupTest.groovy | 2 +- .../executor/BridgeExecutorTest.groovy | 11 +- .../executor/CondorExecutorTest.groovy | 2 +- .../nextflow/executor/CrgExecutorTest.groovy | 8 +- .../executor/ExecutorConfigTest.groovy | 2 +- .../executor/ExecutorFactoryTest.groovy | 4 +- .../nextflow/executor/ExecutorTest.groovy | 16 + .../nextflow/executor/FluxExecutorTest.groovy | 2 +- .../nextflow/executor/GridExecutorTest.groovy | 2 +- .../executor/GridTaskHandlerTest.groovy | 5 +- .../executor/HyperQueueExecutorTest.groovy | 7 +- .../nextflow/executor/LsfExecutorTest.groovy | 6 +- .../nextflow/executor/MoabExecutorTest.groovy | 4 +- .../executor/NqsiiExecutorTest.groovy | 2 +- .../nextflow/executor/OarExecutorTest.groovy | 2 +- .../nextflow/executor/PbsExecutorTest.groovy | 4 +- .../executor/PbsProExecutorTest.groovy | 4 +- .../nextflow/executor/SgeExecutorTest.groovy | 6 +- .../SimpleFileCopyStrategyTest.groovy | 4 +- .../executor/SlurmExecutorTest.groovy | 2 +- .../nextflow/executor/TcsExecutorTest.groovy | 2 +- .../local/LocalTaskHandlerTest.groovy | 5 +- .../res/AcceleratorResourceTest.groovy | 2 +- .../executor/res/DiskResourceTest.groovy | 2 +- .../nextflow/extension/BranchOpTest.groovy | 2 +- .../nextflow/extension/BufferOpTest.groovy | 2 +- .../groovy/nextflow/extension/CHTest.groovy | 16 + .../extension/CapturePropertiesTest.groovy | 2 +- .../extension/CollectFileOperatorTest.groovy | 2 +- .../nextflow/extension/CollectOpTest.groovy | 2 +- .../nextflow/extension/CombineOpTest.groovy | 2 +- .../nextflow/extension/ConcatOpTest.groovy | 2 +- .../extension/CountFastaOpTest.groovy | 2 +- .../extension/CountFastqOpTest.groovy | 2 +- .../extension/CountLinesOpTest.groovy | 2 +- .../extension/DataflowHelperTest.groovy | 2 +- .../DataflowMathExtensionTest.groovy | 2 +- .../DataflowMergeExtensionTest.groovy | 2 +- .../extension/DataflowTapExtensionTest.groovy | 2 +- .../nextflow/extension/DumpOpTest.groovy | 2 +- .../nextflow/extension/GroupKeyTest.groovy | 4 +- .../extension/GroupTupleOpTest.groovy | 4 +- .../nextflow/extension/JoinOpTest.groovy | 2 +- .../nextflow/extension/LoggerTest.groovy | 2 +- .../nextflow/extension/MixOpTest.groovy | 2 +- .../nextflow/extension/MultiMapOpTest.groovy | 16 +- .../nextflow/extension/OpCallTest.groovy | 18 +- .../extension/OperatorImplTest.groovy | 4 +- .../nextflow/extension/PhaseOpTest.groovy | 2 +- .../extension/RandomSampleTest.groovy | 2 +- .../nextflow/extension/SetOpTest.groovy | 6 +- .../extension/SplitFastaOperatorTest.groovy | 2 +- .../extension/SplitFastqOperatorTest.groovy | 2 +- .../nextflow/extension/SplitOpTest.groovy | 2 +- .../nextflow/extension/SplitTextOpTest.groovy | 16 + .../extension/SplitterMergeClosureTest.groovy | 2 +- .../nextflow/extension/TransposeOpTest.groovy | 2 +- .../nextflow/extension/UniqueOpTest.groovy | 3 +- .../nextflow/extension/UntilManyOpTest.groovy | 3 +- .../extension/ViewOperatorTest.groovy | 2 +- .../plugin/ChannelFactoryInstanceTest.groovy | 3 +- .../plugin/PluginExtensionProviderTest.groovy | 3 +- .../nextflow/file/DirWatcherTest.groovy | 4 +- .../nextflow/file/DirWatcherV2Test.groovy | 3 +- .../nextflow/file/FilePorterTest.groovy | 4 +- .../nextflow/file/PathVisitorTest.groovy | 2 +- .../file/SequentialFileStoreTest.groovy | 4 +- .../file/SimpleFileCollectorTest.groovy | 2 +- .../groovy/nextflow/file/SlurperExTest.groovy | 8 +- .../file/SortFileCollectorTest.groovy | 2 +- .../nextflow/fusion/FusionConfigTest.groovy | 3 +- .../nextflow/fusion/FusionHelperTest.groovy | 3 +- .../fusion/FusionScriptLauncherTest.groovy | 3 +- .../nextflow/mail/AttachmentTest.groovy | 2 +- .../test/groovy/nextflow/mail/MailTest.groovy | 2 +- .../groovy/nextflow/mail/MailerTest.groovy | 2 +- .../platform/PlatformHelperTest.groovy | 2 +- .../plugin/spec/PluginSpecTest.groovy | 3 +- .../processor/ArchitectureTest.groovy | 2 +- .../processor/BatchContextTest.groovy | 2 +- .../processor/ErrorStrategyTest.groovy | 3 +- .../processor/LocalPollingMonitorTest.groovy | 4 +- .../ParallelPollingMonitorTest.groovy | 2 +- .../nextflow/processor/PublishDirTest.groovy | 2 +- .../processor/TaskArrayCollectorTest.groovy | 2 +- .../processor/TaskArrayExecutorTest.groovy | 5 +- .../processor/TaskArrayRunTest.groovy | 3 +- .../nextflow/processor/TaskBeanTest.groovy | 2 +- .../nextflow/processor/TaskConfigTest.groovy | 8 +- .../nextflow/processor/TaskContextTest.groovy | 4 +- .../processor/TaskEnvCollectorTest.groovy | 2 +- .../processor/TaskErrorFormatterTest.groovy | 2 +- .../processor/TaskFileCollectorTest.groovy | 2 +- .../nextflow/processor/TaskHandlerTest.groovy | 4 +- .../nextflow/processor/TaskHasherTest.groovy | 4 +- .../nextflow/processor/TaskIdTest.groovy | 2 +- .../processor/TaskInputsResolverTest.groovy | 2 +- .../processor/TaskOutputResolverTest.groovy | 2 +- .../nextflow/processor/TaskPathTest.groovy | 8 +- .../processor/TaskPollingMonitorTest.groovy | 6 +- .../processor/TaskProcessorTest.groovy | 4 +- .../nextflow/processor/TaskRunTest.groovy | 6 +- .../processor/TaskTemplateEngineTest.groovy | 2 +- .../nextflow/scm/AssetManagerTest.groovy | 2 +- .../scm/AzureRepositoryProviderTest.groovy | 2 +- .../BitbucketRepositoryProviderTest.groovy | 4 +- ...tbucketServerRepositoryProviderTest.groovy | 2 +- .../scm/GitReferenceHelperTest.groovy | 2 +- .../groovy/nextflow/scm/GitUrlTest.groovy | 2 +- .../scm/GiteaRepositoryProviderTest.groovy | 4 +- .../scm/GithubRepositoryProviderTest.groovy | 8 +- .../scm/GitlabRepositoryProviderTest.groovy | 2 +- .../groovy/nextflow/scm/HubOptionsTest.groovy | 2 +- .../scm/LocalRepositoryProviderTest.groovy | 2 +- ...MultiRevisionRepositoryStrategyTest.groovy | 2 +- .../nextflow/scm/ProviderConfigTest.groovy | 2 +- .../nextflow/scm/ProviderPathTest.groovy | 14 +- .../scm/RepositoryProviderTest.groovy | 46 +- .../nextflow/scm/UpdateModuleTest.groovy | 2 +- .../nextflow/script/BaseScriptTest.groovy | 6 +- .../groovy/nextflow/script/BodyDefTest.groovy | 2 +- .../nextflow/script/ChannelOutTest.groovy | 18 +- .../nextflow/script/ExecutionStackTest.groovy | 16 + .../nextflow/script/FunctionDefTest.groovy | 16 + .../nextflow/script/FusionMetaTest.groovy | 3 +- .../nextflow/script/IncludeDefTest.groovy | 20 +- .../nextflow/script/IterableDefTest.groovy | 3 +- .../nextflow/script/OutputDslTest.groovy | 16 + .../nextflow/script/ParamsDslTest.groovy | 16 + .../script/PlatformMetadataTest.groovy | 3 +- .../nextflow/script/ProcessConfigTest.groovy | 2 +- .../nextflow/script/ProcessDefTest.groovy | 16 + .../script/ProcessEntryHandlerTest.groovy | 12 +- .../nextflow/script/ScriptBindingTest.groovy | 4 +- .../nextflow/script/ScriptDslTest.groovy | 16 + .../nextflow/script/ScriptIncludesTest.groovy | 2 +- .../nextflow/script/ScriptMetaTest.groovy | 2 +- .../nextflow/script/ScriptPipesTest.groovy | 106 ++-- .../script/ScriptProcessRunTest.groovy | 24 +- .../nextflow/script/ScriptRecurseTest.groovy | 29 +- .../nextflow/script/ScriptRunnerTest.groovy | 2 +- .../nextflow/script/TaskClosureTest.groovy | 2 +- .../nextflow/script/WaveMetadataTest.groovy | 3 +- .../script/WorkflowBindingTest.groovy | 6 +- .../nextflow/script/WorkflowDefTest.groovy | 18 +- .../script/WorkflowMetadataTest.groovy | 12 +- .../script/WorkflowNotifierTest.groovy | 4 +- .../script/bundle/ResourcesBundleTest.groovy | 3 +- .../script/dsl/ProcessBuilderTest.groovy | 2 +- .../dsl/ProcessConfigBuilderTest.groovy | 2 +- .../script/dsl/ProcessDslV1Test.groovy | 2 +- .../script/dsl/ProcessDslV2Test.groovy | 2 +- .../script/params/ArityParamTest.groovy | 2 +- .../script/params/CmdEvalParamTest.groovy | 2 +- .../script/params/EachInParamTest.groovy | 2 +- .../script/params/EnvOutParamTest.groovy | 2 +- .../script/params/FileInParamTest.groovy | 4 +- .../script/params/FileOutParamTest.groovy | 2 +- .../script/params/ParamsDsl2Test.groovy | 16 + .../script/params/ParamsInTest.groovy | 2 +- .../script/params/ParamsOutTest.groovy | 2 +- .../script/params/TupleInParamTest.groovy | 2 +- .../script/params/TupleOutParamTest.groovy | 20 +- .../params/v2/ProcessFileInputTest.groovy | 2 +- .../params/v2/ProcessFileOutputTest.groovy | 2 +- .../parser/v1/ScriptLoaderV1Test.groovy | 18 +- .../parser/v2/ScriptLoaderV2Test.groovy | 20 +- .../secret/DummySecretsProvider.groovy | 3 +- .../secret/LocalSecretsProviderTest.groovy | 3 +- .../nextflow/secret/SecretHelperTest.groovy | 3 +- .../nextflow/secret/SecretsLoaderTest.groovy | 3 +- .../groovy/nextflow/sort/BigSortTest.groovy | 2 +- .../nextflow/spack/SpackCacheTest.groovy | 2 +- .../nextflow/spack/SpackConfigTest.groovy | 3 +- .../splitter/AbstractSplitterTest.groovy | 2 +- .../splitter/AbstractTextSplitterTest.groovy | 2 +- .../splitter/BytesSplitterTest.groovy | 2 +- .../splitter/CharSequenceCollectorTest.groovy | 2 +- .../nextflow/splitter/CsvSplitterTest.groovy | 2 +- .../splitter/FastaSplitterTest.groovy | 2 +- .../splitter/FastqSplitterTest.groovy | 2 +- .../nextflow/splitter/JsonSplitterTest.groovy | 8 +- .../splitter/ObjectListCollectorTest.groovy | 2 +- .../nextflow/splitter/SplitterExTest.groovy | 12 +- .../splitter/SplitterFactoryTest.groovy | 2 +- .../splitter/StringSplitterTest.groovy | 2 +- .../splitter/TextFileCollectorTest.groovy | 2 +- .../nextflow/splitter/TextSplitterTest.groovy | 2 +- .../nextflow/trace/AnsiLogObserverTest.groovy | 2 +- .../nextflow/trace/GraphObserverTest.groovy | 2 +- .../nextflow/trace/ProgressRecordTest.groovy | 2 +- .../nextflow/trace/ReportObserverTest.groovy | 2 +- .../nextflow/trace/ReportSummaryTest.groovy | 2 +- .../trace/ResourcesAggregatorTest.groovy | 16 + .../nextflow/trace/StatsObserverTest.groovy | 2 +- .../trace/TimelineObserverTest.groovy | 2 +- .../trace/TraceFileObserverTest.groovy | 2 +- .../nextflow/trace/TraceHelperTest.groovy | 3 +- .../nextflow/trace/TraceRecordTest.groovy | 6 +- .../nextflow/trace/WorkflowStatsTest.groovy | 8 +- .../groovy/nextflow/util/ArrayBagTest.groovy | 4 +- .../groovy/nextflow/util/BarrierTest.groovy | 2 +- .../util/BlankSeparatedListTest.groovy | 4 +- .../util/ClientProxyThrottlerTest.groovy | 4 +- .../nextflow/util/ClusterConfigTest.groovy | 4 +- .../groovy/nextflow/util/ColorUtilTest.groovy | 2 +- .../nextflow/util/ConfigHelperTest.groovy | 6 +- .../groovy/nextflow/util/CsvWriterTest.groovy | 16 + .../nextflow/util/HistoryFileTest.groovy | 2 +- .../groovy/nextflow/util/IdentityTest.groovy | 2 +- .../nextflow/util/KryoHelperTest.groovy | 2 +- .../nextflow/util/LockManagerTest.groovy | 2 +- .../nextflow/util/LoggerHelperTest.groovy | 4 +- .../util/MustacheTemplateEngineTest.groovy | 6 +- .../nextflow/util/NameGeneratorTest.groovy | 2 +- .../nextflow/util/PathSplitterTest.groovy | 3 +- .../groovy/nextflow/util/PathTrieTest.groovy | 2 +- .../nextflow/util/ProxyConfigTest.groovy | 7 +- .../nextflow/util/RemoteSessionTest.groovy | 2 +- .../nextflow/util/SecretHelperTest.groovy | 20 +- .../nextflow/util/SimpleAgentTest.groovy | 4 +- .../nextflow/util/SimpleHttpClientTest.groovy | 5 +- .../util/ThreadPoolBuilderTest.groovy | 3 +- .../nextflow/util/ThreadPoolHelperTest.groovy | 3 +- .../util/ThreadPoolManagerTest.groovy | 3 +- .../util/ThrottlingExecutorTest.groovy | 6 +- .../test/groovy/nextflow/util/TrieTest.groovy | 2 +- .../nextflow/util/TupleHelperTest.groovy | 2 +- .../src/test/resources/logback-test.xml | 2 +- .../nextflow/trace/timeline-expected.html | 2 +- .../testFixtures/groovy/test/BaseSpec.groovy | 2 +- .../testFixtures/groovy/test/Dsl2Spec.groovy | 2 +- .../groovy/test/OutputCapture.java | 18 +- .../groovy/test/ScriptHelper.groovy | 4 +- .../groovy/test/TemporaryPath.groovy | 2 +- .../groovy/test/TestHelper.groovy | 2 +- modules/nf-commons/build.gradle | 2 +- .../src/main/nextflow/BuildInfo.groovy | 5 +- .../nf-commons/src/main/nextflow/Const.groovy | 2 +- .../src/main/nextflow/Global.groovy | 4 +- .../src/main/nextflow/ISession.groovy | 2 +- .../src/main/nextflow/SysEnv.groovy | 7 +- .../exception/AbortOperationException.groovy | 2 +- .../src/main/nextflow/extension/Bolts.groovy | 4 +- .../main/nextflow/extension/FilesEx.groovy | 2 +- .../main/nextflow/file/CopyMoveHelper.java | 2 +- .../src/main/nextflow/file/CopyOptions.groovy | 3 +- .../src/main/nextflow/file/FileHelper.groovy | 2 +- .../src/main/nextflow/file/FileHolder.groovy | 2 +- .../src/main/nextflow/file/FileMutex.groovy | 2 +- .../nextflow/file/FilePatternSplitter.groovy | 2 +- .../file/FileSystemPathFactory.groovy | 2 +- .../file/FileSystemTransferAware.groovy | 16 + .../src/main/nextflow/file/Globs.groovy | 2 +- .../src/main/nextflow/file/StagePath.groovy | 2 +- .../main/nextflow/file/TagAwareFile.groovy | 5 +- .../src/main/nextflow/io/BucketParser.groovy | 17 +- .../io/ByteBufferBackedInputStream.groovy | 2 +- .../src/main/nextflow/io/ByteDumper.groovy | 2 +- .../nextflow/io/DataInputStreamAdapter.groovy | 2 +- .../io/DataOutputStreamAdapter.groovy | 2 +- .../src/main/nextflow/io/LogOutputStream.java | 19 +- .../main/nextflow/io/ReaderInputStream.java | 20 +- .../nextflow/io/SerializableMarker.groovy | 16 + .../nextflow/io/SerializableObject.groovy | 2 +- .../nextflow/io/SkipLinesInputStream.java | 2 +- .../src/main/nextflow/io/ValueObject.groovy | 2 +- .../main/nextflow/io/WriterOutputStream.java | 19 +- .../main/nextflow/plugin/BasePlugin.groovy | 3 +- .../plugin/CustomPluginManager.groovy | 3 +- .../plugin/CustomVersionManager.groovy | 16 + .../nextflow/plugin/DefaultPlugins.groovy | 3 +- .../nextflow/plugin/DevPluginClasspath.groovy | 3 +- .../nextflow/plugin/DevPluginLoader.groovy | 3 +- .../nextflow/plugin/DevPluginManager.groovy | 3 +- .../nextflow/plugin/DevPluginUpdater.groovy | 3 +- .../plugin/EmbeddedPluginManager.groovy | 3 +- .../plugin/HttpPluginRepository.groovy | 20 +- .../nextflow/plugin/LocalPluginManager.groovy | 3 +- .../plugin/LocalPluginRepository.groovy | 3 +- .../plugin/LocalUpdateRepository.groovy | 16 + .../src/main/nextflow/plugin/PluginRef.groovy | 3 +- .../main/nextflow/plugin/PluginUpdater.groovy | 3 +- .../src/main/nextflow/plugin/Plugins.groovy | 3 +- .../main/nextflow/plugin/PluginsFacade.groovy | 9 +- .../plugin/PrefetchUpdateRepository.groovy | 16 + .../src/main/nextflow/plugin/Priority.groovy | 13 +- .../src/main/nextflow/plugin/Scoped.groovy | 13 +- .../plugin/util/PluginRefactor.groovy | 2 +- .../src/main/nextflow/serde/Encoder.groovy | 2 +- .../nextflow/serde/JsonSerializable.groovy | 2 +- .../serde/gson/GStringSerializer.groovy | 2 +- .../nextflow/serde/gson/GsonEncoder.groovy | 2 +- .../nextflow/serde/gson/InstantAdapter.groovy | 2 +- .../serde/gson/OffsetDateTimeAdapter.groovy | 2 +- .../nextflow/serde/gson/PathAdapter.groovy | 2 +- .../serde/gson/RuntimeTypeAdapterFactory.java | 4 +- .../main/nextflow/ui/LabelDecorator.groovy | 2 +- .../src/main/nextflow/ui/TableBuilder.groovy | 2 +- .../src/main/nextflow/ui/TextLabel.groovy | 2 +- .../ui/console/ConsoleExtension.groovy | 16 + .../src/main/nextflow/util/ArrayTuple.java | 2 +- .../src/main/nextflow/util/Base62.java | 16 + .../src/main/nextflow/util/CacheFunnel.groovy | 16 + .../src/main/nextflow/util/CacheHelper.java | 2 +- .../main/nextflow/util/CharsetHelper.groovy | 2 +- .../src/main/nextflow/util/CheckHelper.groovy | 2 +- .../main/nextflow/util/CmdLineHelper.groovy | 4 +- .../nextflow/util/CmdLineOptionMap.groovy | 16 + .../nextflow/util/CollectionHelper.groovy | 2 +- .../src/main/nextflow/util/CsvParser.groovy | 2 +- .../src/main/nextflow/util/DebugUtils.groovy | 5 +- .../src/main/nextflow/util/Duration.groovy | 12 +- .../src/main/nextflow/util/Escape.groovy | 2 +- .../src/main/nextflow/util/HashBuilder.java | 2 +- .../src/main/nextflow/util/IniFile.groovy | 2 +- .../util/InputStreamDeserializer.groovy | 2 +- .../main/nextflow/util/InsensitiveMap.groovy | 5 +- .../src/main/nextflow/util/MathHelper.groovy | 5 +- .../src/main/nextflow/util/MemoryUnit.groovy | 2 +- .../main/nextflow/util/ProcessHelper.groovy | 16 + .../nextflow/util/QuoteStringTokenizer.groovy | 2 +- .../src/main/nextflow/util/RateUnit.groovy | 4 +- .../src/main/nextflow/util/RetryConfig.groovy | 2 +- .../src/main/nextflow/util/Rnd.groovy | 16 + .../nextflow/util/SerializerRegistrant.groovy | 2 +- .../main/nextflow/util/ServiceDiscover.groovy | 2 +- .../src/main/nextflow/util/StringUtils.groovy | 3 +- .../src/main/nextflow/util/SysHelper.groovy | 2 +- .../src/main/nextflow/util/TestOnly.groovy | 3 +- .../src/main/nextflow/util/Threads.groovy | 3 +- .../src/main/nextflow/util/Throttle.groovy | 2 +- .../src/main/nextflow/util/TypeHelper.groovy | 2 +- .../main/nextflow/util/VersionNumber.groovy | 2 +- ...rg.codehaus.groovy.runtime.ExtensionModule | 2 +- .../src/test/nextflow/BuildInfoTest.groovy | 3 +- .../src/test/nextflow/ConstTest.groovy | 3 +- .../src/test/nextflow/SysEnvTest.groovy | 3 +- .../test/nextflow/extension/BoltsTest.groovy | 6 +- .../nextflow/extension/FilesExTest.groovy | 2 +- .../test/nextflow/file/FileHelperTest.groovy | 4 +- .../file/FilePatternSplitterTest.groovy | 2 +- .../src/test/nextflow/file/GlobsTest.groovy | 4 +- .../test/nextflow/file/StagePathTest.groovy | 2 +- .../io/ByteBufferBackedInputStreamTest.groovy | 2 +- .../io/DataInputStreamAdapterTest.groovy | 2 +- .../io/DataOutputStreamAdapterTest.groovy | 2 +- .../io/SkipLinesInputStreamTest.groovy | 2 +- .../test/nextflow/io/ValueObjectTest.groovy | 42 +- .../nextflow/plugin/BasePluginTest.groovy | 3 +- .../plugin/CustomPluginManagerTest.groovy | 16 + .../plugin/CustomVersionManagerTest.groovy | 16 + .../nextflow/plugin/DefaultPluginsTest.groovy | 13 +- .../nextflow/plugin/FakeIndexHandler.groovy | 16 + .../plugin/HttpPluginRepositoryTest.groovy | 32 +- .../test/nextflow/plugin/PluginRefTest.groovy | 18 +- .../nextflow/plugin/PluginUpdaterTest.groovy | 26 +- .../nextflow/plugin/PluginsFacadeTest.groovy | 18 +- .../PluginExtensionMethodsTest.groovy | 3 +- .../plugin/hello/HelloExtensionTest.groovy | 3 +- .../plugin/hello/HelloFactoryTest.groovy | 3 +- .../plugin/util/PluginRefactorTest.groovy | 2 +- .../nextflow/serde/GsonEncoderTest.groovy | 2 +- .../src/test/nextflow/serde/MyEncoder.groovy | 2 +- .../test/nextflow/ui/TableBuilderTest.groovy | 2 +- .../src/test/nextflow/ui/TextLabelTest.groovy | 2 +- .../test/nextflow/util/ArrayTupleTest.groovy | 2 +- .../nextflow/util/BucketParserTest.groovy | 17 +- .../test/nextflow/util/CacheHelperTest.groovy | 2 +- .../nextflow/util/CharsetHelperTest.groovy | 2 +- .../test/nextflow/util/CheckHelperTest.groovy | 2 +- .../nextflow/util/CmdLineHelperTest.groovy | 3 +- .../nextflow/util/CmdLineOptionMapTest.groovy | 18 +- .../nextflow/util/CollectionHelperTest.groovy | 2 +- .../test/nextflow/util/CsvParserTest.groovy | 2 +- .../test/nextflow/util/DurationTest.groovy | 2 +- .../src/test/nextflow/util/EscapeTest.groovy | 6 +- .../test/nextflow/util/HashBuilderTest.groovy | 6 +- .../src/test/nextflow/util/IniFileTest.groovy | 2 +- .../nextflow/util/InsensitiveMapTest.groovy | 3 +- .../test/nextflow/util/MathHelperTest.groovy | 3 +- .../test/nextflow/util/MemoryUnitTest.groovy | 6 +- .../nextflow/util/ProcessHelperTest.groovy | 16 + .../test/nextflow/util/RateUnitTest.groovy | 2 +- .../test/nextflow/util/RetryConfigTest.groovy | 2 +- .../test/nextflow/util/StringUtilsTest.groovy | 3 +- .../test/nextflow/util/SysHelperTest.groovy | 2 +- .../src/test/nextflow/util/ThreadsTest.groovy | 3 +- .../test/nextflow/util/ThrottleTest.groovy | 2 +- .../nextflow/util/VersionNumberTest.groovy | 2 +- .../plugin/TestPluginClasspath.groovy | 3 +- .../plugin/TestPluginDescriptorFinder.groovy | 3 +- .../nextflow/plugin/TestPluginLoader.groovy | 3 +- .../nextflow/plugin/TestPluginManager.groovy | 3 +- .../plugin/hello/HelloExtension.groovy | 3 +- .../nextflow/plugin/hello/HelloFactory.groovy | 3 +- .../plugin/hello/HelloFunctions.groovy | 3 +- .../plugin/hello/HelloObserver.groovy | 3 +- .../nextflow/plugin/hello/HelloPlugin.groovy | 3 +- modules/nf-httpfs/build.gradle | 2 +- .../file/http/FixedInputStream.groovy | 3 +- .../file/http/FtpFileSystemProvider.groovy | 2 +- .../file/http/HttpFileSystemProvider.groovy | 2 +- .../file/http/HttpsFileSystemProvider.groovy | 2 +- .../nextflow/file/http/XAuthProvider.groovy | 3 +- .../nextflow/file/http/XAuthRegistry.groovy | 3 +- .../nextflow/file/http/XFileAttributes.groovy | 2 +- .../nextflow/file/http/XFileSystem.groovy | 2 +- .../file/http/XFileSystemConfig.groovy | 16 + .../file/http/XFileSystemProvider.groovy | 2 +- .../src/main/nextflow/file/http/XPath.groovy | 2 +- .../nextflow/file/http/XPathRegistrant.groovy | 4 +- .../nextflow/file/http/XPathSerializer.groovy | 4 +- .../java.nio.file.spi.FileSystemProvider | 2 +- .../file/http/FixedInputStreamTest.groovy | 3 +- .../nextflow/file/http/HttpFilesTests.groovy | 6 +- .../file/http/XAuthRegistryTest.groovy | 3 +- .../file/http/XFileSystemConfigTest.groovy | 16 + .../file/http/XFileSystemProviderTest.groovy | 2 +- .../test/nextflow/file/http/XPathTest.groovy | 4 +- modules/nf-lang/build.gradle | 4 +- .../config/ast/ConfigApplyBlockNode.java | 2 +- .../nextflow/config/ast/ConfigApplyNode.java | 2 +- .../nextflow/config/ast/ConfigAssignNode.java | 2 +- .../nextflow/config/ast/ConfigBlockNode.java | 2 +- .../config/ast/ConfigIncludeNode.java | 2 +- .../config/ast/ConfigIncompleteNode.java | 2 +- .../java/nextflow/config/ast/ConfigNode.java | 2 +- .../nextflow/config/ast/ConfigStatement.java | 2 +- .../nextflow/config/ast/ConfigVisitor.java | 2 +- .../config/ast/ConfigVisitorSupport.java | 2 +- .../nextflow/config/control/ConfigParser.java | 2 +- .../config/control/ConfigResolveVisitor.java | 6 +- .../config/control/ConfigToGroovyVisitor.java | 2 +- .../config/control/ResolveIncludeVisitor.java | 2 +- .../control/StringReaderSourceWithURI.java | 2 +- .../config/control/StripSecretsVisitor.java | 3 +- .../config/control/VariableScopeVisitor.java | 2 +- .../java/nextflow/config/dsl/ConfigDsl.java | 2 +- .../formatter/ConfigFormattingVisitor.java | 2 +- .../config/parser/ConfigAstBuilder.java | 10 +- .../config/parser/ConfigParserPlugin.java | 2 +- .../parser/ConfigParserPluginFactory.java | 2 +- .../nextflow/config/schema/ConfigOption.java | 2 +- .../nextflow/config/schema/ConfigScope.java | 2 +- .../config/schema/PlaceholderName.java | 2 +- .../nextflow/config/schema/ScopeName.java | 2 +- .../java/nextflow/config/scopes/Config.java | 10 +- .../nextflow/config/scopes/PluginsDsl.java | 2 +- .../nextflow/config/spec/ConfigOption.java | 2 +- .../nextflow/config/spec/ConfigScope.java | 2 +- .../nextflow/config/spec/PlaceholderName.java | 2 +- .../java/nextflow/config/spec/ScopeName.java | 2 +- .../java/nextflow/config/spec/SpecNode.java | 14 +- .../nextflow/script/ast/ASTNodeMarker.java | 2 +- .../java/nextflow/script/ast/ASTUtils.java | 6 +- .../script/ast/AssignmentExpression.java | 2 +- .../nextflow/script/ast/FeatureFlagNode.java | 2 +- .../nextflow/script/ast/FunctionNode.java | 2 +- .../script/ast/ImplicitClosureParameter.java | 2 +- .../nextflow/script/ast/IncludeEntryNode.java | 2 +- .../java/nextflow/script/ast/IncludeNode.java | 2 +- .../nextflow/script/ast/IncompleteNode.java | 2 +- .../script/ast/InvalidDeclaration.java | 2 +- .../nextflow/script/ast/OutputBlockNode.java | 2 +- .../java/nextflow/script/ast/OutputNode.java | 2 +- .../nextflow/script/ast/ParamBlockNode.java | 2 +- .../java/nextflow/script/ast/ParamNodeV1.java | 2 +- .../java/nextflow/script/ast/ProcessNode.java | 2 +- .../nextflow/script/ast/ProcessNodeV1.java | 2 +- .../nextflow/script/ast/ProcessNodeV2.java | 2 +- .../java/nextflow/script/ast/ScriptNode.java | 2 +- .../nextflow/script/ast/ScriptVisitor.java | 2 +- .../script/ast/ScriptVisitorSupport.java | 2 +- .../nextflow/script/ast/TupleParameter.java | 2 +- .../nextflow/script/ast/WorkflowNode.java | 2 +- .../script/control/CallSiteCollector.java | 2 +- .../nextflow/script/control/Compiler.java | 2 +- .../script/control/GStringToLazyVisitor.java | 2 +- .../control/GStringToStringVisitor.java | 2 +- .../script/control/LazyErrorCollector.java | 2 +- .../script/control/ModuleResolver.java | 2 +- .../script/control/OpCriteriaVisitor.java | 2 +- .../script/control/ParanoidWarning.java | 2 +- .../script/control/PathCompareVisitor.java | 2 +- .../nextflow/script/control/PhaseAware.java | 2 +- .../java/nextflow/script/control/Phases.java | 2 +- .../script/control/ProcessNameResolver.java | 2 +- .../control/ProcessToGroovyVisitorV1.java | 2 +- .../control/ProcessToGroovyVisitorV2.java | 2 +- .../control/RelatedInformationAware.java | 2 +- .../script/control/ResolveIncludeVisitor.java | 2 +- .../script/control/ResolveVisitor.java | 2 +- .../nextflow/script/control/ScriptParser.java | 2 +- .../script/control/ScriptResolveVisitor.java | 6 +- .../script/control/ScriptToGroovyHelper.java | 2 +- .../script/control/ScriptToGroovyVisitor.java | 2 +- .../script/control/StripTypesVisitor.java | 2 +- .../script/control/TaskCmdXformVisitor.java | 2 +- .../script/control/TypeCheckingVisitor.java | 2 +- .../script/control/VariableScopeChecker.java | 2 +- .../script/control/VariableScopeVisitor.java | 2 +- .../java/nextflow/script/dsl/Constant.java | 2 +- .../java/nextflow/script/dsl/Description.java | 2 +- .../java/nextflow/script/dsl/DslScope.java | 2 +- .../nextflow/script/dsl/EntryWorkflowDsl.java | 2 +- .../java/nextflow/script/dsl/FeatureFlag.java | 2 +- .../nextflow/script/dsl/FeatureFlagDsl.java | 2 +- .../java/nextflow/script/dsl/Namespace.java | 2 +- .../java/nextflow/script/dsl/Operator.java | 2 +- .../java/nextflow/script/dsl/OutputDsl.java | 2 +- .../java/nextflow/script/dsl/ProcessDsl.java | 4 +- .../java/nextflow/script/dsl/ScriptDsl.java | 2 +- .../java/nextflow/script/dsl/WorkflowDsl.java | 2 +- .../nextflow/script/formatter/Formatter.java | 2 +- .../script/formatter/FormattingOptions.java | 2 +- .../script/formatter/GroovyFormatter.java | 2 +- .../formatter/ScriptFormattingVisitor.java | 2 +- .../script/namespaces/ChannelNamespace.java | 2 +- .../script/namespaces/LogNamespace.java | 2 +- .../script/namespaces/ManifestNamespace.java | 2 +- .../script/namespaces/NextflowNamespace.java | 2 +- .../script/namespaces/WorkflowNamespace.java | 2 +- .../nextflow/script/parser/AbstractLexer.java | 21 +- .../script/parser/AbstractParser.java | 21 +- .../parser/DescriptiveErrorStrategy.java | 21 +- .../script/parser/GroovydocManager.java | 21 +- .../script/parser/PositionConfigureUtils.java | 21 +- .../script/parser/ScriptAstBuilder.java | 6 +- .../script/parser/ScriptParserPlugin.java | 2 +- .../parser/ScriptParserPluginFactory.java | 2 +- .../script/parser/SemanticPredicates.java | 21 +- .../nextflow/script/parser/TokenPosition.java | 2 +- .../main/java/nextflow/script/types/Bag.java | 2 +- .../java/nextflow/script/types/Channel.java | 4 +- .../java/nextflow/script/types/Duration.java | 2 +- .../nextflow/script/types/MemoryUnit.java | 2 +- .../java/nextflow/script/types/ParamsMap.java | 2 +- .../java/nextflow/script/types/Record.java | 2 +- .../nextflow/script/types/TaskConfig.java | 2 +- .../java/nextflow/script/types/Tuple.java | 2 +- .../java/nextflow/script/types/Types.java | 2 +- .../java/nextflow/script/types/Value.java | 2 +- .../nextflow/script/types/VersionNumber.java | 2 +- .../main/java/nextflow/util/PathUtils.java | 2 +- .../config/control/ConfigResolveTest.groovy | 2 +- .../config/control/ResolveIncludeTest.groovy | 2 +- .../formatter/ConfigFormatterTest.groovy | 2 +- .../config/parser/ConfigAstBuilderTest.groovy | 2 +- .../nextflow/config/spec/SpecNodeTest.groovy | 2 +- .../script/control/ResolveIncludeTest.groovy | 2 +- .../script/control/ScriptResolveTest.groovy | 2 +- .../control/ScriptToGroovyHelperTest.groovy | 2 +- .../script/control/TypeCheckingTest.groovy | 2 +- .../formatter/ScriptFormatterTest.groovy | 4 +- .../script/parser/ScriptAstBuilderTest.groovy | 2 +- .../groovy/nextflow/util/PathUtilsTest.groovy | 2 +- .../testFixtures/groovy/test/TestUtils.groovy | 2 +- modules/nf-lineage/build.gradle | 2 +- .../lineage/DefaultLinHistoryLog.groovy | 2 +- .../nextflow/lineage/DefaultLinStore.groovy | 2 +- .../lineage/DefaultLinStoreFactory.groovy | 2 +- .../nextflow/lineage/LinExtensionImpl.groovy | 2 +- .../nextflow/lineage/LinHistoryLog.groovy | 2 +- .../nextflow/lineage/LinHistoryRecord.groovy | 2 +- .../main/nextflow/lineage/LinObserver.groovy | 2 +- .../lineage/LinObserverFactory.groovy | 2 +- .../lineage/LinPropertyValidator.groovy | 2 +- .../src/main/nextflow/lineage/LinStore.groovy | 2 +- .../nextflow/lineage/LinStoreFactory.groovy | 2 +- .../src/main/nextflow/lineage/LinUtils.groovy | 4 +- .../lineage/cli/LinCommandImpl.groovy | 4 +- .../lineage/cli/LinDagRenderer.groovy | 2 +- .../lineage/config/LineageConfig.groovy | 2 +- .../lineage/config/LineageStoreOpts.groovy | 2 +- .../OutputRelativePathException.groovy | 2 +- .../nextflow/lineage/fs/LinFileSystem.groovy | 2 +- .../lineage/fs/LinFileSystemProvider.groovy | 2 +- .../lineage/fs/LinIntermediatePath.groovy | 16 + .../lineage/fs/LinMetadataPath.groovy | 2 +- .../fs/LinMetadataSeekableByteChannel.groovy | 2 +- .../main/nextflow/lineage/fs/LinPath.groovy | 4 +- .../nextflow/lineage/fs/LinPathFactory.groovy | 2 +- .../lineage/model/v1beta1/Checksum.groovy | 2 +- .../lineage/model/v1beta1/DataPath.groovy | 2 +- .../lineage/model/v1beta1/FileOutput.groovy | 2 +- .../lineage/model/v1beta1/LinModel.groovy | 2 +- .../lineage/model/v1beta1/Parameter.groovy | 2 +- .../lineage/model/v1beta1/TaskOutput.groovy | 2 +- .../lineage/model/v1beta1/TaskRun.groovy | 2 +- .../lineage/model/v1beta1/Workflow.groovy | 2 +- .../model/v1beta1/WorkflowOutput.groovy | 2 +- .../lineage/model/v1beta1/WorkflowRun.groovy | 2 +- .../nextflow/lineage/serde/LinEncoder.groovy | 2 +- .../lineage/serde/LinSerializable.groovy | 2 +- .../serde/LinTypeAdapterFactory.groovy | 4 +- .../java.nio.file.spi.FileSystemProvider | 2 +- .../lineage/DefaultLinHistoryLogTest.groovy | 2 +- .../lineage/DefaultLinStoreFactoryTest.groovy | 4 +- .../lineage/DefaultLinStoreTest.groovy | 2 +- .../lineage/LinExtensionImplTest.groovy | 2 +- .../lineage/LinHistoryRecordTest.groovy | 2 +- .../nextflow/lineage/LinObserverTest.groovy | 3 +- .../lineage/LinPropertyValidationTest.groovy | 2 +- .../test/nextflow/lineage/LinUtilsTest.groovy | 2 +- .../lineage/cli/LinCommandImplTest.groovy | 2 +- .../lineage/config/LineageConfigTest.groovy | 2 +- .../fs/LinFileSystemProviderTest.groovy | 6 +- .../lineage/fs/LinPathFactoryTest.groovy | 6 +- .../nextflow/lineage/fs/LinPathTest.groovy | 4 +- .../lineage/model/ChecksumTest.groovy | 2 +- .../lineage/serde/LinEncoderTest.groovy | 2 +- .../serde/LinTypeAdapterFactoryTest.groovy | 2 +- nextflow | 25 +- plugins/nf-amazon/build.gradle | 4 +- .../nextflow/cloud/aws/AmazonPlugin.groovy | 2 +- .../cloud/aws/AwsClientFactory.groovy | 2 +- .../cloud/aws/batch/AwsBatchExecutor.groovy | 28 +- .../aws/batch/AwsBatchFileCopyStrategy.groovy | 2 +- .../cloud/aws/batch/AwsBatchHelper.groovy | 4 +- .../cloud/aws/batch/AwsBatchProxy.groovy | 2 +- .../aws/batch/AwsBatchScriptLauncher.groovy | 2 +- .../aws/batch/AwsBatchTaskHandler.groovy | 6 +- .../batch/AwsContainerOptionsMapper.groovy | 2 +- .../cloud/aws/batch/AwsOptions.groovy | 2 +- .../model/ContainerPropertiesModel.groovy | 6 +- .../model/RegisterJobDefinitionModel.groovy | 6 +- .../cloud/aws/config/AwsBatchConfig.groovy | 5 +- .../cloud/aws/config/AwsConfig.groovy | 3 +- .../cloud/aws/config/AwsS3Config.groovy | 3 +- .../cloud/aws/fusion/AwsFusionEnv.groovy | 2 +- .../cloud/aws/mail/AwsMailProvider.groovy | 3 +- .../main/nextflow/cloud/aws/nio/S3Client.java | 5 +- .../cloud/aws/nio/S3FileAttributes.java | 5 +- .../cloud/aws/nio/S3FileAttributesView.java | 3 +- .../nextflow/cloud/aws/nio/S3FileSystem.java | 7 +- .../cloud/aws/nio/S3FileSystemProvider.java | 3 +- .../nextflow/cloud/aws/nio/S3Iterator.java | 3 +- .../cloud/aws/nio/S3OutputStream.java | 7 +- .../main/nextflow/cloud/aws/nio/S3Path.java | 41 +- .../cloud/aws/nio/ng/ChunkBuffer.java | 3 +- .../cloud/aws/nio/ng/ChunkBufferFactory.java | 3 +- .../cloud/aws/nio/ng/DownloadOpts.java | 3 +- .../cloud/aws/nio/ng/FutureInputStream.java | 3 +- .../cloud/aws/nio/ng/FutureIterator.java | 5 +- .../aws/nio/util/ByteBufferInputStream.java | 3 +- .../nio/util/ExtendedS3TransferManager.java | 3 +- .../nextflow/cloud/aws/nio/util/IOUtils.java | 3 +- .../nio/util/S3AsyncClientConfiguration.java | 3 +- .../aws/nio/util/S3ClientConfiguration.java | 3 +- .../aws/nio/util/S3MultipartOptions.java | 3 +- .../cloud/aws/nio/util/S3ObjectId.java | 3 +- .../aws/nio/util/S3ObjectSummaryLookup.java | 3 +- .../nio/util/S3SyncClientConfiguration.java | 3 +- .../nextflow/cloud/aws/util/AwsHelper.groovy | 2 +- .../cloud/aws/util/ConfigParser.groovy | 3 +- .../nextflow/cloud/aws/util/S3BashLib.groovy | 6 +- .../aws/util/S3CredentialsProvider.groovy | 3 +- .../cloud/aws/util/S3PathFactory.groovy | 4 +- .../cloud/aws/util/S3PathSerializer.groovy | 4 +- .../java.nio.file.spi.FileSystemProvider | 2 +- .../src/test/nextflow/S3ChannelTest.groovy | 3 +- .../src/test/nextflow/S3NextflowTest.groovy | 3 +- .../src/test/nextflow/S3SessionTest.groovy | 3 +- .../cloud/aws/AwsClientFactoryTest.groovy | 3 +- .../batch/AwsBatchFileCopyStrategyTest.groovy | 18 +- .../cloud/aws/batch/AwsBatchHelperTest.groovy | 2 +- .../cloud/aws/batch/AwsBatchProxyTest.groovy | 2 +- .../batch/AwsBatchScriptLauncherTest.groovy | 44 +- .../aws/batch/AwsBatchTaskHandlerTest.groovy | 14 +- .../AwsContainerOptionsMapperTest.groovy | 16 + .../cloud/aws/batch/AwsOptionsTest.groovy | 6 +- .../model/ContainerPropertiesModelTest.groovy | 134 ++-- .../RegisterJobDefinitionModelTest.groovy | 92 +-- .../aws/config/AwsBatchConfigTest.groovy | 5 +- .../cloud/aws/config/AwsConfigTest.groovy | 5 +- .../cloud/aws/config/AwsS3ConfigTest.groovy | 3 +- .../cloud/aws/fusion/AwsFusionEnvTest.groovy | 3 +- .../cloud/aws/nio/AwsS3BaseSpec.groovy | 5 +- .../cloud/aws/nio/AwsS3NioTest.groovy | 11 +- .../aws/nio/S3FileSystemProviderTest.groovy | 3 +- .../cloud/aws/nio/S3OutputStreamTest.groovy | 3 +- .../cloud/aws/nio/ng/DownloadOptsTest.groovy | 3 +- .../aws/nio/ng/FutureInputStreamTest.groovy | 3 +- .../util/ExtendedS3TransferManagerTest.groovy | 7 +- .../nio/util/S3ClientConfigurationTest.groovy | 3 +- .../cloud/aws/util/AwsHelperTest.groovy | 13 +- .../cloud/aws/util/S3BashLibTest.groovy | 74 ++- .../cloud/aws/util/S3PathFactoryTest.groovy | 18 +- .../nextflow/cloud/aws/util/S3PathTest.groovy | 16 + .../executor/AwsBatchExecutorTest.groovy | 13 +- .../BashWrapperBuilderWithS3Test.groovy | 16 +- .../FusionScriptLauncherS3Test.groovy | 13 +- .../nextflow/extension/PublishOpS3Test.groovy | 2 +- .../nextflow/file/FileHelperS3Test.groovy | 3 +- .../processor/PublishDirS3Test.groovy | 2 +- .../nextflow/util/ConfigParserTest.groovy | 17 +- .../nextflow/util/S3PathSerializerTest.groovy | 2 +- .../src/testResources/amazon.properties | 3 +- .../src/testResources/logback-test.xml | 2 +- plugins/nf-azure/build.gradle | 2 +- .../nextflow/cloud/azure/AzurePlugin.groovy | 4 +- .../cloud/azure/batch/AzBatchExecutor.groovy | 2 +- .../azure/batch/AzBatchScriptLauncher.groovy | 2 +- .../cloud/azure/batch/AzBatchService.groovy | 18 +- .../azure/batch/AzBatchTaskHandler.groovy | 4 +- .../azure/batch/AzFileCopyStrategy.groovy | 2 +- .../cloud/azure/batch/AzHelper.groovy | 4 +- .../cloud/azure/batch/AzJobKey.groovy | 5 +- .../cloud/azure/batch/AzTaskKey.groovy | 2 +- .../cloud/azure/batch/AzVmPoolSpec.groovy | 2 +- .../cloud/azure/batch/AzVmType.groovy | 2 +- .../azure/config/AzActiveDirectoryOpts.groovy | 2 +- .../cloud/azure/config/AzBatchOpts.groovy | 2 +- .../cloud/azure/config/AzConfig.groovy | 2 +- .../cloud/azure/config/AzCopyOpts.groovy | 5 +- .../cloud/azure/config/AzFileShareOpts.groovy | 3 +- .../azure/config/AzManagedIdentityOpts.groovy | 2 +- .../cloud/azure/config/AzPoolOpts.groovy | 4 +- .../cloud/azure/config/AzRegistryOpts.groovy | 2 +- .../cloud/azure/config/AzRetryConfig.groovy | 5 +- .../cloud/azure/config/AzStartTaskOpts.groovy | 4 +- .../cloud/azure/config/AzStorageOpts.groovy | 2 +- .../azure/config/CopyToolInstallMode.groovy | 3 +- .../cloud/azure/file/AzBashLib.groovy | 7 +- .../cloud/azure/file/AzPathFactory.groovy | 2 +- .../cloud/azure/file/AzPathSerializer.groovy | 2 +- .../cloud/azure/fusion/AzFusionEnv.groovy | 7 +- .../cloud/azure/nio/AzFileAttributes.groovy | 2 +- .../azure/nio/AzFileAttributesView.groovy | 2 +- .../cloud/azure/nio/AzFileSystem.groovy | 2 +- .../azure/nio/AzFileSystemProvider.groovy | 2 +- .../nextflow/cloud/azure/nio/AzPath.groovy | 2 +- .../cloud/azure/nio/AzPathIterator.groovy | 2 +- .../azure/nio/AzReadableByteChannel.groovy | 2 +- .../azure/nio/AzWriteableByteChannel.groovy | 2 +- .../java.nio.file.spi.FileSystemProvider | 2 +- .../nextflow/cloud/azure/az-locations.sh | 3 +- .../azure/batch/AzBatchServiceTest.groovy | 46 +- .../azure/batch/AzBatchTaskHandlerTest.groovy | 20 +- .../azure/batch/AzFileCopyStrategyTest.groovy | 58 +- .../cloud/azure/batch/AzHelperTest.groovy | 3 +- .../cloud/azure/batch/AzJobKeyTest.groovy | 3 +- .../cloud/azure/batch/AzTaskKeyTest.groovy | 16 + .../cloud/azure/config/AzBatchOptsTest.groovy | 16 + .../cloud/azure/config/AzCopyOptsTest.groovy | 16 + .../config/AzManagedIdentityOptsTest.groovy | 3 +- .../cloud/azure/config/AzPoolOptsTest.groovy | 3 +- .../azure/config/AzRegistryOptsTest.groovy | 16 + .../azure/config/AzRetryConfigTest.groovy | 3 +- .../azure/config/AzStorageOptsTest.groovy | 18 +- .../cloud/azure/config/AzureConfigTest.groovy | 4 +- .../cloud/azure/file/AzBashLibTest.groovy | 46 +- .../cloud/azure/file/AzPathFactoryTest.groovy | 18 +- .../cloud/azure/fusion/AzFusionEnvTest.groovy | 9 +- .../cloud/azure/nio/AzBaseSpec.groovy | 16 + .../azure/nio/AzFileSystemProviderTest.groovy | 18 +- .../cloud/azure/nio/AzFileSystemTest.groovy | 18 +- .../nextflow/cloud/azure/nio/AzNioTest.groovy | 18 +- .../cloud/azure/nio/AzPathTest.groovy | 18 +- .../BashWrapperBuilderWithAzTest.groovy | 42 +- .../nextflow/file/FileHelperAzTest.groovy | 3 +- .../src/testResources/logback-test.xml | 2 +- plugins/nf-cloudcache/build.gradle | 2 +- .../src/main/nextflow/CloudCachePlugin.groovy | 17 +- .../nextflow/cache/CloudCacheConfig.groovy | 3 +- .../nextflow/cache/CloudCacheFactory.groovy | 3 +- .../nextflow/cache/CloudCacheStore.groovy | 3 +- .../cache/CloudCacheConfigTest.groovy | 3 +- plugins/nf-codecommit/build.gradle | 4 +- .../AwsCodeCommitCredentialProvider.groovy | 18 +- .../codecommit/AwsCodeCommitFactory.groovy | 3 +- .../aws/codecommit/AwsCodeCommitPlugin.groovy | 3 +- .../AwsCodeCommitProviderConfig.groovy | 3 +- .../AwsCodeCommitRepositoryProvider.groovy | 16 +- .../AwsCodeCommitFactoryTest.groovy | 3 +- .../AwsCodeCommitProviderConfigTest.groovy | 3 +- ...AwsCodeCommitRepositoryProviderTest.groovy | 7 +- plugins/nf-console/build.gradle | 4 +- .../nextflow/ui/console/ConsolePlugin.groovy | 4 +- .../nextflow/ui/console/ConsoleRunner.groovy | 2 +- .../main/nextflow/ui/console/Nextflow.groovy | 2 +- .../ui/console/SimpleConsoleLayout.java | 2 +- .../ui/console/ConsoleRunnerTest.groovy | 13 +- plugins/nf-google/build.gradle | 2 +- .../cloud/google/GoogleCloudPlugin.groovy | 4 +- .../nextflow/cloud/google/GoogleOpts.groovy | 3 +- .../google/batch/GoogleBatchExecutor.groovy | 3 +- .../batch/GoogleBatchFusionAdapter.groovy | 3 +- .../batch/GoogleBatchLauncherSpec.groovy | 4 +- .../GoogleBatchMachineTypeSelector.groovy | 3 +- .../batch/GoogleBatchScriptLauncher.groovy | 7 +- .../batch/GoogleBatchTaskHandler.groovy | 5 +- .../google/batch/client/BatchClient.groovy | 2 +- .../google/batch/client/BatchConfig.groovy | 4 +- .../batch/client/BatchRetryConfig.groovy | 3 +- .../google/batch/logging/BatchLogging.groovy | 5 +- .../google/config/GoogleRetryOpts.groovy | 3 +- .../google/config/GoogleStorageOpts.groovy | 3 +- .../cloud/google/util/GsPathFactory.groovy | 2 +- .../cloud/google/util/GsPathSerializer.groovy | 2 +- .../nf-google/src/resources/logback-test.xml | 2 +- .../cloud/google/GoogleSpecification.groovy | 2 +- .../batch/GoogleBatchExecutorTest.groovy | 13 +- .../batch/GoogleBatchLauncherSpecMock.groovy | 3 +- .../GoogleBatchMachineTypeSelectorTest.groovy | 16 + .../GoogleBatchScriptLauncherTest.groovy | 5 +- .../batch/GoogleBatchTaskHandlerTest.groovy | 5 +- .../batch/client/BatchClientTest.groovy | 3 +- .../batch/client/BatchConfigTest.groovy | 3 +- .../batch/client/BatchRetryConfigTest.groovy | 3 +- .../batch/logging/BatchLoggingTest.groovy | 5 +- .../google/config/GoogleRetryOptsTest.groovy | 3 +- .../google/util/GsPathFactoryTest.groovy | 2 +- .../google/util/GsPathSerializerTest.groovy | 4 +- .../BashWrapperBuilderWithGoogleTest.groovy | 3 +- .../nextflow/extension/EscapeTest2.groovy | 2 +- .../nextflow/extension/FilesExTest2.groovy | 2 +- .../nextflow/file/FileHelperGsTest.groovy | 2 +- .../src/test/nextflow/file/GsPathTest.groovy | 3 +- plugins/nf-k8s/build.gradle | 2 +- .../src/main/nextflow/k8s/K8sConfig.groovy | 4 +- .../nextflow/k8s/K8sDriverLauncher.groovy | 8 +- .../src/main/nextflow/k8s/K8sExecutor.groovy | 2 +- .../src/main/nextflow/k8s/K8sPlugin.groovy | 2 +- .../main/nextflow/k8s/K8sTaskHandler.groovy | 8 +- .../nextflow/k8s/K8sWrapperBuilder.groovy | 2 +- .../nextflow/k8s/cli/KubeCommandImpl.groovy | 4 +- .../nextflow/k8s/client/ClientConfig.groovy | 2 +- .../k8s/client/ConfigDiscovery.groovy | 2 +- .../main/nextflow/k8s/client/K8sClient.groovy | 10 +- .../nextflow/k8s/client/K8sResponseApi.groovy | 2 +- .../k8s/client/K8sResponseException.groovy | 2 +- .../k8s/client/K8sResponseJson.groovy | 2 +- .../nextflow/k8s/client/K8sRetryConfig.groovy | 5 +- .../client/PodUnschedulableException.groovy | 2 +- .../main/nextflow/k8s/client/SSLUtils.java | 2 +- .../src/main/nextflow/k8s/model/PodEnv.groovy | 2 +- .../nextflow/k8s/model/PodHostMount.groovy | 2 +- .../nextflow/k8s/model/PodMountConfig.groovy | 2 +- .../k8s/model/PodMountCsiEphemeral.groovy | 2 +- .../k8s/model/PodMountEmptyDir.groovy | 2 +- .../nextflow/k8s/model/PodMountSecret.groovy | 2 +- .../nextflow/k8s/model/PodNodeSelector.groovy | 2 +- .../main/nextflow/k8s/model/PodOptions.groovy | 2 +- .../k8s/model/PodSecurityContext.groovy | 2 +- .../nextflow/k8s/model/PodSpecBuilder.groovy | 2 +- .../nextflow/k8s/model/PodVolumeClaim.groovy | 2 +- .../nextflow/k8s/model/ResourceType.groovy | 2 +- .../test/nextflow/k8s/K8sConfigTest.groovy | 6 +- .../nextflow/k8s/K8sDriverLauncherTest.groovy | 8 +- .../nextflow/k8s/K8sTaskHandlerTest.groovy | 6 +- .../nextflow/k8s/K8sWrapperBuilderTest.groovy | 3 +- .../k8s/client/ClientConfigTest.groovy | 2 +- .../k8s/client/ConfigDiscoveryTest.groovy | 4 +- .../nextflow/k8s/client/K8sClientTest.groovy | 26 +- .../client/K8sResponseExceptionTest.groovy | 4 +- .../k8s/client/K8sResponseJsonTest.groovy | 4 +- .../test/nextflow/k8s/model/PodEnvTest.groovy | 2 +- .../k8s/model/PodMountConfigTest.groovy | 2 +- .../k8s/model/PodMountSecretTest.groovy | 2 +- .../k8s/model/PodNodeSelectorTest.groovy | 4 +- .../nextflow/k8s/model/PodOptionsTest.groovy | 8 +- .../k8s/model/PodSpecBuilderTest.groovy | 2 +- .../k8s/model/PodVolumeClaimTest.groovy | 2 +- plugins/nf-seqera/build.gradle | 17 +- .../main/io/seqera/config/ExecutorOpts.groovy | 3 +- .../config/MachineRequirementOpts.groovy | 5 +- .../main/io/seqera/config/RetryOpts.groovy | 3 +- .../main/io/seqera/config/SeqeraConfig.groovy | 3 +- .../seqera/executor/InputFilesProfiler.groovy | 1 - .../src/main/io/seqera/executor/Labels.groovy | 3 +- .../executor/SeqeraBatchSubmitter.groovy | 1 - .../io/seqera/executor/SeqeraExecutor.groovy | 3 +- .../seqera/executor/SeqeraTaskHandler.groovy | 3 +- .../main/io/seqera/plugin/SeqeraPlugin.groovy | 3 +- .../io/seqera/util/SchemaMapperUtil.groovy | 3 +- .../io/seqera/config/ExecutorOptsTest.groovy | 3 +- .../config/MachineRequirementOptsTest.groovy | 5 +- .../io/seqera/config/RetryOptsTest.groovy | 3 +- .../io/seqera/config/SeqeraConfigTest.groovy | 3 +- .../executor/InputFilesProfilerTest.groovy | 1 - .../test/io/seqera/executor/LabelsTest.groovy | 3 +- .../executor/SeqeraBatchSubmitterTest.groovy | 1 - .../seqera/executor/SeqeraExecutorTest.groovy | 3 +- .../executor/SeqeraTaskHandlerTest.groovy | 3 +- .../test/io/seqera/util/MapperUtilTest.groovy | 3 +- plugins/nf-tower/build.gradle | 17 +- .../tower/plugin/BaseCommandImpl.groovy | 16 + .../seqera/tower/plugin/CacheCommand.groovy | 3 +- .../seqera/tower/plugin/CacheManager.groovy | 3 +- .../seqera/tower/plugin/LogsCheckpoint.groovy | 3 +- .../io/seqera/tower/plugin/LogsHandler.groovy | 3 +- .../io/seqera/tower/plugin/TowerClient.groovy | 3 +- .../io/seqera/tower/plugin/TowerConfig.groovy | 3 +- .../seqera/tower/plugin/TowerFactory.groovy | 3 +- .../tower/plugin/TowerFusionToken.groovy | 16 + .../tower/plugin/TowerJsonGenerator.groovy | 3 +- .../io/seqera/tower/plugin/TowerPlugin.groovy | 3 +- .../seqera/tower/plugin/TowerReports.groovy | 3 +- .../tower/plugin/TowerRetryPolicy.groovy | 10 +- .../io/seqera/tower/plugin/TowerXAuth.groovy | 3 +- .../tower/plugin/WorkflowProgress.groovy | 3 +- .../tower/plugin/auth/AuthCommandImpl.groovy | 16 + .../exception/BadResponseException.groovy | 16 + .../exception/UnauthorizedException.groovy | 16 + .../exchange/GetLicenseTokenRequest.groovy | 16 + .../exchange/GetLicenseTokenResponse.groovy | 16 + .../plugin/launch/LaunchCommandImpl.groovy | 2 +- .../nextflow.trace.TraceObserverFactory | 3 +- .../src/resources/tower-schema.properties | 3 +- .../tower/plugin/CacheManagerTest.groovy | 3 +- .../tower/plugin/LogsCheckpointTest.groovy | 3 +- .../tower/plugin/LogsHandlerTest.groovy | 3 +- .../tower/plugin/TowerClientTest.groovy | 7 +- .../tower/plugin/TowerFactoryTest.groovy | 3 +- .../tower/plugin/TowerFusionEnvTest.groovy | 18 +- .../plugin/TowerJsonGeneratorTest.groovy | 3 +- .../tower/plugin/TowerReportsTest.groovy | 3 +- .../tower/plugin/TowerRetryPolicyTest.groovy | 8 +- .../plugin/auth/AuthCommandImplTest.groovy | 2 +- .../launch/LaunchCommandImplTest.groovy | 2 +- plugins/nf-wave/build.gradle | 17 +- .../seqera/wave/plugin/ContainerConfig.groovy | 5 +- .../seqera/wave/plugin/ContainerLayer.groovy | 3 +- .../plugin/DescribeContainerResponse.groovy | 3 +- .../plugin/SubmitContainerTokenRequest.groovy | 3 +- .../SubmitContainerTokenResponse.groovy | 3 +- .../io/seqera/wave/plugin/WaveAssets.groovy | 5 +- .../io/seqera/wave/plugin/WaveClient.groovy | 3 +- .../io/seqera/wave/plugin/WaveFactory.groovy | 3 +- .../io/seqera/wave/plugin/WavePlugin.groovy | 3 +- .../wave/plugin/adapter/InstantAdapter.groovy | 5 +- .../wave/plugin/cli/WaveCmdEntry.groovy | 3 +- .../wave/plugin/cli/WaveDebugCmd.groovy | 3 +- .../seqera/wave/plugin/cli/WaveRunCmd.groovy | 3 +- .../seqera/wave/plugin/config/HttpOpts.groovy | 3 +- .../wave/plugin/config/ReportOpts.groovy | 3 +- .../wave/plugin/config/RetryOpts.groovy | 3 +- .../wave/plugin/config/TowerConfig.groovy | 3 +- .../wave/plugin/config/WaveConfig.groovy | 7 +- .../exception/BadResponseException.groovy | 3 +- .../exception/UnauthorizedException.groovy | 3 +- .../seqera/wave/plugin/packer/Packer.groovy | 5 +- .../resolver/WaveContainerResolver.groovy | 3 +- .../wave/plugin/util/BasicCliOpts.groovy | 5 +- .../io/seqera/wave/plugin/util/CliOpts.groovy | 3 +- .../wave/plugin/util/DigestFunctions.java | 3 +- .../seqera/wave/plugin/util/GnuCliOpts.groovy | 5 +- .../wave/plugin/ContainerConfigTest.groovy | 3 +- .../wave/plugin/ContainerLayerTest.groovy | 5 +- .../seqera/wave/plugin/WaveAssetsTest.groovy | 5 +- .../seqera/wave/plugin/WaveClientTest.groovy | 7 +- .../seqera/wave/plugin/WaveFactoryTest.groovy | 5 +- .../wave/plugin/cli/WaveCmdEntryTest.groovy | 5 +- .../wave/plugin/cli/WaveDebugCmdTest.groovy | 3 +- .../wave/plugin/config/HttpOptsTest.groovy | 3 +- .../wave/plugin/config/RetryOptsTest.groovy | 3 +- .../wave/plugin/config/TowerConfigTest.groovy | 3 +- .../wave/plugin/config/WaveConfigTest.groovy | 3 +- .../wave/plugin/packer/PackerTest.groovy | 3 +- .../wave/plugin/packer/TarHelper.groovy | 3 +- .../resolver/WaveContainerResolverTest.groovy | 3 +- .../wave/plugin/util/BasicCliOptsTest.groovy | 5 +- .../wave/plugin/util/GnuCliOptsTest.groovy | 3 +- .../nextflow/processor/TaskRunTest2.groovy | 3 +- settings.gradle | 2 +- test-ci.sh | 16 + test-e2e/run.sh | 3 +- 1573 files changed, 4907 insertions(+), 3648 deletions(-) create mode 100644 gradle/codenarc.groovy diff --git a/build.gradle b/build.gradle index 31d5ad9966..91858a5323 100644 --- a/build.gradle +++ b/build.gradle @@ -1,5 +1,5 @@ /* - * Copyright 2013-2024, Seqera Labs + * Copyright 2013-2026, Seqera Labs * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -350,29 +350,29 @@ def getRuntimeConfigs() { */ task exportClasspath { dependsOn allprojects.jar - + // Use provider to delay configuration resolution until task execution def configurationFiles = provider { def libs = [] - + // Resolve configurations during provider evaluation (not ideal but functional) ['nextflow','nf-commons','nf-httpfs','nf-lang','nf-lineage'].each { moduleName -> def moduleProject = project(":$moduleName") def cfg = moduleProject.configurations.getByName('runtimeClasspath') libs.addAll(cfg.files.collect { it.canonicalPath }) } - + // Add module jars ['nextflow','nf-commons','nf-httpfs','nf-lang','nf-lineage'].each { libs << file("modules/$it/build/libs/${it}-${version}.jar").canonicalPath } - + return libs.unique() } - + inputs.files(configurationFiles) outputs.file('.launch.classpath') - + doLast { def libs = configurationFiles.get() file('.launch.classpath').text = libs.join(':') diff --git a/compile.sh b/compile.sh index 1a47271ddf..c1bd61d56f 100755 --- a/compile.sh +++ b/compile.sh @@ -1,2 +1,18 @@ #!/bin/bash +# +# Copyright 2013-2026, Seqera Labs +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + ./gradlew -q compile exportClasspath diff --git a/config/codenarc/codenarc.groovy b/config/codenarc/codenarc.groovy index bd082baef6..44be5a7b98 100644 --- a/config/codenarc/codenarc.groovy +++ b/config/codenarc/codenarc.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2013-2024, Seqera Labs + * Copyright 2013-2026, Seqera Labs * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -23,329 +23,329 @@ ruleset { ''' // rulesets/basic.xml - AssertWithinFinallyBlock - AssignmentInConditional - BigDecimalInstantiation - BitwiseOperatorInConditional - BooleanGetBoolean - BrokenNullCheck - BrokenOddnessCheck - ClassForName - ComparisonOfTwoConstants - ComparisonWithSelf - ConstantAssertExpression - ConstantIfExpression - ConstantTernaryExpression - DeadCode - DoubleNegative - DuplicateCaseStatement - DuplicateMapKey - DuplicateSetValue - EmptyCatchBlock - EmptyElseBlock - EmptyFinallyBlock - EmptyForStatement - EmptyIfStatement - EmptyInstanceInitializer - EmptyMethod - EmptyStaticInitializer - EmptySwitchStatement - EmptySynchronizedStatement - EmptyTryBlock - EmptyWhileStatement - EqualsAndHashCode - EqualsOverloaded - ExplicitGarbageCollection - ForLoopShouldBeWhileLoop - HardCodedWindowsFileSeparator - HardCodedWindowsRootDirectory - IntegerGetInteger - RandomDoubleCoercedToZero - RemoveAllOnSelf - ReturnFromFinallyBlock - ThrowExceptionFromFinallyBlock - + AssertWithinFinallyBlock + AssignmentInConditional + BigDecimalInstantiation + BitwiseOperatorInConditional + BooleanGetBoolean + BrokenNullCheck + BrokenOddnessCheck + ClassForName + ComparisonOfTwoConstants + ComparisonWithSelf + ConstantAssertExpression + ConstantIfExpression + ConstantTernaryExpression + DeadCode + DoubleNegative + DuplicateCaseStatement + DuplicateMapKey + DuplicateSetValue + EmptyCatchBlock + EmptyElseBlock + EmptyFinallyBlock + EmptyForStatement + EmptyIfStatement + EmptyInstanceInitializer + EmptyMethod + EmptyStaticInitializer + EmptySwitchStatement + EmptySynchronizedStatement + EmptyTryBlock + EmptyWhileStatement + EqualsAndHashCode + EqualsOverloaded + ExplicitGarbageCollection + ForLoopShouldBeWhileLoop + HardCodedWindowsFileSeparator + HardCodedWindowsRootDirectory + IntegerGetInteger + RandomDoubleCoercedToZero + RemoveAllOnSelf + ReturnFromFinallyBlock + ThrowExceptionFromFinallyBlock + // rulesets/braces.xml //ElseBlockBraces - ForStatementBraces + ForStatementBraces //IfStatementBraces //WhileStatementBraces - + // rulesets/concurrency.xml - BusyWait - DoubleCheckedLocking - InconsistentPropertyLocking - InconsistentPropertySynchronization - NestedSynchronization - StaticCalendarField - StaticConnection - StaticDateFormatField - StaticMatcherField - StaticSimpleDateFormatField - SynchronizedMethod - SynchronizedOnBoxedPrimitive - SynchronizedOnGetClass - SynchronizedOnReentrantLock - SynchronizedOnString - SynchronizedOnThis - SynchronizedReadObjectMethod - SystemRunFinalizersOnExit - ThreadGroup - ThreadLocalNotStaticFinal - ThreadYield - UseOfNotifyMethod - VolatileArrayField - VolatileLongOrDoubleField - WaitOutsideOfWhileLoop - + BusyWait + DoubleCheckedLocking + InconsistentPropertyLocking + InconsistentPropertySynchronization + NestedSynchronization + StaticCalendarField + StaticConnection + StaticDateFormatField + StaticMatcherField + StaticSimpleDateFormatField + SynchronizedMethod + SynchronizedOnBoxedPrimitive + SynchronizedOnGetClass + SynchronizedOnReentrantLock + SynchronizedOnString + SynchronizedOnThis + SynchronizedReadObjectMethod + SystemRunFinalizersOnExit + ThreadGroup + ThreadLocalNotStaticFinal + ThreadYield + UseOfNotifyMethod + VolatileArrayField + VolatileLongOrDoubleField + WaitOutsideOfWhileLoop + // rulesets/convention.xml - ConfusingTernary - CouldBeElvis - HashtableIsObsolete - IfStatementCouldBeTernary - InvertedIfElse - LongLiteralWithLowerCaseL - ParameterReassignment - TernaryCouldBeElvis - VectorIsObsolete - + ConfusingTernary + CouldBeElvis + HashtableIsObsolete + IfStatementCouldBeTernary + InvertedIfElse + LongLiteralWithLowerCaseL + ParameterReassignment + TernaryCouldBeElvis + VectorIsObsolete + // rulesets/design.xml - AbstractClassWithPublicConstructor - AbstractClassWithoutAbstractMethod - BooleanMethodReturnsNull - BuilderMethodWithSideEffects - CloneableWithoutClone - CloseWithoutCloseable - CompareToWithoutComparable - ConstantsOnlyInterface - EmptyMethodInAbstractClass - FinalClassWithProtectedMember - ImplementationAsType - PrivateFieldCouldBeFinal - PublicInstanceField - ReturnsNullInsteadOfEmptyArray - ReturnsNullInsteadOfEmptyCollection - SimpleDateFormatMissingLocale - StatelessSingleton - + AbstractClassWithPublicConstructor + AbstractClassWithoutAbstractMethod + BooleanMethodReturnsNull + BuilderMethodWithSideEffects + CloneableWithoutClone + CloseWithoutCloseable + CompareToWithoutComparable + ConstantsOnlyInterface + EmptyMethodInAbstractClass + FinalClassWithProtectedMember + ImplementationAsType + PrivateFieldCouldBeFinal + PublicInstanceField + ReturnsNullInsteadOfEmptyArray + ReturnsNullInsteadOfEmptyCollection + SimpleDateFormatMissingLocale + StatelessSingleton + // rulesets/dry.xml - DuplicateListLiteral - DuplicateMapLiteral - DuplicateNumberLiteral - DuplicateStringLiteral - + DuplicateListLiteral + DuplicateMapLiteral + DuplicateNumberLiteral + DuplicateStringLiteral + // rulesets/exceptions.xml - CatchArrayIndexOutOfBoundsException - CatchError - CatchException - CatchIllegalMonitorStateException - CatchIndexOutOfBoundsException - CatchNullPointerException - CatchRuntimeException - CatchThrowable - ConfusingClassNamedException - ExceptionExtendsError - ExceptionNotThrown - MissingNewInThrowStatement - ReturnNullFromCatchBlock - SwallowThreadDeath - ThrowError - ThrowException - ThrowNullPointerException - ThrowRuntimeException - ThrowThrowable - + CatchArrayIndexOutOfBoundsException + CatchError + CatchException + CatchIllegalMonitorStateException + CatchIndexOutOfBoundsException + CatchNullPointerException + CatchRuntimeException + CatchThrowable + ConfusingClassNamedException + ExceptionExtendsError + ExceptionNotThrown + MissingNewInThrowStatement + ReturnNullFromCatchBlock + SwallowThreadDeath + ThrowError + ThrowException + ThrowNullPointerException + ThrowRuntimeException + ThrowThrowable + // rulesets/generic.xml - IllegalClassReference - IllegalPackageReference - IllegalRegex - RequiredRegex - RequiredString - StatelessClass - + IllegalClassReference + IllegalPackageReference + IllegalRegex + RequiredRegex + RequiredString + StatelessClass + // rulesets/grails.xml - GrailsDomainHasEquals - GrailsDomainHasToString - GrailsDuplicateConstraint - GrailsDuplicateMapping - GrailsPublicControllerMethod - GrailsServletContextReference + GrailsDomainHasEquals + GrailsDomainHasToString + GrailsDuplicateConstraint + GrailsDuplicateMapping + GrailsPublicControllerMethod + GrailsServletContextReference GrailsSessionReference // DEPRECATED - GrailsStatelessService - + GrailsStatelessService + // rulesets/groovyism.xml - AssignCollectionSort - AssignCollectionUnique - ClosureAsLastMethodParameter - CollectAllIsDeprecated - ConfusingMultipleReturns - ExplicitArrayListInstantiation - ExplicitCallToAndMethod - ExplicitCallToCompareToMethod - ExplicitCallToDivMethod - ExplicitCallToEqualsMethod - ExplicitCallToGetAtMethod - ExplicitCallToLeftShiftMethod - ExplicitCallToMinusMethod - ExplicitCallToModMethod - ExplicitCallToMultiplyMethod - ExplicitCallToOrMethod - ExplicitCallToPlusMethod - ExplicitCallToPowerMethod - ExplicitCallToRightShiftMethod - ExplicitCallToXorMethod - ExplicitHashMapInstantiation - ExplicitHashSetInstantiation - ExplicitLinkedHashMapInstantiation - ExplicitLinkedListInstantiation - ExplicitStackInstantiation - ExplicitTreeSetInstantiation - GStringAsMapKey - GetterMethodCouldBeProperty - GroovyLangImmutable - UseCollectMany - UseCollectNested - + AssignCollectionSort + AssignCollectionUnique + ClosureAsLastMethodParameter + CollectAllIsDeprecated + ConfusingMultipleReturns + ExplicitArrayListInstantiation + ExplicitCallToAndMethod + ExplicitCallToCompareToMethod + ExplicitCallToDivMethod + ExplicitCallToEqualsMethod + ExplicitCallToGetAtMethod + ExplicitCallToLeftShiftMethod + ExplicitCallToMinusMethod + ExplicitCallToModMethod + ExplicitCallToMultiplyMethod + ExplicitCallToOrMethod + ExplicitCallToPlusMethod + ExplicitCallToPowerMethod + ExplicitCallToRightShiftMethod + ExplicitCallToXorMethod + ExplicitHashMapInstantiation + ExplicitHashSetInstantiation + ExplicitLinkedHashMapInstantiation + ExplicitLinkedListInstantiation + ExplicitStackInstantiation + ExplicitTreeSetInstantiation + GStringAsMapKey + GetterMethodCouldBeProperty + GroovyLangImmutable + UseCollectMany + UseCollectNested + // rulesets/imports.xml - DuplicateImport - ImportFromSamePackage - ImportFromSunPackages - MisorderedStaticImports - UnnecessaryGroovyImport - UnusedImport - + DuplicateImport + ImportFromSamePackage + ImportFromSunPackages + MisorderedStaticImports + UnnecessaryGroovyImport + UnusedImport + // rulesets/jdbc.xml - DirectConnectionManagement - JdbcConnectionReference - JdbcResultSetReference - JdbcStatementReference - + DirectConnectionManagement + JdbcConnectionReference + JdbcResultSetReference + JdbcStatementReference + // rulesets/junit.xml - ChainedTest - CoupledTestCase - JUnitAssertAlwaysFails - JUnitAssertAlwaysSucceeds - JUnitFailWithoutMessage - JUnitLostTest - JUnitPublicNonTestMethod - JUnitSetUpCallsSuper - JUnitStyleAssertions - JUnitTearDownCallsSuper - JUnitTestMethodWithoutAssert - JUnitUnnecessarySetUp - JUnitUnnecessaryTearDown - JUnitUnnecessaryThrowsException - SpockIgnoreRestUsed - UnnecessaryFail - UseAssertEqualsInsteadOfAssertTrue - UseAssertFalseInsteadOfNegation - UseAssertNullInsteadOfAssertEquals - UseAssertSameInsteadOfAssertTrue - UseAssertTrueInsteadOfAssertEquals - UseAssertTrueInsteadOfNegation - + ChainedTest + CoupledTestCase + JUnitAssertAlwaysFails + JUnitAssertAlwaysSucceeds + JUnitFailWithoutMessage + JUnitLostTest + JUnitPublicNonTestMethod + JUnitSetUpCallsSuper + JUnitStyleAssertions + JUnitTearDownCallsSuper + JUnitTestMethodWithoutAssert + JUnitUnnecessarySetUp + JUnitUnnecessaryTearDown + JUnitUnnecessaryThrowsException + SpockIgnoreRestUsed + UnnecessaryFail + UseAssertEqualsInsteadOfAssertTrue + UseAssertFalseInsteadOfNegation + UseAssertNullInsteadOfAssertEquals + UseAssertSameInsteadOfAssertTrue + UseAssertTrueInsteadOfAssertEquals + UseAssertTrueInsteadOfNegation + // rulesets/logging.xml - LoggerForDifferentClass - LoggerWithWrongModifiers - LoggingSwallowsStacktrace - MultipleLoggers - PrintStackTrace - Println - SystemErrPrint - SystemOutPrint - + LoggerForDifferentClass + LoggerWithWrongModifiers + LoggingSwallowsStacktrace + MultipleLoggers + PrintStackTrace + Println + SystemErrPrint + SystemOutPrint + // rulesets/naming.xml - AbstractClassName - ClassName - ConfusingMethodName - FactoryMethodName - FieldName - InterfaceName - MethodName - ObjectOverrideMisspelledMethodName - PackageName - ParameterName - PropertyName + AbstractClassName + ClassName + ConfusingMethodName + FactoryMethodName + FieldName + InterfaceName + MethodName + ObjectOverrideMisspelledMethodName + PackageName + ParameterName + PropertyName //VariableName - + // rulesets/security.xml - FileCreateTempFile - InsecureRandom - NonFinalPublicField - NonFinalSubclassOfSensitiveInterface - ObjectFinalize - PublicFinalizeMethod - SystemExit - UnsafeArrayDeclaration - + FileCreateTempFile + InsecureRandom + NonFinalPublicField + NonFinalSubclassOfSensitiveInterface + ObjectFinalize + PublicFinalizeMethod + SystemExit + UnsafeArrayDeclaration + // rulesets/serialization.xml - SerialPersistentFields - SerialVersionUID - SerializableClassMustDefineSerialVersionUID - + SerialPersistentFields + SerialVersionUID + SerializableClassMustDefineSerialVersionUID + // rulesets/size.xml AbcComplexity // DEPRECATED: Use the AbcMetric rule instead. Requires the GMetrics jar AbcMetric // Requires the GMetrics jar - ClassSize + ClassSize CrapMetric // Requires the GMetrics jar and a Cobertura coverage file CyclomaticComplexity // Requires the GMetrics jar - MethodCount - MethodSize - NestedBlockDepth - + MethodCount + MethodSize + NestedBlockDepth + // rulesets/unnecessary.xml - AddEmptyString - ConsecutiveLiteralAppends - ConsecutiveStringConcatenation - UnnecessaryBigDecimalInstantiation - UnnecessaryBigIntegerInstantiation - UnnecessaryBooleanExpression - UnnecessaryBooleanInstantiation - UnnecessaryCallForLastElement - UnnecessaryCallToSubstring - UnnecessaryCatchBlock - UnnecessaryCollectCall - UnnecessaryCollectionCall - UnnecessaryConstructor - UnnecessaryDefInFieldDeclaration + AddEmptyString + ConsecutiveLiteralAppends + ConsecutiveStringConcatenation + UnnecessaryBigDecimalInstantiation + UnnecessaryBigIntegerInstantiation + UnnecessaryBooleanExpression + UnnecessaryBooleanInstantiation + UnnecessaryCallForLastElement + UnnecessaryCallToSubstring + UnnecessaryCatchBlock + UnnecessaryCollectCall + UnnecessaryCollectionCall + UnnecessaryConstructor + UnnecessaryDefInFieldDeclaration //UnnecessaryDefInMethodDeclaration - UnnecessaryDefInVariableDeclaration - UnnecessaryDotClass - UnnecessaryDoubleInstantiation - UnnecessaryElseStatement - UnnecessaryFinalOnPrivateMethod - UnnecessaryFloatInstantiation + UnnecessaryDefInVariableDeclaration + UnnecessaryDotClass + UnnecessaryDoubleInstantiation + UnnecessaryElseStatement + UnnecessaryFinalOnPrivateMethod + UnnecessaryFloatInstantiation //UnnecessaryGString //UnnecessaryGetter - UnnecessaryIfStatement - UnnecessaryInstanceOfCheck - UnnecessaryInstantiationToGetClass - UnnecessaryIntegerInstantiation - UnnecessaryLongInstantiation - UnnecessaryModOne - UnnecessaryNullCheck - UnnecessaryNullCheckBeforeInstanceOf - UnnecessaryObjectReferences - UnnecessaryOverridingMethod - UnnecessaryPackageReference - UnnecessaryParenthesesForMethodCallWithClosure - UnnecessaryPublicModifier - UnnecessaryReturnKeyword - UnnecessarySelfAssignment - UnnecessarySemicolon - UnnecessaryStringInstantiation - UnnecessarySubstring - UnnecessaryTernaryExpression - UnnecessaryTransientModifier - + UnnecessaryIfStatement + UnnecessaryInstanceOfCheck + UnnecessaryInstantiationToGetClass + UnnecessaryIntegerInstantiation + UnnecessaryLongInstantiation + UnnecessaryModOne + UnnecessaryNullCheck + UnnecessaryNullCheckBeforeInstanceOf + UnnecessaryObjectReferences + UnnecessaryOverridingMethod + UnnecessaryPackageReference + UnnecessaryParenthesesForMethodCallWithClosure + UnnecessaryPublicModifier + UnnecessaryReturnKeyword + UnnecessarySelfAssignment + UnnecessarySemicolon + UnnecessaryStringInstantiation + UnnecessarySubstring + UnnecessaryTernaryExpression + UnnecessaryTransientModifier + // rulesets/unused.xml - UnusedArray - UnusedMethodParameter - UnusedObject - UnusedPrivateField - UnusedPrivateMethod - UnusedPrivateMethodParameter - UnusedVariable - - + UnusedArray + UnusedMethodParameter + UnusedObject + UnusedPrivateField + UnusedPrivateMethod + UnusedPrivateMethodParameter + UnusedVariable + + } diff --git a/config/codenarc/codenarc.xml b/config/codenarc/codenarc.xml index aeffe439b9..73d5bbf6fd 100644 --- a/config/codenarc/codenarc.xml +++ b/config/codenarc/codenarc.xml @@ -1,5 +1,5 @@ (*) ModuleReference - | - v +PipelineSpec (1) -----> (*) ModuleReference (nextflow_spec.json) +ModulesConfig (1) -----> (*) ModuleReference (nextflow.config alternative) RegistryConfig (1) -----> (*) Registry URLs ModuleReference (1) -----> (0..1) InstalledModule | v (via registry) -ModuleInfo (1) -----> (1) ModuleManifest - -InstalledModule (1) -----> (1) ModuleManifest - -----> (*) ToolDefinition - -----> (*) ArgDefinition - -ToolArgsContext (1) -----> (*) ToolArgs - -----> (*) ArgDefinition (schema) +ModuleSpec (1) <----- InstalledModule (from meta.yaml) ``` --- @@ -287,15 +244,16 @@ ToolArgsContext (1) -----> (*) ToolArgs ``` project-root/ -├── nextflow.config # modules{}, registry{} blocks +├── nextflow.config # registry{} block; optional modules{} block +├── nextflow_spec.json # auto-managed module version pins ├── main.nf # include { X } from '@scope/name' └── modules/ └── @scope/ └── name/ - ├── .checksum # SHA-256 from registry + ├── .checksum # SHA-256 from registry (download integrity) ├── main.nf # Entry point (required) - ├── meta.yaml # Manifest (optional but recommended) - ├── README.md # Documentation + ├── meta.yaml # Manifest (required for publishing) + ├── README.md # Documentation (required for publishing) └── [other files] # Supporting files ``` @@ -306,9 +264,9 @@ project-root/ | Entity | Field | Validation | |--------|-------|------------| | ModuleReference | fullName | Pattern: `^@[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$` | -| ModuleManifest | version | SemVer: `MAJOR.MINOR.PATCH` | -| ArgDefinition | type | Enum: boolean, integer, float, string, file, path | -| ArgDefinition | enumValues | If set, value must be member | +| ModuleSpec | name | Pattern: `scope/name` or `scope/path/to/name` | +| ModuleSpec | version | SemVer: `MAJOR.MINOR.PATCH[-prerelease]` | +| ModuleSpec | description, license | Required (non-empty) | | InstalledModule | directory | Must contain main.nf | -| ModuleConfig | modules | Keys must be valid module references | +| ModulesConfig | modules keys | Must be valid module fullName | | RegistryConfig | url | Valid HTTPS URL | \ No newline at end of file diff --git a/specs/251117-module-system/plan.md b/specs/251117-module-system/plan.md index 0d12cf3f96..e9d11d552f 100644 --- a/specs/251117-module-system/plan.md +++ b/specs/251117-module-system/plan.md @@ -15,6 +15,7 @@ Implement client-side module system for Nextflow enabling pipeline developers to - Existing config parser (ConfigBuilder, ConfigParser) - Existing HTTP client (HxClient from io.seqera.http) - Existing plugin authentication infrastructure +- Existing npr-api (registry data models and schema validation) **Storage**: Local filesystem (`modules/@scope/name/` per-project, `.checksum` files) **Testing**: Spock Framework for unit tests, integration tests in `tests/` directory **Target Platform**: JVM 17+ (same as Nextflow core) @@ -49,6 +50,7 @@ Implement client-side module system for Nextflow enabling pipeline developers to ```text specs/251117-module-system/ ├── plan.md # This file +├── spec.md # Feature specification ├── research.md # Phase 0 output ├── data-model.md # Phase 1 output ├── quickstart.md # Phase 1 output @@ -61,40 +63,74 @@ specs/251117-module-system/ ```text modules/nextflow/src/main/groovy/nextflow/ ├── cli/ -│ └── CmdModule.groovy # NEW: Module CLI command +│ ├── CmdModule.groovy # Main module command (uses JCommander) +│ └── module/ +│ ├── ModuleInstall.groovy # Install subcommand (extends CmdBase) +│ ├── ModuleRun.groovy # Run subcommand (extends CmdRun) +│ ├── ModuleList.groovy # List subcommand (extends CmdBase) +│ ├── ModuleRemove.groovy # Remove subcommand (extends CmdBase) +│ ├── ModuleSearch.groovy # Search subcommand (extends CmdBase) +│ ├── ModuleInfo.groovy # Info subcommand (extends CmdBase) +│ └── ModulePublish.groovy # Publish subcommand (extends CmdBase) ├── config/ -│ ├── ConfigBuilder.groovy # MODIFY: Add modules/registry DSL -│ └── parser/v1/ -│ ├── ModulesDsl.groovy # NEW: modules {} block parser -│ └── RegistryDsl.groovy # NEW: registry {} block parser -└── module/ - ├── ModuleResolver.groovy # NEW: Core resolution logic - ├── ModuleStorage.groovy # NEW: Local storage management - ├── ModuleChecksum.groovy # NEW: Checksum verification - ├── ModuleManifest.groovy # NEW: meta.yaml parser - └── HttpModuleRepository.groovy # NEW: Registry HTTP client +│ ├── ModulesConfig.groovy # modules{} config scope +│ └── RegistryConfig.groovy # registry{} config scope (fields: url, apiKey) +├── module/ +│ ├── ModuleReference.groovy # @scope/name parser +│ ├── ModuleResolver.groovy # Core resolution logic (version/integrity/install) +│ ├── ModuleStorage.groovy # Local filesystem operations +│ ├── ModuleRegistryClient.groovy # HTTP registry client +│ ├── ModuleChecksum.groovy # SHA-256 integrity verification +│ ├── ModuleSpec.groovy # Module manifest (meta.yaml) entity +│ ├── InstalledModule.groovy # Installed module entity +│ └── DefaultRemoteModuleResolver.groovy # SPI impl: bridges DSL parser → ModuleResolver +└── pipeline/ + └── PipelineSpec.groovy # nextflow_spec.json read/write modules/nf-lang/src/main/java/nextflow/script/ -└── ResolveIncludeVisitor.java # MODIFY: Add @scope/name detection +└── control/ResolveIncludeVisitor.java # MODIFIED: Delegates @scope/name to SPI resolver + +modules/nf-lang/src/main/java/nextflow/module/spi/ +├── RemoteModuleResolver.java # SPI interface (extensible by plugins) +├── RemoteModuleResolverProvider.java # ServiceLoader wrapper (singleton) +└── FallbackRemoteModuleResolver.java # Error fallback when no impl found + modules/nextflow/src/test/groovy/nextflow/ -├── cli/ -│ └── CmdModuleTest.groovy # NEW: CLI unit tests -├── config/ -│ └── ModulesDslTest.groovy # NEW: Config parsing tests +├── cli/module/ +│ ├── ModuleInstallTest.groovy +│ ├── ModuleRunTest.groovy +│ └── [other subcommand tests] └── module/ - ├── ModuleResolverTest.groovy # NEW: Resolution logic tests - ├── ModuleStorageTest.groovy # NEW: Storage tests - └── ModuleChecksumTest.groovy # NEW: Checksum tests - -tests/ -└── modules/ # NEW: Integration tests - ├── install-module.nf # Test module install + include - ├── version-resolution.nf # Test version management - └── checksum-protection.nf # Test local modification protection + ├── ModuleResolverTest.groovy + ├── ModuleStorageTest.groovy + └── [other module tests] + +tests/modules/ +├── install-module.nf # Integration tests +├── run-module.nf +└── [other integration tests] +``` + +**Structure Decision**: Implementation extends existing Nextflow core modules following modular architecture. New code in `modules/nextflow` for CLI and core logic. DSL parser extension in `modules/nf-lang` via SPI. No new plugins required. + +## Architecture Notes + +### Remote Module Inclusion — SPI Pattern + +The DSL parser (`ResolveIncludeVisitor`) detects the `@` prefix in `include` statements and delegates resolution to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader`. This keeps `nf-lang` decoupled from the runtime module resolution logic: + +``` +include { X } from '@nf-core/fastqc' + ↓ +ResolveIncludeVisitor (nf-lang) + source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(...) + ↓ +DefaultRemoteModuleResolver (nextflow module) + auto-installs via ModuleResolver if missing → returns Path to main.nf ``` -**Structure Decision**: Implementation extends existing Nextflow core modules following modular architecture. New code in `modules/nextflow` for CLI and core logic. DSL parser extension in `modules/nf-lang`. No new plugins required. +The `RemoteModuleResolver` interface in `nf-lang` can be overridden by plugins with a higher priority value. ## Complexity Tracking diff --git a/specs/251117-module-system/quickstart.md b/specs/251117-module-system/quickstart.md index f9247dcb74..71e356e534 100644 --- a/specs/251117-module-system/quickstart.md +++ b/specs/251117-module-system/quickstart.md @@ -22,7 +22,7 @@ nextflow module install nf-core/fastqc nextflow module install nf-core/fastqc -version 1.0.0 ``` -This downloads the module to `modules/@nf-core/fastqc/` and updates `nextflow.config`. +This downloads the module to `modules/@nf-core/fastqc/` and updates `nextflow_spec.json` with the installed version. ### Use in your workflow @@ -52,11 +52,8 @@ Execute a module without writing a wrapper workflow: # Basic usage nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -# With tool arguments -nextflow module run nf-core/bwa-align \ - --reads 'samples/*_{1,2}.fastq.gz' \ - --reference genome.fa \ - --tools:bwa:K 100000000 +# Run specific version +nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0 # With Nextflow options nextflow module run nf-core/salmon \ @@ -66,18 +63,43 @@ nextflow module run nf-core/salmon \ -resume ``` +## 3. View Module Information + +```bash +# Show module metadata and a generated usage template +nextflow module info nf-core/fastqc + +# Show a specific version +nextflow module info nf-core/fastqc -version 1.0.0 + +# JSON output for scripting +nextflow module info nf-core/fastqc -json +``` + --- -## 3. Manage Module Versions +## 4. Manage Module Versions -### Configure versions in nextflow.config +### Version tracking -```groovy -// nextflow.config +Module versions are automatically recorded in `nextflow_spec.json` by `nextflow module install`. You can also pin versions manually: + +```json +// nextflow_spec.json +{ + "modules": { + "@nf-core/fastqc": "1.0.0", + "@nf-core/bwa-align": "1.2.0" + } +} +``` + +Alternatively, declare versions in `nextflow.config` (not currently used): + +```nextflow modules { '@nf-core/fastqc' = '1.0.0' '@nf-core/bwa-align' = '1.2.0' - '@nf-core/samtools' = '2.1.0' } ``` @@ -96,17 +118,11 @@ nextflow module list ### Update a module -Change the version in `nextflow.config`, then run your workflow. Nextflow automatically downloads the new version. - -```groovy -modules { - '@nf-core/fastqc' = '1.2.0' // Changed from 1.0.0 -} -``` +Change the version in `nextflow_spec.json` (or `nextflow.config`), then run your workflow. Nextflow automatically downloads the new version. --- -## 4. Search for Modules +## 5. Search for Modules ```bash # Search by keyword @@ -121,69 +137,19 @@ nextflow module search bwa -json --- -## 5. Configure Tool Arguments - -### Define in meta.yaml (module author) - -```yaml -# modules/@nf-core/bwa-align/meta.yaml -tools: - - bwa: - description: BWA aligner - args: - K: - flag: "-K" - type: integer - description: "Process INT input bases in each batch" - Y: - flag: "-Y" - type: boolean - description: "Use soft clipping for supplementary alignments" -``` - -### Configure in nextflow.config (user) - -```groovy -// nextflow.config -process { - withName: 'BWA_ALIGN' { - tools.bwa.args.K = 100000000 - tools.bwa.args.Y = true - } -} -``` - -### Access in script (module author) - -```groovy -// main.nf -process BWA_ALIGN { - script: - """ - bwa mem ${tools.bwa.args} -t $task.cpus $index $reads - """ -} -``` - ---- - ## 6. Work with Private Registries ### Configure authentication -```groovy +```nextflow // nextflow.config registry { // Multiple registries (tried in order) url = [ 'https://private.registry.myorg.com', - 'https://registry.nextflow.io' + 'https://registry.nextflow.io/api' ] - - auth { - 'private.registry.myorg.com' = '${MYORG_TOKEN}' - 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' - } + apiKey = 'MYORG_TOKEN' // Applied to the primary (first) registry only } ``` @@ -255,13 +221,6 @@ nextflow module remove nf-core/fastqc -keep-files ## Common Patterns -### Install all configured modules - -```bash -# Installs all modules listed in nextflow.config -nextflow module install -``` - ### Offline operation Modules are cached locally in `modules/`. Once installed, workflows run without network access. diff --git a/specs/251117-module-system/research.md b/specs/251117-module-system/research.md index 270603da82..d08005fc89 100644 --- a/specs/251117-module-system/research.md +++ b/specs/251117-module-system/research.md @@ -13,28 +13,44 @@ This document captures technical research and decisions for implementing the Nex **Research Question**: How should `nextflow module` CLI commands be implemented? -**Decision**: Follow CmdPlugin pattern with sub-command delegation +**Decision**: JCommander native subcommands — each subcommand extends `CmdBase` directly; no trait needed **Rationale**: -- CmdPlugin.groovy provides proven pattern for multi-action commands -- Uses JCommander `@Parameters` and `@Parameter` annotations -- Sub-commands (install, search, list, remove, publish, run) handled via positional args -- PluginExecAware interface allows plugin extensibility if needed later +- JCommander's subcommand support handles parameter parsing automatically per subcommand +- Each subcommand (install, run, list, remove, search, info, publish) is a separate class extending CmdBase +- `ModuleRun` extends `CmdRun` to reuse pipeline execution logic (PR #6381) +- No custom `ModuleSubCmd` trait needed; cleaner architecture +- `CmdModule` is registered in `Launcher` alongside all other top-level commands -**Reference Implementation**: -``` -Location: modules/nextflow/src/main/groovy/nextflow/cli/CmdPlugin.groovy -Pattern: - - Extends CmdBase - - @Parameters(commandNames = 'module', commandDescription = '...') - - @Parameter(names = ['-h', '--help']) - - args list for sub-command + module name - - run() method dispatches to install(), search(), etc. +**Implemented Pattern**: +```groovy +@Parameters(commandDescription = "Manage Nextflow modules") +class CmdModule extends CmdBase implements UsageAware { + static final List commands = [] + + static { + commands << new ModuleInstall() // extends CmdBase + commands << new ModuleRun() // extends CmdRun + commands << new ModuleList() // extends CmdBase + commands << new ModuleRemove() // extends CmdBase + commands << new ModuleSearch() // extends CmdBase + commands << new ModuleInfo() // extends CmdBase + commands << new ModulePublish() // extends CmdBase + } + + void run() { + final jc = commander() // JCommander with all subcommands registered + jc.parse(args as String[]) + final subcommand = jc.getCommands().get(jc.getParsedCommand()).getObjects()[0] + subcommand.run() + } +} ``` **Alternatives Considered**: -- Separate CmdModuleInstall, CmdModuleSearch classes: Rejected - too many entry points, doesn't match existing patterns -- Plugin-based CLI extension: Rejected - module system is core functionality, not optional +- CmdFs trait pattern: Considered initially; replaced by JCommander native subcommands — simpler and avoids custom parsing +- Separate top-level Cmd classes (CmdModuleInstall, etc.): Rejected — too many entry points +- Plugin-based CLI extension: Rejected — module system is core functionality, not optional --- @@ -42,33 +58,40 @@ Pattern: **Research Question**: How to extend `include` statement parsing for registry modules? -**Decision**: Extend ResolveIncludeVisitor to detect `@` prefix and delegate to ModuleResolver +**Decision**: Extend `ResolveIncludeVisitor` to detect `@` prefix and delegate to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader` **Rationale**: -- IncludeNode already captures source path as string -- Detection: `source.startsWith('@')` distinguishes registry vs local paths -- Resolution happens at parse time (after plugin resolution) per ADR -- Preserves existing local file include behavior +- Keeps `nf-lang` decoupled from runtime module resolution (`nf-lang` has no dependency on `nextflow` module) +- SPI pattern allows plugins or custom implementations to override the default resolver +- Detection: `source.startsWith('@')` distinguishes registry vs local paths — preserves existing include behavior +- Resolution at parse time (after plugin resolution) per ADR -**Reference Implementation**: +**Implemented Architecture**: ``` -Location: modules/nf-lang/src/main/java/nextflow/script/ResolveIncludeVisitor.java -Extension Point: visitInclude() method -Pattern: - 1. Check if source starts with '@' - 2. If yes: call ModuleResolver.resolve(source, configuredVersion) - 3. ModuleResolver returns absolute path to modules/@scope/name/main.nf - 4. Continue with standard include processing +include { X } from '@scope/name' + ↓ +ResolveIncludeVisitor.visitInclude() [nf-lang] + source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(source, baseDir) + ↓ +RemoteModuleResolverProvider [nf-lang] + Java ServiceLoader discovers implementations; picks highest priority + ↓ +DefaultRemoteModuleResolver [nextflow module] + Calls ModuleResolver.installModule(reference, version, autoInstall=true) + Returns Path to modules/@scope/name/main.nf ``` **Key Files**: -- `IncludeNode.java` - AST representation -- `IncludeEntryNode.java` - Individual entries -- `ResolveIncludeVisitor.java` - Visitor for resolution +- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java` — SPI interface +- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java` — ServiceLoader singleton +- `modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java` — error fallback +- `modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java` — MODIFIED +- `modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy` — default impl **Alternatives Considered**: -- New ANTLR grammar token for `@`: Rejected - unnecessary parser complexity -- Dot file marker for local modules: Deferred to Open Questions in ADR +- New ANTLR grammar token for `@`: Rejected — unnecessary parser complexity +- Direct dependency from nf-lang to nextflow module: Rejected — circular dependency risk; SPI decouples cleanly +- Dot file marker for local modules: Deferred in ADR; current impl uses `@` for registry, `.`/`/` for local --- @@ -76,46 +99,75 @@ Pattern: **Research Question**: How to add new config DSL blocks? -**Decision**: Create ModulesDsl and RegistryDsl classes following PluginsDsl pattern +**Decision**: Create ModulesConfig and RegistryConfig classes implementing ConfigScope interface **Rationale**: -- PluginsDsl.groovy provides exact template for DSL block handling -- ConfigBuilder already supports dynamic DSL registration -- Groovy's methodMissing enables clean config syntax +- ConfigScope is an ExtensionPoint (pf4j) that ConfigBuilder automatically discovers +- Classes implementing ConfigScope and annotated with @ScopeName are automatically parsed +- No need to modify ConfigBuilder or create custom DSL parsers +- Pattern used throughout Nextflow: FusionConfig, CondaConfig, DockerConfig, etc. +- Provides type safety via @CompileStatic and validation via @ConfigOption **Reference Implementation**: ``` -Location: modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy +Location: modules/nextflow/src/main/groovy/nextflow/fusion/FusionConfig.groovy Pattern: + @ScopeName("modules") + @Description("Module version declarations") @CompileStatic - class ModulesDsl { - private Map modules = [:] + class ModulesConfig implements ConfigScope { + @ConfigOption + @Description("Module version mappings") + final Map modules = [:] - def methodMissing(String name, args) { - // modules { '@nf-core/fastqc' = '1.0.0' } - modules[name] = args[0].toString() - } + ModulesConfig() {} - Map getModules() { modules } + ModulesConfig(Map opts) { + // Parse from config map + } } ``` -**RegistryDsl Pattern**: +**ConfigScope Interface**: +``` +Location: modules/nf-lang/src/main/java/nextflow/config/spec/ConfigScope.java +public interface ConfigScope extends ExtensionPoint {} +``` + +**RegistryConfig Pattern**: ```groovy -class RegistryDsl { - String url = 'https://registry.nextflow.io' - List urls = [] // For multiple registries - Map auth = [:] - - void url(String value) { this.url = value } - void url(List values) { this.urls = values } - void auth(Closure config) { /* parse auth block */ } +@ScopeName("registry") +@Description("Module registry configuration") +@CompileStatic +class RegistryConfig implements ConfigScope { + static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api' + + @ConfigOption + final Collection url // One or more URLs in priority order + + @ConfigOption + final String apiKey // API key; falls back to NXF_REGISTRY_TOKEN env var + + RegistryConfig() { + url = [DEFAULT_REGISTRY_URL] + apiKey = null + } + + RegistryConfig(Map opts) { + url = opts.url ?: [DEFAULT_REGISTRY_URL] + apiKey = opts.apiKey as String + } + + String getUrl() { url ? url[0] : DEFAULT_REGISTRY_URL } + Collection getAllUrls() { url ?: [DEFAULT_REGISTRY_URL] } + String getApiKey() { apiKey ?: SysEnv.get('NXF_REGISTRY_TOKEN') } } ``` -**Integration Point**: ConfigBuilder.build() instantiates DSL objects +**Integration Point**: ConfigBuilder automatically discovers and parses ConfigScope implementations via ExtensionPoint mechanism **Alternatives Considered**: +- Custom DSL parsers (ModulesDsl/RegistryDsl): Rejected - unnecessary complexity, ConfigScope pattern handles this automatically - JSON/YAML config file: Rejected - inconsistent with Nextflow config style - Dedicated pipeline.yaml: Deferred per ADR Open Questions @@ -169,36 +221,33 @@ POST /api/modules/{name} # Publish (authenticated) **Research Question**: How to handle registry authentication? -**Decision**: Support NXF_REGISTRY_TOKEN env var + registry.auth config block +**Decision**: Support `NXF_REGISTRY_TOKEN` env var + `registry.apiKey` config field **Rationale**: - Environment variable provides CI/CD compatibility -- Config block allows per-registry tokens for private registries -- Follows existing plugin auth patterns +- `apiKey` config field allows explicit token configuration +- Authentication is only applied to the primary (first) registry URL - Bearer token in Authorization header (standard HTTP auth) -**Reference Implementation**: +**Implementation**: ``` -Location: modules/nextflow/src/main/groovy/nextflow/cli/CmdAuth.groovy -Pattern: - 1. Check NXF_REGISTRY_TOKEN environment variable - 2. Fall back to registry.auth.'registry.nextflow.io' in config - 3. Add header: Authorization: Bearer +RegistryConfig.getApiKey() returns: + 1. registry.apiKey config value if set + 2. NXF_REGISTRY_TOKEN environment variable as fallback + 3. null if neither is set (unauthenticated requests) ``` **Config Syntax**: -```groovy +```nextflow registry { - auth { - 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' - 'private.registry.com' = '${PRIVATE_TOKEN}' - } + apiKey = '${NXF_REGISTRY_TOKEN}' } ``` **Alternatives Considered**: +- Per-registry token map (`auth {}` block): Was in initial design; simplified to single `apiKey` since only the primary registry uses authentication - Secrets file (~/.nextflow/secrets.json): Possible future enhancement -- OAuth flow: Rejected for CLI - token-based simpler +- OAuth flow: Rejected for CLI — token-based simpler --- @@ -275,59 +324,7 @@ class ModuleChecksum { ## 8. Tool Arguments Implementation -**Research Question**: How to implement structured tool arguments (`tools..args`)? - -**Decision**: Implement as implicit variable in process scope, validated at parse time - -**Rationale**: -- `tools` variable accessible in script block like `task`, `params` -- Validation at parse time catches errors early (per clarification) -- Schema defined in meta.yaml, parsed by ModuleManifest -- Concatenation logic handles flag formatting - -**Implementation Pattern**: -```groovy -class ToolArgs { - private Map schema // From meta.yaml - private Map values // From config - - String getAt(String argName) { - def def = schema[argName] - def value = values[argName] - if (def.type == 'boolean' && value) { - return def.flag // e.g., "-Y" - } - return "${def.flag} ${value}" // e.g., "-K 100000" - } - - String toString() { - // Concatenate all configured args - values.collect { name, value -> - this[name] - }.join(' ') - } -} -``` - -**Config Access**: -```groovy -withName: 'BWA_MEM' { - tools.bwa.args.K = 100000 - tools.bwa.args.Y = true -} -``` - -**Script Access**: -```groovy -script: -""" -bwa mem ${tools.bwa.args} -t $task.cpus $index $reads -""" -``` - -**Alternatives Considered**: -- Runtime validation only: Rejected - late errors waste compute -- String-only values: Rejected - loses type safety benefits +> **⚠️ REMOVED FROM ADR** — The tool arguments feature (`tools..args` in meta.yaml and process config) was removed from the module system ADR. It is not implemented and not planned in the current scope. The `meta.yaml` format used in the actual implementation (`ModuleSpec`) does not include tool/argument definitions. --- @@ -335,26 +332,21 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads | Area | Decision | Key Reference | |------|----------|---------------| -| CLI | CmdModule extends CmdBase | CmdPlugin.groovy | -| DSL Parser | Extend ResolveIncludeVisitor | ResolveIncludeVisitor.java | -| Config | ModulesDsl + RegistryDsl | PluginsDsl.groovy | -| Registry HTTP | HttpModuleRepository | HttpPluginRepository.groovy | -| Authentication | NXF_REGISTRY_TOKEN + config | CmdAuth.groovy | -| Checksums | SHA-256, .checksum file | Standard Java security | +| CLI | JCommander subcommands; each extends CmdBase (ModuleRun extends CmdRun) | CmdModule.groovy | +| DSL Parser | SPI pattern — ResolveIncludeVisitor delegates to RemoteModuleResolver; DefaultRemoteModuleResolver bridges to ModuleResolver | ResolveIncludeVisitor.java, RemoteModuleResolver.java | +| Config | ModulesConfig + RegistryConfig (ConfigScope) | FusionConfig.groovy, ConfigScope.java | +| Registry HTTP | ModuleRegistryClient using HxClient + npr-api models | HttpPluginRepository.groovy | +| Authentication | `NXF_REGISTRY_TOKEN` env var or `registry.apiKey` config field (primary registry only) | RegistryConfig.groovy | +| Checksums | SHA-256/SHA-512, `.checksum` file, download integrity via X-Checksum header | ModuleChecksum.groovy | +| Version Storage | `nextflow_spec.json` (auto-managed); `modules {}` in nextflow.config (manual alternative) | PipelineSpec.groovy | | Version Syntax | Plugin-compatible constraints | VersionNumber class | -| Tool Args | Implicit variable, parse-time validation | New implementation | +| Tool Args | ~~Implicit variable, parse-time validation~~ — **Removed from ADR** | N/A | --- ## Open Items (Deferred) -These items are noted in the ADR as open questions and do not block implementation: - -1. **Local vs managed module distinction**: Whether local modules use `@` prefix or dot file marker -2. **Tool arguments CLI syntax**: Colon vs dot separator (`--tools:bwa:K` vs `--tools.bwa.K`) -3. **Module version location**: nextflow.config vs dedicated pipeline.yaml - -Current implementation uses: -- `@` prefix for registry modules only (local paths start with `.` or `/`) -- Colon-separated CLI syntax per ADR assumption -- Versions in nextflow.config per ADR decision \ No newline at end of file +1. **Local vs managed module distinction**: Resolved — `@` prefix for registry modules only; local paths start with `.` or `/` +2. **Tool arguments**: Removed from ADR — not in scope +3. **Module version location**: Resolved — `nextflow_spec.json` (auto-managed by `module install`); `modules {}` block in `nextflow.config` supported as alternative +4. **DSL parser `@scope/name` include**: ✅ Resolved — SPI pattern implemented (T017a-d) \ No newline at end of file diff --git a/specs/251117-module-system/spec.md b/specs/251117-module-system/spec.md index 652397bb80..fba59f7923 100644 --- a/specs/251117-module-system/spec.md +++ b/specs/251117-module-system/spec.md @@ -27,7 +27,7 @@ A pipeline developer wants to use a pre-built module from the Nextflow registry **Acceptance Scenarios**: -1. **Given** a new Nextflow project with no modules installed, **When** user runs `nextflow module install nf-core/fastqc`, **Then** the module is downloaded to `modules/@nf-core/fastqc/`, a `.checksum` file is created, and `nextflow.config` is updated with the version +1. **Given** a new Nextflow project with no modules installed, **When** user runs `nextflow module install nf-core/fastqc`, **Then** the module is downloaded to `modules/@nf-core/fastqc/`, a `.checksum` file is created, and `nextflow_spec.json` is updated with the version 2. **Given** a workflow file with `include { FASTQC } from '@nf-core/fastqc'`, **When** user runs `nextflow run main.nf`, **Then** Nextflow resolves the module from local storage and executes the process 3. **Given** a module version declared in `nextflow.config`, **When** user includes the module, **Then** the declared version is used (not latest) @@ -75,7 +75,7 @@ A pipeline developer wants to pin and manage module versions to ensure reproduci **Acceptance Scenarios**: -1. **Given** a module is installed at version 1.0.0, **When** user changes `nextflow.config` to specify version 1.1.0 and runs the workflow, **Then** version 1.1.0 is automatically downloaded and replaces the local copy +1. **Given** a module is installed at version 1.0.0, **When** user changes `nextflow_spec.json` to specify version 1.1.0 and runs the workflow, **Then** version 1.1.0 is automatically downloaded and replaces the local copy 2. **Given** modules installed locally, **When** user runs `nextflow module list`, **Then** configured version, installed version, latest available version, and status are displayed for each module --- @@ -106,7 +106,7 @@ A pipeline developer wants to remove a module they no longer need. **Acceptance Scenarios**: -1. **Given** a module is installed, **When** user runs `nextflow module remove nf-core/fastqc`, **Then** the module directory is deleted and the entry is removed from `nextflow.config` +1. **Given** a module is installed, **When** user runs `nextflow module remove nf-core/fastqc`, **Then** the module directory is deleted and the entry is removed from `nextflow_spec.json` 2. **Given** a module is referenced in workflow files, **When** user runs `nextflow module remove`, **Then** a warning is displayed about the reference but removal proceeds --- @@ -166,7 +166,7 @@ A module author wants to publish their module to the Nextflow registry for other - **FR-001**: System MUST recognize `@scope/name` syntax in `include` statements as registry module references - **FR-002**: System MUST distinguish between local file paths (starting with `.` or `/`) and registry modules (starting with `@`) -- **FR-003**: System MUST resolve module versions from `nextflow.config` `modules {}` block before downloading +- **FR-003**: System MUST resolve module versions from `nextflow_spec.json` before downloading - **FR-004**: System MUST parse and validate `meta.yaml` files for module metadata and dependencies #### Module Resolution @@ -192,12 +192,13 @@ A module author wants to publish their module to the Nextflow registry for other - **FR-017**: System MUST provide `nextflow module remove scope/name` command to delete modules - **FR-018**: System MUST provide `nextflow module publish scope/name` command to upload modules to registry - **FR-019**: System MUST provide `nextflow module run scope/name` command to execute modules directly +- **FR-019b**: System MUST provide `nextflow module info scope/name` command to display module metadata and a usage template #### Configuration -- **FR-020**: System MUST read module versions from `modules {}` block in `nextflow.config` -- **FR-021**: System MUST support `registry {}` block for configuring registry URL and authentication -- **FR-022**: System MUST support `NXF_REGISTRY_TOKEN` environment variable for authentication +- **FR-020**: System MUST persist module versions in `nextflow_spec.json`; MUST also read versions from `modules {}` block in `nextflow.config` as an alternative +- **FR-021**: System MUST support `registry {}` block with `url` and `apiKey` fields for configuring registry URL and authentication +- **FR-022**: System MUST support `NXF_REGISTRY_TOKEN` environment variable as fallback for `registry.apiKey` - **FR-023**: System MUST support multiple registry URLs with fallback ordering #### Module Parameters From b2fa0f833cebab28552df70898a4ab1a93c7c1d5 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 13 Mar 2026 17:09:14 -0500 Subject: [PATCH 68/75] minor edits Signed-off-by: Ben Sherman --- docs/module.md | 6 +++--- .../nextflow/module/DefaultRemoteModuleResolver.groovy | 5 +---- .../src/main/groovy/nextflow/module/ModuleReference.groovy | 4 ++-- .../nextflow/src/main/resources/META-INF/extensions.idx | 2 +- .../src/testFixtures/groovy/test/TestHelper.groovy | 7 +++---- .../src/main/nextflow/config/RegistryConfig.groovy | 6 +++--- settings.gradle | 2 +- 7 files changed, 14 insertions(+), 18 deletions(-) diff --git a/docs/module.md b/docs/module.md index 747ae76395..19346a15cc 100644 --- a/docs/module.md +++ b/docs/module.md @@ -362,8 +362,8 @@ $ nextflow module remove nf-core/fastqc By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour: -- `-keep-files` — Remove the `.module-info` file created at install but keep the rest of files -- `-force` — Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications +- `-keep-files`: Remove the `.module-info` file created at install but keep the rest of files +- `-force`: Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications ### Viewing module information @@ -430,7 +430,7 @@ Registry modules follow a standard directory structure: modules/ └── scope/ └── module-name/ - ├── .checksum # Integrity checksum (generated automatically) + ├── .module-info # Integrity checksum (generated automatically) ├── README.md # Documentation (required for publishing) ├── main.nf # Module entry point (required) ├── meta.yaml # Module metadata (required for publishing) diff --git a/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy b/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy index ef3d0efb45..82e17d735e 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy @@ -65,10 +65,7 @@ class DefaultRemoteModuleResolver implements RemoteModuleResolver { log.debug "Module ${reference} resolved to ${mainFile}" return mainFile } catch (Exception e) { - throw new IllegalModulePath( - "Failed to resolve remote module ${moduleName}: ${e.message}", - e - ) + throw new IllegalModulePath("Failed to resolve remote module ${moduleName}: ${e.message}", e) } } diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy index 23cbf7ba3d..9cb0225e72 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy @@ -31,7 +31,7 @@ import java.util.regex.Pattern @EqualsAndHashCode class ModuleReference { - // Pattern allows: optional @, scope with letters/digits/hyphens/dots/underscores, name segments separated by slashes (no trailing slash) + // Pattern allows: scope with letters/digits/hyphens/dots/underscores, name segments separated by slashes (no trailing slash) // Scope: starts with letter/digit, followed by letters/digits/dots/underscores/hyphens // Name: one or more segments (each starting with letter, followed by letters/digits/underscores/hyphens), separated by slashes private static final Pattern MODULE_NAME_PATTERN = ~/^([a-z0-9][a-z0-9._\-]*)\/([a-z][a-z0-9._\-]*(?:\/[a-z][a-z0-9._\-]*)*)$/ @@ -47,7 +47,7 @@ class ModuleReference { } /** - * Parse a module reference from a string in "@scope/name" or "scope/name" format + * Parse a module reference from a string as "scope/name" * * @param source The module reference string * @return A ModuleReference object diff --git a/modules/nextflow/src/main/resources/META-INF/extensions.idx b/modules/nextflow/src/main/resources/META-INF/extensions.idx index f9075fb3d9..7a5a21cc99 100644 --- a/modules/nextflow/src/main/resources/META-INF/extensions.idx +++ b/modules/nextflow/src/main/resources/META-INF/extensions.idx @@ -18,8 +18,8 @@ nextflow.cache.DefaultCacheFactory nextflow.conda.CondaConfig nextflow.config.ConfigMap nextflow.config.Manifest -nextflow.config.WorkflowConfig nextflow.config.RegistryConfig +nextflow.config.WorkflowConfig nextflow.container.ApptainerConfig nextflow.container.CharliecloudConfig nextflow.container.DockerConfig diff --git a/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy b/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy index 6739f74306..c79ed822d1 100644 --- a/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy +++ b/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy @@ -16,17 +16,16 @@ package test -import com.google.common.jimfs.JimfsPath -import nextflow.util.KryoHelper -import nextflow.util.PathSerializer - import java.nio.file.Files import java.nio.file.Path import java.util.zip.GZIPInputStream import com.google.common.jimfs.Configuration import com.google.common.jimfs.Jimfs +import com.google.common.jimfs.JimfsPath import groovy.transform.Memoized +import nextflow.util.KryoHelper +import nextflow.util.PathSerializer /** * * @author Paolo Di Tommaso diff --git a/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy b/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy index cb232af33b..f43c0f131d 100644 --- a/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy +++ b/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy @@ -36,15 +36,15 @@ import nextflow.script.dsl.Description @CompileStatic class RegistryConfig implements ConfigScope { - final static public String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api' + public static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api' @ConfigOption @Description("Registry URL or list of registry URLs in priority order (primary URL first)") - final private Collection url + private final Collection url @ConfigOption @Description("API key for authenticating with the primary registry") - final private String apiKey + private final String apiKey /* required by extension point -- do not remove */ RegistryConfig() { diff --git a/settings.gradle b/settings.gradle index 22284225f8..bb90d2b520 100644 --- a/settings.gradle +++ b/settings.gradle @@ -49,5 +49,5 @@ include 'plugins:nf-cloudcache' include 'plugins:nf-k8s' include 'plugins:nf-seqera' -//includeBuild '../sched' //includeBuild('../plugin-registry') +//includeBuild '../sched' From 9b647ef756d6e5be0692feeaba5de019365e1104 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 13 Mar 2026 18:08:23 -0500 Subject: [PATCH 69/75] Cleanup module run output Signed-off-by: Ben Sherman --- .../nextflow/cli/module/CmdModuleRun.groovy | 1 - .../nextflow/module/ModuleResolver.groovy | 2 +- .../script/ProcessEntryHandler.groovy | 100 ++++++------------ 3 files changed, 36 insertions(+), 67 deletions(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy index c0f8fe8421..d09bbd0ee4 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy @@ -81,7 +81,6 @@ class CmdModuleRun extends CmdRun { def resolver = new ModuleResolver(baseDir, client ?: new ModuleRegistryClient(registryConfig)) Path moduleFile = resolver.installModule(reference, version) if( moduleFile ) { - println "Executing module..." args[0] = moduleFile.toAbsolutePath().toString() super.run() } diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy index 8e8626a0b9..2c96a89f85 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy @@ -128,7 +128,7 @@ class ModuleResolver { if( storage.isInstalled(reference) ) { def installed = storage.getInstalledModule(reference) if( installed.installedVersion == version ) { - log.info "Module ${reference}@${installed.installedVersion} is already installed (version $version)" + log.debug "Module ${reference}@${installed.installedVersion} is already installed (version $version)" return installed.mainFile } diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy index c538b418c7..e124e27e8a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy @@ -16,18 +16,13 @@ package nextflow.script -import groovyx.gpars.dataflow.DataflowReadChannel -import groovyx.gpars.dataflow.DataflowWriteChannel -import nextflow.exception.AbortOperationException -import nextflow.extension.CH -import nextflow.extension.DataflowHelper -import nextflow.extension.DumpHelper - import java.nio.file.Path import groovy.transform.CompileStatic import groovy.util.logging.Slf4j +import groovyx.gpars.dataflow.DataflowVariable import nextflow.Session import nextflow.Nextflow +import nextflow.extension.DumpHelper import nextflow.script.params.EnvInParam import nextflow.script.params.FileInParam import nextflow.script.params.InParam @@ -36,8 +31,6 @@ import nextflow.script.params.TupleInParam import nextflow.script.params.v2.ProcessInput import nextflow.script.params.v2.ProcessTupleInput -import java.util.concurrent.atomic.AtomicInteger - /** * Helper class for process entry execution feature. * @@ -56,8 +49,6 @@ class ProcessEntryHandler { private final BaseScript script private final Session session private final ScriptMeta meta - // Map to store process outputs - private Map processOutputs ProcessEntryHandler(BaseScript script, Session session, ScriptMeta meta) { this.script = script @@ -96,9 +87,10 @@ class ProcessEntryHandler { final workflowExecutionClosure = { -> // Get input parameter values and execute the process final inputArgs = getProcessArguments(processDef) - final processResult = script.invokeMethod(processName, inputArgs as Object[]) - printOutput(processName, processResult) - return processResult + final output = meta.getProcess(processName).run(inputArgs as Object[]) as ChannelOut + session.addIgniter { + printOutput(processName, output) + } } // Create the body definition with execution logic @@ -108,69 +100,47 @@ class ProcessEntryHandler { return new WorkflowDef(script, workflowBody) } + /** * Prints the process outputs. - * ChannelOut has two structures containing the outputs and anonymous channels. + * * @param processName - * @param processResult ChannelOut containing the process outputs + * @param output */ - private printOutput(String processName, def processResult){ - if( ! processResult instanceof ChannelOut ) { - throw new AbortOperationException("Not a valid process output ($processResult.class)") - } - final results = processResult as ChannelOut - if (results.isEmpty()){ - log.debug("No outputs found for $processName") + private void printOutput(String processName, ChannelOut output) { + if( output.isEmpty() ) { + log.debug("Process ${processName} does not declare any outputs") return } - final named = new HashMap(results.size()) + Object result = null - //Compute reverse index for named outputs - for (String name : results.getNames()){ - named.put(results.getProperty(name) as DataflowWriteChannel, name) - } - // Create the processOutputs map to keep the outputs order, and create the subcriber per output channel to collect the output values - int unnamedIndex = 1 - processOutputs = new LinkedHashMap<>(results.size()) - results.each { - String name = named.get(it) - if (!name) { - name = "anonymous-$unnamedIndex".toString() - unnamedIndex++ - } - processOutputs.put(name, []) - createProcessOutputSubscriber(name, CH.getReadChannel(it)) - } - //Add workflow - session.workflowMetadata.onComplete { - if( processOutputs) { - println "" - println "Process $processName Outputs:" - println DumpHelper.prettyPrintJson(processOutputs) - println "" - } + // print process output directly if it is a single expression + if( output.size() == 1 && (output.getNames().isEmpty() || output.getNames().first() == '$out') ) { + result = (output[0] as DataflowVariable).get() } - } - /** - * Create a subscriber operator to inspect the output channels and build a map. - * At `onNext` event, add the output value in the `processOutputs` map with the output name as key. - * At `onComplete` event, convert single element list to single value. - * - * @param name Process output name - * @param channel Process output channel - */ - private void createProcessOutputSubscriber(String name, DataflowReadChannel channel ){ - def onNextClosure = { it -> - (processOutputs[name] as List).add(it) } - def onCompleteClosure = { - final list = processOutputs[name] as List - if( list.size() == 1) { - processOutputs[name] = list[0] + // otherwise, construct map of process emits + else { + if( output.size() != output.getNames().size() ) + log.warn("Process ${processName} is missing emit names for one or more outputs -- unnamed outputs will be omitted") + + // compute reverse lookup of emit names + final reverseLookup = new HashMap(output.size()) + for( final name : output.getNames() ) + reverseLookup.put(output.getProperty(name), name) + + // combine process emits into map + final combinedOutputs = new LinkedHashMap(output.size()) + for( final ch : output ) { + final name = reverseLookup.get(ch) + combinedOutputs.put(name, (ch as DataflowVariable).get()) } + + result = combinedOutputs } - DataflowHelper.subscribeImpl(channel, [onNext: onNextClosure, onComplete: onCompleteClosure]) + + println DumpHelper.prettyPrintJson(result) } /** From 00824f3a22e6c4530051cd8c886e92a31df5bea5 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 13 Mar 2026 18:35:36 -0500 Subject: [PATCH 70/75] Fix meta.yaml -> meta.yml, module manifest -> module spec Signed-off-by: Ben Sherman --- adr/20251114-module-system.md | 48 +++++++++---------- adr/module-spec-schema.json | 4 +- docs/cli.md | 2 +- docs/module.md | 6 +-- docs/reference/cli.md | 2 +- .../cli/module/CmdModulePublish.groovy | 48 +++++++++---------- .../groovy/nextflow/module/ModuleSpec.groovy | 32 ++++++------- .../cli/module/CmdModulePublishTest.groovy | 2 +- .../nextflow/module/ModuleSpecTest.groovy | 46 +++++++++--------- 9 files changed, 95 insertions(+), 95 deletions(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index e49add710a..123463c2e6 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -11,13 +11,13 @@ ### Version 2.7 (2026-03-09) - **Renamed `.checksum` to `.module-info`**: Leaves room for additional properties in the future - **Removed `@` prefix from module scopes**: Local modules are distinguished from remote modules by presence/absence of `./` prefix -- **Removed version pinning from config**: Installed module versions are now inferred from the `meta.yaml` of each module in the `modules/` directory instead of being declared in `nextflow.config` +- **Removed version pinning from config**: Installed module versions are now inferred from the `meta.yml` of each module in the `modules/` directory instead of being declared in `nextflow.config` ### Version 2.6 (2026-01-28) - **Removed module parameters**: Module parameters specification moved to separate spec document. ### Version 2.5 (2026-01-23) -- **Module parameters**: Replaced structured tool arguments with general module parameters defined in `meta.yaml` +- **Module parameters**: Replaced structured tool arguments with general module parameters defined in `meta.yml` - **Simplified tools section**: Removed `args` property from tools; tool arguments now configured via module parameters - **Simplified `requires` block**: Removed `plugins`, `modules`, and `subworkflows` sub-properties; `requires` now only contains `nextflow` version constraint - **Process modules focus**: Removed sub-workflow references; spec is now focused on process modules only @@ -74,7 +74,7 @@ include { MY_PROCESS } from './modules/my-process.nf' **Module Naming**: Scoped modules `scope/name` (e.g., `nf-core/salmon`, `myorg/custom`). Local paths supported for backwards compatibility. No nested paths with the module are allowed - each module must have a `main.nf` as the entry point. -**Version Resolution**: Installed module versions are inferred from the `meta.yaml` of each module in the `modules/` directory. If a module is not present locally, the latest available version is downloaded from the registry. +**Version Resolution**: Installed module versions are inferred from the `meta.yml` of each module in the `modules/` directory. If a module is not present locally, the latest available version is downloaded from the registry. **Resolution Order**: 1. Check local `modules/scope/name/` exists @@ -86,7 +86,7 @@ include { MY_PROCESS } from './modules/my-process.nf' | Local State | Action | |-------------|--------| | Missing | Download latest from registry | -| Exists, checksum valid | Use local module (version from `meta.yaml`) | +| Exists, checksum valid | Use local module (version from `meta.yml`) | | Exists, checksum mismatch | **Warn**: locally modified, will NOT replace unless `-force` is used | **Key Behaviors**: @@ -120,7 +120,7 @@ registry { } ``` -**Module Spec** (`meta.yaml`): +**Module Spec** (`meta.yml`): ```yaml name: nf-core/bwa-align version: 1.2.4 # This module's version @@ -158,7 +158,7 @@ This avoids introducing new notation that would require additional parser suppor **Module Resolution**: -Installed module versions are inferred from the `meta.yaml` file for each module in the `modules/` directory. +Installed module versions are inferred from the `meta.yml` file for each module in the `modules/` directory. ### 3. Unified Nextflow Registry @@ -190,7 +190,7 @@ Note: The `{name}` parameter includes the namespace prefix (e.g., "nf-core/fastq **Artifact Types**: - **Plugins**: JAR files with JSON metadata, resolved at startup -- **Modules**: Source archives (.nf + meta.yaml), resolved at parse time +- **Modules**: Source archives (.nf + meta.yml), resolved at parse time **Benefits**: - Reuses existing infrastructure (HTTP service, S3 storage, authentication) @@ -318,7 +318,7 @@ Display the status of all modules, comparing what is configured in `nextflow.con **Output columns**: - Module name (`scope/name`) -- Installed version (from `modules/scope/name/meta.yaml`) +- Installed version (from `modules/scope/name/meta.yml`) - Latest available version (from registry) - Status indicator (up-to-date, outdated, missing) @@ -364,7 +364,7 @@ Publish a module to the Nextflow registry, making it available for others to ins - `-dry-run`: Validate without publishing **Behavior**: -1. Validates `meta.yaml` schema and required fields (name, version, description) +1. Validates `meta.yml` schema and required fields (name, version, description) 2. Verifies that `main.nf` exists and is valid Nextflow syntax 3. Verifies that `README.md` documentation is present 4. Authenticates with registry using configured credentials @@ -372,7 +372,7 @@ Publish a module to the Nextflow registry, making it available for others to ins 6. Publishes the release, making it available for installation **Requirements**: -- Valid `meta.yaml` with name, version, and description +- Valid `meta.yml` with name, version, and description - `main.nf` entry point file - `README.md` documentation - Authentication token configured in `registry.auth` or `NXF_REGISTRY_TOKEN` @@ -391,12 +391,12 @@ Everything within the module directory should be uploaded. Module bundle should ``` my-module/ ├── main.nf # Required: entry point for module -├── meta.yaml # Required: Module spec (version, metadata, I/O specs) +├── meta.yml # Required: Module spec (version, metadata, I/O specs) ├── README.md # Required: Module description └── tests/ # Optional tests ``` -**Module Spec extension** (`meta.yaml`): +**Module Spec extension** (`meta.yml`): ```yaml name: nf-core/bwa-align version: 1.2.4 # This module's version @@ -418,16 +418,16 @@ project-root/ ├── nf-core/ │ ├── bwa-align/ │ │ ├── .module-info # Cached registry checksum - │ │ ├── meta.yaml + │ │ ├── meta.yml │ │ └── main.nf # Required entry point │ └── samtools/view/ │ ├── .module-info - │ ├── meta.yaml + │ ├── meta.yml │ └── main.nf # Required entry point └── myorg/ └── custom-process/ ├── .module-info - ├── meta.yaml + ├── meta.yml └── main.nf # Required entry point ``` @@ -455,12 +455,12 @@ project-root/ 1. Parse `include` statements → extract module names (e.g., `nf-core/bwa-align`) 2. For each module: a. Check local `modules/scope/name/` exists - - If exists → read installed version from `modules/scope/name/meta.yaml` + - If exists → read installed version from `modules/scope/name/meta.yml` - If missing → download latest version from registry b. Verify local module integrity against `.module-info` file - Checksum mismatch → warn and do NOT override (local changes detected) 3. On download: store module to `modules/scope/name/` with `.module-info` file -4. Read `meta.yaml` file: Validates Nextflow requirement → Fail if not fulfilled +4. Read `meta.yml` file: Validates Nextflow requirement → Fail if not fulfilled 5. Parse module's `main.nf` file → make processes available **Security**: @@ -484,7 +484,7 @@ project-root/ | Metadata | JSON spec | YAML spec | | Naming | `nf-amazon` | `nf-core/salmon` | | Cache Location | `$NXF_HOME/plugins/` | `modules/scope/name/` | -| Version Config | `plugins {}` in config | `meta.yaml` in `modules/` directory | +| Version Config | `plugins {}` in config | `meta.yml` in `modules/` directory | | Registry Path | `/api/v1/plugins/` | `/api/modules/{name}` | ## Rationale @@ -495,9 +495,9 @@ project-root/ - Lower operational overhead - Type-specific handling maintains separation of concerns -**Why infer versions from `meta.yaml` instead of pinning in a separate file?** +**Why infer versions from `meta.yml` instead of pinning in a separate file?** - Simple: install a version once and it is captured in the module files -- Reproducibility via committing the `modules/` directory (including `meta.yaml`) to the project git repository +- Reproducibility via committing the `modules/` directory (including `meta.yml`) to the project git repository - Reduces configuration burden: no need to keep config in sync with installed state **Why parse-time resolution?** @@ -520,12 +520,12 @@ project-root/ **Positive**: - Enables ecosystem-wide code reuse -- Reproducible workflows via committing the `modules/` directory (including `meta.yaml`) to the project git repository +- Reproducible workflows via committing the `modules/` directory (including `meta.yml`) to the project git repository - Centralized discovery and distribution via unified registry - Minimal operational overhead (single registry for both plugins and modules) - Module scoping enables organization namespaces and private registries - Local `modules/` directory provides project isolation -- No version duplication: installed `meta.yaml` is the single source of truth +- No version duplication: installed `meta.yml` is the single source of truth - Simple module structure: each module has single `main.nf` entry point **Negative**: @@ -548,7 +548,7 @@ project-root/ ## Appendix A: Module Schema Specification -This appendix defines the JSON schema for module `meta.yaml` files. The schema maintains backward compatibility with existing nf-core module metadata patterns while supporting the new Nextflow module system features. +This appendix defines the JSON schema for module `meta.yml` files. The schema maintains backward compatibility with existing nf-core module metadata patterns while supporting the new Nextflow module system features. **Schema File:** [module-spec-schema.json](module-spec-schema.json) **Published URL:** `https://registry.nextflow.io/schemas/module-spec/v1.0.0` @@ -761,7 +761,7 @@ output: #### Schema Validation -Use the schema reference in your `meta.yaml`: +Use the schema reference in your `meta.yml`: ```yaml # yaml-language-server: $schema=https://registry.nextflow.io/schemas/module-spec/v1.0.0 diff --git a/adr/module-spec-schema.json b/adr/module-spec-schema.json index ad972df973..25f77a0bfc 100644 --- a/adr/module-spec-schema.json +++ b/adr/module-spec-schema.json @@ -2,12 +2,12 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://raw.githubusercontent.com/nextflow-io/schemas/main/module/v1/schema.json", "title": "Nextflow Module Schema", - "description": "Schema for Nextflow module meta.yaml files, supporting both nf-core community patterns and the Nextflow module system", + "description": "Schema for Nextflow module meta.yml files, supporting both nf-core community patterns and the Nextflow module system", "type": "object", "properties": { "name": { "type": "string", - "description": "Module name. Can be a simple identifier (e.g., 'fastqc', 'bwa_mem') for local/nf-core modules, or a fully qualified scoped name (e.g., 'nf-core/fastqc', 'myorg/custom') for registry modules. Note: The '@' prefix is only used in DSL include statements, not in meta.yaml", + "description": "Module name. Can be a simple identifier (e.g., 'fastqc', 'bwa_mem') for local/nf-core modules, or a fully qualified scoped name (e.g., 'nf-core/fastqc', 'myorg/custom') for registry modules.", "examples": ["fastqc", "bwa_mem", "nf-core/fastqc", "myorg/salmon-quant"], "pattern": "^([a-z0-9][a-z0-9-]*/)?[a-z][a-z0-9_-]*$" }, diff --git a/docs/cli.md b/docs/cli.md index 3391288f3a..b766eb3c81 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -386,7 +386,7 @@ $ nextflow module publish myorg/my-module $ nextflow module publish myorg/my-module -dry-run ``` -Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or `registry.apiKey` in the Nextflow configuration. The module must include `main.nf`, `meta.yaml`, and `README.md` files. +Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or `registry.apiKey` in the Nextflow configuration. The module must include `main.nf`, `meta.yml`, and `README.md` files. Use `-dry-run` to validate your module structure without uploading. diff --git a/docs/module.md b/docs/module.md index 19346a15cc..5d60879227 100644 --- a/docs/module.md +++ b/docs/module.md @@ -389,7 +389,7 @@ The argument can be either a `scope/name` reference (for an already-installed mo Your module directory must include: - `main.nf`: The module entry point -- `meta.yaml`: Module metadata (name, description, version, etc.) +- `meta.yml`: Module spec (name, description, version, etc.) - `README.md`: Module documentation Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration: @@ -432,8 +432,8 @@ modules/ └── module-name/ ├── .module-info # Integrity checksum (generated automatically) ├── README.md # Documentation (required for publishing) - ├── main.nf # Module entry point (required) - ├── meta.yaml # Module metadata (required for publishing) + ├── main.nf # Module script (required) + ├── meta.yml # Module spec (required for publishing) ├── resources/ # Optional: module binaries and resources └── templates/ # Optional: process templates ``` diff --git a/docs/reference/cli.md b/docs/reference/cli.md index b8fd201188..21f04a72c1 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1308,7 +1308,7 @@ The `module` command provides a comprehensive system for managing reusable, regi : Publish a module to the registry, making it available for others to install. : The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. : Requires authentication via `NXF_REGISTRY_TOKEN` environment variable or `registry.apiKey` configuration. -: The module directory must contain `main.nf`, `meta.yaml`, and `README.md`. +: The module directory must contain `main.nf`, `meta.yml`, and `README.md`. : The following options are available: `-dry-run` diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy index 4c95bc9539..60f59649bb 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy @@ -88,21 +88,21 @@ class CmdModulePublish extends CmdBase { ) } - // Step 2: Load and validate manifest + // Step 2: Load and validate spec def manifestPath = moduleDir.resolve(ModuleStorage.MODULE_MANIFEST_FILE) - def manifest = ModuleSpec.load(manifestPath) + def spec = ModuleSpec.load(manifestPath) - def manifestErrors = manifest.validate() + def manifestErrors = spec.validate() if (!manifestErrors.isEmpty()) { throw new AbortOperationException( - "Module manifest validation failed:\n" + manifestErrors.collect { " - ${it}" }.join('\n') + "Module spec validation failed:\n" + manifestErrors.collect { " - ${it}" }.join('\n') ) } - log.info "Module validated: ${manifest.name}@${manifest.version}" + log.info "Module validated: ${spec.name}@${spec.version}" if (dryRun) { - printDryRunInfo(manifest) + printDryRunInfo(spec) return } @@ -114,11 +114,11 @@ class CmdModulePublish extends CmdBase { def registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() - publishModule(moduleDir, registryConfig, manifest) + publishModule(moduleDir, registryConfig, spec) } - private void publishModule(Path moduleDir, RegistryConfig registryConfig, ModuleSpec manifest){ + private void publishModule(Path moduleDir, RegistryConfig registryConfig, ModuleSpec spec){ log.info "Creating module bundle..." def tempBundleFile = Files.createTempFile("nf-module-publish-", ".tar.gz") @@ -131,7 +131,7 @@ class CmdModulePublish extends CmdBase { // Create publish request as a map (npr-api will serialize it) def request = [ - version: manifest.version, + version: spec.version, bundle: bundleBytes ] @@ -139,7 +139,7 @@ class CmdModulePublish extends CmdBase { final registry = registryUrl ?: registryConfig.url log.info "Publishing module to registry: ${registryUrl ?: registryConfig.url}" def registryClient = new ModuleRegistryClient(registryConfig) - def response = registryClient.publishModule(manifest.name, request, registry) + def response = registryClient.publishModule(spec.name, request, registry) if (useModuleReference) { // If publish is performed using the module reference we should create/update the .module-info with the correct checksum @@ -152,13 +152,13 @@ class CmdModulePublish extends CmdBase { println "✓ Module published successfully!" println "" println "Module details:" - println " Name: ${manifest.name}" - println " Version: ${manifest.version}" + println " Name: ${spec.name}" + println " Version: ${spec.version}" println " DownloadUrl: ${response.downloadUrl}" println "" println "Others can now install this module using:" - println " nextflow module install ${manifest.name}" + println " nextflow module install ${spec.name}" } finally { // Clean up temporary bundle file @@ -172,23 +172,23 @@ class CmdModulePublish extends CmdBase { } } - private void printDryRunInfo(ModuleSpec manifest) { + private void printDryRunInfo(ModuleSpec spec) { println "✓ Module structure is valid" println "" println "Module details:" - println " Name: ${manifest.name}" - println " Version: ${manifest.version}" - println " Description: ${manifest.description}" - println " License: ${manifest.license}" - if( manifest.authors ) { - println " Authors: ${manifest.authors.join(', ')}" + println " Name: ${spec.name}" + println " Version: ${spec.version}" + println " Description: ${spec.description}" + println " License: ${spec.license}" + if( spec.authors ) { + println " Authors: ${spec.authors.join(', ')}" } - if( manifest.keywords ) { - println " Keywords: ${manifest.keywords.join(', ')}" + if( spec.keywords ) { + println " Keywords: ${spec.keywords.join(', ')}" } - if( manifest.requires ) { + if( spec.requires ) { println " Requires:" - manifest.requires.each { name, version -> + spec.requires.each { name, version -> println " - ${name}: ${version}" } } diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy index 7b27b44abf..73ec7d96af 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy @@ -25,7 +25,7 @@ import java.nio.file.Files import java.nio.file.Path /** - * Represents a module manifest (meta.yaml) with validation + * Represents a module spec (meta.yml) with validation * * @author Jorge Ejarque */ @@ -42,38 +42,38 @@ class ModuleSpec { Map requires /** - * Load a module manifest from a meta.yaml file + * Load a module spec from a meta.yml file * - * @param metaYamlPath Path to meta.yaml + * @param metaYamlPath Path to meta.yml * @return ModuleSpec instance */ static ModuleSpec load(Path metaYamlPath) { if( !Files.exists(metaYamlPath) ) { - throw new AbortOperationException("Module manifest not found: ${metaYamlPath}") + throw new AbortOperationException("Module spec not found: ${metaYamlPath}") } try { def yaml = new Yaml() def data = yaml.load(Files.newInputStream(metaYamlPath)) as Map - def manifest = new ModuleSpec() - manifest.name = data.name as String - manifest.version = data.version as String - manifest.description = data.description as String - manifest.authors = data.authors as List ?: [] - manifest.license = data.license as String - manifest.keywords = data.keywords as List ?: [] - manifest.requires = data.requires as Map ?: [:] + def spec = new ModuleSpec() + spec.name = data.name as String + spec.version = data.version as String + spec.description = data.description as String + spec.authors = data.authors as List ?: [] + spec.license = data.license as String + spec.keywords = data.keywords as List ?: [] + spec.requires = data.requires as Map ?: [:] - return manifest + return spec } catch( Exception e ) { - throw new AbortOperationException("Failed to parse module manifest: ${metaYamlPath}", e) + throw new AbortOperationException("Failed to parse module spec: ${metaYamlPath}", e) } } /** - * Validate the module manifest for required fields + * Validate the module spec for required fields * * @return List of validation errors (empty if valid) */ @@ -107,7 +107,7 @@ class ModuleSpec { } /** - * Check if the manifest is valid + * Check if the spec is valid * * @return true if valid, false otherwise */ diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy index 472f0593ef..55d4d9379d 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy @@ -65,7 +65,7 @@ license: MIT def moduleDir = tempDir.resolve('my-module') Files.createDirectories(moduleDir) - // Only create main.nf, missing meta.yaml and README.md + // Only create main.nf, missing meta.yml and README.md moduleDir.resolve('main.nf').text = 'process TEST { }' and: diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy index cf90a07c7c..17aa247679 100644 --- a/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy @@ -32,9 +32,9 @@ class ModuleSpecTest extends Specification { @TempDir Path tempDir - def 'should load valid manifest' () { + def 'should load valid spec' () { given: - def metaYaml = tempDir.resolve('meta.yaml') + def metaYaml = tempDir.resolve('meta.yml') metaYaml.text = ''' name: nf-core/fastqc version: 1.0.0 @@ -50,21 +50,21 @@ requires: ''' when: - def manifest = ModuleSpec.load(metaYaml) + def spec = ModuleSpec.load(metaYaml) then: - manifest.name == 'nf-core/fastqc' - manifest.version == '1.0.0' - manifest.description == 'FastQC quality control' - manifest.authors == ['John Doe'] - manifest.license == 'MIT' - manifest.keywords == ['quality-control', 'fastq'] - manifest.requires == ['nextflow': '>=24.04.0'] + spec.name == 'nf-core/fastqc' + spec.version == '1.0.0' + spec.description == 'FastQC quality control' + spec.authors == ['John Doe'] + spec.license == 'MIT' + spec.keywords == ['quality-control', 'fastq'] + spec.requires == ['nextflow': '>=24.04.0'] } - def 'should fail to load non-existent manifest' () { + def 'should fail to load non-existent spec' () { given: - def metaYaml = tempDir.resolve('meta.yaml') + def metaYaml = tempDir.resolve('meta.yml') when: ModuleSpec.load(metaYaml) @@ -73,9 +73,9 @@ requires: thrown(AbortOperationException) } - def 'should validate complete manifest' () { + def 'should validate complete spec' () { given: - def manifest = new ModuleSpec( + def spec = new ModuleSpec( name: 'nf-core/fastqc', version: '1.0.0', description: 'FastQC quality control', @@ -83,34 +83,34 @@ requires: ) when: - def errors = manifest.validate() + def errors = spec.validate() then: errors.isEmpty() - manifest.isValid() + spec.isValid() } def 'should detect missing required fields' () { given: - def manifest = new ModuleSpec( + def spec = new ModuleSpec( name: 'nf-core/fastqc' // missing version, description, license ) when: - def errors = manifest.validate() + def errors = spec.validate() then: errors.size() == 3 errors.any { it.contains('version') } errors.any { it.contains('description') } errors.any { it.contains('license') } - !manifest.isValid() + !spec.isValid() } def 'should validate version format' () { given: - def manifest = new ModuleSpec( + def spec = new ModuleSpec( name: 'nf-core/fastqc', version: version, description: 'Test', @@ -118,7 +118,7 @@ requires: ) when: - def errors = manifest.validate() + def errors = spec.validate() then: errors.isEmpty() == valid @@ -135,7 +135,7 @@ requires: def 'should validate module name format' () { given: - def manifest = new ModuleSpec( + def spec = new ModuleSpec( name: name, version: '1.0.0', description: 'Test', @@ -143,7 +143,7 @@ requires: ) when: - def errors = manifest.validate() + def errors = spec.validate() then: errors.isEmpty() == valid From 2e183b52c137ae3d63d7b702e57f854d21188366 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 13 Mar 2026 18:41:18 -0500 Subject: [PATCH 71/75] Fix failing test Signed-off-by: Ben Sherman --- .../src/main/groovy/nextflow/script/ProcessEntryHandler.groovy | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy index e124e27e8a..356f262f7d 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy @@ -91,6 +91,7 @@ class ProcessEntryHandler { session.addIgniter { printOutput(processName, output) } + return output } // Create the body definition with execution logic From 0a7ef220c6925f6d2c0f7183bdc8223c9e84b39b Mon Sep 17 00:00:00 2001 From: jorgee Date: Mon, 16 Mar 2026 12:58:49 +0100 Subject: [PATCH 72/75] fix tests Signed-off-by: jorgee --- .../cli/module/CmdModuleRunTest.groovy | 25 ++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy index d93f58be53..9edcbd1600 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy @@ -33,6 +33,7 @@ import test.OutputCapture import java.nio.file.Files import java.nio.file.Path +import java.util.regex.Pattern import java.util.zip.GZIPOutputStream /** @@ -96,15 +97,20 @@ class CmdModuleRunTest extends Specification { Files.write(dest, modulePackage) return dest } + def escapedPath = Pattern.quote(tempDir.toString()) + def pattern = ~/"${escapedPath}\/.+\/test_output\.txt"/ and: def cmd = new CmdModuleRun() + def opts = new CliOptions() + opts.setQuiet(true) cmd.launcher = Mock(Launcher) { - getOptions() >> new CliOptions() + getOptions() >> opts getCliString() >> "nextflow module run nf-core/test-module" } cmd.args = ['nf-core/test-module'] cmd.root = tempDir + cmd.workDir = tempDir.toString() cmd.client = mockClient when: @@ -116,9 +122,7 @@ class CmdModuleRunTest extends Specification { .findResults { line -> !line.contains('INFO') ? line : null }.join(" ") then: - stdout.contains('Executing module...') - stdout.contains('Process CREATE_FILE Outputs:') - stdout.contains("test_output.txt") + assert (stdout =~ pattern).find() and: // Verify module was installed Files.exists(moduleDir) @@ -131,7 +135,7 @@ class CmdModuleRunTest extends Specification { def moduleScript = ''' process CREATE_FILE_V2 { output: - path "test_output_v2.txt" + path "test_output_v2.txt", emit: output_path script: """ @@ -147,6 +151,8 @@ class CmdModuleRunTest extends Specification { Files.createDirectories(moduleDir) moduleDir.resolve('main.nf').text = moduleScript moduleDir.resolve('meta.yml').text = 'name: nf-core/test-module\nversion: 2.0.0' + def escapedPath = Pattern.quote(tempDir.toString()) + def pattern = ~/"output_path": "${escapedPath}\/.+\/test_output_v2\.txt"/ and: def modulePackage = createModulePackage(moduleScript) @@ -159,13 +165,16 @@ class CmdModuleRunTest extends Specification { and: def cmd = new CmdModuleRun() + def opts = new CliOptions() + opts.setQuiet(true) cmd.launcher = Mock(Launcher) { - getOptions() >> new CliOptions() + getOptions() >> opts getCliString() >> "nextflow module run nf-core/test-module" } cmd.args = ['nf-core/test-module'] cmd.version = '2.0.0' cmd.root = tempDir + cmd.workDir = tempDir.toString() cmd.client = mockClient when: @@ -178,9 +187,7 @@ class CmdModuleRunTest extends Specification { .findResults { line -> !line.contains('DEBUG') ? line : null } .findResults { line -> !line.contains('INFO') ? line : null } .findResults { line -> !line.contains('plugin') ? line : null }.join(" ") - stdout.contains('Executing module...') - stdout.contains('Process CREATE_FILE_V2 Outputs:') - stdout.contains("test_output_v2.txt") + assert (stdout =~ pattern).find() } From 6c15bc870da67fabe4548593a8fcfae7f7844771 Mon Sep 17 00:00:00 2001 From: jorgee Date: Mon, 16 Mar 2026 13:02:34 +0100 Subject: [PATCH 73/75] modify module list to skip subdirectories of directories that already contains the .module-info. This folders can't be a module Signed-off-by: jorgee --- .../nextflow/module/ModuleStorage.groovy | 37 ++++++++++++------- .../nextflow/module/ModuleStorageTest.groovy | 31 ++++++++++++++++ 2 files changed, 54 insertions(+), 14 deletions(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy index 6c9a90ca87..825aae004b 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy @@ -27,8 +27,11 @@ import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream import org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream import org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream +import java.nio.file.FileVisitResult import java.nio.file.Files import java.nio.file.Path +import java.nio.file.SimpleFileVisitor +import java.nio.file.attribute.BasicFileAttributes import java.util.stream.Stream import java.util.zip.ZipEntry import java.util.zip.ZipInputStream @@ -136,22 +139,28 @@ class ModuleStorage { List modules = [] - try( final walkStream = Files.walk(modulesDir) ) { - walkStream - .filter { Path path -> Files.isDirectory(path) } - .filter { Path path -> Files.exists(path.resolve(MODULE_INFO_FILE)) } - .each { Path moduleDir -> - try { - def rel = modulesDir.relativize(moduleDir) - if( rel.nameCount < 2 ) return // Need at least scope/name - def reference = ModuleReference.parse(rel.toString()) - def installed = getInstalledModule(reference) - if( installed ) modules.add(installed) - } catch(Exception e){ - // Catching exception to go on inspecting other valid folders - log.debug("Not a valid module reference - $e.message") + try { + Files.walkFileTree(modulesDir, new SimpleFileVisitor() { + @Override + FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) { + if( dir == modulesDir ) return FileVisitResult.CONTINUE + if( Files.exists(dir.resolve(MODULE_INFO_FILE)) ) { + try { + def rel = modulesDir.relativize(dir) + if( rel.nameCount >= 2 ) { + def reference = ModuleReference.parse(rel.toString()) + def installed = getInstalledModule(reference) + if( installed ) modules.add(installed) + } + } catch(Exception e) { + // Catching exception to go on inspecting other valid folders + log.debug("Not a valid module reference - $e.message") + } + return FileVisitResult.SKIP_SUBTREE } + return FileVisitResult.CONTINUE } + }) } catch (IOException e) { log.warn "Failed to scan modules directory ${modulesDir}: ${e.message}" } diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy index bb44f8fa94..9abf819b21 100644 --- a/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy @@ -207,6 +207,37 @@ class ModuleStorageTest extends Specification { ] } + def 'should ignore directories without MODULE_INFO_FILE and not descend into module subdirectories'() { + given: + def storage = new ModuleStorage(tempDir) + + // A valid module with subdirectories (bin/, src/) + def moduleDir = storage.getModuleDir(new ModuleReference('nf-core', 'fastqc')) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process FASTQC { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + ModuleChecksum.save(moduleDir, 'checksum') + // Subdirectories inside the module — must NOT be listed as separate modules + def binDir = moduleDir.resolve('bin') + Files.createDirectories(binDir) + binDir.resolve('fastqc.sh').text = '#!/bin/bash' + def srcDir = moduleDir.resolve('src/main') + Files.createDirectories(srcDir) + srcDir.resolve('helper.groovy').text = 'def foo() {}' + + // A plain directory tree with no MODULE_INFO_FILE anywhere — must be ignored entirely + def orphanDir = tempDir.resolve('modules/other-org/tool/deep/nested') + Files.createDirectories(orphanDir) + orphanDir.resolve('somefile.txt').text = 'not a module' + + when: + def installed = storage.listInstalled() + + then: + installed.size() == 1 + installed[0].reference.fullName == 'nf-core/fastqc' + } + def 'should return empty list when no modules installed'() { given: def storage = new ModuleStorage(tempDir) From 3be2f023c2fa651ab992c06b00e4a2d5c8369f9a Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Mon, 16 Mar 2026 10:49:55 -0500 Subject: [PATCH 74/75] Update docs Signed-off-by: Ben Sherman --- docs/cli.md | 86 +++++---- docs/module.md | 7 +- docs/reference/cli.md | 182 +++++++++--------- .../nextflow/module/ModuleStorage.groovy | 28 +-- 4 files changed, 151 insertions(+), 152 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index b766eb3c81..b09e03c4de 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -269,49 +269,42 @@ See {ref}`cli-secrets` for more information. :::{versionadded} 26.04.0 ::: -Module management commands enable working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and spreading improvements throughout the community. +The `module` command enables working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and sharing improvements with the community. Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use. -### Installing modules +### Searching for modules -The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL. +The `module search` command queries the module registry to discover available modules by keyword or name. -Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions. +Use this to find modules for specific tasks, explore available tools, or discover community modules. ```console -$ nextflow module install nf-core/fastqc -$ nextflow module install nf-core/fastqc -version 1.0.0 +$ nextflow module search alignment +$ nextflow module search "quality control" -limit 10 +$ nextflow module search bwa -output json ``` -After installation, module will be available in `modules/nf-core/fastqc`. +Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for JSON-formatted output. -Use the `-force` flag to reinstall a module even if local modifications exist. - -See {ref}`cli-module-install` for more information. +See {ref}`cli-module-search` for more information. -### Running modules directly +### Installing modules -The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing. +The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL. -Use this to quickly run a module, test module functionality, or execute one-off data processing tasks. +Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions. ```console -$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0 +$ nextflow module install nf-core/fastqc +$ nextflow module install nf-core/fastqc -version 1.0.0 ``` -The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.): +The installed module will be available in `modules/nf-core/fastqc`. -```console -$ nextflow module run nf-core/salmon \ - --reads reads.fq \ - --index salmon_index \ - -profile docker \ - -resume -``` +Use the `-force` flag to reinstall a module even if local modifications exist. -See {ref}`cli-module-run` for more information. +See {ref}`cli-module-install` for more information. ### Listing modules @@ -324,26 +317,10 @@ $ nextflow module list $ nextflow module list -output json ``` -The output shows each module's name, installed version, and whether it has been modified locally. Use `-json` for machine-readable output suitable for scripting. +The output shows each module's name, installed version, and whether it has been modified locally. Use `-o json` for JSON-formatted output. See {ref}`cli-module-list` for more information. -### Searching for modules - -The `module search` command queries the module registry to discover available modules by keyword or name. - -Use this to find modules for specific tasks, explore available tools, or discover community contributions. - -```console -$ nextflow module search alignment -$ nextflow module search "quality control" -limit 10 -$ nextflow module search bwa -output json -``` - -Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for programmatic access. - -See {ref}`cli-module-search` for more information. - ### Viewing module information The `module info` command displays detailed metadata and usage information for a specific module from the registry. @@ -360,6 +337,31 @@ The output includes the module's version, description, authors, keywords, tools, See {ref}`cli-module-info` for more information. +### Running modules directly + +The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing. + +Use this to quickly run a module, test module functionality, or execute one-off data processing tasks. + +```console +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0 +``` + +The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.): + +```console +$ nextflow module run nf-core/salmon \ + --reads reads.fq \ + --index salmon_index \ + -profile docker \ + -resume +``` + +Process inputs can be specified like params on the command line. For example, `--reads reads.fq` corresponds to the `reads` input in the `nf-core/salmon` module. Run `nextflow module info nf-core/salmon` to see the available params for the module. + +See {ref}`cli-module-run` for more information. + ### Removing modules The `module remove` command deletes modules from your project, removing local files and configuration entries. @@ -386,7 +388,7 @@ $ nextflow module publish myorg/my-module $ nextflow module publish myorg/my-module -dry-run ``` -Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or `registry.apiKey` in the Nextflow configuration. The module must include `main.nf`, `meta.yml`, and `README.md` files. +Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or the `registry.apiKey` config option. The module must include `main.nf`, `meta.yml`, and `README.md` files. Use `-dry-run` to validate your module structure without uploading. diff --git a/docs/module.md b/docs/module.md index 5d60879227..eb5361d9fd 100644 --- a/docs/module.md +++ b/docs/module.md @@ -293,7 +293,7 @@ Modules are designed to be easy to share and re-use across different pipelines, :::{versionadded} 26.04.0 ::: -Nextflow provides a module registry that enables you to install, share, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow DSL. +Nextflow provides a module registry that enables you to install, publish, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow language. ### Installing modules from a registry @@ -323,8 +323,7 @@ For ad-hoc tasks or testing, you can run a module directly without creating a wo $ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' ``` -This command accepts all standard Nextflow options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed. - +This command accepts all standard `nextflow run` options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed. ### Discovering modules @@ -374,7 +373,7 @@ $ nextflow module info nf-core/fastqc $ nextflow module info nf-core/fastqc -version 1.0.0 ``` -The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-json` to get machine-readable output. +The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-o json` to get machine-readable output. ### Publishing modules diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 21f04a72c1..74be2fdd20 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1132,7 +1132,7 @@ work/1f/f1ea9158fb23b53d5083953121d6b6 :::{versionadded} 26.04.0 ::: -Manage Nextflow modules from registries. +Manage Nextflow modules. **Usage** @@ -1142,64 +1142,63 @@ $ nextflow module [options] **Description** -The `module` command provides a comprehensive system for managing reusable, registry-based modules. It enables installing modules from registries, running them directly, searching for available modules, and publishing your own modules for community use. +The `module` command provides a comprehensive system for managing registry-based modules. It enables installing modules from registries, running them directly, searching for available modules, and publishing your own modules to a registry. **Subcommands** -(cli-module-install)= +(cli-module-info)= -`install [options] [scope/name]` +`info [options] [scope/name]` -: Install a module from the registry into your project. -: Downloaded modules are stored in the `modules/` directory. -: The `.module-info` file is created in the module directory during the installation to store additional information of the installed module, such as the checksum of the downloaded module files to detect if a module is locally modified and the URL of the registry used to download the module. +: Display detailed information about a module from the registry. +: Shows module name, version, description, and other metadata, as well as example usage. : The following options are available: `-version` - : Specify the module version to install (e.g., `1.0.0`). If not specified, installs the latest version. + : Specify the module version to query (e.g., `1.0.0`). If not specified, displays information for the latest version. - `-force` - : Force reinstall even if the module exists locally with modifications. Without this flag, Nextflow prevents overwriting locally modified modules. + `-o, -output` (`text`) + : Output mode for info results. Options: `text` (default), `json`. : **Examples:** ```console - # Install latest version - $ nextflow module install nf-core/fastqc + # Display information for latest version + $ nextflow module info nf-core/fastqc - # Install specific version - $ nextflow module install nf-core/fastqc -version 1.0.0 + # Display information for specific version + $ nextflow module info nf-core/fastqc -version 1.0.0 - # Force reinstall over local modifications - $ nextflow module install nf-core/fastqc -force + # Get results as JSON + $ nextflow module info nf-core/fastqc -output json ``` -(cli-module-run)= +(cli-module-install)= -`run [options] [scope/name] [-- ]` +`install [options] [scope/name]` -: Execute a module directly from the registry without creating a wrapper workflow. -: Automatically downloads the module if not already installed. Accepts all standard Nextflow run options. +: Install a module from the registry into your project. +: Downloaded modules are stored in the `modules/` directory. +: The `.module-info` file is created in the module directory to store additional information of the installed module. : The following options are available: `-version` - : Specify the module version to run (e.g., `1.0.0`). If not specified, uses the latest version. + : Specify the module version to install (e.g., `1.0.0`). If not specified, installs the latest version. - All standard `run` command options - : The `module run` command extends the `run` command and accepts all its options, including `-profile`, `-resume`, `-c`, etc. + `-force` + : Force reinstall even if the module exists locally with modifications. Without this flag, Nextflow prevents overwriting locally modified modules. : **Examples:** ```console - # Run module with inputs - $ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' + # Install latest version + $ nextflow module install nf-core/fastqc - # Run specific version with Nextflow options - $ nextflow module run nf-core/fastqc \ - --input 'data/*.fastq.gz' \ - -version 1.0.0 \ - -profile docker \ - -resume + # Install specific version + $ nextflow module install nf-core/fastqc -version 1.0.0 + + # Force reinstall over local modifications + $ nextflow module install nf-core/fastqc -force ``` (cli-module-list)= @@ -1207,7 +1206,7 @@ The `module` command provides a comprehensive system for managing reusable, regi `list [options]` : List all modules currently installed in your project. -: Shows module names, versions, and integrity status (whether they've been modified locally). +: Shows each module's name, version, and integrity status (whether it has been modified locally). : The following options are available: `-o, -output` (`table`) @@ -1223,58 +1222,35 @@ The `module` command provides a comprehensive system for managing reusable, regi $ nextflow module list -output 'json' ``` -(cli-module-search)= - -`search [options] [query]` - -: Search for modules in the registry by keyword or name. -: Returns modules matching the query with their names, versions, descriptions, and download statistics. -: The following options are available: - - `-limit` - : Maximum number of results to return (default: varies by registry). - - `-o, -output` (`simple`) - : Output mode for search results. Options: `simple` (default), `json`. - -: **Examples:** - - ```console - # Search for alignment-related modules - $ nextflow module search alignment - - # Search with limited results - $ nextflow module search "quality control" -limit 10 - - # Get results as JSON - $ nextflow module search bwa -output json - ``` - -(cli-module-info)= +(cli-module-publish)= -`info [options] [scope/name]` +`publish [options] [scope/name | path]` -: Display detailed information about a module from the registry. -: Shows module metadata, version, description, authors, keywords, tools, input/output specifications, and generates a usage template. +: Publish a module to the registry, making it available for others to install. +: The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. +: Requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or the `registry.apiKey` config option. +: The module directory must contain `main.nf`, `meta.yml`, and `README.md`. : The following options are available: - `-version` - : Specify the module version to query (e.g., `1.0.0`). If not specified, displays information for the latest version. + `-dry-run` + : Validate the module structure and metadata without uploading to the registry. Useful for testing before publishing. - `-o, -output` (`text`) - : Output mode for info results. Options: `text` (default), `json`. + `-registry` + : Specify the registry to publish the module (default: `https://registry.nextflow.io`) : **Examples:** ```console - # Display information for latest version - $ nextflow module info nf-core/fastqc + # Validate module structure without publishing + $ nextflow module publish myorg/my-module -dry-run - # Display information for specific version - $ nextflow module info nf-core/fastqc -version 1.0.0 + # Publish to nextflow registry + $ export NXF_REGISTRY_TOKEN=your-token + $ nextflow module publish myorg/my-module - # Get results as JSON - $ nextflow module info nf-core/fastqc -output json + # Publish to a custom registry + $ export NXF_REGISTRY_TOKEN=your-token + $ nextflow module publish myorg/my-module -registry 'https://custom.registry.com' ``` (cli-module-remove)= @@ -1301,35 +1277,57 @@ The `module` command provides a comprehensive system for managing reusable, regi $ nextflow module remove nf-core/fastqc -keep-files ``` -(cli-module-publish)= +(cli-module-run)= -`publish [options] [scope/name | path]` +`run [options] [scope/name] [-- ]` -: Publish a module to the registry, making it available for others to install. -: The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. -: Requires authentication via `NXF_REGISTRY_TOKEN` environment variable or `registry.apiKey` configuration. -: The module directory must contain `main.nf`, `meta.yml`, and `README.md`. +: Execute a module directly from the registry without creating a wrapper workflow. +: Automatically downloads the module if not already installed. Accepts all standard Nextflow run options. +: The `module run` command extends the `run` command and accepts all its options, including `-profile`, `-resume`, `-c`, etc. Command-line params (i.e., `--`) are inferred from the module's declared inputs. +: The following additional options are available: + + `-version` + : Specify the module version to run (e.g., `1.0.0`). If not specified, uses the latest version. + +: **Examples:** + + ```console + # Run module with inputs + $ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' + + # Run specific version with Nextflow options + $ nextflow module run nf-core/fastqc \ + --input 'data/*.fastq.gz' \ + -version 1.0.0 \ + -profile docker \ + -resume + ``` + +(cli-module-search)= + +`search [options] [query]` + +: Search for modules in the registry by keyword or name. +: Returns modules matching the query with their names, versions, descriptions, and download statistics. : The following options are available: - `-dry-run` - : Validate the module structure and metadata without uploading to the registry. Useful for testing before publishing. + `-limit` + : Maximum number of results to return (default: varies by registry). - `-registry` - : Specify the registry to publish the module (default: `https://registry.nextflow.io`) + `-o, -output` (`simple`) + : Output mode for search results. Options: `simple` (default), `json`. : **Examples:** ```console - # Validate module structure without publishing - $ nextflow module publish myorg/my-module -dry-run + # Search for alignment-related modules + $ nextflow module search alignment - # Publish to nextflow registry - $ export NXF_REGISTRY_TOKEN=your-token - $ nextflow module publish myorg/my-module + # Search with limited results + $ nextflow module search "quality control" -limit 10 - # Publish to a custom registry - $ export NXF_REGISTRY_TOKEN=your-token - $ nextflow module publish myorg/my-module -registry 'https://custom.registry.com' + # Get results as JSON + $ nextflow module search bwa -output json ``` (cli-plugin)= @@ -1378,7 +1376,7 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo : Update all downloaded projects. `-d, -deep` -: :::{deprecated} 25.12.0-edge. +: :::{deprecated} 25.12.0-edge Ignored for new multi-revision asset management strategy. Still used in legacy assets. ::: : Create a shallow clone of the specified depth. diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy index 825aae004b..2e1b90c1c1 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy @@ -143,22 +143,22 @@ class ModuleStorage { Files.walkFileTree(modulesDir, new SimpleFileVisitor() { @Override FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) { - if( dir == modulesDir ) return FileVisitResult.CONTINUE - if( Files.exists(dir.resolve(MODULE_INFO_FILE)) ) { - try { - def rel = modulesDir.relativize(dir) - if( rel.nameCount >= 2 ) { - def reference = ModuleReference.parse(rel.toString()) - def installed = getInstalledModule(reference) - if( installed ) modules.add(installed) - } - } catch(Exception e) { - // Catching exception to go on inspecting other valid folders - log.debug("Not a valid module reference - $e.message") + if( dir == modulesDir ) + return FileVisitResult.CONTINUE + if( !Files.exists(dir.resolve(MODULE_INFO_FILE)) ) + return FileVisitResult.CONTINUE + try { + final rel = modulesDir.relativize(dir) + if( rel.nameCount >= 2 ) { + final reference = ModuleReference.parse(rel.toString()) + final installed = getInstalledModule(reference) + if( installed ) modules.add(installed) } - return FileVisitResult.SKIP_SUBTREE + } catch(Exception e) { + // Catching exception to continue inspecting other valid folders + log.debug("Not a valid module reference: $e.message") } - return FileVisitResult.CONTINUE + return FileVisitResult.SKIP_SUBTREE } }) } catch (IOException e) { From 34426eab5857564133f466546ce196e563bf152b Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Mon, 16 Mar 2026 18:19:37 +0100 Subject: [PATCH 75/75] Apply suggestion from @pditommaso [ci skip] Signed-off-by: Paolo Di Tommaso --- adr/20251114-module-system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md index 123463c2e6..f819b81068 100644 --- a/adr/20251114-module-system.md +++ b/adr/20251114-module-system.md @@ -1,7 +1,7 @@ # Module System for Nextflow - Authors: Paolo Di Tommaso -- Status: draft +- Status: approved - Date: 2025-01-06 - Tags: modules, dsl, registry, versioning, architecture - Version: 2.7