diff --git a/docs/cli.md b/docs/cli.md index 607a041756..3391288f3a 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -262,7 +262,135 @@ $ nextflow secrets set AWS_ACCESS_KEY_ID $ nextflow secrets delete AWS_ACCESS_KEY_ID ``` -See {ref}`cli-secrets` for more information. +See {ref}`cli-secrets` for more information. + +## Module management + +:::{versionadded} 26.04.0 +::: + +Module management commands enable working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and spreading improvements throughout the community. + +Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use. + +### Installing modules + +The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL. + +Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions. + +```console +$ nextflow module install nf-core/fastqc +$ nextflow module install nf-core/fastqc -version 1.0.0 +``` + +After installation, module will be available in `modules/nf-core/fastqc`. + +Use the `-force` flag to reinstall a module even if local modifications exist. + +See {ref}`cli-module-install` for more information. + +### Running modules directly + +The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing. + +Use this to quickly run a module, test module functionality, or execute one-off data processing tasks. + +```console +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0 +``` + +The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.): + +```console +$ nextflow module run nf-core/salmon \ + --reads reads.fq \ + --index salmon_index \ + -profile docker \ + -resume +``` + +See {ref}`cli-module-run` for more information. + +### Listing modules + +The `module list` command displays all modules currently installed in your project, showing their versions and integrity status. + +Use this to review installed modules, check module versions, or detect local modifications. + +```console +$ nextflow module list +$ nextflow module list -output json +``` + +The output shows each module's name, installed version, and whether it has been modified locally. Use `-json` for machine-readable output suitable for scripting. + +See {ref}`cli-module-list` for more information. + +### Searching for modules + +The `module search` command queries the module registry to discover available modules by keyword or name. + +Use this to find modules for specific tasks, explore available tools, or discover community contributions. + +```console +$ nextflow module search alignment +$ nextflow module search "quality control" -limit 10 +$ nextflow module search bwa -output json +``` + +Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for programmatic access. + +See {ref}`cli-module-search` for more information. + +### Viewing module information + +The `module info` command displays detailed metadata and usage information for a specific module from the registry. + +Use this to understand module requirements, view input/output specifications, see available tools, or generate usage templates before installing or running a module. + +```console +$ nextflow module info nf-core/fastqc +$ nextflow module info nf-core/fastqc -version 1.0.0 +$ nextflow module info nf-core/fastqc -output json +``` + +The output includes the module's version, description, authors, keywords, tools, input/output channels, and a generated usage template showing how to run the module. Use `-json` for machine-readable output suitable for programmatic access. + +See {ref}`cli-module-info` for more information. + +### Removing modules + +The `module remove` command deletes modules from your project, removing local files and configuration entries. + +Use this to clean up unused modules, free disk space, or remove deprecated modules from your pipeline. + +```console +$ nextflow module remove nf-core/fastqc +$ nextflow module remove nf-core/fastqc -keep-files +``` + +By default, both local files and configuration entries are removed. Use `-keep-files` to remove the configuration entry and `.module-info` while keeping local files. + +See {ref}`cli-module-remove` for more information. + +### Publishing modules + +The `module publish` command uploads modules to a registry, making them available for others to install and use. + +Use this to share your modules with the community, contribute to module libraries, or distribute modules within your organization. + +```console +$ nextflow module publish myorg/my-module +$ nextflow module publish myorg/my-module -dry-run +``` + +Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or `registry.apiKey` in the Nextflow configuration. The module must include `main.nf`, `meta.yaml`, and `README.md` files. + +Use `-dry-run` to validate your module structure without uploading. + +See {ref}`cli-module-publish` for more information. ## Configuration and validation @@ -384,7 +512,7 @@ Use this to understand input/output relationships between tasks, trace data flow $ nextflow lineage ``` -See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information. +See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information. ## Seqera Platform diff --git a/docs/module.md b/docs/module.md index 73d92fe4e2..747ae76395 100644 --- a/docs/module.md +++ b/docs/module.md @@ -279,8 +279,165 @@ This feature requires the use of a local or shared file system for the pipeline ## Sharing modules -Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. While Nextflow does not provide an explicit mechanism for sharing modules, there are several ways to do it: +Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules: +- Use the Nextflow module registry (recommended, see below) - Simply copy the module files into your pipeline repository - Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy - Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community + +(module-registry)= + +## Registry-based modules + +:::{versionadded} 26.04.0 +::: + +Nextflow provides a module registry that enables you to install, share, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow DSL. + +### Installing modules from a registry + +Use the `module install` command to download modules from a registry: + +```console +$ nextflow module install nf-core/fastqc +$ nextflow module install nf-core/fastqc -version 1.0.0 +``` + +Installed modules are stored in the `modules/` directory and can be included by name instead of a relative path: + +```nextflow +include { FASTQC } from 'nf-core/fastqc' + +workflow { + reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz') + FASTQC(reads) +} +``` + +### Running modules directly + +For ad-hoc tasks or testing, you can run a module directly without creating a workflow: + +```console +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' +``` + +This command accepts all standard Nextflow options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed. + + +### Discovering modules + +Search for available modules using the `module search` command: + +```console +$ nextflow module search alignment +$ nextflow module search "quality control" -limit 10 +``` + +List installed modules in your project: + +```console +$ nextflow module list +``` + +### Module checksum verification + +Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites: + +```console +$ nextflow module install nf-core/fastqc -version 1.1.0 +Warning: Module nf-core/fastqc has local modifications. Use -force to override. +``` + +Use the `-force` flag to override local modifications when needed. + +### Removing modules + +Use the `module remove` command to uninstall a module: + +```console +$ nextflow module remove nf-core/fastqc +``` + +By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour: + +- `-keep-files` — Remove the `.module-info` file created at install but keep the rest of files +- `-force` — Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications + +### Viewing module information + +Use the `module info` command to display metadata and a usage template for a module: + +```console +$ nextflow module info nf-core/fastqc +$ nextflow module info nf-core/fastqc -version 1.0.0 +``` + +The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-json` to get machine-readable output. + +### Publishing modules + +To share your own modules, use the `module publish` command: + +```console +$ nextflow module publish myorg/my-module +``` + +The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. + +Your module directory must include: + +- `main.nf`: The module entry point +- `meta.yaml`: Module metadata (name, description, version, etc.) +- `README.md`: Module documentation + +Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration: + +```groovy +registry { + apiKey = 'YOUR_REGISTRY_TOKEN' +} +``` + +Use `-dry-run` to validate your module structure without uploading: + +```console +$ nextflow module publish myorg/my-module -dry-run +``` + +### Registry configuration + +By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries: + +```groovy +registry { + url = [ + 'https://private.registry.myorg.com', + 'https://registry.nextflow.io' + ] + apiKey = '${MYORG_TOKEN}' +} +``` + +Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry. + +### Module directory structure + +Registry modules follow a standard directory structure: + +``` +modules/ +└── scope/ + └── module-name/ + ├── .checksum # Integrity checksum (generated automatically) + ├── README.md # Documentation (required for publishing) + ├── main.nf # Module entry point (required) + ├── meta.yaml # Module metadata (required for publishing) + ├── resources/ # Optional: module binaries and resources + └── templates/ # Optional: process templates +``` + +The `modules/` directory should be committed to your Git repository to ensure reproducibility. + +See the {ref}`cli-page` documentation for more information about module commands. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 23f7ed1855..b8fd201188 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1125,6 +1125,213 @@ $ nextflow log tiny_leavitt -F 'process =~ /split_letters/' work/1f/f1ea9158fb23b53d5083953121d6b6 ``` +(cli-module)= + +### `module` + +:::{versionadded} 26.04.0 +::: + +Manage Nextflow modules from registries. + +**Usage** + +```console +$ nextflow module [options] +``` + +**Description** + +The `module` command provides a comprehensive system for managing reusable, registry-based modules. It enables installing modules from registries, running them directly, searching for available modules, and publishing your own modules for community use. + +**Subcommands** + +(cli-module-install)= + +`install [options] [scope/name]` + +: Install a module from the registry into your project. +: Downloaded modules are stored in the `modules/` directory. +: The `.module-info` file is created in the module directory during the installation to store additional information of the installed module, such as the checksum of the downloaded module files to detect if a module is locally modified and the URL of the registry used to download the module. +: The following options are available: + + `-version` + : Specify the module version to install (e.g., `1.0.0`). If not specified, installs the latest version. + + `-force` + : Force reinstall even if the module exists locally with modifications. Without this flag, Nextflow prevents overwriting locally modified modules. + +: **Examples:** + + ```console + # Install latest version + $ nextflow module install nf-core/fastqc + + # Install specific version + $ nextflow module install nf-core/fastqc -version 1.0.0 + + # Force reinstall over local modifications + $ nextflow module install nf-core/fastqc -force + ``` + +(cli-module-run)= + +`run [options] [scope/name] [-- ]` + +: Execute a module directly from the registry without creating a wrapper workflow. +: Automatically downloads the module if not already installed. Accepts all standard Nextflow run options. +: The following options are available: + + `-version` + : Specify the module version to run (e.g., `1.0.0`). If not specified, uses the latest version. + + All standard `run` command options + : The `module run` command extends the `run` command and accepts all its options, including `-profile`, `-resume`, `-c`, etc. + +: **Examples:** + + ```console + # Run module with inputs + $ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' + + # Run specific version with Nextflow options + $ nextflow module run nf-core/fastqc \ + --input 'data/*.fastq.gz' \ + -version 1.0.0 \ + -profile docker \ + -resume + ``` + +(cli-module-list)= + +`list [options]` + +: List all modules currently installed in your project. +: Shows module names, versions, and integrity status (whether they've been modified locally). +: The following options are available: + + `-o, -output` (`table`) + : Output mode for list results. Options: `table` (default), `json`. + +: **Examples:** + + ```console + # Display installed modules in formatted table + $ nextflow module list + + # Output as JSON + $ nextflow module list -output 'json' + ``` + +(cli-module-search)= + +`search [options] [query]` + +: Search for modules in the registry by keyword or name. +: Returns modules matching the query with their names, versions, descriptions, and download statistics. +: The following options are available: + + `-limit` + : Maximum number of results to return (default: varies by registry). + + `-o, -output` (`simple`) + : Output mode for search results. Options: `simple` (default), `json`. + +: **Examples:** + + ```console + # Search for alignment-related modules + $ nextflow module search alignment + + # Search with limited results + $ nextflow module search "quality control" -limit 10 + + # Get results as JSON + $ nextflow module search bwa -output json + ``` + +(cli-module-info)= + +`info [options] [scope/name]` + +: Display detailed information about a module from the registry. +: Shows module metadata, version, description, authors, keywords, tools, input/output specifications, and generates a usage template. +: The following options are available: + + `-version` + : Specify the module version to query (e.g., `1.0.0`). If not specified, displays information for the latest version. + + `-o, -output` (`text`) + : Output mode for info results. Options: `text` (default), `json`. + +: **Examples:** + + ```console + # Display information for latest version + $ nextflow module info nf-core/fastqc + + # Display information for specific version + $ nextflow module info nf-core/fastqc -version 1.0.0 + + # Get results as JSON + $ nextflow module info nf-core/fastqc -output json + ``` + +(cli-module-remove)= + +`remove [options] [scope/name]` + +: Remove a module from your project. +: By default, removes both local files and configuration entries. Use options to control what gets removed. +: The following options are available: + + `-force` + : Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications. + + `-keep-files` + : Remove the `.module-info` but keep local files in the `modules/` directory. + +: **Examples:** + + ```console + # Remove module completely + $ nextflow module remove nf-core/fastqc + + # Remove from config but keep local files + $ nextflow module remove nf-core/fastqc -keep-files + ``` + +(cli-module-publish)= + +`publish [options] [scope/name | path]` + +: Publish a module to the registry, making it available for others to install. +: The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. +: Requires authentication via `NXF_REGISTRY_TOKEN` environment variable or `registry.apiKey` configuration. +: The module directory must contain `main.nf`, `meta.yaml`, and `README.md`. +: The following options are available: + + `-dry-run` + : Validate the module structure and metadata without uploading to the registry. Useful for testing before publishing. + + `-registry` + : Specify the registry to publish the module (default: `https://registry.nextflow.io`) + +: **Examples:** + + ```console + # Validate module structure without publishing + $ nextflow module publish myorg/my-module -dry-run + + # Publish to nextflow registry + $ export NXF_REGISTRY_TOKEN=your-token + $ nextflow module publish myorg/my-module + + # Publish to a custom registry + $ export NXF_REGISTRY_TOKEN=your-token + $ nextflow module publish myorg/my-module -registry 'https://custom.registry.com' + ``` + (cli-plugin)= ### `plugin` @@ -1171,7 +1378,7 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo : Update all downloaded projects. `-d, -deep` -: :::{deprecated} 25.12.0-edge. +: :::{deprecated} 25.12.0-edge. Ignored for new multi-revision asset management strategy. Still used in legacy assets. ::: : Create a shallow clone of the specified depth. diff --git a/modules/nextflow/build.gradle b/modules/nextflow/build.gradle index 2c083183aa..c4323f909b 100644 --- a/modules/nextflow/build.gradle +++ b/modules/nextflow/build.gradle @@ -70,9 +70,12 @@ dependencies { api 'dev.failsafe:failsafe:3.1.0' api 'io.seqera:lib-trace:0.1.0' api 'com.fasterxml.woodstox:woodstox-core:7.1.1' + api 'org.apache.commons:commons-compress:1.27.1' // For tar.gz extraction + api 'io.seqera:npr-api:0.21.4-SNAPSHOT' testImplementation 'org.subethamail:subethasmtp:3.1.7' testImplementation (project(':nf-lineage')) + testImplementation 'org.wiremock:wiremock:3.13.1' // test configuration testFixturesApi ("org.apache.groovy:groovy-test:4.0.30") { exclude group: 'org.apache.groovy' } testFixturesApi ("org.objenesis:objenesis:3.4") diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdBase.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdBase.groovy index 1b115bda15..aef79a7a4e 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdBase.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdBase.groovy @@ -26,9 +26,14 @@ import com.beust.jcommander.Parameter abstract class CmdBase implements Runnable { private Launcher launcher + private List unknownOptions abstract String getName() + protected List getUnknownOptions(){ return this.unknownOptions } + + void setUnknownOptions(List options){ this.unknownOptions = options } + Launcher getLauncher() { launcher } void setLauncher( Launcher value ) { this.launcher = value } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdModule.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdModule.groovy new file mode 100644 index 0000000000..241c8e631f --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdModule.groovy @@ -0,0 +1,152 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli + +import com.beust.jcommander.JCommander +import com.beust.jcommander.Parameter +import com.beust.jcommander.ParameterException +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.cli.module.CmdModuleInfo +import nextflow.cli.module.CmdModuleInstall +import nextflow.cli.module.CmdModuleList +import nextflow.cli.module.CmdModulePublish +import nextflow.cli.module.CmdModuleRemove +import nextflow.cli.module.CmdModuleRun +import nextflow.cli.module.CmdModuleSearch +import nextflow.exception.AbortOperationException + +/** + * Implements `module` command + * + * @author Jorge Ejarque + */ +@CompileStatic +@Slf4j +@Parameters(commandDescription = "Manage Nextflow modules") +class CmdModule extends CmdBase implements UsageAware { + + static final public String NAME = 'module' + + private JCommander jCommander + + static final List commands = new ArrayList<>() + + static { + commands << new CmdModuleInstall() + commands << new CmdModuleRun() + commands << new CmdModuleList() + commands << new CmdModuleRemove() + commands << new CmdModuleSearch() + commands << new CmdModuleInfo() + commands << new CmdModulePublish() + } + + protected JCommander commander() { + if( !this.jCommander ) { + this.jCommander = new JCommander(this) + this.jCommander.setProgramName('nextflow module') + // Register all subcommands + commands.each { cmd -> + cmd.launcher = this.launcher + this.jCommander.addCommand(cmd.getName(), cmd, new String[0]) + } + } + return jCommander + } + + @Parameter + List args + + @Override + String getName() { + return NAME + } + + @Override + void run() { + + + try { + if( !args ) { + usage() + return + } + final jc = commander() + final moduleArgs = args + unknownOptions + jc.parse(moduleArgs as String[]) + + final parsedCommand = jc.getParsedCommand() + if( !parsedCommand ) { + jc.usage() + return + } + + // Get the parsed subcommand instance + final subcommand = jc.getCommands() + .get(parsedCommand) + .getObjects()[0] as CmdBase + + // Execute with fields already populated by JCommander + subcommand.run() + + } catch( ParameterException e ) { + throw new AbortOperationException("${e.getMessage()} -- Check the available commands and options and syntax with 'nextflow module -h'") + } + } + + private CmdBase findCmd(String name) { + commands.find { it.name == name } + } + + /** + * Print the command usage help + */ + @Override + void usage() { + usage(args) + } + + /** + * Print the command usage help + * + * @param args The arguments as entered by the user + */ + @Override + void usage(List args) { + def result = [] + if( !args ) { + result << 'Usage: nextflow module [options]' + result << '' + result << 'Commands:' + commands.each { + def description = it.getClass().getAnnotation(Parameters)?.commandDescription() + result << " ${it.name.padRight(12)}${description}" + } + result << '' + println result.join('\n').toString() + } else { + final sub = findCmd(args[0]) + if( sub ) { + commander().usage(args[0]) + } else { + throw new AbortOperationException("Unknown module sub-command: ${args[0]}") + } + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy index 9a5cd860bc..a4f581e9d8 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy @@ -111,7 +111,8 @@ class Launcher { new CmdPlugin(), new CmdInspect(), new CmdLint(), - new CmdLineage() + new CmdLineage(), + new CmdModule() ] if(SecretsLoader.isEnabled()) @@ -129,6 +130,9 @@ class Launcher { jcommander.addCommand(cmd.name, cmd, aliases(cmd)) } jcommander.setProgramName( APP_NAME ) + + //Allow unknown options for module command + jcommander.getCommands().get(CmdModule.NAME)?.setAcceptUnknownOptions(true) } private static final String[] EMPTY = new String[0] @@ -154,6 +158,11 @@ class Launcher { jcommander.parse( normalizedArgs as String[] ) fullVersion = '-version' in normalizedArgs command = allCommands.find { it.name == jcommander.getParsedCommand() } + //Attach unknown options to command in case of needed + if (command) { + final unknownOptions = jcommander.commands.get(jcommander.getParsedCommand())?.getUnknownOptions() ?: [] + command.setUnknownOptions(unknownOptions) + } // whether is running a daemon daemonMode = command instanceof CmdNode // set the log file name diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy new file mode 100644 index 0000000000..26a438a682 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy @@ -0,0 +1,342 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.IParameterValidator +import com.beust.jcommander.Parameter +import com.beust.jcommander.ParameterException +import com.beust.jcommander.Parameters +import groovy.json.JsonOutput +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.npr.api.schema.v1.ModuleChannel +import io.seqera.npr.api.schema.v1.ModuleChannelItem +import io.seqera.npr.api.schema.v1.ModuleMetadata +import io.seqera.npr.api.schema.v1.ModuleRelease +import io.seqera.npr.api.schema.v1.ModuleTool +import nextflow.cli.CmdBase +import nextflow.config.ConfigBuilder +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.util.TestOnly + +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module info subcommand - displays module metadata and usage template + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "Show module information and usage template") +class CmdModuleInfo extends CmdBase { + + @Parameter(names = ["-version"], description = "Module version") + String version + + @Parameter( + names = ['-o', '-output'], + description = 'Output mode for reporting search results: text, json', + validateWith = OutputModeValidator + ) + String output = 'text' + + static class OutputModeValidator implements IParameterValidator { + + private static final List MODES = List.of('text', 'json') + + @Override + void validate(String name, String value) { + if( !MODES.contains(value) ) + throw new ParameterException("Output mode must be one of $MODES (found: $value)") + } + } + + @Parameter(description = "[scope/name]", required = true) + List args + + @TestOnly + protected Path root + + @TestOnly + protected ModuleRegistryClient client + + @Override + String getName() { + return 'info' + } + + @Override + void run() { + if( !args || args.size() != 1 ) { + throw new AbortOperationException("Incorrect number of arguments") + } + + def reference = ModuleReference.parse(args[0]) + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(baseDir) + .build() + final registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + + // Fetch full metadata from registry to get input/output parameters + def registryClient = this.client ?: new ModuleRegistryClient(registryConfig) + ModuleRelease release = null + + try { + if( version ) { + release = registryClient.fetchRelease(reference.fullName, version) + + } else { + release = registryClient.fetchModule(reference.fullName).latest + } + } catch( Exception e ) { + log.warn "Failed to fetch metadata from registry: ${e.message}" + } + if( !release ) { + throw new AbortOperationException("No release information available for ${reference}") + } + if( !release.metadata ) { + log.info("No metadata found for $reference ${release.version ? "($release.version)" : ''}") + } + def moduleUrl = buildModuleUrl(registryConfig.url, reference, release.version) + if( !output || output == 'text' ) { + printFormattedInfo(reference, release, moduleUrl) + } else if( output == 'json' ) { + printJsonInfo(reference, release, moduleUrl) + } else { + throw new AbortOperationException("Not implemented output mode $output)") + } + } + + private void printFormattedInfo(ModuleReference reference, ModuleRelease release, String moduleUrl) { + ModuleMetadata metadata = release.metadata + println "" + println "Module: ${reference}" + println "Version: ${release.version}" + println "URL: ${moduleUrl}" + println "Description: ${metadata.description ?: release.description ?: 'N/A'}" + + if( metadata.authors ) { + println "Authors: ${metadata.authors.join(', ')}" + } + + if( metadata.maintainers ) { + println "Maintainers: ${metadata.maintainers.join(', ')}" + } + + if( metadata.keywords ) { + println "Keywords: ${metadata.keywords.join(', ')}" + } + + printToolsInfo(metadata?.tools ?: []) + + printInputsInfo(metadata.input ?: []) + + printOutputsInfo(metadata.output ?: [:]) + + // Generate and display usage template + println "" + println "Usage Template:" + println "-" * 80 + println generateUsageTemplate(reference, metadata).join(" \\\n ") + println "" + } + + private void printOutputsInfo(Map outputs) { + if( outputs ) { + println "" + println "Output:" + outputs.each { name, output -> + println "- ${name} ${output.tuple ? '(tuple)' : ''}" + displayChannel("\t", output) + } + } + } + + private void printInputsInfo(List inputs) { + if( inputs ) { + println "" + println "Input:" + inputs.each { input -> + if( input.tuple ) { + println "- (tuple)" + displayChannel("\t", input) + } else + displayChannel("", input) + } + } + } + + private void printToolsInfo(List toolsList) { + if( toolsList ) { + println "" + println "Tools:" + toolsList.each { tool -> + println " - ${tool.name}${tool.version ? ' v' + tool.version : ''}" + if( tool.homepage ) { + println " Homepage: ${tool.homepage}" + } + } + } + } + + private void displayChannel(String prefix, ModuleChannel channel) { + channel.items.each { ModuleChannelItem item -> + println "${prefix}- ${item.name}${item.type ? ' (' + item.type + ')' : ''}" + if( item.description ) { + println "${prefix}\t${item.description.replaceAll(/\R/, ' ')}" + } + if( item.pattern ) { + println "${prefix}\tPattern: ${item.pattern}" + } + } + } + + private List generateUsageTemplate(ModuleReference reference, ModuleMetadata metadata) { + def template = new ArrayList() + template.add("nextflow module run ${reference}".toString()) + if( version ) + template.add(" -version $version".toString()) + + def inputs = metadata?.input ?: [] + inputs.each { input -> + input.items.each { ModuleChannelItem item -> + template.add(reference.scope == 'nf-core' + ? inferNfCoreParam(item.name, item.type) + : inferNormalParam(item.name, item.type)) + + } + } + if( reference.scope == 'nf-core' ) { + template.add('--outdir ') + } + return template + } + + private static String inferNfCoreParam(String paramName, String type) { + if( type?.equalsIgnoreCase("map") && paramName.equalsIgnoreCase("meta") ) { + return "--${paramName}.id " + } + if( type?.equalsIgnoreCase("file") || type?.equalsIgnoreCase("path") ) { + return "--${paramName} ${inferBioFilePlaceholder(paramName)}" + } + return inferNormalParam(paramName, type) + } + + private static String inferBioFilePlaceholder(String paramName) { + final String lower = paramName.toLowerCase() + if( lower.contains("fasta") ) return "" + if( lower.contains("bam") ) return "" + if( lower.contains("fastq") || lower.equals("reads") ) return "" + if( lower.contains("vcf") ) return "" + if( lower.contains("ref") ) return "" + if( lower.contains("bed") ) return "" + if( lower.contains("gff") || lower.contains("gtf") ) return "" + + return "<${paramName.toUpperCase().replaceAll(/[^A-Z0-9]/, '_')}_PATH>" + } + + private static String inferNormalParam(String paramName, String type) { + final paramPlaceholder = paramName.toUpperCase().replaceAll(/[^A-Z0-9]/, '_') + if( type?.equalsIgnoreCase("map") ) { + return "--${paramName}. <${paramPlaceholder}_KEY_VALUE>" + } + if( type?.equalsIgnoreCase("file") || type?.equalsIgnoreCase("path") ) { + return "--${paramName} <${paramPlaceholder}_PATH>" + } + return "--${paramName} <${paramPlaceholder}>" + } + + private static String buildModuleUrl(String registryUrl, ModuleReference reference, String version) { + // Strip /api suffix to get the base UI URL + def baseUrl = registryUrl.endsWith('/api') ? registryUrl[0..-5] : registryUrl + def encodedName = URLEncoder.encode(reference.name, 'UTF-8') + return "${baseUrl}/admin/modules/${reference.scope}/${encodedName}@${version}" + } + + private void printJsonInfo(ModuleReference reference, ModuleRelease release, String moduleUrl) { + def metadata = release?.metadata + def info = [ + name : reference.toString(), + fullName : reference.fullName, + version : release.version, + url : moduleUrl, + description: metadata.description ?: release.description, + authors : metadata.authors, + keywords : metadata.keywords, + ] + + def toolsList = metadata.tools ?: [] + if( toolsList ) { + info.tools = toolsList.collect { tool -> + return [ + name : tool.name, + version : tool.version, + homepage : tool.homepage, + documentation: tool.documentation + ] + } + } + + def inputs = metadata.input ?: [] + if( inputs ) { + info.input = inputs.collect { input -> + return [ + tuple: input.tuple, + items: input.items?.collect { item -> + [ + name : item.name, + type : item.type, + description: item.description, + pattern : item.pattern + ] + } + ] + } + } + + def outputs = metadata.output ?: [:] + if( outputs ) { + info.output = outputs.collectEntries { name, output -> + return [name, [ + tuple: output.tuple, + items: output.items?.collect { item -> + [ + name : item.name, + type : item.type, + description: item.description, + pattern : item.pattern + ] + } + ]] + } + } + + info.usageTemplate = generateUsageTemplate(reference, metadata).join(" ") + + println JsonOutput.prettyPrint(JsonOutput.toJson(info)) + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy new file mode 100644 index 0000000000..0a260c716f --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy @@ -0,0 +1,99 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.Parameter +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.cli.CmdBase +import nextflow.config.ConfigBuilder + +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleResolver + +import nextflow.util.TestOnly + +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module install subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@Parameters(commandDescription = "Install a module from the registry") +@CompileStatic +class CmdModuleInstall extends CmdBase { + + @Parameter(names = ["-version"], description = "Module version") + String version + + @Parameter(names = ["-force"], description = "Force reinstall even if already installed", arity = 0) + boolean force = false + + @Parameter(description = "[scope/name]", required = true) + List args + + @TestOnly + protected Path root + + @TestOnly + protected ModuleRegistryClient client + + @Override + String getName() { + return 'install' + } + + @Override + void run() { + if( !args || args.size() != 1 ) { + throw new AbortOperationException("Incorrect number of arguments") + } + + def reference = ModuleReference.parse(args[0]) + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(baseDir) + .build() + final registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + // Create resolver and install + def resolver = new ModuleResolver(baseDir, client ?: new ModuleRegistryClient(registryConfig)) + + try { + def installedMainFile = resolver.installModule(reference, version, force) + def installedVersion = version ?: resolver.resolveVersion(reference) + + println "Module ${reference}@${installedVersion} installed and configured successfully" + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Installation failed: ${e.message}", e) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy new file mode 100644 index 0000000000..a9ecf87aca --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy @@ -0,0 +1,148 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.IParameterValidator +import com.beust.jcommander.Parameter +import com.beust.jcommander.ParameterException +import com.beust.jcommander.Parameters +import groovy.json.JsonOutput +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.cli.CmdBase +import nextflow.exception.AbortOperationException +import nextflow.module.InstalledModule +import nextflow.module.ModuleIntegrity +import nextflow.module.ModuleStorage +import nextflow.util.TestOnly + +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module list subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "List all installed modules") +class CmdModuleList extends CmdBase { + + @Parameter( + names = ['-o', '-output'], + description = 'Output mode for reporting search results: table, json', + validateWith = OutputModeValidator + ) + String output = 'table' + + static class OutputModeValidator implements IParameterValidator { + + private static final List MODES = List.of('table', 'json') + + @Override + void validate(String name, String value) { + if( !MODES.contains(value) ) + throw new ParameterException("Output mode must be one of $MODES (found: $value)") + } + } + + @TestOnly + protected Path root + + @Override + String getName() { + return 'list' + } + + @Override + void run() { + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + + + // Create resolver and list modules + def storage = new ModuleStorage(baseDir) + + try { + def installed = storage.listInstalled() + + if( installed.isEmpty() ) { + println "No modules installed" + return + } + + if( !output || output == 'table' ) { + printFormattedList(installed) + } else if( output == 'json' ) { + printJsonList(installed) + } else { + throw new AbortOperationException("Not implemented output mode $output)") + } + + } + catch( Exception e ) { + log.error("Failed to list modules", e) + throw new AbortOperationException("List failed: ${e.message}", e) + } + } + + private void printFormattedList(List installed) { + println "" + println "Installed modules:" + println "" + println "Module".padRight(40) + "Version".padRight(15) + "Status" + println("-" * 70) + + installed.each { module -> + def status = getStatusString(module.integrity) + println "${module.reference.toString().padRight(40)}${(module.installedVersion ?: 'unknown').padRight(15)}${status}" + } + println "" + } + + private void printJsonList(List installed) { + def modules = installed.collect { module -> + [ + name : module.reference.toString(), + version : module.installedVersion ?: 'unknown', + integrity: module.integrity.toString(), + directory: module.directory.toString(), + registry : module.registryUrl ?: 'unknown' + ] + } + + // Simple JSON output (could use groovy.json.JsonOutput for better formatting) + println JsonOutput.prettyPrint(JsonOutput.toJson(modules: modules)) + } + + private String getStatusString(ModuleIntegrity integrity) { + switch( integrity ) { + case ModuleIntegrity.VALID: + return 'OK' + case ModuleIntegrity.MODIFIED: + return 'MODIFIED' + case ModuleIntegrity.NO_REMOTE_MODULE: + return 'LOCAL' + case ModuleIntegrity.CORRUPTED: + return 'CORRUPTED' + default: + return 'UNKNOWN' + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy new file mode 100644 index 0000000000..4c95bc9539 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy @@ -0,0 +1,268 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.Parameter +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.Const +import nextflow.cli.CmdBase +import nextflow.config.ConfigBuilder +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleChecksum +import nextflow.module.ModuleInfo +import nextflow.module.ModuleSpec +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleStorage +import nextflow.util.TestOnly + +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module publish subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "Publish a module to the registry") +class CmdModulePublish extends CmdBase { + + @Parameter(names = ["-dry-run"], description = "Validate without uploading", arity=0) + boolean dryRun = false + + @Parameter(names = ["-registry"], description = "Target registry URL.") + String registryUrl + + @Parameter(description = "Module directory path or scope/name") + List args + + @TestOnly + protected Path root + + @TestOnly + protected ModuleRegistryClient client + + //Flag if publish is invoked from a scope/name. In this case we should create/update the .module-info with the correct checksum + private boolean useModuleReference = false + + @Override + String getName() { + return 'publish' + } + + @Override + void run() { + if (!args || args.size() != 1) { + throw new AbortOperationException("Incorrect number of arguments") + } + + Path moduleDir = determineModuleDir(args[0]) + + log.info "Publishing module from: ${moduleDir}" + + // Step 1: Validate module structure + def validationErrors = validateModuleStructure(moduleDir) + if (!validationErrors.isEmpty()) { + throw new AbortOperationException( + "Module validation failed:\n" + validationErrors.collect { " - ${it}" }.join('\n') + ) + } + + // Step 2: Load and validate manifest + def manifestPath = moduleDir.resolve(ModuleStorage.MODULE_MANIFEST_FILE) + def manifest = ModuleSpec.load(manifestPath) + + def manifestErrors = manifest.validate() + if (!manifestErrors.isEmpty()) { + throw new AbortOperationException( + "Module manifest validation failed:\n" + manifestErrors.collect { " - ${it}" }.join('\n') + ) + } + + log.info "Module validated: ${manifest.name}@${manifest.version}" + + if (dryRun) { + printDryRunInfo(manifest) + return + } + + // Step 3: Get authentication token + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(moduleDir) + .build() + + def registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + publishModule(moduleDir, registryConfig, manifest) + + } + + private void publishModule(Path moduleDir, RegistryConfig registryConfig, ModuleSpec manifest){ + log.info "Creating module bundle..." + def tempBundleFile = Files.createTempFile("nf-module-publish-", ".tar.gz") + + try { + def checksum = ModuleStorage.createBundle(moduleDir, tempBundleFile) + log.info "Bundle checksum: ${checksum}" + + // Read bundle content as bytes + def bundleBytes = Files.readAllBytes(tempBundleFile) + + // Create publish request as a map (npr-api will serialize it) + def request = [ + version: manifest.version, + bundle: bundleBytes + ] + + // Publish to registry + final registry = registryUrl ?: registryConfig.url + log.info "Publishing module to registry: ${registryUrl ?: registryConfig.url}" + def registryClient = new ModuleRegistryClient(registryConfig) + def response = registryClient.publishModule(manifest.name, request, registry) + + if (useModuleReference) { + // If publish is performed using the module reference we should create/update the .module-info with the correct checksum + try { + ModuleInfo.save(moduleDir, [checksum: ModuleChecksum.compute(moduleDir), registryUrl: registry] ) + }catch (Exception e){ + log.warn("Unable to save the checksum - ${e.message}") + } + } + println "✓ Module published successfully!" + println "" + println "Module details:" + println " Name: ${manifest.name}" + println " Version: ${manifest.version}" + println " DownloadUrl: ${response.downloadUrl}" + + println "" + println "Others can now install this module using:" + println " nextflow module install ${manifest.name}" + + } finally { + // Clean up temporary bundle file + if (Files.exists(tempBundleFile)) { + try { + Files.delete(tempBundleFile) + } catch (Exception e) { + log.warn "Failed to clean up temporary bundle file: ${e.message}" + } + } + } + } + + private void printDryRunInfo(ModuleSpec manifest) { + println "✓ Module structure is valid" + println "" + println "Module details:" + println " Name: ${manifest.name}" + println " Version: ${manifest.version}" + println " Description: ${manifest.description}" + println " License: ${manifest.license}" + if( manifest.authors ) { + println " Authors: ${manifest.authors.join(', ')}" + } + if( manifest.keywords ) { + println " Keywords: ${manifest.keywords.join(', ')}" + } + if( manifest.requires ) { + println " Requires:" + manifest.requires.each { name, version -> + println " - ${name}: ${version}" + } + } + println "" + println "Dry run complete. Module is ready to publish." + println "Run without --dry-run to publish to the registry." + } + + /** + * Validate that the module directory has the required structure + * + * @param moduleDir The module directory path + * @return List of validation error messages (empty if valid) + */ + private List validateModuleStructure(Path moduleDir) { + List errors = [] + + if (!Files.exists(moduleDir) || !Files.isDirectory(moduleDir)) { + errors << "Module directory does not exist: ${moduleDir}".toString() + return errors + } + + // Check for required files + def mainNf = moduleDir.resolve(Const.DEFAULT_MAIN_FILE_NAME) + if (!Files.exists(mainNf)) { + errors << "Missing required file: $Const.DEFAULT_MAIN_FILE_NAME".toString() + } + + def metaYaml = moduleDir.resolve(ModuleStorage.MODULE_MANIFEST_FILE) + if (!Files.exists(metaYaml)) { + errors << "Missing required file: $ModuleStorage.MODULE_MANIFEST_FILE".toString() + } + + def readme = moduleDir.resolve(ModuleStorage.MODULE_README_FILE) + if (!Files.exists(readme)) { + errors << "Missing required file: $ModuleStorage.MODULE_README_FILE".toString() + } + + // Check bundle size (1MB uncompressed limit) + try (final sizeStream = Files.walk(moduleDir)){ + long totalSize = sizeStream + .filter { Files.isRegularFile(it) } + .mapToLong { Files.size(it) } + .sum() + + def maxSize = 1024 * 1024 // 1MB in bytes + if (totalSize > maxSize) { + def sizeMB = totalSize / (1024 * 1024) + errors << "Module size exceeds 1MB limit (current: ${String.format('%.2f', sizeMB)}MB)".toString() + } + } catch (Exception e) { + log.warn "Failed to check module size: ${e.message}" + } + + return errors + } + /** + * Determine if the specified module is a local path or a reference + * @param module + * @return + */ + private Path determineModuleDir(String module) { + //If local path exists return this path as module dir + if (Paths.get(module).exists()){ + return Paths.get(module).toAbsolutePath().normalize() + } + + final ref = ModuleReference.parse(module) + final localStorage = new ModuleStorage(root ?: Paths.get('.').toAbsolutePath().normalize()) + + if (!localStorage.isInstalled(ref)){ + throw new AbortOperationException("No module diretory found for $module") + } + useModuleReference = true + return localStorage.getModuleDir(ref) + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy new file mode 100644 index 0000000000..cfcea1b1f4 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy @@ -0,0 +1,106 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.Parameter +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.cli.CmdBase +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleInfo +import nextflow.module.ModuleReference +import nextflow.module.ModuleStorage + +import nextflow.util.TestOnly + +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module remove subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "Remove an installed module") +class CmdModuleRemove extends CmdBase { + + @Parameter(description = "", required = true) + List args + + @Parameter(names = ["-keep-files"], description = "Remove only .module-info keeping the local files", arity = 0) + boolean keepFiles = false + + @Parameter(names = ["-force"], description = "Force remove", arity = 0) + boolean force = false + + @TestOnly + protected Path root + + @Override + String getName() { + return 'remove' + } + + @Override + void run() { + if( !args || args.size() != 1 ) { + throw new AbortOperationException("Incorrect number of arguments") + } + if( keepFiles && force ) { + throw new AbortOperationException("Cannot use both -keep-files and -force options") + } + + def reference = ModuleReference.parse(args[0]) + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + + // Create resolver and spec file manager + def storage = new ModuleStorage(baseDir) + + try { + def filesRemoved = false + + // Remove local files unless -keep-files is set + if( !keepFiles ) { + filesRemoved = storage.removeModule(reference, force) + if( filesRemoved ) { + println "Module ${reference} files removed successfully" + } else { + println "Module ${reference} not found locally" + } + } else { + println "Keeping module files for ${reference} (-keep-files flag)" + final moduleInfo = storage.getModuleInfo(reference) + if( Files.exists(moduleInfo) ) { + Files.delete(moduleInfo) + } + } + + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to remove module $reference: ${e.message}", e) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy new file mode 100644 index 0000000000..c0f8fe8421 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy @@ -0,0 +1,89 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.Parameter +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import nextflow.cli.CmdRun +import nextflow.config.ConfigBuilder + +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleResolver + +import nextflow.util.TestOnly + +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module run subcommand + * + * @author Jorge Ejarque + */ +@CompileStatic +@Parameters(commandDescription = "Run a module directly from the registry") +class CmdModuleRun extends CmdRun { + @Parameter(names = ["-version"], description = "Module version") + String version + + @TestOnly + protected Path root + + @TestOnly + protected ModuleRegistryClient client + + @Override + String getName() { + return 'run' + } + + @Override + void run() { + if( !args ) { + throw new AbortOperationException("Arguments not provided") + } + + // Parse and validate module reference + ModuleReference reference + try { + reference = ModuleReference.parse(args[0]) + } catch( Exception e ) { + throw new AbortOperationException("Invalid module reference: ${args[0]}", e) + } + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(baseDir) + .build() + + def registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + def resolver = new ModuleResolver(baseDir, client ?: new ModuleRegistryClient(registryConfig)) + Path moduleFile = resolver.installModule(reference, version) + if( moduleFile ) { + println "Executing module..." + args[0] = moduleFile.toAbsolutePath().toString() + super.run() + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleSearch.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleSearch.groovy new file mode 100644 index 0000000000..2b55de24d8 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleSearch.groovy @@ -0,0 +1,157 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.IParameterValidator +import com.beust.jcommander.Parameter +import com.beust.jcommander.ParameterException +import com.beust.jcommander.Parameters +import groovy.json.JsonOutput +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.npr.api.schema.v1.ModuleSearchResult +import io.seqera.npr.api.schema.v1.SearchModulesResponse +import nextflow.cli.CmdBase +import nextflow.config.ConfigBuilder +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleRegistryClient +import nextflow.util.TestOnly + +import java.nio.file.Paths + +/** + * Module search subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "Search for modules in the registry") +class CmdModuleSearch extends CmdBase { + + @Parameter(names = ["-limit"], description = "Maximum number of results") + int limit = 20 + + @Parameter( + names = ['-o', '-output'], + description = 'Output mode for reporting search results: simple, json', + validateWith = OutputModeValidator + ) + String output = 'simple' + + static class OutputModeValidator implements IParameterValidator { + + private static final List MODES = List.of('simple', 'json') + + @Override + void validate(String name, String value) { + if( !MODES.contains(value) ) + throw new ParameterException("Output mode must be one of $MODES (found: $value)") + } + } + + @Parameter(description = "", required = true) + List args + + @TestOnly + protected ModuleRegistryClient client + + @Override + String getName() { + return 'search' + } + + @Override + void run() { + if( !args || args.size() != 1 ) { + throw new AbortOperationException("Unexpected number of parameters") + } + String query = args[0] + + // Get config + def baseDir = Paths.get('.').toAbsolutePath().normalize() + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(baseDir) + .build() + + final registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + // Create client to search + final client = this.client ?: new ModuleRegistryClient(registryConfig) + + try { + println "Searching for '${query}'..." + final results = client.search(query, limit) + + if( !results || results.totalResults == 0 || !results.results || results.results.isEmpty() ) { + println "No modules found" + return + } + + if( !output || output == 'simple' ) { + printFormattedResults(results) + } else if( output == 'json' ) { + printJsonResults(results) + } else { + throw new AbortOperationException("Not implemented output mode $output)") + } + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + log.error("Failed to search modules", e) + throw new AbortOperationException("Search failed: ${e.message}", e) + } + } + + private void printFormattedResults(SearchModulesResponse response) { + println "" + println "Found ${response.totalResults} module(s):" + println "" + + response.results.each { ModuleSearchResult result -> + println " ${result.name}" + if( result.description ) { + println " Description: ${result.description}" + } + println "" + } + } + + private void printJsonResults(SearchModulesResponse response) { + final modules = response.results.collect { ModuleSearchResult result -> + [ + name : result.name, + repositoryPath: result.repositoryPath, + description : result.description, + relevanceScore: result.relevanceScore, + keywords : result.keywords, + tools : result.tools, + revoked : result.revoked + ] + } + + println JsonOutput.prettyPrint(JsonOutput.toJson( + query: response.query, + totalResults: response.totalResults, + results: modules + )) + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy b/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy new file mode 100644 index 0000000000..ef3d0efb45 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy @@ -0,0 +1,79 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.Global +import nextflow.config.ConfigBuilder + +import nextflow.config.RegistryConfig +import nextflow.exception.IllegalModulePath +import nextflow.module.spi.RemoteModuleResolver + +import java.nio.file.Path + +/** + * Default implementation of RemoteModuleResolver using the Nextflow module registry. + * + *

This implementation: + *

    + *
  • Checks for locally installed modules in the project's modules directory
    • + *
  • Downloads modules from the configured registry if not present
    • + *
  • Reads version constraints from nextflow_spec.json
    • + *
  • Uses the Session's registry configuration
    • + *
+ * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class DefaultRemoteModuleResolver implements RemoteModuleResolver { + + @Override + Path resolve(String moduleName, Path baseDir) { + + final config = Global.config ?: new ConfigBuilder().setBaseDir(baseDir).build() + final registryConfig = config.navigate('registry') as RegistryConfig + + // Create module resolver + def resolver = new ModuleResolver(baseDir, registryConfig) + + try { + log.debug "Resolving remote module: ${moduleName}" + + // Parse module reference + def reference = ModuleReference.parse(moduleName) + + // Resolve module (will auto-install if missing or version mismatch) + def mainFile = resolver.resolve(reference, null, true) + + log.debug "Module ${reference} resolved to ${mainFile}" + return mainFile + } catch (Exception e) { + throw new IllegalModulePath( + "Failed to resolve remote module ${moduleName}: ${e.message}", + e + ) + } + } + + @Override + int getPriority() { + return 0 // Default implementation has lowest priority + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy b/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy new file mode 100644 index 0000000000..e951df7c59 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy @@ -0,0 +1,90 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.transform.ToString +import groovy.util.logging.Slf4j +import io.seqera.npr.api.schema.v1.ModuleMetadata +import org.yaml.snakeyaml.Yaml + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Represents a module installed in the local modules/ directory + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@ToString(includeNames = true) +class InstalledModule { + + ModuleReference reference + Path directory + Path mainFile + Path manifestFile + Path moduleInfoFile + String installedVersion + String expectedChecksum + String registryUrl + + /** + * Get the integrity status of this installed module + * + * @return ModuleIntegrity status + */ + ModuleIntegrity getIntegrity() { + // Check if main.nf exists + if( !Files.exists(mainFile) || !Files.exists(manifestFile) ) { + return ModuleIntegrity.CORRUPTED + } + + // Check if .module-info file exists + if( !Files.exists(moduleInfoFile) ) { + return ModuleIntegrity.NO_REMOTE_MODULE + } + + try { + // Compute actual checksum + def actualChecksum = ModuleChecksum.compute(directory) + + // Compare with expected + if( actualChecksum == expectedChecksum ) { + return ModuleIntegrity.VALID + } else { + log.debug("Actual: $actualChecksum, expected: $expectedChecksum") + return ModuleIntegrity.MODIFIED + } + } catch( Exception e ) { + log.warn "Failed to compute checksum for module ${reference}: ${e.message}" + return ModuleIntegrity.CORRUPTED + } + } +} + +/** + * Module integrity status + */ +@CompileStatic +enum ModuleIntegrity { + VALID, // Checksum matches + MODIFIED, // Checksum mismatch (local changes) + NO_REMOTE_MODULE, // No .module-info file (local-only module, no registry origin) + CORRUPTED // Missing required files +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy new file mode 100644 index 0000000000..0693b3155c --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy @@ -0,0 +1,164 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j + +import java.nio.file.Files +import java.nio.file.Path +import java.security.MessageDigest + +import static nextflow.module.ModuleInfo.MODULE_INFO_FILE + +/** + * Utility class for computing SHA-256 checksums of module directories + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleChecksum { + + public static final String CHECKSUM_ALGORITHM = "SHA-256" + + /** + * Compute the SHA-256 checksum of a module directory + * + * @param moduleDir The module directory path + * @return The hex-encoded SHA-256 checksum + */ + static String compute(Path moduleDir) { + if( !Files.exists(moduleDir) || !Files.isDirectory(moduleDir) ) { + throw new IllegalArgumentException("Module directory does not exist or is not a directory: ${moduleDir}") + } + + try { + def digest = MessageDigest.getInstance(CHECKSUM_ALGORITHM) + + // Collect all files in sorted order for consistent checksums + List files = [] + try( final walkStream = Files.walk(moduleDir) ) { + walkStream + .filter { Path path -> Files.isRegularFile(path) } + .filter { Path path -> !path.fileName.toString().equals(MODULE_INFO_FILE) } + .sorted() + .each { Path path -> files.add(path) } + } + + // Compute checksum over all file contents + byte[] buf = new byte[8192] + for( Path file : files ) { + // Include relative path in checksum for directory structure integrity + def relativePath = moduleDir.relativize(file).toString() + digest.update(relativePath.bytes) + + // Include file contents via streaming to avoid loading large files into memory + Files.newInputStream(file).withCloseable { is -> + int n + while( (n = is.read(buf)) != -1 ) { + digest.update(buf, 0, n) + } + } + } + + def hashBytes = digest.digest() + return bytesToHex(hashBytes) + } + catch( Exception e ) { + log.error("Failed to compute checksum for module directory: ${moduleDir}", e) + throw new RuntimeException("Failed to compute module checksum", e) + } + } + + /** + * Save a checksum to the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @param checksum The checksum to save + */ + static void save(Path moduleDir, String checksum) { + ModuleInfo.save(moduleDir,'checksum', checksum) + } + + /** + * Load a checksum from the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @return The checksum, or null if file doesn't exist + */ + static String load(Path moduleDir) { + ModuleInfo.load(moduleDir, 'checksum') + } + + /** + * Verify that a module directory matches the expected checksum + * + * @param moduleDir The module directory path + * @param expectedChecksum The expected checksum + * @return true if checksums match, false otherwise + */ + static boolean verify(Path moduleDir, String expectedChecksum) { + def actualChecksum = compute(moduleDir) + return actualChecksum == expectedChecksum + } + + /** + * Compute the checksum of a single file + * + * @param file The file path + * @param type checksum algorithm (sha-256 if not provided) + * @return The hex-encoded checksum + */ + static String computeFile(Path file, String type = CHECKSUM_ALGORITHM) { + if( !Files.exists(file) || !Files.isRegularFile(file) ) { + throw new IllegalArgumentException("File does not exist or is not a regular file: ${file}") + } + + try { + final digest = MessageDigest.getInstance(type) + final byte[] buf = new byte[8192] + Files.newInputStream(file).withCloseable { is -> + int n + while( (n = is.read(buf)) != -1 ) { + digest.update(buf, 0, n) + } + } + return bytesToHex(digest.digest()) + } + catch( Exception e ) { + log.error("Failed to compute checksum for file: ${file}", e) + throw new RuntimeException("Failed to compute file checksum", e) + } + } + + /** + * Convert byte array to hex string + * + * @param bytes The byte array + * @return Hex-encoded string + */ + private static String bytesToHex(byte[] bytes) { + def hexChars = new char[bytes.length * 2] + for( int i = 0; i < bytes.length; i++ ) { + int v = bytes[i] & 0xFF + hexChars[i * 2] = Character.forDigit(v >>> 4, 16) + hexChars[i * 2 + 1] = Character.forDigit(v & 0x0F, 16) + } + return new String(hexChars) + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleInfo.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleInfo.groovy new file mode 100644 index 0000000000..a0fd26511f --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleInfo.groovy @@ -0,0 +1,109 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Utility class for managing .module-info + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleInfo { + + public static final String MODULE_INFO_FILE = ".module-info" + + + /** + * Save a property to the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @param property The property to save + * @param value The property value to save + */ + static void save(Path moduleDir, String property, String value) { + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + def props = new Properties() + // If file exists loads to update current just checksum property + if( Files.exists( moduleInfoFile)) + moduleInfoFile.withInputStream { is -> props.load(is) } + props.setProperty(property, value) + moduleInfoFile.withOutputStream { os -> props.store(os, null) } + } + + /** + * Save a property to the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @param properties Map with properties to save + */ + static void save(Path moduleDir, Map properties) { + if( properties ) { + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + def props = new Properties() + // If file exists loads to update current just checksum property + if( Files.exists( moduleInfoFile ) ) + moduleInfoFile.withInputStream { is -> props.load(is) } + + for( final property : properties.entrySet() ) { + props.setProperty(property.key, property.value) + } + moduleInfoFile.withOutputStream { os -> props.store(os, null) } + } + } + + /** + * Return the value of property from the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @param moduleDir The module directory path + * @return The checksum, or null if file doesn't exist + */ + static String load(Path moduleDir, String property) { + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + if( !Files.exists(moduleInfoFile) ) { + log.debug("Module file $moduleInfoFile not found") + return null + } + def props = new Properties() + moduleInfoFile.withInputStream { is -> props.load(is) } + return props.getProperty(property) + } + + /** + * Load all properties from the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @return The checksum, or null if file doesn't exist + */ + static Map load(Path moduleDir) { + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + if( !Files.exists(moduleInfoFile) ) { + log.debug("Module file $moduleInfoFile not found") + return [:] + } + def props = new Properties() + moduleInfoFile.withInputStream { is -> props.load(is) } + return props as Map + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy new file mode 100644 index 0000000000..23cbf7ba3d --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy @@ -0,0 +1,80 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.transform.EqualsAndHashCode +import nextflow.exception.AbortOperationException + +import java.util.regex.Pattern + +/** + * Represents a reference to a module in DSL include statements + * + * @author Jorge Ejarqe + */ +@CompileStatic +@EqualsAndHashCode +class ModuleReference { + + // Pattern allows: optional @, scope with letters/digits/hyphens/dots/underscores, name segments separated by slashes (no trailing slash) + // Scope: starts with letter/digit, followed by letters/digits/dots/underscores/hyphens + // Name: one or more segments (each starting with letter, followed by letters/digits/underscores/hyphens), separated by slashes + private static final Pattern MODULE_NAME_PATTERN = ~/^([a-z0-9][a-z0-9._\-]*)\/([a-z][a-z0-9._\-]*(?:\/[a-z][a-z0-9._\-]*)*)$/ + + final String scope + final String name + final String fullName + + ModuleReference(String scope, String name) { + this.scope = scope + this.name = name + this.fullName = "${scope}/${name}" + } + + /** + * Parse a module reference from a string in "@scope/name" or "scope/name" format + * + * @param source The module reference string + * @return A ModuleReference object + * @throws AbortOperationException if the format is invalid + */ + static ModuleReference parse(String source) { + if( !source ) { + throw new AbortOperationException("Module reference cannot be empty") + } + + // Trim whitespace + source = source.trim() + + def matcher = MODULE_NAME_PATTERN.matcher(source) + if( !matcher.matches() ) { + throw new AbortOperationException( + "Invalid module reference: '${source}'. " + + "Expected format: scope/name where scope is lowercase alphanumeric with dots/underscores/hyphens " + + "and name is lowercase alphanumeric with underscores/hyphens, optionally with slash-separated segments" + ) + } + + return new ModuleReference(matcher.group(1), matcher.group(2)) + } + + @Override + String toString() { + return fullName + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy new file mode 100644 index 0000000000..0528b17609 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy @@ -0,0 +1,494 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import com.google.gson.Gson +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.http.HxClient +import io.seqera.npr.api.schema.v1.Module +import io.seqera.npr.api.schema.v1.ModuleRelease +import io.seqera.npr.api.schema.v1.PublishModuleResponse +import io.seqera.npr.api.schema.v1.SearchModulesResponse +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.serde.gson.GsonEncoder +import nextflow.util.RetryConfig + +import java.net.http.HttpClient +import java.net.http.HttpRequest +import java.net.http.HttpResponse +import java.nio.file.Files +import java.nio.file.Path + +/** + * REST API client for Nextflow module registry using npr-api models + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleRegistryClient { + + private final RegistryConfig config + private final HxClient httpClient + + ModuleRegistryClient(RegistryConfig config) { + this.config = config ?: new RegistryConfig() + this.httpClient = HxClient.newBuilder() + .retryConfig(RetryConfig.config()) + .followRedirects(HttpClient.Redirect.NORMAL) + .build() + } + + private String encodeName(String name) { + return URLEncoder.encode(name, 'UTF-8') + } + + /** + * Fetch module metadata from the registry + * + * @param name The module name (e.g., "nf-core/fastqc") + * @return Module object with metadata + */ + Module fetchModule(String name) { + def registryUrls = config.allUrls + + Exception lastError = null + for( String registryUrl : registryUrls ) { + log.debug "Trying to fetch from $registryUrl" + try { + return fetchModuleFromRegistry(registryUrl, name) + } catch( Exception e ) { + log.debug "Failed to fetch module from ${registryUrl}: ${e.message}" + lastError = e + } + } + + throw new AbortOperationException( + "Unable to fetch module ${name} from any configured registry", + lastError + ) + } + + /** + * Fetch module from a specific registry URL + */ + private Module fetchModuleFromRegistry(String registryUrl, String name) { + def endpoint = "${registryUrl}/v1/modules/${encodeName(name)}" + def uri = URI.create(endpoint) + + def requestBuilder = HttpRequest.newBuilder() + .uri(uri) + .GET() + + // Add authentication if available + log.debug "Getting auth from: ${registryUrl}" + def token = config.getApiKey() + if( token ) { + requestBuilder.header("Authorization", "Bearer ${token}") + } + log.debug "Building request: ${registryUrl}" + def request = requestBuilder.build() + + try { + log.debug "Fetching module from: ${uri}" + def response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()) + def body = response.body() + + log.debug "Registry request: ${response.uri()}\n- code: ${response.statusCode()}\n- body: ${body}" + + if( response.statusCode() == 404 ) { + throw new AbortOperationException("Module not found: ${name}") + } + + if( response.statusCode() != 200 ) { + throw new AbortOperationException( + "Invalid response from registry: ${uri}\n" + + "- http status: ${response.statusCode()}\n" + + "- response: ${body}" + ) + } + + // Parse response using npr-api Module model + def encoder = new GsonEncoder() {} + return encoder.decode(body) + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to fetch module from: ${uri}", e) + } + } + + /** + * Fetch specific module version/release + * + * @param name The module name + * @param version The version string + * @return ModuleRelease object from npr-api + */ + ModuleRelease fetchRelease(String name, String version) { + def registryUrls = config.allUrls + + Exception lastError = null + for( String registryUrl : registryUrls ) { + try { + return fetchReleaseFromRegistry(registryUrl, name, version) + } catch( Exception e ) { + log.debug "Failed to fetch release from ${registryUrl}: ${e.message}" + lastError = e + } + } + + throw new AbortOperationException( + "Unable to fetch module ${name}@${version} from any configured registry", + lastError + ) + } + + /** + * Fetch release from a specific registry URL + */ + private ModuleRelease fetchReleaseFromRegistry(String registryUrl, String name, String version) { + def endpoint = "${registryUrl}/v1/modules/${encodeName(name)}/${version}" + def uri = URI.create(endpoint) + + def requestBuilder = HttpRequest.newBuilder() + .uri(uri) + .GET() + + def token = config.getApiKey() + if( token ) { + requestBuilder.header("Authorization", "Bearer ${token}") + } + + def request = requestBuilder.build() + + try { + log.debug "Fetching module release from: ${uri}" + def response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()) + def body = response.body() + + if( response.statusCode() == 404 ) { + throw new AbortOperationException("Module version not found: ${name}@${version}") + } + + if( response.statusCode() != 200 ) { + throw new AbortOperationException( + "Invalid response from registry: ${uri}\n" + + "- http status: ${response.statusCode()}\n" + + "- response: ${body}" + ) + } + + // Parse response using npr-api ModuleRelease model + return new GsonEncoder() {}.decode(body) + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to fetch module release from: ${uri}", e) + } + } + + /** + * Download a module bundle from the registry + * + * @param name The module name + * @param version The module version + * @param targetPath The target path to download to + * @return URL of the repository used to downloaded file path + */ + String downloadModule(String name, String version, Path targetPath) { + def registryUrls = config.allUrls + if( targetPath.exists() ) { + targetPath.delete() + } + Exception lastError = null + for( String registryUrl : registryUrls ) { + try { + return downloadModuleFromRegistry(registryUrl, name, version, targetPath) + } catch( Exception e ) { + log.debug "Failed to download from ${registryUrl}: ${e.message}" + lastError = e + } + } + + throw new AbortOperationException( + "Unable to download module ${name}@${version} from any configured registry", + lastError + ) + } + + /** + * Download module from a specific registry URL + */ + private String downloadModuleFromRegistry(String registryUrl, String name, String version, Path targetPath) { + def endpoint = "${registryUrl}/v1/modules/${encodeName(name)}/${version}/download" + def uri = URI.create(endpoint) + + def requestBuilder = HttpRequest.newBuilder() + .uri(uri) + .GET() + + def token = config.getApiKey() + if( token ) { + requestBuilder.header("Authorization", "Bearer ${token}") + } + + def request = requestBuilder.build() + + try { + log.debug "Downloading module from: ${uri}" + def response = httpClient.send(request, HttpResponse.BodyHandlers.ofInputStream()) + + if( response.statusCode() == 404 ) { + throw new AbortOperationException("Module bundle not found: ${name}@${version}") + } + + if( response.statusCode() != 200 ) { + throw new AbortOperationException( + "Invalid response from registry: ${uri}\n" + + "- http status: ${response.statusCode()}" + ) + } + + // Create parent directories if needed + if( targetPath.parent ) { + Files.createDirectories(targetPath.parent) + } + + // Write response body to file + Files.copy(response.body(), targetPath) + log.debug "Downloaded module to: ${targetPath}" + + validateDownloadIntegrity(response, uri, targetPath, name, version) + + return registryUrl + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to download module from: ${uri}", e) + } + } + + private void validateDownloadIntegrity(HttpResponse response, uri, Path targetPath, String name, String version) { + def checksumType = ModuleChecksum.CHECKSUM_ALGORITHM + def checksum = getChecksumFromHeaders(response) + + if( !checksum ) { + log.warn "No X-Checksum or Docker-Content-Digest header found in response from ${uri}" + return + } + + // Check if checksum has a digest format including algorithm: "sha256:abc123..." + def parts = checksum.split(':', 2) + if( parts.length == 2 ) { + checksumType = parts[0].toLowerCase() + checksum = parts[1] + } + log.debug "Using checksum: ${checksumType}:${checksum}" + + def actualChecksum = ModuleChecksum.computeFile(targetPath, checksumType) + if( actualChecksum != checksum ) { + // Clean up downloaded file + Files.delete(targetPath) + throw new AbortOperationException( + "Downloaded module checksum mismatch for ${name}@${version}:\n" + + "- expected (${checksumType}): ${checksum}\n" + + "- actual: ${actualChecksum}\n" + + "The download may be corrupted or tampered with." + ) + } + log.debug "Checksum validated successfully: ${checksumType}:${checksum}" + } + + private String getChecksumFromHeaders(HttpResponse response) { + // Get X-Checksum from response headers + def checksum = getChecksumFromHeader(response) + if( checksum ) { + return checksum + } + // If not look if it is a previous redirected response header + Optional> prev = response.previousResponse() + while( prev.isPresent() ) { + HttpResponse r = prev.get(); + checksum = getChecksumFromHeader(r) + if( checksum ) { + return checksum + } + prev = r.previousResponse(); + } + return null + } + + private String getChecksumFromHeader(HttpResponse response) { + def checksum = response.headers().firstValue("X-NF-Module-Checksum").orElse(null) + if( !checksum ) { + checksum = response.headers().firstValue("Docker-Content-Digest").orElse(null) + } + return checksum + } + + + /** + * Search for modules in the registry + * + * @param query The search query + * @param limit Maximum number of results (default: 20) + * @return SearchModulesResponse with results + */ + SearchModulesResponse search(String query, int limit = 20) { + def registryUrls = config.allUrls + + Exception lastError = null + for( String registryUrl : registryUrls ) { + try { + return searchInRegistry(registryUrl, query, limit) + } catch( Exception e ) { + log.debug "Failed to search in ${registryUrl}: ${e.message}" + lastError = e + } + } + + throw new AbortOperationException( + "Unable to search modules in any configured registry", + lastError + ) + } + + /** + * Search in a specific registry + */ + private SearchModulesResponse searchInRegistry(String registryUrl, String query, int limit) { + def endpoint = "${registryUrl}/v1/modules?query=${URLEncoder.encode(query, 'UTF-8')}&limit=${limit}" + def uri = URI.create(endpoint) + + def requestBuilder = HttpRequest.newBuilder() + .uri(uri) + .GET() + + def token = config.getApiKey() + if( token ) { + requestBuilder.header("Authorization", "Bearer ${token}") + } + + def request = requestBuilder.build() + + try { + log.debug "Searching modules: ${uri}" + def response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()) + def body = response.body() + + if( response.statusCode() != 200 ) { + throw new AbortOperationException( + "Invalid response from registry: ${uri}\n" + + "- http status: ${response.statusCode()}\n" + + "- response: ${body}" + ) + } + + // Parse response using npr-api SearchModulesResponse model + def encoder = new GsonEncoder() {} + return encoder.decode(body) + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to search modules in: ${uri}", e) + } + } + + /** + * Publish a module to the registry (authenticated) + * + * @param name The module name + * @param request The publish request from npr-api + * @param authToken The authentication token + * @return PublishModuleResponse from npr-api + */ + PublishModuleResponse publishModule(String name, def request, String registry = null) { + final registryUrl = registry ?: config.url + final authToken = config.apiKey + + if( !authToken ) { + throw new AbortOperationException( + "Authentication required to publish modules.\n" + + "Please set 'NXF_REGISTRY_TOKEN' environment variable or configure 'registry.apiKey' in nextflow.config:\n\n" + + " registry {\n" + + " apiKey = 'YOUR_REGISTRY_TOKEN'\n" + + " }\n" + ) + } + return publishModuleToRegistry(registryUrl, name, request, authToken) + } + + /** + * Publish module to a specific registry + */ + private PublishModuleResponse publishModuleToRegistry( + String registryUrl, + String name, + def request, + String authToken) { + + String endpoint = "${registryUrl}/v1/modules/${encodeName(name)}".toString() + URI uri = URI.create(endpoint) + + // Serialize request to JSON + def gson = new Gson() + String requestBody = gson.toJson(request) + + HttpRequest httpRequest = HttpRequest.newBuilder() + .uri(uri) + .header("Content-Type", "application/json") + .header("Authorization", "Bearer ${authToken}".toString()) + .POST(HttpRequest.BodyPublishers.ofString(requestBody)) + .build() + + try { + log.debug "Publishing module to: ${uri}" + log.trace "Request: \n\t${httpRequest}\n\theaders: ${httpRequest.headers()}\n\tbody: ${requestBody}" + + HttpResponse response = httpClient.send(httpRequest, HttpResponse.BodyHandlers.ofString()) + String body = response.body() + + if( response.statusCode() != 201 ) { + throw new AbortOperationException( + "Failed to publish module: ${uri}\n" + + "- http status: ${response.statusCode()}\n" + + "- response: ${body}" + ) + } + + // Parse response using npr-api PublishModuleResponse model + return new GsonEncoder() {}.decode(body) + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to publish module to: ${uri}", e) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy new file mode 100644 index 0000000000..8e8626a0b9 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy @@ -0,0 +1,174 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j + +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Core module resolution logic that coordinates registry, storage, and version management + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleResolver { + + private final ModuleRegistryClient registryClient + private final ModuleStorage storage + + ModuleResolver(Path baseDir, ModuleRegistryClient registryClient) { + this.registryClient = registryClient + this.storage = new ModuleStorage(baseDir) + } + + ModuleResolver(Path baseDir, RegistryConfig registryConfig = null) { + this(baseDir, new ModuleRegistryClient(registryConfig ?: new RegistryConfig())) + + } + + /** + * Resolve a module reference to an installed module path + * + * @param reference The module reference + * @param version Optional specific version (null = use config or latest) + * @param autoInstall Whether to auto-install if not present (default: false) + * @return Path to the module's main.nf file + */ + Path resolve(ModuleReference reference, String version = null, boolean autoInstall = false) { + + // Check if module is already installed + def installed = storage.getInstalledModule(reference) + + if( installed ) { + // Check integrity + def integrity = installed.integrity + if( integrity == ModuleIntegrity.CORRUPTED ) { + throw new AbortOperationException( + "Module ${reference} is corrupted (missing required files). " + + "Please remove and reinstall." + ) + } + + if( integrity == ModuleIntegrity.MODIFIED ) { + log.warn1 "Module ${reference} has local modifications (checksum mismatch)" + } else if( integrity == ModuleIntegrity.NO_REMOTE_MODULE ) { + log.warn1 "Module ${reference} has no registry origin (.module-info missing)" + } + + // Check if version matches + if( version && installed.installedVersion != version ) { + if( autoInstall ) { + log.info "Upgrading module ${reference} from ${installed.installedVersion} to ${version}" + return installModule(reference, version) + } else { + throw new AbortOperationException( + "Module ${reference} version mismatch: " + + "installed=${installed.installedVersion}, required=${version}. " + + "Run 'nextflow module install ${reference}@${version}' to update." + ) + } + } + + // Module is installed and version matches + return installed.mainFile + } + + // Module not installed + if( autoInstall ) { + return installModule(reference, version) + } else { + throw new AbortOperationException( + "Module ${reference} is not installed. " + + "Run 'nextflow module install ${reference}' to install." + ) + } + } + + String resolveVersion(ModuleReference reference) { + final version = registryClient.fetchModule(reference.fullName)?.latest?.version + if( !version ) { + throw new AbortOperationException("Module ${reference} has no published versions") + } + return version + } + + /** + * Install or update a module + * + * @param reference The module reference + * @param version Optional specific version (null = latest) + * @param force Force reinstall even if already installed + * @return Path to the installed module's main.nf file + */ + Path installModule(ModuleReference reference, String version = null, boolean force = false) { + if( !version ) + version = resolveVersion(reference) + // Check if already installed + if( storage.isInstalled(reference) ) { + def installed = storage.getInstalledModule(reference) + if( installed.installedVersion == version ) { + log.info "Module ${reference}@${installed.installedVersion} is already installed (version $version)" + return installed.mainFile + } + + // No desired version, check for local modifications + def integrity = installed.integrity + if( integrity == ModuleIntegrity.MODIFIED && !force ) { + throw new AbortOperationException( + "Module ${reference} has local modifications. " + + "Use '-force' to override, or save your changes first." + ) + } + if( integrity == ModuleIntegrity.NO_REMOTE_MODULE && !force ) { + throw new AbortOperationException( + "Folder 'modules/${reference}' already exists and is not a valid remote module. " + + "Use '-force' to override, or save your changes first." + ) + } + } + + + log.info "Installing module ${reference}@${version}..." + + // Download module package to temporary location + Path tempFile = Files.createTempFile("nf-module-", ".tgz") + try { + // Download and validate integrity using server checksum + def downloadUrl = registryClient.downloadModule(reference.fullName, version, tempFile) + + // Install to modules directory (will compute directory checksum for future integrity checks) + InstalledModule installed = storage.installModule(reference, version, tempFile, downloadUrl) + + log.info "Module ${reference}@${version} installed successfully at ${installed.mainFile.parent}" + return installed.mainFile + } + finally { + // Clean up temporary file + if( Files.exists(tempFile) ) { + Files.delete(tempFile) + } + } + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy new file mode 100644 index 0000000000..7b27b44abf --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy @@ -0,0 +1,117 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.exception.AbortOperationException +import org.yaml.snakeyaml.Yaml + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Represents a module manifest (meta.yaml) with validation + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleSpec { + + String name + String version + String description + List authors + String license + List keywords + Map requires + + /** + * Load a module manifest from a meta.yaml file + * + * @param metaYamlPath Path to meta.yaml + * @return ModuleSpec instance + */ + static ModuleSpec load(Path metaYamlPath) { + if( !Files.exists(metaYamlPath) ) { + throw new AbortOperationException("Module manifest not found: ${metaYamlPath}") + } + + try { + def yaml = new Yaml() + def data = yaml.load(Files.newInputStream(metaYamlPath)) as Map + + def manifest = new ModuleSpec() + manifest.name = data.name as String + manifest.version = data.version as String + manifest.description = data.description as String + manifest.authors = data.authors as List ?: [] + manifest.license = data.license as String + manifest.keywords = data.keywords as List ?: [] + manifest.requires = data.requires as Map ?: [:] + + return manifest + } + catch( Exception e ) { + throw new AbortOperationException("Failed to parse module manifest: ${metaYamlPath}", e) + } + } + + /** + * Validate the module manifest for required fields + * + * @return List of validation errors (empty if valid) + */ + List validate() { + List errors = [] + + if( !name ) { + errors << "Missing required field: name" + } + if( !version ) { + errors << "Missing required field: version" + } + if( !description ) { + errors << "Missing required field: description" + } + if( !license ) { + errors << "Missing required field: license" + } + + // Validate version format (semantic versioning) + if( version && !version.matches(/^\d+\.\d+\.\d+(-[\w.-]+)?$/) ) { + errors << "Invalid version format: ${version} (expected semantic versioning, e.g., 1.0.0)".toString() + } + + // Validate name format (scope/name or scope/path/to/name for nested modules) + if( name && !name.matches(/^[a-zA-Z0-9._-]+\/[a-zA-Z0-9_-]+(?:\/[a-zA-Z0-9_-]+)*$/) ) { + errors << "Invalid module name format: ${name} (expected scope/name or scope/path/to/name, e.g., nf-core/fastqc or nf-core/gfatools/gfa2fa)".toString() + } + + return errors + } + + /** + * Check if the manifest is valid + * + * @return true if valid, false otherwise + */ + boolean isValid() { + return validate().isEmpty() + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy new file mode 100644 index 0000000000..6c9a90ca87 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy @@ -0,0 +1,452 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.Const +import nextflow.exception.AbortOperationException +import nextflow.file.FileHelper +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveInputStream +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream +import org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream +import org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream + +import java.nio.file.Files +import java.nio.file.Path +import java.util.stream.Stream +import java.util.zip.ZipEntry +import java.util.zip.ZipInputStream + +import static nextflow.module.ModuleInfo.MODULE_INFO_FILE + +/** + * Manages local filesystem storage for modules + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleStorage { + public static final String MODULE_MANIFEST_FILE = "meta.yml" + public static final String MODULE_README_FILE = "README.md" + private final Path modulesDir + + /** + * Create a ModuleStorage instance + * + * @param baseDir The base directory (usually project root) + */ + ModuleStorage(Path baseDir) { + this.modulesDir = baseDir.resolve('modules') + } + + /** + * Get the modules directory path + * + * @return The modules directory + */ + Path getModulesDir() { + return modulesDir + } + + /** + * Get the directory path for a specific module + * + * @param reference The module reference + * @return The module directory path + */ + Path getModuleDir(ModuleReference reference) { + return modulesDir.resolve(reference.scope).resolve(reference.name) + } + + /** + * Get the module info path for a specific module + * + * @param reference The module reference + * @return The module info path + */ + Path getModuleInfo(ModuleReference reference) { + return modulesDir.resolve(reference.scope).resolve(reference.name).resolve(MODULE_INFO_FILE) + } + + /** + * Check if a module is installed locally + * + * @param reference The module reference + * @return true if the module directory exists + */ + boolean isInstalled(ModuleReference reference) { + def moduleDir = getModuleDir(reference) + return Files.exists(moduleDir) && Files.isDirectory(moduleDir) + } + + /** + * Get an installed module + * + * @param reference The module reference + * @return InstalledModule object, or null if not installed + */ + InstalledModule getInstalledModule(ModuleReference reference) { + def moduleDir = getModuleDir(reference) + if (!Files.exists(moduleDir) || !Files.isDirectory(moduleDir)) { + return null + } + + def installed = new InstalledModule( + reference: reference, + directory: moduleDir, + mainFile: moduleDir.resolve(Const.DEFAULT_MAIN_FILE_NAME), + manifestFile: moduleDir.resolve(MODULE_MANIFEST_FILE), + moduleInfoFile: moduleDir.resolve(MODULE_INFO_FILE), + ) + + // Load checksum if available + Map infoProps = ModuleInfo.load(moduleDir) + installed.expectedChecksum = infoProps?.checksum + installed.registryUrl = infoProps?.registryUrl + installed.installedVersion = ModuleSpec.load(installed.manifestFile).version + return installed + } + + /** + * List all installed modules by scanning for directories containing a .module-info marker file. + * + * @return List of InstalledModule objects + */ + List listInstalled() { + if (!Files.exists(modulesDir) || !Files.isDirectory(modulesDir)) { + return [] + } + + List modules = [] + + try( final walkStream = Files.walk(modulesDir) ) { + walkStream + .filter { Path path -> Files.isDirectory(path) } + .filter { Path path -> Files.exists(path.resolve(MODULE_INFO_FILE)) } + .each { Path moduleDir -> + try { + def rel = modulesDir.relativize(moduleDir) + if( rel.nameCount < 2 ) return // Need at least scope/name + def reference = ModuleReference.parse(rel.toString()) + def installed = getInstalledModule(reference) + if( installed ) modules.add(installed) + } catch(Exception e){ + // Catching exception to go on inspecting other valid folders + log.debug("Not a valid module reference - $e.message") + } + } + } catch (IOException e) { + log.warn "Failed to scan modules directory ${modulesDir}: ${e.message}" + } + + return modules + } + + /** + * Install a module from a downloaded package file + * + * @param reference The module reference + * @param version The module version + * @param packageFile The downloaded package file (zip or tar.gz) + * @return The InstalledModule object + */ + InstalledModule installModule(ModuleReference reference, String version, Path packageFile, String downloadUrl) { + def moduleDir = getModuleDir(reference) + + try { + // Remove existing installation if present + if (Files.exists(moduleDir)) { + log.debug "Removing existing module installation: ${moduleDir}" + FileHelper.deletePath(moduleDir) + } + + // Create module directory + Files.createDirectories(moduleDir) + + // Extract package - detect format by file extension + if (packageFile.toString().endsWith('.tgz') || packageFile.toString().endsWith('.tar.gz')) { + extractTarGz(packageFile, moduleDir) + } else { + extractZip(packageFile, moduleDir) + } + + // Compute and save checksum of extracted directory contents + // This checksum is used to detect local modifications + def checksum = ModuleChecksum.compute(moduleDir) + ModuleInfo.save(moduleDir, [checksum: checksum, registryUrl: downloadUrl]) + + log.debug "Installed module ${reference}@${version} to ${moduleDir}" + + return getInstalledModule(reference) + } + catch (Exception e) { + // Clean up on failure + if (Files.exists(moduleDir)) { + try { + FileHelper.deletePath(moduleDir) + } catch (Exception cleanupError) { + log.warn "Failed to clean up after installation failure: ${cleanupError.message}" + } + } + throw new AbortOperationException("Failed to install module ${reference}@${version}", e) + } + } + + /** + * Remove an installed module + * + * @param reference The module reference + * @param force Force local module folder + * @return true if module was removed, false if not installed + */ + boolean removeModule(ModuleReference reference, boolean force) { + final installed = getInstalledModule(reference) + if( !installed ) + return + final integrity = installed.integrity + if( integrity == ModuleIntegrity.NO_REMOTE_MODULE && !force ) { + throw new AbortOperationException( + "Folder 'modules/${reference}' already exists and is not a valid remote module ($MODULE_INFO_FILE missing). " + + "Use '-force' to remove, or save your changes first.") + } + + if( integrity == ModuleIntegrity.MODIFIED && !force ) { + throw new AbortOperationException( + "Module ${reference} has local modifications. " + + "Use '-force' to remove, or save your changes first." + ) + } + + try { + FileHelper.deletePath(installed.directory) + log.debug "Removed module: ${reference}" + + // Clean up empty scope directory + def scopeDir = installed.directory.parent + if (Files.exists(scopeDir) && isEmpty(scopeDir)) { + Files.delete(scopeDir) + } + + return true + } + catch (Exception e) { + throw new AbortOperationException("Failed to remove module ${reference}", e) + } + } + + /** + * Extract a zip file to a target directory + * + * @param zipFile The zip file path + * @param targetDir The target directory + */ + private void extractZip(Path zipFile, Path targetDir) { + Files.newInputStream(zipFile).withCloseable { fis -> + new ZipInputStream(fis).withCloseable { zis -> + ZipEntry entry + while ((entry = zis.nextEntry) != null) { + def targetPath = targetDir.resolve(entry.name) + + // Security check: prevent zip slip + if (!targetPath.normalize().startsWith(targetDir.normalize())) { + throw new AbortOperationException("Invalid zip entry: ${entry.name}") + } + + if (entry.directory) { + Files.createDirectories(targetPath) + } else { + // Create parent directories if needed + if (targetPath.parent) { + Files.createDirectories(targetPath.parent) + } + + // Write file + Files.copy(zis, targetPath) + } + + zis.closeEntry() + } + } + } + } + + /** + * Extract a tar.gz file to a target directory + * + * @param tarGzFile The tar.gz file path + * @param targetDir The target directory + */ + private void extractTarGz(Path tarGzFile, Path targetDir) { + Files.newInputStream(tarGzFile).withCloseable { fis -> + new GzipCompressorInputStream(fis).withCloseable { gzis -> + new TarArchiveInputStream(gzis).withCloseable { tis -> + TarArchiveEntry entry + while ((entry = tis.nextTarEntry) != null) { + def targetPath = targetDir.resolve(entry.name) + + // Security check: prevent tar slip + if (!targetPath.normalize().startsWith(targetDir.normalize())) { + throw new AbortOperationException("Invalid tar entry: ${entry.name}") + } + + if (entry.directory) { + Files.createDirectories(targetPath) + } else { + // Create parent directories if needed + if (targetPath.parent) { + Files.createDirectories(targetPath.parent) + } + + // Write file + Files.copy(tis, targetPath) + } + } + } + } + } + } + + /** + * Check if a directory is empty + * + * @param dir The directory to check + * @return true if directory is empty + */ + private boolean isEmpty(Path dir) { + if (!Files.exists(dir) || !Files.isDirectory(dir)) { + return true + } + + try { + Stream entries = Files.list(dir) + try { + return !entries.findFirst().isPresent() + } finally { + entries.close() + } + } catch (Exception e) { + log.warn "Failed to check if directory is empty: ${dir}", e + return false + } + } + + /** + * Create a module bundle (tar.gz) from a module directory for publishing + * + * @param moduleDir The module directory to bundle + * @param targetFile The target bundle file path + * @return The created bundle file checksum + */ + static String createBundle(Path moduleDir, Path targetFile) { + if (!Files.exists(moduleDir) || !Files.isDirectory(moduleDir)) { + throw new AbortOperationException("Module directory not found: ${moduleDir}") + } + + try { + // Create parent directories if needed + if (targetFile.parent) { + Files.createDirectories(targetFile.parent) + } + + // Create tar.gz bundle + Files.newOutputStream(targetFile).withCloseable { fos -> + new GzipCompressorOutputStream(fos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add all files in module directory to bundle + addToTarArchive(tos, moduleDir, moduleDir) + } + } + } + final checksum = computeBundleChecksum(targetFile) + log.debug "Created module bundle: ${targetFile} (size: ${Files.size(targetFile)} bytes, checksum: $checksum)" + return checksum + } + catch (Exception e) { + // Clean up partial file on failure + if (Files.exists(targetFile)) { + try { + Files.delete(targetFile) + } catch (Exception cleanupError) { + log.warn "Failed to clean up after bundle creation failure: ${cleanupError.message}" + } + } + throw new AbortOperationException("Failed to create module bundle", e) + } + } + + /** + * Add files to tar archive recursively + * + * @param tos The tar archive output stream + * @param sourceDir The source directory being archived + * @param currentPath The current path being added + */ + private static void addToTarArchive(TarArchiveOutputStream tos, Path sourceDir, Path currentPath) { + + try ( def tarStream = Files.list(currentPath)) { + tarStream.each { Path path -> + // Skip .module-info file when creating bundle + if (path.fileName.toString() == MODULE_INFO_FILE) { + return + } + + def relativePath = sourceDir.relativize(path).toString() + + if (Files.isDirectory(path)) { + // Add directory entry + def entry = new TarArchiveEntry(path.toFile(), "${relativePath}/") + tos.putArchiveEntry(entry) + tos.closeArchiveEntry() + + // Recursively add directory contents + addToTarArchive(tos, sourceDir, path) + } else { + // Add file entry + def entry = new TarArchiveEntry(path.toFile(), relativePath) + entry.setSize(Files.size(path)) + tos.putArchiveEntry(entry) + + // Copy file content + Files.copy(path, tos) + tos.closeArchiveEntry() + } + } + } + } + + /** + * Compute the checksum of a module bundle + * + * @param bundleFile The bundle file + * @return The SHA-256 checksum as hex string + */ + static String computeBundleChecksum(Path bundleFile) { + if (!Files.exists(bundleFile)) { + throw new AbortOperationException("Bundle file not found: ${bundleFile}") + } + + try { + return ModuleChecksum.computeFile(bundleFile) + } + catch (Exception e) { + throw new AbortOperationException("Failed to compute bundle checksum", e) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy index b19f814745..531e6c4cc4 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy @@ -29,6 +29,8 @@ import nextflow.NF import nextflow.Session import nextflow.exception.IllegalModulePath import nextflow.exception.ScriptCompilationException +import nextflow.module.ModuleReference +import nextflow.module.spi.RemoteModuleResolverProvider import nextflow.plugin.Plugins import nextflow.plugin.extension.PluginExtensionProvider import nextflow.script.parser.v1.ScriptLoaderV1 @@ -162,14 +164,24 @@ class IncludeDef { @PackageScope Path resolveModulePath(include) { assert include - final result = include as Path if( result.isAbsolute() ) { if( result.scheme == 'file' ) return result throw new IllegalModulePath("Cannot resolve module path: ${result.toUriString()}") } + final str = include.toString() + if( str.startsWith('./') || str.startsWith('../') ) { + return getOwnerPath().resolveSibling(str).normalize() + } + // Not a local path — treat as remote module reference (scope/name) + return resolveRemoteModulePath(str) + } - return getOwnerPath().resolveSibling(include.toString()) + @PackageScope + Path resolveRemoteModulePath(String moduleName) { + // Use SPI to get the remote module resolver implementation + def resolver = RemoteModuleResolverProvider.getInstance() + return resolver.resolve(moduleName, session.baseDir) } @PackageScope @@ -206,9 +218,15 @@ class IncludeDef { throw new IllegalModulePath("Remote modules are not allowed -- Offending module: ${path.toUriString()}") final str = path.toString() - if( !str.startsWith('/') && !str.startsWith('./') && !str.startsWith('../') && !str.startsWith('plugin/') ) - throw new IllegalModulePath("Module path must start with / or ./ prefix -- Offending module: $str") + if( str.startsWith('/') || str.startsWith('./') || str.startsWith('../') || str.startsWith('plugin/') ) + return + // Otherwise must be a valid remote module reference in scope/name format + try { + ModuleReference.parse(str) + } catch( Exception e ) { + throw new IllegalModulePath("Module path must start with '/', './', '../' or 'plugin/' prefix, or be a valid remote module reference (scope/name) -- Offending module: $str") + } } @PackageScope diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy index 221b05f8df..c538b418c7 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy @@ -16,6 +16,13 @@ package nextflow.script +import groovyx.gpars.dataflow.DataflowReadChannel +import groovyx.gpars.dataflow.DataflowWriteChannel +import nextflow.exception.AbortOperationException +import nextflow.extension.CH +import nextflow.extension.DataflowHelper +import nextflow.extension.DumpHelper + import java.nio.file.Path import groovy.transform.CompileStatic import groovy.util.logging.Slf4j @@ -29,6 +36,8 @@ import nextflow.script.params.TupleInParam import nextflow.script.params.v2.ProcessInput import nextflow.script.params.v2.ProcessTupleInput +import java.util.concurrent.atomic.AtomicInteger + /** * Helper class for process entry execution feature. * @@ -47,6 +56,8 @@ class ProcessEntryHandler { private final BaseScript script private final Session session private final ScriptMeta meta + // Map to store process outputs + private Map processOutputs ProcessEntryHandler(BaseScript script, Session session, ScriptMeta meta) { this.script = script @@ -86,7 +97,7 @@ class ProcessEntryHandler { // Get input parameter values and execute the process final inputArgs = getProcessArguments(processDef) final processResult = script.invokeMethod(processName, inputArgs as Object[]) - + printOutput(processName, processResult) return processResult } @@ -97,6 +108,70 @@ class ProcessEntryHandler { return new WorkflowDef(script, workflowBody) } + /** + * Prints the process outputs. + * ChannelOut has two structures containing the outputs and anonymous channels. + * @param processName + * @param processResult ChannelOut containing the process outputs + */ + private printOutput(String processName, def processResult){ + if( ! processResult instanceof ChannelOut ) { + throw new AbortOperationException("Not a valid process output ($processResult.class)") + } + final results = processResult as ChannelOut + if (results.isEmpty()){ + log.debug("No outputs found for $processName") + return + } + + final named = new HashMap(results.size()) + + //Compute reverse index for named outputs + for (String name : results.getNames()){ + named.put(results.getProperty(name) as DataflowWriteChannel, name) + } + // Create the processOutputs map to keep the outputs order, and create the subcriber per output channel to collect the output values + int unnamedIndex = 1 + processOutputs = new LinkedHashMap<>(results.size()) + results.each { + String name = named.get(it) + if (!name) { + name = "anonymous-$unnamedIndex".toString() + unnamedIndex++ + } + processOutputs.put(name, []) + createProcessOutputSubscriber(name, CH.getReadChannel(it)) + } + //Add workflow + session.workflowMetadata.onComplete { + if( processOutputs) { + println "" + println "Process $processName Outputs:" + println DumpHelper.prettyPrintJson(processOutputs) + println "" + } + } + } + + /** + * Create a subscriber operator to inspect the output channels and build a map. + * At `onNext` event, add the output value in the `processOutputs` map with the output name as key. + * At `onComplete` event, convert single element list to single value. + * + * @param name Process output name + * @param channel Process output channel + */ + private void createProcessOutputSubscriber(String name, DataflowReadChannel channel ){ + def onNextClosure = { it -> + (processOutputs[name] as List).add(it) } + def onCompleteClosure = { + final list = processOutputs[name] as List + if( list.size() == 1) { + processOutputs[name] = list[0] + } + } + DataflowHelper.subscribeImpl(channel, [onNext: onNextClosure, onComplete: onCompleteClosure]) + } /** * Gets the input arguments for a process by parsing input parameter structures diff --git a/modules/nextflow/src/main/resources/META-INF/extensions.idx b/modules/nextflow/src/main/resources/META-INF/extensions.idx index 7250b10d76..f9075fb3d9 100644 --- a/modules/nextflow/src/main/resources/META-INF/extensions.idx +++ b/modules/nextflow/src/main/resources/META-INF/extensions.idx @@ -19,6 +19,7 @@ nextflow.conda.CondaConfig nextflow.config.ConfigMap nextflow.config.Manifest nextflow.config.WorkflowConfig +nextflow.config.RegistryConfig nextflow.container.ApptainerConfig nextflow.container.CharliecloudConfig nextflow.container.DockerConfig diff --git a/modules/nextflow/src/main/resources/META-INF/services/nextflow.module.spi.RemoteModuleResolver b/modules/nextflow/src/main/resources/META-INF/services/nextflow.module.spi.RemoteModuleResolver new file mode 100644 index 0000000000..d12a870946 --- /dev/null +++ b/modules/nextflow/src/main/resources/META-INF/services/nextflow.module.spi.RemoteModuleResolver @@ -0,0 +1 @@ +nextflow.module.DefaultRemoteModuleResolver diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy new file mode 100644 index 0000000000..ba90e5d005 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy @@ -0,0 +1,820 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import groovy.json.JsonSlurper +import io.seqera.npr.api.schema.v1.Module +import io.seqera.npr.api.schema.v1.ModuleChannel +import io.seqera.npr.api.schema.v1.ModuleChannelItem +import io.seqera.npr.api.schema.v1.ModuleMetadata +import io.seqera.npr.api.schema.v1.ModuleRelease +import io.seqera.npr.api.schema.v1.ModuleTool +import nextflow.cli.Launcher +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleRegistryClient +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Path + +/** + * Tests for CmdModuleInfo command + * + * @author Jorge Ejarque + */ +class CmdModuleInfoTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + def 'should display module info in formatted output'() { + given: + def metadata = new ModuleMetadata( + description: 'FastQC quality control analysis', + authors: ['nf-core', 'community'], + keywords: ['quality-control', 'fastqc', 'reads'] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + description: 'FastQC module', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module:') + output.contains('nf-core/fastqc') + output.contains('Version:') + output.contains('1.0.0') + output.contains('Description:') + output.contains('FastQC quality control analysis') + output.contains('Authors:') + output.contains('nf-core, community') + output.contains('Keywords:') + output.contains('quality-control, fastqc, reads') + output.contains('Usage Template:') + } + + def 'should display module info with specific version'() { + given: + def metadata = new ModuleMetadata( + description: 'FastQC quality control' + ) + + and: + def release = new ModuleRelease( + version: '0.9.0', + description: 'FastQC module', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.version = '0.9.0' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockClient = Mock(ModuleRegistryClient) { + fetchRelease(_, _) >> release + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Version:') + output.contains('0.9.0') + } + + def 'should display module info in JSON format'() { + given: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + authors: ['nf-core'], + keywords: ['quality-control'] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + description: 'FastQC module', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.output = 'json' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + // Extract JSON part (skip debug/log lines) + def lines = output.readLines() + def jsonStart = lines.findIndexOf { it.trim().startsWith('{') } + def jsonText = lines[jsonStart..-1].join('\n') + def json = new JsonSlurper().parseText(jsonText) + + then: + json.name == 'nf-core/fastqc' + json.fullName == 'nf-core/fastqc' + json.version == '1.0.0' + json.description == 'FastQC quality control' + json.authors == ['nf-core'] + json.keywords == ['quality-control'] + json.usageTemplate != null + } + + def 'should display module info with tools'() { + given: + def tool = new ModuleTool( + name: 'fastqc', + version: '0.12.1', + homepage: URI.create('https://www.bioinformatics.babraham.ac.uk/projects/fastqc/'), + documentation: URI.create('https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/') + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + tools: [tool] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Tools:') + output.contains('fastqc v0.12.1') + output.contains('Homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/') + } + + def 'should display module info with inputs'() { + given: + def inputItem1 = new ModuleChannelItem( + name: 'reads', + type: 'file', + description: 'Input FASTQ files', + pattern: '*.fastq.gz' + ) + def inputItem2 = new ModuleChannelItem( + name: 'meta', + type: 'map', + description: 'Sample metadata' + ) + + and: + def input = new ModuleChannel( + tuple: true, + items: [inputItem1, inputItem2] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [input] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Input:') + output.contains('(tuple)') + output.contains('reads (file)') + output.contains('Input FASTQ files') + output.contains('Pattern: *.fastq.gz') + output.contains('meta (map)') + output.contains('Sample metadata') + } + + def 'should display module info with outputs'() { + given: + def outputItem = new ModuleChannelItem( + name: 'html', + type: 'file', + description: 'FastQC HTML report', + pattern: '*_fastqc.html' + ) + + and: + def outputChannel = new ModuleChannel( + tuple: false, + items: [outputItem] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + output: ['html': outputChannel] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Output:') + output.contains('html') + output.contains('html (file)') + output.contains('FastQC HTML report') + output.contains('Pattern: *_fastqc.html') + } + + def 'should generate usage template with inputs'() { + given: + def inputItem1 = new ModuleChannelItem( + name: 'reads', + type: 'file' + ) + def inputItem2 = new ModuleChannelItem( + name: 'sample-id', + type: 'val' + ) + + and: + def input1 = new ModuleChannel( + tuple: false, + items: [inputItem1] + ) + def input2 = new ModuleChannel( + tuple: false, + items: [inputItem2] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [input1, input2] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Usage Template:') + output.contains('nextflow module run nf-core/fastqc') + output.contains('--reads ') + output.contains('--sample-id ') + } + + def 'should generate usage template with version'() { + given: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [] + ) + + and: + def release = new ModuleRelease( + version: '2.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.version = '2.0.0' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockClient = Mock(ModuleRegistryClient) { + fetchRelease(_, _) >> release + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Usage Template:') + output.contains('nextflow module run nf-core/fastqc') + output.contains('-version 2.0.0') + } + + def 'should generate usage template with map inputs'() { + given: + def inputItem = new ModuleChannelItem( + name: 'params', + type: 'map' + ) + + and: + def input = new ModuleChannel( + tuple: false, + items: [inputItem] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [input] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Usage Template:') + output.contains('--params. ') + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleInfo() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = [] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with multiple arguments'() { + given: + def cmd = new CmdModuleInfo() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['module1', 'module2'] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should display minimal info when metadata is sparse'() { + given: + def metadata = new ModuleMetadata( + description: null, + authors: null, + keywords: null + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + description: 'Module description', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module:') + output.contains('nf-core/fastqc') + output.contains('Version:') + output.contains('1.0.0') + output.contains('Description:') + output.contains('Module description') + } + + def 'should display N/A when no description is available'() { + given: + def metadata = new ModuleMetadata( + description: null + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + description: null, + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Description: N/A') + } + + def 'should display module with tuple outputs'() { + given: + def outputItem1 = new ModuleChannelItem( + name: 'html', + type: 'file', + description: 'HTML report' + ) + def outputItem2 = new ModuleChannelItem( + name: 'zip', + type: 'file', + description: 'ZIP archive' + ) + + and: + def outputChannel = new ModuleChannel( + tuple: true, + items: [outputItem1, outputItem2] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + output: ['qc': outputChannel] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Output:') + output.contains('qc (tuple)') + output.contains('html (file)') + output.contains('HTML report') + output.contains('zip (file)') + output.contains('ZIP archive') + output.contains(')') + } + + def 'should include all tool information in JSON output'() { + given: + def tool = new ModuleTool( + name: 'fastqc', + version: '0.12.1', + homepage: URI.create('https://example.com'), + documentation: URI.create('https://docs.example.com') + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + tools: [tool] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.output = 'json' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + // Extract JSON part (skip debug/log lines) + def lines = output.readLines() + def jsonStart = lines.findIndexOf { it.trim().startsWith('{') } + def jsonText = lines[jsonStart..-1].join('\n') + def json = new JsonSlurper().parseText(jsonText) + + then: + json.tools.size() == 1 + json.tools[0].name == 'fastqc' + json.tools[0].version == '0.12.1' + json.tools[0].homepage.scheme == 'https' + json.tools[0].homepage.host == 'example.com' + json.tools[0].documentation.scheme == 'https' + json.tools[0].documentation.host == 'docs.example.com' + } + + def 'should include input/output information in JSON output'() { + given: + def inputItem = new ModuleChannelItem( + name: 'reads', + type: 'file', + description: 'Input reads', + pattern: '*.fastq.gz' + ) + def inputChannel = new ModuleChannel( + tuple: true, + items: [inputItem] + ) + + and: + def outputItem = new ModuleChannelItem( + name: 'html', + type: 'file', + description: 'HTML report', + pattern: '*.html' + ) + def outputChannel = new ModuleChannel( + tuple: false, + items: [outputItem] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [inputChannel], + output: ['html': outputChannel] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.output = 'json' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + // Extract JSON part (skip debug/log lines) + def lines = output.readLines() + def jsonStart = lines.findIndexOf { it.trim().startsWith('{') } + def jsonText = lines[jsonStart..-1].join('\n') + def json = new JsonSlurper().parseText(jsonText) + + then: + json.input.size() == 1 + json.input[0].tuple == true + json.input[0].items[0].name == 'reads' + json.input[0].items[0].type == 'file' + json.input[0].items[0].description == 'Input reads' + json.input[0].items[0].pattern == '*.fastq.gz' + + and: + json.output.html.tuple == false + json.output.html.items[0].name == 'html' + json.output.html.items[0].type == 'file' + json.output.html.items[0].description == 'HTML report' + json.output.html.items[0].pattern == '*.html' + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy new file mode 100644 index 0000000000..6f3e9d08e9 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy @@ -0,0 +1,404 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import io.seqera.npr.api.schema.v1.Module +import io.seqera.npr.api.schema.v1.ModuleRelease +import nextflow.cli.Launcher +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleChecksum +import nextflow.module.ModuleInfo +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleStorage +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Files +import java.nio.file.Path +import java.util.zip.GZIPOutputStream + +/** + * Tests for CmdModuleInstall command + * + * @author Jorge Ejarque + */ +class CmdModuleInstallTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + def 'should install module with latest version'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + and: + // Create mock module package + def modulePackage = createModulePackage('nf-core', 'fastqc', '1.0.0') + + // Mock registry client + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', + latest: new ModuleRelease(version: '1.0.0') + ) + mockClient.downloadModule('nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Installing') + output.contains('nf-core/fastqc') + output.contains('1.0.0') + + and: + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + Files.exists(moduleDir.resolve(ModuleStorage.MODULE_MANIFEST_FILE)) + Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should install module with specific version'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.version = '2.0.0' + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('nf-core', 'fastqc', '2.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.downloadModule(_, _, _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return 'http://registry.com' + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Installing') + output.contains('nf-core/fastqc') + output.contains('2.0.0') + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + Files.exists(moduleDir.resolve(ModuleStorage.MODULE_MANIFEST_FILE)) + Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + ModuleInfo.load(moduleDir, 'registryUrl') == "http://registry.com" + + } + + def 'should update existing module with force flag'() { + given: + // Pre-install version 1.0.0 + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process OLD { }' + moduleDir.resolve('meta.yml').text = """ + name: nf-core/fastqc + version: '1.0.0' + description: Test module + """.stripIndent() + + and: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.version = '2.0.0' + cmd.force = true + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('nf-core', 'fastqc', '2.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.downloadModule('nf-core/fastqc', '2.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return 'registry' + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Installing') + output.contains('2.0.0') + + and: + moduleDir.resolve('main.nf').text.contains('FASTQC') // New content + } + + def 'should fail when module already installed without force'() { + given: + // Pre-install the module + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process FASTQC { }' + moduleDir.resolve('meta.yml').text = """ + name: nf-core/fastqc + version: '1.0.0' + description: Test module + """.stripIndent() + ModuleChecksum.save(moduleDir, 'wrong-checksum') + + and: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.version = '2.0.0' + cmd.root = tempDir + + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', + latest: new ModuleRelease(version: '2.0.0') + ) + cmd.client = mockClient + + when: + cmd.run() + + then: + def e = thrown(AbortOperationException) + e.message.contains('already installed') || e.message.contains('-force') + } + + def 'should handle module with scope in name'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['myorg/custom-module'] + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('myorg', 'custom-module', '1.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('myorg/custom-module') >> new Module( + name: 'myorg/custom-module', + latest: new ModuleRelease(version: '1.0.0') + ) + mockClient.downloadModule('myorg/custom-module', '1.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + cmd.client = mockClient + + when: + cmd.run() + + then: + def moduleDir = tempDir.resolve('modules/myorg/custom-module') + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + } + + def 'should create modules directory if it does not exist'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('nf-core', 'fastqc', '1.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', + latest: new ModuleRelease(version: '1.0.0') + ) + mockClient.downloadModule('nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + cmd.client = mockClient + + when: + cmd.run() + + then: + Files.exists(tempDir.resolve('modules')) + Files.exists(tempDir.resolve('modules/nf-core')) + Files.exists(tempDir.resolve('modules/nf-core/fastqc')) + } + + def 'should create checksum file after installation'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('nf-core', 'fastqc', '1.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', + latest: new ModuleRelease(version: '1.0.0') + ) + mockClient.downloadModule('nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + cmd.client = mockClient + + when: + cmd.run() + + then: + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.exists(moduleDir.resolve('.module-info')) + + and: + def checksum = ModuleChecksum.load(moduleDir) + checksum != null + !checksum.isEmpty() + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = [] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with too many arguments'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc', 'extra-arg'] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with invalid module reference'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['invalid-module-name'] // Missing scope + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + // Helper method to create a module package (tar.gz) + private byte[] createModulePackage(String scope, String name, String version) { + def baos = new ByteArrayOutputStream() + + // Create tar.gz with module files + def mainNfContent = """ + process ${name.toUpperCase().replaceAll('-', '_')} { + input: + path reads + + output: + path "*.html" + + script: + \"\"\" + echo "Running ${name}" + \"\"\" + } + """.stripIndent() + new GZIPOutputStream(baos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add main.nf + addTarEntry(tos, 'main.nf', mainNfContent.bytes) + + // Add meta.yml + def metaContent = """ + name: ${scope}/${name} + version: ${version} + description: Test module + """.stripIndent() + addTarEntry(tos, 'meta.yml', metaContent.bytes) + } + } + + return baos.toByteArray() + } + + private void addTarEntry(TarArchiveOutputStream tos, String name, byte[] content) { + def entry = new TarArchiveEntry(name) + entry.setSize(content.length) + tos.putArchiveEntry(entry) + tos.write(content) + tos.closeArchiveEntry() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy new file mode 100644 index 0000000000..dace2c8ba0 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy @@ -0,0 +1,186 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import groovy.json.JsonSlurper +import nextflow.module.ModuleChecksum +import nextflow.module.ModuleInfo +import nextflow.module.ModuleReference +import nextflow.module.ModuleStorage +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Tests for CmdModuleList command + * + * @author Jorge Ejarque + */ +class CmdModuleListTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + // No setup needed - using root field directly + + def 'should list installed modules with formatted output'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create test modules + createTestModule(storage, 'nf-core', 'fastqc', '1.0.0') + createTestModule(storage, 'nf-core', 'multiqc', '2.1.0') + + and: + def cmd = new CmdModuleList() + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Installed modules:') + output.contains('nf-core/fastqc') + output.contains('1.0.0') + output.contains('nf-core/multiqc') + output.contains('2.1.0') + output.contains('OK') + } + + def 'should list installed modules with JSON output'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create test module + createTestModule(storage, 'nf-core', 'fastqc', '1.5.0') + + and: + def cmd = new CmdModuleList() + cmd.output = 'json' + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + def json = new JsonSlurper().parseText(output) + + then: + json.modules != null + json.modules.size() == 1 + json.modules[0].name == 'nf-core/fastqc' + json.modules[0].version == '1.5.0' + json.modules[0].integrity != null + } + + def 'should handle no installed modules'() { + given: + def cmd = new CmdModuleList() + cmd.root = tempDir // Use test directory with no modules + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('No modules installed') + } + + def 'should show modified status for locally modified modules'() { + given: + def storage = new ModuleStorage(tempDir) + def moduleDir = createTestModule(storage, 'nf-core', 'fastqc', '1.0.0') + + // Modify the module to trigger checksum mismatch + moduleDir.resolve('main.nf').text = 'process MODIFIED { }' + + and: + def cmd = new CmdModuleList() + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('nf-core/fastqc') + output.contains('MODIFIED') + } + + def 'should list multiple modules sorted by name'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create modules in random order + createTestModule(storage, 'nf-core', 'samtools', '1.0.0') + createTestModule(storage, 'nf-core', 'fastqc', '1.0.0') + createTestModule(storage, 'myorg', 'custom', '2.0.0') + + and: + def cmd = new CmdModuleList() + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('fastqc') + output.contains('samtools') + output.contains('myorg/custom') + } + + private Path createTestModule(ModuleStorage storage, String scope, String name, String version) { + def moduleDir = storage.getModuleDir(new ModuleReference(scope, name)) + Files.createDirectories(moduleDir) + + // Create main.nf + moduleDir.resolve('main.nf').text = """ + process ${name.toUpperCase()} { + input: + path reads + + output: + path "*.html" + + script: + \"\"\" + echo "test" + \"\"\" + } + """.stripIndent() + + // Create meta.yml + moduleDir.resolve('meta.yml').text = """ + name: ${scope}/${name} + version: ${version} + description: Test module + """.stripIndent() + + // Create .module-info + ModuleInfo.save(moduleDir, [checksum: ModuleChecksum.compute(moduleDir), registryUrl: 'http://registry.com']) + + return moduleDir + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy new file mode 100644 index 0000000000..472f0593ef --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy @@ -0,0 +1,143 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import nextflow.cli.Launcher +import spock.lang.Specification +import spock.lang.TempDir + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Tests for CmdModulePublish command + * + * @author Jorge Ejarque + */ +class CmdModulePublishTest extends Specification { + + @TempDir + Path tempDir + + def 'should validate module structure' () { + given: + def moduleDir = tempDir.resolve('my-module') + Files.createDirectories(moduleDir) + + // Create required files + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('README.md').text = '# Test Module' + moduleDir.resolve('meta.yml').text = ''' +name: test/module +version: 1.0.0 +description: Test module +license: MIT +''' + + and: + def cmd = new CmdModulePublish() + cmd.dryRun = true + cmd.args = [moduleDir.toString()] + + when: + def errors = cmd.invokeMethod('validateModuleStructure', moduleDir) + + then: + errors.isEmpty() + } + + def 'should detect missing required files' () { + given: + def moduleDir = tempDir.resolve('my-module') + Files.createDirectories(moduleDir) + + // Only create main.nf, missing meta.yaml and README.md + moduleDir.resolve('main.nf').text = 'process TEST { }' + + and: + def cmd = new CmdModulePublish() + + when: + def errors = cmd.invokeMethod('validateModuleStructure', moduleDir) + + then: + errors.size() == 2 + errors.any { it.contains('meta.yml') } + errors.any { it.contains('README.md') } + } + + def 'should detect oversized module' () { + given: + def moduleDir = tempDir.resolve('my-module') + Files.createDirectories(moduleDir) + + // Create required files + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('README.md').text = '# Test Module' + moduleDir.resolve('meta.yml').text = ''' +name: test/module +version: 1.0.0 +description: Test module +license: MIT +''' + + // Create a large file (>1MB) + def largeFile = moduleDir.resolve('large-file.txt') + def content = 'x' * (1024 * 1024 + 1000) // 1MB + 1000 bytes + Files.writeString(largeFile, content) + + and: + def cmd = new CmdModulePublish() + + when: + def errors = cmd.invokeMethod('validateModuleStructure', moduleDir) + + then: + errors.size() == 1 + errors[0].contains('1MB limit') + } + + def 'should succeed in dry-run mode without authentication' () { + given: + def moduleDir = tempDir.resolve('my-module') + Files.createDirectories(moduleDir) + + // Create required files + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('README.md').text = '# Test Module' + moduleDir.resolve('meta.yml').text = ''' +name: test/module +version: 1.0.0 +description: Test module +license: MIT +''' + + and: + def cmd = new CmdModulePublish() + def launcher = new Launcher() + launcher.options = [:] + cmd.launcher = launcher + cmd.args = [moduleDir.toString()] + cmd.dryRun = true + + when: + cmd.run() + + then: + noExceptionThrown() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy new file mode 100644 index 0000000000..bdc306bdb5 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy @@ -0,0 +1,263 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleChecksum +import nextflow.module.ModuleInfo +import nextflow.module.ModuleReference +import nextflow.module.ModuleStorage +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Tests for CmdModuleRemove command + * + * @author Jorge Ejarque + */ +class CmdModuleRemoveTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + // No setup needed - using root field directly + + def 'should remove all module files'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference, true) + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module nf-core/fastqc files removed successfully') + !Files.exists(moduleDir) + } + + def 'should keep files with -keep-files flag'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference, true) + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.keepFiles = true + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Keeping module files for nf-core/fastqc ') + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + !Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should fail when both keep-files and force are set'() { + given: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.force = true + cmd.keepFiles = true + cmd.root = tempDir + + when: + cmd.run() + + then: + def e = thrown(AbortOperationException) + e.message.contains('Cannot use both -keep-files and -force options') + } + + def 'should fail to remove module without .module-info when force not set'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + createTestModule(storage, reference) // no .module-info created + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + when: + cmd.run() + + then: + def e = thrown(AbortOperationException) + e.message.contains('.module-info missing') + } + + def 'should force remove module without .module-info'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference) // no .module-info created + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.force = true + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module nf-core/fastqc files removed successfully') + !Files.exists(moduleDir) + } + + def 'should fail to remove modified module when force not set'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference, true) + // Modify a file to cause checksum mismatch + moduleDir.resolve('main.nf').text = 'process MODIFIED { }' + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + when: + cmd.run() + + then: + def e = thrown(AbortOperationException) + e.message.contains('local modifications') + } + + def 'should force remove modified module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference, true) + // Modify a file to cause checksum mismatch + moduleDir.resolve('main.nf').text = 'process MODIFIED { }' + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.force = true + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module nf-core/fastqc files removed successfully') + !Files.exists(moduleDir) + } + + def 'should handle removing non-existent module'() { + given: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/nonexistent'] + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module nf-core/nonexistent not found locally') + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleRemove() + cmd.args = [] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with too many arguments'() { + given: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc', 'extra-arg'] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + private Path createTestModule(ModuleStorage storage, ModuleReference reference, boolean withModuleInfo = false) { + def moduleDir = storage.getModuleDir(reference) + Files.createDirectories(moduleDir) + + // Create main.nf + moduleDir.resolve('main.nf').text = ''' + process FASTQC { + input: + path reads + + output: + path "*.html" + + script: + """ + fastqc ${reads} + """ + } + '''.stripIndent() + + // Create meta.yml + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + description: FastQC quality control + '''.stripIndent() + + if( withModuleInfo ) { + ModuleChecksum.save(moduleDir, ModuleChecksum.compute(moduleDir)) + } + + return moduleDir + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy new file mode 100644 index 0000000000..d93f58be53 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy @@ -0,0 +1,248 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import io.seqera.npr.api.schema.v1.Module +import io.seqera.npr.api.schema.v1.ModuleRelease +import nextflow.cli.CliOptions +import nextflow.cli.Launcher +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleStorage +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Files +import java.nio.file.Path +import java.util.zip.GZIPOutputStream + +/** + * Tests for CmdModuleRun command + * + * @author Jorge Ejarque + */ +class CmdModuleRunTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + def 'should run module and create output file'() { + given: + // Create a simple module script that creates a file + def moduleScript = ''' + process CREATE_FILE { + output: + path "test_output.txt" + + script: + """ + echo "Module executed successfully" > test_output.txt + """ + } + '''.stripIndent() + + and: + // Create module directory structure + def storage = new ModuleStorage(tempDir) + def moduleRef = new ModuleReference('nf-core', 'test-module') + def moduleDir = storage.getModuleDir(moduleRef) + Files.createDirectories(moduleDir) + + // Write main.nf + moduleDir.resolve('main.nf').text = moduleScript + + // Write meta.yml + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/test-module + version: 1.0.0 + description: Test module that creates a file + '''.stripIndent() + + and: + // Create mock module package + def modulePackage = createModulePackage(moduleScript) + + // Mock registry client + def mockClient = Stub(ModuleRegistryClient) + def moduleRelease = new ModuleRelease() + moduleRelease.version = '1.0.0' + def module = new Module() + module.name = 'nf-core/test-module' + module.latest = moduleRelease + mockClient.fetchModule(_) >> module // Use wildcard to match any argument + mockClient.downloadModule(_, _, _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + + and: + def cmd = new CmdModuleRun() + cmd.launcher = Mock(Launcher) { + getOptions() >> new CliOptions() + getCliString() >> "nextflow module run nf-core/test-module" + } + cmd.args = ['nf-core/test-module'] + cmd.root = tempDir + cmd.client = mockClient + + when: + cmd.run() + def stdout = capture + .toString() + .readLines()// remove the log part + .findResults { line -> !line.contains('DEBUG') ? line : null } + .findResults { line -> !line.contains('INFO') ? line : null }.join(" ") + + then: + stdout.contains('Executing module...') + stdout.contains('Process CREATE_FILE Outputs:') + stdout.contains("test_output.txt") + and: + // Verify module was installed + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + + } + + def 'should run module with specific version'() { + given: + def moduleScript = ''' + process CREATE_FILE_V2 { + output: + path "test_output_v2.txt" + + script: + """ + echo "Module version 2.0.0 executed successfully" > test_output_v2.txt + """ + } + ''' + + and: + def storage = new ModuleStorage(tempDir) + def moduleRef = new ModuleReference('nf-core', 'test-module') + def moduleDir = storage.getModuleDir(moduleRef) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = moduleScript + moduleDir.resolve('meta.yml').text = 'name: nf-core/test-module\nversion: 2.0.0' + + and: + def modulePackage = createModulePackage(moduleScript) + + def mockClient = Mock(ModuleRegistryClient) + mockClient.downloadModule('nf-core/test-module', '2.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + + and: + def cmd = new CmdModuleRun() + cmd.launcher = Mock(Launcher) { + getOptions() >> new CliOptions() + getCliString() >> "nextflow module run nf-core/test-module" + } + cmd.args = ['nf-core/test-module'] + cmd.version = '2.0.0' + cmd.root = tempDir + cmd.client = mockClient + + when: + cmd.run() + + then: + def stdout = capture + .toString() + .readLines()// remove the log part + .findResults { line -> !line.contains('DEBUG') ? line : null } + .findResults { line -> !line.contains('INFO') ? line : null } + .findResults { line -> !line.contains('plugin') ? line : null }.join(" ") + stdout.contains('Executing module...') + stdout.contains('Process CREATE_FILE_V2 Outputs:') + stdout.contains("test_output_v2.txt") + + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleRun() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = [] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with invalid module reference'() { + given: + def cmd = new CmdModuleRun() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['invalid-module'] // Missing scope + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + // Helper method to create a module package (tar.gz) + private byte[] createModulePackage(String mainNfContent) { + def baos = new ByteArrayOutputStream() + + new GZIPOutputStream(baos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add main.nf + addTarEntry(tos, 'main.nf', mainNfContent.bytes) + + // Add meta.yml + def metaContent = ''' + name: test-module + version: 1.0.0 + description: Test module + '''.stripIndent() + addTarEntry(tos, 'meta.yml', metaContent.bytes) + } + } + + return baos.toByteArray() + } + + private void addTarEntry(TarArchiveOutputStream tos, String name, byte[] content) { + def entry = new TarArchiveEntry(name) + entry.setSize(content.length) + tos.putArchiveEntry(entry) + tos.write(content) + tos.closeArchiveEntry() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleSearchTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleSearchTest.groovy new file mode 100644 index 0000000000..155a3ae994 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleSearchTest.groovy @@ -0,0 +1,228 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import groovy.json.JsonSlurper +import io.seqera.npr.api.schema.v1.ModuleSearchResult +import io.seqera.npr.api.schema.v1.SearchModulesResponse +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleRegistryClient +import nextflow.cli.Launcher +import org.junit.Rule +import spock.lang.Specification +import test.OutputCapture + + +/** + * Tests for CmdModuleSearch command + * + * @author Jorge Ejarque + */ +class CmdModuleSearchTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + // No setup needed - using client field for mocking + + def 'should search and display results in formatted output'() { + given: + def result1 = new ModuleSearchResult( + name: 'nf-core/fastqc', + repositoryPath: 'nf-core/modules', + description: 'FastQC quality control', + relevanceScore: 0.95, + keywords: ['quality-control', 'fastqc'], + tools: ['fastqc'], + revoked: false + ) + def result2 = new ModuleSearchResult( + name: 'nf-core/multiqc', + repositoryPath: 'nf-core/modules', + description: 'MultiQC reporting', + relevanceScore: 0.85, + keywords: ['quality-control', 'reporting'], + tools: ['multiqc'], + revoked: false + ) + + and: + def cmd = new CmdModuleSearch() + cmd.args = ['quality'] + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.limit = 20 + + and: + def response = new SearchModulesResponse( + query: 'quality', + totalResults: 2, + results: [result1, result2] + ) + assert response.results + // Mock the registry client directly using the test field + def mockClient = Mock(ModuleRegistryClient) { + search(_, _) >> response + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Searching for') + output.contains('quality') + output.contains('nf-core/fastqc') + output.contains('FastQC quality control') + output.contains('nf-core/multiqc') + output.contains('MultiQC reporting') + + } + + def 'should search and display results in JSON output'() { + given: + def result1 = new ModuleSearchResult( + name: 'nf-core/fastqc', + description: 'FastQC quality control', + relevanceScore: 0.95, + keywords: ['quality-control'], + tools: ['fastqc'], + revoked: false + ) + + and: + def cmd = new CmdModuleSearch() + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.args = ['fastqc'] + cmd.limit = 10 + cmd.output = 'json' + + and: + // Mock the registry client + def mockClient = Mock(ModuleRegistryClient) + mockClient.search('fastqc', 10) >> new SearchModulesResponse( + query: 'fastqc', + totalResults: 1, + results: [result1] + ) + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString().readLines() + .findResults { line -> !line.contains('DEBUG') ? line : null } + .findResults { line -> !line.contains('INFO') ? line : null } + .findResults { line -> !line.contains('Searching for') ? line : null }.join("\n") + def json = new JsonSlurper().parseText(output) + + then: + json.query == 'fastqc' + json.totalResults == 1 + json.results.size() == 1 + json.results[0].name == 'nf-core/fastqc' + json.results[0].description == 'FastQC quality control' + + } + + def 'should handle no search results'() { + given: + def cmd = new CmdModuleSearch() + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.args = ['nonexistent-module'] + cmd.limit = 20 + + and: + // Mock empty results + def mockClient = Mock(ModuleRegistryClient) + mockClient.search('nonexistent-module', 20) >> new SearchModulesResponse( + query: 'nonexistent-module', + totalResults: 0, + results: [] + ) + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('No modules found') + + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleSearch() + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.args = [] + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should handle search with custom limit'() { + given: + def results = (1..5).collect { i -> + new ModuleSearchResult( + name: "nf-core/module${i}", + description: "Module ${i}", + relevanceScore: 0.9 - (i * 0.1), + keywords: ['test'], + tools: ["tool${i}"], + revoked: false + ) + } + + and: + def cmd = new CmdModuleSearch() + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.args = ['test'] + cmd.limit = 5 + + and: + // Mock the client with 5 results + def mockClient = Mock(ModuleRegistryClient) + mockClient.search('test', 5) >> new SearchModulesResponse( + query: 'test', + totalResults: 5, + results: results + ) + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('nf-core/module1') + output.contains('nf-core/module5') + (1..5).every { i -> output.contains("Module ${i}") } + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/DefaultRemoteModuleResolverTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/DefaultRemoteModuleResolverTest.groovy new file mode 100644 index 0000000000..bda7b7f5ee --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/DefaultRemoteModuleResolverTest.groovy @@ -0,0 +1,46 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import nextflow.module.spi.RemoteModuleResolverProvider +import spock.lang.Specification + +/** + * Test for DefaultRemoteModuleResolver SPI implementation + * + * @author Jorge Ejarque + */ +class DefaultRemoteModuleResolverTest extends Specification { + + def 'should load resolver via SPI'() { + when: + def resolver = RemoteModuleResolverProvider.getInstance() + + then: + resolver != null + resolver.class.name == 'nextflow.module.DefaultRemoteModuleResolver' + resolver.priority == 0 + } + + def 'should return default priority'() { + given: + def resolver = new DefaultRemoteModuleResolver() + + expect: + resolver.getPriority() == 0 + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy new file mode 100644 index 0000000000..e526dde81f --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy @@ -0,0 +1,201 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import java.nio.file.Files +import java.nio.file.Path + +import spock.lang.Specification + +/** + * Test suite for InstalledModule + * + * @author Jorge Ejarque + */ +class InstalledModuleTest extends Specification { + + Path tempDir + + def setup() { + tempDir = Files.createTempDirectory('nf-installed-module-test-') + } + + def cleanup() { + tempDir?.deleteDir() + } + + def 'should report VALID integrity when checksum matches'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + // Compute actual checksum + def actualChecksum = ModuleChecksum.compute(moduleDir) + + // Save checksum + ModuleChecksum.save(moduleDir, actualChecksum) + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleDir.resolve('.module-info'), + expectedChecksum: actualChecksum, + installedVersion: "0.0.1" + ) + + when: + def integrity = installed.getIntegrity() + + then: + integrity == ModuleIntegrity.VALID + } + + def 'should report MODIFIED integrity when checksum differs'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + // Compute initial checksum + def originalChecksum = ModuleChecksum.compute(moduleDir) + ModuleChecksum.save(moduleDir, originalChecksum) + + // Modify the file + mainFile.text = 'process TEST { println "modified" }' + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleDir.resolve('.module-info'), + expectedChecksum: originalChecksum, + installedVersion: "0.0.1" + ) + + when: + def integrity = installed.getIntegrity() + + then: + integrity == ModuleIntegrity.MODIFIED + } + + def 'should report CORRUPTED integrity when main.nf is missing'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + // Don't create main.nf + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + def moduleInfoFile = moduleDir.resolve('.module-info') + // CORRUPTED check happens before .module-info check, so content doesn't matter here + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleInfoFile, + expectedChecksum: 'some-checksum' + ) + + when: + def integrity = installed.getIntegrity() + + then: + integrity == ModuleIntegrity.CORRUPTED + } + + def 'should report NO_REMOTE_MODULE when .module-info file absent'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + def moduleInfoFile = moduleDir.resolve('.module-info') + // Don't create .module-info file + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleInfoFile, + expectedChecksum: null + ) + + when: + def integrity = installed.getIntegrity() + + then: + integrity == ModuleIntegrity.NO_REMOTE_MODULE + } + + def 'should handle checksum computation failure gracefully'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + // Create .module-info with a checksum that won't match the computed checksum + ModuleChecksum.save(moduleDir, 'expected-checksum-that-will-not-match') + def moduleInfoFile = moduleDir.resolve('.module-info') + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleInfoFile, + expectedChecksum: 'expected-checksum-that-will-not-match' + ) + + when: + def integrity = installed.getIntegrity() + + then: + // Should handle gracefully - since checksum won't match, it should report as MODIFIED + integrity in [ModuleIntegrity.VALID, ModuleIntegrity.MODIFIED, ModuleIntegrity.CORRUPTED] + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy new file mode 100644 index 0000000000..4bb9185681 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy @@ -0,0 +1,414 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import java.nio.file.Files +import java.nio.file.Path + +import spock.lang.Specification + +/** + * Test suite for ModuleChecksum + * + * @author Jorge Ejarque + */ +class ModuleChecksumTest extends Specification { + + Path tempDir + + def setup() { + tempDir = Files.createTempDirectory('nf-checksum-test-') + } + + def cleanup() { + tempDir?.deleteDir() + } + + def 'should compute checksum for directory'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + // Create test files + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: test\nversion: 1.0.0' + moduleDir.resolve('README.md').text = '# Test Module' + + when: + def checksum = ModuleChecksum.compute(moduleDir) + + then: + checksum != null + checksum.length() == 64 // SHA-256 produces 64 hex characters + checksum ==~ /^[a-f0-9]{64}$/ + } + + def 'should produce consistent checksums for same content'() { + given: + def moduleDir1 = tempDir.resolve('module1') + def moduleDir2 = tempDir.resolve('module2') + Files.createDirectories(moduleDir1) + Files.createDirectories(moduleDir2) + + // Create identical content in both directories + ['main.nf', 'meta.yml', 'README.md'].each { filename -> + moduleDir1.resolve(filename).text = "content of ${filename}" + moduleDir2.resolve(filename).text = "content of ${filename}" + } + + when: + def checksum1 = ModuleChecksum.compute(moduleDir1) + def checksum2 = ModuleChecksum.compute(moduleDir2) + + then: + checksum1 == checksum2 + } + + def 'should produce different checksums for different content'() { + given: + def moduleDir1 = tempDir.resolve('module1') + def moduleDir2 = tempDir.resolve('module2') + Files.createDirectories(moduleDir1) + Files.createDirectories(moduleDir2) + + // Create different content + moduleDir1.resolve('main.nf').text = 'process TEST1 { }' + moduleDir2.resolve('main.nf').text = 'process TEST2 { }' + + when: + def checksum1 = ModuleChecksum.compute(moduleDir1) + def checksum2 = ModuleChecksum.compute(moduleDir2) + + then: + checksum1 != checksum2 + } + + def 'should exclude .module-info file from computation'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + moduleDir.resolve('main.nf').text = 'process TEST { }' + + // Compute initial checksum + def checksum1 = ModuleChecksum.compute(moduleDir) + + // Add .module-info file + ModuleChecksum.save(moduleDir, 'some-checksum-value') + + // Compute checksum again + def checksum2 = ModuleChecksum.compute(moduleDir) + + expect: + checksum1 == checksum2 // Should be the same, .module-info is ignored + } + + def 'should include subdirectories in checksum'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + moduleDir.resolve('main.nf').text = 'process TEST { }' + + // Compute checksum without subdirectory + def checksum1 = ModuleChecksum.compute(moduleDir) + + // Add subdirectory with file + def subDir = moduleDir.resolve('templates') + Files.createDirectories(subDir) + subDir.resolve('script.sh').text = '#!/bin/bash\necho "test"' + + // Compute checksum with subdirectory + def checksum2 = ModuleChecksum.compute(moduleDir) + + expect: + checksum1 != checksum2 // Checksums should differ + } + + def 'should save checksum to .module-info file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + def checksumValue = 'abc123def456' + + when: + ModuleChecksum.save(moduleDir, checksumValue) + + then: + def moduleInfoFile = moduleDir.resolve('.module-info') + Files.exists(moduleInfoFile) + ModuleChecksum.load(moduleDir) == checksumValue + } + + def 'should load checksum from .module-info file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleChecksum.save(moduleDir, 'abc123def456') + + when: + def checksum = ModuleChecksum.load(moduleDir) + + then: + checksum == 'abc123def456' + } + + def 'should return null when loading non-existent checksum file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + def checksum = ModuleChecksum.load(moduleDir) + + then: + checksum == null + } + + def 'should handle empty directory'() { + given: + def moduleDir = tempDir.resolve('empty-module') + Files.createDirectories(moduleDir) + + when: + def checksum = ModuleChecksum.compute(moduleDir) + + then: + checksum != null + checksum.length() == 64 + } + + def 'should handle files with special characters in names'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + // Create files with special characters (but valid on filesystem) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('file with spaces.txt').text = 'content' + moduleDir.resolve('file-with-dashes.txt').text = 'content' + + when: + def checksum = ModuleChecksum.compute(moduleDir) + + then: + checksum != null + checksum.length() == 64 + } + + def 'should handle binary files'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + // Create text and binary files + moduleDir.resolve('main.nf').text = 'process TEST { }' + def binaryFile = moduleDir.resolve('data.bin') + binaryFile.bytes = [0x00, 0x01, 0x02, 0xFF] as byte[] + + when: + def checksum = ModuleChecksum.compute(moduleDir) + + then: + checksum != null + checksum.length() == 64 + } + + def 'should sort files consistently for checksum computation'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + // Create files in arbitrary order + moduleDir.resolve('zzz.nf').text = 'content' + moduleDir.resolve('aaa.nf').text = 'content' + moduleDir.resolve('mmm.nf').text = 'content' + + def checksum1 = ModuleChecksum.compute(moduleDir) + + // Create another directory with files in different order + def moduleDir2 = tempDir.resolve('module2') + Files.createDirectories(moduleDir2) + moduleDir2.resolve('aaa.nf').text = 'content' + moduleDir2.resolve('mmm.nf').text = 'content' + moduleDir2.resolve('zzz.nf').text = 'content' + + def checksum2 = ModuleChecksum.compute(moduleDir2) + + expect: + checksum1 == checksum2 // Order shouldn't matter + } + + // Tests for computeFile() method + + def 'should compute checksum for a single file with default algorithm'() { + given: + def testFile = tempDir.resolve('test.txt') + testFile.text = 'Hello, World!' + + when: + def checksum = ModuleChecksum.computeFile(testFile) + + then: + checksum != null + checksum.length() == 64 // SHA-256 produces 64 hex characters + checksum ==~ /^[a-f0-9]{64}$/ + } + + def 'should produce consistent checksums for same file content'() { + given: + def file1 = tempDir.resolve('file1.txt') + def file2 = tempDir.resolve('file2.txt') + def content = 'Same content in both files' + file1.text = content + file2.text = content + + when: + def checksum1 = ModuleChecksum.computeFile(file1) + def checksum2 = ModuleChecksum.computeFile(file2) + + then: + checksum1 == checksum2 + } + + def 'should produce different checksums for different file content'() { + given: + def file1 = tempDir.resolve('file1.txt') + def file2 = tempDir.resolve('file2.txt') + file1.text = 'Content A' + file2.text = 'Content B' + + when: + def checksum1 = ModuleChecksum.computeFile(file1) + def checksum2 = ModuleChecksum.computeFile(file2) + + then: + checksum1 != checksum2 + } + + def 'should compute checksum for empty file'() { + given: + def emptyFile = tempDir.resolve('empty.txt') + emptyFile.text = '' + + when: + def checksum = ModuleChecksum.computeFile(emptyFile) + + then: + checksum != null + checksum.length() == 64 + // SHA-256 of empty string + checksum == 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' + } + + def 'should compute checksum for binary file'() { + given: + def binaryFile = tempDir.resolve('binary.dat') + binaryFile.bytes = [0x00, 0xFF, 0x42, 0xAB, 0xCD, 0xEF] as byte[] + + when: + def checksum = ModuleChecksum.computeFile(binaryFile) + + then: + checksum != null + checksum.length() == 64 + checksum ==~ /^[a-f0-9]{64}$/ + } + + def 'should support different hash algorithms'() { + given: + def testFile = tempDir.resolve('test.txt') + testFile.text = 'Test content for different algorithms' + + when: + def sha256 = ModuleChecksum.computeFile(testFile, 'SHA-256') + def sha512 = ModuleChecksum.computeFile(testFile, 'SHA-512') + + then: + sha256 != null + sha512 != null + sha256.length() == 64 // SHA-256: 256 bits = 64 hex chars + sha512.length() == 128 // SHA-512: 512 bits = 128 hex chars + sha256 != sha512 + } + + def 'should handle case-insensitive algorithm names'() { + given: + def testFile = tempDir.resolve('test.txt') + testFile.text = 'Test content' + + when: + def checksum1 = ModuleChecksum.computeFile(testFile, 'sha-256') + def checksum2 = ModuleChecksum.computeFile(testFile, 'SHA-256') + def checksum3 = ModuleChecksum.computeFile(testFile, 'Sha-256') + + then: + checksum1 == checksum2 + checksum2 == checksum3 + } + + def 'should throw exception for non-existent file'() { + given: + def nonExistentFile = tempDir.resolve('does-not-exist.txt') + + when: + ModuleChecksum.computeFile(nonExistentFile) + + then: + thrown(IllegalArgumentException) + } + + def 'should throw exception for directory instead of file'() { + given: + def directory = tempDir.resolve('subdir') + Files.createDirectories(directory) + + when: + ModuleChecksum.computeFile(directory) + + then: + thrown(IllegalArgumentException) + } + + def 'should compute known SHA-256 checksum correctly'() { + given: + def testFile = tempDir.resolve('known.txt') + testFile.text = 'abc' + + when: + def checksum = ModuleChecksum.computeFile(testFile) + + then: + // Known SHA-256 hash of "abc" + checksum == 'ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad' + } + + def 'should handle large file checksum computation'() { + given: + def largeFile = tempDir.resolve('large.txt') + // Create a file with ~1MB of data + def content = 'x' * 1024 * 1024 + largeFile.text = content + + when: + def checksum = ModuleChecksum.computeFile(largeFile) + + then: + checksum != null + checksum.length() == 64 + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleInfoTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleInfoTest.groovy new file mode 100644 index 0000000000..0df03532e7 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleInfoTest.groovy @@ -0,0 +1,183 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import java.nio.file.Files +import java.nio.file.Path + +import spock.lang.Specification + +/** + * Test suite for ModuleInfo + * + * @author Jorge Ejarque + */ +class ModuleInfoTest extends Specification { + + Path tempDir + + def setup() { + tempDir = Files.createTempDirectory('nf-module-info-test-') + } + + def cleanup() { + tempDir?.deleteDir() + } + + def 'should save and load a single property'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + ModuleInfo.save(moduleDir, 'checksum', 'abc123') + + then: + ModuleInfo.load(moduleDir, 'checksum') == 'abc123' + } + + def 'should create .module-info file on save'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + ModuleInfo.save(moduleDir, 'version', '1.0.0') + + then: + Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should update existing property without affecting others'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleInfo.save(moduleDir, 'checksum', 'original') + ModuleInfo.save(moduleDir, 'registryUrl', 'http://registry.com') + + when: + ModuleInfo.save(moduleDir, 'checksum', 'updated') + + then: + ModuleInfo.load(moduleDir, 'checksum') == 'updated' + ModuleInfo.load(moduleDir, 'registryUrl') == 'http://registry.com' + } + + def 'should return null when loading property from non-existent file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + def result = ModuleInfo.load(moduleDir, 'checksum') + + then: + result == null + } + + def 'should return null when loading non-existent property'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleInfo.save(moduleDir, 'registryUrl', 'http://registry.com') + + when: + def result = ModuleInfo.load(moduleDir, 'missing-property') + + then: + result == null + } + + def 'should save and load multiple properties via map'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + def props = [checksum: 'abc123', registryUrl: 'http://registry.com', author: 'test'] + + when: + ModuleInfo.save(moduleDir, props) + + then: + ModuleInfo.load(moduleDir, 'checksum') == 'abc123' + ModuleInfo.load(moduleDir, 'registryUrl') == 'http://registry.com' + ModuleInfo.load(moduleDir, 'author') == 'test' + } + + def 'should merge map properties with existing ones'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleInfo.save(moduleDir, 'existing', 'value') + + when: + ModuleInfo.save(moduleDir, [newprop: 'newval']) + + then: + ModuleInfo.load(moduleDir, 'existing') == 'value' + ModuleInfo.load(moduleDir, 'newprop') == 'newval' + } + + def 'should do nothing when saving null map'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + ModuleInfo.save(moduleDir, (Map) null) + + then: + !Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should do nothing when saving empty map'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + ModuleInfo.save(moduleDir, [:]) + + then: + !Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should load all properties as map'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleInfo.save(moduleDir, [checksum: 'abc123', registryUrl: 'http://registry.com']) + + when: + def result = ModuleInfo.load(moduleDir) + + then: + result['checksum'] == 'abc123' + result['registryUrl'] == 'http://registry.com' + } + + def 'should return empty map when loading all from non-existent file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + def result = ModuleInfo.load(moduleDir) + + then: + result == [:] + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy new file mode 100644 index 0000000000..b906b83c9f --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy @@ -0,0 +1,229 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import nextflow.exception.AbortOperationException +import spock.lang.Specification + +/** + * Test suite for ModuleReference + * + * @author Jorge Ejarque + */ +class ModuleReferenceTest extends Specification { + + def 'should parse valid module reference without @'() { + when: + def ref = ModuleReference.parse('nf-core/fastqc') + + then: + ref.scope == 'nf-core' + ref.name == 'fastqc' + ref.fullName == 'nf-core/fastqc' + } + + def 'should reject module reference with @ prefix'() { + when: + ModuleReference.parse('@nf-core/fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should parse module reference with multiple slashes'() { + when: + def ref = ModuleReference.parse('myorg/samtools/view') + + then: + ref.scope == 'myorg' + ref.name == 'samtools/view' + ref.fullName == 'myorg/samtools/view' + } + + def 'should reject invalid module reference without scope'() { + when: + ModuleReference.parse('fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should reject empty module reference'() { + when: + ModuleReference.parse('') + + then: + thrown(AbortOperationException) + } + + def 'should reject null module reference'() { + when: + ModuleReference.parse(null) + + then: + thrown(AbortOperationException) + } + + def 'should reject bare @ character'() { + when: + ModuleReference.parse('@') + + then: + thrown(AbortOperationException) + } + + def 'should reject module reference with only scope'() { + when: + ModuleReference.parse('nf-core/') + + then: + thrown(AbortOperationException) + } + + def 'should reject module reference with trailing slash'() { + when: + ModuleReference.parse('nf-core/fastqc/') + + then: + thrown(AbortOperationException) + } + + def 'should create module reference from components'() { + when: + def ref = new ModuleReference('nf-core', 'fastqc') + + then: + ref.scope == 'nf-core' + ref.name == 'fastqc' + ref.fullName == 'nf-core/fastqc' + } + + def 'should handle scope names with hyphens'() { + when: + def ref = ModuleReference.parse('my-org/my-module') + + then: + ref.scope == 'my-org' + ref.name == 'my-module' + } + + def 'should handle scope names with underscores'() { + when: + def ref = ModuleReference.parse('my_org/my_module') + + then: + ref.scope == 'my_org' + ref.name == 'my_module' + } + + def 'should handle module names with numbers'() { + when: + def ref = ModuleReference.parse('nf-core/bwa-mem2') + + then: + ref.scope == 'nf-core' + ref.name == 'bwa-mem2' + } + + def 'should implement equals correctly'() { + given: + def ref1 = ModuleReference.parse('nf-core/fastqc') + def ref2 = ModuleReference.parse('nf-core/fastqc') + def ref3 = ModuleReference.parse('nf-core/multiqc') + + expect: + ref1 == ref2 + ref1 != ref3 + } + + def 'should implement hashCode correctly'() { + given: + def ref1 = ModuleReference.parse('nf-core/fastqc') + def ref2 = ModuleReference.parse('nf-core/fastqc') + + expect: + ref1.hashCode() == ref2.hashCode() + } + + def 'should implement toString correctly'() { + given: + def ref = ModuleReference.parse('nf-core/fastqc') + + expect: + ref.toString() == 'nf-core/fastqc' + } + + def 'should be usable as map key'() { + given: + def ref1 = ModuleReference.parse('nf-core/fastqc') + def ref2 = ModuleReference.parse('nf-core/fastqc') + def ref3 = ModuleReference.parse('nf-core/multiqc') + + def map = [:] + map[ref1] = 'value1' + map[ref3] = 'value3' + + expect: + map[ref2] == 'value1' // ref2 equals ref1, should get same value + map[ref3] == 'value3' + map.size() == 2 + } + + def 'should handle org-style scopes'() { + when: + def ref = ModuleReference.parse('mycompany.io/custom-module') + + then: + ref.scope == 'mycompany.io' + ref.name == 'custom-module' + } + + def 'should reject module reference with spaces'() { + when: + ModuleReference.parse('nf-core/fast qc') + + then: + thrown(AbortOperationException) + } + + def 'should reject module reference with special characters'() { + when: + ModuleReference.parse('nf-core/fastqc!') + + then: + thrown(AbortOperationException) + } + + def 'should handle deeply nested module names'() { + when: + def ref = ModuleReference.parse('nf-core/samtools/sort/parallel') + + then: + ref.scope == 'nf-core' + ref.name == 'samtools/sort/parallel' + ref.fullName == 'nf-core/samtools/sort/parallel' + } + + def 'should parse from string with leading/trailing whitespace'() { + when: + def ref = ModuleReference.parse(' nf-core/fastqc ') + + then: + ref.scope == 'nf-core' + ref.name == 'fastqc' + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleRegistryClientTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleRegistryClientTest.groovy new file mode 100644 index 0000000000..66fbee0ff6 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleRegistryClientTest.groovy @@ -0,0 +1,435 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import com.github.tomakehurst.wiremock.WireMockServer +import com.github.tomakehurst.wiremock.client.WireMock +import groovy.json.JsonOutput +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream +import spock.lang.Specification +import spock.lang.TempDir + +import java.nio.file.Files +import java.nio.file.Path +import java.util.zip.GZIPOutputStream + +import static com.github.tomakehurst.wiremock.client.WireMock.* +import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig + +/** + * Integration tests for ModuleRegistryClient using WireMock + * + * @author Jorge Ejarque + */ +class ModuleRegistryClientTest extends Specification { + + @TempDir + Path tempDir + + WireMockServer wireMock + String url + static final String MODULES_API_PATH = "/api/v1/modules" + def setup() { + wireMock = new WireMockServer(wireMockConfig().dynamicPort()) + wireMock.start() + WireMock.configureFor("localhost", wireMock.port()) + url = "http://localhost:${wireMock.port()}/api" + } + + def cleanup() { + wireMock?.stop() + } + + def 'should fetch module metadata from registry'() { + given: + def moduleResponse = [ + name: 'nf-core/fastqc', + description: 'FastQC quality control', + latest: [ + version: '1.1.0', + createdAt: '2024-02-01T00:00:00Z' + ] + ] + + // Note: nf-core/fastqc is URL-encoded as nf-core%2Ffastqc + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson(moduleResponse)))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + def result = client.fetchModule('nf-core/fastqc') + + then: + result != null + result.name == 'nf-core/fastqc' + result.latest != null + result.latest.version == '1.1.0' + + and: 'verify request was made' + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc'))) + } + + def 'should search modules in registry'() { + given: + def searchResponse = [ + query: 'fastqc', + totalResults: 2, + results: [ + [ + name: 'nf-core/fastqc', + description: 'FastQC quality control', + relevanceScore: 0.95, + keywords: ['quality-control'], + tools: ['fastqc'], + revoked: false + ], + [ + name: 'other/fastqc', + description: 'Another FastQC module', + relevanceScore: 0.75, + keywords: ['qc'], + tools: ['fastqc'], + revoked: false + ] + ] + ] + + stubFor(get(urlPathEqualTo(MODULES_API_PATH)) + .withQueryParam('query', equalTo('fastqc')) + .withQueryParam('limit', equalTo('10')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson(searchResponse)))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + def result = client.search('fastqc', 10) + + then: + result != null + result.query == 'fastqc' + result.totalResults == 2 + result.results.size() == 2 + result.results[0].name == 'nf-core/fastqc' + // Use closeTo for Float/BigDecimal comparison + Math.abs(result.results[0].relevanceScore - 0.95) < 0.001 + + and: 'verify query parameters' + verify(getRequestedFor(urlPathEqualTo(MODULES_API_PATH)) + .withQueryParam('query', equalTo('fastqc')) + .withQueryParam('limit', equalTo('10'))) + } + + def 'should download module package from registry'() { + given: + def modulePackage = createTestModulePackage() + def expectedChecksum = "${computeSha256(modulePackage)}" + + // Note: URL-encoded path + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc/1.0.0/download')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/gzip') + .withHeader('X-NF-Module-Checksum', "sha256:${expectedChecksum}") + .withBody(modulePackage))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + def destFile = tempDir.resolve('module.tgz') + + when: + def result = client.downloadModule('nf-core/fastqc', '1.0.0', destFile) + + then: + result == url + Files.exists(destFile) + Files.size(destFile) == modulePackage.length + + and: + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH +'/nf-core%2Ffastqc/1.0.0/download'))) + } + + def 'should successfully fetch module without authentication'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson([name: 'nf-core/fastqc', latest: [version: '1.0.0']])))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + def result = client.fetchModule('nf-core/fastqc') + + then: + result != null + result.name == 'nf-core/fastqc' + + and: 'verify request was made' + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc'))) + } + + def 'should handle 404 not found error'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Fnonexistent')) + .willReturn(aResponse() + .withStatus(404) + .withBody('Module not found'))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/nonexistent') + + then: + def ex = thrown(AbortOperationException) + ex.message.contains('Unable to fetch module') || ex.message.contains('Module not found') + + and: + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Fnonexistent'))) + } + + def 'should handle 500 server error'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(500) + .withBody('Internal Server Error'))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should send user agent header'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson([name: 'nf-core/fastqc', latest: [version: '1.0.0'], releases: []])))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/fastqc') + + then: + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .withHeader('User-Agent', matching('.*'))) + } + + def 'should handle empty search results'() { + given: + stubFor(get(urlPathEqualTo(MODULES_API_PATH)) + .withQueryParam('query', equalTo('nonexistent')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson([ + query: 'nonexistent', + totalResults: 0, + results: [] + ])))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + def result = client.search('nonexistent', 10) + + then: + result != null + result.totalResults == 0 + result.results.isEmpty() + } + + def 'should respect custom search limit'() { + given: + stubFor(get(urlPathEqualTo(MODULES_API_PATH)) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson([ + query: 'test', + totalResults: 0, + results: [] + ])))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.search('test', 25) + + then: + verify(getRequestedFor(urlPathEqualTo(MODULES_API_PATH )) + .withQueryParam('limit', equalTo('25'))) + } + + def 'should handle malformed JSON response'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody('not valid json {]'))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should verify download includes checksum header'() { + given: + def modulePackage = createTestModulePackage() + def checksum = "sha256:${computeSha256(modulePackage)}" + + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc/1.0.0/download')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/gzip') + .withHeader('X-NF-Module-Checksum', checksum) + .withHeader('Docker-Content-Digest', checksum) + .withBody(modulePackage))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + def destFile = tempDir.resolve('module.tgz') + + when: + def result = client.downloadModule('nf-core/fastqc', '1.0.0', destFile) + + then: + result == url + Files.exists(destFile) + + and: 'verify checksum header was present' + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc/1.0.0/download'))) + } + + def 'should handle network errors gracefully'() { + given: + // Stub will not be set up, causing connection refused + def config = new RegistryConfig([url: "http://localhost:9999"]) // Invalid port + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should require authentication for publish'() { + given: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + def publishRequest = [name: 'nf-core/mymodule', version: '1.0.0'] + + when: + client.publishModule('nf-core/mymodule', publishRequest) + + then: + def ex = thrown(AbortOperationException) + ex.message.contains('Authentication required') + } + + def 'should handle publish failure with no auth token'() { + given: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + def publishRequest = [name: 'nf-core/mymodule', version: '1.0.0'] + + when: + client.publishModule('nf-core/mymodule', publishRequest) + + then: + def ex = thrown(AbortOperationException) + ex.message.contains('Authentication required') + } + + // Helper methods + + private byte[] createTestModulePackage() { + def baos = new ByteArrayOutputStream() + + new GZIPOutputStream(baos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add main.nf + def mainContent = 'process TEST { script: "echo test" }' + addTarEntry(tos, 'main.nf', mainContent.bytes) + + // Add meta.yml + def metaContent = 'name: test\nversion: 1.0.0' + addTarEntry(tos, 'meta.yml', metaContent.bytes) + } + } + + return baos.toByteArray() + } + + private void addTarEntry(TarArchiveOutputStream tos, String name, byte[] content) { + def entry = new TarArchiveEntry(name) + entry.setSize(content.length) + tos.putArchiveEntry(entry) + tos.write(content) + tos.closeArchiveEntry() + } + + private String computeSha256(byte[] data) { + def digest = java.security.MessageDigest.getInstance('SHA-256') + def hash = digest.digest(data) + return hash.encodeHex().toString() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy new file mode 100644 index 0000000000..2c87bdde8e --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy @@ -0,0 +1,193 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.file.FileHelper +import spock.lang.Specification +import spock.lang.TempDir + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Tests for ModuleResolver + * + * @author Jorge Ejarque + */ +class ModuleResolverTest extends Specification { + + @TempDir + Path tempDir + + def 'should throw exception when resolving non-installed module without auto-install'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + + when: + resolver.resolve(reference, null, false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('not installed') + e.message.contains('nextflow module install') + } + + def 'should throw exception when installed module is corrupted'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create corrupted module (directory exists but no main.nf) + Files.createDirectories(moduleDir) + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + + when: + resolver.resolve(reference, null, false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('corrupted') + + cleanup: + FileHelper.deletePath(moduleDir) + } + + def 'should warn about locally modified module'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create module with mismatched checksum + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + ModuleChecksum.save(moduleDir, 'wrong-checksum') + + when: + def result = resolver.resolve(reference, null, false) + + then: + result != null + result == moduleDir.resolve('main.nf') + + cleanup: + FileHelper.deletePath(moduleDir) + } + + def 'should throw exception when version mismatch without auto-install'() { + given: + def resolver = new ModuleResolver(tempDir, new RegistryConfig()) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create module with different version + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + + // Compute and save correct checksum + def checksum = ModuleChecksum.compute(moduleDir) + ModuleChecksum.save(moduleDir, checksum) + + when: + resolver.resolve(reference, '2.0.0', false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('version mismatch') + e.message.contains('installed=1.0.0') + e.message.contains('required=2.0.0') + + cleanup: + if( moduleDir ) FileHelper.deletePath(moduleDir) + } + + def 'should resolve installed module with matching version'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create valid module + Files.createDirectories(moduleDir) + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + + // Compute and save correct checksum + def checksum = ModuleChecksum.compute(moduleDir) + ModuleChecksum.save(moduleDir, checksum) + + when: + def result = resolver.resolve(reference, '1.0.0', false) + + then: + result == mainFile + + cleanup: + FileHelper.deletePath(moduleDir) + } + + def 'should throw exception when trying to update a module with local modifications without force'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create module with wrong checksum (simulating local modifications) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + ModuleChecksum.save(moduleDir, 'wrong-checksum') + + when: + resolver.installModule(reference, '2.0.0', false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('local modifications') + e.message.contains('-force') + + cleanup: + FileHelper.deletePath(moduleDir) + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy new file mode 100644 index 0000000000..cf90a07c7c --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy @@ -0,0 +1,165 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import nextflow.exception.AbortOperationException +import spock.lang.Specification +import spock.lang.TempDir + +import java.nio.file.Path + +/** + * Tests for ModuleSpec + * + * @author Paolo Di Tommaso + */ +class ModuleSpecTest extends Specification { + + @TempDir + Path tempDir + + def 'should load valid manifest' () { + given: + def metaYaml = tempDir.resolve('meta.yaml') + metaYaml.text = ''' +name: nf-core/fastqc +version: 1.0.0 +description: FastQC quality control +authors: + - John Doe +license: MIT +keywords: + - quality-control + - fastq +requires: + nextflow: ">=24.04.0" +''' + + when: + def manifest = ModuleSpec.load(metaYaml) + + then: + manifest.name == 'nf-core/fastqc' + manifest.version == '1.0.0' + manifest.description == 'FastQC quality control' + manifest.authors == ['John Doe'] + manifest.license == 'MIT' + manifest.keywords == ['quality-control', 'fastq'] + manifest.requires == ['nextflow': '>=24.04.0'] + } + + def 'should fail to load non-existent manifest' () { + given: + def metaYaml = tempDir.resolve('meta.yaml') + + when: + ModuleSpec.load(metaYaml) + + then: + thrown(AbortOperationException) + } + + def 'should validate complete manifest' () { + given: + def manifest = new ModuleSpec( + name: 'nf-core/fastqc', + version: '1.0.0', + description: 'FastQC quality control', + license: 'MIT' + ) + + when: + def errors = manifest.validate() + + then: + errors.isEmpty() + manifest.isValid() + } + + def 'should detect missing required fields' () { + given: + def manifest = new ModuleSpec( + name: 'nf-core/fastqc' + // missing version, description, license + ) + + when: + def errors = manifest.validate() + + then: + errors.size() == 3 + errors.any { it.contains('version') } + errors.any { it.contains('description') } + errors.any { it.contains('license') } + !manifest.isValid() + } + + def 'should validate version format' () { + given: + def manifest = new ModuleSpec( + name: 'nf-core/fastqc', + version: version, + description: 'Test', + license: 'MIT' + ) + + when: + def errors = manifest.validate() + + then: + errors.isEmpty() == valid + + where: + version | valid + '1.0.0' | true + '1.0.0-alpha' | true + '1.0.0-beta.1' | true + '1.0' | false + 'v1.0.0' | false + '1.0.0.0' | false + } + + def 'should validate module name format' () { + given: + def manifest = new ModuleSpec( + name: name, + version: '1.0.0', + description: 'Test', + license: 'MIT' + ) + + when: + def errors = manifest.validate() + + then: + errors.isEmpty() == valid + + where: + name | valid + 'nf-core/fastqc' | true + 'myorg/my-module' | true + 'org_1/tool_2' | true + 'nf-core/gfatools/gfa2fa' | true // nested module path + 'myorg/tools/sub/module' | true // deeply nested + 'org.name/tool/sub' | true // dot in scope + 'fastqc' | false + '@nf-core/fastqc' | false + 'nf-core/fast qc' | false + 'nf-core/' | false // trailing slash + '/nf-core/fastqc' | false // leading slash + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy new file mode 100644 index 0000000000..bb44f8fa94 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy @@ -0,0 +1,501 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import java.nio.file.Files +import java.nio.file.Path +import java.util.zip.GZIPOutputStream +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream + +import nextflow.exception.AbortOperationException + +import spock.lang.Specification + +/** + * Test suite for ModuleStorage + * + * @author Jorge Ejarque + */ +class ModuleStorageTest extends Specification { + + Path tempDir + + def setup() { + tempDir = Files.createTempDirectory('nf-module-storage-test-') + } + + def cleanup() { + tempDir?.deleteDir() + } + + def 'should get module directory path'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + + when: + def moduleDir = storage.getModuleDir(reference) + + then: + moduleDir == tempDir.resolve('modules/nf-core/fastqc') + } + + def 'should check if module is installed'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + when: + def installed = storage.isInstalled(reference) + + then: + !installed + + when: + Files.createDirectories(moduleDir) + installed = storage.isInstalled(reference) + + then: + installed + } + + def 'should return null for non-existent module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'nonexistent') + + when: + def installed = storage.getInstalledModule(reference) + + then: + installed == null + } + + def 'should get installed module with metadata'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + Files.createDirectories(moduleDir) + + // Create main.nf + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process FASTQC { }' + + // Create meta.yml with version + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = ''' + name: nf-core/fastqc + version: 1.0.0 + description: FastQC quality control + keywords: + - quality-control + - fastqc + '''.stripIndent() + + // Create .module-info file + ModuleChecksum.save(moduleDir, 'abc123def456') + def moduleInfoFile = moduleDir.resolve('.module-info') + + when: + def installed = storage.getInstalledModule(reference) + + then: + installed != null + installed.reference == reference + installed.directory == moduleDir + installed.mainFile == mainFile + installed.manifestFile == moduleDir.resolve('meta.yml') + installed.moduleInfoFile == moduleInfoFile + installed.expectedChecksum == 'abc123def456' + installed.installedVersion == '1.0.0' + } + + def 'should list all installed modules'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create multiple modules + def modules = [ + new ModuleReference('nf-core', 'fastqc'), + new ModuleReference('nf-core', 'multiqc'), + new ModuleReference('myorg', 'custom') + ] + + modules.each { ref -> + def moduleDir = storage.getModuleDir(ref) + Files.createDirectories(moduleDir) + + // Create main.nf + moduleDir.resolve('main.nf').text = 'process TEST { }' + + // Create meta.yml with version + moduleDir.resolve('meta.yml').text = """ + name: ${ref} + version: 1.0.0 + """.stripIndent() + + // Create .module-info + ModuleChecksum.save(moduleDir, 'checksum') + } + + when: + def installed = storage.listInstalled() + + then: + installed.size() == 3 + installed*.reference.fullName.sort() == ['myorg/custom', 'nf-core/fastqc', 'nf-core/multiqc'] + } + + def 'should list nested modules recursively'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create modules with nested paths + def modules = [ + new ModuleReference('nf-core', 'fastqc'), + new ModuleReference('nf-core', 'gfatools/gfa2fa'), + new ModuleReference('nf-core', 'gfatools/gfa2gfa'), + new ModuleReference('myorg', 'tools/subtools/module') + ] + + modules.each { ref -> + def moduleDir = storage.getModuleDir(ref) + Files.createDirectories(moduleDir) + + // Create main.nf + moduleDir.resolve('main.nf').text = 'process TEST { }' + + // Create meta.yml with version + moduleDir.resolve('meta.yml').text = """ + name: ${ref} + version: 1.0.0 + description: Test module + license: MIT + """.stripIndent() + + // Create .module-info + ModuleChecksum.save(moduleDir, 'checksum') + } + + when: + def installed = storage.listInstalled() + + then: + installed.size() == 4 + installed*.reference.fullName.sort() == [ + 'myorg/tools/subtools/module', + 'nf-core/fastqc', + 'nf-core/gfatools/gfa2fa', + 'nf-core/gfatools/gfa2gfa' + ] + } + + def 'should return empty list when no modules installed'() { + given: + def storage = new ModuleStorage(tempDir) + + when: + def installed = storage.listInstalled() + + then: + installed.isEmpty() + } + + def 'should install module from gzip package'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def version = '1.0.0' + def url = "http://registry.com" + + // Create a gzip package file + def packageFile = Files.createTempFile('module-', '.tgz') + createTestPackage(packageFile) + + when: + def installed = storage.installModule(reference, version, packageFile, url) + + then: + installed != null + installed.reference == reference + installed.installedVersion == '1.0.0' + Files.exists(installed.mainFile) + Files.exists(installed.moduleInfoFile) + Files.exists(installed.directory) + installed.registryUrl == url + installed.expectedChecksum == ModuleChecksum.compute(installed.directory) + + cleanup: + packageFile?.delete() + } + + def 'should replace existing module on install'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + def url = "http://registry.com" + + // Create existing installation + Files.createDirectories(moduleDir) + def oldFile = moduleDir.resolve('old-file.txt') + oldFile.text = 'old content' + + // Create package + def packageFile = Files.createTempFile('module-', '.tgz') + createTestPackage(packageFile) + + when: + def installed = storage.installModule(reference, '2.0.0', packageFile, url) + + then: + installed != null + !Files.exists(oldFile) // Old file should be removed + Files.exists(installed.mainFile) + + cleanup: + packageFile?.delete() + } + + def 'should remove installed module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create valid module with correct checksum + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + ModuleChecksum.save(moduleDir, ModuleChecksum.compute(moduleDir)) + + expect: + Files.exists(moduleDir) + + when: + def removed = storage.removeModule(reference, false) + + then: + removed + !Files.exists(moduleDir) + } + + def 'should return false when removing non-existent module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'nonexistent') + + when: + def removed = storage.removeModule(reference, false) + + then: + !removed + } + + def 'should throw when removing module without .module-info and force is false'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create module WITHOUT .module-info (but with required files for NO_REMOTE_MODULE integrity) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + + when: + storage.removeModule(reference, false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('.module-info missing') + Files.exists(moduleDir) + } + + def 'should force remove module without .module-info'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create module WITHOUT .module-info (but with required files for NO_REMOTE_MODULE integrity) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + + expect: + Files.exists(moduleDir) + + when: + def removed = storage.removeModule(reference, true) + + then: + removed + !Files.exists(moduleDir) + } + + def 'should throw when removing modified module and force is false'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create module with mismatched checksum (simulates local modification) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + ModuleChecksum.save(moduleDir, 'stale-checksum-from-install') + // Now modify a file to cause checksum mismatch + moduleDir.resolve('main.nf').text = 'process TEST_MODIFIED { }' + + when: + storage.removeModule(reference, false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('local modifications') + Files.exists(moduleDir) + } + + def 'should force remove modified module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create module with mismatched checksum (simulates local modification) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + ModuleChecksum.save(moduleDir, 'stale-checksum-from-install') + moduleDir.resolve('main.nf').text = 'process TEST_MODIFIED { }' + + when: + def removed = storage.removeModule(reference, true) + + then: + removed + !Files.exists(moduleDir) + } + + def 'should compute and save checksum on install'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def packageFile = Files.createTempFile('module-', '.tgz') + createTestPackage(packageFile) + def url = "http://registry.com" + + when: + def installed = storage.installModule(reference, '1.0.0', packageFile, url) + + then: + installed.expectedChecksum != null + installed.expectedChecksum.length() > 0 + Files.exists(installed.moduleInfoFile) + + cleanup: + packageFile?.delete() + } + + def 'should handle installation failure gracefully'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def invalidPackage = Files.createTempFile('invalid-', '.tgz') + invalidPackage.text = 'not a valid gzip file' + + when: + storage.installModule(reference, '1.0.0', invalidPackage, null) + + then: + thrown(Exception) + + and: + !storage.isInstalled(reference) // Should not leave partial installation + + cleanup: + invalidPackage?.delete() + } + + /** + * Helper method to create a test package file + */ + private void createTestPackage(Path packageFile) { + // Create a temporary directory with module content + def tempModuleDir = Files.createTempDirectory('temp-module-') + + // Create main.nf + tempModuleDir.resolve('main.nf').text = ''' + process FASTQC { + input: + path reads + + output: + path "*.html" + + script: + """ + fastqc ${reads} + """ + } + '''.stripIndent() + + // Create meta.yml + tempModuleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + description: FastQC quality control + keywords: + - quality-control + - fastqc + '''.stripIndent() + + // Create README + tempModuleDir.resolve('README.md').text = '# FastQC Module' + + // Create tar.gz archive using Java libraries + Files.newOutputStream(packageFile).withCloseable { fos -> + new GZIPOutputStream(fos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add all files from tempModuleDir + Files.walk(tempModuleDir).each { Path path -> + if (Files.isRegularFile(path)) { + // Get relative path + def relativePath = tempModuleDir.relativize(path).toString() + + // Create tar entry + def entry = new TarArchiveEntry(path.toFile(), relativePath) + tos.putArchiveEntry(entry) + + // Write file content + Files.copy(path, tos) + + tos.closeArchiveEntry() + } + } + } + } + } + + // Cleanup temp directory + tempModuleDir.deleteDir() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy index b5e1b7425e..527cbbfbbd 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy @@ -45,8 +45,8 @@ class IncludeDefTest extends Specification { expect: include.resolveModulePath('/abs/foo.nf') == '/abs/foo.nf' as Path - include.resolveModulePath('module.nf') == '/some/path/module.nf' as Path - include.resolveModulePath('foo/bar.nf') == '/some/path/foo/bar.nf' as Path + include.resolveModulePath('./module.nf') == '/some/path/module.nf' as Path + include.resolveModulePath('./foo/bar.nf') == '/some/path/foo/bar.nf' as Path when: include.resolveModulePath('http://foo.com/bar') @@ -66,17 +66,17 @@ class IncludeDefTest extends Specification { include.getOwnerPath() >> script when: - def result = include.realModulePath( 'mod-x.nf') + def result = include.realModulePath( './mod-x.nf') then: result == module when: - result = include.realModulePath('mod-x') + result = include.realModulePath('./mod-x') then: result == module when: - include.realModulePath('xyz') + include.realModulePath('./xyz') then: thrown(NoSuchFileException) @@ -98,21 +98,21 @@ class IncludeDefTest extends Specification { // when the module name reference a directory that contains // a file named 'main.nf', it's considered a module 'bundle' when: - def result = include.realModulePath('foo') + def result = include.realModulePath('./foo') then: result == module when: - include.realModulePath('bar') + include.realModulePath('./bar') then: thrown(NoSuchFileException) when: folder.resolve('bar').mkdir() - include.realModulePath('bar') + include.realModulePath('./bar') then: def e = thrown(ScriptCompilationException) - e.message == "Include 'bar' does not provide any module script -- the following path should contain a 'main.nf' script: '${folder.resolve('bar')}'" + e.message == "Include './bar' does not provide any module script -- the following path should contain a 'main.nf' script: '${folder.resolve('bar')}'" } def 'should check valid path' () { @@ -137,6 +137,16 @@ class IncludeDefTest extends Specification { when: include.checkValidPath('this/dir') then: + noExceptionThrown() // valid remote module reference (scope/name) + + when: + include.checkValidPath('nf-core/fastqc') + then: + noExceptionThrown() // valid remote module reference + + when: + include.checkValidPath('invalid!') + then: thrown(IllegalModulePath) when: diff --git a/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy b/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy index 16b45f87d3..6739f74306 100644 --- a/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy +++ b/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy @@ -15,6 +15,11 @@ */ package test + +import com.google.common.jimfs.JimfsPath +import nextflow.util.KryoHelper +import nextflow.util.PathSerializer + import java.nio.file.Files import java.nio.file.Path import java.util.zip.GZIPInputStream @@ -53,6 +58,13 @@ class TestHelper { static private fs = Jimfs.newFileSystem(Configuration.unix()); + static { + // Some tests failed after a Guava update (Guava 33.4+) when using Jimfs. + // Adding a default serializer for JimfsPaths to prevent Kryo's FieldSerializer from recursing into internal + // fields that may contain non-serializable objects such as lambdas + KryoHelper.kryo().addDefaultSerializer(JimfsPath.class, PathSerializer) + } + static Path createInMemTempDir() { Path tmp = fs.getPath("/tmp"); tmp.mkdir() diff --git a/modules/nf-commons/build.gradle b/modules/nf-commons/build.gradle index 958c789478..8ea4b12220 100644 --- a/modules/nf-commons/build.gradle +++ b/modules/nf-commons/build.gradle @@ -38,7 +38,7 @@ dependencies { api 'io.seqera:lib-retry:2.0.0' // patch gson dependency required by pf4j api 'com.google.code.gson:gson:2.13.1' - api 'io.seqera:npr-api:0.6.1' + api 'io.seqera:npr-api:0.21.4-SNAPSHOT' /* testImplementation inherited from top gradle build file */ testImplementation(testFixtures(project(":nextflow"))) diff --git a/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy b/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy new file mode 100644 index 0000000000..cb232af33b --- /dev/null +++ b/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy @@ -0,0 +1,91 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config + +import groovy.transform.CompileStatic +import nextflow.SysEnv +import nextflow.config.spec.ConfigOption +import nextflow.config.spec.ConfigScope +import nextflow.config.spec.ScopeName +import nextflow.script.dsl.Description + +/** + * Configuration scope for module registry settings + * + * @author Jorge Ejarque + */ +@ScopeName("registry") +@Description(""" + The `registry` scope provides configuration for the Nextflow module registry. + This includes the registry URL(s) and API key for authentication. +""") +@CompileStatic +class RegistryConfig implements ConfigScope { + + final static public String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api' + + @ConfigOption + @Description("Registry URL or list of registry URLs in priority order (primary URL first)") + final private Collection url + + @ConfigOption + @Description("API key for authenticating with the primary registry") + final private String apiKey + + /* required by extension point -- do not remove */ + RegistryConfig() { + this.url = [DEFAULT_REGISTRY_URL] + this.apiKey = null + } + + RegistryConfig(Map opts) { + final urlObject = opts.url ?: [DEFAULT_REGISTRY_URL] + if (urlObject instanceof Collection) + this.url = urlObject as Collection + else + this.url = [urlObject.toString()] + this.apiKey = opts.apiKey as String + } + + /** + * Get the primary (first) registry URL + * + * @return The primary registry URL + */ + String getUrl() { + return this.url ? url[0] as String : DEFAULT_REGISTRY_URL + } + + /** + * Get all registry URLs (primary first, fallbacks after) + * + * @return Collection of registry URLs + */ + Collection getAllUrls() { + return this.url ?: [DEFAULT_REGISTRY_URL] + } + + /** + * Get the API key for the primary registry. + * Authentication is only supported for the primary registry. + * + * @return The API key, or 'NXF_REGISTRY_TOKEN' env value if not configured + */ + String getApiKey() { + return apiKey ?: SysEnv.get('NXF_REGISTRY_TOKEN') + } +} diff --git a/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy b/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy index d066f51640..0313b9d0e4 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy @@ -25,7 +25,7 @@ import groovy.transform.CompileStatic import groovy.util.logging.Slf4j import io.seqera.http.HxClient import io.seqera.npr.api.schema.v1.ListDependenciesResponse -import io.seqera.npr.api.schema.v1.Plugin +import io.seqera.npr.api.schema.v1.PluginDependency import nextflow.BuildInfo import nextflow.util.RetryConfig import org.pf4j.PluginRuntimeException @@ -168,11 +168,11 @@ class HttpPluginRepository implements PrefetchUpdateRepository { } try { final ListDependenciesResponse decoded = encoder.decode(body) - if( decoded.plugins == null ) { + if( !decoded.plugins ) { throw new PluginRuntimeException("Failed to download plugin metadata: Failed to parse response body") } final result = new HashMap() - for( Plugin plugin : decoded.plugins ) { + for( PluginDependency plugin : decoded.plugins ) { if( plugin.releases ) { final pluginInfo = mapToPluginInfo(plugin) result.put(plugin.id, pluginInfo) @@ -195,7 +195,7 @@ class HttpPluginRepository implements PrefetchUpdateRepository { * @param plugin The Plugin object from the repository API response * @return A PluginInfo object compatible with pf4j's update repository interface */ - static protected PluginInfo mapToPluginInfo(Plugin plugin) { + static protected PluginInfo mapToPluginInfo(PluginDependency plugin) { assert plugin.releases, "Plugin releases cannot be empty" final pluginInfo = new PluginInfo() diff --git a/modules/nf-commons/src/test/nextflow/config/RegistryConfigTest.groovy b/modules/nf-commons/src/test/nextflow/config/RegistryConfigTest.groovy new file mode 100644 index 0000000000..26c727d960 --- /dev/null +++ b/modules/nf-commons/src/test/nextflow/config/RegistryConfigTest.groovy @@ -0,0 +1,107 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config + +import nextflow.SysEnv +import spock.lang.Specification + +/** + * Tests for RegistryConfig + * + * @author Jorge Ejarque + */ +class RegistryConfigTest extends Specification { + + def 'should create config with default values'() { + when: + def config = new RegistryConfig() + + then: + config.url == RegistryConfig.DEFAULT_REGISTRY_URL + config.allUrls == [RegistryConfig.DEFAULT_REGISTRY_URL] + config.apiKey == null + } + + def 'should initialize with custom URL as string'() { + when: + def config = new RegistryConfig([url: 'https://custom.registry.com']) + + then: + config.url == 'https://custom.registry.com' + config.allUrls == ['https://custom.registry.com'] + } + + def 'should initialize with URL as list'() { + when: + def config = new RegistryConfig([url: ['https://primary.registry.com', 'https://fallback.registry.com']]) + + then: + config.url == 'https://primary.registry.com' + config.allUrls == ['https://primary.registry.com', 'https://fallback.registry.com'] + } + + def 'should use default URL when none provided'() { + when: + def config = new RegistryConfig([:]) + + then: + config.url == RegistryConfig.DEFAULT_REGISTRY_URL + config.allUrls == [RegistryConfig.DEFAULT_REGISTRY_URL] + config.apiKey == null + } + + def 'should initialize with apiKey'() { + when: + def config = new RegistryConfig([url: 'https://registry.com', apiKey: 'token123']) + + then: + config.apiKey == 'token123' + } + + + + def 'should fall back to NXF_REGISTRY_TOKEN env var when apiKey not set'() { + given: + SysEnv.push([NXF_REGISTRY_TOKEN: 'env_var_token']) + def config = new RegistryConfig([url: 'https://registry.com']) + + expect: + // Without env var set, returns null + config.apiKey == 'env_var_token' + + cleanup: + SysEnv.pop() + } + + def 'should preserve order of URLs in list'() { + when: + def config = new RegistryConfig([url: ['https://first.com', 'https://second.com', 'https://third.com']]) + + then: + config.allUrls == ['https://first.com', 'https://second.com', 'https://third.com'] + config.url == 'https://first.com' + } + + def 'should handle empty list gracefully'() { + when: + def config = new RegistryConfig([url: []]) + + then: + config.allUrls == [RegistryConfig.DEFAULT_REGISTRY_URL] + config.url == RegistryConfig.DEFAULT_REGISTRY_URL + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java b/modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java new file mode 100644 index 0000000000..b438dbf7af --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java @@ -0,0 +1,45 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module.spi; + +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * Fallback implementation of RemoteModuleResolver that is used when no other + * implementation is found via the SPI mechanism. + * + *

This implementation throws an exception with a helpful error message + * indicating that remote module resolution is not available. + * + * @author Jorge Ejarque + */ +public class FallbackRemoteModuleResolver implements RemoteModuleResolver { + + @Override + public Path resolve(String moduleName, Path baseDir) { + if (!Files.exists(baseDir.resolve(moduleName))) { + throw new IllegalStateException("Module '" + moduleName + "' not locally found at 'modules' folder - use 'nextflow install' to download module files"); + } + return baseDir.resolve(moduleName).resolve("main.nf"); + } + + @Override + public int getPriority() { + return Integer.MIN_VALUE; // Fallback has lowest possible priority + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java new file mode 100644 index 0000000000..5c6cf1fc9b --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java @@ -0,0 +1,68 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module.spi; + +import java.nio.file.Path; + +/** + * Service Provider Interface for resolving remote modules referenced with '@scope/name' syntax. + * + *

Implementations should handle: + *

    + *
  • Checking if a module is already installed locally
    • + *
  • Downloading modules from a registry if not present
    • + *
  • Version resolution and validation
    • + *
+ * + *

The interface follows the Java SPI pattern. Implementations should be registered + * in META-INF/services/nextflow.module.spi.RemoteModuleResolver + * + * @author Jorge Ejarque + */ +public interface RemoteModuleResolver { + + /** + * Resolve a remote module reference (e.g., '@scope/name') to a local path. + * + *

This method should: + *

    + *
  1. Parse the module reference
    2. + *
  2. Check if the module is already installed locally
    3. + *
  3. Download and install the module if not present (auto-install)
    4. + *
  4. Validate version constraints if specified
    5. + *
+ * + * @param moduleName The module reference string (e.g., '@scope/name' or '@scope/name@version') + * @param baseDir The base directory for the project (used to locate the modules directory) + * @return Path to the resolved module's main.nf file + * @throws IllegalArgumentException if the module reference is invalid or resolution fails + */ + Path resolve(String moduleName, Path baseDir); + + /** + * Get the priority of this resolver. Higher priority resolvers are tried first. + * + *

Use this to allow custom implementations to override the default resolver. + * The default implementation should return 0. Custom implementations can return + * positive values to take precedence. + * + * @return Priority value (higher = tried first), default should be 0 + */ + default int getPriority() { + return 0; + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java new file mode 100644 index 0000000000..da1b2da9e7 --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java @@ -0,0 +1,92 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module.spi; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.ServiceLoader; + +/** + * Provider for accessing RemoteModuleResolver implementations via SPI. + * + *

This class uses the Java ServiceLoader mechanism to discover and load + * implementations of RemoteModuleResolver. It selects the implementation + * with the highest priority. + * + * @author Jorge Ejarque + */ +public class RemoteModuleResolverProvider { + + private static final Logger log = LoggerFactory.getLogger(RemoteModuleResolverProvider.class); + private static RemoteModuleResolver instance; + + /** + * Get the RemoteModuleResolver instance with the highest priority. + * + *

This method lazily loads and caches the resolver. It discovers all + * implementations via ServiceLoader and selects the one with the highest + * priority value. + * + *

If no implementations are found, returns the FallbackRemoteModuleResolver + * which throws an informative exception. + * + * @return The RemoteModuleResolver instance with highest priority + */ + public static synchronized RemoteModuleResolver getInstance() { + if (instance == null) { + instance = loadResolver(); + } + return instance; + } + + private static RemoteModuleResolver loadResolver() { + List resolvers = new ArrayList<>(); + ServiceLoader loader = ServiceLoader.load(RemoteModuleResolver.class); + + // Collect all available resolvers + for (RemoteModuleResolver resolver : loader) { + resolvers.add(resolver); + log.debug("Discovered RemoteModuleResolver: {} with priority {}", + resolver.getClass().getName(), resolver.getPriority()); + } + + // Sort by priority (highest first) + resolvers.sort(Comparator.comparingInt(RemoteModuleResolver::getPriority).reversed()); + + if (resolvers.isEmpty()) { + log.warn("No RemoteModuleResolver implementations found via SPI, using fallback"); + return new FallbackRemoteModuleResolver(); + } + + RemoteModuleResolver selected = resolvers.get(0); + log.debug("Selected RemoteModuleResolver: {} with priority {}", + selected.getClass().getName(), selected.getPriority()); + + return selected; + } + + /** + * Reset the cached instance. Used primarily for testing. + */ + public static synchronized void reset() { + instance = null; + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java b/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java index 5b76af4111..c52931142a 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java @@ -23,6 +23,7 @@ import java.util.Set; import java.util.function.Function; +import nextflow.module.spi.RemoteModuleResolverProvider; import nextflow.script.ast.IncludeNode; import nextflow.script.ast.ScriptNode; import org.codehaus.groovy.control.SourceUnit; @@ -72,8 +73,18 @@ private SourceUnit resolveInclude(IncludeNode node, SourceUnit sourceUnit, Funct var source = node.source.getText(); if( source.startsWith("plugin/") ) return null; - var uri = sourceUnit.getSource().getURI(); - var includeUri = getIncludeUri(uri, source); + + var parent = Path.of(sourceUnit.getSource().getURI()).getParent(); + + // Resolve remote module paths (scope/name format, not starting with local prefixes) + if( isRemoteModule(source) ) { + var modules = Path.of("./modules"); + var resolver = RemoteModuleResolverProvider.getInstance(); + resolver.resolve(source, modules.getParent()); + parent = modules; + } + + var includeUri = getIncludeUri(parent, source); if( compiler.getSource(includeUri) != null ) return null; if( !Files.exists(Path.of(includeUri)) ) @@ -86,8 +97,15 @@ private SourceUnit resolveInclude(IncludeNode node, SourceUnit sourceUnit, Funct return includeSource; } - private static URI getIncludeUri(URI uri, String source) { - Path includePath = Path.of(uri).getParent().resolve(source); + static boolean isRemoteModule(String source) { + if( source.startsWith("/") || source.startsWith("./") || source.startsWith("../") ) + return false; + // Must match scope/name pattern: scope is lowercase alphanumeric with dots/underscores/hyphens + return source.matches("^[a-z0-9][a-z0-9._\\-]*/[a-z][a-z0-9._\\-]*(/[a-z][a-z0-9._\\-]*)*$"); + } + + private static URI getIncludeUri(Path parent, String source) { + Path includePath = parent.resolve(source); if( Files.isDirectory(includePath) ) includePath = includePath.resolve("main.nf"); else if( !source.endsWith(".nf") ) diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java index 7dc9929bbe..b8bf711d93 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java @@ -83,7 +83,10 @@ public void visitInclude(IncludeNode node) { setPlaceholderTargets(node); return; } - var includeUri = getIncludeUri(uri, source); + + var isRemoteModule = ModuleResolver.isRemoteModule(source); + var parent = isRemoteModule ? Path.of("modules") : Path.of(uri).getParent(); + var includeUri = getIncludeUri(parent, source); if( !isIncludeStale(node, includeUri) ) return; changed = true; @@ -121,8 +124,8 @@ private static void setPlaceholderTargets(IncludeNode node) { } } - private static URI getIncludeUri(URI uri, String source) { - Path includePath = Path.of(uri).getParent().resolve(source); + private static URI getIncludeUri(Path parent, String source) { + Path includePath = parent.resolve(source); if( Files.isDirectory(includePath) ) includePath = includePath.resolve("main.nf"); else if( !source.endsWith(".nf") ) diff --git a/settings.gradle b/settings.gradle index fe58cb245b..22284225f8 100644 --- a/settings.gradle +++ b/settings.gradle @@ -50,3 +50,4 @@ include 'plugins:nf-k8s' include 'plugins:nf-seqera' //includeBuild '../sched' +//includeBuild('../plugin-registry') diff --git a/specs/251117-module-system/data-model.md b/specs/251117-module-system/data-model.md index d9345e33a4..0d3ef4cf8a 100644 --- a/specs/251117-module-system/data-model.md +++ b/specs/251117-module-system/data-model.md @@ -2,6 +2,7 @@ **Date**: 2026-01-19 **Feature**: 251117-module-system +**Last Updated**: 2026-02-27 (reflects final implementation) ## Overview @@ -39,88 +40,39 @@ class ModuleReference { --- -### 2. ModuleManifest +### 2. ModuleSpec -Parsed representation of `meta.yaml` file. +Parsed representation of `meta.yaml` file. Class: `nextflow.module.ModuleSpec`. ```groovy @CompileStatic -class ModuleManifest { +class ModuleSpec { String name // e.g., "nf-core/fastqc" (without @) String version // e.g., "1.0.0" String description // Module description List keywords // Discovery keywords List authors // GitHub handles - List maintainers String license // SPDX identifier + Map requires // dependency -> version constraint - ModuleRequirements requires - List tools - List input - Map output -} - -@CompileStatic -class ModuleRequirements { - String nextflow // Version constraint, e.g., ">=24.04.0" - List plugins // e.g., ["nf-amazon@2.0.0"] - List modules // e.g., ["nf-core/samtools@>=1.0.0"] - List workflows // e.g., ["nf-core/fastq-align@1.0.0"] -} - -@CompileStatic -class ToolDefinition { - String name // Tool identifier - String description - String homepage - String documentation - String doi - List license - String identifier // bio.tools identifier - Map args -} - -@CompileStatic -class ArgDefinition { - String flag // CLI flag, e.g., "-K" - String type // boolean, integer, float, string, file, path - String description - Object defaultValue - List enumValues - boolean required = false + static ModuleSpec load(Path metaYamlPath) { ... } + List validate() { ... } // Returns list of validation errors + boolean isValid() { ... } } ``` **Validation Rules**: -- `version`: Must be valid SemVer (MAJOR.MINOR.PATCH) -- `type` in ArgDefinition: Must be one of: boolean, integer, float, string, file, path -- `enumValues`: If present, configured value must be in this list +- `name`: Must match `scope/name` or `scope/path/to/name` pattern +- `version`: Must be valid SemVer (`MAJOR.MINOR.PATCH[-prerelease]`) +- `description`, `license`: Required fields (validate() reports missing) ---- - -### 3. ModuleInfo - -Module metadata returned from registry API. - -```groovy -@CompileStatic -class ModuleInfo { - String name // e.g., "nf-core/fastqc" - String version // Specific version - String latestVersion // Latest available - String description - String checksum // SHA-256 of bundle - long downloadCount - Instant publishedAt - List versions // All available versions -} -``` +**Note**: Tool/argument definitions were removed from the ADR and are not part of `ModuleSpec`. --- -### 4. InstalledModule +### 3. InstalledModule -Represents a module in local `modules/` directory. +Represents a module in the local `modules/` directory. ```groovy @CompileStatic @@ -132,7 +84,6 @@ class InstalledModule { Path checksumFile // e.g., /project/modules/@nf-core/fastqc/.checksum String installedVersion String expectedChecksum - ModuleManifest manifest ModuleIntegrity getIntegrity() { // Compute and compare checksum @@ -158,84 +109,99 @@ enum ModuleIntegrity { --- -### 5. ModuleConfig +### 4. ModulesConfig and RegistryConfig -Module configuration from `nextflow.config`. +Modules configuration loaded from `nextflow_spec.json` ( or the `modules {}` block in `nextflow.config` as alternative). Registry settings from the `registry {}` block in `nextflow.config`. ```groovy +@ScopeName("modules") @CompileStatic -class ModuleConfig { - Map modules = [:] // name -> version - RegistryConfig registry +class ModulesConfig implements ConfigScope { + Map modules = [:] // module fullName -> version + + String getVersion(String moduleName) { ... } + boolean hasVersion(String moduleName) { ... } } +@ScopeName("registry") @CompileStatic -class RegistryConfig { - String url = 'https://registry.nextflow.io' - List urls = [] // Multiple registries - Map auth = [:] // registry -> token expression +class RegistryConfig implements ConfigScope { + static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api' + + Collection url // Registry URL(s) in priority order + String apiKey // API key (falls back to NXF_REGISTRY_TOKEN env var) + + String getUrl() // Returns primary (first) URL + Collection getAllUrls() + String getApiKey() // Returns apiKey or NXF_REGISTRY_TOKEN } ``` **Config Syntax**: -```groovy +```nextflow +// nextflow_spec.json (current approach) +{ + "modules": { + "@nf-core/fastqc": "1.0.0", + "@nf-core/bwa-align": "1.2.0" + } +} + +// nextflow.config (alternative not currently used) modules { '@nf-core/fastqc' = '1.0.0' '@nf-core/bwa-align' = '1.2.0' } registry { - url = 'https://registry.nextflow.io' - auth { - 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' - } + url = [ + 'https://private.registry.myorg.com', + 'https://registry.nextflow.io/api' + ] + apiKey = '${MYORG_TOKEN}' // Only applied to the primary registry } ``` --- -### 6. ToolArgsContext +### 5. DefaultRemoteModuleResolver (SPI) -Runtime context for tool arguments in process scripts. +Bridges the DSL parser to the module resolution runtime. Class: `nextflow.module.DefaultRemoteModuleResolver`. ```groovy -@CompileStatic -class ToolArgsContext { - private Map tools = [:] - - ToolArgs getAt(String toolName) { - return tools[toolName] +// Implements: nextflow.module.spi.RemoteModuleResolver (nf-lang) +class DefaultRemoteModuleResolver implements RemoteModuleResolver { + int getPriority() { return 0 } // Can be overridden by plugins with higher priority + + Path resolve(String moduleName, Path baseDir) { + // 1. Parse ModuleReference from "@scope/name" + // 2. Read version constraints from nextflow_spec.json / ModulesConfig + // 3. Call ModuleResolver.installModule(reference, version, autoInstall=true) + // 4. Return path to modules/@scope/name/main.nf } } +``` -@CompileStatic -class ToolArgs { - private Map schema - private Map values - - String getAt(String argName) { - def def = schema[argName] - def value = values[argName] - if (value == null) return '' - if (def.type == 'boolean') { - return value ? def.flag : '' - } - return "${def.flag} ${value}" - } +The SPI is loaded via Java `ServiceLoader` by `RemoteModuleResolverProvider` (in `nf-lang`), which selects the highest-priority implementation available. - String toString() { - schema.keySet() - .findAll { values.containsKey(it) && values[it] != null } - .collect { this[it] } - .findAll { it } - .join(' ') - } +--- + +### 6. PipelineSpec + +Reads and writes `nextflow_spec.json` in the project root. Class: `nextflow.pipeline.PipelineSpec`. + +```groovy +class PipelineSpec { + PipelineSpec(Path baseDir) + Map getModules() + void addModuleEntry(String name, String version) + boolean removeModuleEntry(String name) } ``` --- -### 7. ModuleResolutionResult +### 6. ModuleResolutionResult Result of module resolution process. @@ -246,7 +212,6 @@ class ModuleResolutionResult { Path resolvedPath // Absolute path to main.nf ResolutionAction action String message // Warning/info message if any - ModuleManifest manifest } enum ResolutionAction { @@ -263,22 +228,14 @@ enum ResolutionAction { ## Relationships ``` -ModuleConfig (1) -----> (*) ModuleReference - | - v +PipelineSpec (1) -----> (*) ModuleReference (nextflow_spec.json) +ModulesConfig (1) -----> (*) ModuleReference (nextflow.config alternative) RegistryConfig (1) -----> (*) Registry URLs ModuleReference (1) -----> (0..1) InstalledModule | v (via registry) -ModuleInfo (1) -----> (1) ModuleManifest - -InstalledModule (1) -----> (1) ModuleManifest - -----> (*) ToolDefinition - -----> (*) ArgDefinition - -ToolArgsContext (1) -----> (*) ToolArgs - -----> (*) ArgDefinition (schema) +ModuleSpec (1) <----- InstalledModule (from meta.yaml) ``` --- @@ -287,15 +244,16 @@ ToolArgsContext (1) -----> (*) ToolArgs ``` project-root/ -├── nextflow.config # modules{}, registry{} blocks +├── nextflow.config # registry{} block; optional modules{} block +├── nextflow_spec.json # auto-managed module version pins ├── main.nf # include { X } from '@scope/name' └── modules/ └── @scope/ └── name/ - ├── .checksum # SHA-256 from registry + ├── .checksum # SHA-256 from registry (download integrity) ├── main.nf # Entry point (required) - ├── meta.yaml # Manifest (optional but recommended) - ├── README.md # Documentation + ├── meta.yaml # Manifest (required for publishing) + ├── README.md # Documentation (required for publishing) └── [other files] # Supporting files ``` @@ -306,9 +264,9 @@ project-root/ | Entity | Field | Validation | |--------|-------|------------| | ModuleReference | fullName | Pattern: `^@[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$` | -| ModuleManifest | version | SemVer: `MAJOR.MINOR.PATCH` | -| ArgDefinition | type | Enum: boolean, integer, float, string, file, path | -| ArgDefinition | enumValues | If set, value must be member | +| ModuleSpec | name | Pattern: `scope/name` or `scope/path/to/name` | +| ModuleSpec | version | SemVer: `MAJOR.MINOR.PATCH[-prerelease]` | +| ModuleSpec | description, license | Required (non-empty) | | InstalledModule | directory | Must contain main.nf | -| ModuleConfig | modules | Keys must be valid module references | +| ModulesConfig | modules keys | Must be valid module fullName | | RegistryConfig | url | Valid HTTPS URL | \ No newline at end of file diff --git a/specs/251117-module-system/plan.md b/specs/251117-module-system/plan.md index 0d12cf3f96..e9d11d552f 100644 --- a/specs/251117-module-system/plan.md +++ b/specs/251117-module-system/plan.md @@ -15,6 +15,7 @@ Implement client-side module system for Nextflow enabling pipeline developers to - Existing config parser (ConfigBuilder, ConfigParser) - Existing HTTP client (HxClient from io.seqera.http) - Existing plugin authentication infrastructure +- Existing npr-api (registry data models and schema validation) **Storage**: Local filesystem (`modules/@scope/name/` per-project, `.checksum` files) **Testing**: Spock Framework for unit tests, integration tests in `tests/` directory **Target Platform**: JVM 17+ (same as Nextflow core) @@ -49,6 +50,7 @@ Implement client-side module system for Nextflow enabling pipeline developers to ```text specs/251117-module-system/ ├── plan.md # This file +├── spec.md # Feature specification ├── research.md # Phase 0 output ├── data-model.md # Phase 1 output ├── quickstart.md # Phase 1 output @@ -61,40 +63,74 @@ specs/251117-module-system/ ```text modules/nextflow/src/main/groovy/nextflow/ ├── cli/ -│ └── CmdModule.groovy # NEW: Module CLI command +│ ├── CmdModule.groovy # Main module command (uses JCommander) +│ └── module/ +│ ├── ModuleInstall.groovy # Install subcommand (extends CmdBase) +│ ├── ModuleRun.groovy # Run subcommand (extends CmdRun) +│ ├── ModuleList.groovy # List subcommand (extends CmdBase) +│ ├── ModuleRemove.groovy # Remove subcommand (extends CmdBase) +│ ├── ModuleSearch.groovy # Search subcommand (extends CmdBase) +│ ├── ModuleInfo.groovy # Info subcommand (extends CmdBase) +│ └── ModulePublish.groovy # Publish subcommand (extends CmdBase) ├── config/ -│ ├── ConfigBuilder.groovy # MODIFY: Add modules/registry DSL -│ └── parser/v1/ -│ ├── ModulesDsl.groovy # NEW: modules {} block parser -│ └── RegistryDsl.groovy # NEW: registry {} block parser -└── module/ - ├── ModuleResolver.groovy # NEW: Core resolution logic - ├── ModuleStorage.groovy # NEW: Local storage management - ├── ModuleChecksum.groovy # NEW: Checksum verification - ├── ModuleManifest.groovy # NEW: meta.yaml parser - └── HttpModuleRepository.groovy # NEW: Registry HTTP client +│ ├── ModulesConfig.groovy # modules{} config scope +│ └── RegistryConfig.groovy # registry{} config scope (fields: url, apiKey) +├── module/ +│ ├── ModuleReference.groovy # @scope/name parser +│ ├── ModuleResolver.groovy # Core resolution logic (version/integrity/install) +│ ├── ModuleStorage.groovy # Local filesystem operations +│ ├── ModuleRegistryClient.groovy # HTTP registry client +│ ├── ModuleChecksum.groovy # SHA-256 integrity verification +│ ├── ModuleSpec.groovy # Module manifest (meta.yaml) entity +│ ├── InstalledModule.groovy # Installed module entity +│ └── DefaultRemoteModuleResolver.groovy # SPI impl: bridges DSL parser → ModuleResolver +└── pipeline/ + └── PipelineSpec.groovy # nextflow_spec.json read/write modules/nf-lang/src/main/java/nextflow/script/ -└── ResolveIncludeVisitor.java # MODIFY: Add @scope/name detection +└── control/ResolveIncludeVisitor.java # MODIFIED: Delegates @scope/name to SPI resolver + +modules/nf-lang/src/main/java/nextflow/module/spi/ +├── RemoteModuleResolver.java # SPI interface (extensible by plugins) +├── RemoteModuleResolverProvider.java # ServiceLoader wrapper (singleton) +└── FallbackRemoteModuleResolver.java # Error fallback when no impl found + modules/nextflow/src/test/groovy/nextflow/ -├── cli/ -│ └── CmdModuleTest.groovy # NEW: CLI unit tests -├── config/ -│ └── ModulesDslTest.groovy # NEW: Config parsing tests +├── cli/module/ +│ ├── ModuleInstallTest.groovy +│ ├── ModuleRunTest.groovy +│ └── [other subcommand tests] └── module/ - ├── ModuleResolverTest.groovy # NEW: Resolution logic tests - ├── ModuleStorageTest.groovy # NEW: Storage tests - └── ModuleChecksumTest.groovy # NEW: Checksum tests - -tests/ -└── modules/ # NEW: Integration tests - ├── install-module.nf # Test module install + include - ├── version-resolution.nf # Test version management - └── checksum-protection.nf # Test local modification protection + ├── ModuleResolverTest.groovy + ├── ModuleStorageTest.groovy + └── [other module tests] + +tests/modules/ +├── install-module.nf # Integration tests +├── run-module.nf +└── [other integration tests] +``` + +**Structure Decision**: Implementation extends existing Nextflow core modules following modular architecture. New code in `modules/nextflow` for CLI and core logic. DSL parser extension in `modules/nf-lang` via SPI. No new plugins required. + +## Architecture Notes + +### Remote Module Inclusion — SPI Pattern + +The DSL parser (`ResolveIncludeVisitor`) detects the `@` prefix in `include` statements and delegates resolution to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader`. This keeps `nf-lang` decoupled from the runtime module resolution logic: + +``` +include { X } from '@nf-core/fastqc' + ↓ +ResolveIncludeVisitor (nf-lang) + source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(...) + ↓ +DefaultRemoteModuleResolver (nextflow module) + auto-installs via ModuleResolver if missing → returns Path to main.nf ``` -**Structure Decision**: Implementation extends existing Nextflow core modules following modular architecture. New code in `modules/nextflow` for CLI and core logic. DSL parser extension in `modules/nf-lang`. No new plugins required. +The `RemoteModuleResolver` interface in `nf-lang` can be overridden by plugins with a higher priority value. ## Complexity Tracking diff --git a/specs/251117-module-system/quickstart.md b/specs/251117-module-system/quickstart.md index f9247dcb74..71e356e534 100644 --- a/specs/251117-module-system/quickstart.md +++ b/specs/251117-module-system/quickstart.md @@ -22,7 +22,7 @@ nextflow module install nf-core/fastqc nextflow module install nf-core/fastqc -version 1.0.0 ``` -This downloads the module to `modules/@nf-core/fastqc/` and updates `nextflow.config`. +This downloads the module to `modules/@nf-core/fastqc/` and updates `nextflow_spec.json` with the installed version. ### Use in your workflow @@ -52,11 +52,8 @@ Execute a module without writing a wrapper workflow: # Basic usage nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -# With tool arguments -nextflow module run nf-core/bwa-align \ - --reads 'samples/*_{1,2}.fastq.gz' \ - --reference genome.fa \ - --tools:bwa:K 100000000 +# Run specific version +nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0 # With Nextflow options nextflow module run nf-core/salmon \ @@ -66,18 +63,43 @@ nextflow module run nf-core/salmon \ -resume ``` +## 3. View Module Information + +```bash +# Show module metadata and a generated usage template +nextflow module info nf-core/fastqc + +# Show a specific version +nextflow module info nf-core/fastqc -version 1.0.0 + +# JSON output for scripting +nextflow module info nf-core/fastqc -json +``` + --- -## 3. Manage Module Versions +## 4. Manage Module Versions -### Configure versions in nextflow.config +### Version tracking -```groovy -// nextflow.config +Module versions are automatically recorded in `nextflow_spec.json` by `nextflow module install`. You can also pin versions manually: + +```json +// nextflow_spec.json +{ + "modules": { + "@nf-core/fastqc": "1.0.0", + "@nf-core/bwa-align": "1.2.0" + } +} +``` + +Alternatively, declare versions in `nextflow.config` (not currently used): + +```nextflow modules { '@nf-core/fastqc' = '1.0.0' '@nf-core/bwa-align' = '1.2.0' - '@nf-core/samtools' = '2.1.0' } ``` @@ -96,17 +118,11 @@ nextflow module list ### Update a module -Change the version in `nextflow.config`, then run your workflow. Nextflow automatically downloads the new version. - -```groovy -modules { - '@nf-core/fastqc' = '1.2.0' // Changed from 1.0.0 -} -``` +Change the version in `nextflow_spec.json` (or `nextflow.config`), then run your workflow. Nextflow automatically downloads the new version. --- -## 4. Search for Modules +## 5. Search for Modules ```bash # Search by keyword @@ -121,69 +137,19 @@ nextflow module search bwa -json --- -## 5. Configure Tool Arguments - -### Define in meta.yaml (module author) - -```yaml -# modules/@nf-core/bwa-align/meta.yaml -tools: - - bwa: - description: BWA aligner - args: - K: - flag: "-K" - type: integer - description: "Process INT input bases in each batch" - Y: - flag: "-Y" - type: boolean - description: "Use soft clipping for supplementary alignments" -``` - -### Configure in nextflow.config (user) - -```groovy -// nextflow.config -process { - withName: 'BWA_ALIGN' { - tools.bwa.args.K = 100000000 - tools.bwa.args.Y = true - } -} -``` - -### Access in script (module author) - -```groovy -// main.nf -process BWA_ALIGN { - script: - """ - bwa mem ${tools.bwa.args} -t $task.cpus $index $reads - """ -} -``` - ---- - ## 6. Work with Private Registries ### Configure authentication -```groovy +```nextflow // nextflow.config registry { // Multiple registries (tried in order) url = [ 'https://private.registry.myorg.com', - 'https://registry.nextflow.io' + 'https://registry.nextflow.io/api' ] - - auth { - 'private.registry.myorg.com' = '${MYORG_TOKEN}' - 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' - } + apiKey = 'MYORG_TOKEN' // Applied to the primary (first) registry only } ``` @@ -255,13 +221,6 @@ nextflow module remove nf-core/fastqc -keep-files ## Common Patterns -### Install all configured modules - -```bash -# Installs all modules listed in nextflow.config -nextflow module install -``` - ### Offline operation Modules are cached locally in `modules/`. Once installed, workflows run without network access. diff --git a/specs/251117-module-system/research.md b/specs/251117-module-system/research.md index 270603da82..d08005fc89 100644 --- a/specs/251117-module-system/research.md +++ b/specs/251117-module-system/research.md @@ -13,28 +13,44 @@ This document captures technical research and decisions for implementing the Nex **Research Question**: How should `nextflow module` CLI commands be implemented? -**Decision**: Follow CmdPlugin pattern with sub-command delegation +**Decision**: JCommander native subcommands — each subcommand extends `CmdBase` directly; no trait needed **Rationale**: -- CmdPlugin.groovy provides proven pattern for multi-action commands -- Uses JCommander `@Parameters` and `@Parameter` annotations -- Sub-commands (install, search, list, remove, publish, run) handled via positional args -- PluginExecAware interface allows plugin extensibility if needed later +- JCommander's subcommand support handles parameter parsing automatically per subcommand +- Each subcommand (install, run, list, remove, search, info, publish) is a separate class extending CmdBase +- `ModuleRun` extends `CmdRun` to reuse pipeline execution logic (PR #6381) +- No custom `ModuleSubCmd` trait needed; cleaner architecture +- `CmdModule` is registered in `Launcher` alongside all other top-level commands -**Reference Implementation**: -``` -Location: modules/nextflow/src/main/groovy/nextflow/cli/CmdPlugin.groovy -Pattern: - - Extends CmdBase - - @Parameters(commandNames = 'module', commandDescription = '...') - - @Parameter(names = ['-h', '--help']) - - args list for sub-command + module name - - run() method dispatches to install(), search(), etc. +**Implemented Pattern**: +```groovy +@Parameters(commandDescription = "Manage Nextflow modules") +class CmdModule extends CmdBase implements UsageAware { + static final List commands = [] + + static { + commands << new ModuleInstall() // extends CmdBase + commands << new ModuleRun() // extends CmdRun + commands << new ModuleList() // extends CmdBase + commands << new ModuleRemove() // extends CmdBase + commands << new ModuleSearch() // extends CmdBase + commands << new ModuleInfo() // extends CmdBase + commands << new ModulePublish() // extends CmdBase + } + + void run() { + final jc = commander() // JCommander with all subcommands registered + jc.parse(args as String[]) + final subcommand = jc.getCommands().get(jc.getParsedCommand()).getObjects()[0] + subcommand.run() + } +} ``` **Alternatives Considered**: -- Separate CmdModuleInstall, CmdModuleSearch classes: Rejected - too many entry points, doesn't match existing patterns -- Plugin-based CLI extension: Rejected - module system is core functionality, not optional +- CmdFs trait pattern: Considered initially; replaced by JCommander native subcommands — simpler and avoids custom parsing +- Separate top-level Cmd classes (CmdModuleInstall, etc.): Rejected — too many entry points +- Plugin-based CLI extension: Rejected — module system is core functionality, not optional --- @@ -42,33 +58,40 @@ Pattern: **Research Question**: How to extend `include` statement parsing for registry modules? -**Decision**: Extend ResolveIncludeVisitor to detect `@` prefix and delegate to ModuleResolver +**Decision**: Extend `ResolveIncludeVisitor` to detect `@` prefix and delegate to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader` **Rationale**: -- IncludeNode already captures source path as string -- Detection: `source.startsWith('@')` distinguishes registry vs local paths -- Resolution happens at parse time (after plugin resolution) per ADR -- Preserves existing local file include behavior +- Keeps `nf-lang` decoupled from runtime module resolution (`nf-lang` has no dependency on `nextflow` module) +- SPI pattern allows plugins or custom implementations to override the default resolver +- Detection: `source.startsWith('@')` distinguishes registry vs local paths — preserves existing include behavior +- Resolution at parse time (after plugin resolution) per ADR -**Reference Implementation**: +**Implemented Architecture**: ``` -Location: modules/nf-lang/src/main/java/nextflow/script/ResolveIncludeVisitor.java -Extension Point: visitInclude() method -Pattern: - 1. Check if source starts with '@' - 2. If yes: call ModuleResolver.resolve(source, configuredVersion) - 3. ModuleResolver returns absolute path to modules/@scope/name/main.nf - 4. Continue with standard include processing +include { X } from '@scope/name' + ↓ +ResolveIncludeVisitor.visitInclude() [nf-lang] + source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(source, baseDir) + ↓ +RemoteModuleResolverProvider [nf-lang] + Java ServiceLoader discovers implementations; picks highest priority + ↓ +DefaultRemoteModuleResolver [nextflow module] + Calls ModuleResolver.installModule(reference, version, autoInstall=true) + Returns Path to modules/@scope/name/main.nf ``` **Key Files**: -- `IncludeNode.java` - AST representation -- `IncludeEntryNode.java` - Individual entries -- `ResolveIncludeVisitor.java` - Visitor for resolution +- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java` — SPI interface +- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java` — ServiceLoader singleton +- `modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java` — error fallback +- `modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java` — MODIFIED +- `modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy` — default impl **Alternatives Considered**: -- New ANTLR grammar token for `@`: Rejected - unnecessary parser complexity -- Dot file marker for local modules: Deferred to Open Questions in ADR +- New ANTLR grammar token for `@`: Rejected — unnecessary parser complexity +- Direct dependency from nf-lang to nextflow module: Rejected — circular dependency risk; SPI decouples cleanly +- Dot file marker for local modules: Deferred in ADR; current impl uses `@` for registry, `.`/`/` for local --- @@ -76,46 +99,75 @@ Pattern: **Research Question**: How to add new config DSL blocks? -**Decision**: Create ModulesDsl and RegistryDsl classes following PluginsDsl pattern +**Decision**: Create ModulesConfig and RegistryConfig classes implementing ConfigScope interface **Rationale**: -- PluginsDsl.groovy provides exact template for DSL block handling -- ConfigBuilder already supports dynamic DSL registration -- Groovy's methodMissing enables clean config syntax +- ConfigScope is an ExtensionPoint (pf4j) that ConfigBuilder automatically discovers +- Classes implementing ConfigScope and annotated with @ScopeName are automatically parsed +- No need to modify ConfigBuilder or create custom DSL parsers +- Pattern used throughout Nextflow: FusionConfig, CondaConfig, DockerConfig, etc. +- Provides type safety via @CompileStatic and validation via @ConfigOption **Reference Implementation**: ``` -Location: modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy +Location: modules/nextflow/src/main/groovy/nextflow/fusion/FusionConfig.groovy Pattern: + @ScopeName("modules") + @Description("Module version declarations") @CompileStatic - class ModulesDsl { - private Map modules = [:] + class ModulesConfig implements ConfigScope { + @ConfigOption + @Description("Module version mappings") + final Map modules = [:] - def methodMissing(String name, args) { - // modules { '@nf-core/fastqc' = '1.0.0' } - modules[name] = args[0].toString() - } + ModulesConfig() {} - Map getModules() { modules } + ModulesConfig(Map opts) { + // Parse from config map + } } ``` -**RegistryDsl Pattern**: +**ConfigScope Interface**: +``` +Location: modules/nf-lang/src/main/java/nextflow/config/spec/ConfigScope.java +public interface ConfigScope extends ExtensionPoint {} +``` + +**RegistryConfig Pattern**: ```groovy -class RegistryDsl { - String url = 'https://registry.nextflow.io' - List urls = [] // For multiple registries - Map auth = [:] - - void url(String value) { this.url = value } - void url(List values) { this.urls = values } - void auth(Closure config) { /* parse auth block */ } +@ScopeName("registry") +@Description("Module registry configuration") +@CompileStatic +class RegistryConfig implements ConfigScope { + static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api' + + @ConfigOption + final Collection url // One or more URLs in priority order + + @ConfigOption + final String apiKey // API key; falls back to NXF_REGISTRY_TOKEN env var + + RegistryConfig() { + url = [DEFAULT_REGISTRY_URL] + apiKey = null + } + + RegistryConfig(Map opts) { + url = opts.url ?: [DEFAULT_REGISTRY_URL] + apiKey = opts.apiKey as String + } + + String getUrl() { url ? url[0] : DEFAULT_REGISTRY_URL } + Collection getAllUrls() { url ?: [DEFAULT_REGISTRY_URL] } + String getApiKey() { apiKey ?: SysEnv.get('NXF_REGISTRY_TOKEN') } } ``` -**Integration Point**: ConfigBuilder.build() instantiates DSL objects +**Integration Point**: ConfigBuilder automatically discovers and parses ConfigScope implementations via ExtensionPoint mechanism **Alternatives Considered**: +- Custom DSL parsers (ModulesDsl/RegistryDsl): Rejected - unnecessary complexity, ConfigScope pattern handles this automatically - JSON/YAML config file: Rejected - inconsistent with Nextflow config style - Dedicated pipeline.yaml: Deferred per ADR Open Questions @@ -169,36 +221,33 @@ POST /api/modules/{name} # Publish (authenticated) **Research Question**: How to handle registry authentication? -**Decision**: Support NXF_REGISTRY_TOKEN env var + registry.auth config block +**Decision**: Support `NXF_REGISTRY_TOKEN` env var + `registry.apiKey` config field **Rationale**: - Environment variable provides CI/CD compatibility -- Config block allows per-registry tokens for private registries -- Follows existing plugin auth patterns +- `apiKey` config field allows explicit token configuration +- Authentication is only applied to the primary (first) registry URL - Bearer token in Authorization header (standard HTTP auth) -**Reference Implementation**: +**Implementation**: ``` -Location: modules/nextflow/src/main/groovy/nextflow/cli/CmdAuth.groovy -Pattern: - 1. Check NXF_REGISTRY_TOKEN environment variable - 2. Fall back to registry.auth.'registry.nextflow.io' in config - 3. Add header: Authorization: Bearer +RegistryConfig.getApiKey() returns: + 1. registry.apiKey config value if set + 2. NXF_REGISTRY_TOKEN environment variable as fallback + 3. null if neither is set (unauthenticated requests) ``` **Config Syntax**: -```groovy +```nextflow registry { - auth { - 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' - 'private.registry.com' = '${PRIVATE_TOKEN}' - } + apiKey = '${NXF_REGISTRY_TOKEN}' } ``` **Alternatives Considered**: +- Per-registry token map (`auth {}` block): Was in initial design; simplified to single `apiKey` since only the primary registry uses authentication - Secrets file (~/.nextflow/secrets.json): Possible future enhancement -- OAuth flow: Rejected for CLI - token-based simpler +- OAuth flow: Rejected for CLI — token-based simpler --- @@ -275,59 +324,7 @@ class ModuleChecksum { ## 8. Tool Arguments Implementation -**Research Question**: How to implement structured tool arguments (`tools..args`)? - -**Decision**: Implement as implicit variable in process scope, validated at parse time - -**Rationale**: -- `tools` variable accessible in script block like `task`, `params` -- Validation at parse time catches errors early (per clarification) -- Schema defined in meta.yaml, parsed by ModuleManifest -- Concatenation logic handles flag formatting - -**Implementation Pattern**: -```groovy -class ToolArgs { - private Map schema // From meta.yaml - private Map values // From config - - String getAt(String argName) { - def def = schema[argName] - def value = values[argName] - if (def.type == 'boolean' && value) { - return def.flag // e.g., "-Y" - } - return "${def.flag} ${value}" // e.g., "-K 100000" - } - - String toString() { - // Concatenate all configured args - values.collect { name, value -> - this[name] - }.join(' ') - } -} -``` - -**Config Access**: -```groovy -withName: 'BWA_MEM' { - tools.bwa.args.K = 100000 - tools.bwa.args.Y = true -} -``` - -**Script Access**: -```groovy -script: -""" -bwa mem ${tools.bwa.args} -t $task.cpus $index $reads -""" -``` - -**Alternatives Considered**: -- Runtime validation only: Rejected - late errors waste compute -- String-only values: Rejected - loses type safety benefits +> **⚠️ REMOVED FROM ADR** — The tool arguments feature (`tools..args` in meta.yaml and process config) was removed from the module system ADR. It is not implemented and not planned in the current scope. The `meta.yaml` format used in the actual implementation (`ModuleSpec`) does not include tool/argument definitions. --- @@ -335,26 +332,21 @@ bwa mem ${tools.bwa.args} -t $task.cpus $index $reads | Area | Decision | Key Reference | |------|----------|---------------| -| CLI | CmdModule extends CmdBase | CmdPlugin.groovy | -| DSL Parser | Extend ResolveIncludeVisitor | ResolveIncludeVisitor.java | -| Config | ModulesDsl + RegistryDsl | PluginsDsl.groovy | -| Registry HTTP | HttpModuleRepository | HttpPluginRepository.groovy | -| Authentication | NXF_REGISTRY_TOKEN + config | CmdAuth.groovy | -| Checksums | SHA-256, .checksum file | Standard Java security | +| CLI | JCommander subcommands; each extends CmdBase (ModuleRun extends CmdRun) | CmdModule.groovy | +| DSL Parser | SPI pattern — ResolveIncludeVisitor delegates to RemoteModuleResolver; DefaultRemoteModuleResolver bridges to ModuleResolver | ResolveIncludeVisitor.java, RemoteModuleResolver.java | +| Config | ModulesConfig + RegistryConfig (ConfigScope) | FusionConfig.groovy, ConfigScope.java | +| Registry HTTP | ModuleRegistryClient using HxClient + npr-api models | HttpPluginRepository.groovy | +| Authentication | `NXF_REGISTRY_TOKEN` env var or `registry.apiKey` config field (primary registry only) | RegistryConfig.groovy | +| Checksums | SHA-256/SHA-512, `.checksum` file, download integrity via X-Checksum header | ModuleChecksum.groovy | +| Version Storage | `nextflow_spec.json` (auto-managed); `modules {}` in nextflow.config (manual alternative) | PipelineSpec.groovy | | Version Syntax | Plugin-compatible constraints | VersionNumber class | -| Tool Args | Implicit variable, parse-time validation | New implementation | +| Tool Args | ~~Implicit variable, parse-time validation~~ — **Removed from ADR** | N/A | --- ## Open Items (Deferred) -These items are noted in the ADR as open questions and do not block implementation: - -1. **Local vs managed module distinction**: Whether local modules use `@` prefix or dot file marker -2. **Tool arguments CLI syntax**: Colon vs dot separator (`--tools:bwa:K` vs `--tools.bwa.K`) -3. **Module version location**: nextflow.config vs dedicated pipeline.yaml - -Current implementation uses: -- `@` prefix for registry modules only (local paths start with `.` or `/`) -- Colon-separated CLI syntax per ADR assumption -- Versions in nextflow.config per ADR decision \ No newline at end of file +1. **Local vs managed module distinction**: Resolved — `@` prefix for registry modules only; local paths start with `.` or `/` +2. **Tool arguments**: Removed from ADR — not in scope +3. **Module version location**: Resolved — `nextflow_spec.json` (auto-managed by `module install`); `modules {}` block in `nextflow.config` supported as alternative +4. **DSL parser `@scope/name` include**: ✅ Resolved — SPI pattern implemented (T017a-d) \ No newline at end of file diff --git a/specs/251117-module-system/spec.md b/specs/251117-module-system/spec.md index 652397bb80..fba59f7923 100644 --- a/specs/251117-module-system/spec.md +++ b/specs/251117-module-system/spec.md @@ -27,7 +27,7 @@ A pipeline developer wants to use a pre-built module from the Nextflow registry **Acceptance Scenarios**: -1. **Given** a new Nextflow project with no modules installed, **When** user runs `nextflow module install nf-core/fastqc`, **Then** the module is downloaded to `modules/@nf-core/fastqc/`, a `.checksum` file is created, and `nextflow.config` is updated with the version +1. **Given** a new Nextflow project with no modules installed, **When** user runs `nextflow module install nf-core/fastqc`, **Then** the module is downloaded to `modules/@nf-core/fastqc/`, a `.checksum` file is created, and `nextflow_spec.json` is updated with the version 2. **Given** a workflow file with `include { FASTQC } from '@nf-core/fastqc'`, **When** user runs `nextflow run main.nf`, **Then** Nextflow resolves the module from local storage and executes the process 3. **Given** a module version declared in `nextflow.config`, **When** user includes the module, **Then** the declared version is used (not latest) @@ -75,7 +75,7 @@ A pipeline developer wants to pin and manage module versions to ensure reproduci **Acceptance Scenarios**: -1. **Given** a module is installed at version 1.0.0, **When** user changes `nextflow.config` to specify version 1.1.0 and runs the workflow, **Then** version 1.1.0 is automatically downloaded and replaces the local copy +1. **Given** a module is installed at version 1.0.0, **When** user changes `nextflow_spec.json` to specify version 1.1.0 and runs the workflow, **Then** version 1.1.0 is automatically downloaded and replaces the local copy 2. **Given** modules installed locally, **When** user runs `nextflow module list`, **Then** configured version, installed version, latest available version, and status are displayed for each module --- @@ -106,7 +106,7 @@ A pipeline developer wants to remove a module they no longer need. **Acceptance Scenarios**: -1. **Given** a module is installed, **When** user runs `nextflow module remove nf-core/fastqc`, **Then** the module directory is deleted and the entry is removed from `nextflow.config` +1. **Given** a module is installed, **When** user runs `nextflow module remove nf-core/fastqc`, **Then** the module directory is deleted and the entry is removed from `nextflow_spec.json` 2. **Given** a module is referenced in workflow files, **When** user runs `nextflow module remove`, **Then** a warning is displayed about the reference but removal proceeds --- @@ -166,7 +166,7 @@ A module author wants to publish their module to the Nextflow registry for other - **FR-001**: System MUST recognize `@scope/name` syntax in `include` statements as registry module references - **FR-002**: System MUST distinguish between local file paths (starting with `.` or `/`) and registry modules (starting with `@`) -- **FR-003**: System MUST resolve module versions from `nextflow.config` `modules {}` block before downloading +- **FR-003**: System MUST resolve module versions from `nextflow_spec.json` before downloading - **FR-004**: System MUST parse and validate `meta.yaml` files for module metadata and dependencies #### Module Resolution @@ -192,12 +192,13 @@ A module author wants to publish their module to the Nextflow registry for other - **FR-017**: System MUST provide `nextflow module remove scope/name` command to delete modules - **FR-018**: System MUST provide `nextflow module publish scope/name` command to upload modules to registry - **FR-019**: System MUST provide `nextflow module run scope/name` command to execute modules directly +- **FR-019b**: System MUST provide `nextflow module info scope/name` command to display module metadata and a usage template #### Configuration -- **FR-020**: System MUST read module versions from `modules {}` block in `nextflow.config` -- **FR-021**: System MUST support `registry {}` block for configuring registry URL and authentication -- **FR-022**: System MUST support `NXF_REGISTRY_TOKEN` environment variable for authentication +- **FR-020**: System MUST persist module versions in `nextflow_spec.json`; MUST also read versions from `modules {}` block in `nextflow.config` as an alternative +- **FR-021**: System MUST support `registry {}` block with `url` and `apiKey` fields for configuring registry URL and authentication +- **FR-022**: System MUST support `NXF_REGISTRY_TOKEN` environment variable as fallback for `registry.apiKey` - **FR-023**: System MUST support multiple registry URLs with fallback ordering #### Module Parameters