Pipeline composition

[bentsherman]

This PR adds an ADR for remote pipeline inclusion, aka "meta-pipelines".

It describes an approach for including remote pipelines into a meta-pipeline in a way that preserves dataflow concurrency between pipeline inputs/outputs.

It discusses alternative approaches such as pipeline chaining / nf-cascade and why they don't satisfy certain use cases (preserving dataflow concurrency).

It also walks through a basic example of fetchngs -> rnaseq.

[ewels]

Great write up, thanks for this Ben!

As you might expect, I'm most concerned about the params. You characterise it as a one-off cost which is mitigated by LLMs, however that doesn't take into account updates to included pipelines (a core functionality with included modules). The params drift with updates would be dangerous and a constant source of dev work.

I'd still love to look into how we could bulk import nested config and apply it at root level. Even if it is a separate import + apply mechanism (eg. like config profiles in a sense?). I think without it, the use of the meta pipeline functionality is substantially limited.

[ewels]

Having unpredictable global scope params blocks is just weird and if we were designed Nextflow today we would never include this behaviour. In other languages, globals need to be used with caution and are generally not advised.

@adamrtalbot agreed, I never said global. I would love it if the pipeline config is imported within a dedicated scope and treated as a baseline default. Then the import-ing pipeline can override anything, but doesn't need to duplicate config that isn't being changed.

Doing this would not be trivial. The only way I can think of is to do something fairly radical like rendering the config at import time and saving that to a locked config file somewhere. Or some other crazy mechanism.

[adamrtalbot]

Having unpredictable global scope params blocks is just weird and if we were designed Nextflow today we would never include this behaviour. In other languages, globals need to be used with caution and are generally not advised.

@adamrtalbot agreed, I never said global. I would love it if the pipeline config is imported within a dedicated scope and treated as a baseline default. Then the import-ing pipeline can override anything, but doesn't need to duplicate config that isn't being changed.

Doing this would not be trivial. The only way I can think of is to do something fairly radical like rendering the config at import time and saving that to a locked config file somewhere. Or some other crazy mechanism.

Config or params? In my mind they are very different concepts, I was referring to parameters here.

[ewels]

Config or params? In my mind they are very different concepts, I was referring to parameters here.

Ideally params, but might need to be config for all the ext stuff..?

Happy to rename the ADR to "remote workflow inclusion" to align with the workflow keyword.

Yeah as it stands I think this basically boils down to the functionality we already have with nf-core subworkflows, right? Which is quite far from what I think of as meta-pipelines. Still good to have and useful..

[bentsherman]

Yeah as it stands I think this basically boils down to the functionality we already have with nf-core subworkflows, right?

Can the nf-core tooling install a workflow from a pipeline repo? e.g. NFCORE_RNASEQ from nf-core/rnaseq? I think that is the main thing that this ADR adds

[edmundmiller]

left a few thoughts on scope/clarity — overall the direction makes sense to me.

[pditommaso]

Note, my previous comment is based on the assumption is focused on sub-workflow only, not plain pipelines.

[bentsherman]

Marking as draft until I finish my deep clean. Example is still valid

[pditommaso]

Thanks Ben, I went through the changes since my last review (07-01). The ADR reads much better framed as pipeline composition, and it's great to see it running end to end with the example. Most of my earlier points are addressed or out of scope now: the spec/lockfile question went away with remote inclusion, and the binding rules are concrete in ParamsHelper. Config is still the open part, and I'll come back to it.

Here's what I found:

1. Including a pipeline from two scripts crashes when the path contains a symlink.
IncludeDef.runModuleV2(path, session) is @Memoized on the raw Path, but ScriptMeta.getScriptByPath falls back to toRealPath(). So /tmp/x/greet.nf and /private/tmp/x/greet.nf find the same BaseScript under two different cache keys. The script runs twice and fails with Workflow params definition must be defined before the entry workflow. That's why two PipelineCompositionTest cases fail on macOS while CI on Linux is green. It's realistic on HPC, where home and scratch directories are often symlinked. Repro: main.nf includes greet.nf and mid.nf, mid.nf also includes greet.nf, and you run it through a symlinked path.

2. Renaming "_NAMED_PARAM" to ASTNodeMarker.NAMED_PARAM breaks the language server.
LanguageServerASTUtils.java:81 in nextflow-io/language-server reads the string key "_NAMED_PARAM", so go-to-definition on named arguments would silently stop working. Nothing in this repo reads the key, so I'd revert the rename.

3. Stale docs.
docs/reference/syntax.mdx:91 still says the params and output blocks of an included pipeline can be included and must be aliased. Output inclusion was dropped in v1.4, and the compiler only handles params.

4. --rnaseq.input is resolved and then silently thrown away.
When the user gives a value for a param that the meta-pipeline overrides through dataflow, the value is still loaded eagerly: the samplesheet is parsed and its files are checked, and then it's replaced. If a file is missing, the run fails on a value that was going to be discarded anyway. If the file is valid, the value is ignored without a word. v1.1 mentioned warning about these phantom inputs, and I think that's worth bringing back, or rejecting the value outright.

5. Unrelated behaviour change: params outside the entry workflow.
VariableScopeVisitor promotes this from a paranoid warning to a regular warning. nextflow lint and the language server will now flag every such use in existing pipelines, and nf-core subworkflows have a lot of them. I'd move this to its own PR so it gets its own discussion.

6. Channel<E> params can only be loaded from a samplesheet.
The element type must be a map, a record or a record type, so something like Channel<Path> from a glob (--reads '*.fq') can't be given on the CLI. That's fine for a first version, but the ADR and docs should state the limit.

Minor

• BaseScript: the new -entry + params block guard can never be hit, because ScriptLoaderV2 already rejects -entry with the strict parser (removed in Simplify pipeline composition implementation #7716).
• The public Session.outputs field is removed. Nothing in this repo uses it, but it's worth a changelog line in case plugins do.

I'll follow up with a separate review focused on the composition semantics.

[bentsherman]

Thanks Paolo. Going through your points:

1. Symlinked paths: already fixed

2. NAMED_PARAM: see my reply on #7716 and the sibling language-server PR

3. Stale docs: fixed

4. Overridden params: agreed that an error would be better than silently dropping the value. I tried it, and it isn't a quick fix. I added my findings to the ADR as an open question, so we can look into it later if it becomes a common issue.

5. params outside the entry workflow: it's back to a paranoid warning.

6. Channel<E> params from the CLI: this behavior comes from the named workflow execution ADR and it is already in the Nextflow docs.

[pditommaso]

Thanks Ben, all my points are addressed. I checked the changes on macOS: the composition tests pass, the example runs end to end, and an output index CSV now loads fine as a Channel param.

A few small things left, none of them blocking:

• Merge order with the language server: nextflow-io/language-server#183 needs to land together with the nf-lang bump, otherwise go-to-definition on named arguments would silently stop working.
• README: "--rnaseq.input is ignored" is only half true, because an invalid value still fails the run. Maybe point to the ADR open question instead.
• Standalone runs: running ./pipelines/nf-core/rnaseq from the example root also loads the meta-pipeline's nextflow.config (the run shows as fetchngs-rnaseq). It's normal Nextflow behaviour, but it may be worth a note next to "it can still be run on its own".
• CSV index (pre-existing): CsvWriter quotes every value but doesn't escape an embedded ", so such values won't load back. Could be a follow-up.

Thanks again for all the work on this!