Take the input samplesheet as a validated row channel for pipeline composition

[pinin4fjords]

Proof of concept for including rnaseq in another pipeline with nextflow#7213 (pipeline composition), without weakening input validation. Targets output-records (#1945) and contains only what composition needs; #1945 stays the modernisation (records, typing, output block). This is not a statement that nf-core/rnaseq supports composition: the standards below need agreeing first.

What changes

• Input is a validated row channel. input is Channel<SampleRow>: loaded from the CSV on the command line, or given as a channel by an including pipeline. Rows are validated against assets/schema_input.json with nf-schema validate() and grouped per sample. The auto-strandedness flag, MultiQC paired-end ids and the SummarizedExperiment colData are derived from the rows. Rows are sorted by path before the runs of a sample are merged, because the channel has no fixed order.
• Params are passed in. NFCORE_RNASEQ, RNASEQ, PIPELINE_INITIALISATION and PIPELINE_COMPLETION take params as an input, params that had defaults only in nextflow.config get them in the params block, assets and the schema are found through moduleDir, and conf/ is split into params.config (config-only params) and modules.config, with alias-independent process selectors.
• The GTF and the primary quantification are returned to an including pipeline.
• Tool arguments travel on the records (the part that needs the nf-core discussion). Process config closures read global params, which an including pipeline does not share. Arguments that follow from a param are built in the workflow (tool_args.nf), collected in one ToolArgs record and attached as args to each module's input record; ext.args replaces them. The affected modules get their own input record types.
• nf-schema and CI. Pins a patched 2.7.2-channel.5 (Validate and print the value of Channel and Value params instead of hanging nextflow-io/nf-schema#230, Major overhaul of Salmon requirements #232), because validateParameters blocks on Channel params. CI sets NXF_PLUGINS_TEST_REPOSITORY and builds Nextflow from a master commit with #7213.

Status

• Pipeline tests and local module and subworkflow tests pass against the existing snapshots. nextflow lint has no errors.
• Included in a meta-pipeline (STAR and Salmon), it runs end to end and returns its outputs.
• nf-core pipelines lint does not run here or on Migrate to Nextflow workflow outputs via per-stage result records #1945 (KeyError on record imports, Skip included names that are not installed components when recreating installed_by tools#4491). Template questions for typed pipelines: Typed pipelines and pipeline composition: lint and template changes tools#4493.

Notes

• HISAT2_BUILD takes its memory threshold as whole GB, converted in the workflow.
• conf/test.config seeds only the rRNA removal alignment.

🤖 Generated with Claude Code

[github-actions]

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 4.0.3.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the Synchronisation documentation.

[github-actions]

❌ nf-test failed with latest Nextflow version

Note

Tests with Nextflow's latest version failed but it will not cause a CI workflow failure.
Please check if the failure is expected with newer (edge-)releases of Nextflow or if it needs fixing.

• ❌ docker | latest-everything | Shard 18/20

See the full run for details.

[bentsherman]

Comparing this PR to my approach with methylseq (nf-core/methylseq#626) and what I plan to do, since this PR goes beyond my methylseq PR in some ways.

Params and input. Taking params.input as a Channel<SampleRow> looks right. Passing params into NFCORE_RNASEQ looks right. Consider collapsing NFCORE_RNASEQ into the entry workflow to reduce boilerplate. I believe the PIPELINE_* workflows are absorbed by the nf-core-utils plugin.

Tool args. We used the same pattern: add ext settings to process inputs, allow runtime override with task.ext.args ?: <args input> ?: ''. We both extracted the nastier tool args to helper functions to avoid bloating the workflow logic. I first build an options map and then render it to a string with a small cli() helper, which avoids string parsing. I don't have a strong opinion on the ToolArgs record vs passing tool args separately.

Module config. conf/modules/ keeps the settings that don't depend on params. I went ahead and moved everything into pipeline code so that I could delete conf/modules/ entirely. I recommend you do the same, since it makes the pipeline easier to compose.

Smaller things:

• Params block should go after includes
• Boolean params default to false so you don't need Boolean = false
• The params.config seems like an unnecessary level of indirection. Nevermind, I see now it is used by the meta-pipeline
• Wrapping process inputs with channel.value() is unnecessary

[pinin4fjords]

Thanks for that @bentsherman ! The reason for the similarity is that I shamelessly stole some of your work once I realised the issues with params scopes etc and and my existing arg handling.

That said, the args handling will be a LOT more contentious in nf-core, I'm a bit more nervous of this one.