Skip to content

Support pipeline composition - #183

Merged
bentsherman merged 4 commits into
mainfrom
pipeline-composition
Sep 30, 2026
Merged

bentsherman merged 4 commits into
mainfrom
pipeline-composition

Conversation

@bentsherman

@bentsherman bentsherman commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

Language server support for pipeline composition (nextflow-io/nextflow#7213): a pipeline -- the params / workflow / output trio of a script -- can be included as a named workflow.

include {
    params as RnaseqParams ;
    workflow as NFCORE_RNASEQ
} from './pipelines/nf-core/rnaseq'

workflow {
    main:
    rnaseq = NFCORE_RNASEQ( params.rnaseq )
    ...
}

Important

Draft: this does not build yet. It needs two nf-lang additions from nextflow-io/nextflow#7213 that are not released. See Release ordering.

The suite here is green only against a local nf-lang built from that branch, resolved via mavenLocal() over the pinned 26.08.0-edge. That build is also ahead of the v26.08.0-edge tag by the twelve nf-lang commits merged to master since, including #7557 and #7342. This branch has never been built against the released artifact, and cannot be until nf-lang is cut -- the two hooks are missing from it.

Include resolution itself comes for free -- the language server runs nf-lang's own ResolveIncludeVisitor -- so this PR covers only what the language server implements itself.

Changes

Calling a pipeline (TypeCheckingVisitorEx)

A pipeline declares its inputs with a params block rather than a take: section, so its entry workflow has no parameters. Every pipeline call therefore reported Workflow `NFCORE_RNASEQ` expects 0 argument(s) but received 1, and the call's return type was inferred as void.

checkPipelineCall handles the call against the pipeline instead:

  • Named arguments are checked against the params block -- unknown param, missing required param, and argument type. A param is required when it has no default and is not nullable.
  • The return type is a record of the output declarations, matching the record the pipeline returns at runtime, so rnaseq.bams resolves and rnaseq.counts is reported.
  • A single non-map argument (NFCORE_RNASEQ( params.rnaseq + record(input: ch) )) is left to the runtime params validation, rather than modelling record arithmetic here.

Named arguments also get _NAMED_PARAM metadata, so go-to-definition on one lands on the param declaration.

Navigation and hover

  • The params and output blocks of an included pipeline are record types synthesized for the include. They are not in any file's AST, so they have no URI and go-to-definition silently did nothing. getDefinition now redirects such a type to the block it was synthesized from.

  • An entry workflow rendered as workflow <entry> on hover. It is now rendered as the pipeline it belongs to, which is the signature a caller needs:

    pipeline {
        params:
        input: Path
        aligner: String = 'star'
    
        output:
        bams: Channel<Path>
    }
    

Null-name guards

An included pipeline is a WorkflowNode with no name of its own. Two places assumed otherwise:

  • WorkspacePreviewProvider.children passed the name straight to Map.of, which rejects null values.
  • DataflowVisitor.visitPipeline labelled a piped call with it, producing an unlabelled DAG node. It now uses the alias.

Tests

ScriptPipelineCompositionTest, six cases: no diagnostics for a full three-block include, definition of an included params block, pipeline hover, unknown param, missing param, and output type inference. Full suite green.

Not included

  • Completion inside include { ... } for workflow, params and output. There is no completion there at all today, so nothing regressed, but it is where these are least discoverable.
  • Go-to-definition on a type annotation. Parameter types are not indexed by ASTParentVisitor, so this has never worked for any record type -- not specific to pipeline composition.
  • Workspace symbols still show every pipeline as workflow <entry>.

Release ordering

  1. Pipeline composition nextflow#7213 merges -- it adds the two hooks this PR needs:
    • ResolveIncludeVisitor.getPipelineBlock(ClassNode), for navigating to an included params/output block.
    • ScriptNode.getPipeline(WorkflowNode), for resolving a pipeline's blocks from a call to it.
  2. nf-lang release
  3. Bump io.nextflow:nf-lang here, mark this PR ready
  4. Language server release

A pipeline -- the params / workflow / output trio of a script -- can be
included as a named workflow. Navigate to the params or output block of
the module for an include of one, and render an entry workflow as the
pipeline it belongs to on hover.

An included pipeline has no name of its own, which the workspace preview
and the DAG preview assumed.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman
bentsherman marked this pull request as ready for review September 30, 2026 17:39
Render an included pipeline with its alias on hover, and drop the
navigation to the params block of an included pipeline.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman
bentsherman merged commit 4742ba4 into main Sep 30, 2026
2 checks passed
@bentsherman
bentsherman deleted the pipeline-composition branch September 30, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant