diff --git a/adr/20260310-typed-workflows.md b/adr/20260310-typed-workflows.md index b02aa6be72..36fd37b7fd 100644 --- a/adr/20260310-typed-workflows.md +++ b/adr/20260310-typed-workflows.md @@ -431,7 +431,7 @@ process SALMON { This syntax has several benefits: - The `map` operation is explicit, making process calls consistent with other operator logic -- The process call matches the process definition, accepting and returning regular values rather than channels or dataflow values +- The process call matches the process definition, accepting and returning plain values rather than channels or dataflow values - Processes can be chained without needing a pipe syntax (e.g. `ch.map(FOO).map(BAR).map(BAZ) ...`) - Processes could be composed with other operators (e.g. an iterative process with the `reduce` operator) - The closure around the process call can be used to handle process inputs and outputs without additional operator calls @@ -528,13 +528,13 @@ vals.subscribe { _vals -> } ``` -It frustrates many users that dataflow values don't quite work like regular values, even though it seems like they should. +It frustrates many users that dataflow values don't quite work like plain values, even though it seems like they should. -A solution could be to make dataflow values *implicit*, so that users write them like regular values (i.e. the first example above) and the compiler translates that code into explicit dataflow logic (i.e. the second example). +A solution could be to make dataflow values *implicit*, so that users write them like plain values (i.e. the first example above) and the compiler translates that code into explicit dataflow logic (i.e. the second example). To do this, the compiler would need to: -1. distinguish implicit dataflow values from regular values via type inference (e.g. the result of a `collect` operator), +1. distinguish implicit dataflow values from plain values via type inference (e.g. the result of a `collect` operator), 2. wrap downstream code in `map` and/or `subscribe` operators as needed to produce the desired dataflow logic. @@ -552,4 +552,4 @@ Type checking has also solved most of the problems that motivated this idea: - While users still can't use a dataflow value in an `if` statement, they can get clear and early feedback on whether their code is valid, which is what ultimately matters. -- Being transparent about regular values vs dataflow values in the language may be for the best anyway. It gives a clear picture of how things work "under the hood", and it is still far simpler than the async programming models used by most languages. +- Being transparent about plain values vs dataflow values in the language may be for the best anyway. It gives a clear picture of how things work "under the hood", and it is still far simpler than the async programming models used by most languages. diff --git a/docs/migrations/dsl1.mdx b/docs/migrations/dsl1.mdx index ff682892d2..ba5954e08c 100644 --- a/docs/migrations/dsl1.mdx +++ b/docs/migrations/dsl1.mdx @@ -105,10 +105,6 @@ In DSL1, the entire Nextflow pipeline must be defined in a single file. For exam DSL2 introduces the ability for scripts to *include* other scripts. While splitting a pipeline into multiple scripts is not mandatory in DSL2, it is useful for organizing a large pipeline and reusing modules created by others. See [Modules][module-page] for more information. -:::note -DSL2 scripts cannot exceed 64 KB in size. Split large DSL1 scripts into multiple smaller scripts to avoid this limit. -::: - ## Deprecations ### Processes diff --git a/docs/process-typed.mdx b/docs/process-typed.mdx index 56cf9321c7..37cb00e3f9 100644 --- a/docs/process-typed.mdx +++ b/docs/process-typed.mdx @@ -292,7 +292,7 @@ process foo { ### Structured outputs -Whereas legacy process outputs could only be structured using specific qualifiers like `val` and `tuple`, typed process outputs are regular values. +Whereas legacy process outputs could only be structured using specific qualifiers like `val` and `tuple`, typed process outputs are plain values. The `record()` standard library function can be used to create a record: diff --git a/docs/process.mdx b/docs/process.mdx index f2e7ac437f..018ce02ab4 100644 --- a/docs/process.mdx +++ b/docs/process.mdx @@ -1104,12 +1104,12 @@ While this option can be used with any process output, it cannot be applied to i ### Singleton outputs -When a process is only supplied with dataflow values, regular values, or no inputs, it returns outputs as dataflow values. For example: +When a process is only supplied with dataflow values, plain values, or no inputs, it returns outputs as dataflow values. For example: ```nextflow file=./snippets/process-out-singleton.nf ``` -In the above example, the `echo` process is invoked with a regular value that is wrapped in a dataflow value. As a result, `echo` returns a dataflow value and `greet` is executed three times. +In the above example, the `echo` process is invoked with a plain value that is wrapped in a dataflow value. As a result, `echo` returns a dataflow value and `greet` is executed three times. If the call to `echo` was changed to `echo( channel.of('hello') )`, the process would instead return a channel, and `greet` would be executed only once. diff --git a/docs/script.mdx b/docs/script.mdx index 263f3eb56f..de584a1b47 100644 --- a/docs/script.mdx +++ b/docs/script.mdx @@ -13,10 +13,6 @@ This page is a practical introduction to the Nextflow language. See [Syntax][syn Nextflow uses UTF-8 as the default character encoding for source files. Make sure to use UTF-8 encoding when editing Nextflow scripts with your preferred text editor. ::: -:::warning -Nextflow scripts have a maximum size of 64 KiB. To avoid this limit for large pipelines, consider moving pipeline components into separate files and including them in the main script. -::: - ## Hello world You can use the `println` function to print to the console: @@ -264,7 +260,7 @@ See [Regex operators][semantics-regex] for more information. ## Closures -A closure is a function that can be used like a regular value. Typically, closures are passed as arguments to *higher-order functions* to express computations in a declarative manner. +A closure is a function that can be used like a plain value. Typically, closures are passed as arguments to *higher-order functions* to express computations in a declarative manner. For example, the following closure takes one parameter named `v` and returns the *square* of `v`: diff --git a/docs/workflow-typed.mdx b/docs/workflow-typed.mdx index 20e247dd8d..ecf33f8e83 100644 --- a/docs/workflow-typed.mdx +++ b/docs/workflow-typed.mdx @@ -38,7 +38,7 @@ Channels and values are modeled using the dataflow types: The element type `V` can be any [standard type][stdlib-types], such as `Path`, `String`, or a record. -Workflow inputs can be channels, dataflow values, or regular values. Workflow outputs can be channels or dataflow values. +Workflow inputs can be channels, dataflow values, or plain values. Workflow outputs can be channels or dataflow values. ## Restricted syntax diff --git a/modules/nf-lang/src/main/java/nextflow/script/ast/ASTUtils.java b/modules/nf-lang/src/main/java/nextflow/script/ast/ASTUtils.java index 60496552e8..9dcd11e0cd 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/ast/ASTUtils.java +++ b/modules/nf-lang/src/main/java/nextflow/script/ast/ASTUtils.java @@ -193,6 +193,16 @@ public static VariableExpression asVarX(Expression expression) { return expression instanceof VariableExpression ve ? ve : null; } + public static String methodType(MethodNode node) { + if( node instanceof ProcessNode ) + return "Process"; + if( node instanceof AgentNode ) + return "Agent"; + if( node instanceof WorkflowNode ) + return "Workflow"; + return "Function"; + } + /** * Given a variable which represents a method being accessed * as a variable, return the underlying method. diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/TypeCheckingVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/control/TypeCheckingVisitor.java index 8c8ca867c9..56a64d1445 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/TypeCheckingVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/TypeCheckingVisitor.java @@ -1040,16 +1040,6 @@ private static VariableExpression outputTarget(Expression output) { return null; } - private static String methodType(MethodNode node) { - if( node instanceof ProcessNode ) - return "Process"; - if( node instanceof AgentNode ) - return "Agent"; - if( node instanceof WorkflowNode ) - return "Workflow"; - return "Function"; - } - private static String className(Expression node) { var receiverType = getType(node); return receiverType != null && receiverType.implementsInterface(ClassHelper.makeCached(Namespace.class)) @@ -1527,12 +1517,6 @@ private boolean checkRecordCast(ClassNode targetType, ClassNode sourceType, ASTN public void visitPropertyExpression(PropertyExpression node) { super.visitPropertyExpression(node); - var mn = asMethodOutput(node); - if( mn instanceof ProcessNode || mn instanceof WorkflowNode ) { - addError("Using the `.out` property to access process/workflow outputs is not supported with static typing -- assign the output to a variable instead", node); - return; - } - var receiver = node.getObjectExpression(); var receiverType = getType(receiver); if( ClassHelper.isDynamicTyped(receiverType) ) diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/VariableScopeVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/control/VariableScopeVisitor.java index 9f9af5f921..1763f445f7 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/VariableScopeVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/VariableScopeVisitor.java @@ -872,10 +872,19 @@ else if( isStdinStdout(name) ) { } if( variable != null ) { checkGlobalVariableInProcess(variable, node); + checkDataflowMethodAsVariable(variable, node); node.setAccessedVariable(variable); } } + private void checkDataflowMethodAsVariable(Variable variable, ASTNode context) { + if( !typingEnabled ) + return; + var mn = asMethodVariable(variable); + if( mn instanceof ProcessNode || mn instanceof WorkflowNode || mn instanceof AgentNode ) + vsc.addError(methodType(mn) + " `" + variable.getName() + "` cannot be used as a variable", context); + } + private boolean isStdinStdout(String name) { var classScope = currentScope().getClassScope(); if( classScope != null ) { diff --git a/modules/nf-lang/src/main/java/nextflow/script/parser/DescriptiveErrorStrategy.java b/modules/nf-lang/src/main/java/nextflow/script/parser/DescriptiveErrorStrategy.java index ec185f119e..0fb4a3652f 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/parser/DescriptiveErrorStrategy.java +++ b/modules/nf-lang/src/main/java/nextflow/script/parser/DescriptiveErrorStrategy.java @@ -71,19 +71,18 @@ public Token recoverInline(Parser recognizer) public void reportError(Parser recognizer, RecognitionException e) { } + private static final String EOF_ERROR_MESSAGE = "Unexpected end of file -- check for an unclosed brace, bracket, parenthesis, or string"; + protected String createNoViableAlternativeErrorMessage(Parser recognizer, NoViableAltException e) { - TokenStream tokens = recognizer.getInputStream(); - String input; - if (tokens != null) { - if (e.getStartToken().getType() == Token.EOF) { - input = ""; - } else { - input = e.getOffendingToken().getText(); - } - } else { - input = ""; + if (e.getOffendingToken().getType() == Token.EOF) { + return EOF_ERROR_MESSAGE; } + TokenStream tokens = recognizer.getInputStream(); + String input = tokens != null + ? e.getOffendingToken().getText() + : ""; + return "Unexpected input: " + escapeWSAndQuote(input); } @@ -96,6 +95,10 @@ protected void reportNoViableAlternative(Parser recognizer, protected String createInputMismatchErrorMessage(Parser recognizer, InputMismatchException e) { + if (e.getOffendingToken().getType() == Token.EOF) { + return EOF_ERROR_MESSAGE; + } + return "Unexpected input: " + getTokenErrorDisplay(e.getOffendingToken()); } diff --git a/modules/nf-lang/src/test/groovy/nextflow/config/parser/ConfigAstBuilderTest.groovy b/modules/nf-lang/src/test/groovy/nextflow/config/parser/ConfigAstBuilderTest.groovy index c917147abb..671b6deee7 100644 --- a/modules/nf-lang/src/test/groovy/nextflow/config/parser/ConfigAstBuilderTest.groovy +++ b/modules/nf-lang/src/test/groovy/nextflow/config/parser/ConfigAstBuilderTest.groovy @@ -139,6 +139,26 @@ class ConfigAstBuilderTest extends Specification { errors[0].getOriginalMessage() == "Variable declarations cannot be mixed with config statements" } + def 'should report an error for unexpected end of file' () { + when: + def errors = check( + '''\ + process { + withName: 'FOO' { + ext.args = '--foo' + + withName: 'BAR' { + ext.args = '--bar' + } + } + ''' + ) + then: + errors.size() == 1 + errors[0].getStartLine() == 9 + errors[0].getOriginalMessage() == 'Unexpected end of file -- check for an unclosed brace, bracket, parenthesis, or string' + } + def 'should report an invalid escape sequence at the exact position' () { when: def errors = check( diff --git a/modules/nf-lang/src/test/groovy/nextflow/script/control/ScriptResolveTest.groovy b/modules/nf-lang/src/test/groovy/nextflow/script/control/ScriptResolveTest.groovy index 0cbd67e2da..a58cd174db 100644 --- a/modules/nf-lang/src/test/groovy/nextflow/script/control/ScriptResolveTest.groovy +++ b/modules/nf-lang/src/test/groovy/nextflow/script/control/ScriptResolveTest.groovy @@ -338,6 +338,79 @@ class ScriptResolveTest extends Specification { errors[0].getOriginalMessage() == '`x` is not defined' } + def 'should report an error when a process or workflow is used as a variable in a typed script' () { + when: + def errors = check( + '''\ + nextflow.enable.types = true + + process hello { + exec: + true + } + + workflow greet { + } + + workflow { + channel.empty().mix(hello) + def x = greet + } + ''' + ) + then: + errors.size() == 2 + errors[0].getStartLine() == 12 + errors[0].getStartColumn() == 25 + errors[0].getOriginalMessage() == 'Process `hello` cannot be used as a variable' + errors[1].getStartLine() == 13 + errors[1].getStartColumn() == 13 + errors[1].getOriginalMessage() == 'Workflow `greet` cannot be used as a variable' + + when: + errors = check( + '''\ + process hello { + exec: + true + } + + workflow { + channel.empty().mix(hello) + } + ''' + ) + then: + errors.size() == 0 + } + + def 'should report an error for process .out property in a typed script' () { + when: + def errors = check( + '''\ + nextflow.enable.types = true + + process hello { + output: + stdout() + + script: + '' + } + + workflow { + hello() + hello.out + } + ''' + ) + then: + errors.size() == 1 + errors[0].getStartLine() == 13 + errors[0].getStartColumn() == 5 + errors[0].getOriginalMessage() == 'Process `hello` cannot be used as a variable' + } + def 'should report an error for an undefined function' () { when: def errors = check( diff --git a/modules/nf-lang/src/test/groovy/nextflow/script/parser/ScriptAstBuilderTest.groovy b/modules/nf-lang/src/test/groovy/nextflow/script/parser/ScriptAstBuilderTest.groovy index b5446345d9..ce12a74396 100644 --- a/modules/nf-lang/src/test/groovy/nextflow/script/parser/ScriptAstBuilderTest.groovy +++ b/modules/nf-lang/src/test/groovy/nextflow/script/parser/ScriptAstBuilderTest.groovy @@ -139,6 +139,47 @@ class ScriptAstBuilderTest extends Specification { errors[0].getOriginalMessage() == "Unexpected input: ','" } + def 'should report an error for unexpected end of file' () { + when: + def errors = check( + '''\ + workflow { + if( true ) { + println 'hello' + } + ''' + ) + then: + errors.size() == 1 + errors[0].getStartLine() == 5 + errors[0].getOriginalMessage() == 'Unexpected end of file -- check for an unclosed brace, bracket, parenthesis, or string' + + when: + errors = check( + '''\ + workflow { + println 'hello' + } + x = [1, 2 + ''' + ) + then: + errors.size() == 1 + errors[0].getOriginalMessage() == 'Unexpected end of file -- check for an unclosed brace, bracket, parenthesis, or string' + + when: + errors = check( + '''\ + workflow { + println """hello + } + ''' + ) + then: + errors.size() == 1 + errors[0].getOriginalMessage() == 'Unexpected end of file -- check for an unclosed brace, bracket, parenthesis, or string' + } + def 'should report the syntax error in an invalid definition' () { when: def errors = check( diff --git a/modules/nf-lang/src/test/groovy/nextflow/script/types/TypeCheckingTest.groovy b/modules/nf-lang/src/test/groovy/nextflow/script/types/TypeCheckingTest.groovy index 6a225f085c..9115c36e22 100644 --- a/modules/nf-lang/src/test/groovy/nextflow/script/types/TypeCheckingTest.groovy +++ b/modules/nf-lang/src/test/groovy/nextflow/script/types/TypeCheckingTest.groovy @@ -1494,29 +1494,6 @@ class TypeCheckingTest extends Specification { Types.getName(type) == 'Value' } - def 'should report error for process .out property' () { - expect: - check( - ''' - nextflow.enable.types = true - - process hello { - output: - stdout() - - script: - '' - } - - workflow { - hello() - hello.out - } - ''', - 'Using the `.out` property to access process/workflow outputs is not supported with static typing -- assign the output to a variable instead' - ) - } - def 'should not allow a void call result to be assigned to a variable' () { expect: check(