Repository navigation
Make the pipeline a typed pipeline that can be included in another pipeline - #758
Draft
pinin4fjords wants to merge 19 commits into
Draft
pinin4fjords wants to merge 19 commits into
pinin4fjords wants to merge 19 commits into
Conversation
…pipeline, not projectDir [skip ci] In an including pipeline projectDir is the including pipeline's project directory, so the nextflow_schema.json used to validate the paramsets, the paramsheet include files and the schema used for the help text were not found. pipelineDir() resolves the pipeline root from the module directory instead. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…p ci] A pipeline that includes this one does not load nextflow.config, so it had no way to get the defaults that the workflow and the process config read. nextflow.config includes the new file at the same place; the resolved config of a standalone run is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…his one [skip ci] buildParamset() builds the paramset of a run from the pipeline params with the given params replacing them, for a pipeline that takes the input files (matrix, feature lengths, annotation) from its own dataflow, where they cannot be params at launch. The default paramset only carries the params that the schema declares, so the params of an including pipeline do not end up in it, and Path values are given to the schema validator as strings. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Processes serialise the paramset, which cannot handle Path objects, so the files that buildParamset takes are converted to strings, as they are when given on the command line. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…her pipeline [skip ci] main.nf declares the params (generated from the schema; the files of a run, input, contrasts, matrix, feature_length_matrix and gtf, are optional values) and an output block, and its entry workflow builds the paramset of a run from the params. The paramsheet stays an optional mode. The params of the pipeline are passed to the init subworkflow and the paramset helpers instead of being read as globals, the default paramset only carries the declared params, params that the schema allows to be integer or boolean are converted from strings, and the params dump skips dataflow values and writes files as strings. The published layout is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
pinin4fjords
changed the title
…skip ci] The affy, soft, maxquant, generic_matrix and propd profiles set it to null so the pipeline guesses which assays to log, but the typed param was not nullable, so those profiles failed at launch. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…atched nf-schema, deploy output via outputDir [skip ci] The typed pipeline needs nextflow-io/nextflow#7213, #7646 and #7674, which are not in a release yet, so the nf-test jobs run a launcher built from a pinned upstream commit, and the patched nf-schema comes from a plugin repository. The AWS test workflows set the output location with the outputDir config setting, since the outdir param is gone. nf-test no longer lists bin/ as a trigger and lists conf/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nment in the module [skip ci] The environment was set in the process config with a projectDir path, which is the including pipeline's directory when this pipeline is included. The module's conda directive now names the report environment relative to the module, and the patch file records it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…k, split config params from pipeline params [skip ci] Aligns the pipeline with nf-core/rnaseq so that it can be included without special cases. - The outdir param is replaced by Nextflow's -output-dir: the output location comes from outputDir, reports go to pipeline_info under it, the params dump and completion use workflow.outputDir, and the schema, tests, docs, profiles and config examples are updated. - The propd and grea results were the last files published with publishDir; they go through typed output targets with the same folders as before, so an including pipeline decides what to publish. - conf/params.config holds only the config params (publish mode, igenomes, custom config, report suffix, config profile fields); every pipeline param is declared once, in the typed params block. The block no longer carries mangled string defaults (the // in URLs had been read as comments). - The secret directive of SHINYNGS_APP reads the paramset from the task instead of the global params. - The cache selectors for the differential filter and functional enrichment subworkflows named the subworkflows, which match no process; they now match the processes inside. - exploratory_log2_assays is nullable, as the profiles that unset it require. - The banner code tolerates a manifest without a DOI, and the minimum Nextflow version is 26.09.0-edge. - The docs section on including the pipeline follows the rnaseq wording. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The parameter summary and the completion summary looked for nextflow_schema.json in the project that runs, which is the including project when the pipeline is included. They now use the pipeline's own schema, as the nf-core template asks of a pipeline that may be included. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tils subworkflow [skip ci] nf-schema reads parameters_schema; parametersSchema was ignored, so the schema given to the parameter summary, the help and the validation was not used. Same as in rnaseq. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… nextflow version [skip ci] Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… for typed params [skip ci] Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… vendored subworkflows that differ from upstream nextflow config resolves the plugins block, and the patched nf-schema is not in the public registry. The edited nf-core subworkflows are recorded with nf-core subworkflows patch, so that check_local_copy passes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
quartonotebook is deprecated in favour of quarto/notebook, which writes a _quarto.yml in the task directory so Quarto does not search parent directories for a project (nf-core/modules#13077). On Fusion that search reached the mount root and failed every report with "attempt to write a readonly database". The module patch keeps the existing changes: the report conda environment, and no versions topic emissions (the report consumes the versions topic). The report writes the versions.csv the module requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the pipeline a typed pipeline (a
params {}block, an entry workflow and anoutput {}block) that can be included in another pipeline with nextflow#7213 (pipeline composition), with the same conventions as nf-core/rnaseq#1966.What changes
main.nfdeclares every param with its type and default.input,contrasts,matrix,feature_length_matrixandgtfare optionalValue<Path>?: loaded from plain paths on the command line, or given by the dataflow of an including pipeline.buildParamset()), and files are located relative to the pipeline (pipelineDir()), notprojectDir.outdirparam. The output location is Nextflow's-output-dir, as in rnaseq. All results, including propd and grea, are published through theoutput {}block.conf/params.configholds only config-only params andconf/modules.configthe process config. Every pipeline param is declared once, in the typed block. Thecache = 'deep'selectors match the processes insideABUNDANCE_DIFFERENTIAL_FILTERandDIFFERENTIAL_FUNCTIONAL_ENRICHMENT, and thesecretof SHINYNGS_APP reads the paramset of the task.26.09.0-edge) and a patched nf-schema2.7.2-channel.5(Validate and print the value of Channel and Value params instead of hanging nextflow-io/nf-schema#230, Mention missing dots in report volcano #232).An including pipeline provides the manifest, the resources, the container profile, the plugin and
conf/modules.config.nf-core lint
nf-core pipelines lintreports the same 5 failures asdev(typed params and lint: nf-core/tools#4493). The defaults live in the typedparams {}block, which the lint (nextflow config) does not see, so.nf-core.ymlignores, with reasons:nextflow_config(126 params),schema_params,nf_test_content(tests pass nooutdir) andfiles_unchangedfor the PR template.Status
nextflow linthas no errors. All 26 nf-test tests pass against the existing snapshots. Real runs oftestandtest_rnaseq_propd_greapublish the same files asdev; md5 differences are limited to files that also differ between two runs ofdev.DESEQ2_DIFFERENTIALwithvs_method rlogcan hang after writing its outputs (OpenBLAS threads).devdoes too, andOPENBLAS_NUM_THREADS=2avoids it.🤖 Generated with Claude Code