Hello everybody! I'm new to Nextflow and nf-core so my questions might be tied of one of thei basic functionalities. I've tried to read through the documentation but I might have missed this information, so I apologize in advance. I'm running a pipeline (nf-core/eager, v2.5.1) using Nextflow v22.10.6 with DSL1 and I've tried two approaches for clean up: • The first one, was to send my pipeline using sbatch and adding to the script

if <full_nextflow_command>
then
    nextflow clean -f -k
fi

• The second one was to just add

cleanup = true

to my custom configuration file. When I tried the second way, my whole work/ directory was cleaned up of everything, including .sh, .run, and .log files, which were never removed using the first method and which I needed to monitor what command was actually sent. Moreover, previous runs on the same sample were not cleaned up (their work/process/ folders were intact) which makes me think that this behaviour was caused by me messing around with the config. Is it a functionality of

cleanup = true

or did something completely different happened? Thank you for anyone that might have an idea.

Yes,

cleanup = true

completely wipes the current run's work directory. It's documented https://docs.seqera.io/nextflow/reference/config#cleanup although it could be cleaner. Here's a simple run script to use instead.

#! /usr/bin/env bash

set -euo pipefail

trap 'echo "=!= ERROR: Workflow failed (exit code: $?) ==="' ERR

echo "=== Starting workflow: ==="
echo "=== Profile: ${NXF_PROFILE:-uppmax} ==="
WORKFLOW=../../code/01_mapping/01_mapping.nf
nextflow run "${WORKFLOW}" \
    -params-file params.yml \
    -profile "${NXF_PROFILE:-uppmax}" \
    ${CLI_OPTS:-""}

echo ""
echo "=== Cleaning up obsolete work directories ==="
nextflow clean -f -before last && find work -type d -empty -delete
echo "=== Workflow complete ==="

•

set -e

means the script exits on an error, so if

nextflow run

errors,

nextflow clean

does not run • update the profile to your needs. This is just a simple example from one of my projects.

Thank you so much for both the answer and also the code snippet, that is super helpful 🙏 Very appreciated.

If you need the command used by a specific process, i.e. the content of its

.command.sh

, you can find that in the

execution_report_.*html

-- there is a table with all of the processes (including exit codes, RAM, CPU, and time allocations and use), and if you scroll over to the

script

column, you'll find the commands (click to expand). That saves you from keeping and searching through all of the

.command.sh

files. And

cleanup = true

only affects the current run -- anything still hanging around from prior runs will not (and should not) be touched. The

nextflow clean

command has options like

-before

and

-after

that give you fairly fine-grained control over which runs to clean and which to keep. (Note that process work directories that are used [either originally or as cache] in other runs will not be deleted, even if they are also part of a run you are deleting.) The reason you are retaining logs, etc., with

nextflow clean

is the

-k

option -- without that the effect would be closer to

cleanup = true

. But if you want ultimate control, a script like Mahesh showed you is the way to go.