I'm trying to run FCS-GX (<github.com/ncbi/fcs>) u...
# helpdesk-hpc-sweden
n
I'm trying to run FCS-GX (github.com/ncbi/fcs) using >512GB memory on dardel. I'm running FCS-GX through the snakemake workflow GEP2. When FCS-GX has sufficient memory it should require less than an hour to run (github.com/ncbi/fcs/wiki/FCS-GX-troubleshooting). I have attempted requesting both 512GB and 880GB, but I keep getting >24h runtimes and the warning
Note: detected page-faults (disk-thrashing) - GX may run slowly.
from FCS-GX. Has anyone else had luck running FCS-GX on Dardel?
m
I've successfully run FCS-GX on dardel. What's your command line invocation?
n
It's being submitted through GEP2, here's the link to the rule definition: github.com/diegomics/GEP2/blob/…/F_cont_analysis.smk#… and this is the resources line from the log file:
Copy code
resources: tmpdir=/tmp, disk_mb=3875, disk=3.88 GB, disk_mib=3696, mem_mb=901120, mem=901.12 GB, mem_mib=859375, slurm_account=naiss2026-3-456, slurm_partition=main, runtime=1440, slurm_extra=--mail-type=ALL
I'm very open to running this through other tools than GEP2. Do you have a recommended way of running this @Mahesh Binzer-Panchal?
m
I usually run through Nextflow. github.com/NBISweden/Earth-Biogenome-Project-pilot/blob/…/rungx.nf I'm no expert on Snakemake. Do you know how
RAM_DB_DIR=""
should be set from the code there? That's basically the key. However, dardel is also notoriously slow lately with disk I/O. I've had jobs being resubmitted over months because of disk access issues, login reboots, and such.
n
There's a config option (FAST_TEMP_DIR) that propagates, should be easy for me to update! I know of PDC_TMP - is there a per-node dir for faster IO?
m
The Nextflow pipeline uses
/dev/shm/gxdb
so point somewhere on
/dev/shm/
Or if I understand the code enough from my quick scan, set it as
/dev/shm/
n
If I wanted to run the rungx nextflow module, I'm assuming that I would need to wrap that in some way. I've used a bunch of different nextflow pipelines but haven't written any myself. Any idea of how complex it would be for a novice to set this up?
And thanks a ton for this pointer! I'll experiment with these!
m
It should be quite simple to take the module and wrap it. I recall there's been something recent that lets one do that automatically, but I may be misremembering. You can write a workflow that calls the module, and sets it up. Give me a few minutes and I'll give you some code
Copy code
#!/usr/bin/env nextflow
// Standalone wrapper around FCSGX_RUNGX that stages the GX database into a RAM disk.
//
//   nextflow run <http://rungx_wrapper.nf|rungx_wrapper.nf> -c rungx_wrapper.config \
//       --assembly 'asm/*.fasta.gz' --taxid 9606 --gxdb /path/to/gxdb
//
include { FCSGX_RUNGX } from "./modules/local/fcsgx/rungx"

params.assembly    = null          // FASTA file or glob
params.taxid       = null          // NCBI taxid of the sample
params.gxdb        = null          // on-disk GX database directory (source for the RAM-disk copy)
params.ramdisk_path = '/dev/shm/gxdb'  // destination in RAM; removed when the task exits
params.outdir      = 'results'

workflow {
    main:
    if( !params.assembly || !params.taxid || !params.gxdb ) {
        error "Required: --assembly, --taxid, --gxdb"
    }
    ch_assembly = channel.fromPath( params.assembly, checkIfExists: true )
        .map { fasta -> [ [ id: fasta.name.replaceFirst(/\.(fa|fasta|fna)(\.gz)?$/, '') ], params.taxid, fasta ] }
    ch_gxdb = channel.fromPath( params.gxdb, checkIfExists: true, type: 'dir' ).collect()

    FCSGX_RUNGX( ch_assembly, ch_gxdb, params.ramdisk_path )

    publish:
    reports = FCSGX_RUNGX.out.fcs_gx_report
    taxonomy = FCSGX_RUNGX.out.taxonomy_report
    hits = FCSGX_RUNGX.out.hits
}

output {
    reports  { path 'fcsgx' }
    taxonomy { path 'fcsgx' }
    hits     { path 'fcsgx' }
}
config:
Copy code
outputDir = params.outdir

process {
    withName: 'FCSGX_RUNGX' {
        // One task at a time: all tasks share the single RAM-disk copy of the DB.
        maxForks = 1
        cpus     = 20
        memory   = 600.GB   // must cover the ~470 GB DB plus run_gx working memory
        time     = 1.h
    }
}
n
This is amazing! Thank you! Will see if I can get this running
Making sure that the GEP2
FAST_TEMP_DIR
was set to
/dev/shm
resolved this issue for me. Thank you for this input!
👍🏽 1