https://nf-co.re logo
Join Slack
Powered by
What datasets would run through beginning to end f...
# wg-meta-pipelines
m
What datasets would run through beginning to end for some the nf-core pipelines? i.e. is there something I could get with fetchngs, and it take me through multiple other pipelines?
j
fetchngs -> taxprofiler - [processed reads] -> mag ? or fetchngs -> mag - [contigs] -> funcscan I dunno if you mean actually 'working' atm, but conceptually those are natural progressions
'working' == functionally feaisble (.e.g with existing pipeline channels etc)
m
I mean working at the moment. If you can elaborate (e.g. with paths), I can connect them
j
Ok, well currently the pipelines don't necessarily have pipeline-level output channels
m
Doesn't need to. I just the input and output files ( or profiles )
j
But for example, taxprofiler samplseheet from fetchngs already exists the 
analysis_ready_reads/
--outdir
directory of taxprofiler could go into mag
then the And then for mag to funcscan
Contigs from 
assembly/<assembler>/contigs*.fa{asta}.gz
could go into funcscan
m
Does there need to be any manual intervention like making a samplesheet?
j
The pipelines only have samplesheet input atm, if that's what you mean? But if/else statement to replace with input channels could be made if valid output channels are also added to the upstream pipeline For taxprofiler -> mag you can set reasonable default 
meta
for example (mag only accepts short reads really)
mag -> funcscan is very easy, simply 
sample_id
and the 
fasta
m
Thanks. I'll have a go later then
Added in Taxprofiler to the workflow, but bbduk.sh errors out with exit code 1, using profile 
test
for both fetchngs and taxprofiler, but the input from fetchngs, and 
--nf_core_pipeline taxprofiler
. There's no real error to make sense of why it exited.
The nice thing is, it's easy to add any workflow. The hard part is connecting output to input when you've no experience of the tools, and the test profiles don't chain well.
j
Uh that's odd, no log entry at all?
Yeah I see what you mean about the test profiles...
m
Copy code
Caused by:
    Process `NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_COMPLEXITYFILTERING:BBMAP_BBDUK (SRX6725035)` terminated with an error exit status (1)
  
  
  Command executed:
  
    maxmem=$(echo "6 GB"| sed 's/ GB/g/g')
    bbduk.sh \
        -Xmx$maxmem \
        in=SRX6725035_SRR9984183.fastp.fastq.gz \
        out=SRX6725035_SRR9984183.fastq.gz \
        threads=2 \
        entropy=0.3 entropywindow=50 entropymask=f \
         \
        &> SRX6725035_SRR9984183.bbduk.log
    cat <<-END_VERSIONS > versions.yml
    "NFCORE_TAXPROFILER:TAXPROFILER:SHORTREAD_COMPLEXITYFILTERING:BBMAP_BBDUK":
        bbmap: $(bbversion.sh | grep -v "Duplicate cpuset")
    END_VERSIONS
  
  Command exit status:
    1
  
  Command output:
    (empty)
j
Nothing in the 
.nextflow.log
either?
But that's very strange... maybe something in the 
bbduk.log
? Or is that also emopty?
m
Same thing
j
weird....
m
Copy code
Set threads to 2
Error when attempting to read SRX6725035_SRR9984183.fastp.fastq.gz
java.lang.RuntimeException: java.io.EOFException
	at fileIO.ReadWrite.getGZipInputStream(ReadWrite.java:1051)
	at fileIO.ReadWrite.getInputStream(ReadWrite.java:848)
	at fileIO.FileFormat.getFirstOctet(FileFormat.java:446)
	at fileIO.FileFormat.testInterleavedAndQuality(FileFormat.java:435)
	at fileIO.FileFormat.testFormat(FileFormat.java:371)
	at fileIO.FileFormat.<init>(FileFormat.java:220)
	at fileIO.FileFormat.testInput(FileFormat.java:162)
	at fileIO.FileFormat.testInput(FileFormat.java:144)
	at fileIO.FileFormat.testInput(FileFormat.java:129)
	at jgi.BBDuk.<init>(BBDuk.java:938)
	at jgi.BBDuk.main(BBDuk.java:78)
Caused by: java.io.EOFException
	at java.base/java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268)
	at java.base/java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258)
	at java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164)
	at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
	at fileIO.ReadWrite.getGZipInputStream(ReadWrite.java:1045)
j
Uhhhhhh
Ok....
Does that happen on a different dataset? That almost looks like a curropted FASTQ file somehow?
m
No idea what other data to try. I'm working in Gitpod and the space is limited. I can't use 
test_full
And I'm over quota for core hours used where I have permission to run
j
Ah crap
Yeah test full won't work
What test data are you using? The taxprofiler one?
m
See the params.yml in the 
nf-cascade
repo. It's the 
test
profile for fetchngs with 
--nf_core_pipeline taxprofiler
, and that is passed to taxprofiler also using 
test
profile to fill in the rest, but the samplesheet is passed via 
--input
👍 1
j
I'll test it this evening (TM)
Maybe I cannsee if it's reproducibile with different data
m
I have to admit, for my purposes, I know the implementation works, and it's easy(-ish) for someone else to copy and use. The major issue would be struggling with the nextflow code connecting up the workflows, e.g. writing a process to turn a set of contigs into a samplesheet.
j
Yeah, that is something that's high on my task list to do actually
I Was planning on trying to adopt the fetxhbgs module for that but haven't had time to study it
Conceptually with a 'convdrstion' schema between a pipeline channel and a samplesheet definition it shouldn't be too hard I wpuld imagine?
m
Maybe explore the new publishing method instead. If I understand it, the index method would automatically make the samplesheet. https://www.nextflow.io/docs/latest/workflow.html#index-files
j
Oooooh
FYI for the testing: indeed the fetchngs data doesn't work with half the tools in taxprofiler (fastQC even! ). It's a single read FASTQ and looks rather odd to me to be honest... I switched out with a small file I know on the ENA and it worked: Line 11 of params.yml gets updated to
Copy code
wf_opts: '-profile docker,test -r dev --nf_core_pipeline taxprofiler --input /workspace/nf-cascade/ids.csv'
Where 
ids.csv
contains
Copy code
ERR3579671
And we have success:
m
Nice! (Although I'd intended the 
input
files to come through 
fetchngs.input:
since for downstream processes, you could override a samplesheet passed from fetchngs e.g. if it needs tweaking like for replicate number or something)
Thanks for trying
j
Not sure I follow?
But I guess my work around was more to show the issue is the fetchNGS data is possibly dodgy
💯 1
Which is why it wasn't working
I probably didn't test it as I should have obviously
But did something quick
m
Sorry, I didn't mean to cause confusion. It's not an issue. The params file could also have been:
Copy code
fetchngs:
  wf_opts: '-profile docker,test -r dev --nf_core_pipeline taxprofiler'
  input: 'ids.csv'
It's a completely separate issue related how I intended the workflow to be used ( and that's my fault for not making a good example ). I really just wanted to say thanks for checking the chaining running through was an issue with the fetchngs test data.
j
Ahh got you now! You're welcome!
m
So using your dataset id, I got all the way to funcscan which errors like:
Copy code
WARN: Undocumented setting `docker.userEmulation` is not supported any more - please remove it from your config
  [22/d7a863] Submitted process > NFCORE_FUNCSCAN:FUNCSCAN:ARG:AMRFINDERPLUS_UPDATE (update)
  [95/3901e7] Submitted process > NFCORE_FUNCSCAN:FUNCSCAN:AMP:DRAMP_DOWNLOAD
  [17/9b1bb9] Submitted process > NFCORE_FUNCSCAN:FUNCSCAN:CUSTOM_DUMPSOFTWAREVERSIONS (1)
  WARN: Undocumented setting `docker.userEmulation` is not supported any more - please remove it from your config
  [30/5af32b] Submitted process > NFCORE_FUNCSCAN:FUNCSCAN:MULTIQC
  ERROR ~ Error executing process > 'NFCORE_FUNCSCAN:FUNCSCAN:AMP:DRAMP_DOWNLOAD'
  
  Caused by:
    Process `NFCORE_FUNCSCAN:FUNCSCAN:AMP:DRAMP_DOWNLOAD` terminated with an error exit status (1)
  
  
  Command executed:
  
    mkdir amp_ref_database/
    ampcombi_download.py
    
    cat <<-END_VERSIONS > versions.yml
    "NFCORE_FUNCSCAN:FUNCSCAN:AMP:DRAMP_DOWNLOAD":
        ampcombi: $(ampcombi --version | sed 's/ampcombi //')
    END_VERSIONS
  
  Command exit status:
    1
  
  Command output:
    (empty)
  
  Command error:
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.11/http/client.py", line 1277, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.11/http/client.py", line 1037, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.11/http/client.py", line 975, in send
        self.connect()
      File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
        conn = self._new_conn()
               ^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 179, in _new_conn
        raise ConnectTimeoutError(
    urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f83546be790>, 'Connection to <http://dramp.cpu-bioinfor.org|dramp.cpu-bioinfor.org> timed out. (connect timeout=None)')
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
        resp = conn.urlopen(
               ^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
        retries = retries.increment(
                  ^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='<http://dramp.cpu-bioinfor.org|dramp.cpu-bioinfor.org>', port=80): Max retries exceeded with url: /downloads/download.php?filename=download_data/DRAMP3.0_new/general_amps.xlsx (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f83546be790>, 'Connection to <http://dramp.cpu-bioinfor.org|dramp.cpu-bioinfor.org> timed out. (connect timeout=None)'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/gitpod/.nextflow/assets/nf-core/funcscan/bin/ampcombi_download.py", line 78, in <module>
        download_DRAMP("amp_ref_database")
      File "/home/gitpod/.nextflow/assets/nf-core/funcscan/bin/ampcombi_download.py", line 30, in download_DRAMP
        r = requests.get(url, allow_redirects=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
        return request("get", url, params=params, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
        return session.request(method=method, url=url, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
        resp = self.send(prep, **send_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
        r = adapter.send(request, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 553, in send
        raise ConnectTimeout(e, request=request)
    requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='<http://dramp.cpu-bioinfor.org|dramp.cpu-bioinfor.org>', port=80): Max retries exceeded with url: /downloads/download.php?filename=download_data/DRAMP3.0_new/general_amps.xlsx (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f83546be790>, 'Connection to <http://dramp.cpu-bioinfor.org|dramp.cpu-bioinfor.org> timed out. (connect timeout=None)'))
  
  Work dir:
    /workspace/nf-cascade/work/nf-core/funcscan/work/95/3901e780c5deb304de95bb03bae458
  
  Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
  
   -- Check '.nextflow.log' file for details
  Execution cancelled -- Finishing pending tasks before exit
  -[nf-core/funcscan] Pipeline completed with errors-
Thank you. This is POC is working perfectly I think
j
sigh
Stupoid databases, but yes, looks like it is working!
Open in Slack
PreviousNext
Powered by