Hello! First message on this Slack, I'm new to nf-core & Nextflow. I'm running this pipeline offline, with a

--fasta

genome and

--digestion hindiii

. I've run into an issue where Bowtie2-based steps want the genome file to be bgzipped, but the

GET_RESTRICTION_FRAGMENTS

step is unable to read the compressed file. Here's the full output:

executor >  slurm (5)
[8b/2a51d4] NFCORE_HIC:HIC:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_hic.csv)    [100%] 1 of 1, cached: 1 ✔
[15/194ceb] NFCORE_HIC:HIC:PREPARE_GENOME:BOWTIE2_BUILD (hg38.bgzip.fa.gz)        [100%] 1 of 1, cached: 1 ✔
[a5/2f28f7] NFCORE_HIC:HIC:PREPARE_GENOME:CUSTOM_GETCHROMSIZES (hg38.bgzip.fa.gz) [100%] 1 of 1, cached: 1 ✔
[c0/aafef0] NFCORE_HIC:HIC:PREPARE_GENOME:GET_RESTRICTION_FRAGMENTS (A^AGCTT)     [100%] 1 of 1, failed: 1 ✘
[84/dae341] NFCORE_HIC:HIC:FASTQC (CHiC_CD34_rep2)                                [100%] 2 of 2, cached: 2 ✔
[69/348fb4] NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:BOWTIE2_ALIGN (CHiC_CD34_rep2)   [100%] 4 of 4 ✔
[-        ] NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:TRIM_READS                       -
[-        ] NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:BOWTIE2_ALIGN_TRIMMED            -
[-        ] NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:MERGE_BOWTIE2                    -
[-        ] NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:COMBINE_MATES                    -
[-        ] NFCORE_HIC:HIC:HICPRO:GET_VALID_INTERACTION                           -
[-        ] NFCORE_HIC:HIC:HICPRO:MERGE_VALID_INTERACTION                         -
[-        ] NFCORE_HIC:HIC:HICPRO:MERGE_STATS                                     -
[-        ] NFCORE_HIC:HIC:HICPRO:HICPRO2PAIRS                                    -
[32/621d19] NFCORE_HIC:HIC:COOLER:COOLER_MAKEBINS (null})                         [100%] 3 of 3, cached: 3 ✔
[-        ] NFCORE_HIC:HIC:COOLER:COOLER_CLOAD                                    -
[-        ] NFCORE_HIC:HIC:COOLER:COOLER_BALANCE                                  -
[-        ] NFCORE_HIC:HIC:COOLER:COOLER_ZOOMIFY                                  -
[-        ] NFCORE_HIC:HIC:COOLER:COOLER_DUMP                                     -
[-        ] NFCORE_HIC:HIC:COOLER:SPLIT_COOLER_DUMP                               -
[-        ] NFCORE_HIC:HIC:HIC_PLOT_DIST_VS_COUNTS                                -
[-        ] NFCORE_HIC:HIC:COMPARTMENTS:COOLTOOLS_EIGSCIS                         -
[-        ] NFCORE_HIC:HIC:TADS:COOLTOOLS_INSULATION                              -
Plus 2 more processes waiting for tasks…
-[nf-core/hic] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_HIC:HIC:PREPARE_GENOME:GET_RESTRICTION_FRAGMENTS (A^AGCTT)'

Caused by:
  Process `NFCORE_HIC:HIC:PREPARE_GENOME:GET_RESTRICTION_FRAGMENTS (A^AGCTT)` terminated with an error exit status (1)


Command executed:

  digest_genome.py -r A^AGCTT -o restriction_fragments.bed hg38.bgzip.fa.gz
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_HIC:HIC:PREPARE_GENOME:GET_RESTRICTION_FRAGMENTS":
      python: $(echo $(python --version 2>&1) | sed 's/Python //')
  END_VERSIONS

Command exit status:
  1

Command output:
  Analyzing hg38.bgzip.fa.gz
  Restriction site(s) AAGCTT
  Offset(s) 1

Command error:
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  INFO:    Environment variable SINGULARITYENV_SNIC_TMP is set, but APPTAINERENV_SNIC_TMP is preferred
  Traceback (most recent call last):
    File "/scale/gr01/shared/nf-core/nf-core-hic_2.1.0/2_1_0/bin/digest_genome.py", line 176, in <module>
      contig_names, all_indices = find_re_sites(filename, sequences, offset=offset)
    File "/scale/gr01/shared/nf-core/nf-core-hic_2.1.0/2_1_0/bin/digest_genome.py", line 32, in find_re_sites
      for line in infile:
    File "/usr/local/lib/python3.9/codecs.py", line 322, in decode
      (result, consumed) = self._buffer_decode(data, self.errors, final)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

The

digest_genome.py

script tries to do an

open(filename, "r")

, but since the file is compressed it gives the error above. I'm thinking the easiest way around this is to just provide a BED file with

--restriction_fragments

directly? But I would love some advice on how to generate such a file. I'm not super familiar with Hi-C analysis in general. Thanks a lot!

I don't think this pipeline has a lot of support at the moment I'm afraid. You should file an issue on the github if you haven't already. But you could probably manually run the

digest_genome.py

on your unzipped fasta to produce the bed file you are after. If you go to the

work

directory where it was trying to execute that code, the .command.sh file tells you what exact command was trying to be executed (which is presented inside the error log).

Hey Simon, thank you for your reply! It didn't occur to me to just run

digest_genome.py

, which is the obvious solution in hindsight. I run HiCUP Digester externally and then gave the resulting file as an input with

--restriction_fragments

. I guess it amounts to the same thing. After that the whole thing run without big issues. Happy to say I got what I needed:)

Great, glad you got the output 🙂