Hi, whats are the maximum resources I can specify ...
# random
s
Hi, whats are the maximum resources I can specify for the test profile on github actions? Resource limits in my
test.config
👇
Copy code
process {

    resourceLimits = [
        cpus: 4,
        memory: '15.GB',
        time: '3.h'
    ]
}
16GB and 4CPUs I don't think there is a time limit
Leaving a little head room on memory is worth it I think
1.h time limit mostly to avoid hanging processes - and there is a time limit on GHA runs sorry @James Fellows Yates 😬
https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/troubleshooting-workflows/actions-limits > Job execution time > 6 hours > Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails.
Workflow run time
35 days / workflow run
If a workflow run reaches this limit, the workflow run is cancelled. This period includes execution duration, and time spent on waiting and approval.
j
Sorry I meant a time limit we have to care about 😬
But thanks for that link, I hadn't seen that!
p
Sorry I meant a time limit we have to care about
I've managed to hit it in the past 😆
👀 1
Can't remember what it was now, but I remember looking into the details and trying to figure out a workaroudn
Ah maybe it was when we were setting up AWS-megatests and we wanted the CI job to wait for the full run to finish so that we could report on whether it worked or not
j
Ahh ok. Like I've had broken jobs that just hung for 6h but they were obviously not working
s
My current settings are at 3 hours with 4 cpus and 15GB memory, and the process still exits without completing. I will try with the 6 hrs time limit.
p
GitHub actions runners are slow
👍 1
So even if the pipeline with test profile runs in 10 mins on your laptop, it might take 30 on GHA (made up numbers but you get the idea)
Hence we push for very very small test datasets a lot in #test-data
s
Yes, I have the smallest test dataset we can get for xenium with just 1 field of view, but the xeniumranger and other segmentation tools takes a lot of time as these are deep learning based
👍🏻 1
I will try with the 6 hr time limit, I hope it works. Will update on this thread
p
We have had some tools where it's been impossible - if that's the case, then it can be ok to make an exception and use a stub test if needed. But only as a last resort.
party this 2
In fact I think it was Spaceranger(?) that this jumps to mind for before
Or maybe Supernova
s
Yes, 10x tools are actually designed to run on high resources. Although xeniumranger runs on low resources provided but the recommended is atleast 128GB memory and 24-28 cores🫠
I will try once else with the extended time limit.
Thanks for all the inputs🙌🏻
👍🏻 1
s
If you are thinking a 6 hour timelimit, I think that is a clear case of needing a stub test only
👍🏻 1
👍 1
s
I will have to go with the stub test, the test failed even after a 6 hr time limit🫠
sobbing cowboy 1
Hi, I am trying to run the pipeline with
-stub
. For this I changed the
ci.yml
file to 👇 Now, when pushing the changes to the dev branch, I have git errors
Copy code
name: nf-core CI
# This workflow runs the pipeline with the minimal test dataset to check that it completes without any syntax errors
on:
  push:
    branches:
      - dev
  pull_request:
  release:
    types: [published]
  workflow_dispatch:

env:
  NXF_ANSI_LOG: false
  NXF_SINGULARITY_CACHEDIR: ${{ github.workspace }}/.singularity
  NXF_SINGULARITY_LIBRARYDIR: ${{ github.workspace }}/.singularity

concurrency:
  group: "${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}"
  cancel-in-progress: true

jobs:
  test:
    name: "Run pipeline with test data (${{ matrix.NXF_VER }} | ${{ matrix.test_name }} | ${{ matrix.profile }})"
    # Only run on push if this is the nf-core dev branch (merged PRs)
    if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/spatialxe') }}"
    runs-on: ubuntu-latest
    strategy:
      matrix:
        NXF_VER:
          - "24.04.2"
          - "latest-everything"
        profile:
          - "conda"
          - "docker"
          - "singularity"
        test_name:
          - "test"
        isMaster:
          - ${{ github.base_ref == 'master' }}
        # Exclude conda and singularity on dev
        exclude:
          - isMaster: false
            profile: "conda"
          - isMaster: false
            profile: "singularity"
    steps:
      - name: Check out pipeline code
        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
        with:
          fetch-depth: 0

      - name: Set up Nextflow
        uses: nf-core/setup-nextflow@v2
        with:
          version: "${{ matrix.NXF_VER }}"

      - name: Set up Apptainer
        if: matrix.profile == 'singularity'
        uses: eWaterCycle/setup-apptainer@main

      - name: Set up Singularity
        if: matrix.profile == 'singularity'
        run: |
          mkdir -p $NXF_SINGULARITY_CACHEDIR
          mkdir -p $NXF_SINGULARITY_LIBRARYDIR

      - name: Set up Miniconda
        if: matrix.profile == 'conda'
        uses: conda-incubator/setup-miniconda@a4260408e20b96e80095f42ff7f1a15b27dd94ca # v3
        with:
          miniconda-version: "latest"
          auto-update-conda: true
          conda-solver: libmamba
          channels: conda-forge,bioconda

      - name: Set up Conda
        if: matrix.profile == 'conda'
        run: |
          echo $(realpath $CONDA)/condabin >> $GITHUB_PATH
          echo $(realpath python) >> $GITHUB_PATH

      - name: Clean up Disk space
        uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1

      - name: "Run pipeline image mode with test data ${{ matrix.NXF_VER }} | ${{ matrix.test_name }} | ${{ matrix.profile }}"
        continue-on-error: ${{ matrix.NXF_VER == 'latest-everything' }}
        run: |
          nextflow run ${GITHUB_WORKSPACE} -profile ${{ matrix.test_name }},${{ matrix.profile }} --outdir ./results --mode image -stub

      - name: "Run pipeline coordinate mode with test data ${{ matrix.NXF_VER }} | ${{ matrix.test_name }} | ${{ matrix.profile }}"
        continue-on-error: ${{ matrix.NXF_VER == 'latest-everything' }}
        run: |
          nextflow run ${GITHUB_WORKSPACE} -profile ${{ matrix.test_name }},${{ matrix.profile }} --outdir ./results --mode coordinate -stub
`ci.yml`change produces the error👇
the lines I changed in the
ci.yml
file 👇
Copy code
- name: "Run pipeline image mode with test data ${{ matrix.NXF_VER }} | ${{ matrix.test_name }} | ${{ matrix.profile }}"
        continue-on-error: ${{ matrix.NXF_VER == 'latest-everything' }}
        run: |
          nextflow run ${GITHUB_WORKSPACE} -profile ${{ matrix.test_name }},${{ matrix.profile }} --outdir ./results --mode image -stub

      - name: "Run pipeline coordinate mode with test data ${{ matrix.NXF_VER }} | ${{ matrix.test_name }} | ${{ matrix.profile }}"
        continue-on-error: ${{ matrix.NXF_VER == 'latest-everything' }}
        run: |
          nextflow run ${GITHUB_WORKSPACE} -profile ${{ matrix.test_name }},${{ matrix.profile }} --outdir ./results --mode coordinate -stub
👀
Hi @Phil Ewels @James Fellows Yates sorry to bug you just wanted to know if you have any inputs on the
ci.yml
file issue 👆??
p
Not sure sorry, might be good to ask on #ci-runners