Hi, I've been testing pixelator 2.3.0 with the PNA test dataset on the Uppmax cluster Miarka. I had some problems with the uppmax profile and `container_env_options`which was discussed here. Later I had a somewhat related issue with duckdb in the

PIXELATOR_PNA_COMBINE_COLLAPSE

process:

_duckdb.IOException: IO Error: Could not read enough bytes from file "/tmp/duckdb_temp_storage_S32K-0.tmp": attempted to read 32768 bytes from location 14581760

If I use

$SNIC_TMP

(local node storage) instead of `/tmp`the run is able to finish successfully

env {
    PIXELATOR_DUCKDB_TEMP_DIR = "\${SNIC_TMP}/.duckdb_cache"
    MPLCONFIGDIR              = "\${SNIC_TMP}/.config/matplotlib"
    NUMBA_CACHE_DIR           = "\${SNIC_TMP}/.numba_cache"
    XDG_CACHE_HOME            = "\${SNIC_TMP}/.cache"
    XDG_DATA_HOME             = "\${SNIC_TMP}/.data"
}

Do you want me to open another issue on this?

Hi Jonas! It looks like the node run out of memory (I'm guessing the

/tmp

partition is quite limited on Uppmax, which is why they provide

SNIC_TMP

). Basically, the issue is that the temporary path pixelator needs are hard coded in the config. I don't have a straightforward solution right now so please create an issue 👍

Hi Adrien 👋 That was my thought as well, but it might be more complicated. I checked `/tmp`on a compute node and got

df -h /tmp

Filesystem      Size  Used Avail Use% Mounted on

/dev/md127      205G   15G  191G   8% /tmp

I also checked

/tmp

in the singularity container

singularity exec quay.io-pixelgen-technologies-pixelator-0.23.0.img df -h /tmp

Filesystem      Size  Used Avail Use% Mounted on

/dev/md126      391G   16G  375G   5% /tmp

I would be surprised if this is not enough, but maybe you are right. I'll open an issue later today 👍

Did you get this issue consistently before using

$SNIC_TMP

?

I got it once, and then fixed it by redirecting to

$SNIC_TMP

. But I can run it again with `/tmp`and see if it happens again

Interesting 🤔 Thank you for checking, we've had similar error messages when running the pipeline on AWS internally, but they seemed to disappear when we increased the allocated memory (although this has to do with the RAM and not temporary storage, and this did not happen consistently). If this reoccur, you can try to increase the memory allocated for the

COMBINE_COLLAPSE

step (should be around this line: https://github.com/nf-core/pixelator/blob/9d1d575fd6967324b1d211da42022133158622ba/conf/base.config#L37) The memory required by this step is highly dependent on the number of cells in the sample, in the future we hope to scale the requirements automatically, but there is nothing in place yet.

Could it be a filesystem sync issue since this occurs occasionally on AWS and Uppmax? I think the

_duckdb.IOException: IO Error:

rather than a `out of memory`error hints to that. Also, when I look at the successful runs, only ~85% of the requested RAM is used by that process.