https://nf-co.re logo
Join Slack
Powered by
Hi! Sorry for a wall of text :grimacing: TL;DR pi...
# pixelator
s
Hi! Sorry for a wall of text 😬 TL;DR pixelator 2.3.0 fails on miarka on task `GENERATE_REPORTS:EXPERIMENT_SUMMARY`*. In addition - some required parameters seems to be unused.* I'm running nf-core/pixelator 2.3.0 on Miarka (Uppmax HPC) and have some issues. Firstly, I created a samplesheet according to the docs with all the required parameters for PNA including 
sample_alias
and 
condition
. But when running the pipeline I found this is the nextflow log
Copy code
Feb-24 16:00:05.437 [main] WARN  n.v.logging.ValidationLogger - Found the following unidentified headers in /proj/ngi2016001/nobackup/NGI/ANALYSIS/YL-4530/YL-4530.SampleSheet.csv:
        - sample_alias
        - condition
This didn't seems to matter and thepipeline ran all the way to 
NFCORE_PIXELATOR:PIXELATOR:PNA:PNA_GENERATE_REPORTS:EXPERIMENT_SUMMARY
before it failed with the following output:
Copy code
Feb-26 05:39:55.712 [TaskFinalizer-10] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_PIXELATOR:PIXELATOR:PNA:PNA_GENERATE_REPORTS:EXPERIMENT_SUMMARY (all-samples); work-dir=/proj/ngi2016001/nobackup/NGI/ANALYSIS/YL-4530/work/e2/bf332fa5fa8c59e70aa3a9cbf91e97
  error [nextflow.exception.ProcessFailedException]: Process `NFCORE_PIXELATOR:PIXELATOR:PNA:PNA_GENERATE_REPORTS:EXPERIMENT_SUMMARY (all-samples)` terminated with an error exit status (1)
Feb-26 05:39:56.635 [TaskFinalizer-10] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_PIXELATOR:PIXELATOR:PNA:PNA_GENERATE_REPORTS:EXPERIMENT_SUMMARY (all-samples)'

Caused by:
  Process `NFCORE_PIXELATOR:PIXELATOR:PNA:PNA_GENERATE_REPORTS:EXPERIMENT_SUMMARY (all-samples)` terminated with an error exit status (1)


Command executed:

  # Copy the full quarto dir from the read-only image into the workdir
  cp -r /workspace/inst/quarto/ ./quarto/
  quarto render ./quarto/pixelatorES.qmd \
      -P sample_sheet="$PWD/YL-4530.SampleSheet.csv" \
      -P data_folder="$PWD/results/" \
       \
  
  mv ./quarto/pixelatorES.html experiment-summary.html

Command exit status:
  1

Command output:
  (empty)

Command error:
  9/17                         
  10/17 [read_counts]           
  ESC[31mError in `h()`:
  ! error in evaluating the argument 'x' in selecting a method for function 't': invalid type "character" in 'R_matrix_as_sparse'
  Backtrace:
       ▆
    1. ├─... %>% merge_data(sample_sheet)
    2. ├─pixelatorES::merge_data(., sample_sheet)
    3. │ └─pg_data %>% ...
    4. ├─purrr::map(...)
    5. │ └─purrr:::map_("list", .x, .f, ..., .progress = .progress)
    6. │   └─purrr:::vctrs_vec_compat(.x, .purrr_user_env)
    7. ├─pixelatorES::load_pxl_data_list(...)
    8. │ └─... %>% ...
    9. ├─base::lapply(., ReadPNA_Seurat, load_proximity_scores = FALSE)
   10. │ └─pixelatorR (local) FUN(X[[i]], ...)
   11. │   └─db$counts()
   12. │     └─... %>% Matrix::t()
   13. ├─Matrix::t(.)
   14. ├─methods::as(., "dgCMatrix")
   15. │ └─Matrix (local) asMethod(object)
   16. │   └─Matrix::.m2sparse(from, "dgC")
   17. └─base::.handleSimpleError(...)
   18.   └─base (local) h(simpleError(msg, call))
  ESC[39m
I then removed the parameters from the samplesheet and restarted the run thinking that maybe the extra input could affect the report generation somehow. But the pipeline failed again at the same step with the same error. I don't really know how to fix this can someone help? In addition to this, how should I treat 
sample_alias
and 
condition
?
a
Hi Sarah! 👋 Regarding 
sample_alias
and 
condition
, we have a fix on the way, it should be solved by the next release (hopefully this week, the release is being reviewed now) Then about the error, I don't think it's related to these two parameters. Would it be possible to see the samplesheet you used for this run?
s
Hi Adrien 👋 Sure it is a test data so nothing secret. (but must check first)
The one called .org is with 
sample_alias
and 
condition
YL-4530.SampleSheet.csv.orgYL-4530.SampleSheet.csv
a
Thanks! I'll check with our bioinformaticians and come back to you 👍
🙌 1
Meanwhile, I see the paths to the fastq files are very long, is it possible for you to have relative path instead?
s
Very long paths is standard for us 😬. This run is started according to in-house routine for nf-core pipelines. I can try with relative paths if you think it helps,
Ok, I updated the script with 
--input_basedir
and shortened the paths in the samplesheet. The two previous runs ran for ~1d and 12h before they failed. We have users waiting for the data so any updates and pointers from your bioinformaticians are welcome. 😄
a
Alright, I got some new info: 1. even though the pipeline failed at the last step, it should have still created some 
.pxl
files in the output directory. You can analyze these file in R with https://github.com/PixelgenTechnologies/pixelatorR (the experiment summary basically just generates an html report with the same data) 2. I talked to our bioinformaticians, but without more info it is hard to say anything. Is there a way you could share the output 
.pxl
file with us? Like with a good old DDS delivery or something?
s
That can be arranged. Do you want the whole results file in case you want to dive deeper or just these files;
Copy code
├── YL-4530-Pixelgen-test-2-mag.layout.pxl
├── YL-4530-Pixelgen-test-3-sampleA.layout.pxl
└── YL-4530-Pixelgen-test-3-sampleB.layout.pxl
a
Might as well include everything, you can use my email for the delivery: adrien.coulier@pixelgen.com Thanks!
s
Great. I just have to double check with our quality manager first if there could be any legal issues with delivering data even though it is test data. I'll start the delivery as soon as I know
a
Thank you!
I tried to run one of our test datasets with long paths, but they worked fine, so unfortunately, I don't think shortening the paths will help 😞 Hopefully we can figure out what's going on once we dig into the output data
s
Hi! We are not allowed to send you data 😞, so we'll run the R packages instead. I've been in contact with another researcher who also failed to get the pipeline to successfully finish the report generation, he ran on a previous version 2.2.0. When we ran the test data available on your website the report generation worked fine. However that dataset seems to not match the data that we provided now. Do you have updated testdata that we can run? We can send you output from that
a
Which testdata did you use? And how did it differ from your data?
j
We used this data https://software.pixelgen.com/datasets/1k-human-pbmcs-v1.0-proxiome-immuno-155/#file-download . I guess there is a difference in the panel, since we used 
proxiome-immuno-155-v2
for the user data and 
proxiome-immuno-155.csv
as 
panel_file
when we ran the test data
a
Alright, I'll ask internally and come back to you with some plan 👍
Did you try to run the 
test_pna
profile? Since it downloads the data from github, you might have to download it manually and set the 
pipelines_testdata_base_path
parameter. It uses version 2 of the panel.
You can download the data from here: https://github.com/nf-core/test-datasets/tree/pixelator
j
ok, I'll give it a try
a
Also, in the pixelatorES, there is a utility script to run the experiment summary from a docker container (https://github.com/PixelgenTechnologies/pixelatorES/blob/main/utility/run_ES_docker.sh). Could you try the following command and see if you consistently get the same error?
Copy code
bash run_ES_docker.sh -d <pipeline output dir> -n <path where you want the ES> -s <path to your samplesheet> -i <http://quay.io/pixelgen-technologies/pixelatores:0.6.0|quay.io/pixelgen-technologies/pixelatores:0.6.0>
I'm don't think docker is available on Miarka though, so you might need to try this locally
You mentioned you were in contact with another researcher who also had issues with the experiment summary. I assume they got the same error message? Do you know if they were also using singularity?
s
I have asked now, I'll report back when I know more
🙏 1
a
Another thing, I've dived a bit deeper into the code, and I suspect the error might come from one of the matrices in one of the pxl files. Could you try to install pixelator and look at the content of the pixelfile eg in a jupyter notebook or something with:
Copy code
import pixelator
f = pixelator.read_pna(<path to one of your pxl files>)
f.adata().X.dtype
The matrix type should be 
uint32
, but the error from the experiment summary suggests it could be a string or something
👍 1
@Jonas Kjellin I'm also available now, in case you'd like to share your screen and try to troubleshoot this together or somethin
j
Thanks, but I will have to leave soon. I will run run_ES_docker.sh and check the pixelfile content with pixelator and let you know what I find
👍 1
a
Sounds good 👍 Once you get a chance, can you also look at the matrix from the R side (and in particular the column names), that is, using the function from PixelatorR:
Copy code
pxl_db <- PixelDB$new("path/to/pxl/file")
pxl_db$fetch_table("__adata__X") %>% colnames() %>% head()
s
Response: (🇸🇪 ) Jag körde inte Nextflow-skriptet i singularity, däremot använder ju skriptet tre containers: blobs-sha256-52-52ccce28d2ab928ab862e25aae26314d69c8e38bd41ca9431c67ef05221348aa-data.img quay.io-pixelgen-technologies-pixelator-0.22.1.img quay.io-pixelgen-technologies-pixelatores-0.4.3.img Jag bifogar loggen med felet:
slurm-11238222.out.txt
a
I see 👍 The error message is different though, so it looks like it's a different issue
j
OK, some updates. I ran the experiment summary docker container which gave the same error as when running it via the pipeline. Also, attempted to inspect the .pxl files from pixelatorR
Copy code
> pxl_db <- PixelDB$new(filename)
Error in `initialize()`:
ℹ The PXL file must contain PNA data.
ℹ The PXL file might be blocked by another connection
✖ Failed to connect to the database.
Run `rlang::last_trace()` to see where the error occurred
We just received some more data that I will analyze today which will tell us if this problem is sample specific or more general.
a
Regarding the error message, it looks like it could be that either the file is corrupted, or it is being used somewhere else. Can you confirm that there is no other process reading the file (I'm thinking if you've started the analysis in Rstudio and it's still running or something)?
j
I'm pretty sure the files are corrupted, since this is the first work I did with these files today. Also, got the same issue yesterday but was at that point not sure if it was a me-problem or a file problem. The file sizes of the three pxl files also indicates that they are corrupted/incomplete (5.3Gb, 500Mb and 16Mb). But the file sizes are consistent over the three runs we have done so far
Can the pna experimental protocol fail so that you get plenty of reads but very little "proximity" information or what you call it
a
Interesting! And I guess you also get some kind of errors when you try the python code above?
Copy code
import pixelator
f = pixelator.read_pna(<path to one of your pxl files>)
f.adata().X.dtype
Yes, this can definitely happen, but usually the pipeline is able to detect it and fails during the graph step, indicating that it only found small components. Would you be able to share the 
nextflow.log
from that run? Or at least scan through it and see if you can see any error messages from there
Also, do you get this R error with all three pixel files?
j
I tried the largest and the smallest one with same error for both
👍 1
This is from the PNA_GRAPH log for the smallest file, there is some warnings but it finished w/o errors
Copy code
2026-03-02 21:06:42 [DEBUG     ]  Labelling connected components
2026-03-02 21:06:42 [DEBUG     ]  Finding connected components
2026-03-02 21:40:23 [DEBUG     ]  Initiating multiplet recovery
2026-03-02 21:47:29 [DEBUG     ]  Running Leiden community detection. At recursion depth 0
2026-03-03 12:21:25 [DEBUG     ]  Running Leiden community detection. At recursion depth 1
2026-03-03 12:51:57 [DEBUG     ]  Running Leiden community detection. At recursion depth 2
2026-03-03 13:02:28 [DEBUG     ]  Running Leiden community detection. At recursion depth 3
2026-03-03 13:10:09 [DEBUG     ]  Running Leiden community detection. At recursion depth 4
2026-03-03 13:17:12 [DEBUG     ]  No more components to refine, will break the refinement loop.
2026-03-03 13:45:55 [DEBUG     ]  Filtering components by dynamic size thresholds
2026-03-03 13:45:56 [DEBUG     ]  Attempting to find size lower limit on 25461771 component sizes where 576 are unique
2026-03-03 13:45:56 [DEBUG     ]  Pre-filtering 734385 (3.0%) component sizes with size smaller than 20 before trying to find a lower size limit
2026-03-03 13:46:01 [DEBUG     ]  Size limit of 23 found
2026-03-03 13:46:01 [WARNING   ]  Dynamic minimum component size found is below the minimum size threshold of 300 edges. Using that as the minimum size to avoid downstream problems.
2026-03-03 13:46:01 [WARNING   ]  Could not find a lower bound for component size filtering, will set the lower bound to 8000
2026-03-03 13:54:08 [DEBUG     ]  Sorting edgelist by component
2026-03-03 14:02:04 [DEBUG     ]  Counting molecules
2026-03-03 14:07:18 [DEBUG     ]  Building edgelist from anndata
2026-03-03 14:07:18 [DEBUG     ]  Constructing counts matrix.
2026-03-03 14:07:18 [DEBUG     ]  Computing component metrics.
2026-03-03 14:07:19 [DEBUG     ]  Computing antibody metrics.
2026-03-03 14:07:19 [DEBUG     ]  Building AnnData instance.
2026-03-03 14:07:19 [DEBUG     ]  Calling aggregates based on tau specificity scores
2026-03-03 14:07:19 [INFO      ]  Found 1 aggregates. This is 100% of the components
2026-03-03 14:07:19 [DEBUG     ]  Building pxl file
2026-03-03 14:07:22 [INFO      ]  Finished pixelator graph in 61903.92s
I don't know what to expect here, but there are some obvious differences to the log from the largest file
Copy code
2026-03-03 12:38:38 [DEBUG     ]  Labelling connected components
2026-03-03 12:38:38 [DEBUG     ]  Finding connected components
2026-03-03 13:21:49 [DEBUG     ]  Initiating multiplet recovery
2026-03-03 13:27:21 [DEBUG     ]  Running Leiden community detection. At recursion depth 0
2026-03-03 14:55:53 [DEBUG     ]  Running Leiden community detection. At recursion depth 1
2026-03-03 15:18:07 [DEBUG     ]  Running Leiden community detection. At recursion depth 2
2026-03-03 15:42:16 [DEBUG     ]  Running Leiden community detection. At recursion depth 3
2026-03-03 16:03:32 [DEBUG     ]  Running Leiden community detection. At recursion depth 4
2026-03-03 16:48:34 [DEBUG     ]  Filtering components by dynamic size thresholds
2026-03-03 16:48:35 [DEBUG     ]  Attempting to find size lower limit on 42588336 component sizes where 1583 are unique
2026-03-03 16:48:35 [DEBUG     ]  Pre-filtering 8473 (0.0%) component sizes with size smaller than 20 before trying to find a lower size limit
2026-03-03 16:48:37 [DEBUG     ]  Size limit of 14601 found
2026-03-03 16:56:49 [DEBUG     ]  Sorting edgelist by component
2026-03-03 17:04:47 [DEBUG     ]  Counting molecules
2026-03-03 17:09:23 [DEBUG     ]  Building edgelist from anndata
2026-03-03 17:09:23 [DEBUG     ]  Constructing counts matrix.
2026-03-03 17:09:51 [DEBUG     ]  Computing component metrics.
2026-03-03 17:10:17 [DEBUG     ]  Computing antibody metrics.
2026-03-03 17:10:17 [DEBUG     ]  Building AnnData instance.
2026-03-03 17:10:17 [DEBUG     ]  Calling aggregates based on tau specificity scores
2026-03-03 17:10:17 [INFO      ]  Found 2 aggregates. This is 0% of the components
2026-03-03 17:10:17 [DEBUG     ]  Building pxl file
2026-03-03 17:10:22 [INFO      ]  Finished pixelator graph in 16548.49s
a
In your output directory, there should be something at 
outdir/pixelator/graph/<sample>.report.json
. This files contains some information about the component sizes and so on, could you share one of these files here?
j
sampleA.report.json
sampleA.report.json
sampleA == 16M pxl file , sampleC == 5.3G pxl file
sampleC.report.json
a
Alright, this makes things a lot clearer. In both samples, it looks like all but one or two components got discarded. I'm not sure why this happened yet, my guess would be an issue with the experimental protocol, but let me come back to you on that
Alright, we suspect the samples were sequenced too shallowly, could you confirm that in any way? If that is the case, using 
--pna_graph_max_refinement_recursion_depth 1
could help a little, this would prevent the pipeline from oversplitting the components and you might still get some data out of it after that
j
It's three samples on one 10B lane and we got 288 - 435 M read pairs per samples. How much data is needed?
a
How many cells per sample do you have?
And what type of cell do you have?
j
The libraries were made by the user so I have very limited information about the samples
I'll rerun with the suggested parameter for the first project and let you know if we get it to work with the other project.
a
Alright, our best guess so far is that the samples contained too many cells (which causes the number of reads per cell to be too low, i.e the graph will get very low connectivity)
j
yes, makes sense. Is there a cell UMI in the data so I can estimate the number of cells from the reads that I have?
a
No unfortunately not 😞
j
ah to bad. Feature request for the pipeline: QC warning instead of crash in these cases
👍 1
@Adrien Coulier The analysis finished w/o errors for the second dataset, so I think we can conclude that the troubles we had with the first project was sample related. I tried rerunning it with 
--pna_graph_max_refinement_recursion_depth 1
and then it failed at 
PIXELATOR_PNA_LAYOUT
due to OOM.
a
Sounds good, I've created a ticket internally so that we can look into making the pipeline more robust to these kind of issues Regarding the OOM, could you share the 
graph.report.json
file again? I suspect this time it didn't split the components enough and that the resulting cell was too big for 
PIXELATOR_PNA_LAYOUT
to process
👍 1
j
sampleA.json
sampleA.json
Another question related to the `condition`column in the samples sheet. We don't always have that information about the sample. Will the set condition actually affect the analyses somehow or is it only about result annotation in the report?
a
Regarding the OOM, it does look like there is one big cell with ca 1.5M umis (while this is typically around 100k). In theory, it is possible to adjust the parameters to try to split the components just right, but it would take a lot of trials and errors and there is no guarantee you would get anything meaningful out of it. I think the root of the issue is the number of reads per cell being too low About the condition column, as far as I understand, this mostly affect the grouping of the samples in the experiment summary. If it is not set, the sample name should be used, i.e. each sample will belong to its own group
j
Great, thanks for all the help!
🙇 1
8 Views
Open in Slack
PreviousNext
Powered by