<!subteam^S0934CW2DPY> I've got my PR ready and passing tests that shows how we can get Nextflow to play nicely with cellprofiler. Thanks to @Erin Weisbart for helping me figure out some of the problems I was having with load_data.csv formatting. You can find the block of code where I write the illuminationcorrection load csvs here. We can use a similar Nextflow grouping and file collecting strategy to create load_data.csvs at any level of grouping that we need. The other key detail was figuring out how to invoke cellprofiler in a way that allows it find to files successfully. Initially I tried to use

stageAs: "images/*"

to stage all the image files into a dedicated directory. It turns out that

cellprofiler

's

--image-directory ./images/

argument doesn't work as intended when invoked by nextflow's combination of symlinks and docker. Instead we had to stage all the images in the task directory itself and use

--image-directory ./

As a point of feedback to @Erin Weisbart. This workaround is not ideal, but will be fine. I am concerned, however, by the fact that in the Cellprofiler wiki the default image directory argument is listed as deprecated. Having some way at the CLI for Nextflow to control where images should be found is pretty important. It would be great to get a couple of PR reviews on this because this will establish the patterns we will be using in other parts of the codebase.