The google doc mentioned in our meeting: https://docs.google.com/document/d/1uXYRxrVWa4WmlIDMqCXCLf30qN0zVXKrpvN-Qlf9MXk/edit?tab=t.0#heading=h.ir1rd8uror2

Thank you everyone for the great meeting today. To summarize what we agreed on to move forward: • Discuss in this thread the different new components that are required for @Josh Shapiro & @Jaclyn Taroni • Draft a full new pipeline workflow map that includes all components • Assign who will work on what to get the pipeline to a working dev branch by hopefully May Im gonna link two more items here in addition to the google doc shared by Erin: • Issue list with existing issues to implement existing components: https://github.com/nf-core/cellpainting/issues • Exaclidraw board to draft pipeline flow: https://excalidraw.com/#room=e2233414750a8f9a37d0,hiE3ey9Z9EDs5BaaD9rnmA

Hello all. Thanks for all the effort you have already put in here! Looking at the Excalidraw flow, it looks to me like all the process steps we would be looking for are generally accounted for, so that looks great. As we talked about in the meeting, I think the biggest additional thing we would be interested in would be an alternative input point which I think would come after the Cellprofiler Analysis step. The goal would be to allow input of an experiment that has already been through initial steps of segmentation and quantification. In particular, we would love it if you could take a dataset available on Cell Painting Gallery and directly use that for downstream steps. With all of those files available via S3, this could be quite efficient for users (like us) already working on AWS, though we might also expect that people could want to use local files that they had manually processed and QCed up to that point. I want to note that I am still extremely new to all this, so I'm still working through thinking about the data structures and file organization that this might require. I'd definitely like to hear any feedback on this idea!

An additional entrypoint for processed data shouldn't be a problem in my opinion and Nextflow will natively support cloud or local files, so no need to worry about that at all @Josh Shapiro. I think the way to achieve this optional input would likely be to have the optional input parameter and if it's not

null

skip the entire first half of the workflow, create the input channel from the processed data and start the workflow from there. What do you think from a Nextflow perspective @Ken Brewer @Edmund?

All data in the CPG should have a version of collated analysis data (the output of Cytotable in this workflow) but it isn't universally consistent what form that data is - for historical reasons it's mostly SQLite, though we do intend to move to Parquet, and occasionally I think you find other file formats like giant .csv's. So we could have the alternate entrypoint of this workflow be to pycytominer which is what we talked about in the last meeting and if someone only has CellProfiler Analysis data (non-collated) then they would need to collate it outside of the workflow before entry. (We could have the entry be to CytoTable, though I think supporting the variety of options we would want/need to support then gets awfully logic heavy - if one already has collated data they wouldn't necessarily want CyoTable to re-collate it, and pycytominer supports many entry datatypes.)

Thanks for the extra info @Erin Weisbart! If the CPG data is (mostly) ready for pycytominer, that would be fine as well! However, when I was looking I seemed to come across mostly projects that had an

analysis

directory but I could not always locate sqlite or parquet files; this was why I was suggesting the original CP output. I do think it makes sense to keep the input formats limited: I am sure it would be simpler if we could expect parquet files as input to the pycytominer process, so it may then make sense to have a separate process for converting from sqlite to parquet if that is needed.

Complete datasets in CPG should all (or at least mostly all) have collated data that can feed into pycytominer. (We don't have our data_validator script fully functional at the moment that will eventually provide nice reports on if any dataset is missing any of its artifacts, so I can't say anything more concrete). (Also, if there is any particular dataset you are interested in that has analysis data but not collated data, we very much welcome you to collate the data and contribute it back to the CPG!)