<!subteam^S016TBDBXHC> -- just running into some i...
# eager
i
<!subteam^S016TBDBXHC> -- just running into some issues here with getting the test_full run snapshot done. the main hangup is that since dsl2-eager runs each BAM_SPLIT_BY_REGION:SAMTOOLS_VIEW as a contig-by-contig job, it quickly approachs 1000s+ jobs for any lower-quality reference genome even with very few samples. the scheduling of these (very low computational load) jobs is therefore significant and causes immense slowdowns on shared clusters, leading at times to a user getting restricted to spawn new jobs or the parent process getting killed/running to runtime limits (current issue i am hitting) . i know that we are right now in feature freeze mode, but i think we should strongly consider adjusting the code to allow users the option of processing deduplication steps all in one go (as was implemented in dsl1 eager) rather than contig-by-contig. many users will be running dsl2-eager in clusters with prioritzation queues which will cause the whole eager run to balloon in runtime when jobs that take <10 seconds to execute are queued in eg. slurm for 10+ minutes happy to hear other thoughts on this