Hi all, I'm a software engineer at Seqera, and for...
# atacseq
n
Hi all, I'm a software engineer at Seqera, and formerly a bioinformatician, and I made a small issue and PR of 162 lines of syntax change to drop the number of nextflow strict syntax warnings to 0. It was AI-assisted using our new agent, but human-reviewed, and end to end tested.
👀 3
m
Hi Nico, could you join the team in #CEB982K2T so that there is no need to approve for the CI to run on any on your I hope many future PRs
✅ 1
j
I approve it, but make a small suggestion for the changelog message
👍 1
Thanks @Nico De Veaux!
m
I made a suggestion on Jose's suggestion
🤣
n
Thanks for the quick feedback
I also looked at updating modules to get past 16 strict syntax warnings on them, and then that led to a larger project, due to calling signatures changing on them
j
Thanks, I c/p it and did not modify the number 😓
n
I decided to scope that to a larger upcoming issue
j
For the newer PR, it would be nice that they are as atomic as possible
party this 2
and @Kevin Brockers has been working in updating #CFP55NZ5G modules lately, so it might be worth to take a look on how this was done there, as this pipeline are closely related
n
Roger that. Thanks for the feedback
j
I wanted to update #CE5EL6326 but did not have the time
n
Absolutely, makes sense. Now that there's a opinionated linter with strict syntax this is a perfectly suited low-hanging fruit for agentic engineering, so I'm using this as a proof of concept, but will still be the human-in-the-loop.
👍🏻 1
m
If you want to work on more pipeline, we always welcome people in sarek
😅 1
😆 2
auxc4v.jpg
😅 2
maxime 1
k
😂 the sarek hunting game.
m
auxd35.jpg
n
I have made a follow-up issue to update nf-core modules: by doing so, the number of strict syntax warnings drops to 0. https://github.com/nf-core/atacseq/issues/447
🔥 1
The full validation run is still on-going. If its successful, I'll open a PR.
❤️ 1
Ok, it seems to be passing all the checks (I ran 4 GRCh38 samples as well): github.com/nf-core/atacseq/pull/448 . However, I'm keeping it in draft until template version 4.0.3 (github.com/nf-core/atacseq/pull/449) gets merged
👍🏻 1
The biggest change caused by
Copy code
modules update --all
is, from what I could see, that the
subread
module changed the most from 2.0.1 -> 2.1.1. So I made a separate PR handling that update, PR 452, to count consensus peaks separately for single-end and paired-end libraries.
github.com/nf-core/atacseq/pull/453 resolves the ro-crate conflict of PR #449 (made by bot) per the PR #443 pattern by @Jose Espinosa-Carrasco
k
@Nico De Veaux, I think Jose is on vacation. I had a look at your PR #453 could you also update the changelog? Then it should be good to go
n
Thanks, updated it
🔥 1
k
I think there was a warning for the newest nextflow version tests, potentially we should dig into it, this week is a bit chaotic on my end...
n
I see that now, taking a look at what might resolve that. There's also a warning about "Newer version of the nf-core template is available."
Strange, because it passed August 3rd, on 26.07.0-edge.
I think the cause is unordered file collection. The failing test is tests/skip_trimming.nf.test, and so the shuffling can be unlucky. I will make a follow-up PR to see if I can get that test to reliably pass.
k
Thank you for looking into it!
n
I did a re-run, and it succeeded (at 17:53 CET), so it's a flaky test.
👍 1
I might need to re-generate snapshots. Here is the new PR: github.com/nf-core/atacseq/pull/454
👍 1
k
Looks good now. I like the sorting idea, even though it adds a bit of overhead.
✅ 1
n
Alright! Now that the template merge is in, it unblocks the big
nf-core module update --all
PR I made: github.com/nf-core/atacseq/pull/448.
🔥 1
k
Nico you are on fire 😉 I'll look at it later, the morning is a bit tight already.
n
Sorry for the radio silence, I was on break, but I'm addressing these changes today.
❤️ 1
Given how many lines of changes this PR 448 has, I'd like to run another few samples through the pipeline. I already did that for a Osmotic SRR dataset, and I picked a few GRCh38 samples from a research paper of mine from a few years ago. I'm curious if there are any other samples that have been used before for benchmarking, that anyone could recommend.
k
Mhmm, I think you could check which samples are used for the full-size dataset test ran upon release. But before doing those large scale computations, how do the test snapshots behave? Do you spot obvious problems?
n
No problems on the test snapshots. The only obvious problem is that subread is a newly imported module, and subread changed its behavior in 2.1.
✅ 1
And as a result, I made a new PR, #452, github.com/nf-core/atacseq/pull/452, that explicitly handles paired-end vs single-end BAMs. (it now splits the cohort by endedness, counts each homogeneous batch with the correct pairing flag, and merges the per-batch matrices into a single consensus table for DESeq2 and MultiQC).
I think I want to merge #452 into #448 instead of having it be 2 separate PRs: one benefit is that we don't have to pin subread to version 2.0.1 and then to a later version
k
Hi Nico, I really like your enthusiasm 🙂 But I think we should stick to a more atomic development / PR strategy. It get's pretty complex and difficult to review ~500 changed files. I'm getting a bit confused 😉 #448 is the modules update? Which was good so far, only the changelog updates were missing, right? #452 is the paired-end and single-end BAM handling? But why are there >200 files changed, is that a copy of #448 plus some additional changes? Tbh, allowing the handling of SE/PE experiments in the same run is a big conceptional change and something we should discuss in a bigger cycle. You may introduce batch effects when doing DE analysis of SE and PE data together, starting with read aggregation (counting both reads of a PE experiment or only a given fragment once). But this actually has an even bigger impact... e.g on PCR deduplication or computing coverage tracks, where SE reads can be shifted by a theoretical fragment length, while in PE data you can use the center of the fragment. Personally, I feel like we should keep SE/PE data separated and don't allow for mixed library types but this would be something up for discussion. Haven't checked the pipeline, not sure what is currently allowed. What about the following: I will check your changes in #448 and merge it. You raise an issue for #452 and we wait for @Jose Espinosa-Carrasco feedback? E.g could be discussed in #C0C30LAV0G7 and then we get more feedback in the general channel?
party this 1
n
Thanks for the feedback. That makes sense. And yes, 452 is big because it has 448 merged in.
The diff between PR #452 to #448 is around 200 lines of code (1000 lines in total when including shapshots and fixtures). You can see the diff here on my fork.
#452 exists to fix a problem #448 creates (pinning subread to an old version because the latest subread rejects mixed single-end/paired-end input)
But maybe no one is running SE and PE together? I don't have that situational awareness, I just noticed that tests were failing and that behavior had changed, but maybe its an unusual edge-case and its simpler to change the spec to not have runs that mix both.
In summary, I completely agree with you, and I'll keep #452 in draft mode, and I'll make an issue describing the situation so it can be debated.
✅ 1
I think in 2025 @Jose Espinosa-Carrasco merged a PR into the chipseq pipeline so that mixed SE/PE groups aren't allowed. But atacseq still has tests to support SE/PE mixing: the mixed atac-seq test samplesheet
k
Perfect! I just approved #448, feel free to merge
✅ 1