I've been thinking about how the nf-core CI is tes...
# team-maintainers
b
I've been thinking about how the nf-core CI is testing against both stable and latest edge. In general, I don't think every pipeline should be doing this across the board. I guess it's being used as an early warning signal, but in that case you should only need to do it for a few flagship pipelines. If shouldn't cause a panic every time it happens (like the Java issue in 24.11.0-edge). Latest edge is by definition not stable so it doesn't make sense to make people conform to latest edge just to satisfy their CI tests. I would just do it for e.g. fetchngs/rnaseq/sarek
party this 4
j
@Phil Ewels @Maxime Garcia or whoever were pro testing against edge I would be happy to just test stable
m
I am pro testing against edge. But not giving failure in CI
I think my ideas go along with what Ben was saying. It's really more Phil who was pro testing
p
Yup, because I feel like there have been cases in the past where problems on edge have been pipeline-specific.
I feel like we can avoid catastrophic org-wide failures by doing better at canary testing on the Nextflow side (eg. Checking a pipeline from the nf-core template regularly / before releasing edge)
And also on the nf-core side, by building Nextflow from source nightly and running on the template
I think that these two approaches (test more) are better than dropping edge testing on pipelines (test less)
n
Maybe we could only test on the edge version when trying to merge
dev
into
master
? It would halve our CI tests without breaking any released pipelines
👀 1
j
That's a veyr nice idea
m
I like that idea
But I think we should keep the edge tests on at least some selected pipelines
j
Yeah that's fine
But this way @mashehu won't shout anytime any of my pipelines run tests 😆
🤣 3
p
Yeah I'd be ok with that 👍🏻
m
Then I'm sold
b
Checking a pipeline from the nf-core template regularly / before releasing edge
we do test a bunch of nf-core pipelines before making a release I recommend doing a similar thing in nf-core. Test only a few pipelines and/or test against edge less often (like merging to master or cutting a release) so that it isn't a constant roadblock
m
I'm happy with testing on all, I just don't want tests with edge be a hard block either. For me CI should report warning if there is failure with edge, that way we know about it, and we can continue working
👍 1
k
I agree with Maxime. I like the idea of edge tests raising a warning when merging to dev and an error when merging to master.
🙏 1
this 1
m
In general I agree, the only problem is that GitHub Actions don’t really have a good warning mechanism… the only thing I can think of now is post a message to the PR with a warning. or were you thinking about something else?
m
Yeah, but then we'd have to rethink the whole logic
m
We really need to find a solution for this…
j
This (practically) sounds sort of similar to the issue with the whole AWS megatests thing, where we want two runs: one 'testing' execution (make sure it will run properly), and the 'final' execution right on actual release (for the website)? What is the route we are going there?
m
with the next template update, we run awsfulltest for every PR review and on release.
m
awsfulltest for every PR review?
j
Nah that's huge
with the next template update, we run awsfulltest for every PR review and on release.
That's going to be VERY expensive
Or do you mean release PR review?
m
I forgot to mention for PRs targeting `main`/`master`
👍🏻 1
j
Ah OK - that's better 😆
m
I'm relieved too
m
help with the wording in the PR comment appreciated
j
I think it look sgood - do you think it's worth extending to leave a comment that 'latest-everything' maybe unstable or somthing?
m
good idea