https://gradle.com/ logo
Join Slack
Powered by
I have a build with 2 apps. I run `./gradlew chec...
# community-support
v
I have a build with 2 apps. I run 
./gradlew check
it runs all the checks everywhere If I run 
./gradlew :app1:check
it runs checks only for 
app1
directly How do I have it run checks for the 
app1
+ its subgraph of dependencies? (other than manually maintaining a whitelist of projects to run) or is this against some gradle philosophy? bazel has this
i
Copy code
./gradlew -p app1 check
I'm not sure how intended this is though. For example, if you have a 
settings.gradle.kts
in 
app1
, the behavior is not what you want.
v
how does that help? check runs everywhere, not just the subgraph? or what am I missing?
i
The syntax is: ā€¢ 
<task>
: Run that task in the current project and all subprojects that have it ā€¢ 
<project>:<task>
: Run that task in that project specifically, fail if it doesn't exist By telling Gradle that 
app1
is the root project, it runs only for 
app1
and subprojects
v
really? gonna try it out
I've only found bunch of issues on this and all basically say to manage a white list which..doesnt scale..
v
I'm not sure how intended this is though. For example, if you have a 
settings.gradle.kts
in 
app1
, the behavior is not what you want.
If you have a settings script in 
appl
and also include it (not includeBuild) in the settings script one directory higher, that is anyway a highly problematic setup that should never be done. One project must always only be part of exactly one build directly.
i
To be fair, 
-p
is not intended to do that, so there may be some weird things I don't know that make it a bad idea.
v
Why do you think it is not intended to do that?
It should be the same as if you do 
cd app1 && ../gradlew check
which should have the same effect
i
My understanding is that 
-p
is intended to target a directory that contains a 
settings.gradle.kts
Honestly I was surprised that if you 
-p
a directory that doesn't have a 
settings.gradle.kts
, it uses the one from the current directory instead.
āž• 1
v
As long as you only need that behavior from commandline, it should be perfectly fine. If you need it for depending in the build script, then you need to model the dependencies.
v
I have just a normal build with one settings.gradle and multiple apps, which do share dependencies, but also not, so those not in the app1 subgraph should not be tested because..no reason to (?)
v
My understanding is that 
-p
is intended to target a directory that contains a 
settings.gradle.kts
That's not my understanding. Afair it is like I said like cd-ing into that directory and call Gradle from that directory.
v
Im not clear, if its just 
cd
, how does it scope down the graph?
i
So @Vlastimil Brecka actually it looks like cd then run is the best solution. I didn't know that worked, that's useful
Copy code
cd app1
../gradlew check
will run 
check
for 
app1
and all subprojects, TIL
v
As I said, it should be exactly the same
The docs for 
-p
say 
-p
, 
--project-dir
Specifies the start directory for Gradle. Defaults to current directory.
So it behaves like if you start Gradle from that directory
v
why does this work? šŸ˜„ is it somehow file system related? does it rely on all the dependencies be in that folder?
v
It then searches upwards for a settings script
As you said, if you also have a settings script in that subdirectory, it will behave differently, but just the same with 
cd
and it is highly bad setup.
m
will run 
check
for 
app1
and all subprojects, TIL
But @Vlastimil Brecka doesn't want subprojects, they want dependencies
v
`cd`ing to a directory and running Gradle from there should be identical to using 
-p
and specifying that directory. If it does not, that would be a bug
m
(I agree with @Ivan CLOVIS Canet this is super confusing. I always thought you needed to run Gradle from a directory that contained a settings script)
v
But yeah, @Martin is right, that will check all 
:app1
and descendent projects, not dependencies that are not descendents
I always thought you needed to run Gradle from a directory that contained a settings script
In former times settings scripts were not even mandatory, so no. šŸ˜„ It was always like that, Gradle searches for the settings script upwards in the directory hierarchy and in former times also a bit sidewards for "includeFlat" setups.
v
Maybe step back, is this something I should want? Or is running everything idiomatic & keeping a build cache on CICD so only changes run?
m
A lot of Gradle is "always been like that", that doesn't make it less confusing for someone that hasn't been there since 1.0
But yea @Vlastimil Brecka I guess a good remote build cache is what you want
v
ughhh maybe later šŸ˜„
m
šŸ˜„
v
I know @tonyā€™s DAGP has some mechanism to pull out a subgraph
m
Is the issue that you want to save time on CI?
If yes, caching is definitely the solution
v
yes, android release builds
i
I do recommend having incremental checks setup correctly and just always running 
./gradlew check
for the entire build, the first build may take longer but everything should be cached And if your up-to-date checks are not correct, you have worse problems (correctness problems)
šŸ‘† 2
v
also, saving carbon šŸ˜„
v
DAGP will not really help here, unless you call Gradle twice, first to generate the DAGP report and then after parsing it to get the projects to check
But yeah, I'd also recommend having a proper build cache setup. That will save you most headache
v
I'm yet to decide if I like incremental builds for releases on CICD, I worry about it hiding issues
v
And if it is a remote build cache, you can even benefit in local builds
v
last time we discussed you were kinda against it
v
using build cache != running incremental build
m
Aren't all builds incremental anyways
v
I still personally think CI builds should be clean builds. But using build cache is fine.
v
how is that different? isnt that corruptable the way incremental ones are?
v
@Martin kind of, but if you run clean builds with a fresh checkout, effectively they are not
m
They are incremental but they start without local state
But they have the remote build cache state (if it exists)
Remote build cache is great, saves sooo much time and carbon
āž• 2
v
how is that different? isnt that corruptable the way incremental ones are?
Theoretically, yes. But a cacheable task should be written properly for being properly cacheable. If that is faulty then yes, you can get bad results. But this is way less likely than the other problem.
It is of course always a trade-off between saving time and risking wrong results
No software is bug-free
But usually tasks that are marked as cacheable should hopefully be properly written cacheable
m
It's not that hard to properly configure your remote build cache these days. And if you get a corrupted output, you might as well consider it like any other bug and fix it
v
still I think having some sort of querying mechanism is reasonable
v
Querying what?
v
subgraph of pdenendecies of the given project I mean I do it now & it works, it's just I get yelled at by Isolated Projects
v
There is an API to get the resolved dependency graph in a CC (and hopefully also IP) safe way. Your problem is, that you would need this information at configuration time to model the dependencies and that would probably not be good.
v
(btw would the build cache be effective in my case where releases are every 2 weeks?) in inbetween theres lots of commits daily?)
v
If you have some convention plugin you apply to all projects in your build, you could make a setup that works probably. Something like ā€¢ add a consumable configuration in all projects that has the 
check
task as task dependency ā€¢ add a dependency scope configuration that depends on the current project ā€¢ add a resolvable configuration that extends the dependency scope configuration and requests the variant of that consumable configuration ā€¢ depend on that resolvable configuration from the 
check
task That way I think (but did not try) you should get the behavior you want.
m
btw would the build cache be effective in my case where releases are every 2 weeks?) in inbetween theres lots of commits daily?
Your daily commits would contribute the build cache
Unless your release build has a different input that invalidates your whole build (in which case I'd probably try to change this)
šŸ‘† 1
v
right but my basic checks for the daily commits only run debug buildtype
for perf reasons, as not to run R8 everytime
wouldnt that invalidate it?
v
(btw would the build cache be effective in my case where releases are every 2 weeks?) in inbetween theres lots of commits daily?)
Heavily depends on how you set it up, but usually yes. Because say you have 20 projects and a commit changes code of 5, then for the other 15 the build cache can be used (unless there was an API change in the 5 where some of the 15 depend on, then those need to be recompiled too of course. And on release most of the cacheable tasks should be reused from cache. Each build should benefit from the cache unless you change everything or very central things.
m
Make your whole app a lib with a single variant. The debug/release variation only need to happen in your app module and change very little things usually (icon, debug menu)
Basically make your app module as thin as possible
Everything else gets reused
v
basically difference between the standard checks and release checks are the android build type (debug vs release), thats all
v
Like Martin said, it depends on whether the cacheable tasks in question have the build type as input or not. As long as a task with the same inputs is run, the build cache can be reused.
Even if you have two compile tasks that compile the same files, the second should benefit from the first already filling the cache
Not that this would make sense, just generally
v
I see but to answer the original question: No there is no proper supported 1st party way of doing this - and one should use caching to mitigate the effect of executing everywhere?
v
Yes, there is no built-in way I'm aware of to run a given task in a given project and its project dependencies.
v
thanks! OT: do most really have build cache on CICD? I was under impression it's a bit taboo
v
Never heard any taboo for that. Especially as without having build cache on CICD, remote build cache makes no sense. Because almost any setup I have seen or done so far fills remote build cache only from CICD and reuse it read-only on local builds. Otherwise local builds could easily poison the remote cache, even intentionally.
v
Im not really a candidate for remote cache, clean local builds on macbooks are under 5mins, so not worth it it just the cicd machine is crappy and also I was wondering how to scale it
okay so massive monorepos do really just 
./gradlew build
and call it a day on their cicd yaml config? šŸ˜„
v
Depends on what you call "massive" maybe. šŸ˜„ The biggest build I maintain personally actively has 60 projects, or when building with docs 145 projects.
v
does it have multiple "apps" in it? i.e. projects would not be in the "slice" of the graph during building one of the apps?
v
Yes and no. Actually the measure is only halfway right anyway, because all of the 60 projects have multiple source sets, at least 6, some more. And you could theoretically only build the backend or the webclient part or the rich client part and so on. But we always build everything together on CI/CD.
t
> okay so massive monorepos do really just ./gradlew build and call it a day on their cicd yaml config? I haven't read the full thread, but I was `@`d and scrolled to the bottom. No, massive monorepos do not do this. How big is your repo? At Cash we had over 2000 modules (all Kotlin/JVM), with hundreds of separate apps. We had a complex CI setup that built only what was necessary based on the changes coming in. PR duration was generally under 10min. We also had a build team of 4 senior engineers (myself included) focused entirely on achieving this goal. And this is an aside, but this repo was more performant than a Java monorepo built with Bazel that a sibling team managed.
v
noob question but didnt you at that point reimplement gradle? isnt it build systems job to figure out whars changed and needs to be rebuilt?
m
Gradle doesn't know about "commits" so I guess it has to scan the whole working tree to find out changes, which is a lot heavier
t
reimplement gradle
gradle has some deficiencies. So, yeah. With a big enough project supporting a large enough cohort of product developers, it makes sense to do this. As I said, we had a team of four senior people (expensive!). But we also supported 500 developers on a flagship project (very very expensive!)
if you think only technical considerations matter, you're missing an important element. Social, economic, and even political considerations come in to play when deciding how and where to deploy engineering effort
v
i mean yes but I doesnt compilation avoidance work im such way that is executes everything in thevgraph anyways, its just that most are up-to-date?
t
that is still very slow by comparison to doing nothing
however, if you have a team of one person, then your cost/benefit analysis will be different
v
im just thinking out loud, gradle doesnt really calculate what needs to be rebuilt in terms of the graph, or does it? suddenly i dont know what comp avoidance meams
can you guesstime how long would it take for it to execute even if most are up to date, at your scale? (intuition builder)
t
no. You didn't even answer my question about how large your build is šŸ˜„
v
400 projects, 3 android apps, rest are jvm projects where possible, otherwise android libs, and im looking at kmp
t
yeah I mean, you can def make your CI faster than just relying on gradle's work avoidance features
v
and each app is released independently hence my urge to handle just the given subgraph
t
you could write code to do that by parsing your version control system (git)
v
isnt that what sort of you plugin does already? i.e. to have my plugin include your core as a library?
t
we're talking about two different things, though related first thing is, knowing what to build based on the incoming changes (e.g., all the entry points). For this, git second thing, is, translating those entry points into the full graph. Various ways to do this. You could invoke gradle to build the graph, but then you're invoking gradle twice. At Square/Block, we had custom tooling that was sub-1s that did this
Spotlight is the open source version of the second thing, sort of
v
I see, but youd opt for this only once even build cache (on CICD) becomes slow?
t
we had years of experience to know in advance CI would be slow. The cash backend project I'm referring to was greenfield. So we took our years of prior experience and built it right the first time
šŸ’Æ 1
v
Did you have build cache layered on top of it anyways (on CI) because of remote cache?
Idk im still kind of apprehensive about build cache on ci, it relies on all the 3rd plugins not to messup etc
t
iirc, we used build cache on PRs but not main branch. Faster PRs, but more safety for main branch builds that would actually deploy things
thank you 1
Open in Slack
PreviousNext
Powered by