https://gradle.com/ logo
Join Slack
Powered by
This message was deleted.
# plugin-development
s
This message was deleted.
šŸ‘ 10
b
By reading this I learned about 
all
vs. 
configureEach
. I was using the latter for everything related to the 
TaskContainer
but not for custom containers.
@tony what do you think about 
project.layout
vs. 
project.file
? Any recommendations here? Iā€™ve started only using the former.
t
ah, good call out. That's an example of something I should add to my "rules". I almost exclusively use 
ProjectLayout
these days and always look for it in PR reviews. In particular, I heavily discourage people from using 
new File()
somewhat amusingly, I actually had occasion to use 
all
just recently, in dealing with a custom domain object collection that I specifically wanted eagerly realized for use at configuration time. šŸ™ƒ
m
the 
all
vs 
configureEach
is not that simple. If everything becomes lazy, then nothing realizes nothing and nothing happens. So this rule works great for tasks or data which needs to be realized only if a task is going to be executed. Otherwise it can lead to surprising behavior: https://github.com/gradle/gradle/issues/19955
t
honestly this really strikes me as an API design problem. If there are container types that should be eager and container types that should be lazy, that should be encoded in the type, rather than using a common parent that can do both, and then expecting non-expert devs to remember when to use which
āž• 3
m
I agree. But I also think that the general agreement, like you wrote in your blog post, is that everything should be lazy. I think the biggest problem is that nobody really knows how to make that happen.
Since the entry point of Gradle is task names, a lazy model which can register tasks lazily is a problem, because nothing would trigger the realization of the model.
t
Yeah. This came up recently here, where I ran into another surprising behavior when using 
all
to iterate over a custom container for the express purpose of registering tasks if some predicate were true on a given item
m
oh that's a different issue, the ugly "creation action" which isn't called early enough. Definitely don't like this behavior, which is counter-intuitive (and actually very problematic)
t
yeah, I'm really just saying I agree that not everything can be lazy. When I started on that problem I knew I needed the container realized immediately and so used 
create()
and 
all
. Rules <> guidelines šŸ¤·
m
I wish some of this (excellent) advice was surfaced in the IDE itself. Something like 
@DelicateEagerApi
maybe? As it is now it's really hard to remember all of it and pitfalls.
šŸ‘Œ 1
c
Iā€™m personally skeptical of the ā€œeverything should be lazyā€ edict for speeding up configuration time. Unless you have a pre-existing issue with how long it takes to configure the build, going on a ā€œconfiguration avoidanceā€ crusade just churns the code, risking introducing new (and harder to diagnose bugs), and all for minimal gain, since you didnā€™t have excessive configure times before.
t
out of curiosity, what kind of projects do you generally work on? Android, Java, Kotlin, something else?
c
Moderately sized Java middleware - with bits of NPM and Docker on the periphery. Iā€™m aware that configuration times are significantly higher for complex Android projects, and in those situations I think this is probably something worth pursuing - but I think itā€™s heavily context sensitive. For plugin authors I think itā€™s worth thinking a little about where your plugin might be used before going all in on the config avoidance. In my builds Iā€™m looking at config times in the 10s of seconds, and assembly times on the order of 2 minutes. Full test-suite runs are 30 minutes and up. In that context investing any time in config avoidance is a false economy.
Iā€™d trade the entirety of the config avoidance feature for a work-stealing algorithm to solve the long pole effects I see in my big test runs. That could probably save me 5 minutes on each full build.
t
the build I work on day to day has a config time of 90s with a warm daemon. So, every single build "wastes" 90s. It used to be a full two min. Config avoidance is a real problem for Android builds. Actually building varies by vertical, can be a couple min to ~10min. Unit tests tend to be very fast, UI tests much slower but generally only run on CI. I can say I'd never use a plugin whose author didn't make an effort to minimize the impact on my project's configuration time. Gradle Enterprise already supports what you've suggested: it can shard your test runs across many machines. Not sure if you're a customer
c
Noā€¦ I get for Android that configuration times dominate, and for so for most/many plugin authors config avoidance makes sense. I just think that advice may not apply to all plugins, especially if itā€™s somewhat internal, and even less so for local plugins. Regarding work-stealing I really mean this: https://github.com/gradle/gradle/issues/3955 or this https://github.com/gradle/gradle/issues/2669
fyi warm daemon configuration time for a typical build here are ~2s:
Copy code
$ loc
--------------------------------------------------------------------------------
 Language             Files        Lines        Blank      Comment         Code
--------------------------------------------------------------------------------
 Java                  3323       419271        61743        50937       306591
 JSON                    24        28826            0            0        28826
 JavaScript             105        20115         1915          986        17214
 XML                    115         3104          220          462         2422
 Sass                    40         2587          151          184         2252
 HTML                    33         2099          204          148         1747
 Batch                    6          939          129            0          810
 Plain Text              14          597          103            0          494
 YAML                    10          475           11           22          442
 CSS                      5          478           48           30          400
 Markdown                10          416          171            0          245
 Awk                      2          226           30           36          160
 Bourne Shell            11          236           27           99          110
--------------------------------------------------------------------------------
 Total                 3698       479369        64752        52904       361713
--------------------------------------------------------------------------------
m
I'd love to see a metric like configuration time per executed task.
c
Just ran a build-scan over an assemble: Total configuration time 2.996s, Time spent executing tasks 2m 31.905s, 731 tasks in 115 projects
šŸ¤Æ 1
ł
Makes you think just how heavy Android plugin must be, I wouldnā€™t expect under 3s configuration with over 100 projects šŸ˜® I suppose you donā€™t use Kotlin either?
m
Android creates an order of magnitude more tasks. That's why I think that it would be nice to have stats which tell the configuration time per executed task, and per configured task (because they can be much more tasks configured than actually needed during a build). Such a stat would also give an idea if configuration time grows linearly with the number of subprojects or not. I suspect not.
t
CĆ©dric, config time just running help is 90s.
m
But running 
help
configures everything
t
are you implicitly asking about the impact of configuration on demand?
m
Not really. We know that configuration times are much higher in Android than they are in any other ecosystem. But we don't have a good indicator to tell if it's simply because there are more tasks configured, more tasks executed, or because Gradle doesn't scale linearly with the number of tasks/projects and ends up thrashing.
t
configuration is a serial process, so don't we know that, by definition, there is at least a linear component to how it scales on the number of projects?
m
In an ideal world, having 2 minutes configuration time is not a problem if configuration is done in parallel with execution and that execution time is much higher than 2 minutes. But if you have 2 minutes config time for any task, even the simplest, then it's a much bigger issue showing that config avoidance isn't working as it should.
c
Are you suggesting that as a solution configuration-avoidance is treating the symptom, and we should be treating the cause?
m
I am suggesting that configuration avoidance is a required step towards interweaving configuration and execution. And that once this happens, you shouldn't suffer anymore from "long configuration times" since you wouldn't probably notice the difference between configuration and execution. It wouldn't solve, however, slow configuration, or the fact that Gradle configures too much or takes too much resources to configure. It's simply a better UX because it would start working almost immediately.
(it's perceived performance rather than actual)
t
would be nice if GE could do the data crunching you suggest out of the box. As it stands, I'd need to wade through the Export API docs and I'd still only come to the conclusion that the config time is Too Damn Long (this is a US-centric political joke)
m
dashboards!
t
what I just learned looking at our dashboard is that there are more people running 
help
than I anticipated
m
when they mean 
--help
?
t
no. It's probably because I taught people that if they run 
help
, it'll have the side effect of downloading plugins. For some reason, this step often fails during an Android Studio sync with an obscure error, but running a build from CLI is more stable
in a fully up to date build with ~200 projects, running 
:foo:bar:assembleDebug
cost 12s of configuration. This involved 21k tasks
m
that's 0.5ms/task, that's in absolute not too bad. Imagine that interweaved with execution. Barely noticeable.
t
I am very excited for a future of parallel configuration. We already have our build mostly set up for it, per the guidelines currently available on the isolated projects page
4 Views
Open in Slack
PreviousNext
Powered by