Hi all! :wave: Not sure if this is the best chann...
# community-support
r
Hi all! ๐Ÿ‘‹ Not sure if this is the best channel to ask about the Github Gradle Action. Particularly about the feature to remove unused files from Gradle User Home before saving to the cache. We added the flag
gradle-home-cache-cleanup: true
(and we can see in the logs that is working), but the results are a bit disappointing. It's true that the home cache entry dropped from
3.6 gb
to
2.3 gb
in our main project. But we verified that three months ago this was smaller than
1 gb
. So I wonder if everything not used is actually being cleaned up. ๐Ÿค” I also tried in a sample tiny internal repo and after enabling this flag, we still see the cache growing after some minor changes that, according to the docs, should not happen (like after bumping gradle). More details in the thread. ๐Ÿงต
Results after enabling this in the sample repo: - Last commit before enabling the flag: Home 29 MB. - First execution with the flag: Home 29 MB. - Change a source file: Home 29 MB. - Bump from Gradle 8.4 to 8.5: New home saved with 56 MB. - Bump some dependencies: New home saved with 88 MB. (The dependencies entry, which is separated, grew from 249MB to 255MB, this makes sense, but why does home need to grow here?) - Bump from Gradle 8.5 to 8.6: New home saved with 203 MB. ๐Ÿคฏ - Newer commits: Home remains at 203 MB (is saving new entries, but keeping the size) This is using
gradle-version: current
in the action setup as it's a requirement for the cleanup. But all our builds are using the gradle wrapper defined in the project (this should be fine, but sharing just in case this impacts somehow). After that I've added the debug flag and noticed something that catch my attention in the logs. The Gradle User Home, after extracting common artifacts that the action stores in separate entries, still lists two
transforms-#
folders
:
Copy code
227M    ./caches/transforms-3
493M    ./caches/transforms-4
I don't have much knowledge about these internals of Gradle, but my understanding is that these folders in the cache increment the number whenever the artifacts are not compatible with the new gradle version. So are they still required after a gradle bump?? ๐Ÿค”
I downloaded the logs generated with the debug flag, which are quite long so no worth sharing here. But this is a summary of all the entries that saved in the cache from the home folder after cleanup. I grouped them by the two main levels and the number of the left is the count of hits from the logs
Copy code
2 .gradle/.setup-gradle/
   1 .gradle/caches/
 210 .gradle/caches/8.6/
   1 .gradle/caches/jars-9/
   4 .gradle/caches/journal-1/
2422 .gradle/caches/modules-2/
18908 .gradle/caches/transforms-3/
8946 .gradle/caches/transforms-4/
   1 .gradle/notifications/
   2 .gradle/notifications/8.4/
   2 .gradle/notifications/8.5/
   2 .gradle/notifications/8.6/
As you can see, also the notifications folder keeps data from Gradle 8.4 and 8.5 even if we are running Gradle 8.6. The contents on this folder are insignificant, but I think they prove that some legacy data still remains in the cache.
d
Hey @Roldan Galan. Thanks for reporting this.
The
gradle-home-cache-cleanup
support relies entirely on Gradle's built-in cleanup mechanism, but the action will touch certain files and "age" certain other to force the cleanup to run more frequently.
For this reason, I'm entirely unsurprised that the
notifications
directories remain in place. Gradle doesn't do any cleanup of these files.
๐Ÿ‘ 1
Can you please raise an issue at https://github.com/gradle/actions with what you're seeing? A few things I can note at this time: โ€ข Instead of using
gradle-version: current
in your action, it would be preferable to code the same version that is used in your project. Fewer Gradle versions being used means retaining fewer files. โ€ข When you enable cache debug logging, the most interesting messages will be found in the "post-action step". Search for
Gradle User Home (directories >5M): after extracting common artifacts
and you'll see the main directories that are contributing to the Gradle User Home cache entry size. It would be very interesting to see if these grow over time, or only when updating to a new Gradle version.
r
Instead of using
gradle-version: current
in your action, it would be preferable to code the same version that is used in your project.
Usually we adopt the latest Gradle version quite quickly after becoming current so they will be usually the same (also we have automation with Renovatebot, and this would require us changing this value manually after each bump). We never needed this parameter before in the gradle action, but is required for the cleanup. So when it was added in the main repo it was already the matching the one used in the project. Current is now 8.6 anyway, so it matches with the latests runs in the sample repo from my second message.
โ€ข When you enable cache debug logging, the most interesting messages will be found in the "post-action step". Search for
Gradle User Home (directories >5M): after extracting common artifacts
and you'll see the main directories that are contributing to the Gradle User Home cache entry size. It would be very interesting to see if these grow over time, or only when updating to a new Gradle version.
Yes, the information from my above messages about the size of the two different
transforms-x
folders comes from that section after extracting common artifacts. So this is what is being currently stored by the action after cleanup
I've just tried something else: We just removed all the cache entries, so the runner started with an empty cache. This is the main repo (the one whose home cached entry was
3.6 gb
and was reduced to
2.3 gb
after enabling the cleanup three days ago). The new entry for the home is just
457 MB
. Then these builds only produce/require 500mb. So there is a lot of legacy data that is being preserved from one run to the other that is not properly captured by the cleanup mechanism. ๐Ÿ‘€ Some other data for context: โ€ข We started using Github Actions 4 months ago, with the gradle action already integrated. We never looked into the caches it was producing until now. โ€ข 3 months ago the home cache had
1 gb
(eldest data I could find) โ€ข It was using Gradle 8.4 then. We bumped to 8.5 couple of months ago. And to Gradle 8.6 two weeks ago. โ€ข Besides Gradle version, something else must contribute to this increase, as we see a constant increase week over week, even without bumping gradle. But many engineers contribute to this repo, so is not a controlled environment and is hard to point what causes these increases. โ€ข Cleanup was enabled three days ago. First screenshot: Last commit that used the already existing cache (Gradle User Home size: 2367 MB) Second screenshot: First commit without cache (Gradle User Home size: 456 MB)
In any case, I will raise this issue in the GH gradle actions repo later with the most relevant info from these messages. I also will keep a close eye in the next Gradle bumps and report back if we see a significant increment. ๐Ÿ™‚
d
So do you think the increase is mostly due to the
transforms-*
directories? Or do you see other large + growing directories in Gradle User Home? It would be good to know what things I should investigate. Is this an Android project? Kotlin? (I mostly tested cache-cleanup with standard java projects, and these don't make much use of Transforms).
r
Yes, this is an Android project, 98% in kotlin. From my initial investigation, yes, the
transforms-*
directories are most likely the main reason for this increase. So it would be the main place to look in my opinion. Let me know if you need any other data I can facilitate from our current setup ๐Ÿ™‚
d
Thanks for reporting. No more information required, but please still raise an issue. I'm not going to have time to investigate until Mid-March, when we should have further data from the upcoming Gradle 8.7 release ๐Ÿ™‚ .
๐Ÿ‘ 1
r
Sure I will raise the issue indeed when I find some time between today and tomorrow ๐Ÿ‘
Raised, thank you for your support @Daz DeBoer! ๐Ÿ™Œ Feel free to reach out if you need other details :)
d
Thanks for filing. As I mentioned, I won't be able to investigate further until I'm back from vacations in mid-March. (And have dealt with everything that arises while I'm AFK ๐Ÿ™‚ ).
๐Ÿ‘ 1