I've successfully configured a remote build cache node for Kotest, and it works wonderfully. However, the GitHub Action cache is still incredibly overloaded. Build Cache is disabled local cache on CI, as is recommended in the Build Cache docs. Is there something I've missed, or something more than can be done?

wow, those transform caches are huge. Its possible to exclude large contents from the GHA cache: docs, docs.

Do you have

gradle-home-cache-cleanup

enabled for your workflow? If not, you might want to try that, or even better switch to the prerelease version of

v4

, which has this enabled by default.

uses: gradle/actions/setup-gradle@prerelease/v4

.

It looks like that is indeed enabled, on the lastest @v3 setup-gradle. From the GHA cache screen & GHA logs it looks like those transforms aren’t stale - they are all updated as part of the build.

It would appear that the

~/.gradle/caches/transforms-4

directory isn't being cleaned up correctly. https://github.com/kotest/kotest/actions/runs/10043939320/job/27757913559#step:16:69

Does that apply even though the cache cleanup is done with a provisioned Gradle 8.9? Is it the case that < Gradle 8.8 doesn’t provide enough cache metadata to enable transforms cleanup?

Thanks! Let's see if the 8.9 PR can be merged before the cache hits 100GB...

GItHub Actions cache entries are immutable. So every time the set of transforms change, we have to write a new entry. I bet the vast majority of that usage is made up of stale entries that will eventually expire.

And artifact transforms can't use the remote build cache?

hmmm. if a change to the transforms results in a new cache entry then there isn’t really much cleanup to do - the older entries will age-out of the cache. BUT: that leaves 4.xGB per-transform-cache-entry, which seems excessive, especially given that a kotest GHA build writes multiple transform cache entries, quickly exceeding the 10GB limit.

buildSrc vs build-logic doesn't really matter much if a convention plugin is re-used in all subprojects :)

it does wrt classpath - buildSrc gets prepended to the classpath, such that every change in buildSrc changes all classpaths. build-logic is more selective based on applied plugins etc.

true, but in Kotest all of the subprojects apply the same convention plugin, so changes in build-logic affect everything

understood, thx.

and yes, I agree about the transforms not being that significant

agreed, the cache overloading is a Gradle-cache-cleaning things, unrelated to whether the transformers were isolated. Whenever I need to say unzip something have waffled between a transformer vs a ziptree (or a custom task), often find the latter to be “less magical”.

if a change to the transforms results in a new cache entry then there isn’t really much cleanup to do - the older entries will age-out of the cache. BUT: that leaves 4.xGB per-transform-cache-entry, which seems excessive, especially given that a kotest GHA build writes multiple transform cache entries, quickly exceeding the 10GB limit.

There are multiple different "caches" here. The

~/.gradle/caches/transforms-3

directory is where the generated transform outputs are placed. Even with the build cache disabled, the outputs are saved there. The

setup-gradle

action then zips up the entire

transforms-3

directory and saves it to the GitHub Actions cache. The core problem is that the old transform outputs are never cleaned up, so the transforms-3 directory grows over time. This is why each of the cache entries is so big. With Gradle 8.8, the transform outputs were moved to

caches/8.9/transforms

, and these are now cleaned up like other things in the Gradle User Home. So before saving to the GitHub Actions cache, the

setup-gradle

action will attempt to purge unused entries. I suspect/hope that the resulting GHA cache entry will be much smaller than 4Gb.