https://gradle.com/ logo
Join Slack
Powered by
About build cache on CI. When used with gitlab i n...
# community-support
v
About build cache on CI. When used with gitlab i need to set a cache key. Anybody have recommendations on how to manage this? From what I understand, is that whole thing is zipped up and uploaded into storage under the key. Which leads me to believe there is going to be cache clobbering if there are parallel feature branches triggering CI (obvious) Anybody got opinions on how to manage this? Maybe only let main branch upload the cache and features only consume?
t
go look at the gradle setup GHA logic and see if you can use a similar strategy for gitlab. they do a complex key and some partial match fallback
v
im looking at it and it seems they do just that, only main writes the cache
t
well by default yes but pretty sure that is customizable
v
im completely new to this and was wondering what the standard was, as there were some prior discussions related to this and someone said they actually didnt use the cache while on main deliberately
t
yeah really comes down to what is right for you. personally I run on GHA, but on self hosted compute, so I reimplemented the setup action to use an S3 bucket instead of GH cache api's so traffic is account local (bandwidth usage becomes cheap/free) plus it means I don't have a 10GB per repo size limit. Combined with the remote cache from develocity in the same aws subnet I opted to not have the setup cache those artifacts, only the bits required to make configuration cache work. So all jobs write to the cache but its a smaller subset using the same dynamic key fallback behavior
v
im not super clear, what artifacts do you not cache? build artifacts? ismt the required to feed the remote cache?
t
well its a blend. i cache lots of what lands in gradle home + the build directories of any 
includeBuild
projects. was really just a lot of trial and error until I got to the point where basically all the dependency downloads get cached by the setup action, and the task/transform outputs don't and lets the develocity plugin download those on demand from the remote cache during the build
image.png
v
hmm whats your motivation? to avoid local configuration time?
t
motivation is avoid internet bandwidth where possible to save on that cost. plus our corporate network situation is a mess so the more I can keep things in my aws subnet things run way faster/reliably. we use artifactory to proxy 100% of deps, but that is a route over internet so zscaler egress plus some jumps through vpc peering to get there, etc etc
v
but this is ci builds?
t
correct
local still just has to internet things. not much I can do on that, but the gradle user home makes the artifactory path mostly moot on most builds
vastly different shape for local
v
yea im just trying to parse what you mean😄 so you effectively use build cache on ci, however not the local one because reasons but the remote one?
t
well so everything gets the regular develocity remote cache config. local jobs only pull and never push to remote cache local jobs all use local cache ci jobs always push to develocity cache gha setup cache helps make configuration cache work in CI + limit round trips out to artifactory
v
whats a local job? why the distinction? do you mean just a ci job? or do you build in two places?
t
dev laptops -> local gha -> ci
v
i see, i wasn’t aware dev laptops could push cache, seems like a security issue
okay so just the final one, your ci runs do use the build cache they created, but in a roundabout way via Remote cache?
t
yup exactly
v
i see do you ever not use the remote cache as input to the build, due to idk stability of release builds or sth like that?
t
its exceptionally rare that we have to worry about that. maybe a handful of times a year we will get one or two tasks that have cache poisoning. IMO if your tasks are configured correctly you basically have zero risk of that
cache poisoning usually surfaces as weird build failures. usually from some missing reference in a ksp generated output or something
v
hmm okay so you can only get false negatives? never false positive? locally i obviously never worry, with ci im new to using build cache there, my reasoning always was why introduce such dimension but recently i thought of i would rather have it run more analysis in that time
but subconsciously i worry if the release builds are going to be correct
t
this whole stack cuts or CI job times by like 50-75% depending on the job. mixed with my new GHA setup using mostly spot instances instead of reserved or on-demand, cut the underlying compute cost by like 80%. mix in the bandwidth savings of a few thousand a year
v
my tasks are fine, its the 3rd party plugins i worry specifically
t
yeah thats totally fair. and why I am a jerk when it comes to people adding random plugins to our build
v
yea this is self hosted but yea burning theough carbon doesnt make sense if the cache is reliable guess i have to live it 😄
would make the build easier I would just 
./gradlew build
and call it a day for ci jobs 😄
t
anyway hopefully this gives you ideas you can turn into something that works well for your situation
main thing is don't be afraid to get dirty, because the layering can make a huge overall impact
v
maybe comment on that, how do you run your checks? literally just 
build
and let gradle deduplicate work?
t
my main project is a big android one. so its lots of jobs in parallel. assemble apks unit tests ui tests linters in each job I try to just invoke gradle a single time so its parallel planning can pack the timeline for me as best as possible
v
hmm why not just the build task? it includes all of what you mentioned
t
some config things don't mix well. so like you can't instrument for code coverage in UI tests in the same build that is running unit tests or the jvm jacoco agent explodes. plus each of those need like a 64 core box to run in a decent time window. so vertical scale is mostly already exhausted
for sense of scale. normal day uses 3-5k cpu cores of spot compute at any given time. UI tests get distributed on .metal instances on top of its 64 core gha instance so that is a big chunk of this too
v
so you never run release variants to check?
t
our assemble job builds all the app variants in parallel so I know at least the R8 config etc is safe on a PR
typically don't install/run those until we get to release testing though
v
by assemble job you mean 
app:assemble
? i,e including the ones assembleXRelease which do run r8?
t
yeah exactly.
Copy code
./gradlew generateArchiveScript app:assembleQaDebug :app:sendQaDebugPullRequestTeamsHook app:sendPenTestReleasePullRequestTeamsHook app:bundleProdRelease -PappVersionName="$(grep "appVersionName" gradle.properties | cut -c 16-).${GITHUB_RUN_NUMBER}" -Dscan.tag.PullRequest
the pr hook tasks will cause aab upload to internal testing. any apk/aab that gets built goes into S3 for a week so people can download them via job summary links generate archive script is sort of a bad name. that task basically just writes a file path list of things to archive at the end of the job. test report htmls etc... every job runs it
v
okay neat so you do run r8 as part of pre merge PR checks thats something i dont do because reasons
t
its slow -> expensive. have to balance your needs
v
so the build cache helps with that a lot yea? so its no longer a bottleneck?
or rather its tolerable?
t
exactly. UI test is our long tail always even with caching
v
the teams hook, is that a chat notification?
t
that is a typical pipeline. crazy to think it runs 8-9 hours of UI tests in there. yeah chat notification with link to play internal sharing and a QR code so you can grab it on a physical device fast
UI test stuff is fully custom. but you can think of it similar to firebase test labs + gradle build cache. so library tests that don't need to run just get a FROM-CACHE app tests pretty much always run
detekt gate does the normal detekt task so we don't waste compute when you didn't even format your code. second detekt job runs with type resolution -> way slower
v
oh, tests get a no op due to the cache?
makes sense, i somehow didnt realize
t
yup. regular task cache behavior if you wire up the apk's as classpath params
downside is the cache artifacts can get huge. I think I limit at like 512MB on the cache now
only stores if all tests passed
v
btw in that gradle invoication you pasted, where are the checks in that? inside bundleProdRelease?
t
not really any checks there other than does it build. if gradle fails, job fails
its all the others running more aggressive asserts
v
right but tests and detekt etc, are those separe GHA jobs?
since in gradle terms build = assemble + check
oh its bundleProdRelease not buildProdRleesse sorry
t
yeah different jobs
then like I said. I try to make 1 job = 1 gradle invocation
v
okay this is new to me, since yea id try to have just a single job so i csn have a single gradlew invoication to be most effective
but between your jobs in a chain, the cache get uploaded and downloaded in each job right?
t
yup exactly
v
thats nice, new things open up
t
summary gets info on that time spend too.
v
since cirrently i just call app:build and let it reuse the compile steps etc
t
I think I need to adjust the wrapper caching. I think its caching multiple versions in there now. probably wasting some time there
v
btw if you had no limitations, would you just use the ci local build cache and not mess with remote?
t
no because the local means a download everything including maybe things we don't need step before the work startns. then zip and upload everything at the end. vs develocity remote cache downloads on demand and amortizes it across the build in parallel too. usually results in the remote cache being faster consistently
but again thats in my world where I have all these tools already spun up in the same subnet so performance is crazy good. for you on gitlab and possibly infra you don't own, the local cache may be the best you can do and still an improvement
v
i see so remote cache has task granularity
local is just a big blob (to fetch)
t
exactly
in my custom cache stuff I even did a ton of weird parallel compress/transfer stuff to squeeze every bit of performance I can from it. (notice total time was 36.1 seconds but adding them up would be longer)
v
about the checks prior to bundleProdRelease do you care about the debug vs release variants there?
t
most pr testing happens on the debug variant build artifact. has all our tooling and mocks environment so its the main thing we look at. releases are just in there so I won't get surprised with release build failures every two weeks when we attempt deploying to prod
forces everyone to think through their full impact on the codebase when they put up a PR
~40 people doing active dev in this project
I tend to be much more fast/loose on tiny side projects that are not shared or customer facing outputs
v
hmm, so you never run 
:app:lintRelease
for example?
t
not explicitly. I think it only runs the vital portion of that as a side effect of other task dependencies
v
yea okay so lets call it 
testReleaseUnitTest
then
t
there is very little variance between our debug/release variants though so the regular lint debug job catches most things
we don't even run the release variant for testing
v
yes thats what I want you opinion on, if there is value in that
okay so release suffixed is just the assemble/bundle task
t
only in tiny situations where you know the release variant needs a special run.
so my test job is this
Copy code
./gradlew generateArchiveScript unitTestSuite --continue -Dscan.tag.PullRequest
and places I know that need to run the release variant I make the unitTestSuite task depend on it
but by default its just getting the jvm modules test task, or an android modules testDebugUnitTest
v
I see what could even be the delta? maybe some resource usage?
stupid question, if I run 
testReleaseUnitTest
it doesnt somehow run the tests on the minimized apk or something? or does it?
t
yeah compute use is the main concern. build variants are work multipliers. so I try to limit it as much as possible to keep compute demand lower
no I don't think that will cause R8 to run since the unit tests run on jvm they don't need dexing. its just going to compile the release src set config
v
okay so its the same + resources/buildconfig class yea
t
but in most cases the release/debug are 100% the same class files output
v
what about ui tests?
t
only debug. getting them to work after R8 mangles things is too much work. we tried it, went poorly. abandoned it
have decent enough success with that plus each team does a manual testing sign off of their own features on each release
v
I see, so anything that runs on the host machine is just a different source set, nothing special about it being release
gotcha
t
yeah I think I only have one place I run tests against the release variant of a class. but even that happens in an emulator since it is testing hardware keystore behavior for encryption
so its technically not even put through R8 in that case
v
same for detekt, same for lint?
let me check what else I have there oh yea 
verifyReleaseAppDatabaseMigration
this is a sqldelight thing
t
detekt we just do 
detekt
and 
detektMain
no variant stuff yet. probably will do that later on once 2.0 stabilizes more. didn't seem to work well last I attempted it
v
mmm variants seem to work fine for me there and im on 2.0
t
lots of yuck in our config. its probably on me why detekt is in rough shape
v
okay so to sum up, even though r8 doesnt apply to jvm tests, they would still get rerun multiple times, i.e. for every variant?
t
here is the lint job invocation
Copy code
./gradlew generateArchiveScript detektMain lintDebug app:lintQaDebug projectHealth --continue -Dscan.tag.PullRequest
projectHealth looks at outputs from DAGP to enforce correct api/impl on most dependencies
okay so to sum up, even though r8 doesnt apply to jvm tests, they would still get rerun multiple times, i.e. for every variant?
yes
v
yea the DAGP is one thing I wanted to add with the perf headroom coming from the cache
I just take objections to some findings 😄
t
yup same. thus my custom task that just asserts on the stuff I do care about. gets us 99% of the way there without drowning in pedantic details
v
it asks me to add these 
runtimeOnly
crap to which I dont subscribe 😄
t
yup same. just help me get api/impl correct so people stop having weird cache issues on recompiles locally
v
my custom task that just asserts on the stuff I do care about.
do you mean to turn off features from it? wasnt aware you could do that
t
there is some config of the plugin to soften it up. then I just parse the output file of what it dumps in the console to look for api/impl swaps that need to happen
v
why do you parse it? dont the devs read the full logs in case of failure?
or does github allow to surface it nicely somehow in the PR ui?
t
so I can reprint just the parts I want them to edit so they only see the important parts vs all the runtimeOnly crap
v
i see
btw about those per feature deploys, so it runs checks, runs assemble, uploads into GP internal testing, sends the teams notification how does the notifying work? I mean its some sort of channel I assume - but isnt there just a lot of spam? do the human testers know which build concerns them?
t
dedicated channel depending on what the build job is message is driven by a power automate flow so I can have it go to a channel, a DM, or a group chat. Usually its just a channel or DM PR channel is very spammy but its best for when devs need to give product people a link. internal app share link for the qa variant also gets put in a PR comment for those that prefer that
release candidate messages also get a scrape of all the jira tickets in a given release relative to the last release to make the release audit teams lives easier
but otherwise functionally the same as the merge/pr channel
merge channel is just tip of default branch so anyone can always grab the latest
product demo's or w/e would pull from that
v
thats líke a scrummasters dream 😄
💯 1
PR channel is very spammy but its best for when devs need to give product people a link.
thats what I wanted to ask, do devs then go into that channel and look for a post with their branch/tag whatever?
or whats the usual human workflow?
t
yeah. a dev digging into it. the message has their github profile picture and username in the body so its fast to scan/search through
all the pr details. link to pr, link to job. link to play. etc etc
v
btw do you use firebase app distribution? or just the GP internal one
t
just gp internal
random history accidents basically makes any gcp/firebase access here a PITA to get, so play internal was just easier when app center shut down
v
yea I wonder how would I implement this there, since there is like a timeline to it, you open the app tester app and theres a list of builds monotonically over time your internal GP is more like a bag of builds
but I like that part here, links suffice & qr code is a nice touch I wonder if I can link to the firebase, I know it generates links and prints them
okay so if youre people want to check the latest main branch build, they go into teams channel to look for it, gotcha
... great stuff btw, shame AI labs will ingest this for profit
😆 1
btw the remote cache granularity thing which I liked, that requires me to setup a server right? and thats free? I remember there being some paid bits with remote cache (?)
oh and one more 😄 about the bundle internal GP builds .. doesnt GP require the app to be registered on google play? or do you not change the application id?
t
if you use develocity its paid
v
whats that exactly? remote cache as a service?
t
its way more than just cache
v
im familiar with build scans, but dont know what else it offers
t
honestly you might be a good candidate for their SaaS platform
them hosting for you is a pretty new feature they offer
if you only want to try remote cache there is OSS options
some you would host. or there is plugins to use redis as a backer, or S3/GCP buckets
v
what else do you use it for?
company where im at is all self hosted bla bla, theyd look poorly at a SaaS what could be run on our infra 😄
t
for me its mainly analytics and build cache. we don't pay for the predictive test selection or test distro since UI tests dominate our long tail
v
test selection? is that a need? (why not run all & rely on up-to-date due to cache)
t
they ship the product to you as a helm chart for hosting. so if you are already in the k8s world for internal hosting it would probably be easy to get things going
why burn cpu on tests that shouldn't be impacted by a given pr change set
v
doesnt gradle do that out of the box?
t
only sort of via cached test tasks
PTS makes it even more incremental
v
yea .. and invalidation causes to run what needs to run unless im missing something
t
say you have 3000 tests, but you only edited one function. why not only run the tests that class could invalidate. ie you only run 20 tests and your job time drops by a ton
very much a your milage may vary situation
v
okay so its for cases where Im not happy with project-level granularity
t
exactly. only works with junit5 and I think a few other config options now
very specific integration
v
but the self hosted way is free, if Im comfortable with setting it up? That task level cache intrigues me
t
no even self hosted you still pay for a license
comes with great support though. you get a direct DM with people on their teams plus a normal zendesk experience for more async type support
for my main team I support it saves us like 3-5 devs worth of time across a pool of 40 people using it. plus it probably lower EC2 spend for GHA runner time by like 50-70% on that project
v
yea I wonder if Im a candidate yet, currently macbook builds the whole android thing in 10min so I guess im not a remote cache candidate howevr I was thinking about turning the backends into a monorepo/gradle build, there it would be handy
t
hard to really say without measuring things
v
so you have your android devs use the remote cache locally as well?
t
yup it tries to. probably helps decently after a pull of main and that first build. then local cache dominates
biggest headache there is corpo zscaler overhead hurting perf
v
but it requires them to be on company network, or vpn? or over internet
t
yeah we are all zscaler 24/7 so effectively always on vpn
my server is in US east so people on that side get slightly better perf
v
yea I hate this company VPN, its sooo slow, I just turn it on when I git push
so I cant picture being on it 24/7 for builds to work/benefit
t
sounds like a valid reason to give your IT team as them failing to provide good enough services
though I am sure they hear it anyway
ours sure do
v
oh boy they do 😄 "it works fine on my end"
😄 1
or better yet "come to office"
t
ironically in office for us would probably actually be better. puts zscaler basically to sleep on the tunneling part since it can see its on a trusted network
probably simplifies the routing into our aws account by a few hops
v
okay but its standard right? or is there a way to have it via internet, or is that a bad idea?
t
I mean having it internet facing would make your life simple. but basically no corpo security team will go for it. thus me saying maybe SaaS is a good option for you. plus makes maintenance not your problem. hell they would probably also do edge node hosting in regions that matter for your team
edge nodes do some sync magic. basically a CDN type thing for their application stack
v
yea im gonna digest all of this a bit more THANK YOU, great info, invaluable
t
no problem. fun to nerd out on this with someone that understands
v
sent you a dm 😄
2 Views
Open in Slack
PreviousNext
Powered by