Assuming multiple calls to Gradle, it should be able to use the files from the buildDirs without needing the local build cache (i.e. everything is 'UP TO DATE', not 'FROM CACHE').
If you only have a single Gradle invocation, then local cache is definitely not needed with ephemeral build agents, as each task will be executed only once anyway.