https://gradle.com/ logo
Join Slack
Powered by
Just to accelerate the resolution of <this issue>,...
# community-support
j
Just to accelerate the resolution of this issue, is it possible to add some directories as configuration cache input but ignore some subdirectories?
The only thing on my mind right now is adding a list of files and then just calling the parent of one of them to get the 
.git
directory... but it would look really awful and mind-blowing to understand why it is doing that without any comment.
v
If those files are considered CC input, you most probably read them at configuration time which automatically make them CC inputs. Depending on use-case you might be able to use a 
ValueSource
instead to read the files and then only the output of the 
ValueSource
becomes CC input, but then this action is evaluated on each build run even if CC is reused (to determine whether the CC entry can be reused) and if the CC entry cannot be reused it even is executed a second time in that build, so that action should optimally run quite fast.
j
Indeed I am passing the git dir as parameter of 
ValueSource
And in a build service too
v
I think parameter to value source should not be a problem, that's the point of a value source. If you use it in a build service and that build service accesses it at configuration time, that is probably the culprit.
j
The build service is used only on tasks
v
Maybe try to comment out its usages and then run with CC and have a look at the CC report whether the git directory is still listed as CC input?
j
Is it possible to track what property is causing the problem?
The report is generated if the config cache does not fail?
v
Or does the CC report contain information where the inputs are used in form of stacktraces? I don't remember, it is a while since I last looked at a CC report.
I think the CC report is always generated.
It is just not linked in the output when CC was successful
j
Locally, I have added a new file to the 
.git
directory and the config cache is being reused, so the problem is on another place
v
Maybe try to publish the CC report from GHA as artifact so you can look at the report the GHA run produced?
👀 1
👍 1
j
Copy code
- `.git`
        - plugin class 'com.javiersc.semver.project.gradle.plugin.SemverProjectPlugin'
    - `.git/.probe-b36c736a-3835-4d22-966c-c5e63f3259ef`
        - class `org.eclipse.jgit.util.FileUtils`
    - `.git/.probe-c1e96c28-ea16-4382-ab17-24e287b661a5`
        - class `org.eclipse.jgit.util.FileUtils`
v
Sad, I thought it shows a full stacktrace. Maybe that was only for CC problems. But as the two 
.git/.probe...
files are accessed by JGit classes and probably created by https://github.com/eclipse-jgit/jgit/blob/cb0935ab6dbcb2c54f0ea67585e608987c87284e/org.eclipse.jgit/src/org/eclipse/jgit/util/FS.java#L508 or https://github.com/eclipse-jgit/jgit/blob/cb0935ab6dbcb2c54f0ea67585e608987c87284e/org.eclipse.jgit/src/org/eclipse/jgit/util/FS.java#L575 I guess you still do use JGit somewhere at configuration time. That you locally do not find those probe files is also clear as JGit deletes those probe files after the probing.
j
But why it is not deleting them on CI?
v
Who says they are not deleted?
j
Anyway I will look where I am using jgit in configuration time
v
Doesn't mean they are not CC inputs
j
I mean, at a different time that in CI
v
If you create the file, read it, then delete it, it is still a CC input technically.
Check your local CC report, I'd guess you have these entries there too
If not, then maybe it depends on the OS whether this is done? Didn't analyze the JGit code further.
j
I am on windows, I will check the windows report later as I test everything on the three OS
Same problem on Windows on CI
I think this is the only reference that is calling git too early. There is no way to track what is the exact problem?
v
Probably if you can reproduce and use a debugger. 🤷‍♂️ Actually, you can maybe use 
org.gradle.configuration-cache.inputs.unsafe.ignore.file-system-check
to exclude that file, but as the name suggests, it is unsafe and questionable. :-)
j
I can't reproduce it locally, I don't even know what are those files tbh.
v
I don't know from the top of my head what it does, but maybe 
org.gradle.configuration-cache.internal.debug
can also shed some light?
I can't reproduce it locally, I don't even know what are those files tbh.
Why not? I told you
j
Why not? I told you
I am doing exactly the same as I do on CI and the issue is not happening
Check your local CC report, I'd guess you have these entries there too
I haven't those entries, the associated files do not exist on my machine.
I have even done a Google search about those files and I find nothing related.
v
Sorry, bad quote, the "why not" was just referring to you not knowing what those files are.
I told you what those files are
I even linked you to the code that generates them
And I also explained why you don't find those files locally
Did you read my answers or just skipped them? :-/
j
I read them but I do not find a way to avoid JGit to generate them.
v
Well, maybe there is none 🤷‍♂️
Actually, as I said, I linked you to the code that is generating those files. If you would look at it, you could see that the code is never reached if 
readFromConfig
returns a non-empty value. So I'd say you can very well prevent those files to be generated by doing some config.
Given the method names are not misleading
j
Yep, I am trying to track where I am calling git in the config phase. Calling it inside a 
provider
wouldn't be a problem, no?
v
Depends on when the provider is resolved. If it for example is configured as value for a task property, there you have it. Provider values are calculated at CC store time and persisted into the CC.
j
Technically it is on an extension but I am not calling it
v
Technically is pointless. If it ends up as value for a task property, it will be persisted to CC and thus evaluated at end of configuration phase.
j
It is not added to a task, it is unused
v
Then it should not cause it of course, no
j
I think I am only using it as input for build services and value sources parameters. If the value source output is assigned to a version and this version is added as input of a task, can it be the problem?
v
I don't think so. The sense of a value source is, that you can do anything inside and out does not become a CC input, only the result is a CC input and that is calculated on each run
j
And the build service is used only on the task runs. I will try to create a minimal repro tomorrow.
👌 1
Minimal repro here I will try to debug/comment/uncomment the semver code tomorrow. Do you know if there is a feature request about getting the trace of this kind of issues in the config cache report?
I have reduced the code to
Copy code
public class SemverProjectPlugin : Plugin<Project> {

    override fun apply(target: Project) {
        target.tasks.register("printSemver") {
            it.doNotTrackState("foo")
        }
        //target.config()
    }
}
and I am not even adding JGit as dependency... the issue is still there
v
Do you know if there is a feature request about getting the trace of this kind of issues in the config cache report?
None I'm aware of.
and I am not even adding JGit as dependency... the issue is still there
Maybe set a breakpoint at the lines I linked you to and debug the build to see where it is executed?
j
Looks like it was caused by using the build cache, after disabling it, I can get correct reports every I upgrade the semver plugin.
I will post the real issue when I find it, a lot of comment/uncomment... 😢
👌 1
I am not even able to reproduce it without build cache
After enabling build cache again, it is not happening. I am deleting all caches, if that reproduces again the issue, it will be a huge pain to find the issue as I would have to delete all GitHub cache in every try... so I would go for 
org.gradle.configuration-cache.inputs.unsafe.ignore.file-system-check
I can confirm that last statement. It is pretty hard to test it
Copy code
val gradleVersionProvider: Provider<String> = VersionValueSource.register(this)
        version = VersionProperty(gradleVersionProvider)
Those lanes are provoking the issue. The registration of the value source is
Copy code
fun register(project: Project): Provider<String> =
            project.providers.of(VersionValueSource::class) { valueSourceSpec ->
                val parameters: Params = valueSourceSpec.parameters
                val semverExtension: SemverExtension = project.semverExtension

                val gitDir: Provider<File> =
                    project.provider { semverExtension.gitDir.get().asFile }

                parameters.versionMapper.set(semverExtension.versionMapper)
                parameters.gitDir.set(gitDir)
                val commitsMaxCount: Int =
                    project.commitsMaxCount.orNull ?: semverExtension.commitsMaxCount.get()
                parameters.commitsMaxCount.set(commitsMaxCount)
                parameters.projectTagPrefix.set(project.projectTagPrefix.get())
                parameters.tagPrefixProperty.set(project.tagPrefixProperty.get())
                parameters.stageProperty.set(project.stageProperty.orNull)
                parameters.scopeProperty.set(project.scopeProperty.orNull)
                parameters.creatingSemverTag.set(project.isCreatingSemverTag)
                parameters.checkClean.set(project.checkCleanProperty.get())
            }
I am not using JGit in any parameter, so I have no clue what it can be happening
More concretely, assigning the result to the version
Copy code
version = VersionProperty(gradleVersionProvider)
If I comment that line it works
Copy code
val gradleVersionProvider: Provider<String> = VersionValueSource.register(this)
        version = "1.0.0"
        //version = VersionProperty(gradleVersionProvider)
VersionProperty
is just
Copy code
public class VersionProperty(
    private val version: Provider<String>,
) : Provider<String> by version {

    override fun toString(): String = "${version.orNull}"
}
But as that 
version: Provider<String>
comes from a value source, it is possible to get it on configuration time without affecting the configuration cache, right?
Or calling 
version.toString()
in configuration phase would break it?
This can be related to https://github.com/gradle/gradle/issues/28591, can it be, @Adam?
a
ummm yes, I think so, but I think because you use a ValueSource it should avoid the issue
is it possible to add some directories as configuration cache input but ignore some subdirectories
If you have a ValueSource, I don't think adding some directories as a CC cache input would give any benefit. ValueSources are always recomputed, even if re-using CC.
My guess would be that this code is triggering the automatic CC input detection:
Copy code
val gitDir: Provider<File> =
                    project.provider { semverExtension.gitDir.get().asFile }
I think about it this way. Basically Gradle intercepts all 
<http://java.io|java.io>.*
calls that access the file system. The exception being if the call is made inside of a ValueSource. And the way Gradle detects that is basically by looking at the stacktrace. But because this call is not technically made inside of a ValueSource, but the configuration of parameters for the ValueSource, it will be treated as a CC input. But I don't know if that's right or not, the behaviour is undocumented.
j
But as I need to provide it, the only solution would be ignoring the 
.git
folder with 
org.gradle.configuration-cache.inputs.unsafe.ignore.file-system-checks
?
a
you need to provide it, but you probably don't need to do 
.get().asFile
what about just passing it in like 
parameters.gitDir.set(semverExtension.gitDir)
?
I suggest create a custom 
@RequiresOptIn
flag https://kotlinlang.org/docs/opt-in-requirements.html and then using it to mark all code that touches JGit. That should help track it, to make sure that JGit is only called from within a ValueSource's obtain(), or a task action.
v
You should have just debugged the code places I gave you. Locally you do not get the 
.probe...
files because JGit persists the probe results. It remembers them in memory and also on disk. So if you kill the daemon (or use 
--no-daemon
) and make sure that neither 
git config -l | grep filesystem
nor 
git config --file ~/.config/git/config -l | grep filesystem
nor 
git config --file ~/.config/jgit/config -l | grep filesystem
returns any value, you will get the probe values in the CC report. You can for example use 
git config --file ~/.config/jgit/config -l | grep filesystem | cut -d = -f 1 | xargs -i git config --file ~/.config/jgit/config --unset "{}"
to kill all entries from that file. The problem is, that JGit does this probing in a different thread by using 
CompletableFuture.supplyAsync
. By this is "circumvents" the 
ValueSource
encapsulation, as only those access are not end up as CC inputs that are done on that same thread that does the configuration. If something is done on a different thread, Gradle cannot see whether this is done in the scope of a 
ValueSource#obtain
or not and thus the file ends up as CC input.
j
what about just passing it in like parameters.gitDir.set(semverExtension.gitDir) ?
That should be possible, but the 
get().asFile
is inside of a 
provider {}
, so it shouldn't affect, no?
You should have just debugged the code places I gave you.
I tried, but the sources were broken on the IDE, so I could not download/attach them. And the sources I was the owner (the plugin itself), didn't help me.
The problem is, that JGit does this probing in a different thread by using 
CompletableFuture.supplyAsync
. By this is "circumvents" the 
ValueSource
encapsulation, as only those access are not end up as CC inputs that are done on that same thread that does the configuration. If something is done on a different thread, Gradle cannot see whether this is done in the scope of a 
ValueSource#obtain
or not and thus the file ends up as CC input.
Is there any way to force using the same thread the 
ValueSource
uses?
v
No
Not without modifying a 
private static final
field in JGit
😕 1
To care about your GHA runs specifically, you can as work-around configure the file where JGit persists the probe result to be also cached and restored.
So add 
~/.config/jgit/config
to the cache config and you will probably start to see CC reuses
j
Couldn't that increase the cache indefinitely due the probe files?
v
No
j
I am going to try that, thank you!
v
There will be no probe files after the first run
If in neither of the files I wrote above the persisted values are found, the probe files are created, checked, and deleted, and as that is done on a different thread and with different unique files each time, you always have a CC miss as inputs changed.
If you add the path where JGit persists the result to the cache, then this file will be CC input as it is still read on that different thread, but it will always be the same file and always have the same two lines content, so it is still CC input but will not change and thus not invalidate the CC entry.
As the values are found in that file, no probe files will be created.
j
I will need to investigate about how to do this on GitHub, using the cache with that path is not working due
Copy code
Calculating task graph as configuration cache cannot be reused because the file system entry '../../../.config/jgit/config' has been created.
v
On first run, sure, run again?
j
No, no, every run
v
Not sure what that is now, but at least so far JGit is happy and does not create the probe files
👍 1
cannot not be reused because the file system entry '...' has been created.
means that in the run when the CC entry was stored the file was checked for existence but did not exist and now the file does exist so the entry is discarded. The same happens when you locally run without the file, then again with the file being created by the previous run, but the third call reuses the CC entry. Your GHA summaries also show that on the 2. and 3. run no cache entries were saved but only entries restored. This is due to the caculated cache key not being different and so on 2. and 3. run the same cache entry from the 1. run being restored as cache entries are immutable on GHA. If you amend the commit without actual changes and push again, you should get the same message but a new cache stored. If you then rerun, the entry should be reused. If you need the rerun to work, - looking at the setup-gradle action source - you can probably use the 
GRADLE_BUILD_ACTION_CACHE_KEY_JOB_EXECUTION
environment variable to cleverly influence this. github.com/gradle/actions/blob/main/docs/setup-gradle.md#cache-keys describes the cache key construction. That environment variable if set is used, otherwise the git-sha as last part of that constructed key. So if you set this environment variable to 
${{ github.sha }}-${{ hashFiles('~/.config/jgit/config') }}
I guess it should work immediately on the first rerun even on the same commit
Actually, just ignore that
Just make a new commit and rerun on that.
New commit means new cache key, but the cache entry from the previous run will be restored
There the JGit file will be available in the restored cache and so the cache entry stored in the end will be reusable on rerun
No need to mess with the cache-key
j
Just make a new commit and rerun on that.
That works 🙂
👌 1
Thank you for clarifying the problem in the GitHub issue.
👌 1
16 Views
Open in Slack
PreviousNext
Powered by