Hello community! I try to migrate from `exec {}` t...
# community-support
k
Hello community! I try to migrate from
exec {}
to
providers.exec {}
to enable configuration cache and get rid of its warnings. How can I get stdin and stdout both merged in one stream from
providers.exec
? an attempt to call
setXxxOutput()
which I used previously throws "Standard streams cannot be configured for exec output provider" Posted this question with code samples on SO
a
hey šŸ‘‹
providers.exec {}
returns a result, so you can get the text (or bytes, using
asBytes()
)
Copy code
val execResult = providers.exec { commandLine("echo", "hello") }
println("stdout " + execResult.standardOutput.asText.orNull)
println("stderr " + execResult.standardError.asText.orNull)
does that work for your case?
It it helps, you can also
@Inject
a
ExecOperations
service, and then you can just call
execOperations.exec {}
and it'll work the same as
Project.exec {}
ā˜ļø 1
for example
šŸ‘€ 1
āœ… 1
k
Hey @Adam Thanks for fast reply. I guess the 1st solution won't work for me, because I need streams to be merged together to get output lines in the right order. Will try the 2nd one
šŸ‘‹ 1
@Adam the one with
ExecOperations
service works for aggregating stdin and stdout into a single stream and its
.exec
configuration-cache-friendly. Thank you! Now I'm struggling with converting
JavaExec
task the same way to
ExecOperations.javaExec()
. I'm getting:
Copy code
- Task `:ktlint` of type `org.gradle.api.DefaultTask`: cannot serialize Gradle script object references as these are not supported with the configuration cache.
  See <https://docs.gradle.org/8.6/userguide/configuration_cache.html#config_cache:requirements:disallowed_types>
While I can't find any more details, I tend to think that the root cause of the problem here is that I reference one of my "configurans" as a
classpath
for
ExecOperations.javaExec()
. Do you have any suggestion in this situation? My code is as simple as (yeah, I'm aware of a Gradle plugin for Ktlint, but had a preference to build my own tasks which I want to make configuration-cache friendly now):
Copy code
register("ktlint") {
    val exec = serviceOf<ExecOperations>()
    doLast {
        val myExec = exec.javaexec {
            classpath = ktlint
            mainClass = "com.pinterest.ktlint.Main"
            args(*ktlintTargets)
            standardOutput = System.out
            errorOutput = System.err
        }
        myExec.assertNormalExitValue()
    }
}
a
no problem!
I see two things to improve in your example: 1: "cannot serialize script references" is a confusing one. It basically means you can't use script-level variables inside of a task-action. It's basically because the script-level variables are intrinsically linked to a Project instance. So you need to redefine them, inside the task configuration block.
Copy code
// script level val
val ktlint by configurations.creating { ... }

tasks.register("ktlint") {
    val ktlintClasspath = ktlint.incoming.files // redefine the val for CC, and use a CC compatible type (FileCollection)
    // ...
}
šŸ¤” 1
2: ktlin isn't set as a task input. Registering task inputs is important, so Gradle can do clever things like figuring out task dependencies, or skipping running tasks if nothing has changed. It's easy to do with the dynamic-task API, although it's a little bit verbose. And we can use classpath normalization, so when Gradle does up-to-date checks it can do clever things like treat identical JARs the same, even if the name is different. There are other normalization strategies for regular files.
Copy code
val ktlint by configurations.creating { ... }

tasks.register("ktlint") {
    val ktlintClasspath = ktlint.incoming.files
    inputs.files(ktlintClasspath).withPropertyName("ktlintClasspath")
        .withNormalizer(ClasspathNormalizer::class)
    
    doLast {
        exec.javaexec {
            classpath = ktlintClasspath
            // ...
        }
    }
}
🤯 1
k
@Adam trying to follow your advice and get a dedicated variable for ktlint configuraiton. And it probably solves the initial issue, but here is another one 😮 : > - Task
:ktlint
of type `org.gradle.api.DefaultTask`: cannot serialize object of type 'org.gradle.api.internal.artifacts.configurations.*DefaultUnlockedConfiguration*', a subtype of 'org.gradle.api.artifacts.Configuration', as these are not supported with the configuration cache. My configuration is created as simply as:
Copy code
val ktlint: Configuration by configurations.creating
a
in the task do you have
val ktlintClasspath = ktlint
? If so, try
val ktlintClasspath = ktlin.incoming.files
.
āœ… 1
I think Configuration isn't cacheable, so it needs to be converted, and
.incoming.files
converts it to a FileCollection
k
@Adam the
val ktlintClasspath = ktlin.incoming.files
worked out! Thank you!
a
great! My pleasure
k
@Adam do you want to publish StackOverflow answer or I will do it?
a
please, go ahead :)
I've earned enough SO points to be able to edit posts, and that's all I need (I like adding code colour highlighting to code blocks)
k
Oh, it turned out my original SO question is still valid. While the problem is solved for tasks like ktlint's
JavaExec
, it's not solved if I just want to execute arbitrary process during build configuration (without any tasks involved at all). The CC does not allow me to use ExecOperations service outside of the task, so I have to rely on recommended `providers.exec`(with which I have troubles with at merging 2 stream into one, it's simply not supported). My use case for executing arbitrary process on Gradle's configuration-phase is pretty simple btw. I have a dumb Gradle project in monorepo's root, so whenever new people open it with Intellij for the 1st time to contribute, Intellij will trigger Gradle, which in turn `exec`s some plain git commands to initialize mandatory git hooks. something like this:
Copy code
fun tryToInitGitHooks() {
    val isGitRepoCheck = executeCommand("git", "rev-parse", "--git-dir")

    if (isGitRepoCheck.exitCode != 0) {
        logger.info("Seems like not a git repo, skipping git hooks initialization")
        logger.info("Here is the result of git repo check. Exit code: {}. stdout:", isGitRepoCheck.exitCode)
        logger.info(isGitRepoCheck.consoleOutput) // <---- Here is where I need stdin and stderr lines merged together in the right order
        return
    }
    // ... success, execute more commands to init git hooks...
}

data class CommandExecutionResult(val exitCode: Int, val consoleOutput: String)

fun executeCommand(vararg args: String): CommandExecutionResult {
    ByteArrayOutputStream().use {
        val execResult = exec {
            commandLine(*args)
            isIgnoreExitValue = true
            standardOutput = it
            errorOutput = it
        }
        return CommandExecutionResult(exitCode = execResult.exitValue, consoleOutput = it.toString())
    }
}
@Adam any ideas how can I solve this supporting CC? Maybe I can make Gradle to execute one mandatory task if no tasks was requested explicitly? In this case I would just wrap those plain git commands as a task and Intellij would execute that on project import... I'm just not aware if that's possible at all. Or maybe Intellij does execute some tasks on project import? In this case I could add
finalizedBy
dependency to Intellij's import task šŸ¤”
a
Ah I see... Does
Project.exec {}
work during configuration?
I'm not sure you need the "merging 2 stream into one" requirement. What's wrong with logging the std and err output seperately?
k
@Adam it does work and it's my current implementation, but as soon as I enable CC I get the following complain on `Project.exec`: > - Plugin 'git-hooks-initializer': external process started 'git config core.hooksPath .githooks' > See https://docs.gradle.org/8.7/userguide/configuration_cache.html#config_cache:requirements:external_processes > What's wrong with logging the std and err output seperately? I can easily imagine git executable (or some other executable) to write lines switching back and forth between stdin and stderr. I just want to see console text in the right order as if I executed it myself from the terminal. UPD: Ok, I've just checked - in my current scenarios with git it's enough to only get stderr on non-zero exit code from it
v
Some notes: •
serviceOf
is internal API, you should not use it •
ktlin.incoming.files
is not necessary, a
Configuration
is-a
FileCollection
already. You just need to specify the type of the variable explicitly to be
FileCollection
and it already is CC-safe. If you do not specify a type, it gets the
Configuration
type implicitly and that is not CC-safe. • If you need to run external process at configuration time, you need to use a
ValueSource
. This is then also always executed, whether CC entry is reused or not and even twice if CC entry is not reused.