https://gradle.com/ logo
Join Slack
Powered by
Context: I have a task on each subproject, that ta...
# plugin-development
c
Context: I have a task on each subproject, that task is slow but relies on an external server. I can launch the task with a "async" flank so it doesn't block, and reconnect later. Issue: When I have many subprojects, all my threads are just waiting for the server to reply, and I can't even start all the requests until some threads are freed. Question: Can I do something to not use a thread all the time while I wait for the server to reply? Maybe using workers? The thing is that I want each task to be cacheable, so I can't have a single task with all the workers. For example I would like to: on each subproject run the task, only the first part (async request), and after some time reconnect.
That's something years ago I did using libevent to have a single threaded server, that most of the time was just waiting for the database to reply
c
possible to use a non-blocking communication layer, such that it isn’t a thread-per-request?
for example, if the calls are HTTPS, using an async HTTPS library would provide for multiple outstanding requests with a fixed set of threads to process responses as they come in, landing in your callbacks.
it isn’t clear, though, what the blocker is in current implementation - is it exhausting a thread pool, and waiting until outstanding requests finish? Or something else that is causing the blocking?
c
The issue is that for example CI has just 4 CPUs, so my tasks for the first modules will make everything else wait
c
number of CPUs shouldn’t affect the threads - you can have hundreds of threads on any number of CPUs (and may want to do so where the threads are waiting on external resources). Is the task making external HTTP/API calls?
by default, the number of parallel Gradle tasks will match the number of CPUs. That doesn’t work for tasks that are light on CPU, waiting on external resources. This is configurable via 
--max-workers=n
or setting 
org.gradle.workers.max=n
in a gradle.properties
c
As plugin author max-workers is not something I can change, I can suggest users, but they have other tasks that may be CPU intensive
I was looking if Gradle had some way of handling this usecase
c
to my knowledge, Gradle could help via the Workers API or a shared build service that could manage the concurrency.
the shared build service does support a concurrency limit on a per-service basis.
🤔 1
a brief example of a shared build service. Note that you can do your own caching within it, for repeated calls with the same inputs (if that’s applicable in your case).
Copy code
abstract class CdkToolchainBuildService : BuildService<CdkToolchainBuildService.CdkToolchainParams> {

    interface CdkToolchainParams : BuildServiceParameters {
        val cacheService: Property<CacheBuildService>
    }

    private val versionCache = ConcurrentHashMap<String, CdkToolchain>()

    @get:Inject
    abstract val execOperations: ExecOperations

    companion object {
        internal fun registerIfAbsent(
            gradle: Gradle,
            configureAction: Action<BuildServiceSpec<CdkToolchainParams>>
        ): Provider<CdkToolchainBuildService> {
            return gradle.sharedServices.registerIfAbsent(
                "cloudshift-cdk-toolchain",
                CdkToolchainBuildService::class,
                configureAction
            )
        }
    }

    fun resolve(versionProvider: Provider<String>): CdkToolchain {
        require(versionProvider.isPresent) { "Missing version" }
        val version = versionProvider.get()
        require(version.isNotBlank()) { "Empty version"}

        return versionCache.computeIfAbsent(version) {
            // omitted for brevity
        }
    }
}
by default, there are no concurrency limits but you can specify a level of parallelism if needed (# of tasks concurrently accessing the service)
c
Thanks for the example, I'll take a look tomorrow and see if I can use it. So far it's something I didn't know (caching with services), so, thank you very much.
👍 2
I did read about it but, if I understood correctly, in my case it doesn't help. At the end there is a task that will be using a gradle thread to wait for the service to finish. The service can have unlimited threads, but tasks are going to be queued until some of them finish and free a Gradle thread/worker
c
yea. that’s accurate (and unfortunate). Needs some thinking on how tasks can be made async.
c
I could split each task in the starter (throw and forget) and the watcher (reconnect and wait for completion) but then I would rely on luck, if Gradlew decides to start multiple "starter" jobs (from different projects) before starting the "watchers", everything is fine. If not I'm in the same situation as before.
It happens the same with workers API, because as I'm playing with multiple projects I can't have a single task orchestrating the jobs
So probably there is no solution for my concern
Actually...I wonder how Gradle does remote test execution, it's not that different but I guess that's a private API
c
only thing that comes to mind is, as you suggested, splitting the task in two - the initiator portion asks the service to start a background request (with name ‘project:task’), and a completion task (that would somehow have to run much later, hmmm…) to ask the service for ‘project:task’ result (via a future, which will block if not available).
c
The issue is that I can't tell gradle to wait in between those 2 tasks... mustRunAfter(tasks.withType(starter)) won't work between projects
(if that was the option I would need to figure out how to declare my inputs&outputs so the tasks are actually cacheable, but anyway it would be worth)
Open in Slack
PreviousNext
Powered by