Hey. I'm a noob at writing custom pre-compiled scripts/convention plugins and need some help. I'm trying to write a pre-compiled script or a plugin in

buildSrc

that would read a custom

.properties

file and let multiple subprojects access the values. I want to do it in a cache-friendly manner so that the file doesn't get re-read unnecessarily. So, for example, I tried something like

// buildSrc/src/main/kotlin/my-variables.gradle.kts

open class MyVariables(project: Project) {

    val customPropertiesFile = providers.fileContents {
        project.rootProject.file("custom.properties")
    }

    val customProperties = customPropertiesFile.asBytes.map { ByteArrayInputStream(it) }.map { Properties().apply { load(it) } }

    // this is the property I would like to make available in subprojects
    val firebaseAppId = provider { customProperties.get().getProperty("FIREBASE_APP_ID") }

}

val extension = project.extensions.create<MyVariables>("myVariables", project)

However, when I apply the plugin, I get this error:

Class My_variables_gradle.MyVariables is a non-static inner class

What's the recommended way to do something like this? 🤔

it's a confusing error message, but the problem it's indicating should become clear if you move your

MyVariables

class into regular

.kt

file.

That's one of the problems. 🙂 You for example also access

project

, same problem. Besides that, I think you would read the properties once per project where you apply that plugin. Maybe you should instead consider using a shared build service. I'd say that would be the right abstraction here. You would read the properties in its constructor and every consumer will get the same service instance.

Surely, a shared build service is an overkill to read a properties file, isn't it?

... no

Then I wonder how come it is not a widely adopted solution and a popular recommendation.

Ask the people conciously not using it, not me 🙂 Probably because they simply are not aware. Many also do not use Kotlin DSL though it has extensive pros. Many do not use version catalogs. Many do not use artifact transforms where they would be appropriate. Many do not use

Property

where it would be better to, ...

> Ask the people conciously not using it I would argue about the "conciously" part...

But for the use-case you described it is exactly the tool of choice. It is one-time initialized during a build, is build scoped, and everyone accessing it throughout one build run will get the same instance.

yeah, exactly for this reason:

Probably because they simply are not aware.

And that's why the "conciously". "I didn't know about" is not a good reason. If people would be aware there is a more appropriate tool but conciously decide against using it, that would be the "interesting" cases. 🙂

@Adam @Vampire is

Project

injectable or something? Somehow, when I changed the script above to this 👇 it no longer fails with the

non-static inner class

error

open class MyVariables(project: Project) {

    val rootProjectDir = project.rootProject.projectDir.absolutePath

    val customPropertiesFile = project.rootProject.file("local.properties")

    val customProperties = project
        .providers
        .fileContents { project.rootProject.file("custom.properties") }
        .asBytes
        .map { ByteArrayInputStream(it) }
        .map { Properties().apply { load(it) } }

    val firebaseAppId = customProperties.get().getProperty("FIREBASE_APP_ID")

}

val extension = project.extensions.create<MyVariables>("myVariables")

but instead I get

> Failed to apply plugin 'my-variables'.
   > Could not create an instance of type My_variables_gradle$MyVariables.
      > Could not isolate value extension 'myVariables' of type FileContentValueSource.Parameters
         > Could not serialize value of type My_variables_gradle.MyVariables.

...

Caused by: java.io.NotSerializableException: My_variables_gradle$MyVariables$customProperties$1
	at org.gradle.internal.snapshot.impl.AbstractValueProcessor.javaSerialization(AbstractValueProcessor.java:175)

Any advice (except for a shared build service)?

I don't understand why you so hesitant against a build service, it is not really much more effort and unlike your solution even if you fix the error will work as intended.

I'm hesitant for the reason I mentioned above ☝️

That you think it is overkill, but it is not

And also I'm afraid it's gonna increase the maintenance burden for everyone in my team

Why? It is not really more maintenance burden than what you showed up there.

I'm not sure why you use

fileContents

anyway here.

Is there an alternative to it that is cache-friendly?

configuration cache you mean? Just read the file with normal means, Gradle should recognize it as CC input automatically.

In the latest version I wasn't `get()`ting it anymore:

open class MyVariables @Inject constructor(buildLayout: BuildLayout, providerFactory: ProviderFactory) {

    val customPropertiesFile = buildLayout.rootDirectory.resolve("local.properties")

    val customPropertiesContent = providerFactory
        .fileContents { customPropertiesFile }

    val bis = customPropertiesContent.asText.map { it.byteInputStream() }

    val properties = bis.map {
        it.use {
            Properties().apply {
                load(it)
            }
        }
    }
    // val firebaseAppId = properties.map { it.getProperty("FIREBASE_APP_ID") } // fails if uncommented
}

I did not in any way imply that the

get()

is related to the isolation problem

I understand. Just wanted to say I was trying to make a proper use of

Provider

instead of

get()

But any

get()

you do at configuration time usually is a bad idea.

Just wanted to say I was trying to make a proper use of

Provider

instead of

get()

Well, so

properties

is a

Provider

and then you do what with that? At another place using

project.myVariables.properties.get()

at configuration time? Then you did not win too much. 😄 Also each call to

get()

would parse the properties file again and again and again, as

map

execution is not something that is cached by default ...

At another place using

project.myVariables.properties.get()

at configuration time?

Not sure if doing so in the project where the plugin is applied is considered a "configuration time" (again, I'm noob). If yes, then when and where can I finally

get()

the value from the

Provider

?

Not sure if doing so in the project where the plugin is applied is considered a "configuration time" (again, I'm noob).

Yes, it is. Configuration time, is during configuration. Execution time is when tasks actually are executed.

If yes, then when and where can I finally

get()

the value from the

Provider

?

Always as late as possible. That means, optimally only at execution phase, so in the implementation of a task's work. Because then configuration is finished and the value will not change anymore. Whenever you call

get()

earlier you introduce ordering problems / timing problems and race conditions. The sense of

Property

/

Provider

is, that you wire things together and only at execution time where really needed get the value, so that you get the finally configured value.

@Vampire do you think a

ValueSource

like in this example would work fine as well (for the purpose of reading a properties file)?

"work" maybe. "work as you intend it to", most probably not. A value source will always be evaluated, even if configuration cache entry is reused as the result of the value source controls whether the configuration cache entry can be reused or not. If the CC entry cannot be reused, it even is evaluated twice in the same run. And you really do not win anything over doing it properly using a build service.

@Vampire Thanks for all the above. Regarding this:

That means, optimally only at execution phase, so in the implementation of a task's work.

what if I don't own those tasks? I want to use the loaded properties in either one or multiple subprojects where

com.android.application

/

com.android.library

plugin is applied and I configure its

android { ... }

extension. E.g.

plugins {
  id("com.android.application")
  id("my-variables") // the plugin
}

android {
  defaultConfig {
    buildConfigField("string", "FIREBASE_APP_ID", myVariables.firebaseAppId) // here
  }
}

is this "late enough"?

I have no idea about Android and Android is always a bit special. Actually, in your case it does not make much difference when you

get()

the value, as it is not a configurable value. But also as it is not a configurable value, it also does not really make much sense to use

Property

/

Provider

at all.

FYI in case anyone is interested, this worked quite well although I wouldn't be surprised if @Vampire said it doesn't work the way I think

// buildSrc/src/main/kotlin/my-variables.gradle.kts

abstract class PropertiesValueSource : ValueSource<Properties, PropertiesValueSource.Params> {

    interface Params : ValueSourceParameters {
        val configFile: RegularFileProperty
    }

    override fun obtain(): Properties {
        val configFile = parameters.configFile.asFile.get()
        val result = Properties()
        if (!configFile.exists()) {
            return result
        }
        configFile.bufferedReader().use { result.load(it) }
        return result
    }
}

val propsProvider = providers.of(PropertiesValueSource::class) {
    parameters.configFile = rootProject.file("custom.properties")
}

open class MyVariables(propertiesProvider: Provider<Properties>) {
    val firebaseAppId: Provider<String> = propertiesProvider.map { it.getProperty("FIREBASE_APP_ID") }
    val nr: Provider<Int> = propertiesProvider.map { it.getProperty("NR").toInt() }
}

val extension = project.extensions.create<MyVariables>("myVariables", propsProvider)

// buildSrc/src/main/kotlin/buildConfigFields.kt

data class BuildConfigField(val name: String, val type: String, val value: String)

fun buildConfigFields(vararg providers: KProperty0<Provider<*>>) =
    providers
        .map {
            val value = it.get().get()
            val (type, literal) = when (value) {
                is String -> "String" to "\"$value\""
                else -> value.javaClass.name to "$value"
            }
            BuildConfigField(it.name, type, literal)
        }

// lib/build.gradle.kts

plugins {
    id("com.android.library")
    id("my-variables")
}

fun getBuildConfigFields() =
    buildConfigFields(
        myVariables::firebaseAppId,
        myVariables::nr,
    )

android {
    defaultConfig {
        getBuildConfigFields().forEach {
            buildConfigField(it.type, it.name, it.value)
        }
    }
}

When I run

./gradlew :lib:assembleDebug

multiple times and observe the task statuses I see that:

> Task :onboarding:generateDebugBuildConfig UP-TO-DATE

and yet if I go to

custom.properties

and change

FIREBASE_APP_ID

inside, then run Gradle again, I see that

generateDebugBuildConfig

task in not up-to-date anymore and that the

BuildConfig.java

file gets generated in

lib/build/generated/source/buildConfig/

again with the updated content

One of your stated requirements was, that you want to read the properties file only once. And that is not the case in your solution, it will read it once per project. And

Provider

just does not really make much sense in your case. I'd still say that a shared build service would be the better way to achieve what you said and using a

ValueSource

here is just an abuse of it as it is totally not what a value source is for. And it even hurts, as then with CC the value source is always evaluated and thus the file read, even if the CC entry can be reused. And if the CC entry cannot be reused it is even done twice. And that of course per project you apply that plugin to. Again, a shared build service is, what you should use and it is not more complex than what you have now, but works better and more like intended and with less performance penalty. I still don't know why you insist on not using the proper way and why you even ask experts if you then ignore and do the opposite of what they tell you. 🤷‍♂️ Well, your build, you have to live with that solution. 🤷‍♂️

which didn't even work for me because it was looking for

custom.properties

in the daemon's home directory rather than in the project's directory 🤦

It works sometimes(TM), as often the working directory of the daemon is the root project directory, but it is not necessarily. As you have seen, sometimes it is the daemon log directory, sometimes it is the IDE installation directory, sometimes it is some other directory, there is no guarantee what it is. So yes, anything that depends on current working directory is inherently broken and at least flaky. Actually using

File(...)

wilh a relative path or any other method that relies on current working directory is almost always the wrong thing to do in any JVM code, not only in Gralde build logic. The only case I'm aware of where it is appropriate is, if you develop a command-line utility where the user specifies some path as argument, because in this case you can safely assume that a relative path should be resolved relative to the current working directory.

> One of your stated requirements was, that you want to read the properties file only once. And that is not the case in your solution, it will read it once per project. Yes, I'm aware. > And

Provider

just does not really make much sense in your case. Do you mean that task relying on those properties would still be

UP-TO-DATE

without the

Provider

wrappers? > I'd still say that a shared build service would be the better way to achieve what you said Yes, I understood it the first time you mentioned it. I want to explore that as well. Do you happen to know about any examples of shared build service? I don't want to spend a week trying to figure out how to write one that works in the intended way. > I still don't know why you insist on not using the proper way and why you even ask experts if you then ignore and do the opposite of what they tell you. 🤷‍♂️ Even though I want to try, I'm still afraid it will turn out to be overcomplicated. Think about it this way: reading a config file for build purposes sounds easy and it should be. If it's so difficult that almost nobody does it right (i.e. it turns out everyone does it in a wrong way), then IMO it means the build system is badly designed and not user friendly. If I created an app and people would then use it incorrectly, it would be my fault, not theirs. Also, nobody wants to reinvent the wheel, writing complicated machinery just to read a few properties from a file.

Do you mean that task relying on those properties would still be

UP-TO-DATE

without the

Provider

wrappers?

What do

Provider

have to do with up-to-dateness? So yes, most probably. (But as I said, Android is always special, so no guarantees for anything 😄)

Do you happen to know about any examples of shared build service?

More than its documentation shows? https://docs.gradle.org/current/userguide/build_services.html

Even though I want to try, I'm still afraid it will turn out to be overcomplicated.

Again, it will imho not be more complicated than what you have, probably even easier and you could already have the solution you intend in half the time you already spent finding a half-woking alternative to the proper way.

Think about it this way: reading a config file for build purposes sounds easy and it should be.

It is super easy. Just read the file and you are done. What makes it not so easy are your requirements which are not "just reading a file", but "reading a file only once in the build and use it in all projects safely". This is a much more complicated requirement and for it there is a pretty easy solution too: Use a shared build service. 😛

If it's so difficult that almost nobody does it right (i.e. it turns out everyone does it in a wrong way),

Not everyone does it wrong, I do it right for example. You shouldn't extrapolate from one sample (you) or even a few samples to the entirety of Gradle users. Especially as if you read something online then most probably from people not doing it right or not knowing how to do it right as the others just silently do it right.

then IMO it means the build system is badly designed and not user friendly. If I created an app and people would then use it incorrectly, it would be my fault, not theirs.

Feel free to complain to Gradle if you think so. I disagree to you, though I'm just a user like you.

Also, nobody wants to reinvent the wheel, writing complicated machinery just to read a few properties from a file.

Then why do you try to? 😄

@Vampire what do you think about this? https://github.com/gradle/gradle/issues/1534#issuecomment-605868060

Well, whenever you use

ext

or

extra

properties it usually is a work-around for doing something not properly. 😄 And additionally it very much depends on how it is used / how the users of the file are aware. Because if users of the file assume they can put any Project Property or Gradle Property in there, they are wrong. Gradle Properties cannot be put there at all. Project Properties can be put there if they are read with

project.findProperty

or similar means as those also look into extra properties. But if you (or some plugin) for example uses

providers.gradleProperty

(which despite its name reads Project Properties unfortunately) then the values from that file will not be considered. ...

The docs say: > Using shared build services from configuration actions > Generally, build services are intended to be used by tasks, and as they usually represent some potentially expensive state to create, you should avoid using them at configuration time. However, sometimes, using the service at configuration time can make sense. This is possible; call

get()

on the provider. So, correct me if I'm wrong, but this kind of usage:

// lib/build.gradle.kts

android {
  // let's assume I already have buildService reference from somewhere
  something = buildService.whatever
}

is also "not intended", right?

build services much outgrew their original intention. Also the same sentence says sometimes it makes sense at configuration time. The original intention is for example a build service that boots up a web server and does a deployment and waits for it to start up. This build service can then be used by 3 different test tasks, so that as soon as the first test task needs the server it is booted up and as soon as the last task needing the service has finished the service and thus web server can be shut down again. Such a service might be unlucky to be used at configuration time. But as I said, build service much outgrew that original intention. You can for example also use a build service to log something CC safe at the end of the build run. Or you can use a build service to control that several tasks never run in parallel for whatever reason without any actual logic in it, just as a kind of semaphore. Or to share information between different projects of the same build safely, also at configuration time.

Would a build service know that a file should be re-read when its content is changed? 🤔

The lifecycle of a build service is one execution of a build. So as soon as you run a new build the file is read freshly.

Will it re-read even if the same Gradle daemon is reused?

Really?

but the class was remaining in the daemon’s classloader

sounds to me as if the service's "state" was preserved across builds

class

not

instance

@Vampire Is this more or less how you imagine it?

// buildSrc/src/main/kotlin/MyBuildService.kt

abstract class MyBuildService : BuildService<MyBuildService.Params> {

    internal interface Params : BuildServiceParameters {
        val file: Property<java.io.File>
    }

    val properties: Properties = Properties()

    init {
        val file = parameters.file.get()
        println("Loading properties from $file")
        file.bufferedReader().use { properties.load(it) }
    }
}

// buildSrc/src/main/kotlin/MyBuildPlugin.kt

class MyBuildPlugin : Plugin<Project> {
    override fun apply(target: Project) {
        target.gradle.sharedServices.registerIfAbsent(
            "myBuildService",
            MyBuildService::class.java
        ) {
            parameters.file.set(target.rootProject.file("custom.properties"))
        }
    }
}

// libs/build.gradle.kts

val mbs = project.gradle.sharedServices.registrations.named("myBuildService").get().service.get() as MyBuildService

android {
    something = mbs.properties.getProperty("FIREBASE_APP_ID")
}

Because it seems to work. Whenever I run a build, I see only one occurrence of that

println("Loading properties from $file")

even though I applied the plugin in two subprojects. (I don't like the

registrations.named

part and the type casting but I couldn't figure out the type-safe way)

Seems fine from a quick look. At least if you will in any case need the service. As you

get()

it in the plugin already you will in any case create the service and thus read the file, even if doing

gradlew help

. It might be slightly better to stuff the

Provider

in the extension and on usage do

mbe.service.get().properties

or store the provider in the extension and in the getter for

service

to

get()

the provider, so that the service is really only created once it is first needed actually.

Ah, so finally a good use case for

Provider

? 😂 So I guess this is how it should be:

open class MyBuildExtension(
    val service: Provider<MyBuildService>
)

class MyBuildPlugin : Plugin<Project> {
    override fun apply(target: Project) {
        val serviceProvider = target.gradle.sharedServices.registerIfAbsent(
            "myBuild",
            MyBuildService::class.java
        ) {
            parameters.file.set(target.rootProject.file("local.properties"))
        }
        target.extensions.create("mbe", MyBuildExtension::class.java, serviceProvider)
    }
}

// libs/build.gradle.kts
android {
    something = mbe.service.get().properties.getProperty("FIREBASE_APP_ID")
}

right?

Ah, so finally a good use case for

Provider

?

That's not really the question here, you already do have a provider. You just prematurely get it instead of only when you need it. Never

get()

a

Provider

before you really need it, optimally at task execution time, especially when it is coming from something configurable eventually.

right?

From a quick look I'd say so, yes.

According to you, when using a

Provider

, it should not initialize the service while running

It initializes the service when you

get()

the provider. So if the line that does it is executed when you execute

help

, it will be initialized. If you for example would do that in a configuration action of a task, the service would not be initialized. If

something

accepts a

Provider

, it should be

provider = mbe.service.map { it.properties.getProperty("FIREBASE_APP_ID") }

then chances are high it also will not be initialized on

help

unless some other part then prematurely `get`s that property.

Ah, so it's the underlying implementation of

something

that must be compliant with `Provider`'s lazyless?

That, and it's users. Typically

something

would be a

Property

. But also everyone using

something

must behave properly and not prematurely get the value.

Got it. OK, so I guess that's it. You helped me immensely @Vampire 🙇 I appreciate all the guidance. Let me know if I can buy you a coffee sometime ☕ or a coffee tank, actually 😅

Sure, come by, I'm here. 🙂 Alternatively, feel free to PayPal me for example.