This message was deleted.

Gradle repo itself is a monorepo with multiple gradle modules which has more than 1 million of lines.

Yep. But as an Android project, the whole build process has many serial actions, e.g. Android Transform. So theoretically it’s impossible for us to build as fast as Gradle. And now we are facing many build performance problems in our monorepo. Maybe it’s Google’s responsibility to optimize this?

how is your project currently structured? what do you mean by "make into a monorepo"?

For now, we have different gradle modules in different git repos. On CI, all the library modules are artifacts(jars and aars). On the local PC, we use dependency substitution to transform some artifacts into source code so we can change the code in library modules. As for “make into a monorepo”, we’re going to put all of the code into one git repo and structure the whole project as one Android Gradle project. To build the app, all the library modules will participate as source code. So it becomes much more slower than before. Because most modules are artifacts before.

What were separate tepid before would become sub projects (modules) in a Gradle multi module project. Gradle will only build what is necessary (what had changed), and run module builds in parallel. As noted earlier the Gradle project itself is an example of this. Modules still create (but don't necessarily publish) artifacts that are consumed by other modules.

Yep. For “Gradle will only build what is necessary”, I think you mean Build Cache or Incremental Build. As we have Java and Kotlin in our project, many compile task will not

UP_TO_DATE

or

FROM_CACHE

if we change some code in base modules, even if the changes are non-abi changes. For “builds in parallel”, as I mentioned before, there are many tasks can not be built in parallel in Android build process. And the configuration phase spends much more time than before, too. And it’s hard for us to use configuration cache. And if there’s no cache, it will be really painful.

You are misreading that doc imo. You would split the project into modules (subprojects) when there is a natural split (common, client, server e.g.)

FWIW, we have an Android app w/ 1000+ modules and 1.6M LoC in a monorepo...Square's repo is up to 3500+ modules last I heard too...

@Steve Ebersole thanks for your kind reply. We have already enabled parallel build. But for Android build, it will handle all of the ByteCode in one kind of task called Transform. Transform tasks run serially and cost lots of time.

M1 builds are around 13min for full build..incremental is 1min, and yes, all modules. Transform runs serially on each modules AFAIK...

@Ken Yee Wow, that’s really fast. As for transform, we have many transforms in the application module to handle all of the bytecode, so it’s really a bottle neck for our build. Maybe we can register them in every library module to make them run in parallel?

Not sure what to tell you... Hibernate is a very complicated build. That's not why I broke things out into separate modules though. In my opinion (well not just mine, this is general best practice) you do this to properly define dependencies and get away from silly stuff like optional xdependencies

@Steve Ebersole Actually we have about 500 modules not one single module. Maybe I didn’t express clearly.

Awesome. Did you read the link I posted?

@Steve Ebersole Yes, I read it and we have already make some optimizations in our projects using build scan. About parallelity, I agree with you. Maybe we can make our modules isolate better then. We didn’t pay much attention on it before. Thanks a lot.

The easy ones to drop are (1) the use of allprojects and subprojects and (2) any other direct refs across projects

@Steve Ebersole Got it. I have read

@CacheableTask

and documents about build cache. We found that build cache doesn’t work well when we have Kotlin together with Java in our project. Kotlin didn’t adapt to Gradle’s compile avoidance and made its own way to perform compile avoidance. This may be the root cause.

No idea. I don't develop in Kotlin

^^^ that’s referring to incremental compilation; caching is still in play - when nothing has changed, pull from the cache. Incremental is about when something has changed, compile the smallest set possible.

@Chris Lee CacheableTask? that absolutely is about build-cache

This:

Kotlin didn’t adapt to Gradle’s compile avoidance and made its own way to perform compile avoidance.

ah

^^^ that for semantic correctness, plus use Property/Provider to ensure lazy evaluation of values (if at all) and inferred dependency tracking between tasks.

Might help to qualify

^^^

😉

I think maybe you misunderstood me? When I said it is critical to get the inputs and outputs correct what I mean is that otherwise it is very likely you get incorrect results from the build cache

that is correct. my comment was to layer on Provider/Property in addition to that.

As an example... on one

processResources

I replace text during the copy

I just did this exact thing recently for Kotlin project going from 3 repos into 1. What I did first was to bang out a prototype of it to see how it all works. All my separate projects were already gradle projects so its relatively straight forward to combine them and test stuff out from a performance and workflow perspective. My over all developer workflow time improve and our build time was faster that building 3 projects separately.