Simply modularizing the project is not a guarantee to faster builds.
Each module need to build all the other modules it depends before being built. Therefore, the key here is how each module depend in each other.
It needs to be modularized carefully so you limit the number of modules depending between each other, maximizing the amount of modules that can be compiled in parallel.
You can see your module dependencies graph as a tree, where the main app module is the root. Then, your goal is to reduce the heigh of your tree from the root to the leaves. Ideally 3-4 levels at most.
If you are not sure about your modules graph, you can use this
plugin (and I encourage you reading also its documentation, as it also has some recommendations to achieve what I was describing above).
And you are quite correct when you mention that each module comes with some overhead. So think carefully which modules make sense for your project and try to be lean, not abusing of modules with just a few files.