This message was deleted.

Have you thought of a kafka stream for it?

What type and volume of data is this?

I wouldn't call it "big data". Hundreds of megabytes, maybe a few gigabytes

What does the data represent?

this is for my dependency analysis plugin (on a very large project, hence the large amount of data). Metadata about the subprojects under analysis, their dependencies, and various inferred relationships

Sqlite could be nice if you have a remote build cache through which you can share the database. It's just a file and as such perfectly compatible with Gradle's caching. Of course, care must be taken who's allowed to update it, but that problem you have with any solution.

Have you thought about just using a more compact binary format than Json for the files (like protobuf, cbor or similar)? I think that what I would try first (and see if that sufficient). Gradle is so file oriented, that I think this is the better direction. Both for implementing things correctly that they work with input/ouput/caching and also for a user (like me) to understand what the plugin is doing.

I agree that files are outputs and everyone understands what the task does, so if it matters, for users it's better if you use a file, which doesn't need to be a json, it can be sqlitle or binary.

don't forget one of my criteria, which was:

which would also permit queries outside of the context of gradle

🙂 Maybe one idea is to continue to use human-readable json and also serialize the data into a database. I kind of just want a better way to query the data than writing custom little one-off gradle tasks and opening a debugger 😄

If it's a requirement and not a nice to have then a real DB is the way to go. I mean, you don't need the caching in this case, because the database keeps track of things. This should be fairly easy to implement with a build service. Just be sure to commit stuff only from the right places, or the DB is going to become an inconsistent mess. 😅

you could also continue using files locally and provide a way to publish them to a remote thing

good thoughts, thank you!