Hey folks, something I learned about Pinot batch ingestion, today: it can be a bit picky with the input configuration, so for example:

inputDirURI: '/foo/bar'
includeFileNamePattern: 'glob:baz/**/*.json'

doesn't work if you want to ingest from JSON files inside

/foo/bar/baz

. Instead, this should be used:

inputDirURI: '/foo/bar/baz'
includeFileNamePattern: 'glob:**/*.json'

notice how

inputDirURI

goes to the deepest possible fixed subdirectory, and then the pattern will start from there.

Hi

**/baz/*.json

this should work for you You can use both

glob

or

regex

matcher here You can test out the patterns here - https://www.digitalocean.com/community/tools/glob?comments=true&glob=%2A%2A%2Fbaz%2F%2A[…]ik%2FDocuments%2FWorkspace%2Ffoo%2Fbar%2Fbacxxc%2Fdef.json For example - https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystem.html#getPathMatcher(java.lang.String)

Hi @User - thanks for sharing that very useful site! I’d written a little tool try glob patterns, but this is much easier and more powerful.

Thank you, guys! 🙂