Does anyone have a sample job spec for loading up ...
# troubleshooting
s
Does anyone have a sample job spec for loading up segments using a file pattern? I tried this and it didn't recognize the pattern
m
@Rong R ^^
k
What’s the pattern? And what version of Pinot are you running?
Oh, I see you posted this as a separate item in troubleshooting, copying here for this thread:
Copy code
outputDirURI: '<gs://blah/data/immutable_events>'
includeFileNamePattern: "glob:**/*1661990400000*"
What’s a file in the
Copy code
<gs://blah/data/immutable_events>
directory which isn’t getting matched?
s
So I passed that include file pattern and the job seemed to ignore it and loaded all the tar files in the directory
k
And what version of Pinot are you running?
s
.11
k
I added support for filtering files that are pushed, in https://github.com/apache/pinot/pull/8191. This uses the new
pushFileNamePattern
config to control which files get pushed. Based on your comment about “…loaded all the tar files” I’m guessing this is what you want, for a push job.
s
Awesome I’ll try it out thanks!