@here is the any max limit on number of prefixes i...
# general
a
@here is the any max limit on number of prefixes in inputsource of type s3. or any max payload size limit on that ?
Copy code
"ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "s3",
        "objectGlob": "**.json",
        "uris": ["<s3://foo/bar/file.json>", "<s3://bar/foo/file2.json>"]
      },
d
I never tried with more than few dozen records, so I'm not sure. But prob the best solution is to structure them into a prefix structure, so instead of "uris"
Copy code
"uris": ["<s3://foo/bar/file.json>", "<s3://bar/foo/file2.json>"]
you can use
Copy code
"prefixes": ["<s3://foo/bar/>", "<s3://bar/foo/>"]
b
I do not think there is any specific limit. You might use splithintspec to create separate tasks depending on the size of the data. https://druid.apache.org/docs/latest/ingestion/native-batch#split-hint-spec
a
may be there is limit on max size of input spec if I am not able to submit task with more than 512Kb input spec