This message was deleted.
# troubleshootings
Slackbot
03/02/2023, 11:42 AMThis message was deleted.
b
Bibek Sahoo
03/02/2023, 11:45 AMThis is not because of SLOTS not available, If slots are not available the jobs will be in PENDING state, Here the state WAITING should be for other reason, could you check if it has any message printed in overlord/inside TASK log for what it is waiting for.
Bibek Sahoo
03/02/2023, 11:49 AMThe jobs are running on same data source and if the running job is locking the time interval then the other jobs which need locking on the same interval will be in WAITING state till the time it gets the lock so that it can run further
a
Abinash Kumar
03/02/2023, 11:52 AMYes, it is the same case. All tasks are for same data source and same time interval or granularity in ingestion spec.
👍 1
b
Bibek Sahoo
03/02/2023, 11:53 AMcorrect so jobs need to wait till it gets the lock on the timechunk
v
Vijay Narayanan
03/02/2023, 11:54 AMyou should run a single task and specify multiple sub tasks. this will ensure that you don't get into lock issues
a
Abinash Kumar
03/02/2023, 11:55 AMIs there any way to parallelize this? because i have huge number of different files for which I initiate different batch tasks..
v
Vijay Narayanan
03/02/2023, 11:57 AMpaste your ingestion spec here
a
Abinash Kumar
03/02/2023, 11:58 AM@Vijay Narayanan Do you mean increase maxNumConcurrentSubTasks": 1, in tuning config? afaik, this would only help if I have a large file. but I have multiple (in thousands) small individual files....
Abinash Kumar
03/02/2023, 12:00 PM@Vijay Narayanan {
"type": "index_parallel",
"spec": {
"ioConfig": {
"type": "index_parallel",
"appendToExisting": true,
"inputSource": {
"type": "s3",
"uris": ["xxxxxxx/some/object"],
"properties": {
"accessKeyId": "xxxxxxx",
"secretAccessKey": "xxxxxxxx"
}
},
"inputFormat": {
"type": "csv",
"columns": ["col1, col2"]
}
},
"tuningConfig": {
"type": "index_parallel",
"maxNumConcurrentSubTasks": 1,
"partitionsSpec" :{
"type":"dynamic",
"maxRowsPerSegment": 7000000
}
},
"dataSchema": {
"dataSource": "d1",
"timestampSpec": {
"column": "ts",
"format": "iso"
},
"dimensionsSpec": {
"dimensions": [
{
"type":"long",
"name":"col1"
},
{
"type": "string",
"name": "col2",
"createBitmapIndex":true
},
]
},
"granularitySpec": {
"segmentGranularity": "DAY",
"queryGranularity": "none",
"intervals" : [ "2022-09-14T000001.000000Z/2022-09-14T235959.000000Z" ],
"rollup":false
}
}
}
}
v
Vijay Narayanan
03/02/2023, 12:04 PMUse a prefix instead of uri to recursively fetch from a folder and set maxnumconcurrentsubtasks to 4
🙏 1
a
Abinash Kumar
03/02/2023, 12:06 PMi will check it out!! Thanks a lot!