https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
t
@Yogesh Singh - Can you share an example of the query used?
v
did I read that right you are on Druid 0.9.1 ?
y
@Vadim yeah you read right , we are using very older and in process to backfill druid data .
@Tish Johnson Sharing example logs , threshold : 5000 and fromnext: true . Using while loop and exit from while loop once we get empty result set but we don't get empty result set because getting same paging identifiers repeatedly . test-ds_2021-10-13T000000.000Z_2021-10-13T080100.000Z_2022-02-07T050925.413Z test-ds_2021-10-13T000000.000Z_2021-10-13T080100.000Z_2022-02-07T050925.413Z_1
Copy code
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 2708, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 2290}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 5451, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 4547}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 9388, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 5610}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 10421, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 9577}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 15421}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 4999}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 4999}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 4547, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 5451}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 9388, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 5610}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
v
I have not used the 
select
query in a while - it was replaced with 
scan
at some point
have you tried not using 
fromNext
and just incrementing the identifiers by 1 yourself?
also are you just exporting all the data without a filter - if so you could also use the dump segment tool (just wanted to mention)
y
yes i tried using fromnext:false and increment offset manually by code but it does not work in case of multi-partitions . i have designed complete backfill logic to pull druid data using backfill framework which runs on workflow engine and scalable , we backfilled around 50 TB raw data using this framework . facing an issue in one of the corner case .
v
yes, I remember that when Plywood (the toolkit I built for querying Druid) used 
select
(it long doesn't anymore) it used to do the increment client side https://github.com/implydata/plywood/blob/d2ff8dff936baaadbc93396e8c65b39486fe837e/src/external/druidExternal.ts#L447
I wonder why the multi-partition stuff does not work for you, did you make sure to increment all the counters like the code ☝️ does?
y
yes , few folks also faced similar issues but those were closed due to inactivity . https://github.com/apache/druid/issues/4849
Also fromnext:true it increment offset automatically and take cares internally and it should work with multi partitions but not sure why it's happening with multi-partitions . logs shared above .
@Vadim @Tish Johnson thanks ! I added a fix at code level .
Copy code
global paging
    paging_check={} 
      
        for key, value in paging.items():
                            if not paging_check:
                                paging_check.update(paging)
                            if key not in paging_check:
                                paging_check[key] = value
                            if value > paging_check[key] and paging_check[key] != -1:
                                paging_check[key] = value
                            if value < paging_check[key]:
                                paging_check[key] = -1
                        <http://logger.info|logger.info>("Custom hashmap {}".format(paging_check))
                        for key1, value1 in paging_check.items():
                            if value1 == -1:
                                for key1 in list(paging.keys()):
                                    del paging[key1]
                            else:
                                paging[key1] = value1
                            <http://logger.info|logger.info>(
                                "pagingIdentifiers for next Query through backfill code: {0}".format(paging))

                        if all(v == -1 for v in paging_check.values()):
                            events = []
Open in Slack
PreviousNext
Powered by