https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team， I tried to add dependency between jobs. A...
# general
k
Hi team， I tried to add dependency between jobs. And Job DAG has been constructed. But Pinot seems don't execute task like i wish. So I hope someone could tell me the mechanism of acquire task in pinot. Below is the code that acquire task in pinot. But it is a callback method. So I don't know the pipeline.
Copy code
TaskFactory taskFactory = context -> {
        try {
          return new Task() {
            private final TaskConfig _taskConfig = context.getTaskConfig();
            private final PinotTaskExecutor _taskExecutor = taskExecutorFactory.create();
            private final MinionEventObserver _eventObserver = eventObserverFactory.create();
            private final MinionMetrics _minionMetrics = MinionContext.getInstance().getMinionMetrics();

            @Override
            public TaskResult run() {
              HelixManager helixManager = context.getManager();
              JobContext jobContext = TaskDriver.getJobContext(helixManager, context.getJobConfig().getJobId());
              // jobContext.getStartTime() return the time in milliseconds of job being put into helix queue.
              long jobInQueueTimeMs = jobContext.getStartTime();
              long jobDequeueTimeMs = System.currentTimeMillis();
@Jackie @Mayank @Haitao Zhang
h
what is 
JobDAD
? what does it mean that 
JobDAD has been constructed
? and what is 
dependency between jobs
?
m
Perhaps DAG?
👍 1
h
the basic task workflow is: the task generator generates PinotTaskConfigs -> submit to helix framework -> helix assign task to minion workers -> minion workers get PinotTaskConfigs and run tasks
k
Sure DAG
@Haitao Zhang @Mayank We develop a new minion task that will generate multi task. But we need those task executed one by one. We learn from helix doc that workflow can add job dependency to achieve this.
Thanks @Haitao Zhang, But I actually want to know pinot minion how to get PinotTask from Helix Framework.
Here are the info in /PinotCluster/Configs/Resource/OurTask. You can find that Dag info. { "id" : "TaskQueue_SegmentMigrationAndBackfillTask", "simpleFields" : { "AllowOverlapJobAssignment" : "false", "Dag" : "{\"parentsToChildren\":{\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_6943500f-a0df-4190-a840-9748300e9773_1670431857430\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_a2d1102f-130a-48c8-bdde-4d75895b8ce6_1670431857674\"],\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_a2d1102f-130a-48c8-bdde-4d75895b8ce6_1670431857674\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_e8f62e7d-f6ca-44f4-ba62-19156a3d7d1c_1670431857913\"],\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_e8f62e7d-f6ca-44f4-ba62-19156a3d7d1c_1670431857913\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_b4cc3b2d-9db9-4451-8421-ca67e9fa033c_1670433175534\"],\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_b4cc3b2d-9db9-4451-8421-ca67e9fa033c_1670433175534\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_5ed99318-5aa9-41a4-b7e6-b8e2e5b98141_1670433175662\"],\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_5ed99318-5aa9-41a4-b7e6-b8e2e5b98141_1670433175662\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_bdcbde6f-4cc0-4df3-9b76-ba5fb8c457ec_1670433175904\"]},\"childrenToParents\":{\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_a2d1102f-130a-48c8-bdde-4d75895b8ce6_1670431857674\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_6943500f-a0df-4190-a840-9748300e9773_1670431857430\"],\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_e8f62e7d-f6ca-44f4-ba62-19156a3d7d1c_1670431857913\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_a2d1102f-130a-48c8-bdde-4d75895b8ce6_1670431857674\"],\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_b4cc3b2d-9db9-4451-8421-ca67e9fa033c_1670433175534\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_e8f62e7d-f6ca-44f4-ba62-19156a3d7d1c_1670431857913\"],\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_5ed99318-5aa9-41a4-b7e6-b8e2e5b98141_1670433175662\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_b4cc3b2d-9db9-4451-8421-ca67e9fa033c_1670433175534\"],\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_bdcbde6f-4cc0-4df3-9b76-ba5fb8c457ec_1670433175904\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_5ed99318-5aa9-41a4-b7e6-b8e2e5b98141_1670433175662\"]},\"allNodes\":[\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_6943500f-a0df-4190-a840-9748300e9773_1670431857430\",\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_b4cc3b2d-9db9-4451-8421-ca67e9fa033c_1670433175534\",\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_bdcbde6f-4cc0-4df3-9b76-ba5fb8c457ec_1670433175904\",\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_e8f62e7d-f6ca-44f4-ba62-19156a3d7d1c_1670431857913\",\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_5ed99318-5aa9-41a4-b7e6-b8e2e5b98141_1670433175662\",\"TaskQueue_SegmentMigrationAndBackfillTask_Task_SegmentMigrationAndBackfillTask_a2d1102f-130a-48c8-bdde-4d75895b8ce6_1670431857674\"]}", "Expiry" : "86400000", "FailureThreshold" : "0", "IsJobQueue" : "true", "JobPurgeInterval" : "1800000", "MONITORING_DISABLED" : "true", "ParallelJobs" : "2147483647", "TargetState" : "START", "Terminable" : "false", "WorkflowID" : "TaskQueue_SegmentMigrationAndBackfillTask", "capacity" : "2147483647" }, "mapFields" : { "JobTypes" : { } }, "listFields" : { } }
h
Tasks are assigned to minion workers by the helix framework. I guess in your case, the key is to set dependency relationship correctly. We don't need to change how tasks are assigned
If you can share more details, we can try to come up with a solution. We can set up a meeting to discuss the details if you want
k
@Haitao Zhang Sure. I think you are right. the key is to set dependency relationship correctly. I could provide all detail that you need. And If you are free, I wish we could have a meeting. When is good time for you ?
h
I can do 5pm ~ 6pm PST or 8pm ~ 11pm PST, let me know which day works for you.
k
Can we have a meeting at Tomorrow 8pm ~ 11pm PST?Are you free? @Haitao Zhang
h
yes, that works for me, can you set up a meeting and share the meeting link with me?
k
@Haitao Zhang I have sent you private message.
Meeting link: https://go.webex.com/go/j.php?MTID=m09378726825ad38083c1d5a3abaa990f Meeting number: 2003 607 2437 Meeting password: jN3HxnXBe88 Join from a video or application Dial 20036072437@webex.com You can also dial 173.243.2.69 and enter your meeting number. Meeting password for video system 56349692 Join by phone +1-415-655-0000 US Toll +1-877-668-4488 US Toll Free Access code: 20036072437 Global call-in numbers https://go.webex.com/go/globalcallin.php?MTID=mbeb30e4dc7429c420eeeee0654750b41 Toll-free calling restrictions https://cisco.com/go/tollfree-restrictions Meeting password for audio 56349692
Thanks @Haitao Zhang for your suggestion.
Open in Slack
PreviousNext
Powered by