This message was deleted.

are you getting this error in ingestion also?

this kind of validation error happens when you reference a table that doesn't exist or doesn't yet have any available data if the ingestion didn't work, i'd start there if the ingestion did work, then it's possible it's a data availability problem, perhaps because servers can't talk to each other or can't talk to deep storage

Agree. @Vijay Narayanan yes, during ingestion as well. task logs-

"sqlQueryContext" : {
    "finalizeAggregations" : false,
    "maxParseExceptions" : 0,
    "sqlQueryId" : "2c61e012-3ec8-458b-bce8-335a9b8c98b3",
    "groupByEnableMultiValueUnnesting" : false,
    "sqlInsertSegmentGranularity" : "\"HOUR\"",
    "maxNumTasks" : 65,
    "sqlReplaceTimeChunks" : "all",
    "queryId" : "2c61e012-3ec8-458b-bce8-335a9b8c98b3"
  },
  "sqlTypeNames" : [ "TIMESTAMP", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "ARRAY", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "BIGINT", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "ARRAY", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "BIGINT", "BIGINT", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "OTHER", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "DOUBLE", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "DOUBLE", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "BIGINT", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "ARRAY", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "BIGINT", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "ARRAY", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "OTHER", "VARCHAR", "VARCHAR", "BIGINT", "VARCHAR", "BIGINT", "BIGINT", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "VARCHAR", "BIGINT" ],
  "context" : {
    "forceTimeChunkLock" : true,
    "useLineageBasedSegmentAllocation" : true
  },
  "groupId" : "query-2c61e012-3ec8-458b-bce8-335a9b8c98b3",
  "dataSource" : "rollupall_pct100-4",
  "resource" : {
    "availabilityGroup" : "query-2c61e012-3ec8-458b-bce8-335a9b8c98b3",
    "requiredCapacity" : 1
  }
}
2023-03-28T18:09:11,067 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Attempting to lock file[var/druid/task/query-2c61e012-3ec8-458b-bce8-335a9b8c98b3/lock].
2023-03-28T18:09:11,071 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Acquired lock file[var/druid/task/query-2c61e012-3ec8-458b-bce8-335a9b8c98b3/lock] in 4ms.
2023-03-28T18:09:11,080 INFO [NodeRoleWatcher[OVERLORD]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [<http://localhost:8081>] of role [overlord] detected.
2023-03-28T18:09:11,080 INFO [NodeRoleWatcher[COORDINATOR]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [<http://localhost:8081>] of role [coordinator] detected.
2023-03-28T18:09:11,081 INFO [NodeRoleWatcher[OVERLORD]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role [overlord] is now initialized with 1 nodes.
2023-03-28T18:09:11,081 INFO [NodeRoleWatcher[COORDINATOR]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role [coordinator] is now initialized with 1 nodes.
2023-03-28T18:09:11,267 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
2023-03-28T18:09:11,276 INFO [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Running task: query-2c61e012-3ec8-458b-bce8-335a9b8c98b3
2023-03-28T18:09:11,279 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [SERVER]
2023-03-28T18:09:11,284 INFO [main] org.eclipse.jetty.server.Server - jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.12+8-LTS-237
2023-03-28T18:09:11,321 INFO [main] org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
2023-03-28T18:09:11,321 INFO [main] org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
2023-03-28T18:09:11,322 INFO [main] org.eclipse.jetty.server.session - node0 Scavenging every 660000ms
2023-03-28T18:09:11,444 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
2023-03-28T18:09:11,986 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@5032b2e{/,null,AVAILABLE}
2023-03-28T18:09:12,003 INFO [main] org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@428e9be2{HTTP/1.1, (http/1.1)}{0.0.0.0:8100}
2023-03-28T18:09:12,004 INFO [main] org.eclipse.jetty.server.Server - Started @9530ms
2023-03-28T18:09:12,004 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [ANNOUNCEMENTS]
2023-03-28T18:09:12,005 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started lifecycle [module]
2023-03-28T18:09:45,655 INFO [task-runner-0-priority-0] org.apache.druid.msq.exec.ControllerImpl - Query [c5981aa4-dfc0-4fea-b50f-58cede004614] starting 2 workers for stage 0.
2023-03-28T18:20:02,630 WARN [task-runner-0-priority-0] org.apache.druid.msq.exec.ControllerImpl - Controller: Work failed; task query-2c61e012-3ec8-458b-bce8-335a9b8c98b3; host localhost:8100: TaskStartTimeout: Unable to launch all the worker tasks in time. There might be insufficient available slots to start all the worker tasks simultaneously. Try lowering 'maxNumTasks' in your query context to lower than [3] tasks, or increasing capacity. (java.lang.RuntimeException: org.apache.druid.msq.indexing.error.MSQException: TaskStartTimeout: Unable to launch all the worker tasks in time. There might be insufficient available slots to start all the worker tasks simultaneously. Try lowering 'maxNumTasks' in your query context to lower than [3] tasks, or increasing capacity.)
2023-03-28T18:20:02,736 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "query-2c61e012-3ec8-458b-bce8-335a9b8c98b3",
  "status" : "FAILED",
  "duration" : 651451,
  "errorMsg" : "TaskStartTimeout: Unable to launch all the worker tasks in time. There might be insufficient availab...",
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}
2023-03-28T18:20:02,755 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2023-03-28T18:20:02,758 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2023-03-28T18:20:02,818 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@428e9be2{HTTP/1.1, (http/1.1)}{0.0.0.0:8100}
2023-03-28T18:20:02,818 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging
2023-03-28T18:20:02,830 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@5032b2e{/,null,STOPPED}
2023-03-28T18:20:02,864 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2023-03-28T18:20:02,870 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[query-2c61e012-3ec8-458b-bce8-335a9b8c98b3].
2023-03-28T18:20:02,870 INFO [main] org.apache.druid.msq.exec.ControllerImpl - Query [c5981aa4-dfc0-4fea-b50f-58cede004614] canceled.
2023-03-28T18:20:02,915 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2023-03-28T18:20:02,934 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2023-03-28T18:20:03,069 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x1000393fdca0011 closed
2023-03-28T18:20:03,070 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000393fdca0011
2023-03-28T18:20:03,119 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task

Did the controller task fail before the workers started? Not sure if this is your issue, but in order for a SQL-based ingestion to begin, it needs to have all the worker slots available for the controller and worker tasks. If they do not all start because there are not enough worker slots, then the whole job will fail.

it looks like controller failed first. for this error, how can i make sure about worker availability all the time?

Unable to launch all the worker tasks in time. There might be insufficient available slots to start all the worker tasks simultaneously. Try lowering 'maxNumTasks' in your query context to lower than [3] tasks, or increasing capacity.   btw i am running druid in my machine using cmd on 25.0 -> ./bin/start-druid

By adding more MMs or adding more capacity to existing MMs (more CPU and memory) and adjusting the druid.worker.capacity of each MM (set to CPUs-1).