This message was deleted Apache Druid #troubleshooting

Join Slack

This message was deleted.

# troubleshooting

Slackbot

09/20/2022, 3:36 AM

This message was deleted.

💪 3

🙌 3

Vadim

09/20/2022, 3:42 AM

There is this: https://druid.apache.org/docs/latest/tutorials/tutorial-msq-extern.html

👍 1

Vadim

09/20/2022, 3:42 AM

Also the new quickstart for which I just made a video:

https://www.youtube.com/watch?v=PZpu8hOfRJw▾

(careful, it is still hot off the press)

Vadim

09/20/2022, 3:44 AM

Also I made a longer video for the stuff right before the 24.0.0 code freeze that 99% still applies

https://www.youtube.com/watch?v=xUzYuPfzARk▾

❤️ 1

Vadim

09/20/2022, 3:45 AM

(I will give $50 to the first person that sports a bug featured in that video that was fixed before the 24.0 release - sometimes I QA by making videos)

Vadim

09/20/2022, 3:45 AM

(hint: it is a console bug :-p)

Didip Kerabat

09/20/2022, 3:45 AM

I got:

Copy code

Error: Unknown exception

org.apache.druid.rpc.HttpResponseException: Server error [400 Bad Request]; body: {"error":"Please make sure to load all the necessary extensions and jars with type 'query_controller' on 'druid/overlord' service. Could not resolve type id 'query_controller' as a subtype of `org.apache.druid.indexing.common.task.Task` known type ids = [archive, compact, index, index_hadoop, index_kafka, index_kinesis, index_parallel, index_realtime, index_realtime_appenderator, index_sub, kill, move, noop, partial_dimension_cardinality, partial_dimension_distribution, partial_index_generate, partial_index_generic_merge, partial_range_index_generate, restore, single_phase_sub_task]\n at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 9]"}

java.lang.RuntimeException

I enabled the multi-stage-query extension already

Vadim

09/20/2022, 3:46 AM

are you sure it is loaded on your overlord? that error means it is not loaded

Didip Kerabat

09/20/2022, 3:46 AM

they are all packaged together in 1 docker image, but I can check

Vadim

09/20/2022, 3:47 AM

Finally saving the best for last: you can take a self guided tour by loading the demo queries

❤️ 1

Vadim

09/20/2022, 3:49 AM

oh, if you are using docker make sure to edit the

environment

file, see https://github.com/apache/druid/pull/13069

Didip Kerabat

09/20/2022, 4:49 AM

EXTERN

not able to read from S3? e.g.

Copy code

{
  "type": "s3",
  "uris": [
    "<s3://mybucket/multi-stage-query/kttm-v2-2019-08-25.json.gz>"
  ]
}

Didip Kerabat

09/20/2022, 4:50 AM

I got

Request failed with status code 403

Didip Kerabat

09/20/2022, 4:53 AM

default

admin

role should have full

EXTERNAL

access, yeah?

Karan Kumar

09/20/2022, 5:10 AM

EXTERNAL

read perms is required

Karan Kumar

09/20/2022, 5:19 AM

Are you connecting to a fresh database or an old db instance with druid data filled in ?

Vadim

09/20/2022, 5:22 AM

See the 3rd paragraph here: https://druid.apache.org/docs/latest/multi-stage-query/index.html#load-the-extension

Didip Kerabat

09/20/2022, 2:12 PM

I upgraded an old instance

Karan Kumar

09/20/2022, 5:26 PM

I will try to repro it tomorrow. I have a hunch into what might be happening.

🙏 1

Gian Merlino

09/21/2022, 12:47 AM

did you end up getting it to work? if it's an old instance w/ security enabled, you do need to manually add the EXTERNAL / EXTERNAL permission to the admin role

Gian Merlino

09/21/2022, 12:48 AM

it is automatically added to new admin roles for fresh installs, but not for an upgrade of an existing install

Didip Kerabat

09/21/2022, 2:04 AM

Yup yup, I think that’s the problem. I haven’t had the time to try it yet, will try tomorrow.

Karan Kumar

09/21/2022, 2:15 AM

it is automatically added to new admin roles for fresh installs, but not for an upgrade of an existing install

I was thinking of changing this part. Upgrades basically add/remove the new roles

Karan Kumar

09/21/2022, 2:17 AM

Copy code

CoordinatorBasicAuthorizerMetadataStorageUpdater#initSuperUsersAndGroupMapping

@Gian Merlino wdyt ?

Karan Kumar

09/21/2022, 2:18 AM

If we cannot do it for all, I was thinking we can make basic auth work seamlessly at-least.

Gian Merlino

09/21/2022, 3:29 AM

yeah, we can't do it for all auth mechanisms, but it would be great to do it automatically for basic

Gian Merlino

09/21/2022, 3:29 AM

we could do that at least

Didip Kerabat

09/21/2022, 3:33 AM

I see resource.type = VIEW, what is that?

Didip Kerabat

09/21/2022, 3:35 AM

Also, is there an EXTERNAL WRITE role?

Vadim

09/21/2022, 3:36 AM

There is external write (by symmetry) but it does not do anything right now

Vadim

09/21/2022, 3:37 AM

VIEW resources are for views created in Druid

Didip Kerabat

09/21/2022, 3:38 AM

Thanks!

Didip Kerabat

09/21/2022, 3:43 AM

Dope, it works!!! So fancy.

Didip Kerabat

09/21/2022, 3:45 AM

The query bombed though…

Copy code

java.lang.RuntimeException: org.apache.druid.msq.indexing.error.MSQException: WorkerFailed: Worker task failed: [query-1986e125-4ca3-484f-853e-44971a8a0cca-worker0] (This task disappeared on the worker where it was assigned. See overlord logs for more details.)
	at org.apache.druid.common.guava.FutureUtils.getUnchecked(FutureUtils.java:78)
	at org.apache.druid.common.guava.FutureUtils.getUncheckedImmediately(FutureUtils.java:90)
	at org.apache.druid.msq.exec.ControllerImpl$RunQueryUntilDone.lambda$startTaskLauncher$1(ControllerImpl.java:1970)
	at org.apache.druid.msq.exec.ControllerImpl$RunQueryUntilDone.runKernelCommands(ControllerImpl.java:1944)
	at org.apache.druid.msq.exec.ControllerImpl$RunQueryUntilDone.run(ControllerImpl.java:1925)
	at org.apache.druid.msq.exec.ControllerImpl$RunQueryUntilDone.access$000(ControllerImpl.java:1877)
	at org.apache.druid.msq.exec.ControllerImpl.runTask(ControllerImpl.java:352)
	at org.apache.druid.msq.exec.ControllerImpl.run(ControllerImpl.java:296)
	at org.apache.druid.msq.indexing.MSQControllerTask.run(MSQControllerTask.java:192)
	at org.apache.druid.indexing.overlord.ThreadingTaskRunner$1.call(ThreadingTaskRunner.java:210)
	at org.apache.druid.indexing.overlord.ThreadingTaskRunner$1.call(ThreadingTaskRunner.java:152)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: org.apache.druid.msq.indexing.error.MSQException: WorkerFailed: Worker task failed: [query-1986e125-4ca3-484f-853e-44971a8a0cca-worker0] (This task disappeared on the worker where it was assigned. See overlord logs for more details.)

Didip Kerabat

09/21/2022, 3:53 AM

So….… looks like MM is involved in querying now. Does that mean I need to have a separate MM pool just for MSQ?

Gian Merlino

09/21/2022, 3:56 AM

It doesn't need to be separate necessarily but it does need to exist

Gian Merlino

09/21/2022, 3:57 AM

We're using MMs since this iteration is focused on batch query and ingest (like

INSERT ... SELECT

). So tasks/MMs are a natural fit

Didip Kerabat

09/21/2022, 4:12 AM

I love the idea of using MM, in 1 of my largest cluster they are sitting idle after the big ingest window is finished. They need to earn their keeps 😝

🚀 1

Open in Slack

Previous Next