This message was deleted.

I haven't tested this but I think you can do this with the

switching

request logger. Something like:

druid.request.logging.type=switching
druid.request.logging.nativeQueryLogger	# don't specify, defaults to None
druid.request.logging.sqlQueryLogger=emitter

ok let me give it a shot!

I didn't expect internal SQL queries, I thought all internal queries are native. Can you show me an example of such a query?

Yea so when logging sql queries, I get one that looks like this:

SELECT TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS

I get this logged when switching to the "Query" tab in the UI.

Yeah... Those are not really internal queries, at least not from the database's perspective. The Druid console behaves like another Druid client. But I agree that it would be good to filter those out. Maybe something like

druid.request.logging.excludeSchemas=["information_schema","sys"]

Do you want to create a github issue with the feature request? You can do that here: https://github.com/apache/druid/issues/new/choose

Ah I see. Yea I can create an issue. I had one last question. Is it possible to put the native queries into one topic and the sql queries into another topic? I tried configuring with

switching

and setting the types of both the

nativeQueryLogger

and

sqlQueryLogger

to

{"type": "emitter", "feed": <feed-name>}

but it seems that it only logs the sql queries to the topic I created using the

kafka-emitter

extension. It logs the proper sql query and feed, but the native query is nowhere to be found. I feel like I'm missing some configs. Thank you!

That sounds interesting, I have not tried that yet, but would love to see what you come up with that works. Seems doable in principle.

Yea will definitely let you know. Also, I was curious if you knew. But do you know if

kafka-emitter

supports emitting with formats other than JSON (e.g. Thrift)?

I am also considering filtering on request logging. Have you had any luck? I tried this but it didn't work for me 😞

druid.request.logging.type=switching
druid.request.logging.sqlQueryLogger=file
druid.request.logging.dir=log/
druid.request.logging.filePattern='request-'yyyy-MM-dd'.log'
druid.request.logging.durationToRetain=P7D

I also tried this:

druid.request.logging.type=filtered
druid.request.logging.sqlQueryTimeThresholdMs=5000  // Log requests that took longer than 5s
druid.request.logging.delegate.type=file
druid.request.logging.dir=log/
druid.request.logging.filePattern='request-'yyyy-MM-dd'.log'
druid.request.logging.durationToRetain=P7D

But again no luck. The only one I got working is this (but it logs WAAAAY too much):

druid.request.logging.type=file
druid.request.logging.dir=log/
druid.request.logging.filePattern='request-'yyyy-MM-dd'.log'
druid.request.logging.durationToRetain=P7D