https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
b
Interesting. It makes sense that read committed would be faster. When you say "Postgres FYI defaults to READ_COMMITTED", you mean when it's used as a metadata store, druid goes happily with that isolation level? If so, it makes sense to me that we could do the same with MySQL.
a
correct
postgres by default, defaults to read_committed, and we have about 2 druid clusters using it today with the default READ_COMMITTED isolation level. No issues weve observed even when scaling them to around 1200 peons.
task/action/runtime is around 425ms - 1 sec
b
Have you tried setting the default isolation level to read committed in MySQL? I'm not a guru, but if it works in Postgres, I'd expect it to do so in MySQL.
a
yep see the above benchmarks i posted
b
nice... chatAsync is on the kafka or kinesis side, right? You set read_commited for MySQL, not kafka, it sounds like, which makes sense.
a
chatAsync is enabled on the kafka side, and we configured read_commited on the mysql side
b
You don't know how much either one contributed separately, do you (out of curiosity)?
a
we actually do, we ran an apples to apples test, with chatAsync enabled for both REPEATABLE_READ enabled and for READ_COMMITTED as a pre/post on the same mysql database instance.
mysql database version we were using is 5.7
benchmark was executed using the exact same supervisor with a static amount of data flowing into the topic
b
afaik druid doesn't configure MySQL, but maybe the docs could recommend setting that?
a
yep, basically provide it as a recommendation is what I would go with
id argue the above recommendation would also resolve some of the issues users are experiencing with slow task startup times, and stability issues with updating task statuses at a bare minimum
b
+1 I'm going to look into that. (I'm not on docs, but I can ask some colleagues.) btw I also wonder whether a larger chatThreads setting would help; I don't know the details of chatAsync and why the default is false.
p
@Arun C Thanks for posting the thread, READ_COMMITTED make sense for large meta tables like tasks and segments, At least for meta tables, we are not worried about the consistent view in REPEATABLE_READ mode, great to know that it is helping significantly in reducing the latencies. I am still curious about hows is it causing the deadlocks and persist errors, how many realtime replicas do you have for each peon? We had also added additional indexes on the task table in druid 27, that should also help in optimizing the queries on task apis.
a
It seems like when updating the commit offsets (druid_dataSources) prior with REPEATABLE_READ we would often get
Copy code
org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator - Not updating metadata, existing state
this seems to be directly correlated to the number of tasks for a given supervisor + the transaction isolation level configured on the DB, as after the configuration changes we have not noticed this issue, but a further deep dive is needed to understand exactly what was causing it.
b
What was the performance difference for the apples-to-apples test, out of curiosity? (Only changing isolation level.)
@Didip Kerabat you have a big cluster with lots of peons, don't you? Do you use Postgres for metadata, or if not, do you set isolation level? (If so, any observations?)
d
We do use PG for all of our clusters. We use the default which is 
READ_COMMITTED
by default as mentioned here. We never have any issues related to this.
a
Ben it was 15 to 23 sec for task/action/run/time compared to 1 seconds after changing to read committed
Didip what version of pg are u running
same with Druid version + peon count
d
PG 11, all druids are 25
a
How many tasks on average do they handle
d
In the largest cluster, thousands of sub tasks.
2 Views
Open in Slack
PreviousNext
Powered by