This message was deleted.
# general
s
This message was deleted.
e
These are the metrics exported by a “server” service. I would like to know the disk space used by a “store” could you please point me to the metrics I should read ? or in case that there is no metric, to the admin tool command I should use ? it seems that “describe-stores” doesn’t show that piece of information
I would also like to have a metric about the number (and size?) of queries on the router service
cc @Felix GV
f
Disk space used would be:
venice_server_test_store_persons_disk_usage_in_bytes_Gauge
(the prefix is
venice_server_
+ the store name +
_
)
e
thanks
I will try
the value in my snippet makes sense
👍
f
For router queries, they are broken down by type. The “single gets” have no special identifier, the “batch gets” are labeled “multi_get” and the read compute are labeled “compute”… for the last two, there is also a streaming version of them (e.g. “streaming_multi_get”). Then besides that, you can look at the total this router instance is getting (e.g.
venice_server_total_success_request_OccurrenceRate
for single gets,
venice_server_total_multiget_success_request_OccurrenceRate
,
venice_server_total_compute_success_request_OccurrenceRate
) and there is per-store for all of these also, e.g.
venice_server_test_store_persons_success_request_OccurrenceRate
BTW, the metrics dump you shared is for the server only, right?
e
yes
👌 1
I can extract the router metrics as well
f
On the router there is a
response_size
metric, also broken down by request type, as well as by total and by store.
e
this is the router
f
Sure, LMK if you have any more metrics questions. I believe I answered all of those you originally asked (?)
e
yes
🤝 1
perfect !
btw, it is not a problem if nobody complains 😉
f
Great. One more thing on metrics… Venice uses Tehuti as the lib for its metrics… this was forked from Kafka’s internal metrics package many many years ago… At LinkedIn, we implement the MetricsReporter interface and extract metrics this way. Seems like Kafka still uses some descendant of this code, and it may be possible to have some kind of adapter so that any plugin for reporting metrics out of a Kafka broker/client could be used to report Venice metrics too.
Yeah, the slowness of some metrics is on our radar, including this one… it does cause issues when there are too many stores in the same host. We will optimize that.
e
I discovered Tehuti today 🙂 It doesn’t seem too bad. In Pulsar we use Prometheus and DataSketches
the JMXExporter works decently for my initial POC
🤝 1
we can build a native Prometheus exporter on day
f
Sounds great.
Looking at Kafka -> Prometheus, I haven’t found a single guide/solution that implements this interface from the Kafka codebase… seems like they all pull from JMX, e.g https://epsagon.com/observability/a-complete-guide-for-monitoring-apache-kafka-part-1/
I would have thought people would be interested in skipping JMX, but apparently not. Oh well, that’s fine 🤷