Hi everyone, we had some issues with our database volumes and had to do a recovery from a backup that was a few months old. Everything seems to be back in place, but queries seem slower than before. Can we do something to improve the performance? Thanks!

hey couple of questions to understand what went wrong here: • Which version of OpenObserve are you running? • Is the slowness affecting all queries or only specific streams/time ranges? • After restoring from the backup, did you notice high CPU, memory, or disk I/O on the OpenObserve nodes?

thanks for getting back @Simran(OpenObserve) • v0.91.3 • all streams in general (logs and metrics) • nothing particular that we noticed, everything seems to be running as usual

okay let me check this.

whenever we open a dashboard or do a (very simple) log query, completion takes around 20-30 seconds, re-running the identical query completes after 6-7 seconds. Both values are much higher than what we experienced before the recovery. We are running O2 on a AKS Kubernetes cluster, storage is an Azure Blob Store Container and the database is running in-cluster (CNPG).

no idea, maybe you can track the query logs, where was it slow at. db slow, file slow, or something else.

@Hakan Kaya this could be a lot of things. A query that took 30 seconds and later 7 seconds is the cache that is in place. How much are you ingesting and what are your resource limits? Take a look at https://openobserve.ai/docs/enterprise-setup/capacity-planning/

ingestion is about 150-170GB per day, resource limits have not been changed. My point is, the performance was excellent before, then we did the recovery and boom - performance degraded significantly. I don't think it's about ingestion volume or resource allocation, none of these have been changed. I am aware that first and second query is different because of the cache, I was just trying to tell you that even with cache, it's still way too slow since the recovery.

in the api response.