image.png

GreptimeDB uses a portion of memory as a write buffer, which is released after a flush. Another portion of memory remains resident, used to store file metadata and cache data. • For write buffer size: If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. • For cached data: run

curl -s <HOST-IP>:4000/metrics  | grep greptime_mito_cache_bytes

to check the cached data set size. If you notice steadily increasing memory usage or an OOM incident, please follow this doc to collect a memory profile (*.hprof). This will help us identify the memory issue, though such issues are rare.

I just deployed the instance so this most definitely will be a me issue

On the machine, run :

curl -s 0:4000/metrics  | grep greptime_mito_cache_bytes

to get the resident cache size

# HELP greptime_mito_cache_bytes mito cache bytes # TYPE greptime_mito_cache_bytes gauge greptime_mito_cache_bytes{type="file"} 169923223 greptime_mito_cache_bytes{type="index"} 0 greptime_mito_cache_bytes{type="index_result"} 7798321 greptime_mito_cache_bytes{type="index_staging"} 0 greptime_mito_cache_bytes{type="manifest"} 20464244 greptime_mito_cache_bytes{type="page"} 182890074 greptime_mito_cache_bytes{type="prefilter_result"} 37033421 greptime_mito_cache_bytes{type="range_result"} 188104387 greptime_mito_cache_bytes{type="recycle_bin"} 0 greptime_mito_cache_bytes{type="sst_meta"} 128079707 greptime_mito_cache_bytes{type="vector"} 5813968

These numbers add up to 750 MB. Then run

curl -s localhost:4000/metrics | grep greptime_mito_write_buffer_bytes

greptime_mito_write_buffer_bytes 20467123

Given the ingestion rate you can just do that. GreptimeDB will adapt the memory requirement based on your machine's available RAM.

Yeah but I still don't know

increasing: GreptimeDB tends to use as much memory as possible, based on the RAM available. For 8GB RAM, it may use 1G as write buffer and ~3GB as cache. If you reduce the RAM to 4G, both write buffer and cache size will cut down to roughly half, without any config change.

So you recommend not setting any parameter regarding ram and resource usage in config and just let it autoscale?

Can you share the config with us by running this on the machine?

curl -s localhost:4000/config

the problem here I think is that greptime sees the total ram of the task which is 8Gb instead of the real 7Gb that it has assigned to that particular container, or 3/4 in the case of 4Gb

Maybe remove: •

global_write_buffer_size = "512MB"

•

page_cache_size = "512MB"

And let greptime figure out itself.

the problem here I think is that greptime sees the total ram of the task which is 8Gb instead of the real 7Gb that it has assigned to that particular container, or 3/4 in the case of 4Gb

Start with explicit memory limit by

docker run -m 4g greptime/greptimedb standalone start

and check the memory usage. For docker compose it would be:

services:
     greptimedb:
       image: greptime/greptimedb
       mem_limit: 4g
       command: standalone start

Im running an oci container (produced by nix) in ECS, not using compose

Try this:

[[region_engine]]
[region_engine.mito]

global_write_buffer_size = "256MB"
global_write_buffer_reject_size = "512MB"

page_cache_size = "256MB"
sst_meta_cache_size = "128MB"
selector_result_cache_size = "128MB"
range_result_cache_size = "128MB"
vector_cache_size = "64MB"
prefilter_result_cache_size = "64MB"
manifest_cache_size = "128MB"

[region_engine.mito.index]
result_cache_size = "64MB"
content_cache_size = "64MB"
metadata_cache_size = "32MB"

@aitor I just dealt with the same type of thing, while greptime is very efficient, if you don't partition the tables and get a very hot table, it will do exactly what your seeing. In my case it was with 100s of millions of rows and I was pushing a single table to drive a datanode to use 30Gb of ram, but it is the same in theory. While adjusting cache sizing will help, it won't fix it if it is a large table, I would personally start there before changing the cache if your cache levels have been stable up to this point.

how did you fix it?

I looked at the data in my tables and in my case the data was coming in from 20-25k unique VMs, so I built out alter table statements to partition on a column based on the IP. I split mine due to volume into 16 partitions. I did the same on the

greptime_physical_table

so it would auto add partitions to any generated tables from dynamic metrics ingestion. I have my datanodes set to 8c/32g and run 12 pods to 6 pods of frontend in my current PoC test. Yesterday this is what I was able to do with my volume of data after finding that it was hot tables and partitioning. First image is current state, in 24 hours we ingested at an average of 61.4k rows/s for a total of 417Gb of data across 8.35B rows. Second image shows my datanode resources over the 24 hours. You can see when we had much more data coming in due to the ram and cpu usage. But with the partitions, it handled it like a champ with a ton of room to grow. (avg under 7Gb of ram used across each data node)

So the solution in your case was to partition the tables by the IP of the sender... right? I don't think this can be related to my usecase given I currently have literally 3 stable ingestion rows and that's it?

I am suggesting if cache usage is not high, but table usage is, you would have to add partitions