How can I add partitions to the auto generated phy...
# general
p
How can I add partitions to the auto generated physical table for metrics and traces coming in from OTEL (and the logical tables if needed)? I am shipping in my PoC 5% or our total data so far and I have 1 datanode using 29Gi out of 32Gi on the pod limit. I ran into the same issue to the point that I crashed the datanode going OOM due to log volume and broke it down across 16 partitions. I would like to do the same to protect the datanodes but can't figure out how. When I did an alter table and added partitions on the
greptime_physical_table
all querying of data from the logical tables returned errors about invalid regions.
w
After adding the partition, did you see any new regions when you ran
SHOW CREATE TABLE {your_table}
?
p
@Weny Xu So it took some work and some digging through docs to find a single line that states if you want to use the generated metrics engine with partitioning that you have to pre-create the table with a set of low cardinality fields and then partition on those. But the docs are not very clear that you can't use a field that is of INT type as a primary key on a metrics table, so had to move my shard key to a STRING and pad the values from the OTel collector.
I now have all of my tables working partitioned. a running job before would put enough data that I would see 1 pod jump to 18-20Gi of ram usage, now all of the datanode pods are sitting while getting load at sub-2Gi.
🙌 1
👏 1
image.png
higher lines are my 12 datanodes, stacked lines on the bottom are my 6 frontend pods and 2 meta pods
Upwards of 160k rows/s over a 3 hour period for just under 2B rows and around 30gb of data all with no issues. 16 datanodes all sitting less then 4Gi of ram usage max.
I did hit some issues with frontend pods restarting and isito keeping up. Along with a meta pod restart due to an issue with accessing a region and a failure to create an auto table.
Next steps of planning for load will be frontend groups.