https://linen.dev logo
Join Slack
Powered by
Hi, I am currently working on a PoC using greptime...
# general
p
Hi, I am currently working on a PoC using greptime to replace our current telemetry storage for our CI systems. I have the OSS version deployed as a cluster on K8s and I am trying to find better guidance on scaling and the ability to support our use case. We are looking to start out small and grow but I am struggling to get the best patterns of usage. 1 full CI run of our platform is about 1.5 billion log lines totaling about 50Gb of raw text. There is another 10Gb of traces and 3Gb of metrics all collected from 960+ VMs, all via the Otel Collector. I started with the base deployment using 4c and 8gb of ram across 3 pods of frontend, datanode, and meta server and a single run caused both the most used datanode to OOMKill as well as multiple of the frontend pods. Currently with it at 8c/16Gi they stay up but ram usage comes up to 13-14Gi on a few datanodes after 2 full Ci runs. I am currently for the PoC trying to figure out scaling and the optimum patterns of table creation and usage to support stages of our Ci Platform where we would have to support 25 and then 50 full runs per day, and where we are at times running multiple of these full runs concurrently. Any tips or suggestions would be appreciated.
As a note. We are interested in the enterprise version but need to prove out the worth of the system over storing logs raw in S3 as we do now and not having traces and metrics. The key is proving that we can support the data partially first before growing beyond. Worth noting our monthly storage of these CI logs are about 200Tb.
e
Hi, could you collect a memory flamegraph for the datanode whose ram came to 13GB? docs.greptime.com/user-guide/deployments-administration/performance-tuning/performance-tuning-tips#… I have some assumptions but a memory profile would be helpful.
p
@evenyag will it work with the cluster version. It only has how to add the MALLOC_CONF for standalone
e
Official Docker images enable heap profiling by default.
Yes, it should work by default.
p
I will try to run it this evening.
@evenyag are there any guides better then the docs to resource requirements for different levels of scale. Or things like number of frontends to run or data nodes to run, as well as best practices for partitioning to maximize load and performance.
e
It depends on your workload. We generally deploy frontend and datanode in a 1:2 ratio. Generally, the frontend’s specifications are smaller than those of the datanode. We use 4c8g for frontend and 8c16g for datanode for a common setup. For performance tuning or suggestions, you can also try the skills I added recently. • 
greptimedb-performance-diagnosis
• 
greptimedb-performance-tuning
• 
greptimedb-table-design
github.com/GreptimeTeam/docs/…/skills
p
For my initial POC I most likely will be storing 2-4Tb a day.
The bulk will be in most likely 50-100 billion rows of logs per day.
e
Do you know how many bytes (or rows) of log data are ingested per second? Based on the ingestion rate, we can estimate the number of CPU cores for datanodes. And we have to partition the log table. For example, we can consider using the label/pod ip of the collected VM to partition the table.
The ingestion speed of a datanode instance with 8c16g is about 4GB/s. But it vary with the size of the log and the number of columns.
p
Most log lines are small. They are direct lines from shell redirection from running apps, shell scripts, maven, Javac, nginx, etc
Rough math with 1 full job running which takes about an hour there would be around 400-425k rows per second
I added via the otel collector a transform doing an FNV of the IP addr and a mod 256 on it giving us 256 possible values. I was planning on building partitions using this and was going to most likely start with blocks of 16 numbers per partition.
e
I added via the otel collector a transform doing an FNV of the IP addr and a mod 256 on it giving us 256 possible values. I was planning on building partitions using this and was going to most likely start with blocks of 16 numbers per partition.
Looks good. I think we can collect a memory flamegraph for the memory issue. If you have enabled our self-monitor compotent, you can also share the memory usage panel with us. docs.greptime.com/user-guide/deployments-administration/monitoring/cluster-monitoring-deployment We also have a 
self-monitoring-export
skill to help export metrics/logs. You may try that.
p
@evenyag I ran again some of our tests with the 1 full job, and I had added the partitions as mentioned above for the logs. Now we see this in our dashboard.
Working on perms from our org to share the direct data and flamegraph out. Have to make sure no internal data is in it to get approval
wrong graph.
image.png
Here is the one that I meant.
It did not jump up high in ram usage this time at all
Question, Do number of partitions need to match number of datanodes? or can we have more partitions then datanodes?
@evenyag also, you suggested possible sizing for frontend and datanodes, but what about the meta?
e
Question, Do number of partitions need to match number of datanodes? or can we have more partitions then datanodes?
You can have more partitions then datanodes. We use 2c4g or 2c2g pods for meta. Usually 2 meta pods are sufficient.
The memory usage after partition looks fine.
p
We just started shipping 1-2% of our ci load to it. I should have more data by morning.
trying to pull what I can now. Here was what we saw when 2 full jobs ran concurrently.
image.png
image.png
This has 12 datanodes each 8c/16gi
sorry, 8c/32gi
e
You can also collect a memory framegraph for datanode-8.
p
It was in a restart loop due to 
max_concurrent_scan_files
passing the default 384 files.
Is there a command I can run to get a flamegraph of just the datanode-8?
Also, how can I identify what table or tables are causing that one node to ballon, I think it is greptime_physical_table as I am using auto create for tables for metrics.
I believe the partitioning that I did for logs kept those low
e
max_concurrent_scan_file
shoudn’t cause restart. Do you mean datanode-8 is in a restart loop now?
p
It was
Copy code
too many files to read concurrently: 763, max allowed: 384
I upped it to 1024
got up to 963 files
e
This error is unrelated. It only rejects queries. Did you observe restarts in k8s? You can check the reason via describing a pod. Maybe it was got OOM killed. There is a system table to get the location of a region and datanode. docs.greptime.com/reference/sql/information-schema/region-peers You can get a flamegraph after a pod restarts. If possible, can you share the pod logs to us?
p
Copy code
Incident B — 04:10:00–04:24:59 UTC 08-28, unresolved. Queries against greptime-cluster-datanode-8 fail with Too many files to read concurrently: N, max allowed: 384 from mito2 scan paths. The datanode exited code 1 at 04:23:42 and restarted at 04:24:05. That stopped the log flood but not the cause — reported file count was 512 during the flood and 698–960 after, i.e. the backlog grew across the restart. Frontends were still failing queries at 05:39. Compaction on that node isn't keeping pace with ingest.
I had tried earlier today to add the same partitioning that I did to the greptime_physical_table, that was a bad idea. everything died related to metrics access, had to drop the tables and let them re-create
Just trying to identify the issue now
@evenyag how can I get a flamegraph for just datanode-8, or do I have to do the full cluster?
e
You only need to curl the endpoint of datanode-8.
For example, port-forward the http port of datanode-8.
p
greptime-mem.svg
greptime-mem.svg
e
If you enabled self-monitor, you can also use the greptimedb logs dashboard in the grafana to view logs of datanode-8. It’d be helpful to get the logs around the time that the datanode exited code 1.
p
I have the standalone self monitoring deployed
I used the skill to investigate
e
The ram is about 4GB. Can you check the error logs of the datanode? You can also specify the pod name here. The logs during that time.
Copy code
04:23:42 and restarted at 04:24:05
p
@evenyag unfortunately, logs are not an easy thing as they are in binary format and I need to review that no internal data is included before I could share them
I am using perses over grafana, but if you can link me to the dashboard json, I can have it convert and create them
e
It’s greptimedb’s log.
Did you enable the grafana in the self-monitor? It’ll include the dashboards automatically.
p
no, I do not believe that I enabled grafana
e
Could you enable that? Then you can view the logs. Or you can get the dashboard json here. github.com/GreptimeTeam/greptimedb/blob/…/dashboard.json If the datanode was crash, we can also grep “panic” via 
kubectl logs <pod-name> --previous
p
converting it and deploying it to perses now
👍 1
While I am at it, are there any other useful dashboards from the grafana deploy that you would suggest that I import as well?
e
All our dashboards are available at github.com/GreptimeTeam/greptimedb/…/grafana We usually use metrics and logs.
By the way, do you know the approximate ingestion rate of the metrics?
p
We were seeing north of 150k per second during the window when things went sideways
This was from 2 jobs across 2000 hosts.
trying to get the logs dashboard working, seems it is having an sql error
Copy code
GreptimeDB error: 400 - {"code":3000,"error":"Failed to plan SQL: No field named info. Valid fields are _gt_logs.pod_ip, _gt_logs.namespace, _gt_logs.cluster, _gt_logs.file, _gt_logs.module_path, _gt_logs.level, _gt_logs.target, _gt_logs.role, _gt_logs.pod, _gt_logs.message, _gt_logs.err, _gt_logs.timestamp.","execution_time_ms":1}
e
Are there some issues with the converted dashboard? We don’t query 
info
in our sql. Could you check the json of the perses dashboard?
p
I don't see it in the query run which is what is weird about it
Copy code
SELECT `timestamp`,
  CONCAT('[', `level`, ']', ' ', '<', `target`, '>', ' ', `message`) AS `line`,
  `role`,
  `pod`,
  `pod_ip`,
  `namespace`,
  `cluster`,
  `err`,
  `file`,
  `module_path`
FROM `_gt_logs`
WHERE
  ('$level' = '$__all' OR `level` IN ($level))
  AND ('$role' = '$__all' OR `role` IN ($role))
  AND ('$pod' = '' OR `pod` = '$pod')
  AND ('$target' = '' OR `target` = '$target')
  AND ('$search' = '' OR matches_term(`message`, '$search'))
  AND ('$exclude' = '' OR NOT matches_term(`message`, '$exclude'))
  AND `timestamp` BETWEEN to_timestamp_millis($__from) AND to_timestamp_millis($__to)
ORDER BY `timestamp` DESC
LIMIT $limit
e
Is it possible to get the request via the Network panel in the browser DevTools? It may contains the SQL.
p
so the issue is in WHERE clause
$level
is being translated and it thinks it is doing a where 
info = ...
I think the grafana to perses conversion fail, let me check it
yeah, SQL conversion issue
ok, have logs now working
🙌 1
@evenyag so the volume of metrics was a lot higher for those 2 full jobs then I was expecting.
image.png
During the run it peaked north of 600k rows/s
This lines up with that datanode-8 spiking to 22.1Gi of ram usage.
As well as the files open error and the restarts
e
Could you provide the logs before it restarted?
p
I can try to look for things if you can suggest what to look for, but until I look through the logs for things that need to be sanitized, I cannot blanket share them, it is a corp policy.
e
Maybe several hundreds lines of logs before the db restartsed. It prints 
GreptimeDB version
when it restarts so you can get the logs before that. You can also grep the error logs of the node and share them.
Grafana has a download feature in Inspect. Maybe Perses also provides a similar feature.
p
Let me see what I can get
2 Views
Open in Slack
PreviousNext
Powered by