I imported the data using a variation of the instructions at docs.greptime.com/user-guide/migrate-to-greptimedb/migrate-from-influxdb#… The instructions first convert the entire bucket into a single text file, then split that file on disk, and then load the files one-by-one over HTTP. To avoid using hundreds of GB of temporary disk space, I simplified that into a single command:

sudo -u influxdb2 influxd inspect export-lp --bucket-id 53a349a12da3bc3e --engine-path /var/lib/influxdb2/engine/ --output-path - | split -l 100000 -d -a 10 --filter='curl --fail-with-body -i --retry 3 -H @auth.header -X POST "<https://greptimedb.isz.wtf/v1/influxdb/write?db=profinet>" --data-binary @- && sleep .1'

This avoids any temporary files and directly invokes

curl

for each chunk. (I also reduced the sleep from 1s to 0.1s to slightly improve the speed.)

It took about 3 days to import the 125GB of data. This seemed slow to me and it looked like a ton of CPU were spent compacting and re-compacting the Parquet files as it was imported.

I wonder if there are knobs that could be adjusted to improve the import rate (e.g. reducing the frequency of compactions, increasing the size of the WAL before compaction is triggered, making multiple HTTP requests in parallel, etc.)

Before I imported the whole table, I first imported a chunk of a couple weeks of data, and then (re)pointed the live data stream at GreptimeDB using the existing InfluxDB v2 client library (great job with compatibility!). I loved how I could then import the whole bucket and the duplicate points were cleanly handled with the default

merge_mode

.

While the import was running, I `watch`ed the checkpoint files to observe the (re)compaction process:

watch -cd "cat \$(ls -t /var/lib/greptimedb/data/data/greptime/profinet/1052/1052_0000000000/manifest/*.checkpoint | head -1)  | jq -C -c '[.checkpoint.files[] | [((.time_range[].value/1000000000) | todateiso8601), .level, .file_id, .num_rows]] | sort | .[]'"

I also converted all the Grafana dashboards that used this data to use the GreptimeDB datasource. I ran into some problems here - the Grafana datasource has several sharp edges that make it hard to get the data you want in the correct schema for Grafana's visualizations. I can file some issues on the datasource repo.

In the end I was able to get all the graphs updated to use GreptimeDB queries instead of Flux queries.

Query performance does seem to be a bit slower than InfluxDB 2.0. Equivalent queries on the same data seem to be anywhere from 25% to ~5x slower.

Even if that is as good as I can get with GreptimeDB, it's well worth it for the massive disk + RAM savings.

I filed #60-#63 in the Grafana datasource repo for the bugs I've encountered so far.