Hi Greptimians! 👋 We are currently migrating a medium-large dataset (~100GB) from InfluxDB v1.8 to GreptimeDB v1.0.0-rc.1. We’re using a custom bulk migrator that cleans/transforms data in flight and pushes it to Greptime via the ILP (InfluxDB Line Protocol) endpoint. The Goal: We want to avoid the default

greptime_timestamp

prefix. We tried setting the prefix to empty in the config, but since that didn't work, we decided to pre-create the table schema to explicitly define the time index as

ts

. The Issue: When we pre-create the table using the SQL below, we experience partial data loss in specific columns during ingestion:

CREATE TABLE IF NOT EXISTS $table (
    ts TIMESTAMP(9) TIME INDEX,
    periph_id STRING,
    PRIMARY KEY (periph_id)
) ENGINE=mito;

• Symptoms: Most columns are created automatically via auto-schema evolution, but data only populates in some of them. For example, for accelerometer data (

g_x

,

g_y

,

g_z

), only

g_z

contains data; the others remain null or empty. • The Twist: If we do not pre-create the table (starting with an empty database), all columns (

g_x

,

g_y

,

g_z

) are created and populated correctly—but we are stuck with the default

greptime_timestamp

name. Our Questions: 1. Is there a known limitation when mixing pre-defined

TIME INDEX

columns with auto-evolving ILP ingestion? 2. Why would

g_z

succeed while

g_x

and

g_y

fail when the table is pre-created? 3. Is there a way to force the default timestamp name to

timestamp

(empty prefix) in the configuration without pre-creating every table? Thanks for the help!

Thanks for the detailed question. 1. There is no limitation when mixing a predefined time index with auto-evolving ingestion. 2. I don’t think this is data loss, but rather an issue related to

merge_mode

. For the InfluxDB data model,

merge_mode

must be set to

last_non_null

. This affects the

UPDATE

semantics. Please refer to the following documentation: • https://docs.greptime.com/user-guide/ingest-data/for-iot/influxdb-line-protocol/#data-model • https://docs.greptime.com/reference/sql/create/#create-a-table-with-merge-mode 3. You can set

default_column_prefix=""

in the configuration (

standalone

and

datanode

). This will change the default time index column from

greptime_timestamp

to`timestamp`.

Yep, the

with('merge_mode'='last_non_null');

did the trick! Still I don't get why the merge mode is different in a non-initialized table? Is this done to gain influxdb compatibility when a table is initialized purely by ILP, and follow "SQL semantics" when created by a

create table

SQL statement? Anyway cleared up now, many thanks! 🙂 About the setting of `default_column_prefix=""`I'm running greptime in a Docker container, passsing most of the config params via Env vars. When i try to pass an set, but empty env var (in my case

GREPTIMEDB_STANDALONE__DEFAULT_COLUMN_PREFIX=""

) greptime complaints about illegal chars in the prefix seemingly referring to the literal

"

interpreted as prefix. So setting the prefix works, but not setting it empty.. I could introduce a config file purely for this, but there must be a cleaner way IMO. Furthermore would it hurt to ad a config param for setting the default primary timestamp name directly (instead of just making the prefix configurable)

1. Tables auto-created via ILP will have

merge_mode

set to

last_non_null

by default. You can run

SHOW CREATE TABLE [table]

to see the table definition. 2. Good catch. Would you mind creating an issue in the GreptimeDB repo so we can look into it? 3. We discussed this before, but decided to use the prefix approach because there are other columns such as

greptime_value

.

> Would you mind creating an issue in the GreptimeDB repo Yep, will do. > ..., but decided to use the prefix approach... For the timestamp field specifically I'd vouch for a configurable default. We intend to use GreptimeDB primarily (currently only) as a TSDB. In that use case the timestamp field is kinda special (as it was in Influx). We prefer to have the primary timestamp field to be called `ts`as that is (very) short and not a reserved keyword. Currently there's no way to do this without pre-defining the table schema which can be less ergonomic (or possibly hard or impossible in some scenarios).

Make senses. Please create a feature request issue too.

yes, will do so.. ! ✅