Hi team, I am running some Routine Loads test for ...
# questions-and-troubleshooting
a
Hi team, I am running some Routine Loads test for Kafka with Shared-data architecture. I am trying to measure the latency between the ingestion time in Kafka and Starrocks, so I added a dummy column (
ingestion_time
) which records the timestamp when data rows get ingested in the table. My database schema looks like:
Copy code
CREATE TABLE table (
    `__time` datetime NOT NULL COMMENT "event timestamp",
    `data` string NOT NULL COMMENT "data value",
    `user_id` VARCHAR(64) NOT NULL COMMENT "User ID",
    `ingestion_time` DATETIME DEFAULT CURRENT_TIMESTAMP(6) COMMENT "ingestion timestamp"
)
However, after some tests, ingestion_time seems to be the creation time of routine load tasks, instead of the time when the data records get ingested
Copy code
SELECT * FROM table ORDER BY __time DESC LIMIT 5;
+-------------------------------------------+-----------+---------
| user_id | __time                      | ingestion_time        
+---------+-----------------------------+---------------+--------
| 1234    | 2026-02-18 18:03:38.174127  | 2026-02-18 17:55:25.963000 |
| 1234    | 2026-02-18 18:03:38.174127  | 2026-02-18 17:55:25.963000 |
| 1234    | 2026-02-18 18:03:38.174127  | 2026-02-18 17:55:25.963000 |
| 2345    | 2026-02-18 18:03:38.174127  | 2026-02-18 17:55:25.963000 |
| 2345    | 2026-02-18 18:03:38.089570  | 2026-02-18 17:55:25.963000 |
+-------------------------------------------+-----------+-------------
As you can see in the table above, the ingestion_time is < than the event timestamp
__time
, which does not make sense. 1. What does the
ingestion_time DATETIME DEFAULT CURRENT_TIMESTAMP(6)
column really mean in my case? 2. And How can we log the ingestion timestamp of rows in the table, my main purpose is to measure the latency of the data ingestion in starrocks for each record Thanks
k
can open an issue for this, could be something wrong with the default CURRENT_TIMESTAMP taking as constant folding.
a
Thanks @Kevin Cai for the response. Is there anyway or metrics we can measure the latency between 1. Event Timestamp and the ingestion timestamp, which is when the data record is inserted to the table 2. Event Timestamp and the queryable timestamp, which is when the data is ready for query