https://linen.dev logo
Join Slack
Powered by
Hey everyone:wave: I'm running into an issue on o...
# general
c
Hey everyone👋 I'm running into an issue on our GreptimeDB frontend pod during OTLP ingestion.
Copy code
*Error:* `Procedure poison key already exists with a different value`
*Key:* `/__procedure_poison/table/1026` (`opentelemetry_traces`)
*Procedure UUID:* `3b525f48-c72e-4e7a-b403-8daa8570cf4c`
What's happening: schema-on-write ALTER TABLE (adding a new span attribute column) is failing because a stale or conflicting DDL procedure lock exists. This is causing OTLP ingestion to fail entirely for affected requests. Can anyone please help me out with the RCA? Thank You! 🙏
d
@Weny Xu Could you provide guidance on how to recover? I think we should add it to the FAQ.
w
Hi @Chitranshu Bisht, this is typically caused by an alter procedure eventually failing after 12 retry attempts. In this case, we need to manually remove the poison key:
Copy code
/__procedure_poison/table/1026

greptime cli meta del key \
    --store-addrs=$ENDPOINT \
    --backend=postgres-store \
    /__procedure_poison/table/1026
if you are using another type of backend store, please follow the docs here: https://docs.greptime.com/reference/command-lines/utilities/metadata-interaction/#delete-key-value-pair After that, you need to run the table reconciliation procedure to ensure the region metadata is consistent:
Copy code
USE {schema};

ADMIN reconcile_table('opentelemetry_traces');
Please follow the docs here for table reconciliation: https://docs.greptime.com/user-guide/deployments-administration/maintenance/table-reconciliation
c
Thanks, this got resolved but now after the restart I'm seeing this:
Copy code
2026-04-28T12:02:25.009644Z WARN servers::error: Failed to handle HTTP request err=0: Execute gRPC query error, at src/frontend/src/instance/otlp.rs:112:18
1: Table operation error, at src/frontend/src/instance/grpc.rs:469:14
2: Failed to insert data, at /greptimedb/src/operator/src/insert.rs:422:26
3: External error, at src/client/src/region.rs:65:31
4: Failed to request RegionServer 10.233.121.232:4001, code: Some requested entity was not found, at src/client/src/region.rs:272:31
5: Column kv_cache_dtype_skip_layers not found in logical region 4419521347584(1029, 0), at src/client/src/error.rs:173:1
d
Can you confirm that your Metasrv version and datanode version are the same? Looks like it’s a protocol compatibility issue.
c
Yes, they are the same.
d
@Weny Xu Please take a look
👀 1
c
I'm using this version: 
v1.0.0-beta.2
w
Hi @Chitranshu Bisht , Did you know the table name for table ID 
1029
? You can run the following SQL:
Copy code
SELECT *
FROM information_schema.tables
WHERE table_id = 1029;
Once you find the table name, please run:
Copy code
show create table {table_name}
c
Done, now shall I restart the pods?
w
No—could you share the SQL output?
c
show create table vllm_cache_config_info
Copy code
Table

vllm_cache_config_info
CREATE TABLE IF NOT EXISTS "vllm_cache_config_info" (
  "BUILD_VERSION" STRING NULL,
  "CLUSTER_ID" STRING NULL,
  "ENVIRONMENT" STRING NULL,
  "PROJECT_NAME" STRING NULL,
  "block_size" STRING NULL,
  "cache_dtype" STRING NULL,
  "calculate_kv_scales" STRING NULL,
  "cpu_kvcache_space_bytes" STRING NULL,
  "cpu_offload_gb" STRING NULL,
  "enable_prefix_caching" STRING NULL,
  "engine" STRING NULL,
  "gpu_memory_utilization" STRING NULL,
  "greptime_timestamp" TIMESTAMP(3) NOT NULL,
  "greptime_value" DOUBLE NULL,
  "instance" STRING NULL,
  "is_attention_free" STRING NULL,
  "job" STRING NULL,
  "k8s_container_name" STRING NULL,
  "k8s_deployment_name" STRING NULL,
  "k8s_namespace_name" STRING NULL,
  "k8s_pod_name" STRING NULL,
  "k8s_replicaset_name" STRING NULL,
  "key_block_size_resolved" STRING NULL,
  "kserve_inferenceservice" STRING NULL,
  "kserve_pod" STRING NULL,
  "kubernetes_namespace" STRING NULL,
  "kv_cache_dtype_skip_layers" STRING NULL,
  "kv_cache_memory_bytes" STRING NULL,
  "kv_offloading_backend" STRING NULL,
  "kv_offloading_size" STRING NULL,
  "kv_sharing_fast_prefill" STRING NULL,
  "mamba_block_size" STRING NULL,
  "mamba_cache_dtype" STRING NULL,
  "mamba_cache_mode" STRING NULL,
  "mamba_page_size_padded" STRING NULL,
  "mamba_ssm_cache_dtype" STRING NULL,
  "num_cpu_blocks" STRING NULL,
  "num_gpu_blocks" STRING NULL,
  "num_gpu_blocks_override" STRING NULL,
  "prefix_caching_hash_algo" STRING NULL,
  "service_instance_id" STRING NULL,
  "service_name" STRING NULL,
  "sliding_window" STRING NULL,
  "swap_space" STRING NULL,
  "swap_space_bytes" STRING NULL,
  "user_specified_block_size" STRING NULL,
  TIME INDEX ("greptime_timestamp"),
  PRIMARY KEY ("BUILD_VERSION", "CLUSTER_ID", "ENVIRONMENT", "PROJECT_NAME", "block_size", "cache_dtype", "calculate_kv_scales", "cpu_kvcache_space_bytes", "cpu_offload_gb", "enable_prefix_caching", "engine", "gpu_memory_utilization", "instance", "is_attention_free", "job", "k8s_container_name", "k8s_deployment_name", "k8s_namespace_name", "k8s_pod_name", "k8s_replicaset_name", "key_block_size_resolved", "kserve_inferenceservice", "kserve_pod", "kubernetes_namespace", "kv_cache_dtype_skip_layers", "kv_cache_memory_bytes", "kv_offloading_backend", "kv_offloading_size", "kv_sharing_fast_prefill", "mamba_block_size", "mamba_cache_dtype", "mamba_cache_mode", "mamba_page_size_padded", "mamba_ssm_cache_dtype", "num_cpu_blocks", "num_gpu_blocks", "num_gpu_blocks_override", "prefix_caching_hash_algo", "service_instance_id", "service_name", "sliding_window", "swap_space", "swap_space_bytes", "user_specified_block_size")
)

ENGINE=metric
WITH(
  on_physical_table = 'greptime_physical_table',
  otlp_metric_compat = 'prom',
  ttl = '2days'
)
w
@Chitranshu Bisht OK, I see. You can run:
Copy code
ADMIN reconcile_database('{your_db_name}');
Then make sure the reconcile database procedure has completed by checking:
Copy code
ADMIN procedure_state('procedure_id');
The 
reconcile_database
procedure will repair all metadata inconsistency issues for all logical and physical tables within the database.
c
ADMIN procedure_state('c9a4fd8c-4f51-43c0-a94b-4172946858df');
Copy code
ADMIN procedure_state('c9a4fd8c-4f51-43c0-a94b-4172946858df')
{"status":"Failed","error":"Schema nod found, schema: {metrics_schema}"}
w
Run this
Copy code
ADMIN reconcile_database('metrics_schema');
c
ADMIN reconcile_database('metrics_schema');
Copy code
ADMIN reconcile_database('metrics_schema')
f8f1cdc9-a8ba-42e1-b389-ae216da5a8d4
w
then check the procedure state:
Copy code
ADMIN procedure_state('f8f1cdc9-a8ba-42e1-b389-ae216da5a8d4')
c
ADMIN procedure_state('f8f1cdc9-a8ba-42e1-b389-ae216da5a8d4')
Copy code
ADMIN procedure_state('f8f1cdc9-a8ba-42e1-b389-ae216da5a8d4')
{"status":"Done"}
I think now its done.
w
Yes, then you can check the frontend logs. Are there any new errors?
c
Should I restart frontend pod before checking?
w
No
c
Frontend latest logs:
Copy code
2026-04-29T07:52:38.835257Z WARN servers::error: Failed to handle HTTP request err=0: Execute gRPC query error, at src/frontend/src/instance/otlp.rs:112:18
1: Table operation error, at src/frontend/src/instance/grpc.rs:469:14
2: Failed to insert data, at /greptimedb/src/operator/src/insert.rs:422:26
3: External error, at src/client/src/region.rs:65:31
4: Failed to request RegionServer 10.233.121.252:4001, code: Some requested entity was not found, at src/client/src/region.rs:272:31
5: Column kv_cache_dtype_skip_layers not found in logical region 4419521347584(1029, 0), at src/client/src/error.rs:173:1
👀 1
w
Can you run this SQL? Once 
procedure_state
returns 
Done
, please share the logs from metasrv
Copy code
use metrics_schema;

ADMIN reconcile_table('vllm_cache_config_info');
c
Meta latest logs after running the above SQL:
Copy code
2026-04-29T07:44:43.953460Z INFO meta_srv::election::rds::postgres: PgElection uses default search_path (no schema provided)
2026-04-29T07:44:49.045414Z INFO tokio_postgres::connection: NOTICE: relation "greptime_metakv" already exists, skipping
2026-04-29T07:44:49.045854Z INFO meta_srv::bootstrap: Using selector from options, selector type: RoundRobin
2026-04-29T07:44:49.046248Z INFO common_procedure::store: The procedure state store path is: procedure/
2026-04-29T07:44:49.046268Z INFO common_event_recorder::recorder: Start the background processor in event recorder to handle the received events.
2026-04-29T07:44:49.046699Z INFO common_stat::resource: Starting to collect CPU usage periodically for every 5 seconds
2026-04-29T07:44:49.062440Z INFO cmd: Starting app: greptime-metasrv
2026-04-29T07:44:49.062480Z INFO meta_srv::bootstrap: starting http server at 0.0.0.0:4000
2026-04-29T07:44:49.062644Z INFO servers::http: HTTP server timeout is disabled
2026-04-29T07:44:49.063106Z INFO servers::http: HTTP server is bound to 0.0.0.0:4000
2026-04-29T07:44:49.083367Z INFO common_greptimedb_telemetry: Attention: GreptimeDB now collects anonymous usage data to help improve its roadmap and prioritize features.
2026-04-29T07:44:49.083409Z INFO common_greptimedb_telemetry: To learn more about this anonymous program and how to deactivate it if you don't want to participate, please visit the following URL:
2026-04-29T07:44:49.083415Z INFO common_greptimedb_telemetry: <https://docs.greptime.com/reference/telemetry>
2026-04-29T07:44:49.083677Z INFO meta_srv::metasrv: Metasrv started
2026-04-29T07:44:49.083883Z INFO meta_srv::bootstrap: gRPC server is bound to: 0.0.0.0:3002
w
Did you have two metasrv instances? BTW, for a quick recovery of this issue, for a logical table we can also just drop the missing column without data loss. (The real data is stored in the physical table; the logical table only maintains column metadata)
Copy code
ALTER TABLE vllm_cache_config_info
DROP COLUMN kv_cache_dtype_skip_layers;
c
No, I have only one metasrv instance running. Sure let me drop this missing column.
w
No, I have only one metasrv instance running.
Ok, it's a little weird. You should see some logs related to the reconcile procedure...
c
Yeah.. and this is what I'm getting now after running this:
Copy code
ALTER TABLE vllm_cache_config_info
DROP COLUMN kv_cache_dtype_skip_layers;
👀 1
w
Ok, let's try the reconcile table procedure again. Please share the SQL output as well.
Copy code
ADMIN reconcile_table('metrics_schema.vllm_cache_config_info');
c
Copy code
ADMIN reconcile_table('metrics_schema.vllm_cache_config_info')
0f8bb48a-9d81-4c4d-b4e6-cce3159c1796
 
ADMIN procedure_state('0f8bb48a-9d81-4c4d-b4e6-cce3159c1796')
{"status":"Done"}
w
Did you see any logs from metasrv?
c
Nope
Same logs as the last one.
w
It's weird. Are there any logs from the frontend?
You should see some logs like: 
Reconciling table: xxx with resolve_strategy
BTW, did you configure our standalone monitor? Your metasrv logs look like pod previous logs instead of the latest logs.
c
Frontend logs:
Copy code
2026-04-29T08:35:08.830821Z WARN servers::error: Failed to handle HTTP request err=0: Execute gRPC query error, at src/frontend/src/instance/otlp.rs:112:18
1: Table operation error, at src/frontend/src/instance/grpc.rs:469:14
2: Failed to insert data, at /greptimedb/src/operator/src/insert.rs:422:26
3: External error, at src/client/src/region.rs:65:31
4: Failed to request RegionServer 10.233.121.252:4001, code: Some requested entity was not found, at src/client/src/region.rs:272:31
5: Column kv_cache_dtype_skip_layers not found in logical region 4419521347584(1029, 0), at src/client/src/error.rs:173:1
Not sure about the standalone monitor, I'm running this on the cluster mode. And this was running fine from last one month it has started to occur recently.
w
BTW, how are you managing these logs? Can you share the command you use to view the logs?
c
Copy code
(base) PS C:\Users\cbisht> kubectl logs greptimedb-cluster-meta-6b4f7c4ddb-5sc2z -n greptime
2026-04-29T07:44:43.883718Z  INFO cmd: GreptimeDB version:
branch:
commit: 0177f244e9b266b3ca2a43ae6d9ead1a6a760e60
clean: true
version: 1.0.0-beta.2
2026-04-29T07:44:43.883774Z  INFO cmd: command line arguments
2026-04-29T07:44:43.883783Z  INFO cmd: argument: greptime
2026-04-29T07:44:43.883794Z  INFO cmd: argument: metasrv
2026-04-29T07:44:43.883798Z  INFO cmd: argument: start
2026-04-29T07:44:43.883801Z  INFO cmd: argument: --rpc-bind-addr
2026-04-29T07:44:43.883805Z  INFO cmd: argument: 0.0.0.0:3002
2026-04-29T07:44:43.883809Z  INFO cmd: argument: --http-addr
2026-04-29T07:44:43.883812Z  INFO cmd: argument: 0.0.0.0:4000
2026-04-29T07:44:43.883818Z  INFO cmd: argument: --rpc-server-addr
2026-04-29T07:44:43.883822Z  INFO cmd: argument: 10.233.121.232:3002
2026-04-29T07:44:43.883825Z  INFO cmd: argument: --config-file
2026-04-29T07:44:43.883828Z  INFO cmd: argument: /etc/greptimedb/config.toml
2026-04-29T07:44:43.887426Z  INFO cmd: Heap profile is active
2026-04-29T07:44:43.887526Z  INFO cmd: GreptimeDB start with cpu limit in millicores: 1000
2026-04-29T07:44:43.887576Z  INFO cmd: GreptimeDB start with memory limit in bytes: 4294967296
2026-04-29T07:44:43.887591Z  INFO cmd::metasrv: Metasrv start command: StartCommand {
    rpc_bind_addr: Some(
        "0.0.0.0:3002",
    ),
    rpc_server_addr: Some(
        "10.233.121.232:3002",
    ),
    store_addrs: None,
    config_file: Some(
        "/etc/greptimedb/config.toml",
    ),
    selector: None,
    use_memory_store: None,
    enable_region_failover: None,
    http_addr: Some(
        "0.0.0.0:4000",
    ),
    http_timeout: None,
    env_prefix: "GREPTIMEDB_METASRV",
    data_home: None,
    store_key_prefix: "",
    max_txn_ops: None,
    backend: None,
}
2026-04-29T07:44:43.887645Z  INFO cmd::metasrv: Metasrv options: MetasrvOptions {
    store_addrs: [
        "postgres-cluster-rw.shared-services.svc.cluster.local:5432/greptime",
    ],
    backend_tls: None,
    selector: RoundRobin,
    use_memory_store: false,
    enable_region_failover: false,
    allow_region_failover_on_local_wal: false,
    grpc: GrpcOptions {
        bind_addr: "0.0.0.0:3002",
        server_addr: "10.233.121.232:3002",
        max_recv_message_size: 512.0MiB,
        max_send_message_size: 512.0MiB,
        max_total_message_memory: 0B,
        flight_compression: ArrowIpc,
        runtime_size: 8,
        tls: TlsOption {
            mode: Disable,
            cert_path: "",
            key_path: "",
            ca_cert_path: "",
            watch: false,
        },
        max_connection_age: None,
    },
    http: HttpOptions {
        addr: "0.0.0.0:4000",
        timeout: 0ns,
        disable_dashboard: true,
        body_limit: 64.0MiB,
        max_total_body_memory: 0B,
        prom_validation_mode: Strict,
        cors_allowed_origins: [],
        enable_cors: true,
    },
    logging: LoggingOptions {
        dir: "/data/greptimedb/logs",
        level: Some(
            "info",
        ),
        log_format: Text,
        max_log_files: 720,
        append_stdout: true,
        enable_otlp_tracing: true,
        otlp_endpoint: Some(
            "<http://otelcollector-client-daemonset.observability-client.svc.cluster.local:4318/v1/traces>",
        ),
        tracing_sample_ratio: Some(
            TracingSampleOptions {
                default_ratio: 1.0,
                rules: [],
            },
        ),
        otlp_export_protocol: None,
        otlp_headers: {},
    },
    procedure: ProcedureConfig {
        max_retry_times: 12,
        retry_delay: 500ms,
        max_metadata_value_size: Some(
            1.5MiB,
        ),
        max_running_procedures: 128,
    },
    failure_detector: PhiAccrualFailureDetectorOptions {
        threshold: 8.0,
        min_std_deviation: 100ms,
        acceptable_heartbeat_pause: 10s,
    },
    datanode: DatanodeClientOptions {
        client: ClientOptions {
            timeout: 10s,
            connect_timeout: 1s,
            tcp_nodelay: true,
        },
    },
    enable_telemetry: true,
    data_home: "./greptimedb_data",
    wal: RaftEngine,
    store_key_prefix: "",
    max_txn_ops: 128,
    flush_stats_factor: 3,
    tracing: TracingOptions,
    backend: PostgresStore,
    event_recorder: EventRecorderOptions {
        ttl: 7776000s,
    },
    stats_persistence: StatsPersistenceOptions {
        ttl: 0ns,
        interval: 600s,
    },
    meta_table_name: "greptime_metakv",
    meta_election_lock_id: 1,
    meta_schema_name: None,
    node_max_idle_time: 86400s,
}
2026-04-29T07:44:43.953460Z  INFO meta_srv::election::rds::postgres: PgElection uses default search_path (no schema provided)
2026-04-29T07:44:49.045414Z  INFO tokio_postgres::connection: NOTICE: relation "greptime_metakv" already exists, skipping
2026-04-29T07:44:49.045854Z  INFO meta_srv::bootstrap: Using selector from options, selector type: RoundRobin
2026-04-29T07:44:49.046248Z  INFO common_procedure::store: The procedure state store path is: procedure/
2026-04-29T07:44:49.046268Z  INFO common_event_recorder::recorder: Start the background processor in event recorder to handle the received events.
2026-04-29T07:44:49.046699Z  INFO common_stat::resource: Starting to collect CPU usage periodically for every 5 seconds
2026-04-29T07:44:49.062440Z  INFO cmd: Starting app: greptime-metasrv
2026-04-29T07:44:49.062480Z  INFO meta_srv::bootstrap: starting http server at 0.0.0.0:4000
2026-04-29T07:44:49.062644Z  INFO servers::http: HTTP server timeout is disabled
2026-04-29T07:44:49.063106Z  INFO servers::http: HTTP server is bound to 0.0.0.0:4000
2026-04-29T07:44:49.083367Z  INFO common_greptimedb_telemetry: Attention: GreptimeDB now collects anonymous usage data to help improve its roadmap a
nd prioritize features.
2026-04-29T07:44:49.083409Z  INFO common_greptimedb_telemetry: To learn more about this anonymous program and how to deactivate it if you don't want
 to participate, please visit the following URL:
2026-04-29T07:44:49.083415Z  INFO common_greptimedb_telemetry: <https://docs.greptime.com/reference/telemetry>
2026-04-29T07:44:49.083677Z  INFO meta_srv::metasrv: Metasrv started
w
Try this
Copy code
kubectl logs greptimedb-cluster-meta-6b4f7c4ddb-5sc2z -n greptime --tail=200
I think the namespace might have multiple metasrv instances. Can you run this command and share the output?
Copy code
kubectl get pods -n greptime -o wide | grep meta
c
Copy code
(base) PS C:\Users\cbisht> kubectl logs greptimedb-cluster-meta-6b4f7c4ddb-5sc2z -n greptime --tail=200
2026-04-29T07:44:43.883718Z  INFO cmd: GreptimeDB version:
branch:
commit: 0177f244e9b266b3ca2a43ae6d9ead1a6a760e60
clean: true
version: 1.0.0-beta.2
2026-04-29T07:44:43.883774Z  INFO cmd: command line arguments
2026-04-29T07:44:43.883783Z  INFO cmd: argument: greptime
2026-04-29T07:44:43.883794Z  INFO cmd: argument: metasrv
2026-04-29T07:44:43.883798Z  INFO cmd: argument: start
2026-04-29T07:44:43.883801Z  INFO cmd: argument: --rpc-bind-addr
2026-04-29T07:44:43.883805Z  INFO cmd: argument: 0.0.0.0:3002
2026-04-29T07:44:43.883809Z  INFO cmd: argument: --http-addr
2026-04-29T07:44:43.883812Z  INFO cmd: argument: 0.0.0.0:4000
2026-04-29T07:44:43.883818Z  INFO cmd: argument: --rpc-server-addr
2026-04-29T07:44:43.883822Z  INFO cmd: argument: 10.233.121.232:3002
2026-04-29T07:44:43.883825Z  INFO cmd: argument: --config-file
2026-04-29T07:44:43.883828Z  INFO cmd: argument: /etc/greptimedb/config.toml
2026-04-29T07:44:43.887426Z  INFO cmd: Heap profile is active
2026-04-29T07:44:43.887526Z  INFO cmd: GreptimeDB start with cpu limit in millicores: 1000
2026-04-29T07:44:43.887576Z  INFO cmd: GreptimeDB start with memory limit in bytes: 4294967296
2026-04-29T07:44:43.887591Z  INFO cmd::metasrv: Metasrv start command: StartCommand {
    rpc_bind_addr: Some(
        "0.0.0.0:3002",
    ),
    rpc_server_addr: Some(
        "10.233.121.232:3002",
    ),
    store_addrs: None,
    config_file: Some(
        "/etc/greptimedb/config.toml",
    ),
    selector: None,
    use_memory_store: None,
    enable_region_failover: None,
    http_addr: Some(
        "0.0.0.0:4000",
    ),
    http_timeout: None,
    env_prefix: "GREPTIMEDB_METASRV",
    data_home: None,
    store_key_prefix: "",
    max_txn_ops: None,
    backend: None,
}
2026-04-29T07:44:43.887645Z  INFO cmd::metasrv: Metasrv options: MetasrvOptions {
    store_addrs: [
        "postgres-cluster-rw.shared-services.svc.cluster.local:5432/greptime",
    ],
    backend_tls: None,
    selector: RoundRobin,
    use_memory_store: false,
    enable_region_failover: false,
    allow_region_failover_on_local_wal: false,
    grpc: GrpcOptions {
        bind_addr: "0.0.0.0:3002",
        server_addr: "10.233.121.232:3002",
        max_recv_message_size: 512.0MiB,
        max_send_message_size: 512.0MiB,
        max_total_message_memory: 0B,
        flight_compression: ArrowIpc,
        runtime_size: 8,
        tls: TlsOption {
            mode: Disable,
            cert_path: "",
            key_path: "",
            ca_cert_path: "",
            watch: false,
        },
        max_connection_age: None,
    },
    http: HttpOptions {
        addr: "0.0.0.0:4000",
        timeout: 0ns,
        disable_dashboard: true,
        body_limit: 64.0MiB,
        max_total_body_memory: 0B,
        prom_validation_mode: Strict,
        cors_allowed_origins: [],
        enable_cors: true,
    },
    logging: LoggingOptions {
        dir: "/data/greptimedb/logs",
        level: Some(
            "info",
        ),
        log_format: Text,
        max_log_files: 720,
        append_stdout: true,
        enable_otlp_tracing: true,
        otlp_endpoint: Some(
            "<http://otelcollector-client-daemonset.observability-client.svc.cluster.local:4318/v1/traces>",
        ),
        tracing_sample_ratio: Some(
            TracingSampleOptions {
                default_ratio: 1.0,
                rules: [],
            },
        ),
        otlp_export_protocol: None,
        otlp_headers: {},
    },
    procedure: ProcedureConfig {
        max_retry_times: 12,
        retry_delay: 500ms,
        max_metadata_value_size: Some(
            1.5MiB,
        ),
        max_running_procedures: 128,
    },
    failure_detector: PhiAccrualFailureDetectorOptions {
        threshold: 8.0,
        min_std_deviation: 100ms,
        acceptable_heartbeat_pause: 10s,
    },
    datanode: DatanodeClientOptions {
        client: ClientOptions {
            timeout: 10s,
            connect_timeout: 1s,
            tcp_nodelay: true,
        },
    },
    enable_telemetry: true,
    data_home: "./greptimedb_data",
    wal: RaftEngine,
    store_key_prefix: "",
    max_txn_ops: 128,
    flush_stats_factor: 3,
    tracing: TracingOptions,
    backend: PostgresStore,
    event_recorder: EventRecorderOptions {
        ttl: 7776000s,
    },
    stats_persistence: StatsPersistenceOptions {
        ttl: 0ns,
        interval: 600s,
    },
    meta_table_name: "greptime_metakv",
    meta_election_lock_id: 1,
    meta_schema_name: None,
    node_max_idle_time: 86400s,
}
2026-04-29T07:44:43.953460Z  INFO meta_srv::election::rds::postgres: PgElection uses default search_path (no schema provided)
2026-04-29T07:44:49.045414Z  INFO tokio_postgres::connection: NOTICE: relation "greptime_metakv" already exists, skipping
2026-04-29T07:44:49.045854Z  INFO meta_srv::bootstrap: Using selector from options, selector type: RoundRobin
2026-04-29T07:44:49.046248Z  INFO common_procedure::store: The procedure state store path is: procedure/
2026-04-29T07:44:49.046268Z  INFO common_event_recorder::recorder: Start the background processor in event recorder to handle the received events.
2026-04-29T07:44:49.046699Z  INFO common_stat::resource: Starting to collect CPU usage periodically for every 5 seconds
2026-04-29T07:44:49.062440Z  INFO cmd: Starting app: greptime-metasrv
2026-04-29T07:44:49.062480Z  INFO meta_srv::bootstrap: starting http server at 0.0.0.0:4000
2026-04-29T07:44:49.062644Z  INFO servers::http: HTTP server timeout is disabled
2026-04-29T07:44:49.063106Z  INFO servers::http: HTTP server is bound to 0.0.0.0:4000
2026-04-29T07:44:49.083367Z  INFO common_greptimedb_telemetry: Attention: GreptimeDB now collects anonymous usage data to help improve its roadmap a
nd prioritize features.
2026-04-29T07:44:49.083409Z  INFO common_greptimedb_telemetry: To learn more about this anonymous program and how to deactivate it if you don't want
 to participate, please visit the following URL:
2026-04-29T07:44:49.083415Z  INFO common_greptimedb_telemetry: <https://docs.greptime.com/reference/telemetry>
2026-04-29T07:44:49.083677Z  INFO meta_srv::metasrv: Metasrv started
2026-04-29T07:44:49.083883Z  INFO meta_srv::bootstrap: gRPC server is bound to: 0.0.0.0:3002
hmm... I think there is one more instance running on some other namespace
Copy code
(base) PS C:\Users\cbisht> kubectl get pods -n greptime -o wide | Select-string meta

greptimedb-cluster-meta-6b4f7c4ddb-5sc2z        2/2     Running   0              63m    10.233.121.232   genainode3   <none>           <none>
There is one more greptime deployed on this cluster but on some other namespace, maybe someone is using this for testing. I'm assuming this is creating some conflict here
w
There is one more greptime deployed on this cluster
Yes, I think this might be the reason why we didn't see any new logs from the metasrv.
Maybe we should first figure out which cluster we're operating on right
c
Hi @Weny Xu, I removed the other instance which was deployed and then again triggered the reconcile procedure. here are the logs in the metasrv:
Copy code
2026-04-29T11:32:23.538069Z INFO common_greptimedb_telemetry: Attention: GreptimeDB now collects anonymous usage data to help improve its roadmap and prioritize features.
2026-04-29T11:32:23.538088Z INFO common_greptimedb_telemetry: To learn more about this anonymous program and how to deactivate it if you don't want to participate, please visit the following URL:
2026-04-29T11:32:23.538091Z INFO common_greptimedb_telemetry: <https://docs.greptime.com/reference/telemetry>
2026-04-29T11:32:23.538628Z INFO meta_srv::metasrv: Metasrv started
2026-04-29T11:32:23.539118Z INFO meta_srv::bootstrap: gRPC server is bound to: 0.0.0.0:3002
2026-04-29T11:32:23.588069Z INFO meta_srv::metasrv: Leader's cache has bean cleared on leader change: Elected(LeaderKey { name: 10.233.127.12:3002, key: __metasrv_election, rev: 0, lease: 0 })
2026-04-29T11:32:23.588252Z INFO meta_srv::election: [10.233.127.12:3002] is elected as leader: "10.233.127.12:3002", lease: 0
2026-04-29T11:32:23.679518Z INFO common_procedure::local: LocalManager is start.
2026-04-29T11:32:23.679544Z INFO common_procedure::local: LocalManager start to recover
2026-04-29T11:32:23.684365Z INFO common_procedure::local: LocalManager finish recovery, cost: 4ms
2026-04-29T11:32:23.685689Z INFO common_meta::node_expiry_listener: On leader start, node expiry listener started with max idle time: 86400s
2026-04-29T11:32:28.314861Z INFO meta_srv::handler: Pusher register: Flownode-0
2026-04-29T11:32:28.336602Z INFO meta_srv::handler::remap_flow_peer_handler: Successfully updated flow `NodeAddressValue`: Peer { id: 0, addr: "10.233.72.23:4001" }
2026-04-29T11:32:28.336637Z INFO meta_srv::handler::remap_flow_peer_handler: Invalidate flow node cache for new address with cache idents: [FlowNodeAddressChange(0)]
2026-04-29T11:32:29.682724Z INFO meta_srv::handler: Pusher register: Datanode-1
2026-04-29T11:32:29.682838Z INFO meta_srv::handler::extract_stat_handler: New handshake request: RequestHeader { protocol_version: 1, member_id: 1, role: Datanode, tracing_context: {} }
2026-04-29T11:32:29.881247Z INFO meta_srv::handler::collect_stats_handler: Successfully updated datanode `NodeAddressValue`: Peer { id: 1, addr: "10.233.72.31:4001" }
2026-04-29T11:32:31.546684Z INFO meta_srv::handler: Pusher register: Datanode-2
2026-04-29T11:32:31.546771Z INFO meta_srv::handler::extract_stat_handler: New handshake request: RequestHeader { protocol_version: 1, member_id: 2, role: Datanode, tracing_context: {} }
2026-04-29T11:32:31.640326Z INFO meta_srv::handler::collect_stats_handler: Successfully updated datanode `NodeAddressValue`: Peer { id: 2, addr: "10.233.89.223:4001" }
2026-04-29T11:32:42.786033Z INFO meta_srv::handler: Pusher register: Frontend-0
2026-04-29T11:32:43.525853Z INFO meta_srv::handler: Pusher register: Frontend-1
2026-04-29T11:33:06.611222Z INFO meta_srv::handler: Pusher register: Datanode-0
2026-04-29T11:33:06.611365Z INFO meta_srv::handler::extract_stat_handler: New handshake request: RequestHeader { protocol_version: 1, member_id: 0, role: Datanode, tracing_context: {} }
2026-04-29T11:33:06.625648Z INFO meta_srv::handler::collect_stats_handler: Successfully updated datanode `NodeAddressValue`: Peer { id: 0, addr: "10.233.121.222:4001" }
2026-04-29T11:35:58.965539Z INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Runner metasrv-procedure::ReconcileLogicalTables-b64d8831-34f2-4b09-8418-3fee91dcc516 starts
2026-04-29T11:35:58.968918Z INFO LocalManager::submit_root_procedure: common_grpc::channel_manager: ChannelManager: 1, channel recycle is started, running in the background!
2026-04-29T11:35:59.010757Z INFO LocalManager::submit_root_procedure: common_meta::reconciliation::reconcile_logical_tables::reconciliation_start: Starting reconciliation for logical tables: [1029], physical_table_id: 1027, table_name: greptime.metrics_schema.greptime_physical_table, procedure_id: b64d8831-34f2-4b09-8418-3fee91dcc516
2026-04-29T11:35:59.028289Z INFO LocalManager::submit_root_procedure: common_meta::reconciliation::reconcile_logical_tables::resolve_table_metadatas: Resolving table metadatas for physical table: greptime.metrics_schema.greptime_physical_table, table_id: 1027, updating table infos: [], creating tables: []
2026-04-29T11:35:59.032412Z INFO LocalManager::submit_root_procedure: common_meta::reconciliation::reconcile_logical_tables::reconciliation_end: Logical tables reconciliation completed. logical tables: [1029], physical_table_id: 1027, table_name: greptime.metrics_schema.greptime_physical_table, procedure_id: b64d8831-34f2-4b09-8418-3fee91dcc516, metrics: column_metadata_consistent_count: 1, total_table_count: 1, elapsed: 72.975765ms
2026-04-29T11:35:59.035874Z INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::ReconcileLogicalTables-b64d8831-34f2-4b09-8418-3fee91dcc516 done
2026-04-29T11:35:59.036037Z INFO common_meta::reconciliation::manager: Reconciliation procedure b64d8831-34f2-4b09-8418-3fee91dcc516 is finished successfully!
2026-04-29T11:35:59.055379Z INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Runner metasrv-procedure::ReconcileLogicalTables-b64d8831-34f2-4b09-8418-3fee91dcc516 exits
Frontend pod logs:
Copy code
5: Column kv_cache_dtype_skip_layers not found in logical region 4419521347584(1029, 0), at src/client/src/error.rs:173:1
2026-04-29T11:35:58.870925Z INFO sql{protocol="http" request_type="sql"}:do_query:execute_sql:execute_admin_command: common_function::admin::reconcile_table: Reconciling table: greptime.metrics_schema.vllm_cache_config_info with resolve_strategy: UseLatest
2026-04-29T11:35:58.870998Z INFO sql{protocol="http" request_type="sql"}:do_query:execute_sql:execute_admin_command: common_grpc::channel_manager: ChannelManager: 0, channel recycle is started, running in the background!
2026-04-29T11:36:02.818670Z WARN servers::error: Failed to handle HTTP request err=0: Execute gRPC query error, at src/frontend/src/instance/otlp.rs:112:18
1: Table operation error, at src/frontend/src/instance/grpc.rs:469:14
2: Failed to insert data, at /greptimedb/src/operator/src/insert.rs:422:26
3: External error, at src/client/src/region.rs:65:31
4: Failed to request RegionServer 10.233.121.222:4001, code: Some requested entity was not found, at src/client/src/region.rs:272:31
5: Column kv_cache_dtype_skip_layers not found in logical region 4419521347584(1029, 0), at src/client/src/error.rs:173:1
2026-04-29T11:36:38.822923Z WARN servers::error: Failed to handle HTTP request err=0: Execute gRPC query error, at src/frontend/src/instance/otlp.rs:112:18
1: Table operation error, at src/frontend/src/instance/grpc.rs:469:14
2: Failed to insert data, at /greptimedb/src/operator/src/insert.rs:422:26
3: External error, at src/client/src/region.rs:65:31
4: Failed to request RegionServer 10.233.121.222:4001, code: Some requested entity was not found, at src/client/src/region.rs:272:31
5: Column kv_cache_dtype_skip_layers not found in logical region 4419521347584(1029, 0), at src/client/src/error.rs:173:1
2026-04-29T11:37:39.252000Z WARN servers::error: Failed to handle HTTP request err=0: Execute gRPC query error, at src/frontend/src/instance/otlp.rs:112:18
1: Table operation error, at src/frontend/src/instance/grpc.rs:469:14
2: Failed to insert data, at /greptimedb/src/operator/src/insert.rs:422:26
3: External error, at src/client/src/region.rs:65:31
4: Failed to request RegionServer 10.233.121.222:4001, code: Some requested entity was not found, at src/client/src/region.rs:272:31
5: Column kv_cache_dtype_skip_layers not found in logical region 4419521347584(1029, 0), at src/client/src/error.rs:173:1
2026-04-29T11:38:08.811931Z WARN servers::error: Failed to handle HTTP request err=0: Execute gRPC query error, at src/frontend/src/instance/otlp.rs:112:18
1: Table operation error, at src/frontend/src/instance/grpc.rs:469:14
2: Failed to insert data, at /greptimedb/src/operator/src/insert.rs:422:26
3: External error, at src/client/src/region.rs:65:31
4: Failed to request RegionServer 10.233.121.222:4001, code: Some requested entity was not found, at src/client/src/region.rs:272:31
5: Column kv_cache_dtype_skip_layers not found in logical region 4419521347584(1029, 0), at src/client/src/error.rs:173:1
2026-04-29T11:39:08.815275Z WARN servers::error: Failed to handle HTTP request err=0: Execute gRPC query error, at src/frontend/src/instance/otlp.rs:112:18
1: Table operation error, at src/frontend/src/instance/grpc.rs:469:14
2: Failed to insert data, at /greptimedb/src/operator/src/insert.rs:422:26
3: External error, at src/client/src/region.rs:65:31
4: Failed to request RegionServer 10.233.121.222:4001, code: Some requested entity was not found, at src/client/src/region.rs:272:31
5: Column kv_cache_dtype_skip_layers not found in logical region 4419521347584(1029, 0), at src/client/src/error.rs:173:1
👀 1
Please let me know if there are any other steps which I need to follow before restarting.
w
The logs show that the table metadata between the metasrv and datanode is consistent. Could you restart the whole cluster?
c
I restarted but again the same error is there
Do I need to restart the operator as well?
w
No, I think we can try this approach: First, enter the metasrv pod. Then, check your configuration to identify the 
store-addrs
and backend settings:
Copy code
cat /etc/greptimedb/config.toml
After that, run the following command:
Copy code
greptime cli meta repair logical-tables --store-addrs=$ENDPOINT \
  --backend=postgres-store \
  --table-names=vllm_cache_config_info \
  --schema-name=metrics_schema \
  --catalog-name=greptime
https://docs.greptime.com/reference/command-lines/utilities/repair-logical-tables/
Please replace the 
backend
and 
store-addrs
values in the command with the corresponding settings from your 
config.toml
. e.g.,
And please share the output after running the command.
c
Here:
Copy code
2026-04-30T09:16:01.526038Z  INFO tokio_postgres::connection: NOTICE: relation "greptime_metakv" already exists, skipping
2026-04-30T09:16:01.526494Z  INFO cmd: Starting app: greptime-cli
2026-04-30T09:16:01.552118Z  INFO cli::metadata::repair: Sending alter table requests to all datanodes for table: greptime.metrics_schema.vllm_cache
_config_info, number of regions:1.
2026-04-30T09:16:01.552307Z  INFO common_runtime::global: Creating runtime with runtime_name: global, thread_name: global-worker, work_threads: 8.
2026-04-30T09:16:01.554119Z  INFO common_runtime::global: Creating runtime with runtime_name: compact, thread_name: compact-worker, work_threads: 4.
2026-04-30T09:16:01.555100Z  INFO common_runtime::global: Creating runtime with runtime_name: heartbeat, thread_name: hb-worker, work_threads: 2.
2026-04-30T09:16:01.556140Z  INFO common_grpc::channel_manager: ChannelManager: 0, channel recycle is started, running in the background!
2026-04-30T09:16:28.097874Z  INFO cli::metadata::repair: All alter table requests sent successfully for table: greptime.metrics_schema.vllm_cache_co
nfig_info
2026-04-30T09:16:28.097933Z  INFO cli::metadata::repair: Repair logical tables result: 1 tables repaired, 0 tables skipped
2026-04-30T09:16:28.097945Z  INFO cmd: Goodbye!
w
Ok, are there any new error logs from the frontend?
c
Those are resolved now, but when querying any trace using a trace_id, I see this error
Copy code
2026-04-30T11:45:22.408927Z ERROR handle_get_trace{protocol="jaeger" request_type="get_trace"}: servers::http::jaeger: Failed to get trace '14f24fc8b79b96b689c6002ce0d57675': 0: Failed to collect recordbatch, at src/servers/src/http/jaeger.rs:719:22
1: <transparent>, at /rustc/fa3155a644dd62e865825087b403646be01d4cef/library/core/src/task/poll.rs:291:44
2: External(0: External error, at src/client/src/region.rs:177:55
1: NotFound (persistent) at => NotFound (permanent) at read, context: { uri: <https://dummy-storage.com/greptime/data/greptime/traces_schema/1026/1026_0000000001/4a58675f-4e58-4378-86a2-3ed85e8b7e0d.parquet>, response: Parts { status: 404, version: HTTP/1.1, headers: {"accept-ranges": "bytes", "content-length": "431", "content-type": "application/xml", "server": "istio-envoy", "x-amz-request-id": "1777549522345056324", "date": "Thu, 30 Apr 2026 11:45:22 GMT", "x-envoy-upstream-service-time": "11"} }, service: s3, path: data/greptime/traces_schema/1026/1026_0000000001/4a58675f-4e58-4378-86a2-3ed85e8b7e0d.parquet, range: 5957844-6023379 } => S3Error { code: "NoSuchKey", message: "The specified key does not exist.", resource: "/greptime/data/greptime/traces_schema/1026/1026_0000000001/4a58675f-4e58-4378-86a2-3ed85e8b7e0d.parquet", request_id: "1777549522345005477" }, at src/client/src/error.rs:173:1)
Logs from the frontend pod ^^
w
Does it reproduce on retries?
c
The first time I restarted the cluster I was able to see the traces, but after few minutes when I tried again it is now continuously giving:
Copy code
HTTP Error: 3001 - Failed to collect recordbatch
Status500URL/api/traces/dff3a2e521798e5ad3d09eb8e8678d10Response body{
  "data": null,
  "total": 0,
  "limit": 0,
  "offset": 0,
  "errors": [
    {
      "code": 3001,
      "msg": "Failed to collect recordbatch"
    }
  ]
}
there were no such errors when I restarted just INFO logs, but the moment I hit any trace query we are seeing this
w
Did you see any errors from frontend pods？or datanode pods
c
nothing, just info logs. This is what I got in frontend pod after hitting the trace query though:
Copy code
2026-04-30T11:45:22.408927Z ERROR handle_get_trace{protocol="jaeger" request_type="get_trace"}: servers::http::jaeger: Failed to get trace '14f24fc8b79b96b689c6002ce0d57675': 0: Failed to collect recordbatch, at src/servers/src/http/jaeger.rs:719:22
1: <transparent>, at /rustc/fa3155a644dd62e865825087b403646be01d4cef/library/core/src/task/poll.rs:291:44
2: External(0: External error, at src/client/src/region.rs:177:55
1: NotFound (persistent) at => NotFound (permanent) at read, context: { uri: <https://dummy-storage.com/greptime/data/greptime/traces_schema/1026/1026_0000000001/4a58675f-4e58-4378-86a2-3ed85e8b7e0d.parquet>, response: Parts { status: 404, version: HTTP/1.1, headers: {"accept-ranges": "bytes", "content-length": "431", "content-type": "application/xml", "server": "istio-envoy", "x-amz-request-id": "1777549522345056324", "date": "Thu, 30 Apr 2026 11:45:22 GMT", "x-envoy-upstream-service-time": "11"} }, service: s3, path: data/greptime/traces_schema/1026/1026_0000000001/4a58675f-4e58-4378-86a2-3ed85e8b7e0d.parquet, range: 5957844-6023379 } => S3Error { code: "NoSuchKey", message: "The specified key does not exist.", resource: "/greptime/data/greptime/traces_schema/1026/1026_0000000001/4a58675f-4e58-4378-86a2-3ed85e8b7e0d.parquet", request_id: "1777549522345005477" }, at src/client/src/error.rs:173:1)
w
Ok, does the error consistently occur on the frontend whenever a query fails? Could you share the logs?
c
Yes, and these are some new recent errors in the frontend:
Copy code
2026-04-30T12:11:34.935198Z ERROR servers::error: Failed to handle HTTP request err=0: Execute gRPC query error, at src/frontend/src/instance/otlp.rs:112:18
1: Table operation error, at src/frontend/src/instance/grpc.rs:469:14
2: Failed to insert data, at /greptimedb/src/operator/src/insert.rs:422:26
3: External error, at src/client/src/region.rs:65:31
4: Failed to request RegionServer 10.233.121.231:4001, code: Unknown error, at src/client/src/region.rs:272:31
5: transport error, at src/client/src/error.rs:173:1
w
Any errors from datanode?
c
Hey @Weny Xu I restarted all the pods after some time again and all the errors were resolved. Thank you so much for the help 🙏
👍 1
🙌 1
Open in Slack
PreviousNext
Powered by