:::writing{variant="email" id="58321" subject="Grafana GreptimeDB plugin: intermittent "column not found" errors"} Hi, I have another issue, this time I believe it may be related to the Grafana plugin. I installed the latest version of the GreptimeDB plugin on the latest version of Grafana. I have a logs table populated through an identity pipeline, where JSON fields are parsed into columns, and the pipeline itself appears to be working as expected. However, I’m seeing inconsistent behavior on the Grafana side. When I query the table, I sometimes receive a 500 error indicating that a column cannot be found. If I simply refresh the Grafana page and run the same query again, I get the expected results. The error then comes back again later, so the behavior is not consistent. Here is an example of the query:

{
  "request": {
    "url": "api/ds/query?ds_type=info8fcc-greptimedb-datasource&requestId=SQR123",
    "method": "POST",
    "data": {
      "queries": [
        {
          "refId": "A",
          "hide": false,
          "datasource": {
            "type": "info8fcc-greptimedb-datasource",
            "uid": "Greptimedb"
          },
          "editorType": "sql",
          "format": 1,
          "meta": {
            "builderOptions": {
              "columns": [],
              "database": "",
              "limit": 1000,
              "meta": {},
              "mode": "list",
              "queryType": "table",
              "table": ""
            },
            "timezone": "America/New_York"
          },
          "pluginVersion": "2.1.4",
          "queryType": "table",
          "rawSql": "SELECT greptime_timestamp as time, \"message.traceId\", \"message.spanId\", \"message.error.message\", \"message.message\", FROM logs.\"k8s_app_logs\" WHERE greptime_timestamp BETWEEN $__fromTime and $__toTime AND level = 'ERROR' ORDER BY greptime_timestamp DESC LIMIT 1000",
          "datasourceId": 7,
          "intervalMs": 5000,
          "maxDataPoints": 1066
        }
      ],
      "from": "1787087854703",
      "to": "1787094539531"
    },
    "hideFromInspector": false
  },
  "response": {
    "results": {
      "A": {
        "error": "greptime http 400: {\"code\":3000,\"error\":\"Failed to plan SQL: No field named \\\"message.traceId\\\". Valid fields are logs.k8s_app_logs.greptime_timestamp, logs.k8s_app_logs.k8s_cluster, logs.k8s_app_logs.message, logs.k8s_app_logs.namespace, logs.k8s_app_logs.observed_timestamp, logs.k8s_app_logs.level.\",\"execution_time_ms\":3}",
        "errorSource": "downstream",
        "status": 500,
        "frames": [],
        "refId": "A"
      }
    }
  }
}

The interesting part is that the error says the parsed fields such as

message.traceId

do not exist, while the identity pipeline is successfully creating/populating these fields. There is also another related issue when I use the MySQL connector. In that case, I don't get an error, but instead of returning the field value, Grafana displays the column name itself as a string. For example, if I query:

SELECT "k8s_app_logs.k8s_cluster" FROM ...

the result is literally

k8s_app_logs.k8s_cluster

rather than the value stored in that field. Could you please take a look and let me know if this is a known issue with the Grafana plugin, or if there is something I may be missing in the configuration? Thanks! :::

Hi, The MySQL result doesn’t tell us whether the column exists. In MySQL, double quotes are string literals, so

SELECT "k8s_app_logs.k8s_cluster"

just returns that text and never reads the column. try `DESC TABLE logs.`k8s_app_logs`;` you can make this query to confirm the table schema when the error occured.

Hi! DESC TABLE logs.`k8s_app_logs`; returns different column set each time I execute it. One time its what I except, and another just a few fields

that's weird. how about

show create table logs.k8s_app_logs

? could you show us those different output?

Hi, Sorry for taking so long. So there are two results I'm getting. One is:

CREATE TABLE IF NOT EXISTS "k8s_app_logs" ( "greptime_timestamp" TIMESTAMP(9) NOT NULL, "k8s_cluster" STRING NULL, "message" STRING NULL, "namespace" STRING NULL, "observed_timestamp" STRING NULL, "application" STRING NULL, "environment" STRING NULL, "timestamp" STRING NULL, "application_instance" STRING NULL, "level" STRING NULL, TIME INDEX ("greptime_timestamp") ) ENGINE=mito WITH( append_mode = 'true' )

Second have much more columns (this is partial response - its tool large) We have hundreds of applications and each have its own log format 🤕:

It seems your table is automatically created from logs, so the table structure depends on your log structure. If you need structured logs in the table, make sure the logs ingested into this table share similar structure. For example, there should be

message.traceId

in log entries, so the column will be created. Otherwise, you will need to create the table with all required fields ahead of time, or use alter table command to add column to existing tables to add those essential columns.

Thank you! So if I just want to dynamically add columns - it wont work?

It will work. But your query is not guaranteed to work if the column in projection is not found