Hi team, we'd like to temporarily read JSON-formatted files (totaling over 10 TB) from object storage, then count occurrences grouped by

appId

. We're planning to use an external table for this. Do you have any good solutions or suggestions? Thanks in advance. docs.greptime.com/reference/sql/create#… Json content

{
  "_index": "xlog-sd-2026-07",
  "_type": "_doc",
  "_id": "ZYDmuJ8B1n1_pofbyNyl",
  "_score": 1,
  "_source": {
    "@timestamp": "2026-07-31T15:59:20.089Z",
    "kafkaTopic": "it-business-log-filebeat",
    "appId": "58311ae4a190491ca564bc3d",
    "xlog": {
      "level": "DEBUG",
      "branchType": "",
      "zipkinParentId": "e5252737946dafa6",
      "message": "表达式：priceListType ns '',参数：{\"prodModelPrefix\":\"0#NotMatchRuleField\",\"functionDescVal\":\"0#NotMatchRuleField\",\"itemCode\":\"SU-070101-00007\",\"mainSaleModelVal\":\"0#NotMatchRuleField\",\"marketProdModelVal\":\"0#ST33-M\",\"priority\":\"7\",\"matchRuleField\":\"MARKET_PROD_MODEL\",\"prodModel\":\"ST33-M\",\"eori\":\"I\",\"priceListType\":4,\"itemCodeVal\":\"0#NotMatchRuleField\",\"prodModelVal\":\"0#NotMatchRuleField\",\"dealerType\":\"\",\"saleModelVal\":\"0#NotMatchRuleField\",\"currency\":\"CNY\",\"priceListIds\":\"06957fb9213544728bfeada94b47c8b2,143be18efdd640c2913f47f0946f95c9,9e3c4a81a52c4c149511a34f46e8d8c8,beb97f2712124517b829501fbc031cab,c32d214bde82493b886414ed4e2a2b43,e54b464191a34bdf8aec21ac8e8ae136\"},执行结果：true",
      "id": "6f7312e31d5647ae96ab678c539556ed",
      "auditLog": null,
      "optUser": "营销客服",
      "zipkinTraceId": "e5252737946dafa6",
      "ccloudTransactionId": "2901a901905c763f",
      "domain": null,
      "optTime": "2026-07-31T23:59:11.570",
      "branchId": null,
      "zipkinSpanId": "e5252737946dafa6",
      "billIds": [
        ""
      ],
      "txId": "",
      "esIndex": "xlog-sd-2026-07",
      "typeName": "业务日志",
      "application": "cwp",
      "ccloudTraceId": "795623963275fe6b20e560bec1edc30e",
      "type": "BUSINESS",
      "apiLog": null,
      "env": "PROD",
      "prodLine": "SD",
      "stackTrace": null
    },
    "podName": "app-58311ae4a190491ca564bc3d-15-7d95cb5b86-p2rsf",
    "@version": "1"
  }
}

hi Jun, if there is any chance you will be analyzing the data again, I would recommend you to ingest it into greptimedb using the pipeline system. But if it's purely one-shot task, duckdb or a custom script can be better choice for this.

Thanks @Ning, I'll try to use duckdb first.