Hi team, we would appreciate guidance on the cost ...
# general
j
Hi team, we would appreciate guidance on the cost of a narrowly filtered information_schema.partitions query. Operational context We ingest Prometheus metrics using remote_write into metric-engine logical tables backed by a shared physical table. During investigation of uneven data distribution and an SST compaction backlog, we repartitioned prometheus_risingwave.greptime_physical_table online from one data region to 15 namespace ranges. We then needed to list partition boundaries and greptime_partition_id, and correlate those IDs with region_peers and ssts_manifest to understand placement and per-region SST distribution. This is an administrative metadata lookup, not a scan of raw metric samples and not an attempt to obtain row counts from the partitions dictionary. The query targets exactly one schema/table and returns only 15 rows:
Copy code
SELECT *
FROM information_schema.partitions
WHERE table_schema = 'prometheus_risingwave'
  AND table_name = 'greptime_physical_table'
ORDER BY partition_ordinal_position;
For profiling we prepend EXPLAIN ANALYZE VERBOSE to this exact SQL. Observed timeline (different measurements, not a controlled benchmark) • Original user-observed latency: over 14 seconds. Historical plan: RepartitionExec.fetch_time=13.4s, sort compute 12us, filter compute 154us, 15 output rows. • 2026-10-03 12:32 UTC historical re-test: the same EXPLAIN query reported fetch_time=7.67s and 15 rows. • Fresh collection 2026-10-05 08:37 UTC, via Grafana datasource HTTP proxy: original SELECT returned 15 rows in 4.739s client wall time. • A separate EXPLAIN execution immediately afterward took 2.566s client wall time, with 2.45s fetch_time, SortExec 11us and FilterExec 71us. Full current plan attached in latest-plan.txt. Client wall time includes Grafana/proxy/network overhead; fetch_time is an execution-plan metric. They must not be equated. These were sequential single executions without cache reset, while normal ingestion continued; background activity was not controlled. We are NOT claiming a constant 14-second latency or a measured p95. The latest plan still shows:
Copy code
FilterExec: table_schema = prometheus_risingwave AND table_name = greptime_physical_table
  RepartitionExec: ... fetch_time: 2.45s
    StreamScanAdapter
Filter output is 15 rows; the plan reports 847.7KiB output_bytes at FilterExec/RepartitionExec. We do not interpret that byte counter as exact catalog input volume or infer a root cause from it alone. Environment • Current SELECT version() reverified: 1.3.0-beta.1. • Previously verified binary commit: 133296f298c504b920772292c22f169cd46c54c9 (not freshly read from every Pod in this collection). • Latest partition readback: 15 RANGE partitions on namespace, p0–p14, region IDs 4823248273408–4823248273422. • Fresh table inventory in the two metric schemas: prometheus has 78 metric-engine tables, 13 mito tables and 3 entries with blank engine; prometheus_risingwave has 1707 metric-engine tables. Metric counts include physical tables, not only logical tables. This is not an all-catalog table count. The table-count query itself took 13.116s client wall time, a separate observation without a plan. • Last Kubernetes inventory (2026-10-03, NOT refreshed today): 3 frontend (2 CPU/8Gi each), 3 metasrv (2 CPU/4Gi), 8 datanode (8 CPU/16Gi), 3 flownode (4 CPU/8Gi); requests=limits. Four Kubernetes hosts, shared S3-compatible storage, local WAL/cache. Question Why does a metadata query filtered to a single table and returning only 15 partition rows still take more than 2 seconds in GreptimeDB? We would expect this basic single-table metadata lookup to be much faster. Could you help explain the observed latency?
p
I have only seen this slow down the way stated if you repartitioned after data was already in the table. As stated in my other comment on the other thread, make sure you do the partitioning prior to the first ingested metric and have the field for the partitioning defined in that original DDL
e
This latency is consistent with the current implementation: the schema/table filter is not pushed down into metadata retrieval, so even a query returning 15 rows can retrieve metadata for thousands of unrelated tables. This administrative query was not previously treated as a frequent, latency-sensitive operation. A single-table lookup should ideally avoid unrelated metadata work. We’ve opened github.com/GreptimeTeam/greptimedb/issues/9442 to track a future improvement.