Hi, quick question about moving data across environments — looking for best practices. I have a Pinot setup with 36 REALTIME tables (all Kafka-ingested) spanning roughly 2k customers. Some customers are small (100–1k rows per table), some are significantly larger. I’ve built an isolated DEV environment with all my tooling (MCP, Trino, etc.) connected to it, and I deliberately don’t want any of those tools to touch prod directly. What I’m trying to solve: selectively copy a single customer’s data from prod to dev, across all tables. Constraints: - Kafka retention in prod is only 7 days, so replaying from the topic isn’t an option for historical data. - Several tables have large blob columns (noDictionary, multi-MB raw JSON stored in attributes/relationships) that are never used in queries — they’re only there for backfill purposes. I don’t need to copy those. - My main concern is not breaking prod with expensive, large-volume exports — even with pagination, I’m wary of putting too much read pressure on the Pinot cluster. Current approach I’m considering: - Query prod via Trino (which sits in front of Pinot), filtering by meta.customer, selecting only the lightweight indexed columns (skipping the heavy blob columns). - Paginate with LIMIT/OFFSET in manageable batches. - Reconstruct the original Kafka JSON message format by inverting the transformConfigs (since I only have the derived columns, not the raw source JSON). - Push the reconstructed messages into the dev Kafka topic using kcat or a Kafka producer, and let dev Pinot ingest them normally. Questions for the community: 1. Is querying through Trino a reasonable way to extract per-customer data without hammering Pinot directly? Any tips on throttling or batching? 2. Are there better patterns for cross-env data sync in Pinot that I’m not seeing? (Segment-level copy seems too coarse since segments aren’t partitioned by customer.) Thanks in advance!

1. it sounds like you’re hammering Pinot the same amount, just indirectly instead of directly. 2. segment level copy does seem best but mainly if you have them in deep store. there’s also been some work on realtime table replication like https://github.com/apache/pinot/issues/16563

Thks for you feedback. We will give it a try by just running query using python.