Hi team, in there any way to Query using Cursors with trino?

Hi team, i have thousands of users for which i need to apply rls, so is there any dynamic way , or i have to onboard the users and add them in broker.conf? pinot.broker.access.control.principals.alice.password=alice123 pinot.broker.access.control.principals.admin.password=admin pinot.broker.access.control.principals.alice.tables=user_data_REALTIME pinot.broker.access.control.principals.alice.user_data_REALTIME.rls=userId='102'

Hi Team I see DropwizardJmxReporter and YammerJmxReporter in apache pinot; Could you please let me know which is default one and how can we change to other one like DropwizardJmxReporter if YammerJmxReporter is default one.

Hi team, How can we overwrite OFFLINE tables?

Has anyone else encountered the

Expression cycle

problem when using an ingest transform function where the source field name matches the pinot column name? In my case I have geojson that has a geometry field containing coordinates (

{"geometry": {"coordinates": [-82.41959120338983, 35.61020539194915]}}

). I'm looking for a way to convert those coordinates using

toSphericalGeography(stPoint(lon, lat))

into a column also named geometry.

Hello, our team is interested in exploring realtime upsert tables, but I am curious about the maintenance and scaling of the table going forward. Conceptually I understand that the same partition of data has to always route through the same server for this to work. Referencing these docs: https://docs.pinot.apache.org/manage-data/data-import/upsert-and-dedup/upsert#enable-upserts-in-pinot I see we must use

{
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  }
}

and also

numInstancesPerPartition=1

With this is mind, is it still possible to scale up the number of servers horizontally once the table is set up? And what is the process then to do so?

Hello Team I have deployed Apache Pinot on Amazon EKS and configured Horizontal Pod Autoscaling (HPA) for the Pinot Broker using both CPU and memory metrics. However, I am observing that the Broker pods always scale up even when CPU usage is low and no queries are being executed. The memory consumption keeps increasing slowly over time, which continuously triggers the memory-based HPA threshold. Could you please confirm the recommended autoscaling strategy for Pinot Broker? Should we scale only based on CPU, or should memory also be included? Below is the HPA configuration I am currently using: apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: hpa-pinot-broker namespace: pinot-system # <-- Add the namespace here spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: pinot-broker minReplicas: 2 maxReplicas: 8 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 75 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 75

Hi Team, What should be end point for these APIs controller or broker ? also are these enabled default if not what we need to set to get running queries ?

Hi Team, This is regarding null handling in Pinot. I have read about table based, column based and advanced query time null handling. My requirement is simple : Good old way of having nulls when there is no value for a column. Right now, I have enabled column based null handling, but it gives default values for non existent values for eg. -infinity for decimal type columns I tried advanced query based null handling along with column based null handling, but now it gives 0.0 for decimal types. How can I achieve simple nulls when there is no value for a column ie. null for all data types if there is no value for that column in the row?

Couple open questions about upsert tables and the metadataTTL config: 1. If no

metadataTTL

is set, do primary keys then expire and free up memroy at the end of the data TTL (set by the

segmentsConfig

retention settings)? Or do primary keys live forever in this case? 2. Should

metadataTTL

always be set to longer than the segment config retention setting? I imagine if you went to update a record that fell out of the

metadataTTL

, but the old segment data was still around, you could have duplicate data at that point?

Hello, I want row based TTL on a particular table. Could anyone suggest any solution. Thanks.

I have a table on production, it was not able to commit the consuming segments from last 3 days, my kafka has retention of 2 days, so now i have deleted my consuming segments and tried to invoking the

RealtimeSegmentValidationManager

task, after that also triggered this api tables/{{table_name}}/resumeConsumption?consumeFrom=largest still consuming segments are not spawning. Please help to resolve this

I noticed that the Pinot server's /debug/memory/offheap API shows 512MB mmap files for each real-time consuming segment (path: data/table_name/consumers/segment_name) The same was mentioned in the javadoc of MmapMemoryManager

We create files of length 0.5g (or the requested buffer length, whichever is higher), and map areas of the file for each allocation request within a segment

Can someone confirm if at least 512MB is allocated per Kafka partition consumer, and is there a way to configure this size to reduce RES memory usage incase of high partition count ?

Hello we are observing wrong result for below query select dateTrunc('hour', 1765295114678, 'MILLISECONDS', 'America/New_York', 'MILLISECONDS') AS original_event_time_hours_buckets2,dateTrunc('hour', 1765295114678, 'MILLISECONDS', 'Europe/Berlin', 'MILLISECONDS') AS original_event_time_hours_buckets from test limit 10 its giving us same Epoch milliseconds as UTC. 1765292400000 1765292400000 it should have different epoch milliseconds for different timezone.How ever ASIA/Kolkota timezone is giving correct result. is it bug for timezone aware query for pinot?

can you please help me?

For multi-stream ingestion, there was a bug in KafkaConsumerFactory (using partitionGroupId instead of streamPartitionGroupId). I'm assuming the fix in PR #15094 isn't in 1.3.0 since it's missing from the release tag and was opened after the Feb 17 release date. Should we explicitly mention in the 1.3.0 release doc that Kafka multi-topic ingestion is broken in this version? Let me know if I'm missing something

Hi Team, we are using

Apache Pinot 1.2.0

and we are facing a scaling issue with realtime upserts enabled for our use case, due to the nature of the primary key the updates can fall in after the expiration window of the

metadataTTL

defined (~8hrs) in our table config, since we cannot keep infinite retention so we are not able to show accurate information on our views due to duplicate entries being recorded We are thinking to write a custom minion job, which should do stuffs like 1. Fetch all such duplicate entries within a time window 2. Only keep the latest one and mark others as invalid so that they can be taken up for deletion in future We wanted to get community help as we didn't find any resources which provides such functionality open source , can anyone help with any references or starting point so that it can help us to use pinot efficiently cc: @Abhigyan Bharati @Xiang Fu

Hi Team, I use Apache Pinot 1.4.0 for explorational analysis of raw events (small time windows) and for upserts based on primary key. The raw events are flat json format (~3.000 Events/second). The problem is that the attribute names are containing "-" like "event-timestamp" or "event-id". I guess Pinot does not like "-" in attribute names. My first idea was to use Apache Nifi for transformation and replace "-" with "_". Nifi seems not to be the right solution for this use case. What do you think would be an efficient solution (3.000-5.000Events/second)? Thank you :)

Hi Team, I have an upsert table with uniq key (STRING, 64 chars). Ingest rate ~4–5k events/sec. Quick math for the raw ID payload: 64 bytes per row 86,400 sec/day 4–5k rows/sec → ~345–432M rows/day • ⇒ ~22–28 GB/day (≈26 GB/day avg) only for raw uniqId bytes My question: how do you roughly estimate RAM usage for upsert/dedup in Pinot (PK → docId map, validDocIds, etc.) given: • key type: 64-char STRING • high cardinality • hot window target: 3 days or more if it passible Any rule-of-thumb or example numbers would help

Hi, I am looking to consider Pinot for our company however, I am concerned I can not see anything about release schedules either in the opensource or closed source version - not any policies on security patches / updates? Have I missed a resource in my search? I notice until last year releases were roughly every 3-6 months but there seems to have been nothing since September 25 - so I am guessing that means a main release is due but what also about security updates and patches inbetween? Thanks

Here are the recordings and slides from meetup Apache Pinot: What’s Advancing Across Query, Ingestion, and the Core Engine. Recording: Here is the recording of the session Slides: https://docs.google.com/presentation/d/1Eiwx8cOXjZfGayX3nLV6ixBJkqLOgwAzGJ0_jMrZp00/edit?usp=sharing

Cross-posting https://apache-pinot.slack.com/archives/C013WKLT5T7/p1773722199562819

Hi Team, does pinot supports multi region (Live-Live within US) deployment ?

Another question, what is the recommendation for migrating realtime segments from one pinot server to another (different cluster) ? For migrating offline segments we are using segment uri push method. Please note that the migration should be idempotent.

Is there any way to update hdfs uri (segment.download.url) for all segments in a pinot table at a time? I would like to change hdfs hardcoded namenode uri to NN HA nameservice. cc: @Mayank @Xiang Fu @Jackie

Hey Pinot team 👋 Slack is recently using Pinot upsert to build the realtime table Could you please you share the recommended metrics to evaluate if the realtime cluster is in the healthy status thanks Based on what we know, heap utilization could be a critical metrics to validate is the node is overwhelming and we can confirm that our heap utilization is under 80% based on this calculation

jvm_memory_bytes_used{instance=~'.*staging.*',area="heap"} / on(instance) group_left jvm_memory_bytes_max{instance=~"pinot-.*",instance=~".*staging-.*",instance=~".*realtime-.*",area="heap"} * 100

cc: @Neeraja Sridharan @Sai Tarun Tadakamalla

Hi team 👋 Pinot 1.4.0 — getting

BrokerResourceMissingError (410)

on a REALTIME table after adding this to broker config:

pinot.broker.adaptive.server.selector.enable.stats.collection=true
pinot.broker.adaptive.server.selector.type=HYBRID

After updating the config, I am encountering the following error while running the query: Error Code: 410 (BrokerResourceMissingError) BrokerResourceMissingError What am I missing here? Observations: • All 3 brokers:

idealState: ONLINE

but

externalView: ERROR

for this table only • Other tables querying fine •

rebuildBrokerResourceFromHelixTags

returns

"ideal state is the same"

@Mayank @Xiang Fu do we have any tool or documentation for migrating Druid to Pinot?

Hello team We have offline table in pinot and that table has a country name and transaction. We want to set a retention period per country.for example india we want 6 days and if japan we want to set 10 days .how can we do it

Hi all, Apache Pinot 1.5.0 has been released! https://github.com/apache/pinot/releases/tag/release-1.5.0 https://pinot.apache.org/download/ https://downloads.apache.org/pinot/apache-pinot-1.5.0/ https://hub.docker.com/layers/apachepinot/pinot/1.5.0