Hi < Mayank> < Xiang Fu> please have a look at this Ever sin Apache Pinot #troubleshooting

Hi <@UDQU92KBK> <@UGRJA9TEH>, please have a look a...

Rajat

09/19/2025, 6:28 AM

Hi @Mayank @Xiang Fu, please have a look at this. Ever since I added DeepStore configs to Pinot (started a new cluster with DeepStore settings in the controller), I will share the YAML if required. I have been facing this issue a lot. The segments are going a long way in this Bad state. And in the segments, I see errors in Pinot segments. What could be going wrong? Is it some known issue?

Jayesh Asrani

09/19/2025, 8:39 AM

Did you look at the server/controller logs? verified deep store is accessible?

Rajat

09/19/2025, 8:45 AM

Yeah Deep store is working fine all the backed up segments are going into S3 and other Deleted segments I can see in the bucket as well.

Rajat

09/19/2025, 8:46 AM

I couldn't check the logs at that the time of failure

Rajat

09/19/2025, 8:46 AM

after some time when I checked it was showing connection with consumer closed

Xiang Fu

09/19/2025, 9:13 AM

can you click the table and check per segment status?

Xiang Fu

09/19/2025, 9:14 AM

Does pinot server have valid connection to pinot server?

Rajat

09/19/2025, 9:22 AM

Does pinot server have valid connection to pinot server? --- you mean server has valid connection to pinot controller?

Rajat

09/19/2025, 9:23 AM

per segment status, most of them are in updating state and a few are in BAD state and a few in online state

Xiang Fu

09/19/2025, 9:26 AM

no, pinot servers need to download segments from deep store

Xiang Fu

09/19/2025, 9:27 AM

You can also check pinot server logs for segment loading error

Xiang Fu

09/19/2025, 9:27 AM

https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot

Rajat

09/19/2025, 9:29 AM

Yeah I followed this same documentation. Here's the yaml file:

Copy code

image:
  repository: <http://615177075440.dkr.ecr.ap-south-1.amazonaws.com/pinot|615177075440.dkr.ecr.ap-south-1.amazonaws.com/pinot>
  tag: 1.0.1
  pullPolicy: IfNotPresent

cluster:
  name: pinot-engage

  #pinotAuth:
  #enabled: true
  #controllerFactoryClass: org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory
  #brokerFactoryClass: org.apache.pinot.broker.broker.BasicAuthAccessControlFactory
  #configs:
  #- access.control.principals=admin,user
  #- access.control.principals.admin.password=rzNZhSF^Zkk@q8B%
  #- access.control.principals.user.password=xW)w(zKaj)s7!ytw
     # - access.control.principals.user.tables=universal_cookie
     #- access.control.principals.user.permissions=READ  

# ----------------------------------------------------------------------------
# ZOOKEEPER: 3 replicas
# ----------------------------------------------------------------------------
zookeeper:
  name: pinot-zookeeper
  replicaCount: 3
  persistence:
    enabled:      true
    storageClass: gp3      # ← GP3 EBS
    size:         10Gi
  resources:
    requests:
      cpu:    100m
      memory: 256Mi
    limits:
      cpu:    300m
      memory: 512Mi

  port: 2181

# ----------------------------------------------------------------------------
# CONTROLLER: 2 replicas, internal LB
# ----------------------------------------------------------------------------
controller:
  name: pinot-controller
  replicaCount: 2
  startCommand: "StartController"
  # podManagementPolicy: Parallel
  resources:
    requests:
      cpu:    100m
      memory: 1Gi
    limits:
      cpu:    300m
      memory: 3Gi
  
  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=9010:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -XX:ActiveProcessorCount=2 -Xms512M -Xmx2G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-controller.log -Djute.maxbuffer=4000000"
    
  # Persist controller metadata
  persistence:
    enabled:      true
    accessMode:   ReadWriteOnce
    storageClass: gp3
    size:         50Gi
    mountPath:    /var/pinot/controller/data

  service:
    name: controller
    annotations:
      <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
      <http://prometheus.io/path|prometheus.io/path>: /metrics
      <http://prometheus.io/port|prometheus.io/port>: "9010"
    # labels:
    #   prometheus-monitor: "true"
    extraPorts:
      - name: controller-prom
        protocol: TCP
        containerPort: 9010
  extra:
    configs: |-
      pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      controller.local.temp.dir=/var/pinot/controller/data/local_temp/
      pinot.controller.storage.factory.s3.region=ap-south-1
      pinot.controller.segment.fetcher.protocols=file,http,s3
      pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
      pinot.controller.storage.factory.s3.disableAcl=false
      pinot.controller.storage.factory.s3.pathStyleAccess=true
      pinot.set.instance.id.to.hostname=true
      controller.task.scheduler.enabled=true

  data:
    dir: <s3://sr-datapipeline/data-platform/pinot/deep_store/pinot-engage>
  
    
    
  podAnnotations:
    <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    <http://prometheus.io/path|prometheus.io/path>: /metrics
    <http://prometheus.io/port|prometheus.io/port>: "9010"

  # Expose via Kubernetes Ingress on port 9000
  # ingress:
  #   v1:
  #     enabled: true
  #     ingressClassName: nginx          # or your ingress controller
  #     annotations:
  #       <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "HTTP"
  #       <http://nginx.ingress.kubernetes.io/rewrite-target|nginx.ingress.kubernetes.io/rewrite-target>: /
  #     hosts: [<http://pinot-eks.sr-bi-internal.in|pinot-eks.sr-bi-internal.in>]
  #     path: /controller
  #     tls: []
  external:
    enabled: false

# ----------------------------------------------------------------------------
# BROKER: 2 replicas, internal LB
# ----------------------------------------------------------------------------
broker:
  name: pinot-broker
  startCommand: "StartBroker"
  # podManagementPolicy: Parallel
  replicaCount: 2

  resources:
    requests:
      cpu:    200m
      memory: 1Gi
    limits:
      cpu:    500m
      memory: 3Gi

  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=9020:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -XX:ActiveProcessorCount=2 -Xms512M -Xmx2G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-broker.log -Djute.maxbuffer=4000000"

  service:
    name: broker
    #type: LoadBalancer
    annotations:
      # <http://service.beta.kubernetes.io/aws-load-balancer-internal|service.beta.kubernetes.io/aws-load-balancer-internal>: "true"
      <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
      <http://prometheus.io/path|prometheus.io/path>: /metrics
      <http://prometheus.io/port|prometheus.io/port>: "9020"
    # labels:
    #   prometheus-monitor: "true"
    extraPorts:
      - name: broker-prom
        protocol: TCP
        containerPort: 9020
    
  podAnnotations:
    <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    <http://prometheus.io/path|prometheus.io/path>: /metrics
    <http://prometheus.io/port|prometheus.io/port>: "9020"

    # extraEnv:
    #- name: JAVA_TOOL_OPTIONS
    # value: >
    #   -Dpinot.broker.controller.authentication.type=Basic 
    #   -Dpinot.broker.controller.authentication.authToken=admin:rzNZhSF^Zkk@q8B%

  # Expose via Kubernetes Ingress on port 8099
  # ingress:
  #   v1:
  #     enabled: true
  #     ingressClassName: nginx          # or your ingress controller
  #     annotations:
  #       <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "HTTP"
  #       <http://nginx.ingress.kubernetes.io/use-regex|nginx.ingress.kubernetes.io/use-regex>: "true"
  #       <http://nginx.ingress.kubernetes.io/rewrite-target|nginx.ingress.kubernetes.io/rewrite-target>: /$2
  #     hosts: [<http://pinot-eks.sr-bi-internal.in|pinot-eks.sr-bi-internal.in>]
  #     path: /broker(/|$)(.*)
  #     pathType: ImplementationSpecific
  #     tls: []
  external:
    enabled: false


# ----------------------------------------------------------------------------
# PINOT SERVER: 2 replicas, each with 100 Gi gp3 PVC
# ----------------------------------------------------------------------------
server:
  name: pinot-server
  startCommand: "StartServer"
  # podManagementPolicy: Parallel
  replicaCount: 3
  
  resources:
    requests:
      cpu:    2000m    # 2 vCPU
      memory: 5Gi
    limits:
      cpu:    4000m    # 4 vCPU
      memory: 10Gi

  persistence:
    enabled: true
    accessMode: ReadWriteOnce
    size: 150G
    mountPath: /var/pinot/server/data
    storageClass: gp3
  
  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=9030:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms4G -Xmx8G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log -Djute.maxbuffer=4000000"

  service:
    name: server
    # type: LoadBalancer
    annotations:
      # <http://service.beta.kubernetes.io/aws-load-balancer-internal|service.beta.kubernetes.io/aws-load-balancer-internal>: "true"
      <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
      <http://prometheus.io/path|prometheus.io/path>: /metrics
      <http://prometheus.io/port|prometheus.io/port>: "9030"
    # labels:
    #   prometheus-monitor: "true"
    extraPorts:
      - name: server-prom
        protocol: TCP
        containerPort: 9030

  extra:
    configs: |-
      pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      pinot.server.storage.factory.s3.region=ap-south-1
      pinot.server.segment.fetcher.protocols=file,http,s3
      pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
      pinot.server.storage.factory.s3.disableAcl=false
      pinot.server.storage.factory.s3.pathStyleAccess=true
      realtime.segment.serverUploadToDeepStore=true
      pinot.set.instance.id.to.hostname=true
      pinot.server.instance.segment.store.uri=<s3://sr-datapipeline/data-platform/pinot/deep_store/pinot-engage>
      pinot.server.instance.realtime.alloc.offheap=true
      pinot.query.server.port=7321
      pinot.query.runner.port=7732

  dataDir: /var/pinot/server/data/index
  segmentTarDir: /var/pinot/server/data/segment
    
  podAnnotations:
    <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    <http://prometheus.io/path|prometheus.io/path>: /metrics
    <http://prometheus.io/port|prometheus.io/port>: "9030"

    # extraEnv:
    #- name: JAVA_TOOL_OPTIONS
    # value: >
    #   -Dpinot.server.controller.authentication.type=Basic 
    #   -Dpinot.server.controller.authentication.authToken=admin:rzNZhSF^Zkk@q8B%


# ----------------------------------------------------------------------------
# MINION: 1 replica (for background tasks / retention, compaction, etc.)
# ----------------------------------------------------------------------------
minion:
  enabled: false           # run the minion pod
  name: pinot-minion
  startCommand: "StartMinion"
  # podManagementPolicy: Parallel
  replicaCount: 1         # scale up if you have heavy compaction/merge workloads

  resources:
    requests:
      cpu:    100m
      memory: 512Mi
    limits:
      cpu:    200m
      memory: 1Gi

  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -XX:ActiveProcessorCount=2 -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-minion.log -Djute.maxbuffer=4000000"

Xiang Fu

09/19/2025, 9:43 AM

This should be good, can you check server logs for error

Xiang Fu

09/19/2025, 9:43 AM

you can also reload a segment and see exactly the errors that server prints

Rajat

09/19/2025, 9:44 AM

sure I'll check for Updating and Bad segments

Rajat

09/19/2025, 9:45 AM

this was showing updating in the previous page but when I clicked on it, it is showing online with this error: { "code": 500, "error": "Segment: Universal_cookie__0__0__20250908T0609Z does not exist in the ideal state of table: Universal_cookie_REALTIME" }

Rajat

09/19/2025, 9:47 AM

when reloading too got this error: { "code": 500, "error": "Segment: Universal_cookie__12__64__20250910T0103Z does not exist in the ideal state of table: Universal_cookie_REALTIME" }

Xiang Fu

09/19/2025, 4:37 PM

can you check table idealstates in the zookeeper

Rajat

09/20/2025, 5:12 PM

Copy code

{
  "id": "Universal_cookie_REALTIME",
  "simpleFields": {
    "BATCH_MESSAGE_MODE": "false",
    "IDEAL_STATE_MODE": "CUSTOMIZED",
    "INSTANCE_GROUP_TAG": "Universal_cookie_REALTIME",
    "MAX_PARTITIONS_PER_INSTANCE": "1",
    "NUM_PARTITIONS": "39",
    "REBALANCE_MODE": "CUSTOMIZED",
    "REPLICAS": "2",
    "STATE_MODEL_DEF_REF": "SegmentOnlineOfflineStateModel",
    "STATE_MODEL_FACTORY_NAME": "DEFAULT"
  },
  "mapFields": {
    "Universal_cookie__0__70__20250910T0607Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__0__71__20250910T0643Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__10__74__20250910T0517Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__10__75__20250910T0552Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__11__64__20250910T0616Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__12__70__20250910T0556Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__12__71__20250910T0631Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__13__75__20250910T0539Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__13__76__20250910T0612Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__14__63__20250910T0534Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__14__64__20250910T0616Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__15__71__20250910T0557Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__15__72__20250910T0632Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__16__76__20250910T0515Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__16__77__20250910T0550Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__17__63__20250910T0533Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__17__64__20250910T0617Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__18__71__20250910T0546Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__18__72__20250910T0621Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__19__74__20250910T0508Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__19__75__20250910T0544Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__1__76__20250910T0525Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__1__77__20250910T0600Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__2__65__20250910T0534Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__2__66__20250910T0614Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__3__70__20250910T0556Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__3__71__20250910T0631Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__4__75__20250910T0533Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__4__76__20250910T0608Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__5__65__20250910T0535Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__5__66__20250910T0613Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__6__70__20250910T0605Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__6__71__20250910T0641Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__7__76__20250910T0524Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__7__77__20250910T0559Z": {
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__8__64__20250910T0535Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__8__65__20250910T0613Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-1.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    },
    "Universal_cookie__9__70__20250910T0608Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "ONLINE"
    },
    "Universal_cookie__9__71__20250910T0643Z": {
      "Server_pinot-pinot-server-0.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098": "CONSUMING"
    }
  },
  "listFields": {}
}

Rajat

09/20/2025, 5:12 PM

This was in the idealState of the table

Xiang Fu

09/20/2025, 5:26 PM

hmm, weird, you have way smaller segments in IdealStates than loading.. this is very abnormal

Xiang Fu

09/20/2025, 5:26 PM

what's your table retention

Rajat

09/20/2025, 8:13 PM

It was 7 days but the data keeps on filling the server very soon so I made it 4

Xiang Fu

09/21/2025, 9:31 AM

can you check segment metadata start/end time and see if end time passed the retention

Xiang Fu

09/21/2025, 9:32 AM

I somehow feel old data are deleted due to retention

Xiang Fu

09/21/2025, 9:32 AM

only latest data are there for servers

Xiang Fu

09/21/2025, 9:32 AM

also, can you get controller/server logs(not console log, the log file inside the container at

./logs

directory ) when you see this issue again

Rajat

09/22/2025, 7:45 AM

this was the data for segment start time end time. the retention is 4 set in tableConfig. Plus the ingestion has stopped

Pinot_Data Explorer (10).csv

Xiang Fu

09/22/2025, 7:58 AM

then those segments are expected to be deleted as the end time is already passed the retention date

Rajat

09/22/2025, 8:10 AM

Untitled

Rajat

09/22/2025, 8:10 AM

what is happening here is that this issue was persistent in this cluster so I was deleting the table and ingesting them again as it was a direct kafka realtime data. but I was getting this same issue each time I was creating new table. here are some logs at the time I lost data>

Rajat

09/22/2025, 8:15 AM

I can't find logs for particular time also: bash-5.2# zgrep -H -i "2025/09/10 123[0 9].universal_cookie" pinotServer.2025-09-10..log.gz bash-5.2# zgrep -H -i "2025/09/10 12:3[0-9]" pinotServer.2025-09-10.*.log.gz these are returning blank

Xiang Fu

09/22/2025, 1:06 PM

are you deleting the table and recreate immediately? It might be issue of async table cleanup is not finished

Xiang Fu

09/22/2025, 1:07 PM

For large table suggesting recreate the table after the old table is deleted for a while, you can check disk util or zk metrics to ensure table is deleted

Xiang Fu

09/22/2025, 1:07 PM

or just create a new table with suffix like myTable_v1, myTable_v2 etc

Rajat

09/22/2025, 8:01 PM

yeah I was changing the table name as well but I guess it was just a font change plus maybe I had to have a different consumer name

Rajat

09/22/2025, 8:01 PM

I will ingest the data again and monitor the issue again

Open in Slack

Previous Next