Hi, After setting up the deepstore why the new rec...
# troubleshooting
r
Hi, After setting up the deepstore why the new records are not coming after this limit of 800000. Please anyone help. I am running this with helm chart in EKS. @Mayank @Xiang Fu please connect need to get it resolved. also I am not able to see the segments on S3.
also, after deepstore setup it is creating new controllers and all the older data is gone after that due to all the older segments are in local dir, and now it is searching the S3 locations. How can we migrate existing segments to S3 Why is it creating new controllers at the first place. C.C: @Kavya
m
You cannot change deepstore location on the fly. You have to ensure older data is moved to new deepstore manually
It is likely stopping ingestion because you haven’t configured deepstore correctly so it is unable to commit
r
Mayank, i have tried moving data from one cluster by pausing consumer on old cluster but segments are not loading. But here i have used HDFS to HDFS migration.
m
Well the ZK metadata may be pointing to old deepstore in the segment metadata. You need to fix that too
r
ok any steps do we have to update zk metadata as per old (source) cluster ?
m
It might be simple to just push the segments
r
Okay so you’re saying after creating s3 bucket we will have to push the older segments there and then just run the cluster again. But how will the missing data get managed and also earlier when i set up the deepstore realtime table is showing this behaviour that no new data comes after some threshold like this screenshot and in the segments none of them were online all of them were in consuming state
m
Taking a step back, deepstore config simply cannot be changed on a live cluster and assume everything works. I am not sure what changes you did and the current state of your cluster so hard to say what’s wrong in your setup
r
This is the values.yaml that I used for deployment:
Copy code
image:
  repository: <http://615177075440.dkr.ecr.ap-south-1.amazonaws.com/pinot|615177075440.dkr.ecr.ap-south-1.amazonaws.com/pinot>
  tag: 1.0.1
  pullPolicy: IfNotPresent

cluster:
  name: pinot-engage

  #pinotAuth:
  #enabled: true
  #controllerFactoryClass: org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory
  #brokerFactoryClass: org.apache.pinot.broker.broker.BasicAuthAccessControlFactory
  #configs:
  #- access.control.principals=admin,user
  #- access.control.principals.admin.password=rzNZhSF^Zkk@q8B%
  #- access.control.principals.user.password=xW)w(zKaj)s7!ytw
     # - access.control.principals.user.tables=universal_cookie
     #- access.control.principals.user.permissions=READ  

# ----------------------------------------------------------------------------
# ZOOKEEPER: 3 replicas
# ----------------------------------------------------------------------------
zookeeper:
  name: pinot-zookeeper
  replicaCount: 3
  persistence:
    enabled:      true
    storageClass: gp3      # ← GP3 EBS
    size:         10Gi
  resources:
    requests:
      cpu:    100m
      memory: 256Mi
    limits:
      cpu:    300m
      memory: 512Mi

  port: 2181

# ----------------------------------------------------------------------------
# CONTROLLER: 2 replicas, internal LB
# ----------------------------------------------------------------------------
controller:
  name: pinot-controller
  replicaCount: 2
  startCommand: "StartController"
  # podManagementPolicy: Parallel
  resources:
    requests:
      cpu:    100m
      memory: 1Gi
    limits:
      cpu:    300m
      memory: 3Gi
  
  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=9010:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -XX:ActiveProcessorCount=2 -Xms512M -Xmx2G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-controller.log -Djute.maxbuffer=4000000"
    
  # Persist controller metadata
  persistence:
    enabled:      true
    accessMode:   ReadWriteOnce
    storageClass: gp3
    size:         50Gi
    mountPath:    /var/pinot/controller/data

  service:
    name: controller
    annotations:
      <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
      <http://prometheus.io/path|prometheus.io/path>: /metrics
      <http://prometheus.io/port|prometheus.io/port>: "9010"
    # labels:
    #   prometheus-monitor: "true"
    extraPorts:
      - name: controller-prom
        protocol: TCP
        containerPort: 9010

  data:
    dir: <s3://sr-datapipeline/data-platform/pinot/deep_store/pinot-engage/>

  extra:
    configs: |-
      pinot.set.instance.id.to.hostname=true
      pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      controller.local.temp.dir=/var/pinot/controller/data/local_temp/
      controller.host=pinot-pinot-controller
      pinot.controller.storage.factory.s3.region=ap-south-1
      pinot.controller.segment.fetcher.protocols=file,http,s3
      pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
      pinot.controller.storage.factory.s3.disableAcl=false
      pinot.controller.storage.factory.s3.pathStyleAccess=true
    
    
  podAnnotations:
    <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    <http://prometheus.io/path|prometheus.io/path>: /metrics
    <http://prometheus.io/port|prometheus.io/port>: "9010"

  # Expose via Kubernetes Ingress on port 9000
  # ingress:
  #   v1:
  #     enabled: true
  #     ingressClassName: nginx          # or your ingress controller
  #     annotations:
  #       <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "HTTP"
  #       <http://nginx.ingress.kubernetes.io/rewrite-target|nginx.ingress.kubernetes.io/rewrite-target>: /
  #     hosts: [<http://pinot-eks.sr-bi-internal.in|pinot-eks.sr-bi-internal.in>]
  #     path: /controller
  #     tls: []
  external:
    enabled: false

# ----------------------------------------------------------------------------
# BROKER: 2 replicas, internal LB
# ----------------------------------------------------------------------------
broker:
  name: pinot-broker
  startCommand: "StartBroker"
  # podManagementPolicy: Parallel
  replicaCount: 2

  resources:
    requests:
      cpu:    200m
      memory: 1Gi
    limits:
      cpu:    500m
      memory: 3Gi

  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=9020:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -XX:ActiveProcessorCount=2 -Xms512M -Xmx2G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-broker.log -Djute.maxbuffer=4000000"

  service:
    name: broker
    #type: LoadBalancer
    annotations:
      # <http://service.beta.kubernetes.io/aws-load-balancer-internal|service.beta.kubernetes.io/aws-load-balancer-internal>: "true"
      <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
      <http://prometheus.io/path|prometheus.io/path>: /metrics
      <http://prometheus.io/port|prometheus.io/port>: "9020"
    # labels:
    #   prometheus-monitor: "true"
    extraPorts:
      - name: broker-prom
        protocol: TCP
        containerPort: 9020
    
  podAnnotations:
    <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    <http://prometheus.io/path|prometheus.io/path>: /metrics
    <http://prometheus.io/port|prometheus.io/port>: "9020"

  external:
    enabled: false


# ----------------------------------------------------------------------------
# PINOT SERVER: 2 replicas, each with 100 Gi gp3 PVC
# ----------------------------------------------------------------------------
server:
  name: pinot-server
  startCommand: "StartServer"
  # podManagementPolicy: Parallel
  replicaCount: 3
  
  resources:
    requests:
      cpu:    2000m    # 2 vCPU
      memory: 5Gi
    limits:
      cpu:    4000m    # 4 vCPU
      memory: 10Gi

  persistence:
    enabled: true
    accessMode: ReadWriteOnce
    size: 150G
    mountPath: /var/pinot/server/data
    storageClass: gp3
  
  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=9030:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms4G -Xmx8G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log -Djute.maxbuffer=4000000"

  service:
    name: server
    # type: LoadBalancer
    annotations:
      # <http://service.beta.kubernetes.io/aws-load-balancer-internal|service.beta.kubernetes.io/aws-load-balancer-internal>: "true"
      <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
      <http://prometheus.io/path|prometheus.io/path>: /metrics
      <http://prometheus.io/port|prometheus.io/port>: "9030"
    # labels:
    #   prometheus-monitor: "true"
    extraPorts:
      - name: server-prom
        protocol: TCP
        containerPort: 9030

  extra:
    configs: |-
      pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      pinot.server.storage.factory.s3.region=ap-south-1
      pinot.server.instance.segmentTarDir=<s3://sr-datapipeline/data-platform/pinot/deep_store/pinot-engage/>
      pinot.server.segment.fetcher.protocols=file,http,s3
      pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
      pinot.server.storage.factory.s3.disableAcl=false
      pinot.server.storage.factory.s3.pathStyleAccess=true
    
  podAnnotations:
    <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    <http://prometheus.io/path|prometheus.io/path>: /metrics
    <http://prometheus.io/port|prometheus.io/port>: "9030"

    # extraEnv:
    #- name: JAVA_TOOL_OPTIONS
    # value: >
    #   -Dpinot.server.controller.authentication.type=Basic 
    #   -Dpinot.server.controller.authentication.authToken=admin:rzNZhSF^Zkk@q8B%


# ----------------------------------------------------------------------------
# MINION: 1 replica (for background tasks / retention, compaction, etc.)
# ----------------------------------------------------------------------------
minion:
  enabled: false           # run the minion pod
  name: pinot-minion
  startCommand: "StartMinion"
  # podManagementPolicy: Parallel
  replicaCount: 1         # scale up if you have heavy compaction/merge workloads

  resources:
    requests:
      cpu:    100m
      memory: 512Mi
    limits:
      cpu:    200m
      memory: 1Gi

  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -XX:ActiveProcessorCount=2 -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-minion.log -Djute.maxbuffer=4000000"
Here I used whatever was in the documentation.
m
That isn’t my point. If you have a live cluster you can’t simply change the deepstore configs on the fly
x
is this realtime table or offline table ?
r
Realtime table
x
ok, then check if there is any CONSUMING segments
if no, then try to pause ingestion for the table , then resume the ingestion for the table
r
this happened after deepstore setup and after that when added this new table, it got stuck after reaching this limit. All the segments were in Consuming State none was in Online state yet. I waited for 1-2 hours still none of them were in Online state and thus were not in the S3 deepstore.
x
ok, did you configure deep store for servers ? will server be able to commit segments to deep store ?
r
yeah that I checked now in the server.conf that was deployed I see two dataDir: .... and segmentTarDir: ..... Fixing the values.yaml and checking again