https://pinot.apache.org/ logo
Join SlackCommunities
Powered by
# troubleshooting
  • b
    Somehow ingestion from kafka stopped in a RELATIME table...
  • b
    getting this error in logs
  • d
    Folks, I have a sample data set like this: 
    "9/1/2014 6:04:00",40.7513,-73.935,"B02512"
    "9/1/2014 6:08:00",40.7291,-73.9813,"B02512"
    "9/1/2014 6:14:00",40.7674,-73.9841,"B02512"
    Time is in minute granularity throughout the data set. So I mapped the time column like this in my schema file: 
    "dateTimeFieldSpecs": [{
    "name": "pickupTime",
    "dataType": "STRING",
    "format" : "1:MINUTES:SIMPLE_DATE_FORMAT:MM/dd/yyyy HH:mm:ss",
    "granularity": "1:MINUTES"
    }
    And then in the table configuration: 
    "segmentsConfig" : {
    "timeColumnName": "pickupTime",
    "timeType": "MINUTES",
    "replication" : "1",
    "schemaName" : "pickups"
    },
    Hope this is fine?
  • m
    Hi Team, We are running a hybrid table setup in our Pinot cluster. We have configured task to move data from RealTimeToOffline table. However, we are not seeing any data being moved from Realtime to Offline table. On checking the controller logs, I see the below errors.
    Copy code
    2021-10-13 07:41:57.360 ERROR [ZkBaseDataAccessor] [grizzly-http-server-4] paths is null or empty
2021-10-13 07:41:58.956 ERROR [ZkBaseDataAccessor] [grizzly-http-server-17] paths is null or empty
2021-10-13 00:06:19.325 ERROR [JobDispatcher] [HelixController-pipeline-task-PinotCluster-(275fe39b_TASK)] Job configuration is NULL for TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1633995887529
  • a
    ]
  • s
    eieecckbhjerbubnbdhjrtilhuritcduvnirujijhfbf
  • g
    Hello - Has anyone tried to connect to Pinot from Tableau using JDBC driver?
  • e
    More importantly, happy Friday everyone! We are loving pinot 0.8.0:)
    🎉 1
  • e
    If we find a fix we will update the thread also...
  • a
    Hey everyone, i have a few questions regarding to deep storage. •Is it possible to use Linux filesystem as deep storage? If so how can i configure it? •What is actually stored in the folder controller.dir.data property value pointing at? •Is Peer Download functionality still supported in version 0.7.1? I would appreciate it if you could share your knowledge with me
  • a
    I am running a cluster with HDFS as deep store. I can see that segment are pushed to hdfs and there are no error in debug endpoint but when i am running a simple query , it hangs and there is exception on server side. Attached is the stack trace.could you help ?
    pinot_error.txt
  • o
    @Orbit has left the channel
  • k
    Hi Team, I am using version 0.8.0. We are in the process of testing related to UPSERT. I am testing upsert by creating a table with the following config: In a development environment with replicas = 1, partition = 1, broker = 1, and server = 1, id-based upsert worked fine as needed. However, in an operating environment with replicas = 2, partition = 3, broker = 6, server = 6 One id is upsert to 3 rows. Any idea why it happens? -- Upsert Schema
    Copy code
    {
  "schemaName": "upsert_test",
  "primaryKeyColumns": [
    "id"
  ],
  "dimensionFieldSpecs": [
    {
      "name": "id",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "dt1",
      "dataType": "DOUBLE",
      "transformFunction": "JSONPATHDOUBLE(\"dt\", '$.1', -999.0)"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "ts_asia_seoul_datetime",
      "transformFunction": "toDateTime((ts*1000)+(timezoneHour('Asia/Seoul')*3600000), 'yyyy-MM-dd HH:mm:ss')",
      "dataType": "STRING",
      "format": "1:SECONDS:EPOCH",
      "granularity": "1:SECONDS"
    },
    {
      "name": "ts_utc",
      "transformFunction": "ts*1000",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]
}
    -- FULL Upsert TABLE
    Copy code
    {
  "tableName": "upsert_test",
  "tableType": "REALTIME",
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "ingestionConfig": {
    "filterConfig": {
      "filterFunction": "Groovy({ts < 1000000000}, ts)"
    }
  },
  "segmentsConfig": {
    "schemaName": "upsert_test",
    "timeColumnName": "ts_utc",
    "timeType": "DAYS",
    "replicasPerPartition": "2",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "5",
    "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "invertedIndexColumns": [
      "id"
    ],
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.topic.name": "monitor_1",
      "stream.kafka.broker.list": "upsert_test-0.kafka-headless.prod.svc.cluster.local:9092,upsert_test-1.kafka-headless.prod.svc.cluster.local:9092,upsert_test-2.kafka-headless.iotops-prod.svc.cluster.local:9092",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.desired.size": "450M"
    }
  },
  "metadata": {
    "customConfigs": {}
  },
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  },
  "upsertConfig": {
    "mode": "FULL",
    "comparisonColumn": "ts_utc"
  }
}
  • t
    image.png
  • a
    Just found some config in broker which is equal to this number 2147483647
  • r
    that number is 
    Integer.MAX_VALUE
  • m
    Can you try 0.9.0 in the meantime? https://downloads.apache.org/pinot/apache-pinot-0.9.0/apache-pinot-0.9.0-bin.tar.gz Will ask @Xiang Fu what's happened to those other ones when he wakes up!
    ❤️ 1
  • d
    Hi folks! First of all, congratulations on the work on 0.9.0! However, bumping up from 0.8.0 in my docker compose image references makes Pinot not work anymore for me when running it locally on my computer. For reference, here's my compose file:
    Copy code
    version: "3.3"

services:
  zookeeper:
    image: zookeeper:latest
    container_name: zookeeper
    hostname: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    restart: unless-stopped
    ports:
      - "2181:2181"
    volumes:
      - ./pinot-volumes/zookeeper/data:/data
      - ./pinot-volumes/zookeeper/datalog:/datalog

  kafka:
    image: wurstmeister/kafka:latest
    container_name: kafka
    restart: unless-stopped
    depends_on:
    - zookeeper
    ports:
    - 9094:9094
    environment:
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      # Topic will have 10 partitions and 1 replica
      KAFKA_CREATE_TOPICS: "bb8_api_logs:10:1,bb8_analyses_logs:10:1,bb8_search_logs:10:1,bb8_visits_logs:10:1"
      # Other configs
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,<OUTSIDE://127.0.0.1:9094>
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    volumes:
      - ./pinot-volumes/kafka/data:/var/lib/kafka/data

  pinot-controller:
    image: apachepinot/pinot:0.9.0
    container_name: pinot-controller
    restart: unless-stopped
    ports:
    - "9000:9000"
    - "8888"
    command: StartController -zkAddress zookeeper:2181
    volumes:
      - ./config:/config
      - ./sensitive-data:/sensitive-data
      - ./pinot-volumes/controller:/tmp/data/controller
    depends_on:
      - zookeeper
      - kafka

  pinot-broker:
    image: apachepinot/pinot:0.9.0
    container_name: pinot-broker
    restart: unless-stopped
    ports:
    - "8099:8099"
    - "8888"
    command: StartBroker -zkAddress zookeeper:2181
    volumes:
      - ./config:/config
      - ./sensitive-data:/sensitive-data
    depends_on:
      - zookeeper
      - kafka
      - pinot-controller

  pinot-server:
    image: apachepinot/pinot:0.9.0
    container_name: pinot-server
    restart: unless-stopped
    ports:
    - "8098:8098"
    - "8888"
    command: StartServer -zkAddress zookeeper:2181
    volumes:
      - ./config:/config
      - ./sensitive-data:/sensitive-data
      - ./pinot-volumes/server:/tmp/data/server
    depends_on:
      - zookeeper
      - kafka
      - pinot-controller
    do you see any problem in the above?
  • d
    @Trust Okoroego seems to be having the same problem as me, right Trust?
  • t
    @Diogo Baeder Yes, I face the issue too. My docker compose file is similar to yours.
  • m
    what's the error?
  • m
    is it that it shuts down each component as soon as they start up?
  • m
    I am working around that by overriding the env var:
    Copy code
    environment:
      JAVA_OPTS: "-Dpinot.admin.system.exit=false"
  • m
    Copy code
    version: '3.7'
services:
  zookeeper:
    image: zookeeper:3.5.6
    hostname: zookeeper
    container_name: manual-zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  pinot-controller:
    image: apachepinot/pinot:0.9.0-SNAPSHOT-d1606cd0f-20211116-jdk11
    command: "StartController -zkAddress manual-zookeeper:2181"
    container_name: "manual-pinot-controller"
    volumes:
      - ./config:/config
      - ./data:/data
    ports:
      - "9000:9000"
    environment:
      JAVA_OPTS: "-Dpinot.admin.system.exit=false"
    depends_on:
      - zookeeper
  pinot-broker:
    image: apachepinot/pinot:0.9.0-SNAPSHOT-d1606cd0f-20211116-jdk11
    command: "StartBroker -zkAddress manual-zookeeper:2181"
    container_name: "manual-pinot-broker"
    volumes:
      - ./config:/config
      - ./data:/data
    ports:
      - "8099:8099"
    environment:
      JAVA_OPTS: "-Dpinot.admin.system.exit=false"
    depends_on:
      - pinot-controller
  pinot-server:
    image: apachepinot/pinot:0.9.0-SNAPSHOT-d1606cd0f-20211116-jdk11
    command: "StartServer -zkAddress manual-zookeeper:2181"
    container_name: "manual-pinot-server"
    volumes:
      - ./config:/config
      - ./data:/data
    restart: on-failure
    environment:
      JAVA_OPTS: "-Dpinot.admin.system.exit=false"
    depends_on:
      - pinot-broker
  • d
    Interesting, I'll try that, thanks!
  • m
    btw how does the 
    restart: unless-stopped
    on your broker/server work
  • m
    is that to work around the issue where the controller might not have yet written metadata to ZK before the broker/server startup?
  • m
  • m
    oh cool! That's good 😄
  • m
    that env var was always there, it just started defaulting to 'true' since 0.9.0
1...155156157...166Latest