https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • n
    Hi team, one of the requirements for supporting stream ingestion w/ upsert is to partition the input stream by the primary key. What if the input stream partition key (e.g. company id) is different from the record primary key (e.g. employee id)?
    t
    k
    +3
    • 6
    • 15
  • p
    Is there a tool that helps us visualise the schema of Pinot OLAP Tables (across all tables)? Since there is no joins support, we need to have data redundancy (same field in multiple tables). And I want to see all fields and their relationships (redundant fields) across all tables. I am looking something similar to ER diagram that postgres allows us to make.
    m
    • 2
    • 1
  • c
    Hi. Team. https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations In my tests, I had to use the groovy syntax used in the 'injection transformation filterFunction' differently for the realtime table and the offline table. I don't know if this is a bug or wrong use case or the correct case, but I do report it as it seems confusing to users. • 'score' field's datatype is float. • 'filterFunction" : "Groovy({score >= 4 && score < 6}, score)" REALTIME working, OFFLINE not working(casting exception, creationAndPush injestion job) • 'filterFunction" : "Groovy({(score as float) >= 4 && (score as float) < 6}, score)" OFFLINE working
    m
    • 2
    • 4
  • f
    Hi amazing pinot team. If I read well the docs there is a mecanism for GDPR compliance. Have you any example to share arround here. For eg I’ve customer_Id I want to purge. How can I do that ? And what happen for the curent consumming segment ? Thx for your help 😉
    m
    j
    v
    • 4
    • 36
  • p
    Hi Pinot folks 🙂 I am getting some segments of my table as unavailable (error code 305):
    Copy code
    [
  {
    "message": "7 segments...unavailable",
    "errorCode": 305
  }
]
    I am okay to let go of these segments to query my table. How can I do that? Or anything I can do to bring these back up? I tried reloading but doesn’t help. Also, this is an REALTIME table.
    m
    t
    • 3
    • 6
  • c
    I'm looking for best practices for choosing hardware (cpu, memory, storage). This is the only documentation I found. https://docs.pinot.apache.org/basics/getting-started/frequent-questions/operations-faq#how-much-heap-should-i-allocate-for-my-pinot-instances Of course I know it depends on my workload, but can someone give me a general hint or experience? Anything is fine.
    m
    • 2
    • 6
  • a
    pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars
    m
    • 2
    • 1
  • a
    for server node, the above properties are used to config local attached disk, right?
    m
    • 2
    • 8
  • a
    pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.server.storage.factory.s3.region=us-west-2 pinot.server.segment.fetcher.protocols=file,http,s3 pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
    m
    • 2
    • 3
  • a
    By default, no retention is set. In this case, the data in the server local storage will grow increasingly? And if so, what happens if there is not enough space for local storage?
    m
    • 2
    • 1
  • s
    Hi Folks, I'm trying to load test Apache pinot with jmeter, From where can I download the PINOT JDBC driver jar ? I have included the pinot-jdbc-client-0.9.3.jar in the JMETER path however it's resulting in internal dependency related errors ... Is there a PINOT JDBC Driver jar that i could use directly ?
    m
    k
    • 3
    • 5
  • a
    Hi, I’m a little confused about the instance name. If i specify a instanceId for a server, take pinot-server for example, the instance name will be Server_pinot-server_port. So what’s the best practice for setting up a pinot cluster with 3 or more servers using Docker or Kubernetes? Should i set a unique instanceId for each one of them or maybe there is a better way?
    m
    • 2
    • 2
  • a
    My question is, could every single one of the properties in a config file be set in the above way?
    m
    • 2
    • 1
  • s
    Does that mean , I cannot use that index too if I want upsert to be configured for a table?
    m
    • 2
    • 3
  • s
    I got one more question regarding lookup tables. I have created an offline dimension only table with 3 records
    m
    • 2
    • 37
  • a
    I met the following error when i query EXPLAIN PLAN FOR select * from streams_metrics_flat_data, any idea of the cause for it?
    r
    a
    • 3
    • 8
  • a
    I created a star tree index for a realtime table, how can I verify if it works or not? and there is any tool to view this index data?
    m
    • 2
    • 7
  • s
    Copy code
    Hello Community,

We are pleased to announce that Apache Pinot 0.10.0 is released!
The release can be downloaded at <https://pinot.apache.org/download>
The release note is available at <https://docs.pinot.apache.org/basics/releases/0.10.0>

Additional resources -
Project website: <https://pinot.apache.org>
Getting started: <https://docs.pinot.apache.org/getting-started>
Pinot developer blogs: <https://medium.com/apache-pinot-developer-blog>
Intro to Pinot Video: <https://www.youtube.com/watch?v=T70jTTYhYyM>

Best Regards,
Apache Pinot Team
    🙌 4
    🎉 24
    d
    m
    +4
    • 7
    • 14
  • m
    Anyone faced following error ? Once this happens I have to restart Pinot server and the data during these errors are not consumed by realtime table.
    Copy code
    16:20:24.624 SegmentColumnarIndexCreator - Caught exception org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed while refreshing realtime lucene reader for segment: mydata__1__25531__20220323T2114Z
    t
    s
    • 3
    • 6
  • e
    Question about tiered storage+deep storage. We want to use pinot to query mostly fresh data (last week or so) and then store data long term on s3. We have deployed pinot in our k8s cluster using the helmchart and configuring the controller to backup to s3 seems to work. If I understand it correctly this is only to backup data which is also stored on local disks however and we want to keep the local disk usage low! Can we use tiered storage and configure one server to use s3 and another server to use local disks?
    👀 1
    k
    n
    • 3
    • 6
  • a
    when I run an explain plan for query multiply times, I got different query results. And the totalDocs number is different. A very simple query like explain plan for select * from table_name. Any idea why is that?
    r
    m
    +2
    • 5
    • 14
  • a
    Hi, I’ve a question. What’s the name of consumer group in kafka when pinot ingests the kafka stream data?
    m
    • 2
    • 1
  • a
    Hi, I am going to setup apache-pinot for production on AWS EKS and I want to setup multi AZ setup So how would I manage the data because ebs volume support only in single AZ. Can we use efs ? Please suggest Setup should be scaleable and fault tolerance Thanks
    m
    d
    • 3
    • 6
  • a
    If I want to run pinot-admin.sh StartController command using args instead of using a config file. How can I set properties like controller.host, controller.data.dir, etc?
    k
    • 2
    • 6
  • d
    Is there something similar to 
    RealtimeToOfflineTask
    for an upsert table?
    m
    y
    +2
    • 5
    • 10
  • a
    Hi team, I’am a little confused. My table stopped consuming data from a kafka topic. But a new table with the same schema and table can ingest data from the same topic. There’s new data consistently written to this Kafka topic. Any idea why this is happening? I’ve tried and restarted controller, broker, and server nodes, but it didn’t help.
    m
    • 2
    • 13
  • e
    We have configured our server to use s3 deepstore. Does anyone know where the documentation on how to restore a server from the deepstore backup is located?
    k
    • 2
    • 5
  • e
    Also: does anyone know if using NFS as storage with works well with pinot (as a secondary storage tier, not for real time writes).
    k
    • 2
    • 1
  • t
    Hello, what would be impact on storage footprint if we set maxLength of string column(SV) to 1MB?
    r
    • 2
    • 12
  • d
    Hey there 👋 Is there a way to change the log level for the instances?
    m
    • 2
    • 2
1...373839...160Latest