Thanks in advance!

@Carlos Yes, ingesting realtime + offline data to pinot and use it as the backend storage for Superset is a common usage pattern for Pinot. We have a documentation around Superset-Pinot https://docs.pinot.apache.org/integrations/superset

Hi @Seunghyun, thanks for your response. I have succesfully connected Superset to Pinot. My problem is knowing if it is posible to dedupe data in offline tables.

I want to ingest data from an AVRO-encoded Pulsar topic into a Pinot REALTIME table. Is that currently possible? If so, how can I do it?

Hi, How can I pass basic auth credentials to INSERT INTO statement. My Pinot controller has basic auth enabled. I tried below and got an error:

INSERT INTO "orders" FROM FILE '<s3://xxxxxx/>'
OPTION(
  taskName=myTask-s3,
  input.fs.className=org.apache.pinot.plugin.filesystem.S3PinotFS,
  input.fs.prop.accessKey=xxxxxxx,
  input.fs.prop.secretKey=xxxxxxxxx,
  authToken='Basic xxxxxx=='
  input.fs.prop.region=us-east-1
)

I got bellow error message

ProcessingException(errorCode:450, message:InternalError:
org.apache.pinot.sql.parsers.SqlCompilationException: Caught exception while parsing query: INSERT INTO "orders"
FROM FILE '<s3://xxxxxxxx>'
	at org.apache.pinot.sql.parsers.CalciteSqlParser.compileToSqlNodeAndOptions(CalciteSqlParser.java:136)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:135)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:103)
...
Caused by: org.apache.pinot.sql.parsers.SqlCompilationException: OPTION statement requires two parts separated by '='
	at org.apache.pinot.sql.parsers.CalciteSqlParser.extractOptionsMap(CalciteSqlParser.java:486)
	at org.apache.pinot.sql.parsers.CalciteSqlParser.compileToSqlNodeAndOptions(CalciteSqlParser.java:131)
	... 27 more)

removing the authToken line I get below error

[
  {
    "message": "QueryExecutionError:\norg.apache.commons.httpclient.HttpException: Unable to get tasks states map. Error code 400, Error message: {\"code\":400,\"error\":\"No task is generated for table: orders, with task type: SegmentGenerationAndPushTask\"}\n\tat org.apache.pinot.common.minion.MinionClient.executeTask(MinionClient.java:123)\n\tat org.apache.pinot.core.query.executor.sql.SqlQueryExecutor.executeDMLStatement(SqlQueryExecutor.java:102)\n\tat org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:145)\n\tat org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:103)",
    "errorCode": 200
  }
]

Hello, I am a newbie to pinot! referencing this doc: https://docs.pinot.apache.org/users/tutorials/schema-evolution

Pinot only allows adding new columns to the schema. In order to drop a column, change the column name or data type, a new table has to be created.

Is this applicable only for realtime tables (or) offline tables as well? Are there any plans to support renaming of a column? Any reasons why renaming is not supported?

Hello All, Need some help on theoretical perspective. I need to know where pinot store the data, it's like since pinot keeps very new data inside it self and after some time it push it back to some where else. What I need to know this other DB where old data remains . Can I use postgreSQL for that one

Hi All,

Im new to pinot and trying to connect pinot connector with trino and work there. So i just run a quickstart in pinot and connect it with trino. Then try to run a simple select query but it give errors. Is there any limitaions in this or wht?..

Anyone knows a quick (scripted) way to convert mysql ddl into pinot schema.json?

Hi, I am currently testing pinot and am not able to create realtime and offline table with same name. I want to injest data from kafka in the realtime table and let pinot handles creating and offloading segments to offline table. Here is the error I get

2023/02/09 13:00:24.206 INFO [AddTableCommand] [main] Executing command: AddTable -tableConfigFile null -offlineTableConfigFile /var/pinot/data/epoch_table_offline_table_config.json -realtimeTableConfigFilenull -schemaFile /var/pinot/data/epoch_table_schema.json -controllerProtocol http -controllerHost pinot-controller -controllerPort 9000 -user null -password [hidden] -exec

2023/02/09 13:00:24.514 INFO [AddTableCommand] [main] {"code":400,"error":"TableConfigs: epoch_table already exists. Use PUT to update existing config"}

You need to call AddTable with the

-update

flag

when you add the second table

Hi and Happy Friday to y'all! I have a question about Pinot and Zookeeper. We already have a Zookeeper (and Kafka) deployed in production. Do we need a second Zookeeper to handle Pinot separately or can we point Pinot to the existing one? Perhaps the answer is obvious, since the name Zookeeper would imply having more than one animal to attend to! 😄

Is there a way to use minion based ingestion to replace single segments? like what I'm trying to is: • let's say a segment (

my-table_0_10_0

) points at parquet file in s3

<s3://my-bucket/data/part-0000-abc.parquet>

• I'd like to replace that segment; I have REST API access to everything, but can't exec into my controller or anything like that. • My table has, let's say 100 segments • If I just replace that file and rerun minion based ingestion via my tables

SegmentGenerationAndPush

task, it's going to kick off 100 tasks, which I don't really want to do, since that'll take a while and I just want my one segment So is there a way to kick off only the subtask for that single file/segment?

Any idea how to configure the pinot-controller kubernetes manifest and/or the pinot-controller.conf file for connecting to zookeeper when zookeeper requires SASL authentication? The pinot documentation talks about setting "controller.zk.str", but nothing about how to set zookeeper auth as sasl and specifying zookeeper userid and password

Hey Everyone! I'm a beginner and using pinot for the first time! Can anyone help me with how to run pinot with kafka on docker-compose? just a basic tutorial maybe!will be much helpful!

We have a 'Getting Started' document on that topic at:

<https://dev.startree.ai/docs/pinot/getting-started/ingest-streaming-data-source>

Hey everyone! Hope you are doing well! Kindly recommend me an article on the working flow and architecture of apache pinot!

Newbie question 🙂 : How does using Pinot compare with a situation say using a "snowflake warehouse with connectors" for real time analytics? One obvious difference is being open source. Are there any more advantages (like operational simplicity, cost, etc?)

Hi, was wondering if changing the config :

realtime.segment.flush.threshold.rows

from 100000 to 10000 and then refreshing segments would also refresh the already committed segments i.e. segments would have 10000 entries now? Please let me know

Hi, We are in the process of setting up ETL pipelines. The processed data will be stored on S3 in parquet format using datetime column for partitioning. The idea is to connect pinot offline table with S3 and let minions handle SegmentGenerationAndPush tasks. Can pinot handle parquet files with snappy/lz4 compression? What about dictionary encoding? I do not see any documentation on how to add custom config for ParquetRecordReader here https://docs.pinot.apache.org/basics/data-import/pinot-input-formats Could someone point me to the right place to read more about it? Or any tips for me in general on how to efficiently store data on S3 using parquet format? Thanks!

Hi, I am trying to deploy quickstart pinot helmchart on kubernetes cluster and getting connection timeout from Controller

Hi team, been exploring apache pinot for the first time. I'm unable to make the filter function work on pinot tables consuming events from kafka. I wanted to filter events based on event_names field in each kafka event. I get the below error and I tried setting up the Groovy field in controller.conf file, still no luck. org.apache.pinot.segment.loca^Cjava.lang.RuntimeException: Caught exception while executing filter function: Caused by: java.lang.NumberFormatException: For input string: "{event_name}" Any help would be appreciated.

I’ve only tried it out for a basic example here - https://dev.startree.ai/docs/pinot/recipes/filtering-ingestion Can you give a bit more info on what your events look like & what the filter fn looks like too?

Hello! If I call

INSERT INTO

command in PQL for batch data import, what defines the degree of parallelism in this case? Number of minions' processors?

Hey Folks 👋 I am looking at our test setup and feeling in bit of dire straits. Currently we develop under Windows and build on Linux servers. During the build we start required environment locally (mainly Zookeeper, Kafka and Flink), run tests against it and then tear it down at the end of the build. I'm trying to figure out what is the best way to incorporate Apache Pinot into this setup. Has anyone got experience or can recommend best practices for testing against Apache Pinot under Windows? Is it possible to run it locally under MS Windows 10 or start it directly from within JVM process?

hi team, we have a realtime pinot table configured with ingestion-aggregation. can we also setup start-tree index on this table?

Hi, all. Is it possible to change the Key of JSON Field to lower case? For example,

{
  "columnName": "converted_json",
  "transformFunction": "map_keys(raw_json, x -> lower(x))"
}

Hi everybody