ran out of space

yeah, for kube, better to have your own volume to mount or storageclass defined

Leaving a few questions here 1. During setup what creates the topic

flights-realtime

? 2. I imagine I can use the same

pinot-example-loader

to load my sample data into the pinot cluster? 3. The

StreamAvroIntoKafka

reads from a provided file, how are the records written to the file? I am thinking of using the same tooling to load my data 4. If I have to change my table schema after I load data, what is the recommendation? 5. I imagine once I set the

timeFieldSpec

I cannot alter it? 6. Imagine metrics coming in at a min granularity, what controls roll-up? Say I want to rollup every 15 mins, would that be

outgoingGranularitySpec

? 7. Should there always be a 1:1 mapping between schema -> table schema?

QuickStart setup starts a stand-alone Kafka cluster and we invoke Kafka admin api to create the topic.

2 and 3, yes that should work.

4. Invoke Pinot admin api to update schema

Run reload segment api. As long as the schema is backwards compatible, Pinot will generate default values for ew columns

5. Yes altering time column will be tricky. Retention and routing depends on that column

6. Yes.

7. Which schema are you referring to?

For 7 take for example

airlineStats_Schema

and the realtime table for it i.e.

airlineStats_realtime_table_config

Table config is separate from schema

But yes 1-1 mapping

Can I have multiple

tableIndexConfig

point to the same topic?

Yes

Multiple tables can consume from the same topic.

Think of this as different messages (messages have different schemas) being fed into the same topic which then feeds into the multiple tables

But one table can only consume from 1

@User I am thinking m : 1 : n where m = number of schemas and n = number of tables. Any entry in m points to exactly one entry in n

Messages on one topic should have same schema right?

That was my question

Does it have to be?

I think that’s what Kafka supports

Kafka does not dictate the schema of the message

Kafka is only the transport layer

I can have multiple consumers on Kafka each reading a different message and ignoring ones it does not understand

Yeah you can do that

Each table will have separate schema, and it will pull out the part of the message as per its schema

May not be the most efficient way of doing it but it would work

Interesting idea. Why would you do that?