hi folks 🙂 Good to be here! I’d really love to learn more about the capabilities of the star-tree index. In particular, I’m curious to know how it might enable ingesting raw data and creating “materialized views” for specific use cases later once they are known. This might be considered an anti-pattern for Pinot, and if so that would be good to know too 👍

Currently, starTree index is created based on user configs; https://docs.pinot.apache.org/basics/indexing/star-tree-index

Hi Evan! It's definitely not an anti-pattern. You've come to the right place. Creating materialized views by transforming raw data sources can be done using a variety of techniques during ingestion. Specifically, transform functions can mutate your raw data using Groovy scripts. We have a variety of other transform capabilities worth checking out in the docs. For star-tree index, that's a good way to speed up your query responses once you've figured out what your materialized view is going to look like.

@User that’s interesting to know RE Groovy scripts. Thinking about it from a traditional RDBMS perspective and using a classic “blog posts” example, would it make sense to ingest

posts

,

comments

,

likes

etc as distinct data sets, then later make use of Groovy scripts to effectively join those data sets at ingestion time (rather than search time)?

There's multiple strategies here. If you're streaming your data to a real-time table using Kafka, you have two potential approaches. You can implement your messaging at the application-level, which would send your domain objects as a payload to three separate topics. At that point, you will need to use something like Flink to join those three streams together into a single materialized view (rather simple in practice). The joined materialized view is then sent to a view topic that is ingested into Pinot and ready for query.

ok ya, that makes sense using an ETL platform to do the work of joining data sets. I suppose my concern is more in the area of product evolution and the maintenance that would go along with it. For example with an RDBMS, maybe a new table is added later on resulting in a new dimension for

posts

. To then start making use of the new dimension, the new table can be joined; it might not be very efficient but it can start answering questions about the data right away. And testing/local development can be done relatively cheaply by inserting data and joining at query time. What does the developer workflow look like to support the same kind of feature evolution in Pinot? It seems complex to mimic an ETL pipeline for local development, for example

Just double checking with Mayank to make sure I answer this right. One sec.

Yep, all makes sense 👍 Thanks for sticking with me.  I wonder if you can comment on another area of developer experience. As a developer who would be building dashboards consuming the data (ie making queries), making use of Postgres (just as a concrete example RDBMS) for local development is pretty feasible: spin up a Docker container of Postgres, apply migrations as applicable, insert some data, and start making queries.  With what’s described above, local development sounds a bit heavy. One would need all the described components running locally in order to test changes and develop new features. Is there a trick to that, or established pattern for local development with an analytics DB which requires ETL? Maybe just synthesizing data locally that adheres to the expected format which the ETL job will produce/input to the DB? Thanks again for all your time 🙏

I got caught up today in some other things. Let me get back to you by tomorrow morning. Thanks Evan!

Thanks for all the answers here! I still need to digest the content 🙏 Thanks for all your time here

👍