Hello, I am new in Pinot. I have a question about modeling tables. I have one real-time table and one dimension table, the dimension table is larger than 200MB.. Is there a solution/best practice how to handle lookup\joins with dimension tables that are larger than 200MB? thanks 🙏

Hi Jonathan, welcome! This would be a better question for #CDRCA57FC rather than #C013WKLT5T7. But let me answer briefly: You can create dim tables larger than 200MB. To change the size limit you can use the configuration:

controller.dimTable.maxSize

But mind you, dim tables are directly loaded into heap, so make sure you are not running out of memory. If you need larger dim tables that won’t fit into memory, you can try the newly introduced off-heap dim tables. Add this to your dim table configuration:

"dimensionTableConfig": {
            "disablePreload": true
        },

Disclaimer, I haven’t used offheap dim tables in production

@balci thanks! I will try there also