Disregard, I think we might be dumb, we should be using replication not replicasPerPartition b/c we aren't using partitioning

Hmm, I'm actually still getting this error and we aren't using partitioning

Assigning segment: immutable_events_1657670400000_1657670400000_0 with instance partitions: {"instancePartitionsName":"immutable_events_OFFLINE","partitionToInstancesMap":{"0_0":["Server_ica-pinot-server-immutable-offline-0.ica-pinot-server-immutable-offline-headless.default.svc.cluster.local_8098"]}} for table: immutable_events_OFFLINE Caught exception while updating ideal state for resource: immutable_events_OFFLINE _java.lang.IllegalStateException_: There are less instances: 1 in instance partitions: immutable_events_OFFLINE than the table replication: 2

Our issue here was local testing and we only had 1 of each server, 1 realtime and 1 offline. When the offline rollup job ran it failed b/c it could only fine one serer, but gave us an error message that didn't exactly explain that. What's weird is w/ replication 2 on the realtime table w/ 1 realtime server that worked fine. It was only when pushing to offline that we failed.

So our thought based on looking at the segments is that realtime segments don't get replicated at all. We set replication to 2 for our realtime table and they only got 2 segments when pushing to offline.

Is that true?

Our findings seem to indicate that replication is the field name for offline and replicasPerPartition for realtime are what is needed. That definitely feels like a bug unless I'm not understanding things @Mayank / @Neha Pawar

@Stuart Millholland you can get the history in the issue