Sorry to revive a very old thread... @Allen Li is this still the case? From our testing, it seems to be true with routine load and kafka connector https://starrocks.slack.com/archives/C02FACZSNJV/p1762556208094909?thread_ts=1760900809.319769&cid=C02FACZSNJV. Any ideas on how we can work around this or support it? @Ivan Georgiev did you figure out a way around this? We have the exact same use case as you - 1000 columns with a kafka topic updating only a few columns at a time, and are pretty stuck at the moment

@Allen Li facing the same issue

Hey! Yeah this is still a limitation with Routine Load - you have to specify the column list upfront and it applies to every batch. For your use case with different columns updating each time, Routine Load isn't going to work unfortunately. Your best bet is probably Stream Load with partial update mode instead of Routine Load. You can send individual payloads with just the columns you're updating plus the primary key, and StarRocks will merge them properly without nulling out the other columns. You'd need to handle the streaming consumption yourself (read from Kafka, batch up records, send via Stream Load), but it gives you the flexibility to have different column sets per request. The other option is using the Flink connector if you're in that ecosystem already - it handles partial updates better and you can dynamically specify which columns to update per record. Bit more infrastructure to manage though. For the versioning concern with lots of small updates, yeah you'll create more versions but compaction should handle it as long as you're not absolutely hammering it. Keep an eye on the tablet version count and compaction metrics. If it gets out of hand you might need to batch your updates a bit more or tune the compaction settings. But honestly for most workloads it's fine, the compaction process is pretty good at keeping up.